All Apps and Add-ons

Custom email parsing

jjpeet
New Member

Hi,
The only way I can get log data out of my home router is via email, so I'm using Splunk for iMAP to achieve this.
I'm only interested in the body of the emails which contains log data, and want them to be separated out as separated records in Splunk.

At the moment, I'm getting emails which look like what I've included below. How might I process these a little further to get just the log data in the body (and hopefully each line of the log data is a separate record).

Thanks
James

One email as it appears in Splunk for IMAP
Date = "12-Sep-2015 07:01:04 +0000"
DATE = "12 Sep 15 17:01:04"
FROM = "<logs@jjpeet.com>"
To = "<logs@jjpeet.com>"
Subject = "NETGEAR VEGN2610 Log [86:9C:A1]"
mailbox = "[Gmail]/All Mail"
size = 27951
____________________ Message Body ____________________
[Site allowed: su.ff.avast.com] from source 192.168.0.96, Saturday, Sep 12,2015 17:00:53
Firewall: packet drop. 172.192.255.255 -->172.192.184.226, Protocol ICMP, Message type 3.
Saturday, Sep 12,2015 17:00:46
[Site allowed: static.ess.apple.com:80] from source 192.168.0.126, Saturday, Sep 12,2015 17:00:34
Firewall: packet drop. 172.192.255.255 -->172.192.184.226, Protocol ICMP, Message type 3.
Saturday, Sep 12,2015 17:00:33
[Site allowed: t.tcactivity.net] from source 192.168.0.150, Saturday, Sep 12,2015 17:00:33
[Site allowed: deliver.oztam.com.au] from source 192.168.0.173, Saturday, Sep 12,2015 17:00:31
[Site allowed: static.ess.apple.com:80] from source 192.168.0.126, Saturday, Sep 12,2015 17:00:30
Firewall: packet drop. 172.192.255.255 -->172.192.184.226, Protocol ICMP, Message type 3.
Saturday, Sep 12,2015 17:00:24
Firewall: packet drop. 172.192.255.255 -->172.192.184.226, Protocol ICMP, Message type 3.
Saturday, Sep 12,2015 17:00:11
[TR-069] Send Inform, Saturday, Sep 12,2015 17:00:07

Tags (1)
0 Karma
1 Solution

Richfez
SplunkTrust
SplunkTrust

So, as it stands right now, when you search Splunk for, say, "172.192.184.226" you get that entire message, and not just the 3 or 4 lines from inside that message, right?

Yet what you want is to skip all those headers and just grab the message body, and have that message body parsed off into individual lines so that when you run that same search as above, you'd just get 4 events (or whatever) that are properly separated and time-stamped?

I think if that's the case, we may be able to fix this using the app's options indexHeaders = False, indexBody = True and filter = regex, then possibly configure event line breaking and timestamp recognition. But first, we need to make sure how it's coming in, so confirm all the above and we can start picking this job apart and getting it working the way you want.

View solution in original post

0 Karma

Richfez
SplunkTrust
SplunkTrust

So, as it stands right now, when you search Splunk for, say, "172.192.184.226" you get that entire message, and not just the 3 or 4 lines from inside that message, right?

Yet what you want is to skip all those headers and just grab the message body, and have that message body parsed off into individual lines so that when you run that same search as above, you'd just get 4 events (or whatever) that are properly separated and time-stamped?

I think if that's the case, we may be able to fix this using the app's options indexHeaders = False, indexBody = True and filter = regex, then possibly configure event line breaking and timestamp recognition. But first, we need to make sure how it's coming in, so confirm all the above and we can start picking this job apart and getting it working the way you want.

0 Karma

jjpeet
New Member

Hi,
What you describe is exactly what I'm after.

Thx
JP

0 Karma

Richfez
SplunkTrust
SplunkTrust

Great!

I don't have imap or the app installed, so you'll have to go down that road alone. Hopefully it'll be straightforward to change/set the indexHeaders and indexBody and see if they get your events coming in without the headers. I suspect if you look through inputs.conf in the app you'll figure it out - it'll be mostly configured, you'll just need to add a line or two.

Your event line breaking and timestamp recognition may actually "just work" once you get the above changes made. Splunk's pretty good about figuring out timestamps that are in a fairly normal format, and those look relatively non-weird. It should linebreak properly as well, I think, because your data looks line-oriented with carriage returns/line breaks as appropriate.

Can you give those a try then let us know how it went? If you have specific issues with any particular piece, please post back your problems with specifics (or create a new question, which will get new people looking at the problem).

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...