Parsing Windows DHCP Trace Logs using HF -> Indexer -> SH

[reswob4]

OK, I've been looking at collecting and parsing the Windows DHCP Trace Logs and after reviewing several forum posts and splunk docs, I had not found anything suggesting how to parse the log. There was a free app, but it seems to have disappeared from the apps.

So I'm posting this description for how I did it in the hopes that a: someone else can use it and b: someone else can improve it.

On my HF, I set up remote file monitoring (under Settings - Data Inputs - Files & Directories) and pointed to the Windows DHCP Trace Log file (I am not allowed to install locally on the server). The HF is sending events to an Indexer and in turn, I am searching/viewing those events on a SH. The challenge with parsing the DHCP log file, is that it has several line variations.

For example, the following four lines types can be found in the trace file

11,10/01/15,08:39:59,Renew,1.1.0.2,,MAC,,3463970816,0,,,
24,10/01/15,08:37:49,Database Cleanup Begin,,,,,0,6,,,
11,10/01/15,08:35:27,Renew,1.1.0.1,hostname,MAC,,1027190399,0,,,
32,10/01/15,09:36:09,DNS Update
Successful,1.1.0.10,hostname,,,0,6,,AAEBZaMAKV,

As you can see, sometimes the hostname is there and sometimes it isn't.
Also, sometimes the MAC is there and sometimes it isn't.
And sometimes the line is for information or status with no IP, hostname, or MAC.

The columns correspond to the following headers: ID,Date,Time,Description,IP,hostname,MAC
(see the following link for more info on the DHCP Trace Log file format:
https://technet.microsoft.com/en-us/library/dd183591(v=ws.10).aspx )

I tried doing a field extraction at the SH, but it seems that when Splunk builds the regex using the web gui, Splunk doesn't know how to handle a double ,, when the field is empty. More likely, I don't know how to make Splunk figure that out through the gui.

So I ended up manually configuring the following setup. On the HF I crafted the <splunk>/etc/apps/search/local/inputs.conf:

[monitor://\\<DHCP Server IP>\c$\Windows\System32\dhcp\DhcpSrvLog-*.log]
disabled = false
host = DHCPSERVER
sourcetype = WinDHCP

On the SH, I created the props.conf, I crafted the <splunk>/etc/apps/search/local/props.conf:

[WinDHCP]
EXTRACT-DHCP_EventID,Description,src,hostname,MAC = (?P<DHCP_EventID>[^,]*),[^,]*,[^,]*,(?P<Description>[^,]*),(?P<src>[^,]*),(?P<hostname>[^,]*),(?P<MAC>[^,]*),[^,]*,[^,]*,[^,]*,.*

This props.conf could also be configured on the indexer, if you want.

Now everything is parsed correctly when I perform a search.

Feel free to comment, suggest improvements or point to someone/somewhere else that has a better/different solution.

[bmacias84]

I would use a Report Transform and specify the DELIMS. This should be faster than using regex. I think i got the delims right.

# Transforms.conf
[WinDHCP_csv]
DELIMS = ","
FIELDS = "DHCP_EventID", "Date", "Time","Description","src","hostname","MAC","field1","field2","field3","field4","field5","field6"

11,10/01/15,08:39:59,Renew,1.1.0.2,,MAC,,3463970816,0,,,
# Props.conf
[WinDHCP]
REPORT-WinDHCP = WinDHCP_csv

[mreynov_splunk]

I would be careful with DELIMS if there is a possibility of field values containing quotes: http://docs.splunk.com/Documentation/Splunk/latest/admin/transformsconf