Splunk Search

How to Handle Metadata in File Headers

chrissale
Explorer

I am using Splunk to collect data from log files generated by a thick client application. The log files contain metadata in the header relating to the user that logged on. I want to be able to search for events using the metadata in the file header (example below).

username: myUser
hostname: myHost
10/02/2014 13:12:03 INFO User did some stuff
10/02/2014 13:12:41 INFO User did some more stuff
10/02/2014 13:14:26 WARNING User did some stuff they weren't supposed to!

In this example a search for 'username=myUser' would return all three events shown. Is that possible?

0 Karma

yannK
Splunk Employee
Splunk Employee

Not easily because

  • the fields username and hostname will be present in the first events (with a timestamp that will be unpredictable)
  • Then the other events will be all individuals, and not contain the username and hostname information.

If your log files are unique (unique filename), you could build a lookup to link : the source and the username and hostname
then when searching use the lookup to add those fields to the events of this particular source.
see http://docs.splunk.com/Documentation/Splunk/6.0.1/Search/Useexternalfieldlookups

Otherwise, you should change you method of login, and maybe add the username and hostname in the source.
the extract them at search time with a rex command.

or in the path, example : /path/to/my/file/<host>/<username>/file.log
you can have the host field extracted at index time with host_segment or host_regex, see http://docs.splunk.com/Documentation/Splunk/6.0.1/Data/Setadefaulthostforaninput

0 Karma

yannK
Splunk Employee
Splunk Employee

There is a possibility is to add a meta field at index time per monitor in the inputs.conf, but it will not be dynamic per files, So not really what you want.

The only real solution is to format your events to add the fields to every lines before monitoring, Or create a custom scripted input to replace the monitor, and add then on the fly.
see http://docs.splunk.com/Documentation/Splunk/6.0.1/AdvancedDev/ScriptedInputsIntro

0 Karma

chrissale
Explorer

Thanks for the response yannK. I like where you are going with this but unfortunately I can't guarantee that the source name is unique or influence the naming strategy for the file. I am going to see if I can create a custom field at indexing time that uniquely identifies the file (CRC?) and use that for the lookup.

0 Karma

yannK
Splunk Employee
Splunk Employee

I forgot, a subsearch could work if you have unique file sources.

[search username=myUser | dedup source | table source] myothercondition=condition

the sub search will return the source name of the events that contains the username, and it will become a search condition in the man search.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...