Splunk Search

How to Handle Metadata in File Headers

chrissale
Explorer

I am using Splunk to collect data from log files generated by a thick client application. The log files contain metadata in the header relating to the user that logged on. I want to be able to search for events using the metadata in the file header (example below).

username: myUser
hostname: myHost
10/02/2014 13:12:03 INFO User did some stuff
10/02/2014 13:12:41 INFO User did some more stuff
10/02/2014 13:14:26 WARNING User did some stuff they weren't supposed to!

In this example a search for 'username=myUser' would return all three events shown. Is that possible?

0 Karma

yannK
Splunk Employee
Splunk Employee

Not easily because

  • the fields username and hostname will be present in the first events (with a timestamp that will be unpredictable)
  • Then the other events will be all individuals, and not contain the username and hostname information.

If your log files are unique (unique filename), you could build a lookup to link : the source and the username and hostname
then when searching use the lookup to add those fields to the events of this particular source.
see http://docs.splunk.com/Documentation/Splunk/6.0.1/Search/Useexternalfieldlookups

Otherwise, you should change you method of login, and maybe add the username and hostname in the source.
the extract them at search time with a rex command.

or in the path, example : /path/to/my/file/<host>/<username>/file.log
you can have the host field extracted at index time with host_segment or host_regex, see http://docs.splunk.com/Documentation/Splunk/6.0.1/Data/Setadefaulthostforaninput

0 Karma

yannK
Splunk Employee
Splunk Employee

There is a possibility is to add a meta field at index time per monitor in the inputs.conf, but it will not be dynamic per files, So not really what you want.

The only real solution is to format your events to add the fields to every lines before monitoring, Or create a custom scripted input to replace the monitor, and add then on the fly.
see http://docs.splunk.com/Documentation/Splunk/6.0.1/AdvancedDev/ScriptedInputsIntro

0 Karma

chrissale
Explorer

Thanks for the response yannK. I like where you are going with this but unfortunately I can't guarantee that the source name is unique or influence the naming strategy for the file. I am going to see if I can create a custom field at indexing time that uniquely identifies the file (CRC?) and use that for the lookup.

0 Karma

yannK
Splunk Employee
Splunk Employee

I forgot, a subsearch could work if you have unique file sources.

[search username=myUser | dedup source | table source] myothercondition=condition

the sub search will return the source name of the events that contains the username, and it will become a search condition in the man search.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...