hello, i have a subset of results from a search. i now that if I have a clientIP=x.x.x.x, this is proxied and i need to do another search to the outer proxy logs to retrieve the real IP address of the client; this can be achieved by doing a separate search on another sourcetype in the same exact timestamp.
what i need to do, if possibile, is to rewrite the content of the clientIP value with another retrieved with a subsearch, passing the time range and username field. something like this (but this one cannot work):
the hard part (for me) is passing the field values to the subsearch.
i need to present the data as a single result, thus i need to to do all the trasnforms inside that search.
thanks in advance
If I'm understanding what your trying to do correctly, it sounds like you want to launch a sub-search for each and every event returned by your base search (
First off, I don't think there is any way to conditionally launch a subsearch. Each subsearch is only run once, and is evaluated and expanded into the main search, and then the main search runs. Now, you can launch a search per-event using the
For the record, this is a complete guess. This would take lots of examples and probably a few hours of messing around to get something that actually works properly...
I'm not sure if you can get
Good luck ;-)
Alternate approaches: Don't forget that you can always programatically call splunk searches. It sounds like you have something complex enough that it may warrant that kind of effort, and that would give you FULL control. It's also possible that generating lookups periodically could be a much better way to handle this.
answered 19 Sep '11, 11:56
Thank you both for you exhaustive and helpful answers.
After a lot of effort in trial and error, i ended up piping search results to a custom lookup script.
The script does as search by itself, but only if needed; it leaves the field as-is, spitting out the same if the ip is NOT proxied, and not doing anything to preserve resources) .
The script works by passing the old (proxied - in the script below - is 18.104.22.168) SourceIP value, and a timestamp (to be matched against the other log) and a discriminator (in my case, since this is a pop3 authentication, is the customer username). the _time value needs to be rewritten, so do an eval (see TS below).
Please note that you need to pass an empty value to be populated by the output. you can 'construct' this field in the search by using a eval.
This took me a lot of time to understand, otherwise it doesn't work.
transform stanza needed:
for sake of completeness: the script works by being passed (via STDIN) by the splunk server a partially filled CSV file, and spitting out a filled csv file by stdout. you also need to pass the field names as arguments (see the former transform above) the script is not perfect to take it as-is, maybe can be useful to others :)
Hope will be useful to others.
In addition to Lowell's answer that covers the most: from what I've gathered, what you want to achieve cannot be easily/efficiently achieved by issuing searches on indexed data. I've been wishing for a similar feature myself, i.e. something that can perform "lookups" on indexed data as easily as you can perform lookups using .csv files and external scripts.
With manageable volumes of data, Splunk can perform time-based lookups that automatically grab values that were valid for an event's specific time period. Typical use-case is to feed Splunk with a DHCP log with time, IP and MAC address, and then use it as a time-based lookup in order to automatically tie an IP to a MAC address for each event in your searches (as long as the IP address exists in the DHCP log of course). I'm guessing your proxy logs are pretty big so feeding them to Splunk as a lookup csv is far from an ideal solution. Nevertheless, should you want to try it there's information on time-based lookups available in the docs: http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Addfieldsfromexternaldatasources#Example_of_time-based_fields_lookup
answered 19 Sep '11, 14:30