Webintelligence: filter by host name

[opticsplanet]

I have logs coming from a load balancer. Naturally, everything is in one file. There is a field for the host name. I figured out how to properly feed this data in Splunk, and host name shows up fine in the search tool.
However, Webintelligence seems to be working with file names for source only. I need to be able to filter different web sites in the reports based on a host name. How can I do that? One way is to split log file, but I'd like to avoid that 🙂

Sample of the log file:

68.4.236.243 - - [19/Jul/2012:22:06:34 +0000] "GET /msgboard/styles/prosilver/imageset/icon_back_top.gif HTTP/1.1" 200 514 "http://www.opticsplanet.com/msgboard/about9-13081.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2" - www.opticsplanet.com
107.0.100.18 - - [19/Jul/2012:22:06:34 +0000] "GET /brunton-explorer-foldable-solar-panel-qegxz.html HTTP/1.1" 200 16502 "-" "Zend_Http_Client" - www.opticsplanet.com
189.120.181.234 - - [19/Jul/2012:22:06:34 +0000] "GET /s/search.php?query=Ray+ban+3311 HTTP/1.1" 302 531 "http://www.opticsplanet.com/s/Rb3311+black/" "Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3" - www.opticsplanet.com
72.148.43.212 - - [19/Jul/2012:22:06:34 +0000] "GET /ray-ban-sunglasses.html HTTP/1.1" 200 16396 "http://www.opticsplanet.com/sunglasses.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:6.0.1) Gecko/20100101 Firefox/6.0.1" - www.opticsplanet.com

I have Splunk set up to parse the host name from the file, so that's populated properly.

[opticsplanet · Accepted solution]

I adjusted macros.conf to this:

[sourcename_lookup]
definition = lookup sourcenames host | eval sourcename=if(sourcename==" " OR isnull(sourcename),host,sourcename)

Then adjusted sourcenames.csv to this:

host,sourcename
"*","*"
"www.site.com","site"

And finally, adjusted saved search:

[Sourcenames Lookup]
search = eventtype=web-traffic | stats count by host | eval sourcename=" " | inputlookup append=t sourcenames.csv | stats last(sourcename) as sourcename by host | outputlookup sourcenames.csv

Now when I "Select site," it works by host name, rather than source file name.

[casspugh]

I noticed that I only received this error for the 5 minute interval. I actually, uninstalled this app, reinstalled, and made the changes above, and then ran it for the first time, including the back fill. It seems to be happy now.

The splunk start up script also stated that there was a possible typo in the macros.conf file, so I deleted the "~" at the end of the file. I am not sure if that was supposed to be there, but again, it seems happier now.

[opticsplanet]

As far as I understand, the problem is with saved search adjustment, last "outputlookup" statement. Some of the host names are not in your sourcenames.csv file, and that is causing the error. See which ones are not listed, and add them to csv.

[casspugh]

Thank you very much for this.

It is exactly what I am looking for.
However, I am running into this problem when I try to implement this.

I ran the search with the modifications in place, and a new .csv was created. I entered in the names I wanted, but now when I try to use the dashboard, I receive this error:

PARSER: Applying intentions failed Error in 'lookup' command: Could not find all of the specified lookup fields in the lookup table.

Do you know how I can fix this error?
Thanks,
Cassandra

[araitz]

A sample of the file name and a sample event, as well as a specific example of how web intelligence is using the source.

[opticsplanet]

What kind of sample can I provide?

[araitz]

It would be very helpful if you could provide a sample of your data, otherwise all anyone can do is speculate.