How to get top user agent distribution

[xvxt006]

Hi, we have a filed User_Agent which gets the user agents distribution. But what i would like to get it by browser family instead of showing by version.
Meaning right now the output is

S.No User Agent Count Percentage

1 Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MS-RTC LM 8; .NET4.0C; .NET4.0E) 26513 8.818970
2 Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) 16544 5.503000
3 - 16041 5.335688
4 Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; CWADS32; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E) 15727 5.231243
5 Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) 10227 3.401788
6 Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/20100101 Firefox/17.0 6462 2.149443
7 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; CWADS32; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E) 5867 1.951529
8 Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 5464 1.817480
9 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 4655 1.548384
10 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 4555 1.515121

I want the output to be something like this

Browser Requests %
FireFox 4000 2
IE 12000 4

[elesinolalekan]

Anyone Help, please. I'm new to Splunk. Anytime I search with the command top, no result is returned. Any help?

sourcetype=access_combined_wcookie| top limit=20 url

[elesinolalekan]

Thanks for the prompt reply. The search returns "0 Events Found".
Yes there is a field named url.

Sorry about my late reply

[lguinn2]

First, what happens if you simply run the base search

sourcetype=access_combined_wcookie

and second, when you look at the results of the base search, is there a field named url?

[xvxt006]

Hi Dave, i have asked my admin to install your addon. I think he installed it under search App. But i am not seeing any of the fields that you have mentioned (I have field discover On). How to make sure that we have installed it properly.

[xvxt006]

Great. Thank you for both of your answers. I will try these.

[lguinn2]

I suggest that you build a lookup table that contains the following fields:

useragent,browser
"Mozilla/5.0 (compatible; Googlebot...",Googlebot

I didn't put in the full useragent field, as it is pretty long!

To figure out what user agents should be in the lookup table, you could run this search

sourcetype=whatever | stats count by useragent

Export the results and use them to build the .csv file. Read the transforms.conf file section for lookups, and you will find that you can use wildcards in your lookups! This feature is not available from the Splunk Manager UI; you have to edit the transforms.conf file directly.

Use the wildcard feature to manage the slight variations in useragents. You will still probably have a lot of entries in your lookup file. Here are some other answers that may help with the lookup:

http://splunk-base.splunk.com/answers/52580/can-we-use-wild-characters-in-lookup-table

http://splunk-base.splunk.com/answers/28566/how-to-use-wildcard-in-lookup-based-searches-and-alerts

And links to the docs:

http://docs.splunk.com/Documentation/Splunk/5.0/knowledge/Addfieldsfromexternaldatasources

http://docs.splunk.com/Documentation/Splunk/5.0/admin/transformsconf

Once you have the lookup table in Splunk, and the lookup defined, your search will be easy:

sourcetype=whatever 
| lookup lookup-name useragent OUTPUT browser
| top browser

[dshpritz]

Hey xvxt006,

Sorry to plug my own stuff, but this may help:
http://splunk-base.splunk.com/apps/48017/ta-uas_parser

Put simply, user-agent strings suck, extra parsing is required. This lookup does for you, which should allow you to get the types of stats you are looking for. Something like:

index=web_data | lookup uas_lookup http_user_agent | stats count by ua_family

HTH,

Dave

[dshpritz]

You will need to rename or copy your field to the http_user_agent field. You can do:

index=web | rename useragent AS http_user_agent | lookup_uas http_user_agent

or

index=web | eval http_user_agent = useragent | lookup_uas http_user_agent

[xvxt006]

Dave,

i tried sourcetype=access_combined_wcookie | lookup uas_lookup http_user_agent and i see the events but when i added sourcetype=access_combined_wcookie | lookup uas_lookup http_user_agent | stats count by ua_family i am not getting any results (even though it has matching events). our user agent field name is useragent. Do i need to change anything in the lookup table to match our field names, etc?

[dshpritz]

Without looking at the data coming into it, it's not really something I can debug off of the top of my head. I haven't gotten other reports of that being a problem.

[marcellodesales]

Not sure if it is a current problem, but I couldn't run the script... The red error message "Script for lookup table 'uas_lookup' returned error code 1. Results may be incorrect." is displayed after taking a while running... The columns "os_company os_family os_name ua_build_version ua_company ua_family ua_info_url ua_major_version ua_minor_version ua_name ua_type" are displayed, but empty... Any way to verify that?