How to count Unique User Agents in the message log?

[davidsumner]

I would like to setup a dashboard that tracks the totals for user agents in incoming requests.  I couldn't find a "user agent", "user-agents" or any other field listed.  When I exported the search results to CSV, I saw the following heading: 

"_raw","_time",cloudaccount,host,index,linecount,message,source,sourcetype,"splunk_server"

It appears that the info containing the user agent is contained in the message field, enclosed in double double-quotes.

I assume that in order to count each type of user agent, I first need to isolate these values, then count them.  What's the best way to do that?

[somesoni2]

Give this a try

Find count of events by userAgent

Your base search
| rex "\]\s+(\"+[^\"]+){3}\"+\s+\"+(?<userAgent>[^\"]+)"
| stats count by userAgent

 Trend of distinct count of userAgents over time

Your base search
| rex "\]\s+(\"+[^\"]+){3}\"+\s+\"+(?<userAgent>[^\"]+)"
| timechart dc(userAgent) as distinct_userAgents

[davidsumner]

When I did a table view I realized that the data I'm looking for is actually part of the _raw field.  I'm trying to figure out how isolate the user-agent portion (in bold) and count the different unique values that get reported.  Here's a sanitized version of a record.

"<133>1 2023-09-21T14:53:43+00:00 host-29490 example.apache-access - - - 208.207.1.214 - - [21/Sep/2023:14:53:43 +0000] ""GET / HTTP/1.1"" 302 46779 ""https://edit.onlineshop.example.com/"" ""Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake)"" vhost=example.prod.my-sites.com host=edit.example.com hosting_site=example pid=24164 request_time=106271 forwarded_for=""208.51.62.14, 64.220.85.15, 23.120.51.94"" request_id=""reqid-a88558b0-5a8e-1ee-6e0-ea57887e2d"" location=""/user/login"" ","2023-09-21T10:53:43.000-04:00",778910529448,"52.22.171.60",application,1,,example,"tcp-raw","splunk-indexer-ip-10-128-128-5.ec2.internal"

[richgalloway · Accepted solution]

This works with the sample event

<<your query>>
| rex "https?\S+\s\\\"+(?<UA>[^\\\"]+)"
| stats count by UA

[richgalloway]

Since rex is my favorite command, I'll say that's the best way to extract the field.  Of course, that depends on the details.  Please share (sanitized) sample events with message fields.