Dashboards & Visualizations

How to label search results and create a "Time since last crawls" dashboard panel?

pentago
New Member

2 questions

1. I search my Nginx access logs for various Googlebot traffic (web, image, ads, mobile, smartphone, adsense, feedfetcher).

All of those bots have their own (https://goo.gl/sUcGvU) useragent which I use to perform the search. Currently I just paste those useragents and get results, but I'd like to get search results labeled.

For instance, all search results for "Googlebot/2.1 (+http://www.google.com/bot.html)" I'd like to be labeled as "Googlebot" in statistics tab.

Example:
alt text
alt text

2. Is there a chance to create somehow dashboard panel like this or smililar? alt text

0 Karma

Runals
Motivator

I'd use a case statement and a regex match or maybe like against whatever your user agent field is called

sourcetype = nginx | eval bot = case(match(useragent,"Googlebot"), Googlebot, match(useragent, "foo"), "Foobot", match(useragent, "Bar"), "Barbot", 1=1, "Unknown Bot")

I'd have to think about your dashboard but I'd probably start with something like

sourcetype = nginx | eval bot = case(match(useragent,"Googlebot"), Googlebot, match(useragent, "foo"), "Foobot", match(useragent, "Bar"), "Barbot", 1=1, "Unknown Bot")  | stats max(_time) as last_seen by bot | eval current_time = now() | eval delta = current_time - last_seen | timechart max(delta) by bot 

The problem is that search doesn't account for the number of days ago by individual days so you might need to bake in a bins command before the stats and actually convert the stats to eventstats. I'm not sure what the colors mean; I'm guessing individual bots.

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust
  1. Use eventtypes. Once you define an eventtype, use it like such: stats count by eventtype | where eventtype="bot_*"

    [bot_googlebot]
    search = Googlebot/2.1

http://docs.splunk.com/Documentation/Splunk/6.2.3/Admin/Eventtypesconf?r=splunky

  1. Yes. But for performance reasons, you would probably want to use a Data model acceleration or Summary Indexing to do the relevant counts. Summarize the data into daily buckets. Also, it depends on what your time ranges are refering to. Is it counts of activity? or ...?
0 Karma

pentago
New Member

Awesome. Thanks man. Any chance i could rename those searches directly from search? I saw that some stuff could be renamed with AS or something like that. Not quite sure as I'm a newbie.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...