How to label search results and create a "Time since last crawls" dashboard panel?

[pentago]

2 questions

1. I search my Nginx access logs for various Googlebot traffic (web, image, ads, mobile, smartphone, adsense, feedfetcher).

All of those bots have their own (https://goo.gl/sUcGvU) useragent which I use to perform the search. Currently I just paste those useragents and get results, but I'd like to get search results labeled.

For instance, all search results for "Googlebot/2.1 (+http://www.google.com/bot.html)" I'd like to be labeled as "Googlebot" in statistics tab.

Example:

2. Is there a chance to create somehow dashboard panel like this or smililar?

[Runals]

I'd use a case statement and a regex match or maybe like against whatever your user agent field is called

sourcetype = nginx | eval bot = case(match(useragent,"Googlebot"), Googlebot, match(useragent, "foo"), "Foobot", match(useragent, "Bar"), "Barbot", 1=1, "Unknown Bot")

I'd have to think about your dashboard but I'd probably start with something like

sourcetype = nginx | eval bot = case(match(useragent,"Googlebot"), Googlebot, match(useragent, "foo"), "Foobot", match(useragent, "Bar"), "Barbot", 1=1, "Unknown Bot")  | stats max(_time) as last_seen by bot | eval current_time = now() | eval delta = current_time - last_seen | timechart max(delta) by bot 

The problem is that search doesn't account for the number of days ago by individual days so you might need to bake in a bins command before the stats and actually convert the stats to eventstats. I'm not sure what the colors mean; I'm guessing individual bots.

[alacercogitatus]

1. Use eventtypes. Once you define an eventtype, use it like such: stats count by eventtype | where eventtype="bot_*"

   [bot_googlebot]
   search = Googlebot/2.1

http://docs.splunk.com/Documentation/Splunk/6.2.3/Admin/Eventtypesconf?r=splunky

1. Yes. But for performance reasons, you would probably want to use a Data model acceleration or Summary Indexing to do the relevant counts. Summarize the data into daily buckets. Also, it depends on what your time ranges are refering to. Is it counts of activity? or ...?

[pentago]

Awesome. Thanks man. Any chance i could rename those searches directly from search? I saw that some stuff could be renamed with AS or something like that. Not quite sure as I'm a newbie.