Alerting

alter index=_internal query to ignore a specific index

rnolette
Path Finder

Below is teh query I am using to alert me of excessive hourly usage by a host. Is there a way to modify the first portion of the query "index=_internal source=*license_usage.log" to ignore a specific index?

index=_internal source=*license_usage.log | eval lastReceived = _time | rename s as source st as mysourcetype h as host b as bytes o as originator | eval my_splunk_server = splunk_server | fields lastReceived source mysourcetype host bytes pool originator my_splunk_server | stats sum(bytes) as bytes max(lastReceived) as lastReceived by host | eval mbytes=((bytes/1024)/1024) | fields host mbytes lastReceived | stats max(lastReceived) as lastReceived sum(mbytes) as mbytes by host | stats max(lastReceived) as lastReceived first(mbytes) as MBytes by host | search MBytes > 25| sort by MBytes Desc

I have tried:

  1. (index=_internal source=*license_usage.log) NOT index=other
  2. (index=_internal NOT index=other) source=*license_usage.log
  3. (index=* NOT index=other) source=*license_usage.log
  4. (index=* source=*license_usage.log) NOT index=other

All of the above modifications have failed. None of them give any results. I am currently assuming this is because the index=_internal statement is covering all indexes and I am unable to create exceptions for that statement. Can anyone speak to this? thoughts? comments? questions? concerns? Any ideas are greatly appreciated.

thanks!

Tags (4)
0 Karma
1 Solution

rnolette
Path Finder

I believe i have found my answer. After looking at the output for

index=_internal  source=*license_usage.log 

i saw that there is a variable called "h" that contains the hostname.

5/6/13
    11:22:24.171 AM 
    05-06-2013 11:22:24.171 -0400 INFO  LicenseUsage - type=Usage s="udp:514" st=syslog h="otherhost" o="" i="<removed>" pool="auto_generated_pool_enterprise" b=21067 poolsz=5368709120
    host=<removed>   Options|  sourcetype=splunkd   Options|  source=<removed>

I then edited my original query to ignore the hostnames i wanted to ignore (index=_internal source=*license_usage.log) NOT(h="ignoreME") and below is the full results.

(index=_internal source=*license_usage.log) NOT(h="ignoreME") | eval lastReceived = _time | rename s as source st as mysourcetype h as host b as bytes o as originator | eval my_splunk_server = splunk_server | fields lastReceived source mysourcetype host bytes pool originator my_splunk_server | stats sum(bytes) as bytes max(lastReceived) as lastReceived by host | eval mbytes=((bytes/1024)/1024) | fields host mbytes lastReceived | stats max(lastReceived) as lastReceived sum(mbytes) as mbytes by host | stats max(lastReceived) as lastReceived first(mbytes) as MBytes by host | search MBytes > 25| sort by MBytes Desc

View solution in original post

0 Karma

dkuk
Path Finder

It sounds like you've got where you want to be now though it's worth also checking out this link for some useful ways of troubleshooting license usage - some more searches.

In addition to this, Splunk Deployment Monitor app and Splunk on Splunk apps provide some useful dashboards for tracking usage and splitting it by host, sourcetype etc. These apps are free and available on Splunkbase.

Another place to look is in the search app under the Status > Index Activity menu option. There are some useful dashboards there also, notably Index Volume.

rnolette
Path Finder

thank you for the information dkuk. I have been using those apps for information gathering in the past but they are not able to generate alerts which is what i really needed, so i had to create this manual queries.

0 Karma

rnolette
Path Finder

I believe i have found my answer. After looking at the output for

index=_internal  source=*license_usage.log 

i saw that there is a variable called "h" that contains the hostname.

5/6/13
    11:22:24.171 AM 
    05-06-2013 11:22:24.171 -0400 INFO  LicenseUsage - type=Usage s="udp:514" st=syslog h="otherhost" o="" i="<removed>" pool="auto_generated_pool_enterprise" b=21067 poolsz=5368709120
    host=<removed>   Options|  sourcetype=splunkd   Options|  source=<removed>

I then edited my original query to ignore the hostnames i wanted to ignore (index=_internal source=*license_usage.log) NOT(h="ignoreME") and below is the full results.

(index=_internal source=*license_usage.log) NOT(h="ignoreME") | eval lastReceived = _time | rename s as source st as mysourcetype h as host b as bytes o as originator | eval my_splunk_server = splunk_server | fields lastReceived source mysourcetype host bytes pool originator my_splunk_server | stats sum(bytes) as bytes max(lastReceived) as lastReceived by host | eval mbytes=((bytes/1024)/1024) | fields host mbytes lastReceived | stats max(lastReceived) as lastReceived sum(mbytes) as mbytes by host | stats max(lastReceived) as lastReceived first(mbytes) as MBytes by host | search MBytes > 25| sort by MBytes Desc
0 Karma

rnolette
Path Finder

as per a suggestion from /k i have modified my overall query to:

index=_internal source=license_usage.log NOT h="hosts" | bucket _time span=1h | eval when=strftime(_time, "%F %H:00") | stats sum(b) as mbytes by h, when | eval mbytes = round(mbytes/1024/1024,2) | where mbytes>15 | sort by mbytes desc

0 Karma

kristian_kolb
Ultra Champion

I believe there are simpler ways of achieving what you're trying to do.

index=_internal source=*license_usage.log  
| bucket _time span=1h
| eval when=strftime(_time, "%F %H:00") 
| stats sum(b) as mbytes by h, when 
| eval mbytes = round(mbytes/1024/1024,2) 
| where mbytes>25 

rnolette
Path Finder

thank you for the suggestion. This is much cleaner and simpler than my current query. I have modified mine and posted the new query in the answer. Thank you very much for showing me a better way.

0 Karma

rnolette
Path Finder

thank you but that still does not allow me to ignore certain indexes or hosts. but i do think i have found a solution.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

If your search only looks in index _internal it will not look into any other index, any other index already is excluded.

rnolette
Path Finder

thanks! I will try that now and see if i can improve my query.

0 Karma

kristian_kolb
Ultra Champion

What goes in the internal index _internal?

general splunkd messages (splunkd.log)
searches that have been performed (searches.log)
metrics data from forwarders (metrics.log)
...

try this;

index=_internal | dedup 3 source sourcetype

This will give you a small sample of the types of events/sources that will end up in _internal

rnolette
Path Finder

to be honest I am confused then to what _internal uses. Is my whole query wrong then based off of that first statement? i was think that "(index=_internal source=*license_usage.log) NOT [search index=dogs]" might do what i needed but after you last comment i am not sure anymore. Can you elaborate a bit on what indexes contain what data? I am going to start googling to find more information. Thanks k!

0 Karma

kristian_kolb
Ultra Champion

Internal indexes in general start with an underscore. There are several of these, e.g. _audit, _blocksignature and _internal.

If you specify an index as part of the search, only that index will be searched.

main is not an internal index, and there is no such thing as 'index enveloping'.

/k

rnolette
Path Finder

does index=_internal envelope all internal indexes like main, dhcp, win_dns, etc?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...