Getting Data In

Finding forwarders that are 3.4.x and older.

sanju005ind
Communicator

I have a around 800 forwarders in my distributed environment.Most of them would be 3.4.11 or 3.3.x and only around 50 odd with ver 4.0.9.These devices are tagged as AD or DHCP or DNS along with the Business Tag. I am attempting to create a dashboard that displays the forwarders by business which are not upgraded to 4.0.9. I am using the following query

(tag::host=AD OR  tag::host=DHCP OR tag::host=DNS) AND tag::host=$buss$ AND linecount > 50 | regex _raw!="^[0-9][0-9]\/[0-9][0-9]\/10" | stats count by host | sort -count

Though I get a fair result. I would like to know if there is any other efficient way of writing this query.

1 Solution

sanju005ind
Communicator

Currently we are using linecount as an indicator since the old forwarders are sandwiching the logs. We are using the following search and excluding EventCode 565 since it contains around 200 line at times.We get a fair amount of result.

tag::host=AD OR tag::host=DHCP OR tag::host=DNS) AND tag::host=$buss$ AND linecount > 50 AND EvenCode!=565 | regex _raw!="^[0-9][0-9]\/[0-9][0-9]\/10" | stats count by host | sort -count

View solution in original post

0 Karma

sanju005ind
Communicator

Currently we are using linecount as an indicator since the old forwarders are sandwiching the logs. We are using the following search and excluding EventCode 565 since it contains around 200 line at times.We get a fair amount of result.

tag::host=AD OR tag::host=DHCP OR tag::host=DNS) AND tag::host=$buss$ AND linecount > 50 AND EvenCode!=565 | regex _raw!="^[0-9][0-9]\/[0-9][0-9]\/10" | stats count by host | sort -count

0 Karma

Lowell
Super Champion

I don't know what you mean by "sandwich the AD logs". I'm guessing that you're referring to some kind of glitch that only occurs in certain releases of splunk, and therefore you've built a search to detect that anomaly and using that as the basis for determining which machines are running which version. Is that correct? If that's all you have to work with, then this search may be the best option that you have from within splunk.

0 Karma

Lowell
Super Champion

Oh, I understand your regex now. I was seeing "\/" as a "V", whoops. (Fonts make a big difference) BTW, you don't need the "\" at all in this case, you could just write: | regex _raw!="^[0-9][0-9]/[0-9][0-9]/10"

0 Karma

Lowell
Super Champion

You could try cross referencing all of the forwarders reported build numbers with a cross-reference table to get the splunk version numbers using a simple lookup table.

It appears that you have a couple of different ways to get the build information. Some of this will depend on whether or not you are forwarding _internal events or not. Here are a few options to try:

Search 1: Use Internal startup event to capture build number

 index=_internal sourcetype=splunkd loader "Splunkd starting" | rex "build (?<build>\d+)" | stats max(build) as build by host

Search 2: Use metrics info.

Note: This info may only be generated with deployment clients, so this may not work for you. I'm not 100% sure

index=_internal sourcetype=splunkd Metrics "group=ds_connections_default" | stats max(build) as build by dns, ip, hostname


If you can get one of these searches to give you a host/build breakdown, then it's simply a matter of adding a lookup command like so. | lookup splunkbuild build OUTPUT version

Splunk doesn't have such a lookup table by default, but you can build your own pretty easily. There's a short script on the Splunk build number to version table? post that you can use for this.


Another completely different approach would be to use a script to contact splunkd on all your forwarders and get each to report the server version. You could use splunk to export a list of your forwarders which you could then use in your script. Of course, I'm not sure if this service API was available in previous versions so this may not work. Also, if you have a different username/password setup for your various forwarders that would complicate things as well.

The REST end point can be accessed via:

https://splunk.example.com:8089/services/server/info/server-info

The XML output will contain the version key, something like this:

<s:key name="version">4.1.2</s:key> 

You probably want to try calling this URL on a couple of forwarders manually first. (Try ones that you know are running 3.3 or 3.4 first and see if they will return a version this way or not. If they do, then this may be an option for you.)

Well, that's it. I'm out of ideas.

Lowell
Super Champion

I'm guessing that your not using the deployment client/server either? (I added one more last-ditch-approach to my answer.) Good luck.

0 Karma

sanju005ind
Communicator

The forwarders are not forwarding _internal events.

0 Karma

Lowell
Super Champion

Do you forward events in your _internal index?

0 Karma

sanju005ind
Communicator

noticed that a lot of the "bad" log entries didn't start with a date. The regex looks for a date in the beginning of the raw event.. 06/15/10

0 Karma

sanju005ind
Communicator

In this particular case I am searching for all of the windows forwarders since the indexers are 4.1.2 and the older forwarders seem to sandwich the AD logs sometimes the line count is around 257.Basically I am looking for all those forwarders that are not 4.0.9.

0 Karma

Lowell
Super Champion

Can you explain the purpose of (1) linecount>50 and (2) what you regex is looking for? Is this part of determining what version of splunk is running, or can you determine the version based on just the assigned tags?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...