Splunk Search

Searching for all field values in order to flag log changes

Dark_Ichigo
Builder

So basically What im looking for is a search where I can search for the values of fields, for example a httResponse has to values "200" and "404", I want to be able to count only "200" and "400" not how many are found, but only if they are there.

I want this to be done for all fields, in one search!

So in the end I can create a Summary index that will run everyday to check if for example the value "200" in the field isnt there any more...Basically flagging an error.

How can this be done?

index="index" httpResponse="*" | stats count by httpResponse ---> something like this but for all fields!

0 Karma
1 Solution

bbingham
Builder
 | stats values(*)| fields - raw | stats count(*)

Stats values(*) will list all unique values for a field, use list if you want every occurrence. Append fields - with what ever fields you'd like to exclude before the count. Using the * in the stats command tells splunk to do the command for all fields.

You can also then append a rename function to change the column titles to only the field names:

| rename count(values(*)) AS *

View solution in original post

bbingham
Builder
 | stats values(*)| fields - raw | stats count(*)

Stats values(*) will list all unique values for a field, use list if you want every occurrence. Append fields - with what ever fields you'd like to exclude before the count. Using the * in the stats command tells splunk to do the command for all fields.

You can also then append a rename function to change the column titles to only the field names:

| rename count(values(*)) AS *

Dark_Ichigo
Builder

It worked fine thanks, but one issue remains, Now all I want is to search for all the field names alone, so I only want the results of field names.

Is that Possible?

http://splunk-base.splunk.com/answers/31164/retrieve-names-of-all-fields

0 Karma

bbingham
Builder

The issue you're going to run into is you'll have to keep pushing the summarized data forward. Also when your searching the summarized data also is based on time, so you'll have to either run a constant summary or run a subsearch to gather the data your after. In the use case you described above, I'd personally use a lookup table. You can populate the lookup table with a list of fields, then use |inputlookup append=t to the end of a stats command and append the list of fields your expecting. Since lookups aren't time sensitive you won't have to use a subsearch to gather your data.

0 Karma

Dark_Ichigo
Builder

My summary index is basically to limit the load on the system, rather than having something running in the background continue-sly, a summary index will run once a day and through that I can determine if all of a sudden a field disappears.

Im currently not familiar with static Lists in Splunk if you can help shed some light on the issue.

My main goal is just to have a real-time alert based on a summary index that checks for any fields that may have disappeared due to changes in the structure of logs for example.

0 Karma

bbingham
Builder

It really just depends on what the ultimate goal is, if you're just trying to get notified if a field is empty, you may be better off doing a real-time alert. The summary index really in the case your using, only tells you if you had a field and it disappeared, since new fields probably don't just "appear" you may be better off building a static list.

0 Karma

Dark_Ichigo
Builder

Thanks, that was really cool!!

I was also wondering that if I could use this to my advantage, basically after turning this into a summary index and using a backfill script to populate it, to search for the fields that have no values or missing values?

Basically I don't want to have to write a long search checking for every field in the summary index to check that there are no missing values within fields, or even missing fields all together.
Is there a way to simplify this?

0 Karma

Dark_Ichigo
Builder

Yes, basically I want to know in the end that the field httpResponse has both "200" and "404" within it.

So I want it to return 2, as in it contains 2 values!

0 Karma

bbingham
Builder

can you elaborate a little more on what you're looking for? You are wanting a count of events where httpResponse = 200 or 404? or you're looking for how many fields have 200 and 404 in them? Or are you looking for a list of values that each field has at the end of the day, basically the "values" or "list" function of timechart/stats/chart?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...