Splunk Search

Need help using Tstats getting count of a string in raw logs

aamer86
Path Finder

I want to show the count of logs where a string appeared

I have a string and need to know how many times it appears in logs

Tags (1)
0 Karma
1 Solution

woodcock
Esteemed Legend

Like this:

| tstats count WHERE index="foo" AND sourcetype="bar" AND TERM(SearchStringHere)

View solution in original post

landen99
Motivator

It won't work with tstats, but rex and mvcount will work.

For the tstats to work, first the string has to follow segmentation rules.
Second, you only get a count of the events containing the string as presented in segmentation form. If the string appears multiple times in an event, you won't see that.

The best you can get is a count of the number of events containing the string if it follows the segmentation rules or it's contained in an indexed field. Any string with major segment breakers in it will fail in tstats TERM.

| tstats count WHERE index=foo sourcetype=bar TERM(mySegment) TERM(mySegment2)

Also, if you ever consider using NOT with tstats, you'll also need to consider the effects of the following Splunk bug: https://answers.splunk.com/answers/787657/not-term-removes-results.html?childToView=787658#answer-78...

The better solution is to extract and count each string with rex:

index=<yourIndexName> "yourString"
| rex max_match=0 "(?<yourStringName>yourString)" 
| eval str_cnt = mvcount(yourStringName)
| stats sum(str_cnt) AS "No. of occurrences"
0 Karma

woodcock
Esteemed Legend

Like this:

| tstats count WHERE index="foo" AND sourcetype="bar" AND TERM(SearchStringHere)

aamer86
Path Finder

Thanks @woodcock

This is exactly what I was looking for

landen99
Motivator

It won't work.
First, the string has to follow segmentation rules.
Second, you only get a count of the events containing the string as presented in segmentation form. If the string appears multiple times in an event, you won't see that.

woodcock
Esteemed Legend

Agreed, it is a somewhat limited solution but it is the best that can be done.

0 Karma

aamer86
Path Finder

Thanks @landen99

does that mean if im looking for a string like this

"eventName": "DeleteBucketPolicy

it wont work due to space and double qoute ?

0 Karma

woodcock
Esteemed Legend

Yes, there is no way to use TERM with a string with double-quotes.

0 Karma

landen99
Motivator

TERM will not work with that explicit string, but you can search for the combination of the two substrings: eventname AND deletebucketpolicy (case insensitive). The odds of false positive event matches on those two strings surely must be extremely low to non-existent. A much slower way to do it is just to extract the string with rex multiple times and then count the number of extractions (in the event that an event may contain the string more than once and you want to count every instance).

0 Karma

arjunpkishore5
Motivator

if your data has indexed extractions, then you can use tstats as follows

|tstats count as total where index=<your index> AND <your field>=<your value>

by default, metadata fields such as host, source, sourcetype, and _time are indexed. unless <your field> is defined as an indexed extraction, the tstats won't work.

The other alternative is to build a datamodel on your data to be able to use tstats. if you have a datamodel defined, your tstats would be like this.

| tstats count as total from datamodel=< your datamodel name> where nodename=<your node name> AND <your node name>.<your field>=<your value>

If you have neither

index=<your index> <your field filter>
| stats count as total
0 Karma

aamer86
Path Finder

Thanks @arjunpkishore5 but this doesnt get me what I want

So the field is not from the indexed extraction fields i.e. it is not hostname, index, sourcetype or sourse.

I dont want to create a DataModel

I want to use tstats as it is faster

0 Karma

landen99
Motivator

tstats only counts events, not string occurrences.

0 Karma

arjunpkishore5
Motivator

As performant as tstats is, it cannot be used without an indexed field extractions or a datamodel. If you must use tstats, have the fields added to your indexed extractions. This however has performance impact and increased disk usage during indexing. Or, create a datamodel.

0 Karma

woodcock
Esteemed Legend

This is not exactly true; it can also leverage the terms in the tsidx file. See my answer.

0 Karma

niketn
Legend

@aamer86 if the field/fields containing the string you want to count are
(1) indexed extracted or
(2) has accelerated Data Model created then only you can use the tstats command.

Otherwise you will have to perform regular index search.

index=<yourIndexName> "<yourString>"
| stats count as "No. of occurrences"

Please provide more details for the community to assist you better.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

aamer86
Path Finder

the string is indexed but i need to get numbers over long period so i want to use tstats but the index is not in any Data Model

I remember once I saw tstats used with raw but cant remember the context

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...