Splunk Search

using foreach at the beginning of the query fetching less records than expected

angelinealex
Communicator
index="index1" tdr=tdr1 OR tdr=tdr2
transaction id
| foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]
| eval _time=strptime(Qtime,"%Y-%m-%d %H:%M:%S.%3N") 
| eval LoS=if(tdr=1,Cos,null)
| search LoS != 1 Los !=3 Los!=H LoS!=C
....
....
| table .....

The above code returning 140 records. But the actual records are 200.

index="index1" tdr=tdr1 OR tdr=tdr2
transaction id    
| eval _time=strptime(Qtime,"{%Y-%m-%d %H:%M:%S.%3N}") 
| eval LoS=if(tdr=1,Cos,null)
| search LoS != {1} Los !={3} Los!={H} LoS!={C}
....
....
| table .....
| foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]

The above code is returning 200 records.

Why foreach behaves differently in these two cases?
Please help me to understand the issue.

Tags (3)
0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Change line 6 in your first example to

| search LoS != 1 OR isnull(LoS)

and see if they reappear.

angelinealex
Communicator

Its not working. It always has value with {1}

0 Karma

rmuraly
Explorer

In Example 1, Los always has a value = 1, since we strip off {} using foreach at the top before applying the condition check.

0 Karma

micahkemp
Champion

I don't have an answer, but I wonder if it would help you (and us) track down the issue if you look at the events that are not returned, perhaps with a search like this:

 index="index1" tdr=tdr1 OR tdr=tdr2
 transaction id
 | foreach * [ rex field=<<FIELD>> mode=sed "s/{|}//g"]
 | eval _time=strptime(Qtime,"%Y-%m-%d %H:%M:%S.%3N") 
 | eval LoS=....
 | search LoS = 1
 | table .....

It's worth pointing out that since we can't see your eval expression for LoS, it's tough to see the full picture of why searching for !=1 and !={1} may return different sets of results.

0 Karma

angelinealex
Communicator

Updated my queries. Please have a look.
tdrs always have the values with {} eg: {H} or {1}

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...