Receiving "OR OR" error message

[genesiusj]

Hello,
My colleague and I noticed an issue in the following SPL.
If there is data, the SPL works. If there isn't any events, this error message comes up. How can this situation be handled cleanly?

"Error in 'search' command: Unable to parse the search: Invalid search: OR OR."

`comment("THIS WORKS. 912 events new code. ~2:15 mins")`
index="linuxevents" 
                AND earliest=-24h@h AND latest=@h
                AND host=server1
                AND (source="/var/log/sudo.log" OR source="/var/log/secure" OR source="/var/log/audit/audit.log")

`comment("70 events -- audit.log -- acct or auid -- overlap with first 'OR subsearch'")`
AND [ search index="linuxevents" 
                AND earliest=-24h@h AND latest=@h
                AND host=server1
                AND source="/var/log/audit/audit.log" 
                      AND (acct="user1" OR auid="1014")]

`comment("46 events -- audit.log -- ip address")`
OR [ search index="linuxevents"
                AND earliest=-24h@h AND latest=@h
                AND host=server1
                AND source="/var/log/audit/audit.log"
                      AND acct="user1"
                      AND (auid!="0" AND auid!="4294967295")
                      AND addr!="?"
                          | head limit=1
                          | eval addr="\"".addr."\""
                          | return $addr  ]  

`comment("2 events -- audit.log -- new auid")`
OR [ search index="linuxevents"
                AND earliest=-24h@h AND latest=@h
                AND host=server1
                AND source="/var/log/audit/audit.log"
                      AND (_raw="*new auid*" AND "1014") ]

              | dedup _raw
              | transaction event_id 

`comment("7 events -- secure -- user or ip address.")`
| append
    [ search index="linuxevents" 
                AND earliest=-24h@h AND latest=@h
                AND (source="/var/log/sudo.log" OR source="/var/log/secure")
                AND host=server1
|  rex field=_raw "for\suser\s(?<userId>[\w\-]*)"
|  rex field=_raw "publickey\sfor\s(?<userId>[\w\-]*)"
|  rex field=_raw "\sfrom\s(?<ipAddr>(?:\d{1,3}\.){3}\d{1,3})|\sfrom\s(?<pcName>[0-9a-zA-z\.\-]*)"

| where userId="user1" OR ipAddr="10.10.10.10" ]


              | eval log=case(source=="/var/log/sudo.log", "sudo", source=="/var/log/secure", "secure", source=="/var/log/audit/audit.log", "audit")
              | sort +_time
              | table _time, log, host, addr, ipAddr, auid, acct, userId, pcName, ses, pid, _raw

Thanks and God bless,
Genesius

[maciep]

Well I think the problem is that with no events, your search resolves to something like this, which is invalid

index=blah OR OR

You already have incredibly complicated search here, why not make a bit more complicated. In your subsearches add some logic to determine if there's no events and return something that results to false?

[genesiusj]

@maciep ,

Blockquote why not make a bit more complicated.

😞

I believe that is what I said. If there are no events, then the search is OR OR. You mention "logic", would you please mention where in the documentation I could find information concerning "logic".

Thanks and God bless,
Genesius

[maciep]

I'm having hard time understanding what exactly is getting returned in your subsearches, but maybe for the one returning an address, something like below. This is based on a "similar" scenario of wanting dashboard panel to have "0" instead of "no results found" (https://answers.splunk.com/answers/582253/replacing-no-results-found-with-0.html)

So essentially, append a count of events to your results, and if it's 0, then set the return to something that results to false? Not sure if you have to do it sooner in your search or not, but just take each subsearch by itself and try to get that logic working - finding some way to identify no results and then return something false.

....
| eval addr="\"".addr."\""
| appendpipe [ | stats count | where count=0| eval $addr="1=0" ]
| return $addr  ]  

[genesiusj]

@maciep ,
I want to thank you for your reply. I will check into later this afternoon. I'm prepping for a meeting.
Thanks and God bless,
Genesius

[adonio]

@genesiusj, i read the comments above and couldn't agree more with @skoelpin, also seems like @maciep observation is true.
please share some sample data and your desired result. there must be a better way to accomplish your goal

[genesiusj]

@adonio ,
As I mentioned I agree with them as well, though your computer-side manner is much more inviting.

I see that my comment post showing event results was finally permitted. I think I explained things clearly. If I haven't, please let me know.

I am not in the office today, and I cannot access my search head. I will comment tomorrow morning.

UPDATE: Something I forgot to mention is that those fields acct, auid, addr are all tokens. This SPL is from a single dashboard panel.

Thanks and God bless,
Genesius

[skoelpin]

Why are you appending on the same index? This query looks very expensive and needs to be redone. This is not sustainable and will not scale

[jacobpevans]

I'm going to have to agree with @skoelphin, even the following search for the past 1 minute takes longer than I'm willing to wait:

index=_internal
OR [ search index=_internal ]

[genesiusj]

@jacobpevans ,
I agree with both of you, and in my previous comment, which is right now being held up by the moderators and I don't know why, I explain why I need to search the same index, time and source over and over again.
Using your mini-example above, with some modifications, I'll try to explain again.

index=_internal AND acct="user1"
OR [ search index=_internal AND addr="10.10.10.10"]

Here is the issue. The first event to have acct="user1" is found. However, without the OR (which actually should be an AND) I would never see the first event with the addr="10.10.10.10" in it.

I hope this is making sense. And I hope there is an easier, more efficient way to do this. Either way, my original question hasn't been replied to. If there are no events, I receive an error concerning OR OR. If I change the OR to AND, there error is AND AND. So how to I correct this error?

Thanks and God bless,
Genesius

[genesiusj]

@skoelpin ,
You are correct. This is VERY expensive. However, in order to pull ALL events for a user login/logoff, the same index, time, and source need to (sub)searched multiple times.

When user1 ssh's into the server1 the first event that appears in audit.log is 5 events into the actual login.

type=USER_AUTH msg=audit(1568646439.241:9915945): user pid=4317 uid=0 auid=4294967295 ses=4294967295 msg='op=pubkey_auth rport=5879 acct="user1" exe="/usr/sbin/sshd" hostname=? addr=10.1.10.10 terminal=? res=success'

In this event we also have the addr 10.10.10.10. The addr will used in another subsearch of the entire audit.log based on the same index, time, and source, because the very first event from audit.log contains events with only the addr.

type=CRYPTO_KEY_USER msg=audit(1568646439.123:9915941): user pid=4320 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=28:6a:cb:a7:ab:67:d4:85:ff:34:99:b7:c9:f5:55:5c direction=? spid=4320 suid=0  exe="/usr/sbin/sshd" hostname=? addr=10.10.10.10 terminal=? res=success'

The only information that ties this first event with user1 is the addr of 10.10.10.10.

The same case is for the auid=1014. The first event with auid=1014 is the 12th event.

type=LOGIN msg=audit(1568646439.246:9915951): pid=4317 uid=0 old auid=4294967295 new auid=1014 old ses=4294967295 new ses=21486

The auid was from a dropdown at the top of the dashboard. As you can see from above there is no addr or acct in this event. If we did not know the auid was 1014 using the dropdown, the first event with 1014, and user1 and/or 10.10.10.10 would be the 15th event.

type=USER_START msg=audit(1568646439.258:9915954): user pid=4317 uid=0 auid=1014 ses=21486 msg='op=PAM:session_open acct="user1" exe="/usr/sbin/sshd" hostname=10.37.21.237 addr=10.10.10.10 terminal=ssh res=success'

Either way, the entire audit.log needs to be searched again for events containing only auid=1014.

One last subsearch is performed which uses the event_ids from the previous subsearches with the transaction command, to find all remaining events from the audit.log.

A similar logic is used with secure.log. The first event from secure.log will contain user1.

2019-09-16T11:07:19.260210-04:00 server1 sshd[4317]: pam_unix(sshd:session): session opened for user user1 by (uid=0)

However, when exiting from the Linux box, there is also an event from secure.log which only has the IP address.

2019-09-16T12:07:19.840956-04:00 ruby01-s sshd[4321]: Received disconnect from 10.10.10.10: 11: disconnected by user

I hope that explains it clearer.

Thanks and God bless,
Genesius