Hello, New to using base searches, and could not find the answer to my issue.
I am trying to convert a Antivirus dashboard used by the desktop team to a base search, in hopes to improve performance and be less of a hit on the search heads. Dashboard takes data from two products, gives a near real time refresh of status.
The issue I am experiencing is one of the panels is a graph for the desktop lead. By itself its fine, the base search, then sub search misses one catagory, but when going to the search on that panel, the catagory is there in the chart.
Here is the base search I am using.
<search id="baseSearch">
<query>index=sophos sourcetype="sophos:threats" | regex FullFilePath!="eicar" | regex FullFilePath!="pagefile.sys" | eval _time=strptime(InsertedAt,"%Y-%m-%d %H:%M:%S.%N") |rename user as User |search (Status="Cleanable" OR Status="Not Cleanable" OR Status="Cleanup failed" OR Status="Threat type not cleanable") </query>
<earliest>-10d@d</earliest>
<latest>-0d@d</latest>
Here is the sub search.
<search base="baseSearch">
<query> |stats count by ActionTaken, Status, _time |timechart span=1d count(Status) by ActionTaken | append [ search (Status="Cleanable" OR Status="Not Cleanable" OR Status="Cleanup failed" OR Status="Threat type not cleanable") |stats count by ActionTaken, Status, _time |timechart span=1d count(Status) by Status ]|
timechart first(*) as *</query>
</search>
Here is what the dashboard panel looks like.
Here is what I get when I search. (via the dashboard panel)
I am sure its something simple I am missing. Thanks in advance
Let's start with cleaning up your SPL. I'm not sure why the append either.
Base Search
index=sophos sourcetype="sophos:threats" NOT ( FullFilePath="eicar" OR FullFilePath="pagefile.sys") Status="Cleanable" OR Status="Not Cleanable" OR Status="Cleanup failed" OR Status="Threat type not cleanable")| eval _time=strptime(InsertedAt,"%Y-%m-%d %H:%M:%S.%N")
Post Process
timechart count by Status span=1d
TO NOTE: Passing raw events from a base to post process is not recommended, as there is a limit of either 10K or 50K events that can be passed. It is best to transform the base search results prior to passing to another search.
If you could also enlighten us as to why the use of such a broad append
we can refine this query further towards your goal.
Edit: Updated post process to do the correct counting.
Let's start with cleaning up your SPL. I'm not sure why the append either.
Base Search
index=sophos sourcetype="sophos:threats" NOT ( FullFilePath="eicar" OR FullFilePath="pagefile.sys") Status="Cleanable" OR Status="Not Cleanable" OR Status="Cleanup failed" OR Status="Threat type not cleanable")| eval _time=strptime(InsertedAt,"%Y-%m-%d %H:%M:%S.%N")
Post Process
timechart count by Status span=1d
TO NOTE: Passing raw events from a base to post process is not recommended, as there is a limit of either 10K or 50K events that can be passed. It is best to transform the base search results prior to passing to another search.
If you could also enlighten us as to why the use of such a broad append
we can refine this query further towards your goal.
Edit: Updated post process to do the correct counting.
The issue I ran into, without the append I would only get one column, like this.
index=sophos sourcetype="sophos:threats" | regex FullFilePath!="eicar" | regex FullFilePath!="pagefile.sys" | eval _time=strptime(InsertedAt,"%Y-%m-%d %H:%M:%S.%N") |rename user as User |search (Status="Cleanable" OR Status="Not Cleanable" OR Status="Cleanup failed" OR Status="Threat type not cleanable") | stats count by ActionTaken, Status, _time |timechart span=1d count(Status) by ActionTaken
Adding the append gives me this.
Since this is my first time trying the base search option, I am completely open to suggestions and other options.
I will adjust my searches with your suggestions, thank you for them.
Edit. The other reason for the second search was to add the action, looking to see how many were blocked.
@ alacercogitatus, in line with what you have said while base search should not pass on raw data rather should have statistically aggregated data, following are the two options @bworrellZP should try.
Option 1
Make the last function in the base search | field *
. This is obviously not recommended but might work.
Option 2
Move the first stats command on the post process search as the last command in the base search instead |stats count by ActionTaken, Status, _time
Do read about limitations of Post Process Search. Somethimes you are better off running separate searches than post processing: https://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches#Post-process_searches