Splunk Enterprise Security

In Splunk Enterprise Security, why is the eval field from our correlation search missing in a notable event?

arlombar
Explorer

I have a correlation search in which I use a simple eval command to create a new field (ex. eval test=123). This fields shows in the search, however, when I set this search as an alert, the eval field I created is missing on the notable event. How do I ensure that this custom field is being sent along with the other data that is sent by default as a notable event?

0 Karma

smoir_splunk
Splunk Employee
Splunk Employee
0 Karma

arlombar
Explorer

As I stated above, I've followed these steps and am still not getting the correct result. When I look under the notable index, then the event in question, I cannot see the eval field I created which shows in the correlation search. Is there anything else that would cause this not to work?

0 Karma

lakshman239
SplunkTrust
SplunkTrust

The key thing is that you need to add your custom field (e.g test) to Incident Review - Event Attributes. [ under Config->Incident Review Settings ].

you can follow the steps in the link from p_gurav above

0 Karma

arlombar
Explorer

I've checked this setting was in place as well, but I am still not getting the data. Under the notable index, then the event in questions, the eval field I created is not present even though it shows on the correlation search.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

in your eval, you are using ceil to roundup. If you just want to capture the time to a field, you can do eval first_date = _time . or first_time = now() . There is also an existing incident review field called Modication Time, so you could use eval modtime = _time or modify_time = _time to capture time . would this help?

0 Karma

arlombar
Explorer

Thanks for your reply. The ceil function was more of just an example, but I tried with your recommendations and still get the same results, no field showing in the notable event, but shows on the correlation search results. The only work around I've found so far is to create the field using the calculated field option, which does work, but this is for all events which is not idea. I'm not sure what I am doing wrong here.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

did you try .... your base search | eval mycustomField = now() | fields + mycustomField as per step 2 in https://docs.splunk.com/Documentation/ES/5.1.1/Admin/Customizenotables#Add_a_field_to_the_notable_ev...?

Basically, once you a field defined (your custom field) and added in the 'Incident review-> event attributes, you can force it to appear using fields + . let me know if this fixes and also indicate the version of splunk core and ES.

0 Karma

arlombar
Explorer

I just tried using a new simple search with the same eval command you referenced with the fields function, however it seems like you need to add each field individually which is not ideal. Regardless, the new search triggered on an event, I checked the notable event index to look at what was sent over and again the custom eval field is missing. I've also added this field as an event attribute, but this does not matter if the field/data are not found when passing to the notable index. Again just for clarification, my issue is that the correlation search shows the custom field I created in the results with data, when this is a set as an alert and sent to the notable events index, the notable event is now missing that field (both when I tried to look for it using index=notable | search custom_field=* and under the incident review dashboard.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

Just to be clear, if you want your custom field to appear in Incident review dashboard against the list of fields in the notable/alert, you need to create that field in Incident-Review -> Event Attributes. There are already about 200+ fields available which can be used to your needs, by way of renaming, e.g. your search/event has name as 'lastname', you can use | eval user_last = lastname, to make use of existing notable event fields. That way you can reuse an existing field. As far as i know, if you cannot re-use an existing field, you will need to define/add it before it can be displayed in the incident review screen for that notable.

0 Karma

arlombar
Explorer

Yes, I've gone through adding about a dozen event attributes and have never had this issue before. I have other alerts in which use eval commands and they are passing the eval field I created along to the notable event when generated (I didn't even need to configure anything on incident review to get this to work). I find it strange that in other alerts eval commands are working and when you search for the event in the notable index they are showing. Would a recycle of the services be the best move at this point or is there some way to refresh Splunk from within the UI? Never experienced anything like this before.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

may be delete the correlation search, restart splunk instance and re-add/create a new search and check it out.

0 Karma

arlombar
Explorer

Splunk: 7.0.4
ES: 5.0.1

0 Karma

p_gurav
Champion
0 Karma

arlombar
Explorer

Thanks for the link, I've followed all of these steps and am still not getting the results I need. When I look at the notable event index and at the event in question, the eval field I created within the correlation search is not present.

0 Karma

smoir_splunk
Splunk Employee
Splunk Employee

So you're creating the field in the search but it's not showing in the results? I'd test this out further separate from a correlation search, and run the search directly in Splunk and see if there is something incorrect in the search syntax.

0 Karma

arlombar
Explorer

When I run the search manually and look at the results I can see the field I created. I have this search set as an alert, once triggered it sends the event to the notable event index. When I go and look at the notable event index and specifically at this event my eval field I created does not come over with the other data. Everything else comes over as expected, its just this eval field I created in the search is no longer present.

0 Karma

smoir_splunk
Splunk Employee
Splunk Employee

That's odd. Can you share the search syntax (even if it's a bit obscured)?

0 Karma

smoir_splunk
Splunk Employee
Splunk Employee

Also, does the same thing happen if you search the notable index using the notable macro rather than searching the index directly?

0 Karma

arlombar
Explorer

I looked at the index and macro and had the same problem. I can't post the search, but I can show an example of the eval statement I am writing, If something seems off please let me know.

eval first_date=ceil(_time), second_date=ceil(_time)*1000

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...