Splunk Search

Can you help me with my field extraction weirdness?

a212830
Champion

Hi,

I have a field extraction situaton that I've never come across before, and hoping someone can help me.

We have a number of fields setup to do search-time extractions and transformations. One of the fields is named "action", which looks at the values in the field and transforms them. The transformation works when you do a query that doesn't directly query that field, but if you query the field directly, it isn't found. However, if you wildcard it, the field is found.

Here's my transforms.conf:

[stonesoft_action_blocked]
REGEX = |(Connection_Discarded)|
FORMAT = action::blocked

[stonesoft_action_teardown]
REGEX = |(Connection_Closed(?:-Abnormally)?)|
FORMAT = action::teardown

[stonesoft_action_allowed]
REGEX = action=(Allow|Permit)
FORMAT = action::allowed

If I query "index=myIndex", then the field "action" field appears under "Interesting Fields", with each option — teardown, allowed, and blocked". However, if I click on any of these values, and they get added to the search, it now comes back with zero events. So, "index=myIndex action=blocked" returns nothing. If I enter that directly in the search (rather than clicking on it from the event), it also returns zero events.

If I wildcard the search, and type "index=myIndex action=*blocked*", then I get events returned.

Hope this makes sense. Appreciate any advise.

1 Solution

woodcock
Esteemed Legend

You have to tell the Search Head that these fields are not indexed values (they do not fall between to major/minor breakers) by adding this to fields.conf:

[action]
INDEXED_VALUE = false

See details here:
https://www.splunk.com/blog/2011/10/07/cannot-search-based-on-an-extracted-field.html

View solution in original post

woodcock
Esteemed Legend

You have to tell the Search Head that these fields are not indexed values (they do not fall between to major/minor breakers) by adding this to fields.conf:

[action]
INDEXED_VALUE = false

See details here:
https://www.splunk.com/blog/2011/10/07/cannot-search-based-on-an-extracted-field.html

sloshburch
Splunk Employee
Splunk Employee

Also, would this all be resolved if he didn't do this type of indexed field approach and used props.conf with a case or if eval to calculate the field?

0 Karma

FrankVl
Ultra Champion

It isn't indexed, it is search time already (REPORT in props.conf). Just the way it is being done in transforms.conf, with those static values, has some disadvantages that require that setting that woodcock mentioned in order for Splunk to properly handle this field in searches.

0 Karma

a212830
Champion

Thanks. Can you elaborate? If there a better alternative?

0 Karma

woodcock
Esteemed Legend

If you can do it from an automatic lookup, that is best, otherwise my solution is your best option. Don't forget to UpVote any helpful answers and click Accept, @a212830.

0 Karma

a212830
Champion

Thanks. Answer should be accepted now. When you say lookup, I'm assuming that you mean some sort of eval?

0 Karma

FrankVl
Ultra Champion
0 Karma

sloshburch
Splunk Employee
Splunk Employee

Word. (and thanks for keeping me straight that it is NOT indexed). Check the props.conf details on EVAL-, FIELDALIAS-, EXTRACT-, and so forth. A ton of options.
Honestly, I would suggest going through the UI and creating the field extraction with the Interactive Field Extractor and seeing the config it produces. That's a great way to learn.

BTW: Since we know each other, I recall you used to be primary on the backend while your peer handled more of this UI stuff. Given your new role, might be a good time to get the boss to send you for some of those courses like https://www.splunk.com/en_us/training/learning-path/courses-for-users/advanced-searching-and-reporti...

0 Karma

a212830
Champion

Thanks. Good idea.

And yes, the fields.conf entry fixed the issue. That said, I'll revist how this was done.

0 Karma

FrankVl
Ultra Champion

Typically people will just extract the 'raw' action as defined in the event into some field (e.g. vendor_action) and then use a lookup, or an EVAL with an if/case statement to calculate the normalized 'action' field.

0 Karma

a212830
Champion

Hey Burch,

My intent is not to make these indexed fields. Is there something in this setup that is making these appear as indexed fields?

0 Karma

FrankVl
Ultra Champion

So just for my understanding, that is because he uses a REPORT transform which assigns a static value which doesn't occur in the event itself?

When you would use an eval in props.conf instead, that wouldn't be an issue, right?

0 Karma

woodcock
Esteemed Legend

More than that, actually; it is because the string does not occur inside the event OR is in the event but not bounded by segmenters. If the field came from EVAL or LOOKUP, yes, splunk would then understand it. Also, you did not supply your props.conf so I don't know if these are tied to TRANSFORMS- or REPORT-. If the former, you could use action::blocked syntax and it would work.

0 Karma

FrankVl
Ultra Champion

Thanks 🙂

He did share his props in one of the nested comments: https://answers.splunk.com/comments/702388/view.html

It uses REPORT.

0 Karma

CarlosMena
New Member

Hello,
Just started to use splunk and i dont have many knowledge about programing language.
Anyone can help me extragting latitude and Longitude so i can use it in geolocation?
Here is a sample of na event:

"2018-07-26 15:29:59 192.168.1.5 GET /igeoearcweb/igeoesig/ExecCmd.asp Lat=34.224167&Lng=-118.063111&SC=WGS84&Scale=25000&cmd=Center 81 - 192.168.1.5 Mozilla/5.0+(compatible;+SemrushBot/2~bl;++http://www.semrush.com/bot.html) - 404 0 2 0"

Thanks

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Hi Carlos. I think your post might have been better suited as a new question rather than an answer to the question posted on this page. If so, this page may be effective for getting you help: http://docs.splunk.com/Documentation/Splunkbase/latest/Answers/Questions

0 Karma

FrankVl
Ultra Champion

Are you literally searching with those double quotes around your entire query? Have you tried not doing that?

Normally you only put quotes around field values and such, not around the entire search query. I've tried searching like that on my own splunk box and that indeed gives weird behavior with no results when you have multiple search criteria (just "index=bla" works fine, "index=bla field=foo" doesn't work).

0 Karma

a212830
Champion

No, those quotes are not part of the search.

0 Karma

FrankVl
Ultra Champion

What does the props.conf look like for this?

0 Karma

a212830
Champion
[stonesoft]
KV_MODE = none
REPORT-leef = stonesoft_transport_id,stonesoft_dest_port,stonesoft_src_port,stonesoft_dest,stonesoft_src,stonesoft_sender,stonesoft_dvc,stonesoft_action_al
lowed,stonesoft_action_blocked,stonesoft_action_teardown
LOOKUP-transport_protocols = transport_protocols transport_id
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...