Splunk Search

extracting search fields using regexp in transforms.conf not working

builder
Path Finder

I refered to the following documentation to try and get this working:

http://www.splunk.com/base/Documentation/3.0.2/Admin/CreateAdditionalSearchFields

I have a standard rails log file where every new request starts with a line like:

Started (GET|POST) "/some-path"

I want to extract "/some-path" into a field rails-action-path. I added the following to $SPLUNK_HOME/etc/system/local/transforms.conf (note that the documentation says to add it to $SPLUNK_HOME/etc/bundles/local/transforms.conf, but there is no bundles directory in my installation and my other conf files have all been in the system directory).

[get-rails-action-path]
REGEX = Started \S ([\S\/"\.]+)
FORMAT = $0 rails-action-path::$1
DEST_KEY = _meta

My sourcetype stanza for the rails logs in $SPLUNK_HOME/etc/apps/search/local/props.conf is as follows (again note, my props.conf has always lived here although the documentation refers to $SPLUNK_HOME/etc/bundles/local/props.conf).

[(?:::){0}*rails]
LINE_BREAKER = ([\r\n]).* [\r\n]+Started (POST|GET)
TRANSFORMS-rap = rails-action-path

Note that the last line was the only part I just added. The wildcard source type for the rails logs and the LINE_BREAKER have been there for awhile and behave as I expect.

After restarting my indexer with these changes, I do not see rails-action-path in the field list.

Any ideas?

0 Karma
1 Solution

southeringtonp
Motivator

You're looking at pretty old documentation. You're not really using Splunk version 3.0.2, right? All of the "bundles" stuff went out a long time ago and was replaced by "apps". Start here for the documentation, or use the "product version" dropdown in the upper left to make sure you're looking at something current.

Putting it in etc/system/local is alright, though the more common place in Splunk 4.x would be in /etc/apps/search/local, or in a new app of your own creation.

Suggestions / Issues:

  • The regex is a little off. You need \S+ (note the plus sign) to match more than one character.
  • You can leave out the $0
  • Use REPORT instead of TRANSFORMS for search-time extraction, and make sure that the name referenced in props.conf matches the stanza name used in transforms.conf.
  • DEST_KEY isn't needed

With those things in mind, try this:

#transforms.conf
[get-rails-action-path]
REGEX = Started \S+ ([\S\/"\.]+)
FORMAT = rails-action-path::$1

#props.conf
[(?:::){0}*rails]
LINE_BREAKER = ([\r\n]).* [\r\n]+Started (POST|GET)
REPORT-rap = get-rails-action-path

(Assuming your stanza name in props.conf is ok, since you say it's working for line breaker.)

View solution in original post

southeringtonp
Motivator

You're looking at pretty old documentation. You're not really using Splunk version 3.0.2, right? All of the "bundles" stuff went out a long time ago and was replaced by "apps". Start here for the documentation, or use the "product version" dropdown in the upper left to make sure you're looking at something current.

Putting it in etc/system/local is alright, though the more common place in Splunk 4.x would be in /etc/apps/search/local, or in a new app of your own creation.

Suggestions / Issues:

  • The regex is a little off. You need \S+ (note the plus sign) to match more than one character.
  • You can leave out the $0
  • Use REPORT instead of TRANSFORMS for search-time extraction, and make sure that the name referenced in props.conf matches the stanza name used in transforms.conf.
  • DEST_KEY isn't needed

With those things in mind, try this:

#transforms.conf
[get-rails-action-path]
REGEX = Started \S+ ([\S\/"\.]+)
FORMAT = rails-action-path::$1

#props.conf
[(?:::){0}*rails]
LINE_BREAKER = ([\r\n]).* [\r\n]+Started (POST|GET)
REPORT-rap = get-rails-action-path

(Assuming your stanza name in props.conf is ok, since you say it's working for line breaker.)

builder
Path Finder

Just going to start a new thread as this one seems to have died. : P

0 Karma

builder
Path Finder

I was using the instructions here for search time extraction and EXTRACT seemed simpler:

http://www.splunk.com/base/Documentation/4.2.1/Knowledge/Createandmaintainsearch-timefieldextraction...

Also, the original confs were on my indexer. The EXTRACT is in props.conf on my search-head. The field is showing up using this method, but it didn't solve the problem. It's pretty much the same as before. I see the field and all extracted values. If I search by rails_action_path="*", I get all results, but any specific value returns 0 results/No matching events found.

0 Karma

southeringtonp
Motivator

Not sure why it wouldn't show up and say "waiting for events", unless having DEST_KEY in there was interfering with things, which is quite possible.

0 Karma

southeringtonp
Motivator

Using REPORT as above, it will be a search-time transaction. DEST_KEY shouldn't be in there though; I copied and pasted from your example and failed to remove it until just now.

0 Karma

builder
Path Finder

So, I was reading over the 4.2.1 docs, per your advisement, and it sounds like the recommendation is actually search-time field extraction. Thus, I have reverted all the changes mentioned in this thread to my transform.conf and props.conf on my indexer machines. I have instead set up search-time extraction in my props.conf, but it's not working either (the field is back to not showing up). I will start a new thread for that.

0 Karma

builder
Path Finder

I updated the conf files per the above and restarted my indexer. Now, rails-action-path shows up on my summary page as a field with properly parsed values. If I click on the field and select 'show only events with this field', it does the right thing (searching with sourcetype="space-rails" rails_action_path="*" and showing numerous events). However, if I select any specific listed value for the field (e.g., searching with sourcetype="space-rails" rails_action_path="/client_api/updateGameData"), it returns 0 events and says "Waiting for data" indefinitely.

What could be going on here?

0 Karma

builder
Path Finder

I'm going to have to reply in two comments since it has a comment limit.

You are correct, I am using 4.2.1. I didn't notice the documentation was old. Sorry about that.

The missing + on the regexp and the non-matching stanza name were both just brain farts on my part, but I probably wouldn't have caught them after staring at this for so long.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...