Getting Data In

Troubleshooting props.cof and transforms.conf

zislin
Explorer

I am trying to add new app to do some field extraction at index time. I've used a working app to get my folders and files created. I've tested my regex in online regex tester tools. Restarted service but no luck.

Can somebody direct me to which processor I need to set to debug logging mode to troubleshoot my issue?

Thanks
= ==============================
we have multiple servers performing different roles in splunk. So to make search heads life easier, I was thinking on doing this particular extraction on heavy forwarder.

Here is the props file:

[someDevice]
MAX_TIMESTAMP_LOOKAHEAD = 15
TIME_FORMAT = %b %d %H:%M:%S
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRANSFORMS-getMyFields = get_type

Here is transforms:

[get_type]
REGEX = ]\:\s(.+?\s[\d]?[\d]?[\d]?)
FORMAT = event_type::$1

My regex is a bitty cumbersome but that is due to various data coming.
Sample of data is as follows:

Sep 11 15:47:20 111.111.111.111 blabla[123]: sometype on bla bla bla the rest doesnt matter
Sep 11 15:47:20 111.111.111.111 blabla[123]: othertype at bla bla bla the rest doesnt matter
Sep 11 15:47:20 111.111.111.111 blabla[123]: newtype 11 for bla bla bla the rest doesnt matter

I want to extract "sometype", "othertype", "newtype 11".
Last one makes my extraction tricky.

0 Karma

kristian_kolb
Ultra Champion

Like lguinn says, do it search-time on the SH instead;

props.conf

[your_sourcetype/source/host]
EXTRACT-blah = \]:\s(?<event_type>\S+(\s\d+)?)

Should work.

/K

0 Karma

kristian_kolb
Ultra Champion

zislin,

I think you should escape the initial closing square bracket. And you don't need to put your \d's in character classes ([]).

If this regex works in pcregextest - fine - but does it work in Splunk? I think you should test that first, before committing it to your production environment.

0 Karma

zislin
Explorer

At this point, the only thing I am trying to do, is to extract one field from data coming from one existing sourcetype. App is not visible. That's it

0 Karma

sowings
Splunk Employee
Splunk Employee

The answer depends greatly upon the application.

0 Karma

zislin
Explorer

I've tested my regex with pcregextest command line tool and it was happy.

My regex is
]\:\s(?<event_type>.+?\s[\d]?[\d]?[\d]?)

Does anybody know if I have to restart Splunk services when I am adding a new app via command line on search head?

0 Karma

kristian_kolb
Ultra Champion

hi lguinn, that could be good to know, though I tested it and it did not complain on the nested, so-to-speak, capturing groups.

/K

0 Karma

lguinn2
Legend

I would use the following regex, though

[your_sourcetype/source/host]
EXTRACT-blah = \]:\s(?<event_type>\S+(?:\s\d+)?)

As I have found that sometimes Splunk is picky about capturing vs. non-capturing groups. Or even

[your_sourcetype/source/host]
EXTRACT-blah = \]:\s(?<event_type>.*?)\s(?:on|at|for)\s

0 Karma

kristian_kolb
Ultra Champion

The example I provided does indeed work. Tested on the sample events you provided. /K

0 Karma

kristian_kolb
Ultra Champion

just what does your regex look like. What you have in your original question has errors.

0 Karma

zislin
Explorer

Alright. I moved my extraction to SH and did a search-time extraction via props.conf. No luck.
WHen I search for my source type I dont see that new field appear. I have only done debug/refresh not service restart. In addition, I've tested my regex via pcregextest command and it was successful.

I need to find out which debugger to turn on, to see errors. thx

0 Karma

lguinn2
Legend

No idea. However, if you post props.conf and transforms.conf stanzas here, someone will probably be able to help you.

Also, I might as well be the first to ask: why an index time field extraction? A search time field extraction is just as fast, easier to create, and much more flexible.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...