Splunk Search

field extractor question with regex

gnovak
Builder

I'm trying to use the field extractor for this to prevent having to do a restart and putting in props.conf

2012-11-26 05:39:35.933549-> Finished CREATE_IDX for REPLICA->[{'dbs': 'edr', 'host': 'boo-bee-db1.he1.booberry-boo.eeek', 'port': 5492, 'user': 'etl_user'}]. Time: 01:27:12.449

2012-11-26 13:49:51.935960-> Finished CREATE_IDX for BOOBERRY->[{'dbs': 'edr', 'host': 'boo-bee-db1.he1.booberry-boo.eeek', 'port': '5491', 'user': 'etl_user'}]. Time: 02:16:58.666

I'm interested in pulling out the Time in both lines and for each Time, putting it into a separate field. Example: Time in first line would be called "replicaend" field. Time in second line would be called "booberryend" field.

When Using the field extractor, i gave it the time and it generated results. I narrowed it down to the time I wanted. After testing it appeared it was still giving me additional results I didn't want. Here's the regex the field extractor made:

(?i)^(?:[^:]*:){7}\\s+(?P*FIELDNAME>[^\\.]+)

NOTE: I put a * instead of a less than symbol because it wasn't showing up if i used a less than symbol!

From this regex, is there a way to include the Finished CREATE_IDX for REPLICA-> pattern in the regex when extracting as well? I tried but wasn't able to get it to match anything. Using the search bar, this search gives me exactly what I want. I could put something like this in props.conf but I'm trying to use the field extractor, avoid a restart and see my old data extracted right away.

sourcetype="BOO" | rex "Finished CREATE_IDX for AFILIAS.*Time: (?P*Finished>[0-9:]*)" | top 50 Finished

NOTE: I put a * instead of a less than symbol because it wasn't showing up if i used a less than symbol!

Any ideas?

Tags (1)
0 Karma

jonuwz
Influencer

would something like this do :

... | rex ".*Finished CREATE_IDX for (?<created_index>[\w]+).*Time: (?<index_create_finish_time>[0-9.:]+)$" | table created_index index_create_finish_time

Then you can have the name of the index and the finish time in 2 seperate fields.

You could just drop this EXTRACT into props.conf

i.e.

[your_source_type]
EXTRACT-index_create = .*Finished CREATE_IDX for (?<created_index>[\w]+).*Time: (?<index_create_finish_time>.+)$

then run

| extract reload=t

to pick up the change

0 Karma

gnovak
Builder

It's crazy but I've tried this over and over and it just doesn't work. I noticed all of my other extractions in props.conf usually start with like (?i) and such. This one doesn't. Not sure why it's not working. Works through search but trying to do this automatically isn't working. 😞 I even tried putting (?i) in the front and nothing. It works great in search...but automatic extractions are a no go. ;(

0 Karma

jonuwz
Influencer

ah.

run | extract reload=t as a single search (nothing before it)
this re-reads the props.conf

then you can search sourcetype=edr

0 Karma

jonuwz
Influencer

Work here. Updated answer with what I have in my props.conf

I just put this in system/local/props.conf

0 Karma

gnovak
Builder

i was putting mine in /opt/splunk/etc/system/local. The ones I did through the field extractor are in /opt/splunk/etc/apps/search/local/props.conf. I tried adding an extraction manually using field extractor. It put this in my props.conf for the search app and I still do not see these fields showing up. EXTRACT-IndexTime = Finished CREATE_IDX for (?P[\w]+).Time: (?P[0-9:]). I tried searching for just sourcetype=edr | extract reload=t and I still didn't see these new fields. If I do this regex manually it works fine. I tried field extractions in manager too. 😞

0 Karma

jonuwz
Influencer

It should work straight away. All the field extractor does is put EXTRACT-xxx entries in props.conf.

search time extractions affect all your data.
index time extractions only affect new data

0 Karma

gnovak
Builder

also yes your regex is good to extract the name of the index too. good call on that one. That also works great!

0 Karma

gnovak
Builder

I don't see any of the data yet because I think only new data will work once you do props.conf, isn't that correct?

0 Karma

gnovak
Builder

well I added something to props.conf and did a restart to make the changes happen. I was wondering if it could be done using the field extractor instead by editing the regex right there. I tried some stuff but it didn't take using field extractor.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...