I'm trying to use the field extractor for this to prevent having to do a restart and putting in props.conf
2012-11-26 05:39:35.933549-> Finished CREATE_IDX for REPLICA->[{'dbs': 'edr', 'host': 'boo-bee-db1.he1.booberry-boo.eeek', 'port': 5492, 'user': 'etl_user'}]. Time: 01:27:12.449
2012-11-26 13:49:51.935960-> Finished CREATE_IDX for BOOBERRY->[{'dbs': 'edr', 'host': 'boo-bee-db1.he1.booberry-boo.eeek', 'port': '5491', 'user': 'etl_user'}]. Time: 02:16:58.666
I'm interested in pulling out the Time in both lines and for each Time, putting it into a separate field. Example: Time in first line would be called "replicaend" field. Time in second line would be called "booberryend" field.
When Using the field extractor, i gave it the time and it generated results. I narrowed it down to the time I wanted. After testing it appeared it was still giving me additional results I didn't want. Here's the regex the field extractor made:
(?i)^(?:[^:]*:){7}\\s+(?P*FIELDNAME>[^\\.]+)
NOTE: I put a * instead of a less than symbol because it wasn't showing up if i used a less than symbol!
From this regex, is there a way to include the Finished CREATE_IDX for REPLICA-> pattern in the regex when extracting as well? I tried but wasn't able to get it to match anything. Using the search bar, this search gives me exactly what I want. I could put something like this in props.conf but I'm trying to use the field extractor, avoid a restart and see my old data extracted right away.
sourcetype="BOO" | rex "Finished CREATE_IDX for AFILIAS.*Time: (?P*Finished>[0-9:]*)" | top 50 Finished
NOTE: I put a * instead of a less than symbol because it wasn't showing up if i used a less than symbol!
Any ideas?
would something like this do :
... | rex ".*Finished CREATE_IDX for (?<created_index>[\w]+).*Time: (?<index_create_finish_time>[0-9.:]+)$" | table created_index index_create_finish_time
Then you can have the name of the index and the finish time in 2 seperate fields.
You could just drop this EXTRACT into props.conf
i.e.
[your_source_type]
EXTRACT-index_create = .*Finished CREATE_IDX for (?<created_index>[\w]+).*Time: (?<index_create_finish_time>.+)$
then run
| extract reload=t
to pick up the change
It's crazy but I've tried this over and over and it just doesn't work. I noticed all of my other extractions in props.conf usually start with like (?i) and such. This one doesn't. Not sure why it's not working. Works through search but trying to do this automatically isn't working. 😞 I even tried putting (?i) in the front and nothing. It works great in search...but automatic extractions are a no go. ;(
ah.
run | extract reload=t
as a single search (nothing before it)
this re-reads the props.conf
then you can search sourcetype=edr
Work here. Updated answer with what I have in my props.conf
I just put this in system/local/props.conf
i was putting mine in /opt/splunk/etc/system/local. The ones I did through the field extractor are in /opt/splunk/etc/apps/search/local/props.conf. I tried adding an extraction manually using field extractor. It put this in my props.conf for the search app and I still do not see these fields showing up. EXTRACT-IndexTime = Finished CREATE_IDX for (?P
It should work straight away. All the field extractor does is put EXTRACT-xxx entries in props.conf.
search time extractions affect all your data.
index time extractions only affect new data
also yes your regex is good to extract the name of the index too. good call on that one. That also works great!
I don't see any of the data yet because I think only new data will work once you do props.conf, isn't that correct?
well I added something to props.conf and did a restart to make the changes happen. I was wondering if it could be done using the field extractor instead by editing the regex right there. I tried some stuff but it didn't take using field extractor.