Splunk Search

Issue with props.

theouhuios
Motivator

Hello

This is the props.conf which I have now

[xxx]
TIME_FORMAT = %a %m/%d/%Y %H:%M:%S.%2N (Is this supposed to be %3N? Even then it doesn't work. Becuase of this I have a regex to extract the time.But somehow even that doesn't populate timestamp field.Instead to make the regex work,I have to call it with "time" as field)
SHOULD_LINEMERGE = False
LINE_BREAKER =  \r\n\s+\(.*\)([\r\n])
TRUNCATE = 999999
MAX_TIMESTAMP_LOOKAHEAD = 12

and this is how each event is in the data

Tue 06/25/2013 19:55:46.91 

SERVICE_NAME: TL_SearchSvr 
        TYPE               : 10  WIN32_OWN_PROCESS  
        STATE              : 4  RUNNING 
                                (STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)

I am kinda lost on why is the timestamp field not being extracted. Any help please

Tags (1)
0 Karma

Ayn
Legend

I think you're making this much more complicated than it needs to be. All you need to do to make everything work properly is make sure your timestamps are being recognized - that should also solve your event breaking problems, because by default Splunk breaks into a new event whenever it recognizes a valid timestamp.

The obvious problem I see with your config is that you have MAX_TIMESTAMP_LOOKAHEAD set to 12, so you're telling Splunk to only look at the first 12 characters and then try to match those against your TIME_FORMAT. So for your sample event, it tries to match Tue 06/25/20 (rest is cut off). You should just remove the MAX_TIMESTAMP_LOOKAHEAD altogether, along with...well, the rest of your configuration directives really 🙂 Remove LINE_BREAKER, TRUNCATE, SHOULD_LINEMERGE...Possibly even including TIME_FORMAT because I suspect Splunk will recognize the timestamp out of the box, but this might not be the case.

So the resulting stanza would be something like

[xxx]
TIME_FORMAT = %a %m/%d/%Y %H:%M:%S.%2N

(btw, your TIME_FORMAT should indeed have %2N, not %3N, because there's 2 digits after the decimal point, not 3.)

Also as a general note I find that it's often much easier to work with event breaking settings instead of trying to setup custom line breakers.

asimagu
Builder

try this one:

TIME_FORMAT = %a\s%m/%d/%Y\s%H:%M:%S.%2N
LINE_BREAKER =  \r\n\s+\(.*\)([\r\n])
TRUNCATE = 999999
MAX_TIMESTAMP_LOOKAHEAD = 27

alt text

0 Karma

asimagu
Builder

I was just advising what worked for me when I replicated theouhuios' problem on my laptop, I am editing my answer to add a screenshot of my results.
Of course I always respect what your workmate has to say 😉

0 Karma

Ayn
Legend

_time ALWAYS gets populated. If a string matching your TIME_FORMAT is found in the event that's used. If no match is found Splunk will resort to other methods. See more here: http://docs.splunk.com/Documentation/Splunk/5.0/Data/Configuretimestamprecognition

0 Karma

theouhuios
Motivator

Should Time_format config effect the timestamp field in the data? I thought timestamp field gets populated when the Time_FORMAT config works fine. Am I right?

0 Karma

kristian_kolb
Ultra Champion

You should set SHOULD_LINEMERGE = false if you intend to use LINE_BREAKER. While I like the LINE_BREAKER approach over the SHOULD_LINEMERGE, for performance reasons, I think that Ayn has some solid advice below.

0 Karma

asimagu
Builder

well I did not see the should_linemerge line there, but I don´t think it should affect. when cleaning the index, maybe you can try cleaning the one where the data got indexed and the _thefishbucket
I´m guessing the problem is around the issue of reindexing the events with this new config, either that or create new test events

0 Karma

theouhuios
Motivator

still doesnt work. Di i need to remove Should_linemerge? I don't think will affect it.

0 Karma

theouhuios
Motivator

Yup, I know how to clean the index. But its still test data so no worries about it.

0 Karma

theouhuios
Motivator

oh.. missed that.. let me try that..I guess I missed that part while checking things.

0 Karma

asimagu
Builder

did u notice that I changed the MAX_TIMESTAMP_LOOKAHEAD ?

0 Karma

asimagu
Builder

I replicated your log on my laptop and it worked.
The data that was already indexed is not going to change unless you clean the index, do u know how to do that?

0 Karma

theouhuios
Motivator

Still the same.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...