Getting Data In

How do I recognize a time in epoch seconds?

bselig
Explorer

Total newbie here.

I have a data file (a few lines here):

1280718483,204.28.227.23:53;5;5.49;13;2183;2183;0;0;0-2103;2-0;3-48;5-32;15-0;*-0;2183;0;0;0;0
1280718543,204.28.227.23:53;5;5.75;6;16;16;0;0;0-16;2-0;3-0;5-0;15-0;*-0;16;0;0;0;0
1280804716,204.28.227.23:53;4;6.74;77;2412;2412;0;0;0-2332;2-0;3-48;5-32;15-0;*-0;2410;2;0;0;0
1280804776,204.28.227.23:53;5;5.57;14;2391;2391;0;0;0-2343;2-0;3-0;5-48;15-0;*-0;2391;0;0;0;0

The actual file has 500+ lines (events?) going back several months.

The first number in each line (e.g.128071848) is the date in seconds since the epoch.

How can I get splunk (using 4.1.5) to recognize this as the date?

The file is called "tns-stats-0.log.0" located in /home/lis/log/lis and I have the following in etc/system/local/props.conf.

[source::.../lis/tns-stats-0.log.0]
TIME_FORMAT=%s

which is supposed to, from what I can gather, treat the format as seconds since epoch.

Yet, splunk insists on assigning all of the events the time associated with the file itself.

Someone please tell me what I'm missing here. Based on what I've read in other answers and the splunk docs, this should work.

1 Solution

bselig
Explorer

The REAL answer is that you appear to have to use sourcetype and not just [source::] in props.conf:

[tns-stats]
TIME_FORMAT=%s

AND, then you have to define the sourcetype in apps/search/input.conf:

[monitor:///home/lis/log/lis/tns-stats-0.log.0]
sourcetype = tns-stats

which seems a bit odd to me since I thought the global spec would be seen before the app level spec, but then what do I know.

Anyway, this now works.

View solution in original post

bselig
Explorer

The REAL answer is that you appear to have to use sourcetype and not just [source::] in props.conf:

[tns-stats]
TIME_FORMAT=%s

AND, then you have to define the sourcetype in apps/search/input.conf:

[monitor:///home/lis/log/lis/tns-stats-0.log.0]
sourcetype = tns-stats

which seems a bit odd to me since I thought the global spec would be seen before the app level spec, but then what do I know.

Anyway, this now works.

ftk
Motivator

As for markup in comments, you can use the backtick (above the ~) to escape code.

bselig
Explorer

Thanks to ftk for nudging me towards the sourcetype route. I did the "answer my own question" so I could better format a succinct answer fo those that come after.

0 Karma

ftk
Motivator

Add the following to props.conf:

TIME_FORMAT=%s
TIME_PREFIX=^

bselig
Explorer

CRAP! - what I put gets munged together by this input box - the "[tns-stats]" and "TIME_FORMAT=%s" should be on separate lines. Same for the "[monitor:///home/lis/log/lis/tns-stats-0.log.0]" and "sourcetype = tns-stats" that goes in input.conf.
--- hope that's readable (is there markup for these comment boxes so one can be more informative?)

bselig
Explorer

OK - here's what seems to be necessary.

First - in etc/system/local/props.conf put the sourcetype and the format:
[tns-stats]

TIME_FORMAT=%s

Doing it with a source path spec seems to not take.

THEN (and this seems to be the secret sauce), in etc/apps/search/input.conf have

[monitor:///home/lis/log/lis/tns-stats-0.log.0]

sourcetype = tns-stats

and then it assigns the right timestamp to each event line.

0 Karma

bselig
Explorer

Yeah. The open question I had was if the props.conf file is hitting. I did try the full absolute path to the file to no avail.
One of the things that isn't real clear to a newbie like myself is which of the various props.conf one should be modifying. This is currently in system/local. Will try sourcetype next.

0 Karma

ftk
Motivator

Hmm, are we certain that your props.conf line is hitting? With the ... it should hit, but would you mind using the full path to the log file (or using sourcetype instead) and trying this again?

0 Karma

bselig
Explorer

Have tried that in that order and reversed (though maybe the prefix info should be seen first). After I changed props.conf in etc/system/local I stopped splunk, cleaned evendata and restarted splunk.
Still getting the same results where in the events table it shows "_time" as the file time and "timestamp" as 'none'.

Seems like your suggestion is how it should work (that's what I've been trying), but it insists on not behaving that way and I'm at a loss as to where to look to see why. Any insight into splunk logs that might have info about either not finding the data or not interping it right?

0 Karma

cfrantsen
Explorer

Try adding TIME_PREFIX=^ and perhaps take a look at MAX_TIMESTAMP_LOOKAHEAD if the rest of the event text might include something that looks like an epoch time.

0 Karma

bselig
Explorer

No luck. I added
TIME_PREFIX=^
and then did
splunk stop
splunk clear eventdata
splunk start
and the data from that file still shows up identified with "timestamp=none" and the time of all the events reading as the file time.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...