How to make the _time from the source path?

[power12]

I have the following source path which I am monitoring..I want splunk to use the _time from the source file.

/svc-logs/host*/ads/stdout.230621-125941

Thanks in Advance

[power12]

Thank you it worked

[isoutamo]

Hi

maybe these helps you?

•  https://community.splunk.com/t5/Getting-Data-In/How-to-get-props-and-transforms-to-extract-time-from...
• https://community.splunk.com/t5/Getting-Data-In/How-to-apply-source-file-date-using-INGEST-as-Time/m...

r. Ismo

[power12]

@isoutamo  Thank you so much for you reply. I tried the below but I am not sure what mistake I am doing

| makeresults 
| eval source="/logs/abc*/mods/stdout.230626-203719" 
| fields - _time
``` above set test data ```
| eval _time = strptime(replace(source, ".*\/stdout.(\d{6}\-\d{6}).*","\1") + tostring(random() % 86400,"duration"),"%Y%m%d%H:%M:%S")

[isoutamo]

Your replace seems to be incorrect. It should be something like 

| eval _time = strptime(replace(source, ".*/stdout.(\d{6}-\d{6}).*","20\1"),"%y%m%d-%H%M%S")

[power12]

@isoutamo  This is what I used and I dont see _time field

| makeresults 
| eval source="/svc-logs/hostabc/mods/stdout.230626-203719" 
| fields - _time
``` above set test data ```
| eval _time = strptime(replace(source, ".*/stdout.(\d{6}-\d{6}).*","20\1"),"%y%m%d-%H%M%S")

[isoutamo]

This _time is on your Time field on this event. Actually you should remove “| fields - _time” if you didn’t see Time on this one event.
Above is just test to get correct version for INGEST_EVAL expression to transforms.conf.

[power12]

I get the below

| makeresults
| eval source="/svc-logs/hostabc/mods/stdout.230626-203719"

``` above set test data ```
| eval _time = strptime(replace(source, ".*/stdout.(\d{6}-\d{6}).*","20\1"),"%y%m%d-%H%M%S")

[isoutamo · Accepted solution]

As I expecting that century is 20xx, you should use %Y instead of %y on strptime function.

[power12]

@isoutamo 

I Added below conf...but somehow the data still did not take the source path time.Is there other steps I need to add
props.conf

[< sourcetype name>]
TRANSFORMS-set_time = set_time_from_source_path

Added below in transforms.conf

[set_time_from_file_path]
INGEST_EVAL = _time = strptime(replace(source, ".*/stdout.(\d{6}-\d{6}).*","20\1"),"%Y%m%d-%H%M%S")

[isoutamo]

Was it typo or are those name different on your conf files?

[power12]

I tried this as well

[< sourcetype name>]
TRANSFORMS-set_time = set_time_from_file_path

Added below in transforms.conf

[set_time_from_file_path]
INGEST_EVAL = _time = strptime(replace(source, ".*/stdout.(\d{6}-\d{6}).*","20\1"),"%Y%m%d-%H%M%S")

Should we have the "set_time" extension after the TRANSFORMS

[isoutamo]

That should works. Have you restarted splunk and then remember that this is applied only a new ingested events, not for those which are already indexed.

[power12]

Yes...I did restart splunk after applying that...I tried mimicking the same on my local.

I went to my local (C:\Program Files\Splunk\etc\apps\search\local) where I set the transforms to be

[set_time_from_source_path]
INGEST_EVAL = _time = strptime(replace(source, ".*/stdout.(\d{6}-\d{6}).*","20\1"),"%Y%m%d-%H%M%S")


and in the props.

[sourcetime]
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
disabled = false
pulldown_type = true
TRANSFORMS-set_time = set_time_from_source_path

Restarted>

Then from WEB UI--->ADD DATA -->SELECTED FILE (_svc-logs_hostbac_mods_stdout.230629-053900) -->selected sourcetype (sourcetime) to see the preview I get errors(attached) that its failed to parse time

[isoutamo]

On your example the source was “foo/bar/stdout.230629-123456” and now you are trying to use source “foo.bar.stdout.230629-123456”. Obviously that doesn’t match our regex on replace function. You must fix that to match your real source path!

[power12]

In the regex  its looking for .*/stdout so even if its _ or . ..it matches everything until stdout right

INGEST_EVAL = _time = strptime(replace(source, ".*/stdout.(\d{6}-\d{6}).*","20\1"),"%Y%m%d-%H%M%S")

[isoutamo]

It match everything before “/stdout…..” and you have “_stdout….” which obviously doesn’t match that.
So you must remove that / from regexp, then it should work.
https://regex101.com/ Is good place to test those.

[power12]

I can try that but I am testing on my windows local splunk installed trying to do the same from 

C:\Users\vv\OneDrive-xx\Pictures\svc-logs\hostabc\mods\stdout.230629-081216.txt for this I changed it to 

[set_time_from_source_path]
INGEST_EVAL = _time = strptime(replace(source, ".*\\stdout.(\d{6}-\d{6}).*","20\1"),"%Y%m%d-%H%M%S")

This also did not work.

I am assuming something with "20\1"..how do we change that..Can you please let me know what should be changed for this case

[isoutamo]

You need just change this

INGEST_EVAL = _time = strptime(replace(source, ".*stdout.(\d{6}-\d{6}).*","20\1"),"%Y%m%d-%H%M%S")

then it works both windows and Linux.

When you are escaping \, the amount of needed \ is depending where you are using it (props, rex, …). Good rule of thumb is start with one and adding those as many as needed (sometimes until 4-6 is needed).

[power12]

Ya even tried that but somehow it says "could not use regex to parse timestamp from source::source::C:\Users\vv\OneDrive-xx\Pictures\svc-logs\hostabc\mods\stdou."