Hello
I am trying to extract a timestamp from this type of events. Here, 04 is the day of month and 12 is the month, Dec
on the search head, these events currently appear as 12th April
[04/12/2018 10:16:04] CAUAJM_I_40245 EVENT: CHANGE_STATUS STATUS: SUCCESS JOB: esysprod_NOA_5_min_box
[04/12/2018 10:26:03] CAUAJM_I_40245 EVENT: CHANGE_STATUS STATUS: SUCCESS JOB: esysprod_EX900 MACHINE:
It looks pretty straightforward, but I cannot figure out what I am doing wrong.
The source type for these events is called : "autosys_events_prod"
So, I created a props.conf as below, and located it in the app that gets distributed from my deployment server:
I also verify on the server where the log is created that the props.conf file is updated, and I also restart Splunk on the Universal Forwarder.
[splunk@msplunkutil01 local]$ cat props.conf
[autosys_events_prod]
TIME_PREFIX = ^[
TIME_FORMAT = %d/%m/%Y %H:%M:%S
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 19
I have tried different time prefix(es) without success.
How do I know if my props.conf is actually being used?
Everything I have tried seems to have no effect so far.
What is the best way to troubleshoot this ?
Thank you for your help in advance.
it is the first time I am trying to extract a timestamp from an event, so I might be doing something wrong.
Blaise
Hello
I have finally resolved the issue, the problem was I have a distributed environment ...
so like Prakash suggested, the props.conf needs to be on the indexers, where the timestamp extraction is done.
I have completely removed the props.conf from the universal forwarder server, where I only left the inputs.conf to define the inputs.
Thank you for all your help
Blaise
Hello
I have finally resolved the issue, the problem was I have a distributed environment ...
so like Prakash suggested, the props.conf needs to be on the indexers, where the timestamp extraction is done.
I have completely removed the props.conf from the universal forwarder server, where I only left the inputs.conf to define the inputs.
Thank you for all your help
Blaise
@blaise Please accept an answer to help future readers.
Hi Rich and Prakash
I have tried both suggestions and it still is not working
thank you both for your replies, you both suggested to use :
TIME_FORMAT to %m/%d/%Y %H:%M:%S
but my raw events timestamps shows as : [05/12/2018 10:32:03] text text ...
where 05 is the day of the month %d
and 12 is the month %m
so the correct TIME FORMAT should be : %d/%m/%Y %H:%M:%S
please explain why you suggested otherwise, I am getting really confused ...
I am also wondering why all my attempts are failing, is it possible that another definition or config somewhere could take precedence over the app's props.conf ?
Thank you again
Blaise
@blaise: I tested it on my local with your sample data, it's working for me, except you need to make changes to TIME_FORMAT based on your requirements...
##this configs should be on indexers(data parsing happens on indexers)
props.conf
[autosys_events_prod]
SHOULD_LINEMERGE = false
TIME_PREFIX = ^\[
TIME_FORMAT = %d/%m/%Y %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
try running this command to check all the props in that particular app...
./splunk btool props list --debug
./splunk btool props list --debug --app=search
Hi Prakash,
thank you , you are correct and that was my mistake, the props.conf definition needs to be on the indexers.
Once I did that , it started working
Thank you heaps for your help, it is appreciated 🙂
Blaise
@blaise, I originally recommended %d/%m/%Y %H:%M:%S
, but you said it was wrong so I suggested %m/%d/%Y %H:%M:%S
.
This should work, give it a try....
[autosys_events_prod]
SHOULD_LINEMERGE = false
TIME_PREFIX = ^\[
TIME_FORMAT = %m/%d/%Y %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
TIME_PREFIX
is a regular expression, but yours is not a valid regex. ^[
starts a character set, but doesn't finish it. Try `^[', which treats the bracket as a literal character.
Hi Rich
I have tried your suggestion and it still is showing events for the 12th April, instead of the 4th Dec
I tried those 2:
TIME_PREFIX = '^['
TIME_PREFIX = '['
to confirm my props setttings on the universal forwarder, I found this great command:
[splunk@bautoprod01 local]$ splunk cmd btool --app=autosys props list
[autosys_events_prod]
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = false
TIME_FORMAT = %d/%m/%Y %H:%M:%S
TIME_PREFIX = '['
[splunk@bautoprod01 local]$ pwd
/opt/splunkforwarder/etc/apps/autosys/local
[splunk@bautoprod01 local]$
So the above confirms that the settings are applied ("distributed"), but yet it still is not working
Thank you for your help anyway
Blaise
Change TIME_FORMAT
to %m/%d/%Y %H:%M:%S
.