Hi
We index logfiles which have date time strings on the beginning of each event in hex format.
50FD316C = Mon Jan 21 13:15:40 2013
Is it possible to change this hex datetimestamp to an readable timestamp with default splunk features?
Or is it possible to execute a script on each event and change this timestamps in the _raw field?
Thanks
Rob
Do you want to use the hex code as the timestamp for the event ?
Thats not possible without pre-processing the event beforehand ( before splunk sees it )
If all you want to do is convert the code to a readable time after the event has been indexed then assuming the field you want to convert is called 'hex_time'
... | eval my_time=tonumber(hex_field,16) | convert ctime(my_time) | table my_time
1st part converts the hex to epoch seconds
2nd part formats the time into a readable date format
If the 'hex_field' is not automatically extracted, this'll do it (based on the fact the hex is at the start of the line) :
... | rex ^(?<hex_field>\S+) | ...
Update
If you need to convert the hex to a timestamp beforehand, a (very) hacky way to achieve this would be to configure syslog-ng to follow the files, and run each line through a program.
i.e. (in syslog-ng.conf)
source ibm_file {
file("/opt/IBM/log/kbb.log" follow_freq(1)
flags(no-parse));
};
destination hex_converter { program("/home/user/hex_converter.pl"); };
log { source(ibm_file); destination(hex_converter); };
the 'hex_converter' program would read from STDIN, convert the hex, and output the converted line to a file that splunk would then index ( or write it out to syslog, or send it as a TCP stream to splunk )
IBM normally ship a binary called ras1log that'll you could use in the hex_converter before redirecting to another file.
Do you want to use the hex code as the timestamp for the event ?
Thats not possible without pre-processing the event beforehand ( before splunk sees it )
If all you want to do is convert the code to a readable time after the event has been indexed then assuming the field you want to convert is called 'hex_time'
... | eval my_time=tonumber(hex_field,16) | convert ctime(my_time) | table my_time
1st part converts the hex to epoch seconds
2nd part formats the time into a readable date format
If the 'hex_field' is not automatically extracted, this'll do it (based on the fact the hex is at the start of the line) :
... | rex ^(?<hex_field>\S+) | ...
Update
If you need to convert the hex to a timestamp beforehand, a (very) hacky way to achieve this would be to configure syslog-ng to follow the files, and run each line through a program.
i.e. (in syslog-ng.conf)
source ibm_file {
file("/opt/IBM/log/kbb.log" follow_freq(1)
flags(no-parse));
};
destination hex_converter { program("/home/user/hex_converter.pl"); };
log { source(ibm_file); destination(hex_converter); };
the 'hex_converter' program would read from STDIN, convert the hex, and output the converted line to a file that splunk would then index ( or write it out to syslog, or send it as a TCP stream to splunk )
IBM normally ship a binary called ras1log that'll you could use in the hex_converter before redirecting to another file.
Thank you jonuwz for your detailed explainaition.
I'm not firm with syslog but I will give it a try and check how this works.
I know the ras1log tool, this was my prefered tool to troubleshoot.
Yes thats IBM logfiles, which came from a heavy forwarder.
How do the logs get to the indexer ? syslog-ng ? universal forwarder ? Whats the application (looks IBM .. ) ?
Thanks jonuwz for your answer.
Yes I want to use this hex date for the actual event.
I'm also consider to make the time conversion on the forwarder side,
cause in case of debuging, there will be produced a lot of events on top.Do you see a possiblity to do so?Thanks
Robert