Getting Data In

Custom month names?

gljiva
Path Finder

Hi, I'm having problems timestamp extraction of custom month names (written in my language). Here is a sample log entry:

[26/stu/2010:14:37:40 +0200]

First i tried using TIME_PREFIX and

TIME_FORMAT=%d/%b/%Y:%H:%M%S

but splnuk was unable to understand it and generates wrong timestamp. Next i tried modifying datetime.xml and replacing english month notation with this regex (sij|vel|ozu|tra|svi|lip|srp|kol|ruj|lis|stu|pro) and setting it as argument to DATETIME_CONFIG in props.conf. This also produces wrong timestamp.

What is correct procedure to map english notation (eg. jan feb mar...) to my language (sij, vel, ozu...)? I suppose i need to add transformation and replace my localized names to english ones? How can i do such mapping?

thanks

gljiva
Path Finder

Hi, i tried using sed to rewrite month names, it is overwritten fine:

[23/Nov/2010:14:37:40 +0200]

But timestamp is wrong, it seems that splunk first resolves timestamp and then does sed replace.

SEDCMD-sij = s/sij/Jan/
SEDCMD-vel = s/vel/Feb/
SEDCMD-ozu = s/ozu/Mar/
SEDCMD-tra = s/tra/Apr/
SEDCMD-svi = s/svi/May/
SEDCMD-lip = s/lip/Jun/
SEDCMD-srp = s/srp/Jul/
SEDCMD-kol = s/kol/Aug/
SEDCMD-ruj = s/ruj/Sep/
SEDCMD-lis = s/lis/Oct/
SEDCMD-stu = s/stu/Nov/
SEDCMD-pro = s/pro/Dec/
TIME_PREFIX = \d*.\d*.\d*.\d* - \w* \[
TIME_FORMAT = %d/%b/%Y:%H:%M%S
0 Karma

gljiva
Path Finder

So how do i rewrite timestamps?

0 Karma

jrodman
Splunk Employee
Splunk Employee

Timestamps have already been extracted before the events exist. They're part of how we find event boundaries. Thus all event transformations are too late.

0 Karma

jrodman
Splunk Employee
Splunk Employee

If Splunk is running in the that locale, then I would expect this %b conversion to work. What does the command

# date +'%b'

show for you, in the environment in which splunk is used?

UPDATE: We don't have locale handling in our date parsing at all, currently. I had somehow imagined that we made use of the system library for the specific string decoding, but apparently it is a custom implementation for cross-platform consistency, existence at all on windows, and performance goals.

Essentially this becomes an enhancement request (although a fairly important one) for handling localized european dates. In Asia this hasn't come up (yet) because mostly numerics are used for months, rather than names.

The only short-term workarounds I can recommend are to pre-process the file, or to alter the date format in which it is emitted. Obviously neither is ideal but it's what's possible today.

jrodman
Splunk Employee
Splunk Employee

There's a defect or a missing step. Please work with splunk support to resolve.

0 Karma

gljiva
Path Finder

Incoming datastream is similar to sample provided in first post, every event contains timestamp inside []. I tested locale settings on splunk server indexing data and indeed locale is set to right one (locale matches log locale), but splunk still doesn't understand timestamps. How can i manually rewrite timestamp or tell it real mapping between names?

0 Karma

jrodman
Splunk Employee
Splunk Employee

What's the incoming datastream like? What hosts are running in a locale where the timestamps will look like this? How does it arrive? The goal here is to have a splunk running in the desired locale, handling that data.

0 Karma

gljiva
Path Finder

Hi, Splunk is running in a different locale than one used in logs.

date +'%b' returns
May

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...