Getting Data In

When indexing don't create an event every time EPOCH date is shown, but still index the data

j666gak
Communicator

Hi,

I'm having a bit of a headache. I am trying to index an XML file however I want the event date to be the date the file is updated, and not each EPOCH date listed in the file. Although I do still want to index this data.

The second thing that I want to do is is convert the EPOCH date in to a human readable format.

Example of log file

domain created="1364516883">url.com /domain>

domain created="1364516882">url.net /domain>

domain created="1364516882">url.info /domain>




I would appreciate any help anybody can give me.


Thanks

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

You can do this force this by creating a datetime.xml file containing only:

<timePatterns />
<dataPatterns />

and then indexing with

DATETIME_CONFIG = /etc/apps/myapp/local/mydatetime.xml

or whatever the appropriate relative path to your file is. This will cause date parsing from the data and from the file name to fail (because no patterns exist that can be matched), until the remaining option is to use the modification date of the input file. Instead of trying to modify the epoch date when the data is indexed, you should just convert it at search time when viewing the data, e.g., using the | convert search command or strptime() eval function.

Ayn
Legend

Use DATETIME_CONFIG = CURRENT in props.conf.

http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

As for converting epoch to human readable in the actual raw event, that's not possible to do via an index-time transform at least. If it's really important to you, you could consider switching to a script-based input and put that kind of transforming logic into the script instead.

0 Karma

Ayn
Legend

Yes. The assumption would be that Splunk will pick events up right as the file is updated, so index time will be roughly the same as file modification time.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

This will set the timestamp to the time the file is indexed, rather than the modification time of the file.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...