Recognising individual events

[fox]

A new type of log file has been added to an existing data input by amending the whitelist and blacklist. (new data input could not be raised as this would cause an overlap in data monitoring lower down in the directory tree)

Hence this new data could not be given a different sourcetype or indexed to its own index.

The new log file type has a slightly different timestamp at the beginning of each new event:

Existing: 21-Mar-11 09:10:34 GtrGe...

New: 21/03/2011 09:53:58: Runni...

originally the timeformat is explicitly configured in props.conf as follows: TIME_FORMAT = %d-%b-%y %H:%M:%S

When the new log file is added, the events are not split (this may also be down to the fzct that there is the occassional blank line in the log file.

Ideally I need to give an either or timeformat... any ideas if this is possible?

Following this, i will need to find a work around or repair for already indexed new log files that have the wrong event splitting... ideas here would also be welcomed. Bear in mind that I cannot clean this index and start again.

[jbsplunk · Accepted solution]

I would think that the best method to achieve this might be to use the datetime.xml, which is what splunk uses to recognize timestamps out of the box. If you update it to include both of these, then splunk should see them and treat both of them as valid time formats.

http://www.splunk.com/base/Documentation/4.1.4/Admin/TrainSplunktorecognizeatimestamp#Create_a_custo...

With regard to the data that has already been indexed, you'd need to use exporttool to write the events out of splunk, then to reindex them with the new configuration options in place. There isn't a way to surgically remove particular events from an index based on criteria of your choosing. Data is aged out via the bucket configuration settings.

[fox]

thanks jb - I'be found some success by applying props config on the specific source values - waiting to test results. I think datetime.xml is best practice...