I was wondering: Is there a way to index past logs and still have them show up as just one source?
Example:
I have a directory with a bunch of logs in it. They look like:
BEK02132013.log
BEK02142013.log
BEK02152013.log
BEK02162013.log
BEK02172013.log
BEK02182013.log
....
....
etc., etc.,
So a new log is made every day with the date in it. This means if I setup a monitor inputs for this directory, all the files are indexed. They all show up as a source and this makes my source list huge!
Considering there are timestamps in the logs, I was wondering is there a way for all these logs to just be under one source? Example: All this data is under the source BEK.log.
Yes, that can be done, but it will not alter already indexed data, just new stuff coming in.
Assuming you have an inputs.conf
that looks like this;
[monitor:///var/logs/BEKLOGS]
index = blah
sourcetype = bek
you would want to have a props.conf
entry like this;
[bek]
TRANSFORMS-foo = set_bek_source
and a transforms.conf
like this
[set_bek_source]
REGEX = .
DEST_KEY = MetaData:Source
FORMAT = source::BEK.log
For more examples, see:
http://splunk-base.splunk.com/answers/5544/override-source-tcpxxxx-of-a-tcp-input-using-transforms
http://docs.splunk.com/Documentation/Splunk/5.0.2/Admin/Transformsconf
Hope this helps,
Kristian
If you're just setting the source to a static value, you can do it via transforms, as kristian.kolb suggested, but it might be simpler to just do:
[monitor:///my/log/path/BEK*.log]
sourcetype=BEKlogs
source=BEK.log
However, if you're going to do that, I might just suggest you ignore "source" and use "sourcetype" anyway. If on the other hand, you want to preserve part of the source path, e.g., you are monitoring files like:
[monitor:///my/path/*/logs/BEK*.log]
and you want the source to read like /my/path/group1/logs/BEK.log
, you would use kristian's method of a transform, but you would need a more complex REGEX and FORMAT to extract and use the appropriate parts of the source you want.
aah, I knew there was a simpler way... just never done much of source overriding, just index, host etc.
Yes, that can be done, but it will not alter already indexed data, just new stuff coming in.
Assuming you have an inputs.conf
that looks like this;
[monitor:///var/logs/BEKLOGS]
index = blah
sourcetype = bek
you would want to have a props.conf
entry like this;
[bek]
TRANSFORMS-foo = set_bek_source
and a transforms.conf
like this
[set_bek_source]
REGEX = .
DEST_KEY = MetaData:Source
FORMAT = source::BEK.log
For more examples, see:
http://splunk-base.splunk.com/answers/5544/override-source-tcpxxxx-of-a-tcp-input-using-transforms
http://docs.splunk.com/Documentation/Splunk/5.0.2/Admin/Transformsconf
Hope this helps,
Kristian
just to clarify; if you use a heavy forwarder, the props and transforms should go there and not on the indexer.
For universal or lightweight forwarder, the settings should be on the indexer.
I have the inputs on the forwarder and made entries in props.conf and transforms.conf on the indexer. So far don't have the logs showing up but will look at things. I had a crc salt error so i added crcsalt =
Ok i will try this. I assumed it used transforms but wasn't sure the exact way to go about it. Let me test this.