Rename sourcetype to a non-existent type

[bdunstan]

Hi,

I am trying to reset/rename the sourcetype based on the filename - which appears to work fine, if the sourcetype it is being renamed to exists in props.conf. But, what happens if it doesnt exist ?

I have an inital sourcetype based on json.

props.conf:
[clone-json]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = none
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
TIMESTAMP_FIELDS = timeStamp
TRANSFORMS-fs = force-sourcetype-st

This works perfectly, but now I need to change the sourcetype based on the filename, therefore the 'TRANSFORMS-fs = force-sourcetype-st' setting at the bottom.

If the source file is, /DATA/12345/interfaces.20160611.gz

transforms.conf:
[force-sourcetype-st]
DEST_KEY = MetaData:Sourcetype
SOURCE_KEY = MetaData:Source
REGEX = \/\d+\/(\w+).\d+.gz$
FORMAT = sourcetype::$1
INDEXED_EXTRACTIONS = json
WRITE_META = true

So with the above, configurations, I am able to reset the sourcetype to interfaces and that works, however, when I look at the data in splunk it is duplicated. That is, if I pass in 1 record and do .... | stats count by id - it returns 2 instead of 1.

In this instance there is no sourcetype interfaces defined in the props.conf, so although I can change the sourcetype to interfaces, that type doesnt actually defined anywhere.

If I create the sourcetype interfaces, it all works fine.

You ask, why not just create the type if that makes it work ?
Well, I dont know what types are likely to come into the system, so I am trying to make it completely dynamic in nature.

I dont know if the problem is due to the INDEXED_EXTRACTION not being known, or its set to a default that is not json - or if there is some other metadata value I need to change to tell splunk the format and only to create 1 record.

Or worst case, I have to predefine all possible sourcetypes - even though that are all json in nature.

[bdunstan · Accepted solution]

Solved, the issue was a props.conf miss-understanding by me - indextime vs runtime:

[source::.../json-files/*/*.gz]
KV_MODE = none
INDEXED_EXTRACTIONS = json

[clone-json]
pulldown_type = true
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
TIMESTAMP_FIELDS = timeStamp
TRANSFORMS-fs = force-sourcetype-st

Setting a source with KV_MODE to none and the INDEX_EXTRACTIONS there as well seems to have solved the issue.