I have an application writing out JSON formatted logfile entries that we're using the universal forwarder to get over to the indexer system. The log entries (client-side) could be several lines per second. I didn't define a sourcetype when I added the monitor on the forwarder system (yes, I know 'now'). So the result is that the indexer is 'helping' too much and it sometimes puts multiple entries into one event as seen on the splunk console.
Questions:
or
I poked around a little with "splunk btool props list <sourcetype_here>" and can see which types do or don't linemerge, but there are a 'lot' of known sourcetypes. Any suggestions on which one to pick if we can (hopefully) not need to create our own ?
Generally what you do, if you are not using somebody else's configuration files (e.g. from an app on apps.splunk.com), is you create your own app directory like $SPLUNK_HOME/etc/apps/MyApp/default
(yes, since you are the developer of this app, you use default
, not local
) and you create your files there. Inside this directory, you should put your inputs.conf
file and inside this file you should have something like this:
[monitor:///path/to/my/file.log]
sourcetype=MyApp
You might also add index=MyIndex
if you would like to get your events out of index=main
.
In the same directory structure, you should put your props.conf
file and inside this file you should have something like this:
[MyApp]
INDEXED_EXTRACTIONS = json
This set of files needs to be put on your Forwarders and the Splunk instances there all restarted.
That is mostly it but you will probably like to do some other things, too. For example, there's a TIMESTAMP_FIELDS setting that exploits the JSON structure rather than specifying TIME_FORMAT or TIME_PREFIX expressions to manually walk through the structure; see more here:
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf
http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileheadersatindextime
If your'e using a universal forwarder, the sourcetype definition should be on Indexers (server side). If your data is in just single line, SHOULD_LINEMERGE should be false.
Makes a devops deployment kind of hard to do. No way to control should_linemerge from the client side using the universal forwarder ? Does the forwarder support props/transforms ? Setting a known sourcetype that has should_linemerge=false on the server side already ?