Getting Data In

Can't get sourcetype right

Branden
Builder

Hi.

Seems like a lot of people have a question similar to this, but maybe I am missing something simple. I'm monitoring logs in a directory (/foo/barr/logs/APPSRV*). Logs are sent to the indexer via the SplunkLightForwarder. The wild card is because the log file names are dynamic (they have a timestamp on the end of them)

I created a props.conf in /splunk/etc/apps/search/local with the following entry on the SplunkLightForwarder:

[source:///foo/barr/logs/APPSRV*]
sourcetype=appsrv

Unfortunately, the sourcetype still comes out wrong. It shows up as APPSRV-too_small. I then tried putting props.conf in /splunk/etc/apps/search/default, but that didn't help either.

I think I'm missing something fundamental here...

Thanks in advance for your help.

Tags (1)
1 Solution

treinke
Builder

In $SPLUNK_HOME/etc/system/local/inputs.conf

If you are using the file system change:

[fschange:/foo/barr/logs/APPSRV*]
sourcetype = appsrv

For monitoring the files/folders:

[monitor:///foo/barr/logs/APPSRV*]
sourcetype = appsrv

A great resource to find out more about the inputs.conf is: http://www.splunk.com/base/Documentation/4.1.4/admin/Inputsconf

There are no answer without questions

View solution in original post

rotten
Communicator

Splunk support showed us how to do it using an approach like this:


inputs.conf on the lightweight forwarder:

[monitor:///foo/bar/logs/]
disabled = false
host = myServer_myApplication
crcSalt = <SOURCE>
blacklist = \.(tar|gz|bz2)

props.conf on the indexer:

[source::/foo/bar/logs/.../*]
TRANSFORMS-foobarlogs = fix_foo_bar_logs_sourcetype

transforms.conf on the indexer:

[fix_foo_bar_logs_sourcetype]
REGEX=.
FORMAT=sourcetype::foo_bar_log
DEST_KEY=MetaData:Sourcetype


This approach is the only way we have found to reliably set the sourcetype for the vast majority of our logs.

If we pick up the logs on the indexer, we can simplify this by setting the sourcetype in the inputs.conf directly:

[monitor:///foo/bar/logs/]
disabled = false
sourcetype = foo_bar_log
host = myServer_myApplication
crcSalt = <SOURCE>
blacklist = \.(tar|gz|bz2)
0 Karma

BunnyHop
Contributor

If you starting seeing "**-too_small" it means that the indexing is being learned, and not using your props.conf or transforms.conf settings. So, a couple things:

First, you shouldn't replace any of the documents/configuration files within the Default folder. Use either the Apps folder or the Local folder for your custom config. Second, there are a few bugs in the past (true with IIS at some point) that no matter how you specify the sourcetype, it will only grab the name, and learn the rest automatically. So that's something that needs to be fixed on the software. Third, when you re-add/re-index, make sure you clear the learned configurations. This is located on the Apps folder under learned.

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

Your first approach didn't work because of the difference in naming convention for stanzas in inputs.conf and props.conf.

In inputs.conf, stanzas look like URIs, so they have a scheme (say monitor, tcp or udp) followed by "://" followed by a path.

On the other hand, props.conf just looks at the fields source, sourcetype or host. So the stanzas in props.conf should be named: [<sourcetype>] [source::<source>] [host::<host>]

The "::" here is a inherited from when all fields in Splunk were indicated by field::value rather than field=value.

Lowell
Super Champion

Your first approach should have worked, but I image you had a typo or permissions setting getting in the way. I would also like to point out that there are limitations in simply forcing a sourcetype on all of your inputs.conf stanzas as was suggested above. For example, say you had two different types of logs in /foo/barr/logs/APPSRV* and you want to use two different sourcetypes? How would you get the proper sourcetype assigned? (Splunk 4.1 makes this slightly easier, but there are still some limitations.)

I highly recommend using the following command line tool to debug your .props files:

splunk test sourcetype /foo/barr/logs/APPSRV/some_log_file.log

This should return the sourcetype associated with your log file and show you all the props settings associated with it.


I put together a list of common debugging techniques on a different post. You may find some of it helpful:

What’s the best way to track down props.conf problems?

0 Karma

treinke
Builder

In $SPLUNK_HOME/etc/system/local/inputs.conf

If you are using the file system change:

[fschange:/foo/barr/logs/APPSRV*]
sourcetype = appsrv

For monitoring the files/folders:

[monitor:///foo/barr/logs/APPSRV*]
sourcetype = appsrv

A great resource to find out more about the inputs.conf is: http://www.splunk.com/base/Documentation/4.1.4/admin/Inputsconf

There are no answer without questions

Branden
Builder

Worked great, thanks!

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...