Getting Data In

How to add new logs to monitor on same index but different sourcetype

rparagas
Explorer

Hi,

I have a forwarder installed on one of our hosts and is currently sending logs to the server, please see contents of stanza on inputs.conf:

[monitor:////home/me/logs] 
sourcetype = Core
index = CoreLog
whitelist = me\.log(\.1)?
ignoreOlderThan = 1d
recursive = false

We have a request to expand forwarder to monitor another log file and this is what I've appended on the inputs.conf:

[monitor:////home/me/logs/sms]
sourcetype = CoreSMS
index = CoreLog
whitelist = smpp\.log(\.\d+\-\d+\-\d+)?
ignoreOlderThan = 1d
recursive = false

Then restarted the forwarder and doesn't work.

I have not modified any other config file, do I need to?

Also, note that I am using the same index, is this possible?

Cheers,

Rex

Tags (2)
0 Karma

elof
Path Finder

Not really an answer (since I think this is a bug), but here's a workaround:
http://answers.splunk.com/answers/126064/bug-in-universal-forwarder-inputsconf-monitor-and-recursive...

0 Karma

Dark_Ichigo
Builder

I don't really see any issues with your configuration, Can you give us more information on what's happening, it would be worth checking the splunkd logs as well.

0 Karma

kristian_kolb
Ultra Champion

How do you mean "doesn't work"? Forwarder crash, strange error messages, or no data in indexer?

You should probably have a look at the what the tailing processor on the forwarder is up to.
https://your_forwarder:8089/services/admin/inputstatus/TailingProcessor:FileStatus

read more here;
http://docs.splunk.com/Documentation/Splunk/5.0.3/Data/Troubleshoottheinputprocess

Also, look through the splunkd.log (/opt/splunkforwarder/var/log/splunk/splunkd.log) for interesting warning/error messages shortly after a restart. "Permission Denied" , "File not found" etc.

Other than that;

It is certainly not a problem to have one index with many sourcetypes (or the opposite).

You have four consecutive slashes in the monitor-stanza header. Don't know if that could screw things up. Should probably only be three.

Make sure your regexes match.

----------
UPDATE:
----------

Perhaps you should also check the _internal index on the receiving indexer. Things to look for include, but are not limited to;

blocked queues
unconfigured indexes

I seem to remember that there is an issue with mixed-case index names in some cirumstances. Check the release notes. However, if you already have data coming in to that index from the same forwarder, then that may not be the problem.

Finally, the most obvious thing that should really be tested first:

If your forwarder seems to be working and reading the files, make sure that timestamps are being parsed correctly for the new logs. Otherwise events might be indexed far (in time) from where you'd expect.

Make a search for 'All time' and let it run. Or make a metadata search like so;

| metadata type=sourcetypes | eval firstTime=strftime(firstTime, "%F %T") | eval lastTime=strftime(lastTime, "%F %T")

The metadata search should return a table with the sourcetypes you have indexed. If your new sourcetype does not show, then the data is not there, and if it shows, you'll know what timestamps Splunk used for the oldest and newest event.

/K

rparagas
Explorer

Thank you very much for your help!

I was able to make it work just this morning - there is now data on the index from the new log added to be monitored.

From the link you've provided, I've change logging settings to DEBUG for TrailingProcessor and found these:

TailingProcessor - Adding watch on path: /home/me/logs.
06-03-2013 10:28:45.440 +1000 DEBUG TailingProcessor - Will not call watch on path '/home/me/logs/sms' due to stanza: monitor:///home/me/logs.

...

TailingProcessor - File state notification for path='/home/me/logs/sms' (first time).
TailingProcessor -   Item '/home/me/logs/sms' matches stanza: /home/me/logs.
TailingProcessor -     Is dir, but recursion is disabled.
TailingProcessor -     Not using stanza for this item (ignored item (recursion disabled)).
TailingProcessor -   Item '/home/me/logs/sms' matches stanza: /home/me/logs/sms.
TailingProcessor -     Is dir, but recursion is disabled.
TailingProcessor -     Not using stanza for this item (ignored item (recursion disabled)).
TailingProcessor -   Entry is associated with 0 configuration(s).
TailingProcessor - No configurations match, will ignore path='/home/me/logs/sms'.

so, I guess what this means is that the recursive setting on the initial stanza made splunk ignore the directory on the new stanza because they use the same directory (/home/me/logs & /home/me/logs/sms).

After removing recursive=false on both stanzas and doing a restart I can now see the new directory is recognized:

TailingProcessor - File state notification for path='/home/me/logs/sms' (first time).
TailingProcessor -   Item '/home/me/logs/sms' matches stanza: /home/me/logs.
TailingProcessor -   Item '/home/me/logs/sms' matches stanza: /home/me/logs/sms.
TailingProcessor -   Storing config '/home/me/logs/sms' for app ''.
TailingProcessor -   Entry is associated with 1 configuration(s).

Regards,
rparagas

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...