Getting Data In

UF not sending logs from all folders monitored

guimilare
Communicator

Hello Splunkers.

I have an issue that I've been dealing with for the past 2 days but no success in solving it.
I'm working on a Splunk cluster environment, 3 SH and 2 IDX.

I have an UF installed in a SunOS machine.
This UF monitors a file called runlog.098880020 (the number is actually just an ID, it doesn't really matters).
This log can be found at the path /export/tsi/tsi/tsiout.1509/runlog.098880020
The thing is: the application creates a new folder every month (tsiout.1505, tsiout.1506, tsiout.1507, tsiout.1508, tsiout.1509....)

this is how I've setted my inputs.conf:

[monitor:///export/home/tsi/tsi/.../runlog*]
index = tsi
sourcetype = tsi_logs

However when Splunk starts to indexing the files, it indexes only a few folders (e.g., tsiout.1406 and tsiout.1409).
If I set my inputs.conf as following, I can see the current log beeing indexed:

[monitor:///export/home/tsi/tsi/tsiout.1509/runlog*]
index = tsi
sourcetype = tsi_logs

Do you guys know why this is happening?
Shouldn't the ... tell Splunk to search in every folder for the runlog* file?

Thank you guys!
Regards!

0 Karma
1 Solution

nnmiller
Contributor

Let's start with some basics:

Do all the directories and files underneath /export/home/tsi/tsi have the same permissions? Does the user the UF is running as have appropriate permissions for those files and directories? Directories will need at least x to traverse them; files will need both r and x.

You may also need to look at the privileges assigned using usermod on Solaris if the UF is not running as root (it should not be, that is a security risk). Look at /etc/user_attrand review the bottom portion of this docs page to see if permissions match:
Run Splunk as Non-Root User

Are any errors being generated on the UF? You can search index = _internal for the UF's hostname or IP; if nothing is showing up there, check on the UF itself. Logs will be in /opt/splunkforwarder/var/log/splunk/

View solution in original post

nnmiller
Contributor

Let's start with some basics:

Do all the directories and files underneath /export/home/tsi/tsi have the same permissions? Does the user the UF is running as have appropriate permissions for those files and directories? Directories will need at least x to traverse them; files will need both r and x.

You may also need to look at the privileges assigned using usermod on Solaris if the UF is not running as root (it should not be, that is a security risk). Look at /etc/user_attrand review the bottom portion of this docs page to see if permissions match:
Run Splunk as Non-Root User

Are any errors being generated on the UF? You can search index = _internal for the UF's hostname or IP; if nothing is showing up there, check on the UF itself. Logs will be in /opt/splunkforwarder/var/log/splunk/

woodcock
Esteemed Legend

What makes you think that it is not indexing? Tell the WHOLE story (did it ever get indexed before and now you are trying to reindex it)?

0 Karma

guimilare
Communicator

OK..
I don't need old logs..
I need to index this month logs and the ones that are coming.
I can see at the server that runlog.0901000208 exists under the folder tsiout.1509, but Splunk is not indexing it (never indexed).

0 Karma

richgalloway
SplunkTrust
SplunkTrust

For your purposes, [monitor:///export/home/tsi/tsi/*/runlog*] should work.

---
If this reply helps you, Karma would be appreciated.
0 Karma

guimilare
Communicator

No luck in that either..

The files indexed where:
/export/home/tsi/tsi/tsiout.1409/runlog.0901000145
/export/home/tsi/tsi/tsiout.1209/runlog.0905001532
/export/home/tsi/tsi/tsiout.1209/runlog.0901001129
/export/home/tsi/tsi/tsiout.1209/runlog.0904213433

Maybe is there somthing to do with the month..
Only the folders tsiout.1209 and tsiout.1409 were indexed...

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you checked the permissions on the folders that are not indexed?

---
If this reply helps you, Karma would be appreciated.
0 Karma

woodcock
Esteemed Legend

Have you ever used ignoreolderthan on these inputs?

0 Karma

guimilare
Communicator

I've never used ignoreolderthan

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...