Getting Data In

File not being read by Splunk in a directory while others are

SirHill17
Communicator

Hi,
I have a directory which is defined in inputs.conf on a host (which has UF running), directory is:

/var/middleware/inventory/var

As per the logs (splunkd.log), the directory is now monitored:

10-04-2017 11:50:50.105 +0200 INFO TailingProcessor - Adding watch on path: /var/middleware/inventory/var.

In this directory there are nine different files. But only eight of them are read. They all have the same permissions and the content format is also the same.

Does anyone know why the last file is not being read by Splunk? There is no log about it.

Thanks for your help.

1 Solution

SirHill17
Communicator

Since the UF version was upgraded it resolved the problem, not sure what was the issue but thanks for the useful command you provided which helps for other troubleshooting.

View solution in original post

SirHill17
Communicator

Since the UF version was upgraded it resolved the problem, not sure what was the issue but thanks for the useful command you provided which helps for other troubleshooting.

SirHill17
Communicator

Are you sure this command can be run on the UF ? I got the below error message when running it (but I am able to do it on the Splunk Servers):

./splunk list inputstatus Command
error: The subcommand 'inputstatus' is
not valid for command 'list'. Data
forwarding configuration management
tools. Commands:
enable local-index [-parameter ] ...
disable local-index [-parameter ] ...
display local-index
add [forward-server|search-server] server
remove [forward-server|search-server] server
list [forward-server|search-server]

Objects:
forward-server a Splunk forwarder to forward data to be
indexed
search-server a Splunk server to forward searches
local-index a local search index on the Splunk server

Searching for index=_internal source=*splunkd.log tailreader ERROR there is no log for the host where I am trying to get the file to be read.

Thanks

0 Karma

mattymo
Splunk Employee
Splunk Employee

What version is the UF? If it is pre 6.3-ish then you may not have the option to run it like that. You would need to try this:

https://www.splunk.com/blog/2011/01/02/did-i-miss-christmas-2.html

Also, try grep ERROR splunkd.log on the on the UF located at $SPLUNK_HOME/var/log/splunk/splunkd.log

Do you see any logs from this UF if you run index=_internal source=*splunkd.log ?

- MattyMo

SirHill17
Communicator

Got it, UF version is 6.2.7. The servers are on 6.4.3.
It explains, if I could have it upgraded soon I will have a try but anyway I also already tried with crcSalt = .

0 Karma

mattymo
Splunk Employee
Splunk Employee

ok so try the rest call on the UF. Just need to make sure the UF is serving 8089, you can get the same output as the ./splunk list inputstatus command. Also check the splunkd.log locally for the TailReader logs.

- MattyMo
0 Karma

SirHill17
Communicator

Ok so I upgraded the forwarder to version 6.4.3 and run the command ./splunk list inputstatus

The output contains the file which is not read and there is still no error in the logs.

I did test to renamed the file to tomcat_jvm2.out and Splunk is picking it up...

Any other suggestion?

Thanks!

0 Karma

mattymo
Splunk Employee
Splunk Employee

Please post that status of the file you are looking for. We are not just looking that the ouptut contains the file...we want the status. If you change the name and it gets picked up, it must be failing CRC Check, or have some other issue with rotation, etc:

/var/log/secure
    file position = 677
    file size = 677
    parent = /var/log
    percent = 100.00
    type = finished reading
- MattyMo
0 Karma

SirHill17
Communicator

That's the output:

/var/middleware/inventory/var/tomcat_jvm.out
                file position = 764
                file size = 764
                parent = /var/middleware/inventory/var/
                percent = 100.00
                type = open file
0 Karma

hardikJsheth
Motivator

This output suggest that Splunk has already read the file so it won't re-read unless you clean fish bucket.

The issue has with the rate at which logs are written and then rotated. Splunk UF is unable to match this load.

In my environment we faced similar issue with reading firewall logs which were generated/rotated at very fast rate (5GB/30minutes). We were able to solve our problem after switching to Heavy Forwarder.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Hey hardikJsheth, can you please post your own answer and thread? I think you are getting way ahead of yourself, but feel free to work your own answer post above, rather than hijacking this one.

switching to a hf is a bad and downvote worthy idea and could have been avoided completely by simply editing limits.conf of the UF (defaults to maxKBps=256) or using parallel pipes, etc. a uf can easily keep up if tuned properly. not to mention the HF impacts the host and network more

I havent gotten the feeling they are trying to re-read anything...so fishbucket might help if OP is trying to reread but as far as i can tell they cant even find the original events. Furthermore the file is open, which means tailreader is watching for new events! lets find those before we go too far.

- MattyMo

SirHill17
Communicator
/var/middleware/inventory/var/tomcat.out
                file position = 74
                file size = 74
                parent = /var/middleware/inventory/var/
                percent = 100.00
                type = finished reading

/var/middleware/inventory/var/tomcat_jvm.out
                file position = 764
                file size = 764
                parent = /var/middleware/inventory/var/
                percent = 100.00
                type = open file

What's the difference between "open file" and "finished reading" ? Really strange behaviour today, only the file that was not read before has been read today.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Open file means it being tailed, finished reading probably means we hit an end of file.

Ok, so since we have ruled out Splunk not being able to read the files, how many files are you monitoring on this server? Just this directory? just these 9? or many files???

In my dealings with middleware teams, especially when I see jvms involved, the UF may require tuning to ensure we can move data fast enough to remain "realtime".

Take a look at $SPLUNK_HOME/var/log/splunk/metrics.log and grep for "blocked=true" are you seeing any blocking?

Also check to see if you are perhaps hitting large files that are taking up all the bandwidth in splunk.log. (search for "large file" in splunkd.log and you should see the batch processor being invoked)

THEN we can start to talk about tuning, like @hardikJsheth alluded to

- MattyMo
0 Karma

mattymo
Splunk Employee
Splunk Employee

Thats better. The file is being read.

Perhaps your timestamps are messed up? Have you tried:

index=* host=<yourForwarder> source=/var/middleware/inventory/var/tomcat_jvm.out over all time??

If you do:

tail -10 tomcat_jvm.out

What do the last few events look like?

- MattyMo
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Try to run this command, it will display which files has been read by splunk and which one is not and also gives reason why it didn't read.

/opt/splunkforwarder/bin/splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus

I hope this helps.

Thanks,
Harshil

0 Karma

mattymo
Splunk Employee
Splunk Employee

This is the exact same command I provided, fyi. Same output.

- MattyMo

SirHill17
Communicator

It helps thanks!
I will let the forwarder running for few days and monitor if the file will be read or not and use the command to get info.
I will come back with comments.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Try running:

./splunk list inputstatus on the UF and looking for the file in question. It should show you why the tailReader may not have actioned it. If you have many files, it can be easier to output the command to a file.

Or try searching:

index=_internal source=*splunkd.log tailreader ERROR

might turn up something like:

10-03-2017 21:27:33.978 -0400 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/splunker/splunk/var/log/splunk/splunk_app_stream.log.8). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

- MattyMo
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...