Getting Data In

How can I view the names of files that have 0 KB of data?

robertlynch2020
Motivator

Hi

I have lots of file in a directory, some with data some with no data.

If i understand correctly Splunk will forward on files only with data, however this is an issues for me as i parse the name and i use it.
So for example 535211-11233-service.log. I take out the value 11233 and use it.
In this case i also need the value 11235 on the 0KB file, but as the forwarder is not sending it i cant access it.

So how can i get the forwarder to send the file with 0KB?

535211-11233-service.log 10kb
535613-11234-service.log 100kb
535614-11235-service.log 0kb
etc...

Thanks in advance
Robert

0 Karma
1 Solution

woodcock
Esteemed Legend

Splunk did monitor the file, but it never obtained any events so no events could be forwarded. What you can do is set up a cron like this on that server:

01 23 * * * find /Your/Directory/To/Files/Here -type f -name "log*.txt" -size 0 -mtime +1 | xargs -i sh -c "echo 'This file will always be empty!' >> '{}' "

Be sure to modify the cron time, the start directory, the -name details, and the -mtime value for your situation. This will cause empty files to obtain a fake event following the rules that you give it.

View solution in original post

0 Karma

woodcock
Esteemed Legend

Splunk did monitor the file, but it never obtained any events so no events could be forwarded. What you can do is set up a cron like this on that server:

01 23 * * * find /Your/Directory/To/Files/Here -type f -name "log*.txt" -size 0 -mtime +1 | xargs -i sh -c "echo 'This file will always be empty!' >> '{}' "

Be sure to modify the cron time, the start directory, the -name details, and the -mtime value for your situation. This will cause empty files to obtain a fake event following the rules that you give it.

0 Karma

robertlynch2020
Motivator

Hi

Thanks for this, Ideally i would love to use the forwarder (Or have the option).
However i understand why it cant, but it means i have to set this cron up on multiple environments on multiple machines etc...

My argument would be there is data in the file name, just not in the file.

I
in the end we are going to use this.

find logs/mxsession -name "*-*service.log" | xargs ls -ltr | grep "murex     0" | awk '{print $9}' > ListOfFileToFill.txt
while read file ; do
DATE=`date  +%Y-%m-%d`
TIME=`date  +%H:%M:%S`
echo "$DATE $TIME [QCST] adding any line" >> $file
done < ListOfFileToFill.txt 
0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Okay, shooting from the hip here. If there is no data to index, Splunk can't index it. Not that it doesn't see it and so forth, but it is looking for data IN the file to index. So it did index the file, but you just have no data associated with the file. The file is in the fishbucket on the forwarder. So you have a problem of how to get information about the file into Splunk.

This is one way that you can solve your dilemma. I suspect that you say you need to know about the file because you need to know about the name of the file (that it exists) in this case, and not the data that is or isn't in the file. That is my assumption for the next section.

There are a couple of approaches that you can take, The first is to run a script regularly on your host (I don't know if you are using Linux or windows) that does a listing of the directory and outputs the result to a file. Then you index that file. You then have the contents of the directory that shows the files that are in it, including your zero length ones.

The second would be to do something similar (file listing), but doing it the same way the *nix and Windows TA apps do (they just run the command and send that output regularly to the indexers).

Does either of these approaches satisfy your need?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...