Getting Data In

How to monitor .dat.gz files?

Genti
Splunk Employee
Splunk Employee

I have set up an inputs.conf monitor stanza to index a few .dat.gz files but do not see them getting indexed.

There is no blacklist or null queue routing going on.
Also, there are no error messages in splunkd.log.

Tags (1)
0 Karma
1 Solution

Genti
Splunk Employee
Splunk Employee

There are two issues going on here.

  1. Why are the files not being moitored and how to actually monitor them
  2. Why are there no errors in splunkd.log

First, Splunk will not index binary files. Even if you have a file.dat.gz that is not binary, and you specify NO_BINARY_CHECK, Splunk will still fail to index the file.

This is because there is stanza in /etc/system/default/props.conf that tells splunk not to.

[source::....(0t|a|ali|asa|au|bmp|cg|cgi|class|d|dat|deb|del|dot|dvi|dylib|elc|eps|exe|ftn|gif|hlp|hqx|hs|icns|ico|inc|iso|jame|jin|jpeg|jpg|kml|la|lhs|lib|lo|lock|mcp|mid|mp3|mpg|msf|nib|o|obj|odt|ogg|ook|opt|os|pal|pbm|pdf|pem|pgm|plo|png|po|pod|pp|ppd|ppm|ppt|prc|ps|psd|psym|pyc|pyd|rast|rb|rde|rdf|rdr|rgb|ro|rpm|rsrc|so|ss|stg|strings|tdt|tif|tiff|tk|uue|vhd|xbm|xlb|xls|xlw)]
sourcetype = known_binary

Note that this is not just happening for .dat files but happens for all the above extensions..
In order to actually have the files indexed then we need to create (or edit) a props.conf within /etc/system/local/ with the following stanza:

[source::....(dat)]
sourcetype = mysourcetype
priority = 20

This will overwrite the default/props.conf and will tell splunk to use a different sourcetype for the .dat.gz files, and hence allow them to actually be indexed.

The reason why there was no message in splunkd.log is because of a bug, which has been filed and our friendly devs will soon be working on it.
Hope this helps someone out there..
.gz

View solution in original post

Genti
Splunk Employee
Splunk Employee

There are two issues going on here.

  1. Why are the files not being moitored and how to actually monitor them
  2. Why are there no errors in splunkd.log

First, Splunk will not index binary files. Even if you have a file.dat.gz that is not binary, and you specify NO_BINARY_CHECK, Splunk will still fail to index the file.

This is because there is stanza in /etc/system/default/props.conf that tells splunk not to.

[source::....(0t|a|ali|asa|au|bmp|cg|cgi|class|d|dat|deb|del|dot|dvi|dylib|elc|eps|exe|ftn|gif|hlp|hqx|hs|icns|ico|inc|iso|jame|jin|jpeg|jpg|kml|la|lhs|lib|lo|lock|mcp|mid|mp3|mpg|msf|nib|o|obj|odt|ogg|ook|opt|os|pal|pbm|pdf|pem|pgm|plo|png|po|pod|pp|ppd|ppm|ppt|prc|ps|psd|psym|pyc|pyd|rast|rb|rde|rdf|rdr|rgb|ro|rpm|rsrc|so|ss|stg|strings|tdt|tif|tiff|tk|uue|vhd|xbm|xlb|xls|xlw)]
sourcetype = known_binary

Note that this is not just happening for .dat files but happens for all the above extensions..
In order to actually have the files indexed then we need to create (or edit) a props.conf within /etc/system/local/ with the following stanza:

[source::....(dat)]
sourcetype = mysourcetype
priority = 20

This will overwrite the default/props.conf and will tell splunk to use a different sourcetype for the .dat.gz files, and hence allow them to actually be indexed.

The reason why there was no message in splunkd.log is because of a bug, which has been filed and our friendly devs will soon be working on it.
Hope this helps someone out there..
.gz

rakesh_498115
Motivator

Hi ..i am facing the similar issue for indexing dat files..i have removed the dat extension in system/default/props.conf file ..but then its not indexing...pls help

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...