Splunk Search

Log data source received but only some of it searchable

pj
Contributor

I have an odd issue occurring. Essentially I have a high volume log source which is getting picked up by a Splunk forwarder and forwarding the log source to a specific index using an autoLB DNS location (pointing to 2 indexers).

I can see from the internal metrics that both indexers are receiving the log data (over 10Gb per day spread over the 2 indexers). The problem is that when I perform a search for this sourcetype, I can only see a handful of the events (i.e. 1000 over the past day when there should be a few million). If I select an event that i can see in the simple sourcetype search, then do a 'view source', I see 1000s of events in the source, which suggests the data is getting in Splunk fine, but is just not searchable.

The forwarder is version 4.0.9 (our current standard). Other sources are set up in a similar fashion with this forwarder version and work fine. Additionally, we pointed the source to our lab/test indexer and it shows up fine.

Thoughts?

0 Karma
1 Solution

pj
Contributor

We were able to confirm that the logs were getting indexed, they just were not availabel for viewing in Splunk. The data concerned was a series of zipped up delimited log files containing a header row, a load of data rows, then a summary row. For some reason, Splunk would only make searchable the header and summary rows of these files, despite indexing everything. After much looking around, the fix seems to have been to add crcSalt = <SOURCE> to the inputs.conf file. I also did a splunk clean all on the forwarder. Now the data comes in and is fully searchable...

View solution in original post

0 Karma

pj
Contributor

We were able to confirm that the logs were getting indexed, they just were not availabel for viewing in Splunk. The data concerned was a series of zipped up delimited log files containing a header row, a load of data rows, then a summary row. For some reason, Splunk would only make searchable the header and summary rows of these files, despite indexing everything. After much looking around, the fix seems to have been to add crcSalt = <SOURCE> to the inputs.conf file. I also did a splunk clean all on the forwarder. Now the data comes in and is fully searchable...

0 Karma

Simeon
Splunk Employee
Splunk Employee

Please supply your exact search query in addition to any details about how the source data is classified or input to Splunk

0 Karma

Simeon
Splunk Employee
Splunk Employee

If you can confirm the appropriate metrics on the indexing side, then the data is likely in the index. However, you need to run a search that will find out where Splunk stored it. Typically, you can run a search over all-time just for that specific source or sourcetype and that will show you where the data has been stored. Most likely, the event data is getting incorrectly time stamped or bunched into a large multi line event.

0 Karma

pj
Contributor

A search over all time does not bring up the logs. One thing i have noticed is that when performing a search for the sourcetype, I see things like:

11 matching events|295,308 scanned events

The scanned events corresponds closely with the actual events that would be expected for the source.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...