Data being indexed but unable to search

[rmcdougal]

Ok so here is the issue, I have installed a forwarder on my Snort box to forward over the data to Splunk. It appears to be sending the data over and it appears to be getting indexed but, I am not able to search the information.

This is my search summary page

Notice the last update time.

Now I am going to click on the source type and search for the events.

Notice that the latest event showing up has a timestamp of 12/6/2012 at 12:56 AM. This contradicts the search summary page.

One last thing, from the deployment monitor, this is the status of the forwarder on my snort box.

[rmcdougal · Accepted solution]

I found the solution and it wasn't very intuitive. The timestamp was not being indexed properly by splunk so the events were getting indexed but there was an invalid timestamp associated with them preventing them from showing when searching for them. (I still haven't been able to find them).

After changing the TIME_FORMAT in props.conf the events started to display.

[lguinn2]

My guess is that your Splunk admin did NOT set up the security index to be searched by default. That setting is under Manager -> Access Control -> Roles. For each role, the admin can determine which indexes are visible and which indexes are searched by default.

If the security index is NOT one of your default indexes, you may be able to search it explicitly:

index=security sourcetype=snort

If that doesn't work, perhaps the Splunk admin has not given you access to the security index at all.

[Drainy]

The summary page only displays data in the main index by default, so it won't register detail on other indexes.

Related: http://splunk-base.splunk.com/answers/47879/cannot-see-data-that-gets-indexed-on-summary-page

[rmcdougal]

It is configured to forward to the "security" index. It is using a heavy forwarder because that is what my system admin felt most comfortable installing.

[Ayn]

Also, if I recall correctly the time on the summary page is when the last event was INDEXED, not necessarily when it was actually generated.

[miteshvohra]

And, why 'Heavy Forwarder' and not 'Universal'?

[lguinn2]

What index does the forwarder specify for the snort data in inputs.conf?