Please advise. How do I Check for data gaps, broken events & incorrectly indexed data in Splunk Ent. Thank u

[SamHTexas]

As a deep dive into my data sources / data integrity. I need to learn what SPLs /Apps needs to be used for this purpose. I appreciate your help.

[richgalloway]

That sounds very site-specific.  Tell us more about your needs.  What constitutes a data gap?  What is a broken event?  What do you mean by "incorrectly indexed data"?

[SamHTexas]

Thank u for your message. Broken events are events that looks like they are there but don't make sense - they are incomplete, same with incorrect indexed data. I get errors that are caused when someone re-starts Splunk instances in the middle of the day or Indexers lose communication with each other or there are Internet lapses. I hope I made sense. Is there a way to see incomplete events & why they occurred ? Thank u

[codebuilder]

If you want to see gaps in data ingestion, such as days or hours where no data came in you can run this:

| tstats count where index=your_index_name by _time

Then just click on "visualization" and you'll get a nice graph of event count over a timeline (controlled by your date/time picker).

You can drill down further on the search to visualize by day, hour, seconds, etc.

[SamHTexas]

Thank u. So what is Data gap per your experience. I did get bar code & data but could not make sense of it. Does data gap really mean no data was ingested? What causes that per your knowledge? 

[codebuilder]

The bars represent the number of events indexed by Splunk on a given day/time and is based on the range you selected with the date/time picker. The timestamp is shown below the bars. You can left click/hold on the graph and drag across it to drill down into smaller time ranges.

To me, a data gap would mean no events received for some period of time. That would indicate a problem with the forwarders, hosts, the indexers, etc.

Gaps can be normal, such as on weekends or holidays, for example. It just depends on your specific environment as @richgalloway  mentioned.