Getting Data In

Why am I unable to search data from Splunk servers when the time period is set for previous week ?

Hemnaath
Motivator

Hi All, Can any one guide me why I am unable to fetch the data from index=_internal host=splunk1 sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log" when time period is set for last month? This is happening only servers related with the Splunk instances. So kindly help us in troubleshooting this issue.

thanks in advance.

0 Karma

Hemnaath
Motivator

Hi Somesoni, we are getting data when we keep the duration for 15min or even I could see data for last 7 days but when we set for more that then we are getting no result found. But I need to how to check whether the _internal indexes are full ? as we have 15 splunk instance running in our environment.
Kindly guide me on this..

Wish you a merry Christmas and Happy New Year.

thanks in advance

0 Karma

lguinn2
Legend

Are your indexes full? Regardless of the retention period, once the _internal index fills the allocated space, it will remove the oldest data to ensure that it does not exceed that size.

0 Karma

Hemnaath
Motivator

thanks somesoni for quick response, Yes I am able to get the data when we search with the query index=_internal sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log.

Retention period is set as 30 days

but when we search with the time period for last 10 days, we are getting no result found. As per retention we should get the data right .

Kindly guide me to trouble shoot this issue.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Are you getting the data for recent time ranges like last 15 mins, last 24 hrs etc? Every Splunk instance generates splunkd logs rather frequently so if your Splunk servers (search head/deployment servers etc) are sending data to your Splunk indexers, you'll see the data for these recent time ranges. If no then probably your Splunk servers are not sending their internal data to Splunk at all. For that you need to check outputs.conf on those Splunk servers to see if it exists and if yes, are they referencing your Splunk indexers?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

So if you just search this you get data?

index=_internal sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log" 

What is the retention period of _internal index in your indexers? Run this query and check

| rest /services/data/indexes/_internal | table title splunk_server frozenTimePeriodInSecs | eval RetentionDays=frozenTimePeriodInSecs/86400 | rename splunk_server as Indexer
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...