Splunk Search

Why is the count of indexed events shown under “What to Search” on the default page of the Search & Reporting app much lower than expected?

sat94541
Communicator

Upon login to Splunk, when we navigate to default page for the Search & Reporting app, the count under “What to Search” is incorrect. The number showing is 721,000+ when it should in the billions.

Refer picture to see the issue
all text

Tags (3)
1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

1)First thing to eliminate will be that caused by the fact that "| metadata" (and therefore, that UI Panel) is scoped by default to the default set of indexes that the user in question is set to search - as per http://docs.splunk.com/Documentation/Splunk/6.3.0/SearchReference/Metadata :

As mentioned by the following section in above link

Optional arguments[edit]
index-specifier
Syntax: index=
Description: Specify the index from which to return results. Wildcards () can be used. index= matches non-internal indexes. index=_* matches internal indexes.
Default: The default index, which is usually main
So I'm pretty sure that this is governed by the following role-specific setting in authorize.conf:

srchIndexesDefault =
* Semicolon delimited list of indexes to search when no index is specified
* These indexes can be wildcarded, with the exception that '' does not
match internal indexes
* To match internal indexes, start with ''. All internal indexes are
represented by '
'
* Defaults to none, but the UI will automatically populate this with 'main'
in manager

2)One very simple way to check this is to check what kind of event count you get with:

| metadata type=sourcetypes | stats sum(totalCount) AS total_event_count_of_indexes_searched_by_default

and

| metadata type=sourcetypes index=* | stats sum(totalCount) AS total_event_count_of_all_searchable_indexes

3) In case the search in (2) both return incorrect count, next you can try and rebuild metadata - using the following command

splunk cmd splunkd recover-metadata /localhome/igor/local/core/var/lib/splunk/defaultdb/db/db_1442694953_1442694953_0

This command is referred at link --http://docs.splunk.com/Documentation/Splunk/6.3.0/Troubleshooting/Buckets

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

1)First thing to eliminate will be that caused by the fact that "| metadata" (and therefore, that UI Panel) is scoped by default to the default set of indexes that the user in question is set to search - as per http://docs.splunk.com/Documentation/Splunk/6.3.0/SearchReference/Metadata :

As mentioned by the following section in above link

Optional arguments[edit]
index-specifier
Syntax: index=
Description: Specify the index from which to return results. Wildcards () can be used. index= matches non-internal indexes. index=_* matches internal indexes.
Default: The default index, which is usually main
So I'm pretty sure that this is governed by the following role-specific setting in authorize.conf:

srchIndexesDefault =
* Semicolon delimited list of indexes to search when no index is specified
* These indexes can be wildcarded, with the exception that '' does not
match internal indexes
* To match internal indexes, start with ''. All internal indexes are
represented by '
'
* Defaults to none, but the UI will automatically populate this with 'main'
in manager

2)One very simple way to check this is to check what kind of event count you get with:

| metadata type=sourcetypes | stats sum(totalCount) AS total_event_count_of_indexes_searched_by_default

and

| metadata type=sourcetypes index=* | stats sum(totalCount) AS total_event_count_of_all_searchable_indexes

3) In case the search in (2) both return incorrect count, next you can try and rebuild metadata - using the following command

splunk cmd splunkd recover-metadata /localhome/igor/local/core/var/lib/splunk/defaultdb/db/db_1442694953_1442694953_0

This command is referred at link --http://docs.splunk.com/Documentation/Splunk/6.3.0/Troubleshooting/Buckets

aljohnson_splun
Splunk Employee
Splunk Employee

Those are just the indexes that are searched by default.

If you go to:

  • Settings > User Access > Roles
  • chose a specific role
  • Scroll to the bottom

You will see:
alt text

The counts you're talking about in the data summary are only reflective of the indexes that are searched by default.


Here are a few other ways to get an event count:

| eventcount index=*

Or internal indexes:

| eventcount index=_*

Or try using the metadata command:

| metadata type=sourcetypes index=*

There are many other ways of course.

Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...