Why is the data in index from using collect is not being indexed?

[caseyra]

Hello,

I'm trying to take a list of IDs that were given to me and confirm that they are in our system. I imported the data into a lookup table and then use the collect command to put the data into a temporary index that was created some time ago for other reasons. The data ends up in the index, but the data does not appear to be indexed. For example, if I run a query like this, I get results:

index=temp sourcetype=idlist

However, the following query will bring back 0 results:

| tstats count as total where index=temp by ID

ID is the new field that has the ID that we're looking for. Why is the second query not working?

Some additional information: we're doing this on a stand alone search head connected to a indexer cluster. Originally we tried to create a local index for the IDs so that we could easily delete it afterwards, but it looked like it was expecting the index to be in the cluster and not local, hence why we switched to using the temp index.

Thanks.

[tiagofbmm · Accepted solution]

Hey

The reason for that is ID is not a Splunk indexed field by default. Only index, source, sourcetype and _time are.

So tstats, working only on indexed fields, can;t give you the results

[tiagofbmm]

More, if you did not specify a sourcetype when you did the collect command, the sourcetype is stash, and with KV_MODE=none, you will get no search time extractions.

Only the metadata fields are shown (index, source, sourcetype, host, _time)

[caseyra]

I did specify a source type, but time field isn't what we need.

Can we force Splunk to index the ID field? Or, is that something that needs to be configured when you create the index and before you ingest any data?

[tiagofbmm]

To force Splunk to have a new indexed field, you need to do that on your sourcetype.

Follow this doc to do that:

http://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Configureindex-timefieldextraction

As collect indexes data in a summary index, you should take a look here to find out how to configure the summary index to have your fields there:

http://docs.splunk.com/Documentation/Splunk/7.0.2/Knowledge/Configuresummaryindexes

[caseyra]

Thanks. I'll check it all out. Right now, it looks like I have an alternative query running faster than expected, so I think I'm good to go for now.