Why are we receiving inconsistent data?

[vxl65703]

I am a first time, I have a user who says his search heads are kicking different results on two different search heads.

We are using Splunk 6.3

[DalJeanis]

Have them give you the exact search they are running, and the results.

First, make sure that the search they are kicking off is a consistent search. Make sure it has a fixed earliest and latest value, and so on. If they are kicking off "last 30 minutes" on one head then later on another had, then of course the answer will be different.

Second, run that query yourself on each head. See if your results are the same or different from his. Ideally, limit the search to as small a time range and amount of detail as possible, as long as he gets a different result on each head.

If there are lookups in the search, or joins, then run those subsearches independently and check whether they are consistent on the two heads. Perhaps a lookup isn't propagating fully, or whatever.

[vxl65703]

Yes they are a part of a search head cluster

[pradeepkumarg]

Are the search heads in cluster? or independent? You might want to verify the search peers for both of these search heads. You will have to ask more details to your user like what is the search? time range selected? Is he using any lookups that are available on one SH and not other? Are the results always consistently different ? Did he take a look at the job inspector to see if there were any errors from a search peer that had trouble sending data back ?

[vxl65703]

the search heads are in a cluster, one instance pulls up data the other instance only pulls up a part of the data, would.

Would need to look at the indexers or forwarders to see if either is corrupt ?

Would I need to SSH into the servers to review the indexers/forwarders ?

[jkat54]

To be sure, these search heads are part of a SEARCH HEAD CLUSTER? Yes/no?

Noting SEARCH HEAD CLUSTER is not the same as a SPLUNK CLUSTER (which is the general term used for a cluster of indexers, a cluster master, license master, and search head(s))

[vxl65703]

Yes these are search heads apart of the cluster.

[jkat54]

-hangs head-

Are they part of a search head cluster?