Splunk (Search Head 1) fields extracted is different from (Search Head 2) fields

[Oracle]

Hi Guys,

I am confused right now with the OS nix data that are ingesting right now in our splunk, we have 2 search head btw.

When i search this query "(index=* tag=oshost tag=performance tag=cpu) " on both search head the fields are different. What would be the problem why the fields are different from each other?

Search head 1 Result:
---> The fields on this search head 1 was extracted the way we need it like for E.g (mem used & mem free).

Search head 2 Result:
---> The fields that we are seeing is the splunk default fields like for E.g (host, line count, index, tag). For us to be able to see the same fields on search head 1 we need to add/used "multikv" on our query.

I already checked the tag, eventtype, & user permission that we are using, seems to be fine.

Any suggestions would be appreciated. Thanks,

--
Michael

[akocak]

Technically, same settings from Search head deployer is sent to all search head instances. The way you sounded in your question first reminds me search modes of Splunk: fast, smart and verbose
Are you sure search is made on smart or verbose on both searches ? is it possible second search is made on fast mode ?

if it is not the case:
one thing would be comparing props.conf and transforms.conf on both search heads as well as SH deployer as mentioned.

if this is not the case also, on splunk CLI

    ./splunk cmd btool props list --debug | grep <Field_that_you_are_looking for>

you should look if SH gets setting from same file on each search head. Especially a search head is added to cluster later than initial members, I had couple cases where previous users added some stuff to etc/system/local that conflicts with my changes from search head deployer.
Thanks.

[Oracle]

Hi akocak,

Sorry for the late response
Will try to look the configuration file and get back to you as soon.

Thanks..

[deepashri_123]

Hey@ Oracle,

Try comparing the props.conf and transforms.conf on both the Search Heads.This should help you find where the problem is. You can use btool on both the search heads for comparison.

Let me know if this helps!!

[Oracle]

Hi deeparshri,

Sorry for the late response
Will try to look the configuration file and get back to you as soon.

Thanks..