No tcpin_connections (group) for _internal index

[edwinmae]

Hi,

Does anybody know what could be the cause why the tcpin_connections (group) is missing entirely from _internal index?

This search for checking the Forwarders (see below) worked just fine in the past. Currently our server and Forwarders run 6.5.0. Now it says that 'No results are found' (as there is no tcpin_connections group). tcpout_connections group is visible though.

Also netstat -an shows established connections for port 9997 on Linux (Splunk) server

index=_internal source=*metrics.log group=tcpin_connections   | eval sourceHost=if(isnull(hostname), sourceHost,hostname)   | rename connectionType as connectType  | eval connectType=case(fwdType=="uf","Universal Forwarder", fwdType=="lwf", "lightwt fwder",fwdType=="full", "heavy fwder", connectType=="cooked" or connectType=="cookedSSL","Splunk fwder", connectType=="raw" or connectType=="rawSSL","legacy fwder")  | eval version=if(isnull(version),"pre 4.2",version)  | rename version as Ver   | fields connectType sourceIp sourceHost destPort kb tcp_eps tcp_Kprocessed tcp_KBps splunk_server Ver os arch  | eval Indexer= splunk_server  | eval Hour=relative_time(_time,"@h")  | stats avg(tcp_KBps) sum(tcp_eps) sum(tcp_Kprocessed) sum(kb) by sourceHost sourceIp os arch connectType destPort Indexer Ver  | sort Ver

--

Thanks in advance for Support!

[jcrabb_splunk]

That is strange, I tested your search in my 6.5 environment and I get results. If you just look at the ingested metrics logs, do you see that group? Rather, if you run:

index=_internal source=*metrics.log group=tcpin_connections

Does that yield results? Or:

index=_internal source=*metrics.log | stats count by group

Do you see the various groups? If the answer is no, if you search previous 30 days is there any change in the results? Your search and the ones I've listed above work in 6.5 on my instance so hopefully its just a straight forward issue.

[edwinmae]

• index=_internal source=metrics.log group=tcpin_connections (for let's say last 24 hours) does not provide any results
• When searching for e.g Last 30 days I do get the (normal) results -- last event was before the upgrade --

10-03-2016 02:32:38.451 +0100 INFO StatusMgr - destPort=9997, eventType=connect_close, group=tcpin_connections, sourceHost=xx.xx.xx.xx, sourceIp=xx.xx.xx.xx, sourcePort=58796, statusee=TcpInputProcessor

• index=_internal source=*metrics.log | stats count by group (for let's say last 24 hours)

still no tcpin_connections

I also hope it's a straight forward issue, except I have not been able to find it yet ...

[lguinn2]

Hmm, did you accidentally change some settings that control either the log channels or the indexing of internal logs (.../var/log/splunk)?

[edwinmae]

During the upgrade to 6.5 there were some challenges as we got an error:

Exception: , Value: [Errno 13] Permission denied: '/opt/splunk/etc/system/local/indexes.conf'

We decided to delete that file, after which the upgrade 'process' went just fine.
Now that the tcpin_connections 'group' seems to be missing, the upgrade probably not went as it should (for 100 %)
The forwarders itself work fine as we have the incoming 'data'.

Is there an easy way to fix this or how can this be resolved?

[edwinmae]

We removed props.conf and transform.conf (from local) after which the functionality was restored