I have an forwarder that's set up to monitor a log file at the location: /var/log/mhn/mhn-splunk.log.
inputs.conf on forwarder:
[monitor:///var/log/mhn/mhn-splunk.log]
sourcetype = mhn
index = mhn
disabled = false
outputs.conf on forwarder:
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = Dest IP:9997
[tcpout-server://Dest IP:9997]
On the forwarder
I have verified connection using netstat
tcp 0 0 0.0.0.0:8089 0.0.0.0: LISTEN 5600/splunkd*
tcp 0 0 Source IP:48652 Dest IP:9997 ESTABLISHED 5600/splunkd
Checked splunkd.log
02-22-2018 02:04:04.790 -0500 INFO TcpOutputProc - Connected to idx=Dest IP:9997, pset=0, reuse=0.
02-22-2018 02:27:07.846 -0500 INFO TcpOutputProc - Connected to idx=Dest IP:9997, pset=0, reuse=0.
02-22-2018 02:29:03.860 -0500 INFO TcpOutputProc - Connected to idx=Dest IP:9997, pset=0, reuse=0.
On the indexer:
I have verified the index, mhn, exists and is enabled.
Listener is setup on the right port
tcp 0 0 0.0.0.0:9997 0.0.0.0: LISTEN 31490/splunkd
tcp 0 0 0.0.0.0:8089 0.0.0.0: LISTEN 31490/splunkd*
tcp 0 0 0.0.0.0:8000 0.0.0.0: LISTEN 31490/splunkd*
tcp 0 0 Dest IP:9997 Source IP:48652 ESTABLISHED 31490/splunkd
metrics.log is showing as receiving the events from the forwarder
02-21-2018 23:40:19.593 -0800 INFO Metrics - group=tcpin_connections, Source IP:48652:9997, connectionType=cooked, sourcePort=48652, sourceHost=Source IP, sourceIp=Source IP, destPort=9997, kb=7.95, _tcp_Bps=262.59, _tcp_KBps=0.26, _tcp_avg_thruput=0.52, _tcp_Kprocessed=346.17, _tcp_eps=0.19, _process_time_ms=0, evt_misc_kBps=0.00, evt_raw_kBps=0.23, evt_fields_kBps=0.00, evt_fn_kBps=0.00, evt_fv_kBps=0.00, evt_fn_str_kBps=0.00, evt_fn_meta_dyn_kBps=0.00, evt_fn_meta_predef_kBps=0.00, evt_fn_meta_str_kBps=0.00, evt_fv_num_kBps=0.00, evt_fv_str_kBps=0.00, evt_fv_predef_kBps=0.00, evt_fv_offlen_kBps=0.00, evt_fv_fp_kBps=0.00, build=03bbabbd5c0f, version=7.0.2, os=Linux, arch=x86_64, hostname=ubuntu, guid=BEB9358D-17D6-4C65-B408-99DF4C038DFA, fwdType=uf, ssl=false, lastIndexer=Dest IP:9997, ack=false
Can't quite figure out why I'm not seeing the events in index=mhn. I was hoping the Splunk community might be able to tell me if there was anything I was missing.
Just in case timestamps are not being parsed correctly, try searching index=mhn
over All Time.
You should check the logs and see if your forwarder is sending over data. You can also check the forwarder logs
Run this search
index=_internal sourcetype=splunkd
Just in case timestamps are not being parsed correctly, try searching index=mhn
over All Time.
@richgalloway, you were right. There's something messed up with the timestamps. I'll write another question on how to untangle that mess. Thank you for pointing me in the right direction.