Getting Data In

Cannot see the data that is being forwarded/indexed in the Splunk web interface

ghoskiller
New Member

Hi everyone,
I am currently facing an issue which am not getting my head around it. I have installed the universal forward in win srv 2012r2 to send every log to Splunk server. However, In the Splunk web interface, I cannot see the data that is being forwarded/indexed. I have done a Tcpdump to monitor traffics on port 9997.

I can see that the communication is being made between the Splunk server and the windows machine on that port, however, I cannot see the data being indexed or displayed on the graphic. Can anyone tell me where does the data that is being collected usually stored? it is indexed on the default index or somewhere else. Because so far I cannot find it in the default index or where ever.
Thanks in advance.

0 Karma

adonio
Ultra Champion
0 Karma

pruthvikrishnap
Contributor

can you help me with the inputs and outputs which you have used while configuring on UF.

0 Karma

ghoskiller
New Member

Hi adonio, the info inside the

outputs.conf

Version 7.3.1

[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
forwardedindex.filter.disable = false

input.conf

Version 7.3.1

these here just override and disable stuff that in system/default.

Data thru parsingQueue always

[splunktcp]
route=has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue

Make sure these get forwarded

[monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log]
_TCP_ROUTING = *
index = _internal

[monitor://$SPLUNK_HOME\var\log\splunk\metrics.log]
_TCP_ROUTING = *
index = _internal

I hope this helps. thanks in the advance.
Harguilar Nhanga.

0 Karma

ghoskiller
New Member

I just had a look at the logs files this is what am getting. However I do dont understand why this is refusing connection if I can see from the tcpdump the connection hitting on the server and I do not have firewall configure in the linux Machine. My scenario I am using Windows Universal Foward to Fowards logs to a SPlunk server that is a Linux Machine. Below you can see some of the logs.

09-18-2019 17:44:31.351 -0700 INFO WatchedFile - Will begin reading at offset=5800411 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\health.log'.
09-18-2019 17:44:31.367 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage.log'.
09-18-2019 17:44:31.367 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\btool.log'.
09-18-2019 17:44:31.367 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage_summary.log'.
09-18-2019 17:44:31.383 -0700 INFO TailReader - Registering metrics callback for: batchreader0
09-18-2019 17:44:31.383 -0700 INFO TailReader - Starting batchreader0 thread
09-18-2019 17:44:31.399 -0700 INFO UiHttpListener - Web UI disabled in web.conf [settings]; not starting
09-18-2019 17:44:32.398 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:44:32.398 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:44:33.413 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:44:33.413 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:44:33.413 -0700 WARN TcpOutputProc - Applying quarantine to ip=192.168.0.12 port=9997 _numberOfFailures=2
09-18-2019 17:45:00.742 -0700 INFO TcpOutputProc - Removing quarantine from idx=192.168.0.12:9997
09-18-2019 17:45:00.882 -0700 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
09-18-2019 17:45:00.882 -0700 INFO FileAndDirectoryEliminator - Enabled
09-18-2019 17:45:01.773 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:45:01.773 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:45:02.789 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:45:02.789 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:45:02.789 -0700 WARN TcpOutputProc - Applying quarantine to ip=192.168.0.12 port=9997 _numberOfFailures=2
09-18-2019 17:45:30.680 -0700 INFO TcpOutputProc - Removing quarantine from idx=192.168.0.12:9997
09-18-2019 17:45:31.679 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:45:31.679 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:45:32.679 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:45:32.679 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:45:32.679 -0700 WARN TcpOutputProc - Applying quarantine to ip=192.168.0.12 port=9997 _numberOfFailures=2
09-18-2019 17:47:00.320 -0700 INFO TcpOutputProc - Removing quarantine from idx=192.168.0.12:9997
09-18-2019 17:47:01.351 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:47:01.351 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:47:02.351 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:47:02.351 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:47:02.351 -0700 WARN TcpOutputProc - Applying quarantine to ip=192.168.0.12 port=9997 _numberOfFailures=2

0 Karma

itrimble1
Path Finder

Have you checked your firewall settings ? Is port 9997 open on 192.168.0.12 ?
Have you checked on both on the Windows Side and the Linux side ?

Are you using SELinux ?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...