Deployment Architecture

ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

ahmemohs03
Explorer

07-18-2018 21:20:40.725 +0000 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see:
07-18-2018 21:20:40.736 +0000 INFO WatchedFile - Will begin reading at offset=392049 for file='/welldata/splunk/var/log/introspection/disk_objects.log'.
07-18-2018 21:20:40.740 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/welldata/splunk/var/log/introspection/http_event_collector_metrics.log'.
07-18-2018 21:20:40.799 +0000 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.
07-18-2018 21:20:40.967 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/welldata/splunk/var/log/splunk/btool.log'.
07-18-2018 21:20:40.970 +0000 INFO WatchedFile - Will begin reading at offset=3894 for file='/welldata/splunk/var/log/splunk/splunkd-utility.log'.
07-18-2018 21:20:40.977 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/welldata/splunk/var/log/splunk/searchhistory.log'.
07-18-2018 21:20:40.981 +0000 INFO WatchedFile - Will begin reading at offset=238867 for file='/welldata/splunk/var/log/splunk/splunkd_access.log'.
07-18-2018 21:20:40.998 +0000 INFO WatchedFile - Will begin reading at offset=3141787 for file='/welldata/splunk/var/log/splunk/audit.log'.
07-18-2018 21:20:41.001 +0000 INFO WatchedFile - Will begin reading at offset=933 for file='/welldata/splunk/var/log/splunk/conf.log'.
07-18-2018 21:20:41.020 +0000 INFO WatchedFile - Will begin reading at offset=2076287 for file='/welldata/splunk/var/log/splunk/health.log'.
07-18-2018 21:20:43.337 +0000 INFO IntrospectionGenerator:resource_usage - RU_main - I-data gathering (Resource Usage) starting; period=10s
07-18-2018 21:20:43.349 +0000 INFO IntrospectionGenerator:resource_usage - RU_main - I-data gathering (IO Statistics) starting; interval=60s
07-18-2018 21:20:46.023 +0000 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
07-18-2018 21:20:48.590 +0000 INFO ExecProcessor - message from "python /welldata/splunk/etc/apps/splunk_monitoring_console/bin/dmc_config.py" Cannot detect SHC status because of License Restriction. Will not disable DMC.
07-18-2018 21:21:10.392 +0000 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
07-18-2018 21:52:08.785 +0000 WARN TcpInputProc - Stopping all listening ports. Queues blocked for more than 300 seconds
07-18-2018 21:52:08.785 +0000 INFO TcpInputProc - Stopping IPv4 port 9997
07-18-2018 21:59:59.999 +0000 INFO ExecProcessor - setting reschedule_ms=3600002, for command=python /welldata/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py
07-18-2018 23:00:00.003 +0000 INFO ExecProcessor - setting reschedule_ms=3599997, for command=python /welldata/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py
07-19-2018 00:00:00.000 +0000 INFO ExecProcessor - setting reschedule_ms=3600000, for command=python /welldata/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py
07-19-2018 00:00:00.000 +0000 INFO ExecProcessor - setting reschedule_ms=86400000, for command=python /welldata/splunk/etc/apps/splunk_instrumentation/bin/schedule_delete.py
07-19-2018 00:00:00.945 +0000 INFO LMStackMgr - should rollover=true because _lastRolloverTime=1531872000 lastRolloverDay=1531872000 snappedNow=1531958400
07-19-2018 00:00:00.945 +0000 INFO LMStackMgr - finished rollover, new lastRolloverTime=1531958400
07-19-2018 00:00:28.945 +0000 INFO LMSlaveInfo - Detected that masterTimeFromSlave(Wed Jul 18 23:59:27 2018) < lastRolloverTime(Thu Jul 19 00:00:00 2018), meaning that the master has already rolled over. Ignore slave persisted usage.
07-19-2018 01:59:59.999 +0000 INFO ExecProcessor - setting reschedule_ms=3600002, for command=python /welldata/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py
07-19-2018 03:00:00.001 +0000 INFO ExecProcessor - setting reschedule_ms=3599999, for command=python /welldata/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py
07-19-2018 03:01:00.599 +0000 WARN TelemetryHandler - 1531872000.000000

Tags (1)
0 Karma

lmaclean
Path Finder

Hi @ahmemohs03,

So after the other comments think I know what part of the issue is...

Splunk by default does not have the listening ports enabled i.e. 9997. So you will need to edit the inputs.conf file and insert then restart splunkd:

[splunktcp://9997]
connection_host = dns

Note: By default Splunk will use "connection_host = ip" meaning that the "host" field will come up as the IP address and not use the DNS name...

Note 2: You could also use the command line to enable the listening port:

/opt/splunk/bin/splunk enable listen 9997

Confirm the configuration by the command line via:

/opt/splunk/bin/splunk display listen

Note: You will have to login but you should see the following line (or whatever port you enabled)

Receiving is enabled on port 9997.

The "X509Verify" message is more of a hint/tip that for enterprises or for production use, you should use a proper PKI solution (Certificates) either from a 3rd-party or if you have your own Certificate Service then use that. As the Splunk CA is one that gets shipped out in every download and as such the communications aren't as secure...

For in the logs on your Linux A system (Splunk Enterprise) you should see something similar to the below lines for the various port inputs usually after all of the "HotDBManager" & "IndexWriter" messages...

07-22-2018 12:49:57.748 +1000 INFO  TcpInputConfig - IPv4 port 517 is reserved for raw input
07-22-2018 12:49:57.748 +1000 INFO  TcpInputConfig - IPv4 port 517 will negotiate s2s protocol level 4
07-22-2018 12:49:57.748 +1000 INFO  TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk
07-22-2018 12:49:57.748 +1000 INFO  TcpInputConfig - IPv4 port 9997 will negotiate s2s protocol level 4
07-22-2018 12:49:57.754 +1000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 514 with Non-SSL
07-22-2018 12:49:57.755 +1000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 515 with Non-SSL
07-22-2018 12:49:57.768 +1000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 516 with Non-SSL
07-22-2018 12:49:57.768 +1000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 517 with Non-SSL
07-22-2018 12:49:57.768 +1000 INFO  TcpInputProc - Creating fwd data Acceptor for IPv4 port 9997 with Non-SSL

Regarding the web GUI on Linux A, any configuration on Linux B will have zero effect upon this... What I would say is confirm that the user that which Splunk is running under has permissions to open ports, or has ownership of the entire "/opt/splunk/" folder structure.

For this would you mind sending in what your web.conf looks like, also check your splunkd.log for any errors, I believe if you look after the "TailReader" / "TailingProocessor" entries there should be couple items from "loader" talking about REST HTTP server, then two X509Verify entries... If there are issues this is where it probably will mention what is causing the web gui from working. Could possibly need to look at your mongod.log file as well to make sure that it starting correctly (esp. if you have changed the sslPassword setting within server.conf):

...
07-22-2018 12:50:05.910 +1000 INFO  TailingProcessor - Adding watch on path: /var/adm.
07-22-2018 12:50:05.910 +1000 INFO  TailingProcessor - Adding watch on path: /var/log.
07-22-2018 12:50:05.917 +1000 INFO  loader - Limiting REST HTTP server to 1365 sockets
07-22-2018 12:50:05.917 +1000 INFO  loader - Limiting REST HTTP server to 303 threads
07-22-2018 12:50:05.917 +1000 WARN  X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts y
our Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-
signcertificates>
07-22-2018 12:50:06.291 +1000 WARN  X509Verify - X509 certificate (O=SplunkUser,CN=splunk) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instan
ce at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates>

When you restart Splunk from the command line, I would recommend using the splunk binary instead of "systemctl" if that is what you are using as the "splunk" binary will provide some basic output on each stage... e.g.

Stopping splunk helpers...
                                                           [  OK  ]
Done.

Splunk> Like an F-18, bro.

Checking prerequisites...
        Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking appserver port [127.0.0.1:8065]: open
        Checking kvstore port [8191]: open
        Checking configuration...  Done.
        Checking critical directories...        Done
        Checking indexes...
                Validated: _audit _internal _introspection _telemetry _thefishbucket add_on_builder_index car_data cim_modactions cim_summary firedalerts history main os os_metrics perfmon pos_pu sophos summary synology syslog unifi windows wineventlog
        Done
        Checking filesystem compatibility...  Done
        Checking conf files for problems...
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunk/splunk-7.1.0-2e75b3406c5b-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done                                                           [  OK  ]
Waiting for web server at https://127.0.0.1:8000 to be available......................... Done

If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at https://splunk:8000
0 Karma

sudosplunk
Motivator

@ahmemohs03, I see you're getting good help from the folks here.
But, let me ask you some questions,
1. Can you ping your indexer (Linux A) from your forwarder (Linux B) and vice versa?
2. Did you check to see if ports 9997 and 8000 are open on Linux A?

0 Karma

auraria1
Path Finder

Is this your first time starting this splunk forwarder?
I've add issues after running accept license where I need to stop the splunk instance and re-start it on versions 7.1 and above.

0 Karma

ahmemohs03
Explorer

Hi Auraria

Yes, first time after installating universal splunk forwarder..its 7.1.1 version.
splunk on A Linux server where splunk installed, Universal forwarder is on B Linux server, were the logs of Linux B will be seen on Linux A.
Do I need to start splunkforwarder also, along with splunk?

0 Karma

auraria1
Path Finder

Both need to be running for logs to be indexed.

I'd try restarting the forwarder:

/directory to splunk/splunkforwarder/bin/splunk stop
/directory to splunk/splunkforwarder/bin/splunk start

Then check the splunkd logs, let me know if that fixes the issue.

Quick tip on troubleshooting and splunkd logs, rename the log to splunkd1 before starting the splunk forwarder and increase the number everytime so it creates a fresh log and you can compare the two.

0 Karma

ahmemohs03
Explorer

Thanks for quick reply...

Restarted splunk forwarder, getting this error on splunkd.logs
Weburl is not coming up. http://hostanme:8000

WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see:
07-19-2018 18:31:56.698 +0000 INFO WatchedFile - Will begin reading at offset=508858 for file='/welldata/splunk/var/log/introspection/disk_objects.log'.
07-19-2018 18:31:56.702 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/welldata/splunk/var/log/introspection/http_event_collector_metrics.log'.
07-19-2018 18:31:56.750 +0000 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

Please advice

0 Karma

patilsonali1729
Path Finder

1.Splunk universal forwarder does not have web UI.
2.Where is your outputs.conf file located? Please run ./splunk btool --debug outputs list and paste the response here.

0 Karma

patilsonali1729
Path Finder

Also, verify if the user running Splunk has read permission to outputs.conf (the user should have access to all the conf files, in short to /opt/splunkforwarder)

0 Karma

ahmemohs03
Explorer

Do you mean -rw------- 1 root root 140 Jul 17 18:39 outputs.conf?

0 Karma

auraria1
Path Finder

I would just change ownership to all the files and folders as it can cause issues in the future in the splunkforwarder directory.

Did the bracket and the disabled = 0 fix the issue or is it still persisting?

0 Karma

ahmemohs03
Explorer

after bracket and the disabled = 0 fix , I did a splunkforwarder restart, but see the error again.
ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

0 Karma

ahmemohs03
Explorer

Do I need to restart splunk also on another server?

0 Karma

auraria1
Path Finder

That wouldn't effect the outputs.conf issue.

Comment this out in the outputs.conf:
from this:
[tcpout-server://12.34.342.87:9997]

to this:

[tcpout-server://12.34.342.87:9997]

And try again.

0 Karma

ahmemohs03
Explorer

in forwarder(splunkd.logs) I saw this error:

20:28:34.393 +0000 WARN TcpOutputFd - Connect to 12.34.342.87:9997 failed. Connection refused
07-19-2018 20:28:34.393 +0000 ERROR TcpOutputFd - Connection to host=12.34.342.87:9997 failed.

0 Karma

lmaclean
Path Finder

Hi ahmemohs03,

What does the inputs.conf look like on your Linux A to receive these connections? Also what do you mean about weburl not coming up?

Is this actually about two issues:
One the data from the UF (Linux B) -> Splunk Enterprise (Linux A)?
Two the issue with Splunk Enterprise Web not coming up?

0 Karma

ahmemohs03
Explorer

Q. What does the inputs.conf look like on your Linux A to receive these connections?
inputs.conf (below)
[default]
host = ijckollwl12(hostname)

Q. what do you mean about weburl not coming up?
A. Weburl of splunk(linux A)were the logs need to monitor.
Weburl : http://hostame:8000(not coming up after installation of universal forwarder on Linux B.

One the data from the UF (Linux B) -> Splunk Enterprise (Linux A)?
Yes, UF(Linux B) logs need to forward at Splunk Enterprise (Linux A) weburl(http:hostname:8000).

Two the issue with Splunk Enterprise Web not coming up?
Yes, Splunk Enterprise Web not coming up(http://hostname:8000)

0 Karma

patilsonali1729
Path Finder

DO you get any warning/error when you restart Splunk Enterprise instance??
Ideally, once the restart is complete it should give a message like: The Splunk web interface is at https://hostname:8000

0 Karma

ahmemohs03
Explorer

Thanks with this error is gone, but splunk weburl not coming up, any suggestions how I can up splunk weburl?

0 Karma

auraria1
Path Finder

You're welcome! there is nowebui on the forwarder.

0 Karma

ahmemohs03
Explorer

Dnt got you, you mean to remove [tcpout-server://12.34.342.87:9997] from outputs.conf file?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...