Why is there Missing data on ingestion?

[ReginaP]

Brand news servers. Not receiving all data from the UF.
Confirmed connectivity.
Confirmed inputs via "/opt/splunkforwarder/bin/splunk btool inputs list | grep bc_ | grep "\["",
Only getting 2 sourcetypes when there should be at least 16 for the index.

Getting this error message:
Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false).

Getting this when starting splunkd:

Splunk> Take the sh out of IT.

Checking prerequisites...
        Management port has been set disabled; cli support for this configuration is currently incomplete.
        Checking conf files for problems...
                Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.0.3-dd0128b1f8cd-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done

[ReginaP]

It was a RHEL8 python issue. Thank you for your responses

[isoutamo]

Hi

which user you are running splunkd on UF? You should use separate non root user. But this means that you must give access to this user to access those files which you want to read. You should try to check if user have access those e.g.

sudo -s root bash
sudo -s <your splunk user> bash
less /path/to/log/file

If you cannot see the content of that file you need to give access by e.g. "setfacl" command. 

r. Ismo

[ReginaP]

We have root running splunkd on all our UFs with no issues.

[isoutamo]

Actually this is huge security issue, but it shouldn't affect to your current issue.

When you do that "splunk btool inputs ...." you get 16+ sourcetypes of inputs, but on splunk there is only 2 of them?

What you get with 

splunk list inputstatus

For your missing bc_* inputs? There should be something like

/your/input/file/bc_something
	file position = 631
	file size = 631
	parent = /var/log
	percent = 100.00
	type = finished reading

[gcusello]

Hi @ReginaP,

did you disabled the local firewall on the system?

did you checked the connectivity between the forwarder and the Indexer using telnet on the port you configure for receiving?

did you enabled receiving on the Indexers?

the "Invalid key" error isn't relevant, is instead very strange the message "Management port has been set disabled; cli support for this configuration is currently incomplete".

for the second message, you could see the solution at https://community.splunk.com/t5/Getting-Data-In/Unable-to-start-Splunk-forwarder/m-p/386916 but anyway, iy shouldn't have effects on the log forwarding.

Ciao.

Giuseppe

Ciao.

Giuseppe

[ReginaP]

I checked the connection via telnet to the correct port.