Unknown source sending logs

[dhs_harry08]

Hi,

I am in a weird situation. I have recently joined a new company. The guy who setup splunk left the company after upgrading the splunk and the PCI app "Splunk App for PCI Compliance" to latest version. After upgrade we are getting logs from various sources which I am not sure if they are genuine or not. I only have access to the splunk server but not the clients. I have previous knowledge of splunk setup.

When I check the logs in splunk the source is something like this source=/opt/splunk/var/spool/splunk/singlehost.sample.sav hostname="splunk server", and sourcetype="sav".

Now I logged into the server console through putty to see the original logs in /opt/splunk/var/spool/splunk/ but I cant find any. Could someone suggest me how to troubleshoot this issue or if anyone else faced this issue please help me.

Regards,
Harish

[dhs_harry08]

I found the solution at last. These events were generated by an app by splunk SA-Eventgen. We were using PCI compliance app and while installation it seems this app was configured by default. After disabling the app these sample logs disappeared.

I found the solution in the Splunk security app FAQs.
http://docs.splunk.com/Documentation/ES/2.4/User/FAQ#How_do_I_manually_enable_eventgen.3F

But the same is not included in PCI compliance app documentation. Hope splunk sees this and update the PCI compliance app documentation.

Regards,
Harish

[kristian_kolb]

move_policy=sinkhole is indeed connected with batch - and the docs state that you must not change that to any other value.

[martin_mueller]

Stuff in var/spool/splunk may be an indicator for custom collect statements doing some data processing - check if there are any scheduled searches that end with a call to collect - do not run them without knowing what you're doing though!

[dhs_harry08]

I think it is the "move_policy = sinkhole" in the stanza in inputs.conf that is causing this. But is it possible to remove this, that is it could be a default configuration in splunk and if I edit it will it affect my splunk server negetively.

[kristian_kolb]

Isn't there a batch stanza for that directory in /etc/system/default/inputs.conf. I.e. that splunk will eat files in there destructively.

Don't have a splunk in front of me right now. can't check.