How do we overcome high number of sourcetypes and incorrect ingestion in Splunk Cloud instance?

[AL3Z]

Hi All,

In our Splunk Cloud instance, we are experiencing a significant increase in the number of sourcetypes, and it seems that a considerable portion of these sourcetypes are being ingested incorrectly. This incorrect ingestion is likely causing an inflation in the overall count of sourcetypes.
How do we over come from this issue?

[glc_slash_it]

Hi,

where is this information coming from? On-prem forwarder? If so, check your any recently updated inputs.conf files.

Maybe there is an incorrect configuration to index all files in a folder or sub-folder.

[AL3Z]

@glc_slash_it ,

I believe when I query the data   | metadata index=* type=sourcetypes it  appear to be ingesting incorrectly.
How do we correct the ingestion of these sourcetypes ?

[glc_slash_it]

Hi,

That command:  "| metadata index=* type=sourcetypes" is just a way of seeing metadata on which sourcetypes are being applied in splunk. It has nothing to do with the actual events/logs of your information.

For that you should search the specific indexes (e.g. index=abc) and:

- first check if the important fields are correct: _time, host, sourcetype and source

- second check for any field extractions, if they are also correct and then analyze if you need any more field extractions for you use cases.

[AL3Z]

@glc_slash_it ,

I'm seeing  a huge number of buffer.* sourcetype which are present in one of the index, how do we change the sourcetype  into single sourcetype ?

Thanks

[glc_slash_it]

That depends on the source of the logs and how you are ingesting them into Splunk.

Usually you need to configure the inputs.conf file to tell Splunk which files to ingest and which sourcetype should be applied to those files. Something like this:

[monitor:///home/jenkins/folder/dir]

host = <name_of_the_server_producing_logs>

sourcetype = <desired_sourcetype> (can be a known sourcetype like "cisco:asa" or a custom one)

index = abc

disabled = false

But again this is just an example, it depends on how you are collecting the logs and Splunk has many ways to configured log collection.