Why is the Splunk App for Stream putting all data into _internal, and can I change the target index?

[scc508]

It looks like the Splunk App for Stream puts everything into _internal. In general, we don't like to do that. In this case, do we really need to? It seems like too much volume to put in _internal.

My plan is to capture summary data on most network connections for hundreds of product servers. I can't find anything about index choices in the docs. Does anyone have any recommendations on which index(es) to use? This data will be part of our security app we're building, and access will be restricted as well.

I have some data on a test server, just with the tcp and udp streams enabled as they come by default. The only messages I see in the events are: "decodePacket: skip bogus packet with length less than IP header length". Any idea what that means?

thank you!

Scott

[DenM]

Hello yes you can.

I use this technique to put my streams in different indexes and manage the permissions.

You need to declare the index on your Splunk stream app indexes.conf. to do that follow this steps:

1) On your stream server : SPLUNK_HOME/etc/apps/splunk_app_stream/local
2) edit or create the indexes.conf file
3) declare your indexes (already created on your indexers) like this
[yourindex]
coldpath = $SPLUNK_DB/yourindex/colddb
enableDataIntegrityControl = 0
enableTsdixReduction = 0
homePath = $SPLUNK_DB/yourindex/db
MaxTotalDataSizeMB = 512000 #(in my case)
thawedPath = $SPLUNK_DB/yourindex/thaweddb

4) restart your splunkd service
5) now when you configure your streams in the stream app you can see your indexes in the dropdown menu

Hope it helps

[johnberwick]

This is due to tcp segmentation offload which generates a 0 length IP Header so the stream forwarder skips the data.

A Fix is due out soon, try disabling tcp offload

Regards

[mdickey_splunk]

Stream only puts internal logs and statistics into the "_internal" index (this doesn't count towards your license volume). By default, it will put all events derived from network traffic into the "main" index. You can change the index used for each stream within the UI (at the top of the page, after you click on a particular stream in the list), or the default index for all streams by setting the following parameter in the streamfwd section of your inputs.conf file:

index=somethingelse

Note that the priority for index selection is:

• The config for a specific stream
• index defined in inputs.conf
• main

[scc508]

Thank you both, that helps explain it. Does anyone know what to check why we can't see any data? All we see are the error messages to _internal about "decodePacket: skip bogus packet with length less than IP header length".

Clearly something is not working. This is win2012r2 hosted in Azure, so has a hyperv network card. splunkd is running as localsystem.

Thank you for any insight or steps to try.

[jsie_splunk]

By default only Stream internal information goes to _internal.

The real event data (by default) goes to the main/default index. You can also configure on a per Stream basis and destination index if you like. And finally, if you want to set a global destination, you can modify Splunk_TA_stream/local/inputs.conf and add an "index = foo" attribute.

-jsie