Splunk Search

Streaming and non streaming commands

splunkn
Communicator

Can someone explain exact difference between streaming and non-streaming commands in laymen terms?

Thanks

1 Solution

ddrillic
Ultra Champion

The following explains it nicely - How to create custom search commands using Splunk SDK for Python

It says -

-- There are two subtypes of custom search commands:
• A streaming custom search command is one to which data is streamed. You can think of it as applying a "function"/"transformation" to each event and then writing out the result of that operation. It is a kind of mapper. An example of such a command might be a command that adds a field to each event.
• A non-streaming custom search command expects to have all the data before it operates on it. As such, it is usually "reducing" the data into the output by applying some sort of summary transformation on it. An example of a non-streaming command is the stats command, which will collect all the data before it can calculate the statistics.

I see a correlation to the hadoop world - the streaming custom search command is something like the map phase of the MapR job while a non-streaming custom search command is the reduce part.

The following streaming command says

-- A command that runs on the indexer and can be applied to subsets of index data in a parallel manner. A streaming command applies a transformation to each event returned by a search. For example, the rex command is streaming because it extracts and adds fields to events at search time.

-- In contrast, non-streaming commands are centralized and run at the search head. These commands analyze the entire set of data available at the search head, and then derive the search result output from that set.

View solution in original post

ddrillic
Ultra Champion

The following explains it nicely - How to create custom search commands using Splunk SDK for Python

It says -

-- There are two subtypes of custom search commands:
• A streaming custom search command is one to which data is streamed. You can think of it as applying a "function"/"transformation" to each event and then writing out the result of that operation. It is a kind of mapper. An example of such a command might be a command that adds a field to each event.
• A non-streaming custom search command expects to have all the data before it operates on it. As such, it is usually "reducing" the data into the output by applying some sort of summary transformation on it. An example of a non-streaming command is the stats command, which will collect all the data before it can calculate the statistics.

I see a correlation to the hadoop world - the streaming custom search command is something like the map phase of the MapR job while a non-streaming custom search command is the reduce part.

The following streaming command says

-- A command that runs on the indexer and can be applied to subsets of index data in a parallel manner. A streaming command applies a transformation to each event returned by a search. For example, the rex command is streaming because it extracts and adds fields to events at search time.

-- In contrast, non-streaming commands are centralized and run at the search head. These commands analyze the entire set of data available at the search head, and then derive the search result output from that set.

woodcock
Esteemed Legend

It really doesn't make any difference, except for a few commands that you might like to use that require it (e.g. multireport). I really never think about this distinction.

0 Karma

jplumsdaine22
Influencer

The documentation here is pretty clear: http://docs.splunk.com/Documentation/Splunk/6.4.1/Search/Typesofcommands

Can you be more specific on which parts you don't understand?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...