Splunk Search

Real time event tracing

leonid_komarovs
Explorer

I have a simple setup of forwarder->indexer and I want to display real time events coming from the forwarder. The data coming from the forwarder is very small, every event produces about 20 log lines and there are about 1-3 events per minute. Almost all log lines are recorded at the same second.

I need to show the events as fast as possible, the ideal would be that they appear as they happen and in correct order.

Currently I have two problems:
1. Events are appearing with at least 20 seconds delay after they really happen, they also come in chunks with big delay between them, meaning I can get 4 lines and other 16 after another delay.
2. Log lines are coming in wrong order (I sort by _indextime), if they happen on the same second.

When I tried same search with local files without the forwarder everything worked as expected.

How can I improve forwarder->indexer communication and why there is a problem with sorting?

Update: I was able to pin down the problem with the delay, the forwarder had about +40 seconds offset and events coming from it were displayed in chunks. This happens if you set specific real time window only.

Now the question is how to sort the events that happen on the same second in correct order, I tried _time,_indextime and their combinations.

Tags (2)

_d_
Splunk Employee
Splunk Employee

A forwarder sends data to an indexer in blocks and a block of data is approximately 64KB. In a small volume implementation, like yours, this could be one of the reasons that events are not being displayed in "real-time" or as fast as they're being written to.

0 Karma

leonid_komarovs
Explorer

I did some tests with bigger data amounts and it still comes in chunks of various size. For example I got chunks of 4, 183, 435 events with few seconds delay between them.
Are there config options to handle values of blocks etc. on forwarder? I can see only limits.conf file with only maxKBps parameter which is currently set to 0.

0 Karma

Drainy
Champion

Welcome.
Firstly there will be some unavoidable delay with the forwarder to indexer process.
20 seconds actually sounds like a fantastic time. If you imagine that the device has an event happen that causes it to write to its log, the log is then checked by the forwarder and updated data is found. This data is then pulled from the log and a connection made to the indexer, data is forwarded.

The indexer then receives the data and does its initial parsing to check for field extractions, timestamps etc. This data is then written to disc (into a hot bucket). Given both systems could be busy doing other things, there could be network traffic and packets could fail and require a re-send (TCP, also a reason why events may be the wrong way round at indextime as well as slightly larger events to parse).

Anyway, although others may have some suggestions firstly you can adjust how fast the forwarder forwards data. It has a rate limiter to prevent it flooding the network / indexer, have a look here. You want the part about MAXKBPS.

Also, best practice is generally to snap your search to at least -1m@m to allow for time delay (although on a busier setup/network -5m@m is best)

For your interest;

How indexing works

leonid_komarovs
Explorer

While I generally agree with you, there was very significant delay which I was finally able to solve (I updated the question) Do you have any ideas regarding the second part?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...