Getting Data In

Checkpoint LEA pointer is reset, events are re-indexed

pajohnston
Explorer

I've recently set up LEA-LogGrabber, which is working fine from a communication point of view - the logs are being successfully retrieved from the Checkpoint Manager and fed into Splunk. However, I've noticed that I seem to be getting identical log entries repeated in my Splunk logs. This means that searches tend to return more data than necessary and also it's using up more of my daily Splunk license than it should.

It looks like the reference pointer in the file "lea_log_rec_num.cache" in $SPLUNK_HOME/etc/apps/lea-loggrabber-splunk/bin is somehow getting reset back to 0 regularly and therefore the whole logfile is being reread into Splunk on a regular basis.

For example, see the repeated commands below:

/opt/splunk/etc/apps/lea-loggrabber-splunk/bin$ cat lea_log_rec_num.cache

13271

/opt/splunk/etc/apps/lea-loggrabber-splunk/bin$ cat lea_log_rec_num.cache

13271

/opt/splunk/etc/apps/lea-loggrabber-splunk/bin$ cat lea_log_rec_num.cache

0

/opt/splunk/etc/apps/lea-loggrabber-splunk/bin$ cat lea_log_rec_num.cache

13271

Does anyone know what could be causing this behaviour and how to stop it from happening?

Tags (1)
1 Solution

hexx
Splunk Employee
Splunk Employee

This is a known bug in the current implementation of checkpointing for the OPSEC LEA app. Its internal defect reference is APP-70. The checkpoint which determines the offset of the last event we received from the OPSEC API gets reset to 0 when a connection attempt to the API fails. We are working to resolve this problem in a future release of this app.

View solution in original post

lmyrefelt
Builder

We had this problem as well and in our case it was enough to increase the time between the data pulls against the Checkpoint Manager. I think we had problems with any setting under 15 or 20 minutes.

So my tip is to increase the time elapsed between your data pulls, hope that can help you.

pajohnston
Explorer

OK, I'll try increasing the time between polls and see if that makes a difference. Thanks for the tip!

0 Karma

hexx
Splunk Employee
Splunk Employee

This is a known bug in the current implementation of checkpointing for the OPSEC LEA app. Its internal defect reference is APP-70. The checkpoint which determines the offset of the last event we received from the OPSEC API gets reset to 0 when a connection attempt to the API fails. We are working to resolve this problem in a future release of this app.

hexx
Splunk Employee
Splunk Employee

@pajohnston : There is no known work-around for this issue at this time, other than investigating and resolving the failed connection attempts to the Checkpoint API that cause the event read offset to be reset. This defect is currently being investigated, but unfortunately there is no ETA for the fix that I can share with you at this time.

pajohnston
Explorer

OK, thanks for that. I'm glad that it's known about.

Do you know whether there is any way to work around the problem, and when it might be fixed?

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...