Getting Data In

|delete hang indefinetely

rbal_splunk
Splunk Employee
Splunk Employee

Issue :
I am trying to "delete" bunch of events from the index by using "| delete" command.
The search without "| delete" returns results with no problem. But when I add "| delete" to the end of the command line it just hangs.

Search query:
$ /opt/splunk/bin/splunk search 'index=security source=billing.log earliest=-39d latest=-38d'

Search query with "| delete"
/opt/splunk/bin/splunk search 'index=security source=billing.log earliest=-39d latest=-38d | delete'

The same query in the "Runnig jobs" list in UI looks like the following (http://goo.gl/8xITae):
index=security source=billing.log earliest=-39d latest=-38d | delete | head 100 | export add_timestamp=f add_offset=t segmentation=raw

Tags (1)
0 Karma
1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

It was found that bunch of .db_*.rbsentinel files have been created in the db/ folder. One of those files had a .lock extension, figured that the name of the *.lock file is related to the bucket that contains events which supposed to be deleted by the search query.

Had to stopped Splunk and removed all the .rbsentinel files and started splunk again. After that the same search query piped to the "delete" was able to delete events as expected.

View solution in original post

sat94541
Communicator

This behavior is caused due to bugs

SPL-84276-If a search process is waiting to acquire a lock then it should print lock file information in search.log
SPL-81481-If a search process is waiting to acquire a lock then it should print lock file information in search.log

Bug is expected to be fixed din 6.0.6

Steps to reproduce

  1. Search index=_internal sourcetype=splunk_web_access | eval bkt=_bkt | stats count by bkt, sourcetype and select a bucket drilldown on a slightly older bucket. For me, I had buckets 11 to 38 and I selected 35. Note the number of event that bucket

  2. The drilldown search will look something like index=_internal sourcetype=splunk_web_access | eval bkt=_bkt | search bkt="_internal~35~687F80CD-2706-4248-8773-C09142624BB2". From the search results, select a unique requestid.

  3. Run the search index=_internal sourcetype=splunk_web_access _bkt="35" 536a6a176e7fc09c5209d0 where 536a6a176e7fc09c5209d0 is my unique requestid and 35 is my selected bucket number. This should return only one result.

  4. Now cd into the bucket var/lib/splunk/internaldb/db/db*35 and create a lock file with the same name as tsidx file. In my case the tsdix file name is 1399792823-1399758630-17977910403059112245.tsidx. The lock file created was 1399792823-1399758630-17977910403059112245.tsidx.lock

  5. Now run index=_internal sourcetype=splunk_web_access _bkt="35" 536a6a176e7fc09c5209d0 | delete. This search should keep running indefinitely.

  6. To replicate what the user behavior and get the matching pstacks, run the above search in another tab without closing this tab. Basically you have 2 searches running indefinetely

hogan24
Path Finder

I'm also currently on v6.1.1 if that helps. Thanks.

0 Karma

hogan24
Path Finder

Are there any updates to this? I followed the steps above and deleted the mentioned files under "C:/Program Files/Splunk/var/lib/splunk/defaultdb/db" and restarted splunk. Then reran the |delete command and my delete is still hanging. Any additional help would be appreciated. Thanks.

0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

It was found that bunch of .db_*.rbsentinel files have been created in the db/ folder. One of those files had a .lock extension, figured that the name of the *.lock file is related to the bucket that contains events which supposed to be deleted by the search query.

Had to stopped Splunk and removed all the .rbsentinel files and started splunk again. After that the same search query piped to the "delete" was able to delete events as expected.

asetyyli
Explorer

Removing the .rbsentinel files solved the issue for me. Thanks.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...