Splunk Search

Why does search keep getting killed by 'Signal 9'?

dkoops
Path Finder

For a certain search I keep getting the following error:

Search process did not exit cleanly, exit_code=0, description="killed by signal 9: Killed". Please look in search.log for this peer in the Job Inspector for more info.

Now the search.log doesn't show much useful information except that the job is indeed killed. Searching a bit on the error learned me that 'Signal 9' means that the job is killed 'externally', probably due to memory issues.

Strange thing is that this is only happening for this specific query, by far not the heaviest or most complex one running and that I never had any memory issues before. Also, the query works like a charm when the timespan is several days, but setting the timespan on 1 month or more gives me this error.

Anyone?

Labels (1)
Tags (2)

flynegal
Explorer

This post is a few years old but to aid those who have spent hours trying to find the answers and end up here for help.... In my case I saw the "killed by signal 9" because of the proxy configuration...  I had 2 different cases of 'signal9'. 

The solution for one case was the application itself allowed the proxy to be set in the app settings and in the other case this was the solution Can you configure the Duo Splunk Connector to use ... - Splunk Community.  

iamsahil
Explorer

Did you get the answer, Please let me know, I have a same concern.😐

0 Karma

dkoops
Path Finder

Thank you for the reply. The search doesn't contain memory intensive stuff like transactions. It is only used to calculate some variables imported in another search that does a simple regression on all volumes:

source='source1' OR source='source2' earliest=@d-31d latest=@d
| fields _time 'volume-id' name 'storage_used_percent'
| eval name=if(source=="source1", 'volume-id', name)
| stats count as count, sum(storage_used_percent) as Y sum(_time) as X, sum(eval(_time*storage_used_percent)) as XY, sum(eval(_time
_time)) as X2 latest(_time) as T1 earliest(_time) as T0 by name*

While the environment is quite big (>6000 volumes), even larger searches over a longer timespan work just fine. Also, after testing a bit, if I set the timerange to up to 26 days -> now, it works. A period longer than 26 days -> now, gives the signal 9 error..

0 Karma

lguinn2
Legend

What is the search? If it includes a transaction command, then you may very well be running out of memory.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...