Splunk Search

inputlookup + Join search = parsing job

bdondlinger
Explorer

I have scheduled search jobs that run nightly. The first search adds fields A and B for the day to the lookup. The second search imports the CSV adds field C. When the second job executes it gets stuck "parsing job" for 30 minutes before finally progressing into the execution phase.

My lookup file is only ~5MB, why so slow to parse the job?

|inputlookup mylookup.csv append=true
| join type=left host, _time
[search index=my_idx FieldC
|  bucket _time span=d
|  dedup host, _time]
|  where _time>=relative_time(now(),"-30d") 
|  table host, FieldA, FieldB, FieldC
|  outputlookup mylookup.csv

chintan2181997
Loves-to-Learn

@bdondlinger  ,

                    Were you able to find the root cause behind this?. We are experiencing same problem.

0 Karma

HiroshiSatoh
Champion

Is not it only because the subsearch part is slow?
I think that it will be quicker if you first bring the condition of the where clause behind.

I do not understand the purpose of this search sentence. Is this sample a whole search statement?
For example, this search statement never adds a record of mylookup.csv. Is this correct?

  | join type=left host, _time

Records are not increased here.

  | where _time>=relative_time(now(),"-30d")

Records older than 30 days are excluded.

0 Karma

bdondlinger
Explorer

Is not it only because the subsearch part is slow?

No, if I run the search alone

index=my_idx FieldC
 |  bucket _time span=d
 |  dedup host, _time

It will complete in less than a minute.

When I add the inputlookup and join

|inputlookup mylookup.csv append=true
 | join type=left host, _time
 [search

This is when the job seems to hang in the parsing state much longer than it should.

The delay isn't normally a problem because it's run as a scheduled job. However if I run it manually I risk getting logged out due to inactivity before it completes. While the job is in the parsing state I cannot send it to the background.

I do not understand the purpose of this search sentence... Is this correct?
The search works as expected, there is one mistake in my example, my table and lookup include the _time field which the join references.

Records older than 30 days are excluded.
This is correct, I want my lookup file to be a rolling 30 day window. Each day I add to the lookup and filter out anything older than 30 days. The |where is probably only required in the first job that adds to the lookup file daily but I left it in this search just because.

HiroshiSatoh
Champion

Did you investigate the portion that takes time with "Search job inspector"?
※It is not waiting for JOB to run due to lack of CPU core?

In the sample search sentence, although it is deleted, there is no part to be added. Is this correct?
Is this sample all search sentences? Is there no other search sentence?

0 Karma

bdondlinger
Explorer

Nothing stands out as wrong in the job inspector however when I open search.log i see this logged every ~10s

ERROR DispatchThread - Failed to read runtime settings: File :/opt/splunk/var/run/splunk/dispatch/subsearch_tmp_1528303728.1/runtime.csv does not exist

in my example search I am importing a lookup file that looks like this.

_time       host        FieldA  FieldB  FieldC
2018-06-04  Host1       foo     bar     baz
2018-06-04  Host2       foo     bar     baz
2018-06-05  Host1       foo     bar     baz
2018-06-05  Host2       foo     bar     baz
2018-06-06  Host1       foo     bar 
2018-06-06  Host2       foo     bar 

The subsearch finds the values to for FieldC for the day and those results are then joined to the table in the corresponding host, _time row.

0 Karma

HiroshiSatoh
Champion

Can you give the full text of the search sentence you are using and the contents of "Search job inspector"?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...