Splunk Rest API not getting all results.

[sidmod25]

Hello,

Situation: I have uploaded little more than 1 million data rows to one of the splunk indexer via csv file. When I am doing the search from search head I am getting exact number of rows. But when I am searching it through a rest api using sdk, it is unable to return entire data and stopping at some arbitrary number (different everytime).

Config Changes: Under search header I change the maxresultrows to 40000 (default 50000) to keeps iterations of 40000 each. Also< i change max_count to 1.2 million (default 500,000) so that I can get all of my data.

When I trigger my search I can see query reached to Splunk visible from audit.log and in query mentions the exact rows. Also, Iat the sam time I can see iterations happening in splunkd_access.log for 40000 rows for each iteration. But suddenly the search stops at without completing total iteration. The stooping number is also different each time. Sometime at 760000, other 800000 and some other time at 920000. But never completed. i am looking into there logs in search head

I am not able to find nat logs where it could have mentioned when its not able to get all data. In the end I should have got a CSV for all rows

I am using Splunk 6.5.9.

Any suggestions. Thanks

[tiagofbmm]

What kind of iteration are you really trying to do?

[sidmod25]

Hello, I am not trying any iteration from y end. Its the Splunk feature where if the count of rows is more than 40000 then it search in offset mode where results are gathered 40000 each time and offset increases by 40000 each time. But this offset value stops or pauses sometime at 760000, other 800000 and some other time at 920000. But never completed.
Need to know why this is not going till the end and generating a csv for me.

[tiagofbmm]

I wouldn't change those parameters. The Java sdk is explicit that you can Target count and offset to iterate it with a regular loop. That version your're using is quite old and if your code is really bugless, there may be a problem on 6.5.9.

I'm assuming you're going through https://dev.splunk.com/enterprise/docs/javascript/sdk-javascript/howtousesdkjavascript/howtosearchsd...