Splunk Add-on for Salesforce Ingestion Problem

[lawrence_magpoc]

I am able to collect data from Salesforce using Splunk Add-on for Salesforce. However:
Let's say the data I am collecting has the fields "CreatedDate", "CompletedDate", and "Status" which can either be "Processing" or "Completed", if an item of status "Processing" had been ingested to Splunk, when the status of that item is updated to "Completed" in Salesforce, for some reason, Splunk doesn't ingest it. I need the events for both processing and completed as I have to compute the time it took for a job item to be completed. Is there a way solve this problem?

[pensword]

The data is not being pulled into Splunk when Status changed, right?

Take a look at the ORDER BY clause (SOQL). I wonder if ORDER BY Status might work for you, at least in terms of catching the change. ORDER BY CompletionDate might also work. You can find the query in the api conf files and can also see it in add-on logs. You should also be able to see the SOQL query in the add-on logs.,

[jadengoho · Accepted solution]

This may not be the best way to solve this:
I encounter this issue on ingesting ApexJobs , apparently this logs changed every second and Splunk along with Salesforce Add-on doesn't re-ingest data after it updates.
For example:

1/11/2020 7:04:03.000 AM   - ID:00000001 Status="Processing"
1/11/2020 7:04:05.000 AM   - ID:00000001 Status="Completed"

Splunk will ingest the "Processing" but not the "Complete". In my observation it will only re-ingest the data if the update is greater than one minute.

Solution:

1. Lower the interval time to 20-60 seconds in Salesforce-addon.
2. Create a script that will ingest the APEXJOBS. If you can find a way to ingest all data along with its ID and Fields using wildcard(*).
3. Since "Processing" status are ingested, i need to find a way to run the script and get the latest data. But according to this document https://developer.salesforce.com/docs/api-explorer/sobject/AsyncApexJob I need to specify the ID to get a specific information.

In a search i get all Processing Status and run a REST API query. Curl application helps me to accomplish that, and dump the data in the same index but different sourcetype. And i save this query as a savesearch to recapture information that are not being ingested.

Again, this might not be the best solution out there, but this workaround help me.