I have 2 logs: an error log and a success log. When an item fails (error log), it is retried. I would like to filter out the items that succeed on retry and only display items that are still failing - In other words, items that only appear in the error log.
Sample search:
index=fail_log "Processing failed" | dedup event_id | transaction event_id | rex "\"itemId\":(?P<itemIdInt>\d+)" | fields itemIdInt | search NOT [search index=success_log Order processed successfully | eval itemIdInt=substr(itemId, 4) | fields itemIdInt ] | stats count
The reason for the substring is that in the success log, the items show up as ID- and in the fail log, they are just , so we need to just pull out the actual digits.
This seems to work for most things. We have 20,000+ successes and on the order of 10-20 failures. However, only 1 or 2 of those failures have not been successful, and a simple search of:
index=success_log Order processed successfully | eval itemIdInt=substr(itemId, 4) | fields itemIdInt | search itemIdInt=<itemId_from_fail_log>
will return results, indicating that itemId has succeeded on rerun.
Is there a more formal way of removing items from result A (failures) that appear in result B (successes)
Not sure you need the transaction command. Try this
index=fail_log "Processing failed" | dedup event_id | rex "\"itemId\":(?P<itemIdInt>\d+)" | fields itemIdInt | search NOT [search index=success_log Order processed successfully | eval itemIdInt=substr(itemId, 4) | fields itemIdInt ] | stats count
That didn't seem to affect the results. Just curious, why would removing the transaction command potentially work?
Would it be just as useful to only include items whose final state is "failed?" E.g. the last time they ran they were in state failed?
How would I go about that, Rich? It sounds like it could work!