What is the purpose of var/run/splunk/dispatch/.../metadata.csv?

[muebel]

I have been having repeated warnings that the system is unable to read metadata.csv, which looks like it should be located in the above mentioned directory, replacing "..." with what looks to be a datastamp. However, it is absent.

What is metadata.csv's purpose? And, by extension, how worried should I be that it is not actually in the intended directory?

[jrodman · Accepted solution]

metadata.csv appears to be metadata about the search. It contains things like the splunk user who requested the search, the app context it was run in, how long it should be kept alive, and so on. It's also (along with the rest of the dispatch directory) a communication path between splunkd and the running search.

If you're seeing a message from DispatchCommand WARN could not read metadata file: ... Then this suggests (looking at the code path) that splunk probably was unable to create it in the first place, or else that it was wiped before the job finished.

If it's present, review the search.log in the dispatch directory, or provide it to support. You might also review the permissions in the dispatch directory vs the user splunkd is running under, and whether that job might have gotten cancelled at the time (in which case, that's the answer).

The csv files present may depend upon phases in the search job. Additionally the dispatch directory can be short-lived. If this message occurs in any way reliably (ie for a particular search), I would recommend:

1. run the search
2. save the results
3. in the saved results url, you can see the search id
4. go inspect the dispatch directory manually, particularly the search.log file, to see if any erorrs leap out at you.

It's also possible to save a job by creating a zero byte 'save' file in the job's directory.

For what it's worth, splunk diag captures all current dispatch job directories (except for the results!) from 4.0.10 on.