We recently had an issue where Splunk services were up and running, but new data wasn't being indexed. I'd like to capture data on the LATEST EVENT or INDEXED count with HP SiteScope and report it to a dashboard.
My first thought was to match against the LATEST EVENT timestamp from the default user landing page, but SiteScope can't parse the JS. No problem, I isolated the JSON and was able to send the request and retrieve a good response...for a while. But because the request is dynamic (current timestamp), I need to create a dynamic path and query string. Okay, I can do that in SiteScope, but if I manually update the values I get the following:
{"messages":[{"type":"FATAL","text":"Unknown sid."}]}
Is there another way to poll the LATEST EVENT timestamp over http? Alternatively, I could pull the INDEXED Counter and make sure it is greater than the previous run...
Here's the request which works in real time, but becomes stale soon after:
http://SPLUNK/en-US/splunkd/__raw/servicesNS/USER/search/search/jobs/rt_1453156700/results_preview?output_mode=json&count=1&search=%7C+stats+sum(totalCount)+as+cnt%2C+min(firstTime)+as+min%2C+max(lastTime)+as+max&_=r1453156700%27%20-H%20%27Cookie:%20session_id_8000=76573b1cc0db7735330e0f79b98861204c90afa8;%20splunkd_8000=CY1ElvorX1ZiDZacY_qLzN6xLQweT26T7^SQ6cfXUqqAcuiXdcts2vmdDNzCmqAF1JOUg7ZWzgBpdAIFP73l8TzoaHBfhbwDluD538T4YFqW29emwuIdVpZICExcwpjE;%20splunkweb_csrf_token_8000=10741111671149595932
... View more