Reporting on 2 time series on the one chart

[grahammkelly]

Hi,

I've also asked this on SO (http : // bit.ly / 1h9XMd8) [For some strange reason, I'm not allowed to post links to sites. It's on SO, search for 'Splunk - chart 2 time periods on 1 report'].

OK, so I need to compare 1 hour of data on 2 separate weeks against each other. I've seen 2 solutions recomended. I've explained in the SO question toe specifics, but I burnt down to searching for the specific hours data this week, and appending the same search with different earliest and latest parameters, then manipulating the _time to show the data on the same time chart.

The specific query is pasted at the end of this post.

Anyway, the search works correctly if run through the 'Search' app. But the second (appended) search does not seem to run at all if I schedule the search. In this case, the second column is just '0'.

Thanks

Anyway, the search is as follows:

sourcetype="ws-logs" source="/var/local/catalina/logs/localhost_access_log.*" 
     "/importantCall" AND httpStatusCode>=200 AND httpStatusCode<300 
     earliest=-60m@h latest=-0m@m  
| eval marker="today" 
| append [search 
    sourcetype="ws-logs" source="/var/local/catalina/logs/localhost_access_log.*" 
        "/track/sale" AND httpStatusCode>=200 AND httpStatusCode<300 
        earliest=-10140m@h latest=-10080m@m 
    | eval marker="weekAgo" 
    | eval w1_time=_time+(7*24*60*60)] 
| eval _time=if(isnotnull(w1_time), w1_time, _time) 
| chart 
        count(eval(marker=="today")) as lastHour
        ,  count(eval(marker=="weekAgo")) as sameTimeLastWeek
    by _time span=10m 
| rename _time AS Time 
| eval Time=strftime(Time, "%H:%M")

The results I get are always '0' for the second (appended) search.

Any thoughts?

[grahammkelly · Accepted solution]

No, I'm happy with the date manipulation part of the search. The problem is that the 'append [...]' does not seem to be working.

I've had a look at the timewarp docs and the searches shown only compare contiguous time periods. There's one question in the doc FAQ asking to compare Wednesday to Wednesday. The answer given is to search the entire data set and filter out the data to ignore (Thursday->Tuesday) after the timewarp occurs.

That's immediately a non-runner for me as I'm comparing 1 hour's data between different weeks, so ignoring 98.8% of the data being collected. Also, as the data I'm analysing is the throughput of a fairly highly used Web Service, that's a lot of data to search in the first place.

Anyway, I found the answer. The search was NOT failing to return the data, but failing to complete the search. Basically the search through the older data was failing due to load. n the end, I scheduled 2 searches. 1 stores summary information (aggregated to 5 minute batches), the other searches through the summaries and compares the weeks summaries against each other.

[edschembor]

Have you tried looking at the Splunk app "Timewrap"? It seems like it could quickly resolve your issue. See http://apps.splunk.com/app/1645/