I'm trying to monitor a set of hosts that run a batch process, and I want to produce output that dynamically identifies the hosts according to filter key words, then join that list of hosts with subsearches that look for the specific events of interest. For example: index=Idx filter1 filter2 | stats min(_time) as initial_time by host | convert ctime(first_time) | table host, first_time | join type=left host [ | search index=eng_etm [ index=Idx filter1 filter2 | stats count by host | table host | return 2000 host ] starttag1 starttag2 | convert ctime(_time) as start_time | table host, start_time | sort 0 host | dedup host ] | join type=left host [| search index=Idx [ index=Idx filter1 filter2 | stats count by host | table host | return 2000 host ] endtag1 endtag2 | table _time, host | stats min(_time) as end_time by host | convert ctime(end_time) ] Returns something like this: host initial_time start_time end_time 1 8/1/16 00:00:00 8/2/16 00:01:00 8/2/16 00:01:21 2 8/1/16 00:00:00 8/2/16 00:01:00 3 8/1/16 00:00:00 4 8/1/16 00:00:00 8/2/16 00:01:00 8/2/16 00:01:21 This is actually the output I want, but I don't like having to constantly restate the subsearch in each join statement. Is there a simpler way to write this search to allow the subsearch to get the host list from the outer search? ... View more

I have an index "eng_1" that has a max size of 500,000 MB. When I look in SplunkOnSplunk it reports this index to be 25% full, however, the oldest data I can query is about 100 days old. Given this index gets about 5GB/day, that amount of history seems right, but the 25% full seems to imply I should be able to go back much further. | rest /services/data/indexes | search title="eng_1" | table currentDBSizeMB, splunk_server, maxTotalDataSizeMB, maxWarmDBCount, homePath.maxDataSizeMB, coldPath.maxDataSizeMB yields: ` currentDBSizeMB splunk_server maxTotalDataSizeMB maxWarmDBCount homePath.maxDataSizeMB coldPath.maxDataSizeMB 1 index1splunk.au1.domain.net 500000 300 300000 200000 1 index1splunk.br1.domain.net 500000 300 300000 200000 7126 index1splunk.eu1.domain.net 500000 300 300000 200000 134915 index1splunk.us1.domain.net 500000 300 300000 200000 ` So on one hand it seems like the amount of history I see is approximately right given the expected max size and the amount of data I ingest every day, but the metadata and SplunkOnSplunk doesn't show that this index is full. How can I confirm whether the daily incoming data is causing the old data to get evicted? ... View more

I've been searching and experimenting for quite a while and I suspect I'm missing something simple.... I have a CSV lookup file with an epoch time field ("timestamp"). In my search string, I use inputlookup and I want to filter it, like this: ...| inputlookup my.csv | where timestamp >= start AND timestamp <= end but the values I want to use for "start" and "end" I want to be the values that come from the UI in the search App (Last 24 hours, etc.) I would expect these to translate to some sort of pre-defined variables, but I haven't been able to track them down. I've also tried: | inputlookup my.csv | eval _time=timestamp where I hoped the predefined range would apply, but it didn't filter the lookup at all. Is there a way I can get the UI generated search range to use in my where clause? Thanks! ... View more

Is there a performance difference between searching for this: index=foo host=XXXX "a text string bar" vs. defining an eventtype bar_event: index=foo "a text string bar" and then searching for this: index=foo host=XXXX eventtype=bar_event ? I'm wondering if the eventtype definition causes the generation of any additional indexing data to speed up the search. ... View more

I have a scheduled search to extract a tiny subset of my data set and attempt to perform a field extraction on the name of the source file: index=<myindex> host=xyz* <string to find> | eval report="my_summary_detail" | eval logtime=_time | table _time, report, host, source, logtime, _raw | rex field=source "XYZ*-(?<upload_date>\d*-\d\d-\d\d \d\d-\d\d-\d\d)Z.txt" | convert timeformat="%Y-%m-%d %H-%M-%S" mktime(upload_date) as uploadtime When I run this in Search, it returns exactly what I want. When I schedule the search and tell it to use summary indexing, the table of values in the query (report, host, source, logtime, uploadtime) are all lost and all I see in the summary index is the time and _raw. I want to extract this data and then do further processing on it such as creating multiple other summaries against this summary with some stats/timecharts. My goal is to perform the rex only once as well as extracting these records from the other 99% that don't match my query. FYI - When I scheduled the search, I used a new index just for this report. (not index=summary). Am I doing something wrong? Is this use of a summary index impossible? Do I have to do something special when I create my separate index? Thanks, Brad ... View more