Why am I not seeing results from last 60 minutes, but I do from last 15 minutes?

[gtaylor123]

When I run the simple search
host=hostname source="/var/log/audit/audit.log" | transaction fields=msg | search keyword
and select "Last 60 minutes" or larger, I get No results found
but if I run the exact same search and select "Last 15 minutes" I get results.

I am running on a 6 month 50G trial license - and have only ingested under 2G on any given day (only a few day old test system)

Running on a RHEL 6 VM - with 4 CPUs and 5GB memory - plenty of hard drive

[DalJeanis]

This is almost always going to mean that your search is running out of time or space. I'd bet on space.

Transaction is a resource hog. Better to rewrite the search.

Supposing that you need to see three fields, FieldA FieldB FieldC, start with something like this to cut off everything else you don't need...

 | fields FieldA FieldB FieldC msg

Then, since transaction needs to identify the multiple pieces that go together based on their fields, sort the events so that each msg is together.

This should get you a little further (maybe up to an hour or two of data) ...

  host=hostname source="/var/log/audit/audit.log" 
 | fields FieldA FieldB FieldC msg
 | sort 0 msg _time 
 | transaction fields=msg 
 | search keyword 

---

Now, the above can also be rewritten to eliminate the need for "transaction". If "keyword" is fairly rare, you do this...

   host=hostname source="/var/log/audit/audit.log"  "keyword" 
   | table msg 
   | format

Take a look at the output of that. It creates a field called search that looks like...

( ( msg="first message" ) OR ( msg="second message" ) OR ... )

Now ,if you put that in square brackets, whatever is in the field search at the end of the subsearch will be dropped into the outside search, and limit the outside search to only those values of msg.

  host=hostname source="/var/log/audit/audit.log" 
      [host=hostname source="/var/log/audit/audit.log"  "keyword" 
       | table msg | format]
 | fields FieldA FieldB FieldC msg
 | stats min(_time) as _time, range(_time) as duration,
      values(FieldA) as FieldA, values(FieldB) as FieldB,
      values(FieldC) as FieldC
      by msg

For sparse records of "keyword", this is the way to go. It would be pretty inefficient if keyword was dense, however.

---

Which brings up the last point.

In Splunk, efficiency of a search is highly data dependent. If there are three ways to write something, it can be sane... not necessarily smart, but sane... to code all three and see which one has the best effectiveness.