couple of follow up questions: what does the ?: at the beginning do. the query seems to work fine without it any way to prevent all of the empty matches that get returned as a result of changing max_match? ex, I get a bunch of empty mailfrom, mailto, etc. fields since they don't all occur on every line: sample field list from search results: mailfrom="" | mailfrom="" | mailfrom="" | mailfrom=""abc@bvd.com" | mailto="" | mailto="" | mailto="xyz@abc.com" | ... etc ... View more

awesome, that did it. thanks. the reason im running these inline is because I am searching off of a summary index. those fields do already get extracted automatically for regular searches, but since the fields get dropped after doing a " | collect index=summary", im having to recreate them. ... View more

here is the command used to populate the summary index: eventtype=cisco_esa [search | fields mid] | transaction fields=mid mvlist=t | collect index=summary there is more context in this answer, where folks were recommending the summary index for this use case, even though it isn't the standard usage: http://splunk-base.splunk.com/answers/35267/very-large-lookup-tables-exceeding-2gb-bundle-size ... View more

is there a way to preserve the fields while still populating the index with the full set of events like the requestor has? im running into the same issue, but need to run ad hoc search commands... not just preset timecharts. for ex. ironport logs all of the "from, to, subject, etc" attributes as separate events on new lines. most searches i run need to return the full results associated with that mail. so i am using a summary index to store the events in transaction form, and then run my searches against that. ... View more

thanks! that did it. conveniently, it turns out you can store the whole transaction in the summary index without even having to try and squish all of the desired fields onto one line. eventtype=cisco_esa [search | fields mid] | transaction fields=mid mvlist=t | collect index=summary creates the index, and running a search on "index=summary" actually returns the whole transaction for all of the search matches. unfortunately it seems like the field extractions are lost in the process. i need to see about getting those rerun on the index somehow. but at least all of the data is there ... View more

im not sure how I would be able to use the index to accomplish the same thing, without going back to requiring a subsearch. in the search command you have listed, where does the index come into play? also, i don't see how it would work, because the search for "Bounced message..." won't return any mailfrom fields: eventtype=cisco_esa "Bounced message..." | stats count(mailfrom) i would need something conceptually like this: (although this is incorrect syntax) eventtype=cisco_esa "Bounced message..." | fields mid | index=my_ironport_summary mid | table mid,mailfrom,mailto,subject ... View more

Sure! I missed the comment box here... My response with the problem specifics got posted below as one of the answers... ... View more

Sure, to answer your question sdwilkerson. There is a class of search queries that I keep running into that conceptually require a subsearch. Since the subsearch has limitations on how many results can be string interpolated and inserted into the main search command, I am using lookup tables instead. Here is one example: I want to create a list of the top 10 offenders sending mails with the most number of "Bounced" errors. (This is a good way to fingerprint someone who is using their company email address to spam external people, as the spammers list is usually of poor quality--many old or illegitimate email addresses). To calculate this, I need to know the mailfrom field which is associated with each occurrence of the "Bounced" error. The initial way I approached this was to first run a subsearch for the "bounced" message, and pipe that into fields mid to return a list of MIDs. The subsequent search looked for all events related to that list of mids, and then ran a transaction over the MID field. That allowed me to know the mailto, mailfrom and subject of the mails which had the "Bounced" errors. Since I ran into the subsearch limits on this, the solution was to create a lookup table. This removes the need for a subsearch to find the context for each "Bounced" mail, since I can just look up the mailfrom by mid in the table. eventtype=cisco_esa "Bounced message..." | lookup my_ironport_table mid OUTPUT mailfrom,mailto,subject | stats count(mailfrom) There are a good number of other queries similar to this that I need to run. They all follow the pattern of needing to search for something that is part of a conceptual transaction, and then needing to find the other events associated with the rest of that transaction in order to complete the query. ... View more