Large complicated search logic

[howyagoin]

I've got an index with some 80 million events in it (thus far) and I want to search it for a list of some 100+ values. There's got to be a better way than a really complicated series of OR statements -- but I can't see what it is.

I'd prefer not to have to write a custom search command, but that strikes me as one option, where my blacklist or whitelist of data is in a separate file and the custom search command will either include/exclude results based on that, but, is there a better way?

I may also want to search for results which DON'T match a particular pattern from the list...

In some cases, the list may be IP addresses in CIDR format, in other cases it may be hostnames or application names.

Thoughts?

[mw]

It depends on what you're doing. Obviously, Splunk isn't able to read your mind and show you what you want, so you'll have to communicate it somehow. If there's a way to group those items by some method, that should do it, but if they're very unique and subject to your whim at the moment of search, you'll probably have to specify everything explicitly. Can you give some examples of the data and how you would imagine making this more terse?

Obviously, you can use wildcards to search, so if what you're searching for is similar it could be as simple as:

myfield=*val*

It may be possible to use a lookup:

myfield,newfield
someval,1
otherval,1
lastval,0

with a search:

... newfield=1 | ...

It's also possible to do something similar with regular expressions:

... | eval newfield=if(match(myfield, "^...$"), 1, 0)

And there's a cidrmatch() function for eval as well.