What's a scalable to extract key-value pairs where the value matches via exact or substring match but the field is not known ahead of time, and could be in _raw only? Eg, search for the string "alan", which may be associated to fields as follows: index=indexA  user=alan index=indexB username=alan index=indexC loginId=corp\alan index=indexD   _raw=<unstructured text, alan : user>   ... View more

We need to run the same query over a list of values (10k to 100k) without knowing the exact key across various indexes where they might show up.  What's the best way to do this in a scalable way. For example, same search over a list of users (user1 OR user2 OR user3) | stats count by index The desired output should be a table with both user and index as columns, not just the index. But again, the field is not known ahead of time as it varies by index (or could simply be in _raw) Should one use map or how to assign each of user_n to a variable?           ... View more

I get a different result set when using jobs.export of python SDK with a simple stats query compared to the same query (and time range) on the splunk UI.  the jobs.export returns a list of results of the following form, so there is a repeating pattern with several sets of "lastrow":true  and repeating "offset" almost as if partial results are included several times.  Only the last set matches the final results of the query on the UI.  Schematically the results of this call: jobs.export("search my_id | stats count by index").read().decode('utf8').split('\n') have this form ['{"preview":true, "offset":0, "result": {"index": "index_a", "count":"2"}}', '{"preview":true, "offset":1, "result": {"index": "index_b", "count":"4"}}', '{"preview":true, "offset":2, "lastrow":true, "result": {"index": "index_b", "count":"4"}}', '{"preview":true, "offset":0, "result": {"index": "index_a", "count":"6"}}', '{"preview":true, "offset":1, "result": {"index": "index_b", "count":"12"}}', '{"preview":true, "offset":2, "lastrow":true, "result": {"index": "index_b", "count":"50"}}', '{"preview":true, "offset":0, "result": {"index": "index_a", "count":"18"}}', '{"preview":true, "offset":1, "result": {"index": "index_b", "count":"102"}}', '{"preview":true, "offset":2, "lastrow":true, "result": {"index": "index_b", "count":"499"}}', '{"preview":true, "offset":0, "result": {"index": "index_a", "count":"18"}}', '{"preview":true, "offset":1, "result": {"index": "index_b", "count":"102"}}', '{"preview":true, "offset":2, "lastrow":true, "result": {"index": "index_b", "count":"499"}}', ] the last couple of segments ending with "lastrow":true share the same count that matches that in the UI.   Is there some flag we need to insert in kwargs? currently using only earliest_time, latest_time, count:0 and sample_ratio:1 ... View more

This question: How to use IN function with VALUE-LIST as a search or lookup  discusses using IN for a single key and list of values. Can that approach be generalized for lists of  KV lists?  Want to abstract what could be done in a verbose way with  AND and OR's : (keyA=value1 AND keyB=value2) OR (keyA=value3 AND keyB=value4) OR (keyA=value5 AND keyB=value6)...           ... View more

In trying to speed up queries by using buffered export API in the python SDK as discussed here, running into a problem where the service jobs.oneshot query works, while the jobs.export version fails with ParseError . The query parameters are otherwise the same. Moreover the location of the parse error varies each time the query is run (date/time parameters are fixed so it is presumably getting the same data each time). By looping through the generator, ie using results.ResultsReader().next() , and trapping Exceptions and returning python type() , I see that the query seems to work for an initial segment of data in other words, the returned OrderedDict objects are ok - but that there is a XML parse errors that occurs at different records each time I run the query (recall the data extracted should be the same each time) 0 <class 'splunklib.results.Message'> 1 <class 'splunklib.results.Message'> 2 <class 'splunklib.results.Message'> 3 <class 'collections.OrderedDict'> 4 <class 'collections.OrderedDict'> 5 <class 'collections.OrderedDict'> ... 4044 <class 'xml.etree.ElementTree.ParseError'> (rerunning this, the record # will vary) Here's a sample of the Traceback: Traceback (most recent call last): File "/Users/zk8n1ue/miniconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3267, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "<ipython-input-153-517a13bba7fa>", line 1, in <module> tmp3 = [parsefunc(x) for x in tq3['results']] File "<ipython-input-153-517a13bba7fa>", line 1, in <listcomp> tmp3 = [parsefunc(x) for x in tq3['results']] File "/Users/zk8n1ue/miniconda3/lib/python3.7/site-packages/splunklib/results.py", line 210, in next return next(self._gen) File "/Users/zk8n1ue/miniconda3/lib/python3.7/site-packages/splunklib/results.py", line 219, in _parse_results for event, elem in et.iterparse(stream, events=('start', 'end')): File "/Users/zk8n1ue/miniconda3/lib/python3.7/xml/etree/ElementTree.py", line 1222, in iterator yield from pullparser.read_events() File "/Users/zk8n1ue/miniconda3/lib/python3.7/xml/etree/ElementTree.py", line 1297, in read_events raise event File "/Users/zk8n1ue/miniconda3/lib/python3.7/xml/etree/ElementTree.py", line 1269, in feed self._parser.feed(data) File "<string>", line unknown ParseError: not well-formed (invalid token): line 454841, column 54713 Rerunning, yields the same trace except for the location of invalid token. eg ParseError: not well-formed (invalid token): line 169194, column 13753 ParseError: not well-formed (invalid token): line 204476, column 30137 Any ideas what could be causing this and is there a workaround? By manually ... View more

Is it possible, via Splunk's Python SDK, to specify event sampling ratio (say 1:1000) or some equivalent random evaluation in a subsearch which returns a long OR expression while specifying that the outer search does not sample? For concreteness, the subsearch is: [search index=my_index | rex "(?i)deviceId=(?P<DevId>[^ ]+)" | dedup DevId | return 1000000 $DevId] This returns a long OR lists, each of which can match one or more events. It is critical to extract all events associated with the randomly sampled device. ... View more