So you actually solved the problem? I'm trying the same thing using SCPv2 and it still waits for all 1M results before showing them in preview. Same configuration settings and CSC. ... View more

Sorry it took me a while to get back to you. I have played around with those options, but to no avail. And as the documentation states, they default to the most permissive settings anyway. As my response to timbits' question mentions, the problem may have something to do with the search_command.py flush() method, which invokes self._record_writer.flush(). Still no clue what the solution is though. ... View more

This is interesting -- that code actually triggers the error immediately, regardless of the size of the event set. Any thoughts on what this means the problem may be? Could it have something to do with the SearchCommand class's flush() method, which invokes self._record_writer.flush()? ... View more

Thanks for the definitive answer; that is what I suspected. I mainly just prefer the chart annotations, data labels, and select/deselect animations of TimeLineView over Timechart. But now that I have a definitive answer that the exposed configuration interface is as limited for TimeLineView as I thought, I may pivot to a Timechart instead. ... View more

I saw that post too, but unfortunately it didn't work for me... even after adding CSS !important rules. Regarding the timeline viz properties, correct me if I'm mistaken, but I believe TimelineViz and TimelineView are actually different things. That being said, I did try a few of those TimelineViz properties, but they had no effect either. ... View more

Nice, I'm running the same thing on one of my large datasets right now, and the results are looking pretty good. Even the premium MaxMind databases were coming up with misses for about 1/3 of the IP's I check, and I'm seeing similar results from the IP2Location database, but with better performance. And it distributes just like it's supposed to! Shall we transform one of the comments in this chain into a formal answer to the original question? I think we've pretty much concluded that yes, custom search commands do have worse performance, due to the implementation differences between the custom command protocols and the dynamic lookup protocols. Primarily: External lookups can assume idempotence Custom Search Commands are limited to one streaming process External lookups are free to spin up many processes in parallel ... View more

Very cool! I had considered testing out ip2location a while ago, but already had a copy of the premium MaxMind databases and wanted to exhaust that option first. Did you use the Lite version of ip2location or the commercial version in your tests above? And which database, by the way? It looks like there is a free Lite IP-ASN database, but the closest thing I can find in the premium section is "IP-Country-ISP Database"... and it's unclear exactly what fields that would contain. ... View more

By the way, are we clear on whether the dynamic/external lookup protocol is creating new processes or new threads? From what you mentioned about Python overhead on Windows, I'm assuming it's creating separate full Python processes for each chunk of data? Is there any way we could safely write a custom search command using the V2 protocol that uses threads instead of processes? Or maybe a specific number of processes that all stay alive and get eventset chunks via IPC in a round-robin fashion? Depending on the answer to that question, now that we know all this, what do you think the bottom line is for a scenario where you need to perform hundreds of millions of ASN lookups in order to build a set of machine learning model features? Specifically, one of the features is distinct ASN count by domain, and there could be over 10 million domains, with over 200 million total events. Assume the environment is capable of distributed search over 5-10 indexers. Would you go with the process-spam lookup method or the slow-as-death custom search command protocol? Any concerns about eating up too much of the host resources if you know the indexers are pretty busy already? How about if the environment is all Windows? Even possible then with the extra Python process creation overhead? So I guess you could say the real question now is: is any algorithm like this going to be truly scalable if there is no way to do all of the following: Be compatible with distributed search Spin up a specific number of processes on each node and keep them all alive for geoip caching purposes Pass chunks of data through each process on each host via IPC in a similar manner to the single-threaded V2 protocol Would you even attempt to deploy an analytic that requires this many ASN lookups for a large enterprise if those conditions listed above can't be satisfied? ... View more

Nice analysis. I agree with your cookie distinct count of 2 for the above query, but my actual search query is still seeing a huge distinct count: index="scale_med" | rex "^dns,(?<dns_server>\S+),(?<log_file>\S+),(?<log_type>\S+),.*:\s+(?<timestamp>\S+\s+\S+)\s+client\s+(?<client_ip>\S+)#(?<client_port>\d+).*query:\s+(?<domain>\S+)\s+IN\s+(?<query_type>\S+)\s+response:\s+(?<response_code>\S+)\s+[+-]\S*,?\s?(?<raw_response>.*)" | where response_code="NOERROR" and query_type="A" | rex field=raw_response max_match=1000 "(?<record_name>\S+)\.\s+(?<ttl>\S+)\sIN\s(\w\s)?(?<record_type>(A))\s(?<record_ip>[^\s;]+)(;\s)?" | where isnotnull(record_ip) | eval _time = strptime(timestamp,"%d-%b-%Y %H:%M:%S.%f") | fields client_ip, domain, response_code, record_name, ttl, record_ip, eventtype | eval zip=mvzip(record_name,mvzip(ttl,record_ip)) | mvexpand zip | rex field=zip "(?<record_name>[^,]*),(?<ttl>[^,]*),(?<record_ip>.*)" | fields - zip | head 500000 | lookup ipasn_dat_lookup ip as record_ip OUTPUT asn as record_asn asn_error as record_asn_error cookie local=true | stats count dc(cookie) max(record_asn) min(record_asn) count dc(cookie) max(record_asn) min(record_asn) 500000 1987 393342 -2 I wonder what about these two search pipelines is causing such different behavior? I believe there may be some JSON involved in the V2 protocol's IPC, and the documentation indicates that it spins up one single process and streams all the events through. Clearly, this differs from the external lookup method, where we're seeing what looks like multiple processes. Would you agree that there should be one distinct cookie per process? If that is true, the external lookup for my query above must be creating tons and tons of processes for some reason(?) All in all, I knew that the MaxMind lookup would be the most taxing part of the custom scripts. I had assumed that the true streaming interface of the V2 protocol would help me take advantage of the MaxMind API's GeoIP caching flag, but would you conclude that the external lookup's manner of spamming the system with new processes every 255 events is what happens to get the job done faster, even without caching? ... View more

Thanks Martin! I appreciate you taking the time to look into this with me. It is good to see that you got similar performance results, but still disappointing that the custom search command is so slow. A couple questions about your comments: You say that there doesn't seem to be excessive re-launching of python every 255 events, but what led me to that conclusion is the code I inserted to add a "cookie" field, which is a random number computed at the top of each python script. That code might actually be commented out in the ipasn_dat_lookup.py script right now, but you should be able to comment it back in (in that particular script, it might be called the "debug" field, rather than "cookie". sorry for the inconsistency there). I ran the lookup scripts on 1,000,000 events, and stats showed that about (1,000,000/255) unique values for the "cookie" (or "debug") fields existed for the lookup versions of the commands. For the custom search commands, there was only one unique value. So, I concluded that the external lookup scripts were getting re-invoked every 255 events, while the custom search commands were only dispatched once and streamed events in chunks. Would you agree with this line of reasoning? If the old external lookup method really is faster, I would like to find a definitive answer why. The best information I could find on the inner workings of traditional Splunk streaming commands was this post from 2012, but it doesn’t explicitly say how much parallelization there is, nor does it say if that applies to external lookup commands. Overall, nothing has explained why my results differ so radically from the performance benchmarks in this powerpoint on the V2 protocol. Now that you have reviewed my code, note that the MaxMind lookups are being performed the exact same way across both the custom command and external lookup versions. So, with your comment about something "inside the custom command [being] terribly inefficient?" -- do you mean that the custom command protocol itself is tremendously inefficient? If so, is that something we should try to report to the Splunk Dev's? Either way, if external lookups are always going to be this much faster than custom commands, it seems like that should be stated somewhere. This question as a whole is actually closely related with another I have out there. Since you already have the code, I was hoping you might be able to take a look at this: https://answers.splunk.com/answers/496840/custom-streaming-command-wont-distribute-to-indexe.html Try as I may, that custom command refuses to distribute to indexers. I feel like an answer to that question may be more achievable, and I would very much like to find out what the problem is. I have implemented versions with the V2 protocol and the old Intersplunk way of doing things, and the streaming command isn't showing up in remoteSearch either way. ... View more

Sorry for the trouble, Martin. It looks like I had been playing with the dat command's decorators during testing and forgot to change them back. Here's a new version, with the underscores ripped out: ipasn (side note: do you happen to know why the underscores were causing so much grief?) I tested your 8.8.8.8 lookup command right before packaging using "eval ... | ipasndat" and it worked fine on this version. If you encounter any other errors, one thing to try from the command line is: /opt/splunk/bin/splunk cmd python ipasn_dat.py < ipasn.csv With the V2 protocol, I don't think that will yield any actual results like the lookup version of the commands would, but it might at least bring any import errors or things like that to light. One last note, when I first installed these commands a couple weeks ago, I initially got that "error code 1" message several times, and then all of a sudden the commands mysteriously started working. I have seen similar behavior when installing the official GeoASN app as well. Let me know how it goes this time! I'm assuming there must be a better way to propagate errors up through to the job inspector than what I have now? I'll look into that, but hopefully the new package will work better already. ... View more

Hey Martin, Here's a packaged version of both apps: ip_asn_lookup.spl ip_asn.spl Note that there are two commands in each app. The lookup version has ip_asn_lookup and ip_asn_dat_lookup, while the custom search command has ip_asn and ip_asn_dat. The "dat" command versions query the legacy MaxMind binary database, while the others query the newer ".mmdb" format. My results have shown that the legacy API is actually faster, so whichever commands you use for testing, make sure to use the ones with equivalent MaxMind API's. Also on that topic, the "lookups" folders originally contained the paid versions of the MaxMind databases: GeoIP2-ISP.mmdb, and GeoIPASNum.dat. Since I am not free to distribute those files, I have removed them from both apps, and replaced them with just the Lite version of GeoIPASNum.dat (I couldn't find a lite version of the ISP database at the moment). So, without adding your own copy of GeoIP2-ISP.mmdb to the lookups folders, only the "_dat" commands will work. The custom search commands assert that a field named "record_ip" exists in each event, upon which the ASN lookup is performed. Meanwhile, the external lookups operate on just an "ip" field, as defined in transforms.conf. Let me know if you have any questions, and I look forward to hearing back. ... View more

Configuring it as a GeneratingCommand in the streams pipeline was an interesting idea, but it looks like it won't work for my use case. The command needs to perform ASN lookups on an existing set of events, but GeneratingCommands "must be the first command of a search." I did rewrite it as a GeneratingCommand just to see what would happen, but Splunk pitched that error in response. Do you have any ideas about how we might get this issue more visibility? I think it's worthy of bringing up to some Splunk dev's -- if there are widespread distributing issues with streaming commands using splunklib, it's definitely something that should be addressed. ... View more

I have tried passing several combinations of arguments to the Configuration decorator, including those that you mention, but so far no luck. For Streaming commands, both 'type' and 'streaming' are actually fixed parameters (as defined in splunklib/searchcommands/streaming_command.py). So attempting to specify them in the Configuration decorator throws an error. I've also tried leaving @Configuration() empty, since splunklib's streaming_command.py indicates that 'streaming' should be fixed to True, and 'distributed' defaults to True in the V2 protocol. 'local' also defaults to False. So in theory, no extra configuration should be needed to get this to distribute to indexers. Any other thoughts? ... View more

Okay thanks for continuing to look! I'm quite puzzled by why it doesn't work yet. But I'm definitely not seeing it in remoteSearch, and the performance is exactly as bad as it would be running on just the search head. ... View more

Thanks for the response! I can't share the exact command syntax, but trust that it is placed before all non-streaming commands. In fact, this custom command is being used in place of an older distributed steaming command (that does distribute correctly within the same larger search query, but is less efficient). The command is indeed defined on all indexers and deployed in the same manner to the search head and all indexers. I also verified that it can be run manually on each of the indexers. ... View more

Never mind, figured it out. Just need to iterate over events and yield each one. ... View more

Thanks, just figured that part out on my own I think. Does anyone have an example of a streaming command that actually does something? I'm not sure what the syntax is for modifying events in the "def stream(self, events):" function. Having pass in there doesn't seem to work.... the web interface is throwing: TypeError at "/opt/splunk/etc/apps/ip_asn/bin/splunklib/searchcommands/internals.py", line 519 : 'NoneType' object is not iterable ... View more

Okay, so just by playing around with it, I seem to have worked around the errors. Using this did the trick: [handlers] keys = app, splunklib, stderr ... View more