Old post but I just noticed the addon builder doesn't support config or account changes in a search head cluster environment too. Did you ever find a solution? I'm sure I've hand crafted addons that were capable of having config changes replicated across the cluster. ... View more

Hi, Did you ever get a solution to this? I'm seeing the same on Windows as well. With no streams configured the streamfwd.exe process uses 5MB. The second I turn on just one stream all agents start hogging 700MB of memory. This is kind of a deal breaker for a host with only 4GB of RAM. Infrastructure people really didn't like that and are reluctant to install site wide. It's not isolated to busy hosts either. Quiet dev/test hosts with minimal traffic exhibited the same behaviours. I've been playing around with enabling/disabling various streams to see if I can pinpoint the main culprit but no luck yet.   Thanks   ... View more

Replying to my own post of the other requirement if you have large knowledge bundles. You need this on the indexers server.conf as well if you continue to have bundles bigger than 2GB.   [httpServer] # Required on indexers due to increasing knowledge bundle sizes to 5GB. Large Knowledge Bundles caused by machine learning models. Default is 2147483648 (2GB) max_content_length = 5368709120   ... View more

Answering my own question here. I did some testing and the _draft_ ones are updated when you are playing around in the experiments section with each model types interfaces. They update the non _draft_ ones when you save. Which is pretty obvious. As for whether the models are required in the knowledge bundle it all depends on the search you are running to build your data set. If the job inspector phase0 search includes the fit command then you will likely run into issues if you don't distribute these in the bundle. I added this to /opt/splunk/etc/apps/Splunk_ML_Toolkit/local/distsearch.conf to see if I could "break" it. [replicationBlacklist] # non_draft_models = apps/Splunk_ML_Toolkit/lookups/(?!__mlspl__exp_draft)*.mlmodel user_models = users/*/Splunk_ML_Toolkit/lookups/*.mlmodel # all_models = apps/Splunk_ML_Toolkit/lookups/*.mlmodel   Essentially stops all users data models being included in the knowledge bundle. I tried many different combinations of searches using fit and couldn't get a search that tried to run the fit command on the indexers. Like this pointless model index=_internal (status=2* OR status=3* OR status=4*) | fit DensityFunction "status" by "method" dist=auto threshold=0.0001 show_density=true show_options="feature_variables,split_by,params" into "_exp_draft_212803036b7b408a9d467eedf8c6a3a6"   The fit always ran on the search head so I'd say 9 times out of 10 you don't need these in the bundle. ... View more

Thanks for posting this. Solved the problem I was having perfectly. Only difference was I was trying to convert 30 min data to 5 min data which I did with a modified version of your macro Macro Def - linear_interpolate_minutes($field$,$minutes$) sort - _time | streamstats window=2 first($field$) as next_value, first(_time) as next_time | eval delta = ((next_value - $field$) / (next_time - _time)) * ($minutes$ * 60) | makecontinuous span=$minutes$m | filldown delta | streamstats sum(eval(coalesce($field$, delta))) as $field$_interpolated reset_before="ISNOTNULL($field$)" | fields - delta next_value next_time ... View more

IT WORKS!!! Thanks for the clarification. I do recall I saw this a few years ago but it obviously never stuck in my mind. I certainly won't forget it now. ... View more

I just encountered this issue with Fedora 33 My fix was to edit /etc/systemd/system/Splunkd.service then change the ExecStartPost commands to      # old cgroups v1 method installed by Splunk. Comment these out # ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n" # ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n" # new cgroups v2 method ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/system.slice/%n"   Don't forget to run "systemctl daemon-reload" after changing the file. Edit: Not sure if there are other implications with cgroups v2 as I haven't tested workload management but at least Splunk starts with that fix.     ... View more

Good to see modbus has been added to Splunk Streams 8.0  https://docs.splunk.com/Documentation/StreamApp/8.0.0/ReleaseNotes/NewFeatures The customers I've worked with wanted to probe endpoints so I've ended up writing my own modular input. We've also moved to Claroty for security monitoring of modbus data travelling through the network. Claroty can feed Splunk with richer alerts and analysis of "normal" traffic. Had this been in Streams a few years ago prior to using Claroty I would have had a field day.   ... View more

Some debug code I added with results for both Python 2 and 3 run   ########################################################### # Python Code changes - Added debugs where it says TESTING ########################################################### def upload(self, callback=do_nothing, pre_allocate=True): """Test upload speed against speedtest.net""" sizes = [] for size in self.config['sizes']['upload']: for _ in range(0, self.config['counts']['upload']): sizes.append(size) # request_count = len(sizes) request_count = self.config['upload_max'] requests = [] for i, size in enumerate(sizes): # We set ``0`` for ``start`` and handle setting the actual # ``start`` in ``HTTPUploader`` to get better measurements data = HTTPUploaderData( size, 0, self.config['length']['upload'], shutdown_event=self._shutdown_event ) if pre_allocate: data.pre_allocate() requests.append( ( build_request(self.best['url'], data, secure=self._secure), size ) ) def producer(q, requests, request_count): for i, request in enumerate(requests[:request_count]): thread = HTTPUploader( i, request[0], start, request[1], self.config['length']['upload'], opener=self._opener, shutdown_event=self._shutdown_event ) thread.start() q.put(thread, True) callback(i, request_count, start=True) finished = [] def consumer(q, request_count): while len(finished) < request_count: thread = q.get(True) while thread.isAlive(): thread.join(timeout=0.1) finished.append(thread.result) callback(thread.i, request_count, end=True) q = Queue(self.config['threads']['upload']) prod_thread = threading.Thread(target=producer, args=(q, requests, request_count)) cons_thread = threading.Thread(target=consumer, args=(q, request_count)) start = timeit.default_timer() prod_thread.start() # TESTING printer('Upload Start:%d\n' % start, debug=True) cons_thread.start() while prod_thread.isAlive(): prod_thread.join(timeout=0.1) while cons_thread.isAlive(): cons_thread.join(timeout=0.1) stop = timeit.default_timer() # TESTING printer('Upload Stop:%d\n' % stop, debug=True) self.results.bytes_sent = sum(finished) # TESTING printer('Bytes sent:%s\n' % finished, debug=True) printer('Bytes sent:%d\n' % self.results.bytes_sent, debug=True) self.results.upload = ( (self.results.bytes_sent / (stop - start)) * 8.0 ) # TESTING printer('Upload Value:%d\n' % self.results.upload, debug=True) return self.results.upload ########################################################### # Python 2 Run ########################################################### DEBUG: Upload Start:1613343269 .................................................................DEBUG: Upload Timeout: result=114688 DEBUG: Upload Timeout: result=458752 DEBUG: Upload Timeout: result=974848 DEBUG: Upload Timeout: result=483328 ............................... DEBUG: Upload Stop:1613343279 DEBUG: Bytes sent:[524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 974848, 483328, 458752, 114688, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] DEBUG: Bytes sent:26148864 DEBUG: Upload Value:20746227 Upload: 20.75 Mbit/s DEBUG: Results: {'client': {'rating': '0', 'loggedin': '0', 'isprating': '3.7', 'ispdlavg': '0', 'ip': '193.116.x.x', 'isp': 'TPG Internet', 'lon': '153.0215', 'ispulavg': '0', 'country': 'AU', 'lat': '-27.4732'}, 'bytes_sent': 26148864, 'download': 0, 'timestamp': '2021-02-14T22:54:27.902614Z', 'share': None, 'bytes_received': 0, 'ping': 14.774, 'upload': 20746227.40552687, 'server': {'latency': 14.774, 'name': 'Brisbane', 'url': 'http://qld.speedtest.exetel.com.au:8080/speedtest/upload.php', 'country': 'Australia', 'lon': '153.0251', 'cc': 'AU', 'host': 'qld.speedtest.exetel.com.au:8080', 'sponsor': 'Exetel', 'lat': '-27.4698', 'id': '13276', 'd': 0.5187222967744429}} ########################################################### # Python 3 Run ########################################################### .DEBUG: Upload Start:224944 ..DEBUG: Upload Timeout: result=524288 ..DEBUG: Upload Timeout: result=524288 ..DEBUG: Upload Timeout: result=524288 ..DEBUG: Upload Timeout: result=524288 ............................................................................................. DEBUG: Upload Stop:224955 DEBUG: Bytes sent:[524288, 524288, 524288, 524288, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] DEBUG: Bytes sent:2097152 DEBUG: Upload Value:1574428 Upload: 1.57 Mbit/s DEBUG: Results: {'download': 0, 'upload': 1574428.9733984023, 'ping': 13.135, 'server': {'url': 'http://brs1.speedtest.telstra.net:8080/speedtest/upload.php', 'lat': '-27.4728', 'lon': '153.0278', 'name': 'Brisbane', 'country': 'Australia', 'cc': 'AU', 'sponsor': 'Telstra', 'id': '2604', 'host': 'brs1.speedtest.telstra.net:8080', 'd': 0.62311775977947, 'latency': 13.135}, 'timestamp': '2021-02-14T22:55:17.200248Z', 'bytes_sent': 2097152, 'bytes_received': 0, 'share': None, 'client': {'ip': '193.116.x.x', 'lat': '-27.4732', 'lon': '153.0215', 'isp': 'TPG Internet', 'isprating': '3.7', 'rating': '0', 'ispdlavg': '0', 'ispulavg': '0', 'loggedin': '0', 'country': 'AU'}} [root@primary bin]#     I've temporarily resolved by forcing python2 in inputs.conf   [script://$SPLUNK_HOME/etc/apps/speedtest/bin/speedtest-cli.py --json] disabled = 0 interval = 5,35 * * * * python.version = python2 ... View more

The plot thickens. Normal run of script - OK Run script as Splunk user - OK Run script via Splunk binary - Lower result. JSON format slightly different order too. Maybe something to do with Splunk's Python????? I'm running Splunk 8.1.2 (build 545206cc9f70)   Edit: I think I found the issue Manually run with Python2 - OK Python3 - Lower result ... View more

Thanks Mark. Happy to run some more tests if you need me to. It's pretty odd though I have no idea how you would even begin to troubleshoot it or how it would behave like this only when run by Splunk. ... View more

It got approved and is up in the cloud. Executives are a bit excited by the moving pictures. Thanks Daniel ... View more

Thanks Daniel, Yep I submitted it again when another of my colleagues pointed out that you have updated it to 1.0.1 that day. Hopefully it makes it through this time. I'll let you know how it goes. ... View more