It doesn't address it . This was still part of the original question just showing a more complex form. I'm still waiting on a response 2.5 years on 🙂 ... View more

We have been using a python library for a few years now. It is pretty bullet proof but obviously needs custom code written to interract with the devices. This is the library we use. https://github.com/ljean/modbus-tk We wrote a complex set of daemons which actively poll SCADA devices at regular intervals via modbusTCP. It works but having something in Stream would be great. Primarily I wanted it from a security perspective to "catch" any unwanted modbus traffic being sent to SCADA devices. The protocol is pretty basic so wouldn't be too hard to implement. A framework for us to add our own Stream decoders would be even better though. ... View more

Good to know. I haven't looked much at Eventgen since I posted this ~2 years ago it has likely changed a bit since then. https://github.com/splunk/eventgen Back then if eventgen didn't find a timestamp on every line it borked out as shown above. ... View more

Its been a while since I looked at this but I should note the error is in the eventgen code not in the Splunk ingestion config. Eventgen can't handle generating sample data where it is multiple lines. It is breaking as eventgen tries to make fake logs from this sample data. You are right though if I was having problems ingesting this data into Splunk. Changing the breaker would make this work just fine. However changing the breaker will have no effect on eventgen as far as I know. ... View more

It didn't on a Heavy Forwarder on Windows either sadly. I was only testing on a Universal Forwarder to try to debug the issue locally. ... View more

Granted I can see why it would be a safe guaranteed way to stop a imposter forwarder from sending in data. That said though it appears Splunk is completely ignoring the requireClientCert because even when I set it to false it still doesn't work. ... View more

Yeah there is still confusion amongst some of our engineers (and splunk support) who are convinced server.conf settings can affect the port 8000. We are not sure which is right yet. Notably things like cipherSuite, sslVersions and sslVersionsForClient. Thanks for the tip. I forgot you could add images directly (fixed it up) ... View more

One thing that has puzzled me about this "cheat" for ingesting binary data. What happens if new data is written to the end of these files? Does it reingest it all over again when new data is appended? Do I need to keep track of where I left off in my script or will Splunk just send me the new section of data? I'll find out in a few hours after some more testing no doubt. ... View more

Thanks again that was exactly the problem. I'm sure I read somewhere a few days ago that unarchive_cmd had to be applied to sourcetypes only but there it is in black an white. It is working perfectly now. Sorry MuS had to unaccept your answer and give it to micahkemp who beat you by 6 minutes. ... View more

Bingo! That was exactly the problem. I was just coming back to correct my post and answer my own question but you all beat me to it. I did read this a few days ago and was certain I read somewhere it was only valid on sourcetype and not source. ... View more

Thank you. I like your thinking!!!! That works perfectly with a few tweaks but you got me on the right track. http://imgur.com/a/QdEMp Here is the actual dashboard in all its glory. Essentially it is charting the average speed for traffic on freeways over the last 6 months with percentiles to give an idea of variability then comparing to today traffic speeds. <dashboard stylesheet="liveworm.css" refresh="300"> <label>Foyer 1 - Live Freeway Speeds</label> <description>* Upper and lower bands indicate the 95th and 5th percentile of the average speed.</description> <search id="base"> <query>| tstats count AS "Count of BluetoothLinks" avg(BluetoothLinks.speed) AS speed from datamodel=Customer_Bluetooth_Averaged.BluetoothLinks where (nodename = BluetoothLinks) (BluetoothLinks.Freeway="Y") (BluetoothLinks.WORKDAY_FLG= "Y ") (BluetoothLinks.bluetooth_id="*") (BluetoothLinks.date_hour&gt;=5) (BluetoothLinks.date_hour&lt;=20) groupby BluetoothLinks.TIME_KEY_30M, BluetoothLinks.DATE_KEY | rename BluetoothLinks.TIME_KEY_30M AS TIME_KEY | rename BluetoothLinks.DATE_KEY AS DATE_KEY | fields TIME_KEY, DATE_KEY, speed | stats avg(speed) AS "Average speed" perc95(speed) AS "95th Percentile speed" perc5(speed) AS "5th Percentile speed" by TIME_KEY | join type=left TIME_KEY [ search index=Customer* sourcetype=json_bluetooth_links bluetooth_id="*" Freeway="Y" earliest=@d latest=now | eval speed = (length/1000)/(travel_time/60) | eval timeSegmentMin=floor(date_min/30) * 30 | eval timeSegmentMin=if(length(timeSegmentMin)&lt;2,"0"+timeSegmentMin,timeSegmentMin) | eval TIME_KEY = date_hour + ":" + timeSegmentMin | eval today_metric = if (strftime(time(), "%Y%m%d") == strftime(_time, "%Y%m%d"), speed, NULL) | fields today_metric, speed, name, bluetooth_id, _time, TIME_KEY | stats avg(today_metric) as "Today speed" by TIME_KEY ] | eval _lower = "5th Percentile speed" | eval _upper = "95th Percentile speed" | eval _predicted = "Average speed" | rename TIME_KEY AS "Time of Day" | eventstats min("5th Percentile speed") as min max("95th Percentile speed") as max |eval max=max+5 |eval min=min-5</query> <earliest>-6mon</earliest> <latest>@day</latest> <done> <set token="maxaxis">$result.max$</set> <set token="minaxis">$result.min$</set> </done> </search> <row> <panel> <chart id="liveworm"> <title>Min $minaxis$ and Max $maxaxis$</title> <search base="base"> <query>fields "Time of Day", "Average speed", "5th Percentile speed", "95th Percentile speed", "Today speed", _lower, _upper, _predicted</query> </search> <option name="charting.axisLabelsX.majorLabelStyle.rotation">-90</option> <option name="charting.axisY.includeZero">0</option> <option name="charting.chart">line</option> <option name="charting.axisY.minimumNumber">$minaxis$</option> <option name="charting.axisY.maximumNumber">$maxaxis$</option> <option name="charting.chart">line</option> <option name="charting.fieldColors">{"Today speed": 0xE51010, "Average speed": 0x1E93C6, "Median speed":0x9DD1F9}</option> <option name="height">750</option> </chart> </panel> </row> </dashboard> ... View more

Yeah I think you are right. cmerriman did find a solution to my simple test but in my edited post I have a more accurate search simulating the data I am working with and cmerriman's fix doesn't work on that. ... View more

Thanks for the response. Removing that from the basic test did in fact work. However it didn't work on my real search with my own data. I have editted the original post with a search pretty much simulating the data and visualisation I am working with and removing auto minimum value doesn't work sadly. ... View more

Thanks for the response. That was just a test query I built since I knew nobody had my data. Everyone has _internal so I wrote that search to simulate the data I am working with. I have editted the original post with a search pretty much simulating the data and visualisation I am working with. Still not working for me. ... View more

If the client wants to spend the dev time I could whip up a custom search command which would do it. If they do I'll post the code up here or package it up as an app for Splunkbase (if they allow custom search commands as an app???) ... View more

No unfortunately this macro was meant to simplify the creation and population of them. It's all good though don't try too hard. I solved it with 4 different macros as there was some commonality with some of the groupings. I did a very similar method to your examples. Great minds think alike. Two examples below. MacroSimple(fieldName,bitField) eval bankA_$fieldName$=if(floor($bitField$)%2>0,1,0) | eval bankB_$fieldName$=if(floor($bitField$/2)%2>0,1,0) | eval bankC_$fieldName$=if(floor($bitField$/4)%2>0,1,0) | eval bankD_$fieldName$=if(floor($bitField$/8)%2>0,1,0) | eval bankE_$fieldName$=if(floor($bitField$/16)%2>0,1,0) | eval bankF_$fieldName$=if(floor($bitField$/32)%2>0,1,0) The most complicated one which actually spanned 3 integers and instead of making many fields populated a single field with a multivalue set of results based on which bit was set. bitFieldGensetDecoding(fieldName,bitFieldA,bitFieldB,bitFieldC) eval gen$fieldName$=if(floor($bitFieldA$)%2>0,"Gen01$fieldName$","") | eval gen$fieldName$=if(floor($bitFieldA$/2)%2>0,mvappend(gen$fieldName$,"Gen02$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldA$/4)%2>0,mvappend(gen$fieldName$,"Gen03$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldA$/8)%2>0,mvappend(gen$fieldName$,"Gen04$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldA$/16)%2>0,mvappend(gen$fieldName$,"Gen05$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldA$/32)%2>0,mvappend(gen$fieldName$,"Gen06$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldA$/64)%2>0,mvappend(gen$fieldName$,"Gen07$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldA$/128)%2>0,mvappend(gen$fieldName$,"Gen08$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldA$/256)%2>0,mvappend(gen$fieldName$,"Gen09$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldA$/512)%2>0,mvappend(gen$fieldName$,"Gen10$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldA$/1024)%2>0,mvappend(gen$fieldName$,"Gen11$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldA$/2048)%2>0,mvappend(gen$fieldName$,"Gen12$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldA$/4096)%2>0,mvappend(gen$fieldName$,"Gen13$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldA$/8192)%2>0,mvappend(gen$fieldName$,"Gen14$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldA$/16384)%2>0,mvappend(gen$fieldName$,"Gen15$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldA$/32768)%2>0,mvappend(gen$fieldName$,"Gen16$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$)%2>0,mvappend(gen$fieldName$,"Gen17$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/2)%2>0,mvappend(gen$fieldName$,"Gen18$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/4)%2>0,mvappend(gen$fieldName$,"Gen19$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/8)%2>0,mvappend(gen$fieldName$,"Gen20$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/16)%2>0,mvappend(gen$fieldName$,"Gen21$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/32)%2>0,mvappend(gen$fieldName$,"Gen22$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/64)%2>0,mvappend(gen$fieldName$,"Gen23$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/128)%2>0,mvappend(gen$fieldName$,"Gen24$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/256)%2>0,mvappend(gen$fieldName$,"Gen25$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/512)%2>0,mvappend(gen$fieldName$,"Gen26$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/1024)%2>0,mvappend(gen$fieldName$,"Gen27$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/2048)%2>0,mvappend(gen$fieldName$,"Gen28$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/4096)%2>0,mvappend(gen$fieldName$,"Gen29$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/8192)%2>0,mvappend(gen$fieldName$,"Gen30$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/16384)%2>0,mvappend(gen$fieldName$,"Gen31$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldB$/32768)%2>0,mvappend(gen$fieldName$,"Gen32$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldC$)%2>0,mvappend(gen$fieldName$,"Gen33$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldC$/2)%2>0,mvappend(gen$fieldName$,"Gen34$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldC$/4)%2>0,mvappend(gen$fieldName$,"Gen35$fieldName$"),gen$fieldName$) | eval gen$fieldName$=if(floor($bitFieldC$/8)%2>0,mvappend(gen$fieldName$,"Gen36$fieldName$"),gen$fieldName$) Good times 🙂 ... View more

Thanks for the reply. That is essentially what I am doing already with macros but what I was hoping was something like this. Which could create anywhere from 1 - 32 fields depending on what was passed in for the first field. myBitDecoding("alarm1, alarm2, doorSensor1, doorSensor4", bitField) and then reuse the same macro for another decoding of completely different data with only maybe two bitfields to extract myBitDecoding("motionDetectorHallway,motionDetectorFridge", bitField) then again for a massive set of bits myBitDecoding("highTempAlarm,lowTempAlarm,highRPM,lowRPM,noPower,myBrainWentOfflineTryingToMakeThisAGenericMacro", bitField) Dynamically being able to provide the fieldnames is what I was aiming for. ... View more