Knowledge Management

Generic Bitwise field decoding

phoenixdigital
Builder

I'm thinking this might required a custom search command which I'd like to try to avoid if possible.

I have about 10 bitwise fields in a dataset with each bit in each field representing an alarm condition.

This works just fine however I would like to create something more generic if at all possible

| eval bit0Value=if(floor(bitField)%2>0,"on","off") 
| eval bit1Value=if(floor(bitField/2)%2>0,"on","off") 
| eval bit2Value=if(floor(bitField/4)%2>0,"on","off") 
| eval bit3Value=if(floor(bitField/8)%2>0,"on","off") 

So the above takes the field 'bitField' and creates a new field for each bit that is set.

This works and I could implement it as is with 10 different macros for each of my 10 bitwise fields.

However ideally I would prefer to have one macro to do it all. Something along the lines of a call like this

index=blah sourcetype=blah 
| `myBitDecodingMacro("bit0Value, bit1Value, bit2Value, bit3Value",bitField)`

Where the first string passed to the macro is the fieldnames I want to apply to each bit that is set in the bitwise field.

Is this even possible with a macro?

Am I going to have to suck it up and write a custom search command?

or more likely just have 10 different macros one for each bitwise field.

Tags (2)
0 Karma

woodcock
Esteemed Legend

Come back and pick an answer and click Accept to close the question and award your bounty!

0 Karma

gesman_splunk
Splunk Employee
Splunk Employee

I'd create a lookup:

| makeresults count=65536
| streamstats c 
| eval value_dec=tostring(c-1)
| eval value_0xhex=tostring((c-1), "hex")
| eval value_hex=replace(value_0xhex, "^0x", "")
| eval value_bin=value_hex
| rex mode=sed field=value_bin "s/0/0000/g s/1/0001/g s/2/0010/g s/3/0011/g s/4/0100/g s/5/0101/g s/6/0110/g s/7/0111/g s/8/1000/g s/9/1001/g s/a|A/1010/g s/b|B/1011/g s/c|C/1100/g s/d|D/1101/g s/e|E/1110/g s/f|F/1111/g" 
| eval value_bin=substr("0000000000000000", 0, max(16-len(value_bin), 0)) . value_bin
| rex field=value_bin "(?<bit__15>.)(?<bit__14>.)(?<bit__13>.)(?<bit__12>.)(?<bit__11>.)(?<bit__10>.)(?<bit__09>.)(?<bit__08>.)(?<bit__07>.)(?<bit__06>.)(?<bit__05>.)(?<bit__04>.)(?<bit__03>.)(?<bit__02>.)(?<bit__01>.)(?<bit__00>.)" 
| table value_dec value_hex value_0xhex value_bin bit__*
| outputlookup convert_value.csv

And then use this lookup to convert any number to any radix and have all my bits automatically extracted:

| makeresults count=111 | streamstats c 
| lookup convert_value.csv value_dec AS c | table c value_* bit__*

Resulted lookup (even for 65536 values) is <5MB and will work really fast. You can make it even faster by generating smaller lookup table (if you don't need all 16 bits)

0 Karma

phoenixdigital
Builder

No unfortunately this macro was meant to simplify the creation and population of them.

It's all good though don't try too hard. I solved it with 4 different macros as there was some commonality with some of the groupings.

I did a very similar method to your examples. Great minds think alike. Two examples below.

MacroSimple(fieldName,bitField)

eval bankA_$fieldName$=if(floor($bitField$)%2>0,1,0) 
| eval bankB_$fieldName$=if(floor($bitField$/2)%2>0,1,0) 
| eval bankC_$fieldName$=if(floor($bitField$/4)%2>0,1,0)
| eval bankD_$fieldName$=if(floor($bitField$/8)%2>0,1,0)
| eval bankE_$fieldName$=if(floor($bitField$/16)%2>0,1,0)
| eval bankF_$fieldName$=if(floor($bitField$/32)%2>0,1,0)

The most complicated one which actually spanned 3 integers and instead of making many fields populated a single field with a multivalue set of results based on which bit was set.

bitFieldGensetDecoding(fieldName,bitFieldA,bitFieldB,bitFieldC)

eval gen$fieldName$=if(floor($bitFieldA$)%2>0,"Gen01$fieldName$","") 
| eval gen$fieldName$=if(floor($bitFieldA$/2)%2>0,mvappend(gen$fieldName$,"Gen02$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldA$/4)%2>0,mvappend(gen$fieldName$,"Gen03$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldA$/8)%2>0,mvappend(gen$fieldName$,"Gen04$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldA$/16)%2>0,mvappend(gen$fieldName$,"Gen05$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldA$/32)%2>0,mvappend(gen$fieldName$,"Gen06$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldA$/64)%2>0,mvappend(gen$fieldName$,"Gen07$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldA$/128)%2>0,mvappend(gen$fieldName$,"Gen08$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldA$/256)%2>0,mvappend(gen$fieldName$,"Gen09$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldA$/512)%2>0,mvappend(gen$fieldName$,"Gen10$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldA$/1024)%2>0,mvappend(gen$fieldName$,"Gen11$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldA$/2048)%2>0,mvappend(gen$fieldName$,"Gen12$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldA$/4096)%2>0,mvappend(gen$fieldName$,"Gen13$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldA$/8192)%2>0,mvappend(gen$fieldName$,"Gen14$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldA$/16384)%2>0,mvappend(gen$fieldName$,"Gen15$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldA$/32768)%2>0,mvappend(gen$fieldName$,"Gen16$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$)%2>0,mvappend(gen$fieldName$,"Gen17$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/2)%2>0,mvappend(gen$fieldName$,"Gen18$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/4)%2>0,mvappend(gen$fieldName$,"Gen19$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/8)%2>0,mvappend(gen$fieldName$,"Gen20$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/16)%2>0,mvappend(gen$fieldName$,"Gen21$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/32)%2>0,mvappend(gen$fieldName$,"Gen22$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/64)%2>0,mvappend(gen$fieldName$,"Gen23$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/128)%2>0,mvappend(gen$fieldName$,"Gen24$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/256)%2>0,mvappend(gen$fieldName$,"Gen25$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/512)%2>0,mvappend(gen$fieldName$,"Gen26$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/1024)%2>0,mvappend(gen$fieldName$,"Gen27$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/2048)%2>0,mvappend(gen$fieldName$,"Gen28$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/4096)%2>0,mvappend(gen$fieldName$,"Gen29$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/8192)%2>0,mvappend(gen$fieldName$,"Gen30$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/16384)%2>0,mvappend(gen$fieldName$,"Gen31$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldB$/32768)%2>0,mvappend(gen$fieldName$,"Gen32$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldC$)%2>0,mvappend(gen$fieldName$,"Gen33$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldC$/2)%2>0,mvappend(gen$fieldName$,"Gen34$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldC$/4)%2>0,mvappend(gen$fieldName$,"Gen35$fieldName$"),gen$fieldName$) 
| eval gen$fieldName$=if(floor($bitFieldC$/8)%2>0,mvappend(gen$fieldName$,"Gen36$fieldName$"),gen$fieldName$)

Good times 🙂

0 Karma

woodcock
Esteemed Legend

I really do not think anything better can be done with core SPL.

0 Karma

phoenixdigital
Builder

If the client wants to spend the dev time I could whip up a custom search command which would do it. If they do I'll post the code up here or package it up as an app for Splunkbase (if they allow custom search commands as an app???)

0 Karma

woodcock
Esteemed Legend

If you just need a macro, then this will do:

Name=myBitDecodingMacro(5)
Definition=eval $bit0Value$=if(floor($bitField$)%2>0,"on","off") \
| eval $bit1Value$=if(floor($bitField$/2)%2>0,"on","off") \
| eval $bit2Value$=if(floor($bitField$/4)%2>0,"on","off") \
| eval $bit3Value$=if(floor($bitField$/8)%2>0,"on","off")
Arguments=bit0Value, bit1Value, bit2Value, bit3Value, bitField  

Tested A-OK; compare this:

| makeresults 
| eval bitField=11
| eval bit0Value=if(floor(bitField)%2>0,"on","off") 
| eval bit1Value=if(floor(bitField/2)%2>0,"on","off") 
| eval bit2Value=if(floor(bitField/4)%2>0,"on","off") 
| eval bit3Value=if(floor(bitField/8)%2>0,"on","off")

To this:

| makeresults 
| eval bitField=11

| rename COMMENT AS "Everything above fakes your data; everything below is your solution"

| `myBitDecodingMacro(bit0Value, bit1Value, bit2Value, bit3Value ,bitField)`

Or even more simply, like this:

Name=myBitDecodingMacro(2)
Definition=eval $prefix$0Value=if(floor($bitField$)%2>0,"on","off") | eval $prefix$1Value=if(floor($bitField$/2)%2>0,"on","off") | eval $prefix$2Value=if(floor($bitField$/4)%2>0,"on","off") | eval $prefix$3Value=if(floor($bitField$/8)%2>0,"on","off")
Arguments=prefix, bitField  

Used like this:

| makeresults 
| eval bitField=11

| rename COMMENT AS "Everything above fakes your data; everything below is your solution"

| `myBitDecodingMacro(bit ,bitField)`
0 Karma

phoenixdigital
Builder

Thanks for the reply. That is essentially what I am doing already with macros but what I was hoping was something like this. Which could create anywhere from 1 - 32 fields depending on what was passed in for the first field.

myBitDecoding("alarm1, alarm2, doorSensor1, doorSensor4", bitField)

and then reuse the same macro for another decoding of completely different data with only maybe two bitfields to extract

myBitDecoding("motionDetectorHallway,motionDetectorFridge", bitField)

then again for a massive set of bits

myBitDecoding("highTempAlarm,lowTempAlarm,highRPM,lowRPM,noPower,myBrainWentOfflineTryingToMakeThisAGenericMacro", bitField)

Dynamically being able to provide the fieldnames is what I was aiming for.

0 Karma

woodcock
Esteemed Legend

If the fields exist with any non-null value already, then I think that I can make it work, but not if the fields do not already exist. Is mass field initialization to zero doable?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...