Hi!
I created a custom command for a comparison between json.
The steps to create it:
0) created fieldcompare folder in etc/apps (with bin, metadata and default subfolders)
1) fieldcompare.py (in etc/apps/fieldcompare/bin)
# custom command
import splunk.Intersplunk
import json
from splunklib.searchcommands import \
dispatch, StreamingCommand, Configuration, Option, validators
# long def function (searchleveljson) that looks for things in jsons
#etc etc
(isgetinfo, sys.argv) = splunk.Intersplunk.isGetInfo(sys.argv)
args, kwargs = splunk.Intersplunk.getKeywordsAndOptions()
if isgetinfo:
# streaming, generating, retevs, reqsop, preop
splunk.Intersplunk.outputInfo(True, False, False, False, None)
(results, dummyresults, settings) = splunk.Intersplunk.getOrganizedResults()
field1 = kwargs.get("field1", "field1")
field2 = kwargs.get("field2", "field2")
try:
for result in results:
try:
j1 = json.load(field1)
j2 = json.load(field2)
except KeyError:
# If either field is missing, simply ignore
continue
resultKey = list()
resultValues = list()
searchleveljson(j1,j2,resultKey,resultValues)
result["mismatched keys"]=json.dumps(resultKey)
result["Value diff"]=json.dumps(resultValues)
splunk.Intersplunk.outputResults(results)
2) commands.conf (in etc/apps/fieldcompare/default)
[fieldcompare]
filename = fieldcompare.py
supports_getinfo = true
3) default.meta (in etc/apps/fieldcompare/metadata)
[commands/fieldcompare]
access = read : [ * ], write : [ admin ]
export = system
[scripts/fieldcompare.py]
access = read : [ * ], write : [ admin ]
export = system
4) copy splunklib (with modularinput and searchcommands) in etc/apps/fieldcompare/bin from etc\apps\framework\contrib\splunk-sdk-python
5) Restart splunk
6) index=* etc etc | table expected actual | fieldcompare field1=expected field2=actual
but I get an error that says:
Error in 'script': Getinfo probe failed for external search command 'fieldcompare'
What am I doing wrong?
Thanks,
have a good day.
Debora
Intersplunk solution:
apps/fieldcompare/bin/fieldcompare.py:
# custom command script
import splunk.Intersplunk
import json
import sys
def is_json(myjson):
return type(myjson) == type({})
#found difference between two jsons
def searchleveljson(json1,json2,resultKey,resultValues):
keys=set(json1.keys())
keys2=set(json2.keys())
keys.update(keys2)
for json_event in keys:
if json_event not in json1:
resultKey.append(json_event)
resultValues.append("actual."+json_event+"."+json2[json_event])
continue;
if json_event not in json2:
resultKey.append(json_event)
resultValues.append("expected."+json_event+"."+json1[json_event])
continue;
v1 = json1[json_event]
v2 = json2[json_event]
if v1 != v2:
if is_json(v1) and is_json(v2):
searchleveljson(v1,v2,resultKey,resultValues)
else:
resultKey.append(json_event)
diff = "actual."+json_event+"="+json2[json_event]
diff+= " "
diff+= "expected."+json_event+"="+json1[json_event]
resultValues.append(diff)
(isgetinfo, sys.argv) = splunk.Intersplunk.isGetInfo(sys.argv)
args, kwargs = splunk.Intersplunk.getKeywordsAndOptions()
if isgetinfo:
# streaming, generating, retevs, reqsop, preop
splunk.Intersplunk.outputInfo(True, False, False, False, None)
results, dummyresults, settings = splunk.Intersplunk.getOrganizedResults()
field1 = kwargs.get("json1", "field1")
field2 = kwargs.get("json2", "field2")
for result in results:
try:
#corrected errors for converting string to json
#json.load(fieldx) -> json.loads(result[fieldx])
j1 = json.loads(result[field1])
j2 = json.loads(result[field2])
except ValueError:
pass
resultKey = list()
resultValues = list()
searchleveljson(j1,j2,resultKey,resultValues)
result["mismatched_keys"]=json.dumps(resultKey)
result["value_diff"]=json.dumps(resultValues)
splunk.Intersplunk.outputResults(results)
apps/fieldcompare/default/custom.conf:
[fieldcompare]
filename = fieldcompare.py
streaming=true
apps/fieldcompare/metadata/default.meta:
[commands/fieldcompare]
access = read : [ * ], write : [ admin ]
export = system
[scripts/fieldcompare.py]
access = read : [ * ], write : [ admin ]
export = system
I deleted all other folders (spunk and splunklib)
Restart splunk.
| table column1 column2 | fieldcompare __EXECUTE__ json1=column1 json2=column2
the result is a table with the following columns:
column1 | column2 | mismatched_keys | value_diff
mismatched_keys contains all different keys
value_diff contains all different value
Intersplunk solution:
apps/fieldcompare/bin/fieldcompare.py:
# custom command script
import splunk.Intersplunk
import json
import sys
def is_json(myjson):
return type(myjson) == type({})
#found difference between two jsons
def searchleveljson(json1,json2,resultKey,resultValues):
keys=set(json1.keys())
keys2=set(json2.keys())
keys.update(keys2)
for json_event in keys:
if json_event not in json1:
resultKey.append(json_event)
resultValues.append("actual."+json_event+"."+json2[json_event])
continue;
if json_event not in json2:
resultKey.append(json_event)
resultValues.append("expected."+json_event+"."+json1[json_event])
continue;
v1 = json1[json_event]
v2 = json2[json_event]
if v1 != v2:
if is_json(v1) and is_json(v2):
searchleveljson(v1,v2,resultKey,resultValues)
else:
resultKey.append(json_event)
diff = "actual."+json_event+"="+json2[json_event]
diff+= " "
diff+= "expected."+json_event+"="+json1[json_event]
resultValues.append(diff)
(isgetinfo, sys.argv) = splunk.Intersplunk.isGetInfo(sys.argv)
args, kwargs = splunk.Intersplunk.getKeywordsAndOptions()
if isgetinfo:
# streaming, generating, retevs, reqsop, preop
splunk.Intersplunk.outputInfo(True, False, False, False, None)
results, dummyresults, settings = splunk.Intersplunk.getOrganizedResults()
field1 = kwargs.get("json1", "field1")
field2 = kwargs.get("json2", "field2")
for result in results:
try:
#corrected errors for converting string to json
#json.load(fieldx) -> json.loads(result[fieldx])
j1 = json.loads(result[field1])
j2 = json.loads(result[field2])
except ValueError:
pass
resultKey = list()
resultValues = list()
searchleveljson(j1,j2,resultKey,resultValues)
result["mismatched_keys"]=json.dumps(resultKey)
result["value_diff"]=json.dumps(resultValues)
splunk.Intersplunk.outputResults(results)
apps/fieldcompare/default/custom.conf:
[fieldcompare]
filename = fieldcompare.py
streaming=true
apps/fieldcompare/metadata/default.meta:
[commands/fieldcompare]
access = read : [ * ], write : [ admin ]
export = system
[scripts/fieldcompare.py]
access = read : [ * ], write : [ admin ]
export = system
I deleted all other folders (spunk and splunklib)
Restart splunk.
| table column1 column2 | fieldcompare __EXECUTE__ json1=column1 json2=column2
the result is a table with the following columns:
column1 | column2 | mismatched_keys | value_diff
mismatched_keys contains all different keys
value_diff contains all different value
Are you getting any error when you start your splunk server? Any compilation error in your customcommand will get reported only at starting of Splunk server.
Also you can check https://splunkbase.splunk.com/app/3447/. It has different way of writing streaming command.
it doesn't find any errors at startup.
This is right method to use intersplunk?
I would suggest moving to the SDK (referenced in your script as splunklib
but not actually used). Check out the filter.py example that comes with the SDK for guidance.
Yes use of intersplunk is correct.
Few things I just noticed from your code. You are creating tupple when fetching results. You should update that query as follows:
results, dummyresults, settings = splunk.Intersplunk.getOrganizedResults()
When you return result, it should be list of directories.
I tried to create the custom command in both ways.
Intersplunk way:
I added "import sys" and removed import from splunklib.
From splunk search page I read an error that asks to add GETINFO or EXECUTE
I changed the command to:
| fieldcompare EXECUTE field1=x field2=y
but I get a generic error 1.
I add getOrganizedResults before outputResults but nothing has changed.
I would just like to pass two parameters and get two new ones for each line but it seems more complicated than expected.
SDK way:
I followed the documentation.
I started from SDK templates, taking inspiration from the examples and I created a class that extends a GeneratingCommand but the error is always the same:
Error in 'script': Getinfo probe failed for external search command 'jsoncompare'
Thanks 🙂
I solved the problem with Intersplunk!
Please post your solution as an answer and accept that answer so that others may benefit from your findings.