Unable to access event data for comparison reasons

[gsmith93]

I have ingested configuration information from WebSphere Application Server. Specifically, appserver configuration data. The events look like the following:

{ 
Attributes: { 
:
-DFileNet.Content.DownloadServerAffinityEnabled: TRUE
-DFileNet.Content.GetBlockSizeKB: 4096
-DFileNet.Content.PutBlockSizeKB: 4096
-DFileNet.Content.UploadServerAffinityEnabled: TRUE
-DFileNet.WSI.AutoDetectLTPAToken: is not set
-Dappdynamics.agent.applicationName: is not set
-Dappdynamics.agent.nodeName: is not set
-Dcom.filenet.authentication.token.userid: is not set
-Dcom.filenet.authentication.wsi.AutoDetectAuthToken: is not set
-Dcom.filenet.repositoryconfig.allowWSIOnWAS: is not set
-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterAction: is not set
-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterReplacement: is not set
-Dfilenet.pchconfig: is not set
-Djava.awt.headless: TRUE
-Djava.security.auth.login.config: ${USER_DIR}/DocumentRepository/jaas.conf.WebSphere
-Djaxws.payload.highFidelity: is not set
-Xdump: system:none
-Xgcpolicy: is not set
-Xmn2048M: is not set
-Xmn512M: is not set
-Xquickstart: is set
-Xverbosegclog: is not set
-javaagent: is not set
-server: is set
genericJvmArguments: -Xgcpolicy:gencon -Djaxws.payload.highFidelity=true -Dfilenet.pchconfig=${USER_DIR}\HJIPDash\PchConfig.properties -Xmn512M -Dcom.filenet.authentication.token.userid=sso:ltpa -DFileNet.WSI.AutoDetectLTPAToken=true -Dcom.filenet.authentication.wsi.AutoDetectAuthToken=true -Dcom.filenet.repositoryconfig.allowWSIOnWAS=true -DFileNet.Content.PutBlockSizeKB=10240 -DFileNet.Content.GetBlockSizeKB=10240
}
Env: UAT
Object: HJn8server3 (HJn8, UAT-230612150440)
SectionName: JVM Arguments
}

The eveants field are made up of the following:

Objects: AppServer Names

SectionName: Are the various sections of an application server. For example: "JVM Configuratioin"

Attributes: Are all the configurations for a give SectionName

I have been unable to traverse the "Attributes" field of these events. I have tried making the Attributes field into an JSON Array and/or Object but have had no luck.  My search code has gotten so convoluted, I don't know where to start. The code I use to create a table listing the attributes of two appserver is:

index=websphere_cct SectionName
     [ search index=websphere_cct
        | dedup SectionName
        | streamstats count as "RowCount"
        | eval newfield="panel".RowCount
        | head 1
        | tail
        | head 1
        | table SectionName] Object=" HJn6server1 (HJn6, PROD-230612151857)" OR Object=" HJn4server1 (HJn4, PROD-230612151857)"
| table Object, SectionName, Attributes.*
| transpose 0 header_field=Object
| fields - Attributes.*

This produces the table:

[yuanliu]

Can you explain what "traverse the Attributes field" pertains to?  The first prominent problem I see is the fact that the posted sample data is terribly butchered JSON. (Three big problems: keys and values are not properly quoted; newlines are not escaped; backslash characters used in Windows/DOS path are not escaped.)  A conformant notation should read

{
"Attributes": ":\n-DFileNet.Content.DownloadServerAffinityEnabled: TRUE\n-DFileNet.Content.GetBlockSizeKB: 4096\n-DFileNet.Content.PutBlockSizeKB: 4096\n-DFileNet.Content.UploadServerAffinityEnabled: TRUE\n-DFileNet.WSI.AutoDetectLTPAToken: is not set\n-Dappdynamics.agent.applicationName: is not set\n-Dappdynamics.agent.nodeName: is not set\n-Dcom.filenet.authentication.token.userid: is not set\n-Dcom.filenet.authentication.wsi.AutoDetectAuthToken: is not set\n-Dcom.filenet.repositoryconfig.allowWSIOnWAS: is not set\n-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterAction: is not set\n-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterReplacement: is not set\n-Dfilenet.pchconfig: is not set\n-Djava.awt.headless: TRUE\n-Djava.security.auth.login.config: ${USER_DIR}/DocumentRepository/jaas.conf.WebSphere\n-Djaxws.payload.highFidelity: is not set\n-Xdump: system:none\n-Xgcpolicy: is not set\n-Xmn2048M: is not set\n-Xmn512M: is not set\n-Xquickstart: is set\n-Xverbosegclog: is not set\n-javaagent: is not set\n-server: is set\ngenericJvmArguments: -Xgcpolicy:gencon\n-Djaxws.payload.highFidelity=true -Dfilenet.pchconfig=${USER_DIR}\\HJIPDash\\PchConfig.properties -Xmn512M -Dcom.filenet.authentication.token.userid=sso:ltpa -DFileNet.WSI.AutoDetectLTPAToken=true -Dcom.filenet.authentication.wsi.AutoDetectAuthToken=true -Dcom.filenet.repositoryconfig.allowWSIOnWAS=true -DFileNet.Content.PutBlockSizeKB=10240 -DFileNet.Content.GetBlockSizeKB=10240",
"Env": "UAT",
"Object": "HJn8server3 (HJn8, UAT-230612150440)",
"SectionName": "JVM Arguments"
}

If your events do not look like that, Splunk cannot even extract Attributes properly.  Can you clarify whether raw events are conformant JSON and post real samples? (Anonymize as needed.)

Also, your illustrated code implies the presence of field name pattern Attributes.*  Based on your posted event, there is no way Splunk can give you those.  Can you explain?

[gsmith61293]

Thank you for the reply and pointing out my JSON format is incorrect. I will start with that.

Per the events that I ingest, I did not show the raw data. The raw data looks as such

{ "Object":" HJn6server1 (HJn6, PROD-230612151857)","Env":"PROD","SectionName":"JVM Arguments", "Attributes":{"genericJvmArguments": "-Djava.awt.headless=true -server -Xdump:system:none -javaagent:D:\\AppDynamics\\AppServerAgent\\javaagent.jar -Dappdynamics.agent.nodeName=${WAS_SERVER_NAME}","-DFileNet.Content.DownloadServerAffinityEnabled": "TRUE","-DFileNet.Content.GetBlockSizeKB": "4096","-DFileNet.Content.PutBlockSizeKB": "4096","-DFileNet.Content.UploadServerAffinityEnabled": "TRUE","-DFileNet.WSI.AutoDetectLTPAToken": "is not set","-Dappdynamics.agent.applicationName": "is not set","-Dappdynamics.agent.nodeName": "${WAS_SERVER_NAME}","-Dcom.filenet.authentication.token.userid": "is not set","-Dcom.filenet.authentication.wsi.AutoDetectAuthToken": "is not set","-Dcom.filenet.repositoryconfig.allowWSIOnWAS": "is not set","-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterAction": "is not set","-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterReplacement": "is not set","-Dfilenet.pchconfig": "is not set","-Djava.awt.headless": "TRUE","-Djava.security.auth.login.config": "is not set","-Djaxws.payload.highFidelity": "is not set","-Xdump": "system:none","-Xgcpolicy": "is not set","-Xmn2048M": "is not set","-Xmn512M": "is not set","-Xquickstart": "is not set","-Xverbosegclog": "is not set","-javaagent": "D:\\AppDynamics\\AppServerAgent\\javaagent.jar","-server": "is set","": ""}}

From what you stated, the field "Attributes" should read:

{
"Attributes": ":\n-DFileNet.Content.DownloadServerAffinityEnabled: TRUE\n-DFileNet.Content.GetBlockSizeKB: 4096\n-DFileNet.Content.PutBlockSizeKB: 4096\n-DFileNet.Content.UploadServerAffinityEnabled: TRUE\n-DFileNet.WSI.AutoDetectLTPAToken: is not set\n-Dappdynamics.agent.applicationName: is not set\n-Dappdynamics.agent.nodeName: is not set\n-Dcom.filenet.authentication.token.userid: is not set\n-Dcom.filenet.authentication.wsi.AutoDetectAuthToken: is not set\n-Dcom.filenet.repositoryconfig.allowWSIOnWAS: is not set\n-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterAction: is not set\n-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterReplacement: is not set\n-Dfilenet.pchconfig: is not set\n-Djava.awt.headless: TRUE\n-Djava.security.auth.login.config: ${USER_DIR}/DocumentRepository/jaas.conf.WebSphere\n-Djaxws.payload.highFidelity: is not set\n-Xdump: system:none\n-Xgcpolicy: is not set\n-Xmn2048M: is not set\n-Xmn512M: is not set\n-Xquickstart: is set\n-Xverbosegclog: is not set\n-javaagent: is not set\n-server: is set\ngenericJvmArguments: -Xgcpolicy:gencon\n-Djaxws.payload.highFidelity=true -Dfilenet.pchconfig=${USER_DIR}\\HJIPDash\\PchConfig.properties -Xmn512M -Dcom.filenet.authentication.token.userid=sso:ltpa -DFileNet.WSI.AutoDetectLTPAToken=true -Dcom.filenet.authentication.wsi.AutoDetectAuthToken=true -Dcom.filenet.repositoryconfig.allowWSIOnWAS=true -DFileNet.Content.PutBlockSizeKB=10240 -DFileNet.Content.GetBlockSizeKB=10240",

Please affirm.

As for "traverse the Attributes field" , I want to compare the Attributes of one Application Server to another Application Server.  My results table would look something like this:

  Config Matches (y/n)? HJn6server1 (HJn6, PROD-230612151857) HJn3server1 (HJn3, PROD-230612151857)

As for my code, I am not certain of what you are asking. The dashboard I have looks as such:

The user selects an Environment/Application server pair twice, selects a Section then the dashboard shows a table (per above).

[yuanliu]

Thanks for posting sample raw data.  This settles the first and last questions I had.  Now I see that the raw events are valid JSON format.  I guess your original post copied from Splunk's events window without reverting to "Raw text".  Splunk event view is kind of convenient, but is rather difficult for others to revert to raw when copied in text format.  Based on sample event, the Attributes node is indeed a JSON key-value list of its own, therefore the Attributes.* fields.

The illustration of desired output is also critical.  Now I can make a reasonable speculation about your intent.  If I may restate in terms of JSON data, what you are asking is: given two user-selected Object nodes, compare each subnode in Attribute as row; if the subnode in question has identical values, mark the subnode as "y", otherwise "n".

There is one last confusion from the mockup of desired output: the last two columns both with header "Channel HTTP".  The column values seem to be the value of the subnode in some unspecified context.  Here, I will make two bold assumptions:

1. that the two Object nodes are selected by distinct inputs (tokens) from the dashboard you are trying to construct.  (Your screenshot seems to confirm this.)  I'll call them $object_tok1$ and $object_tok2$, respectively;
2. that identical headers "Channel HTTP" and "Channel HTTP" should actually be distinct values of Object.

If these speculations are correct, the task can be expressed as

index = myindex Object IN ($object_tok1$, $object_tok2$)
| fields - _*
| fields Object Attributes.*
| eval Object = ltrim(Object, " ")
| transpose column_name=Attribute header_field=Object
| eval match = if('$object_tok1$' == '$object_tok2$', "y", "n")

Two things to notice:

• You need to list only fields of interest before transpose.  In this case, Object and Attributes.*.
• Values of Object field contains leading white space.  Whereas SPL's single quote (') denotation usually handles special characters well, it doesn't work with leading spaces.  So I have to run trim on this field.

To help you troubleshoot, I composed an emulation for data you demonstrated, but I added a second event with Object=" HJn3server1 (HJn3, PROD-230612151857)" that has some matching attributes, some unmatching ones.

| makeresults
| eval data = mvappend("{ \"Object\":\" HJn6server1 (HJn6, PROD-230612151857)\",\"Env\":\"PROD\",\"SectionName\":\"JVM Arguments\", \"Attributes\":{\"genericJvmArguments\": \"-Djava.awt.headless=true -server -Xdump:system:none -javaagent:D:\\\\AppDynamics\\\\AppServerAgent\\\\javaagent.jar -Dappdynamics.agent.nodeName=${WAS_SERVER_NAME}\",\"-DFileNet.Content.DownloadServerAffinityEnabled\": \"TRUE\",\"-DFileNet.Content.GetBlockSizeKB\": \"4096\",\"-DFileNet.Content.PutBlockSizeKB\": \"4096\",\"-DFileNet.Content.UploadServerAffinityEnabled\": \"TRUE\",\"-DFileNet.WSI.AutoDetectLTPAToken\": \"is not set\",\"-Dappdynamics.agent.applicationName\": \"is not set\",\"-Dappdynamics.agent.nodeName\": \"${WAS_SERVER_NAME}\",\"-Dcom.filenet.authentication.token.userid\": \"is not set\",\"-Dcom.filenet.authentication.wsi.AutoDetectAuthToken\": \"is not set\",\"-Dcom.filenet.repositoryconfig.allowWSIOnWAS\": \"is not set\",\"-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterAction\": \"is not set\",\"-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterReplacement\": \"is not set\",\"-Dfilenet.pchconfig\": \"is not set\",\"-Djava.awt.headless\": \"TRUE\",\"-Djava.security.auth.login.config\": \"is not set\",\"-Djaxws.payload.highFidelity\": \"is not set\",\"-Xdump\": \"system:none\",\"-Xgcpolicy\": \"is not set\",\"-Xmn2048M\": \"is not set\",\"-Xmn512M\": \"is not set\",\"-Xquickstart\": \"is not set\",\"-Xverbosegclog\": \"is not set\",\"-javaagent\": \"D:\\\\AppDynamics\\\\AppServerAgent\\\\javaagent.jar\",\"-server\": \"is set\",\"\": \"\"}}", "{ \"Object\":\" HJn3server1 (HJn3, PROD-230612151857)\",\"Env\":\"PROD\",\"SectionName\":\"JVM Arguments\", \"Attributes\":{\"genericJvmArguments\": \"-Djava.awt.headless=true -server -Xdump:system:none -javaagent:D:\\\\AppDynamics\\\\AppServerAgent\\\\javaagent.jar -Dappdynamics.agent.nodeName=${WAS_SERVER_NAME}\",\"-DFileNet.Content.DownloadServerAffinityEnabled\": \"TRUE\",\"-DFileNet.Content.GetBlockSizeKB\": \"4096\",\"-DFileNet.Content.PutBlockSizeKB\": \"4096\",\"-DFileNet.Content.UploadServerAffinityEnabled\": \"TRUE\",\"-DFileNet.WSI.AutoDetectLTPAToken\": \"is not set\",\"-Dappdynamics.agent.applicationName\": \"myApp\",\"-Dappdynamics.agent.nodeName\": \"${WAS_SERVER_NAME}\",\"-Dcom.filenet.authentication.token.userid\": \"is not set\",\"-Dcom.filenet.authentication.wsi.AutoDetectAuthToken\": \"is not set\",\"-Dcom.filenet.repositoryconfig.allowWSIOnWAS\": \"is not set\",\"-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterAction\": \"is not set\",\"-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterReplacement\": \"is not set\",\"-Dfilenet.pchconfig\": \"is not set\",\"-Djava.awt.headless\": \"TRUE\",\"-Djava.security.auth.login.config\": \"is not set\",\"-Djaxws.payload.highFidelity\": \"is not set\",\"-Xdump\": \"system:none\",\"-Xgcpolicy\": \"M1\",\"-Xmn2048M\": \"is not set\",\"-Xmn512M\": \"is not set\",\"-Xquickstart\": \"is not set\",\"-Xverbosegclog\": \"is not set\",\"-javaagent\": \"D:\\\\AppDynamics\\\\AppServerAgent\\\\javaagent.jar\",\"-server\": \"is set\",\"\": \"\"}}")
| mvexpand data
| spath input=data
``` all above emulates
index = myindex Object IN ($object_tok1$, $object_tok2$)
```

Because '$object_tok1$' and '$object_tok2$' are interpolated at run time, I have to substitute them with fixed interpolations 'HJn6server1 (HJn6, PROD-230612151857)' and 'HJn3server1 (HJn3, PROD-230612151857)' for testing (no leading space):

| fields - _*
| fields Object Attributes.*
| eval Object = ltrim(Object, " ")
| rename Attributes.* AS *
| transpose column_name=Attribute header_field=Object
| eval match = if('HJn6server1 (HJn6, PROD-230612151857)' == 'HJn3server1 (HJn3, PROD-230612151857)', "y", "n")
```
the above line emulates the following line
| eval match = if('$object_tok1$' == '$object_tok2$', "y", "n")
```

If you run them together, they give

Attribute	HJn6server1 (HJn6, PROD-230612151857)	HJn3server1 (HJn3, PROD-230612151857)	match
-DFileNet.Content.DownloadServerAffinityEnabled	TRUE	TRUE	y
-DFileNet.Content.GetBlockSizeKB	4096	4096	y
-DFileNet.Content.PutBlockSizeKB	4096	4096	y
-DFileNet.Content.UploadServerAffinityEnabled	TRUE	TRUE	y
-DFileNet.WSI.AutoDetectLTPAToken	is not set	is not set	y
-Dappdynamics.agent.applicationName	is not set	myApp	n
-Dappdynamics.agent.nodeName	${WAS_SERVER_NAME}	${WAS_SERVER_NAME}	y
-Dcom.filenet.authentication.token.userid	is not set	is not set	y
-Dcom.filenet.authentication.wsi.AutoDetectAuthToken	is not set	is not set	y
-Dcom.filenet.repositoryconfig.allowWSIOnWAS	is not set	is not set	y
-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterAction	is not set	is not set	y
-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterReplacement	is not set	is not set	y
-Dfilenet.pchconfig	is not set	is not set	y
-Djava.awt.headless	TRUE	TRUE	y
-Djava.security.auth.login.config	is not set	is not set	y
-Djaxws.payload.highFidelity	is not set	is not set	y
-Xdump	system:none	system:none	y
-Xgcpolicy	is not set	M1	n
-Xmn2048M	is not set	is not set	y
-Xmn512M	is not set	is not set	y
-Xquickstart	is not set	is not set	y
-Xverbosegclog	is not set	is not set	y
-javaagent	D:\AppDynamics\AppServerAgent\javaagent.jar	D:\AppDynamics\AppServerAgent\javaagent.jar	y
-server	is set	is set	y
genericJvmArguments	-Djava.awt.headless=true -server -Xdump:system:none -javaagent:D:\AppDynamics\AppServerAgent\javaagent.jar -Dappdynamics.agent.nodeName=${WAS_SERVER_NAME}	-Djava.awt.headless=true -server -Xdump:system:none -javaagent:D:\AppDynamics\AppServerAgent\javaagent.jar -Dappdynamics.agent.nodeName=${WAS_SERVER_NAME}	y

Also note that the illustrated data do not contain attributes such as enableLogging or keepAlive, the above demonstration will not include them.  Still, you see two attributes that are non-matching, and the rest are matching.

[gsmith61293]

That seemed to do it. Thank you very much!  Here is the code and results:

index = websphere_cct Object IN (" HJn8server3 (HJn8, UAT-230612150440)"," HJn5server1 (HJn5, PROD-230612151857)") SectionName
| fields - _*
| fields Object Attributes.*
| eval Object = ltrim(Object, " ")
| transpose column_name=Attribute header_field=Object
| eval match = if('HJn8server3 (HJn8, UAT-230612150440)' == 'HJn5server1 (HJn5, PROD-230612151857)', "y", "n")

However, I am confused about how this code works.  Specifically, what the code "| fields - _*"   and "| eval match = if('HJn8server3 (HJn8, UAT-230612150440)' == 'HJn5server1 (HJn5, PROD-230612151857)', "y", "n")"  does?

As I understand it,  "| fields - _*"  just eliminates the fields with an underscore?

if('HJn8server3 (HJn8, UAT-230612150440)' == 'HJn5server1 (HJn5, PROD-230612151857)', "y", "n")"  I would think compares two "Object" fields.

Can you please explain?  

Thank you. 

[yuanliu · Accepted solution]

---

As I understand it,  "| fields - _*"  just eliminates the fields with an underscore?

Absolutely correct.  Because transpose is used, having extraneous fields are rather distracting.  You can also use table to list only fields that interest you.

if('HJn8server3 (HJn8, UAT-230612150440)' == 'HJn5server1 (HJn5, PROD-230612151857)', "y", "n")"  I would think compares two "Object" fields.

---

That is because after transpose header_field=Object, the values of original Object field become column names, i.e., new field names.