Splunk Search

Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

Lucas_Henry_
New Member

Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

edit: here's what I'm trying to do
ie, "eval myField=( "value1", "value2", "value3") | stats count by myField"

Where "value1", "value2", "value3" are literal strings.

I want to get a count for how many "value1"s, "value2"s, and "value3"s there are

0 Karma
1 Solution

woodcock
Esteemed Legend

OK, assuming that the problem is that a field called errorMsg does not exist, try this:

index=its_akana* source="/apps/logs/*" host=ent5*ll5app ("at the below stack trace. Not closed in the same method" OR
"Cannot get a connection, pool exhausted" OR
"com.digev.fw.exception.GException: Execution of a DB command failed" OR
"com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERROR" OR
"com.mongodb.MongoSocketOpenException" OR
"com.mongodb.MongoTimeoutException" OR
"Data truncation" OR
"ERROR [DBStatementAndResultSetTracker] PreparedStatementTracker" OR
"Error encountered in WS-Security engine" OR
"Error in creating Prepared statement for the query" OR
"federation member auth token cannot be refreshed" OR
"GC overhead limit exceeded" OR
"Illegal character" OR
"java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer" OR
"java.lang.StackOverflowError" OR
"Log block not closed correctly. Enable log block tracking to see diagnostic information" OR
"Log frame is closed at the below stack trace" OR
"No corresponding startTraceBlock() is seen" OR
"No key found in WSDL for service" OR
"No process found" OR
"No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error" OR
"OutOfMemoryError" OR
"Timeout waiting for idle object" OR
"Unable to initialize SiteMinder agent" OR
"UsageJDBCWriter.writeUsage" OR
"Wsdl does not conform to wsdl schema" OR
"org.elasticsearch.action.UnavailableShardsException" OR
"None of the configured nodes are available") 

| rename COMMENT AS "You can get rid of the following line if you ever get the field extraction working"
| rex "(?<errorMsg>at the below stack trace. Not closed in the same method|Cannot get a connection, pool exhausted|com.digev.fw.exception.GException: Execution of a DB command failed|com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERROR|com.mongodb.MongoSocketOpenException|com.mongodb.MongoTimeoutException|Data truncation|ERROR [DBStatementAndResultSetTracker] PreparedStatementTracker|Error encountered in WS-Security engine|Error in creating Prepared statement for the query|federation member auth token cannot be refreshed|GC overhead limit exceeded|Illegal character|java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer|java.lang.StackOverflowError|Log block not closed correctly. Enable log block tracking to see diagnostic information|Log frame is closed at the below stack trace|No corresponding startTraceBlock() is seen|No key found in WSDL for service|No process found|No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error|OutOfMemoryError|Timeout waiting for idle object|Unable to initialize SiteMinder agent|UsageJDBCWriter.writeUsage|Wsdl does not conform to wsdl schema|org.elasticsearch.action.UnavailableShardsException|None of the configured nodes are available)"

| chart count BY host errorMsg

View solution in original post

0 Karma

woodcock
Esteemed Legend

OK, assuming that the problem is that a field called errorMsg does not exist, try this:

index=its_akana* source="/apps/logs/*" host=ent5*ll5app ("at the below stack trace. Not closed in the same method" OR
"Cannot get a connection, pool exhausted" OR
"com.digev.fw.exception.GException: Execution of a DB command failed" OR
"com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERROR" OR
"com.mongodb.MongoSocketOpenException" OR
"com.mongodb.MongoTimeoutException" OR
"Data truncation" OR
"ERROR [DBStatementAndResultSetTracker] PreparedStatementTracker" OR
"Error encountered in WS-Security engine" OR
"Error in creating Prepared statement for the query" OR
"federation member auth token cannot be refreshed" OR
"GC overhead limit exceeded" OR
"Illegal character" OR
"java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer" OR
"java.lang.StackOverflowError" OR
"Log block not closed correctly. Enable log block tracking to see diagnostic information" OR
"Log frame is closed at the below stack trace" OR
"No corresponding startTraceBlock() is seen" OR
"No key found in WSDL for service" OR
"No process found" OR
"No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error" OR
"OutOfMemoryError" OR
"Timeout waiting for idle object" OR
"Unable to initialize SiteMinder agent" OR
"UsageJDBCWriter.writeUsage" OR
"Wsdl does not conform to wsdl schema" OR
"org.elasticsearch.action.UnavailableShardsException" OR
"None of the configured nodes are available") 

| rename COMMENT AS "You can get rid of the following line if you ever get the field extraction working"
| rex "(?<errorMsg>at the below stack trace. Not closed in the same method|Cannot get a connection, pool exhausted|com.digev.fw.exception.GException: Execution of a DB command failed|com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERROR|com.mongodb.MongoSocketOpenException|com.mongodb.MongoTimeoutException|Data truncation|ERROR [DBStatementAndResultSetTracker] PreparedStatementTracker|Error encountered in WS-Security engine|Error in creating Prepared statement for the query|federation member auth token cannot be refreshed|GC overhead limit exceeded|Illegal character|java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer|java.lang.StackOverflowError|Log block not closed correctly. Enable log block tracking to see diagnostic information|Log frame is closed at the below stack trace|No corresponding startTraceBlock() is seen|No key found in WSDL for service|No process found|No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error|OutOfMemoryError|Timeout waiting for idle object|Unable to initialize SiteMinder agent|UsageJDBCWriter.writeUsage|Wsdl does not conform to wsdl schema|org.elasticsearch.action.UnavailableShardsException|None of the configured nodes are available)"

| chart count BY host errorMsg
0 Karma

Lucas_Henry_
New Member

You are the man. Thank you for all your help.

0 Karma

woodcock
Esteemed Legend

Why does this not work?

index=its_akana* source="/apps/logs/*" host=ent5*ll5app ("at the below stack trace. Not closed in the same method" OR
"Cannot get a connection, pool exhausted" OR
"com.digev.fw.exception.GException: Execution of a DB command failed" OR
"com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERROR" OR
"com.mongodb.MongoSocketOpenException" OR
"com.mongodb.MongoTimeoutException" OR
"Data truncation" OR
"ERROR [DBStatementAndResultSetTracker] PreparedStatementTracker" OR
"Error encountered in WS-Security engine" OR
"Error in creating Prepared statement for the query" OR
"federation member auth token cannot be refreshed" OR
"GC overhead limit exceeded" OR
"Illegal character" OR
"java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer" OR
"java.lang.StackOverflowError" OR
"Log block not closed correctly. Enable log block tracking to see diagnostic information" OR
"Log frame is closed at the below stack trace" OR
"No corresponding startTraceBlock() is seen" OR
"No key found in WSDL for service" OR
"No process found" OR
"No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error" OR
"OutOfMemoryError" OR
"Timeout waiting for idle object" OR
"Unable to initialize SiteMinder agent" OR
"UsageJDBCWriter.writeUsage" OR
"Wsdl does not conform to wsdl schema" OR
"org.elasticsearch.action.UnavailableShardsException" OR
"None of the configured nodes are available"
| stats count by errorMsg
0 Karma

Lucas_Henry_
New Member

Because errorMsg isn't a native field in the data. It's one I have to create

0 Karma

woodcock
Esteemed Legend

Then my other solution ABSOLUTELY POSITIVELY should work (the one that is now the bottom one in the pair of the other answer).

0 Karma

woodcock
Esteemed Legend

Assuming that you are just matching strings in the raw events (the strings are not accessed by a field name), then like this:

Your Base Search Here | stats 
[| makeresults 
 | eval errorMsg="value1::value2::INFO" 
 | makemv delim="::" errorMsg
 | format "" "" "" "" "" "" 
 | rex field=search mode=sed "s/\( errorMsg=| OR errorMsg=/ count(eval(searchmatch(/g s/\" count\(/\")) count(/g s/\s*\)  $/))/ s/\"([^\"]+)\"\)\)/\"\1\"))) AS \"\1\"/g"]

If you do indeed have field names that contain these strings, then like this (you might have to change the field name errorMsg to your field name):

Your Base Search Here | stats 
[| makeresults 
| eval errorMsg="value1::value2::value2" 
| makemv delim="::" errorMsg 
| rex field=errorMsg mode=sed "s/[\r\n]//g"
| format "" "" "" "" "" "" 
| rex field=search mode=sed "s/\(| OR / count(eval(/g s/\" count\(/\")) count(/g s/\s*\)  $/))/ s/\"([^\"]+)\"\)\)/\"\1\")) AS \"\1\"/g"]

You can make this into a macro.

0 Karma

Lucas_Henry_
New Member

Now it throws the following error: Error in 'stats' command: You must specify a rename for the aggregation specifier on the dynamically evaluated field 'count(eval( errorMsg="at the below stack trace. Not closed in the same method"))'.

0 Karma

woodcock
Esteemed Legend

Try replacing the last line with this:

| rex field=search mode=sed "s/\(| OR / count(eval(/g s/\" count\(/\")) count(/g s/\s*\)  $/))/ s/\"(\w+)\"\)\)/\"\1\")) AS \1/g"
0 Karma

Lucas_Henry_
New Member

This is the full search:

index=its_akana* source="/apps/logs/*" host=ent5*ll5app ("at the below stack trace. Not closed in the same method" OR
"Cannot get a connection, pool exhausted" OR
"com.digev.fw.exception.GException: Execution of a DB command failed" OR
"com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERROR" OR
"com.mongodb.MongoSocketOpenException" OR
"com.mongodb.MongoTimeoutException" OR
"Data truncation" OR
"ERROR [DBStatementAndResultSetTracker] PreparedStatementTracker" OR
"Error encountered in WS-Security engine" OR
"Error in creating Prepared statement for the query" OR
"federation member auth token cannot be refreshed" OR
"GC overhead limit exceeded" OR
"Illegal character" OR
"java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer" OR
"java.lang.StackOverflowError" OR
"Log block not closed correctly. Enable log block tracking to see diagnostic information" OR
"Log frame is closed at the below stack trace" OR
"No corresponding startTraceBlock() is seen" OR
"No key found in WSDL for service" OR
"No process found" OR
"No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error" OR
"OutOfMemoryError" OR
"Timeout waiting for idle object" OR
"Unable to initialize SiteMinder agent" OR
"UsageJDBCWriter.writeUsage" OR
"Wsdl does not conform to wsdl schema" OR
"org.elasticsearch.action.UnavailableShardsException" OR
"None of the configured nodes are available" 
) | stats 
 [| makeresults | eval errorMsg="at the below stack trace. Not closed in the same method::
com.digev.fw.exception.GException: Execution of a DB command failed::
com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERR::
com.mongodb.MongoSocketOpenException::
com.mongodb.MongoTimeoutException::
Data truncation::
ERR [DBStatementAndResultSetTracker] PreparedStatementTracker::
Error encountered in WS-Security engine::
Error in creating Prepared statement for the query::
federation member auth token cannot be refreshed::
GC overhead limit exceeded::
Illegal character::
java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer::
java.lang.StackOverflowError::
Log block not closed correctly. Enable log block tracking to see diagnostic information::
Log frame is closed at the below stack trace::
No corresponding startTraceBlock() is seen::
No key found in WSDL for service::
No process found::
No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error::
OutOfMemoryError::
Timeout waiting for idle object::
Unable to initialize SiteMinder agent::
UsageJDBCWriter.writeUsage::
Wsdl does not conform to wsdl schema::
org.elasticsearch.action.UnavailableShardsException::
None of the configured nodes are available::
Cannot get a connection, pool exhausted" | makemv delim="::" errorMsg
 | format "" "" "" "" "" "" | rex field=search mode=sed "s/\(| OR / count(eval(/g s/\" count\(/\")) count(/g s/\s*\)  $/))/ s/\"(\w+)\"\)\)/\"\1\")) AS \1/g"]
0 Karma

woodcock
Esteemed Legend

Now we are talking. The missing piece was that your values have spaces. See my updated answer; it is tested and working but the field names are TERRIBLE.

0 Karma

Lucas_Henry_
New Member

What I'm trying to get is a count of how many times each string appears per unit time. That doesn't seem to be happening when I run the amended search:

index=its_akana* source="/apps/logs/*" host=ent5*ll5app ("at the below stack trace. Not closed in the same method" OR
"Cannot get a connection, pool exhausted" OR
"com.digev.fw.exception.GException: Execution of a DB command failed" OR
"com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERROR" OR
"com.mongodb.MongoSocketOpenException" OR
"com.mongodb.MongoTimeoutException" OR
"Data truncation" OR
"ERROR [DBStatementAndResultSetTracker] PreparedStatementTracker" OR
"Error encountered in WS-Security engine" OR
"Error in creating Prepared statement for the query" OR
"federation member auth token cannot be refreshed" OR
"GC overhead limit exceeded" OR
"Illegal character" OR
"java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer" OR
"java.lang.StackOverflowError" OR
"Log block not closed correctly. Enable log block tracking to see diagnostic information" OR
"Log frame is closed at the below stack trace" OR
"No corresponding startTraceBlock() is seen" OR
"No key found in WSDL for service" OR
"No process found" OR
"No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error" OR
"OutOfMemoryError" OR
"Timeout waiting for idle object" OR
"Unable to initialize SiteMinder agent" OR
"UsageJDBCWriter.writeUsage" OR
"Wsdl does not conform to wsdl schema" OR
"org.elasticsearch.action.UnavailableShardsException" OR
"None of the configured nodes are available" 
) | stats 
 [| makeresults | eval errorMsg="at the below stack trace. Not closed in the same method::
com.digev.fw.exception.GException: Execution of a DB command failed::
com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERR::
com.mongodb.MongoSocketOpenException::
com.mongodb.MongoTimeoutException::
Data truncation::
ERR [DBStatementAndResultSetTracker] PreparedStatementTracker::
Error encountered in WS-Security engine::
Error in creating Prepared statement for the query::
federation member auth token cannot be refreshed::
GC overhead limit exceeded::
Illegal character::
java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer::
java.lang.StackOverflowError::
Log block not closed correctly. Enable log block tracking to see diagnostic information::
Log frame is closed at the below stack trace::
No corresponding startTraceBlock() is seen::
No key found in WSDL for service::
No process found::
No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error::
OutOfMemoryError::
Timeout waiting for idle object::
Unable to initialize SiteMinder agent::
UsageJDBCWriter.writeUsage::
Wsdl does not conform to wsdl schema::
org.elasticsearch.action.UnavailableShardsException::
None of the configured nodes are available::
Cannot get a connection, pool exhausted" | makemv delim="::" errorMsg
 | rex field=errorMsg mode=sed "s/[\r\n]//g"| format "" "" "" "" "" "" | rex field=search mode=sed "s/\(| OR / count(eval(/g s/\" count\(/\")) count(/g s/\s*\)  $/))/ s/\"([^\"]+)\"\)\)/\"\1\")) AS \"\1\"/g"] | stats count by errorMsg
0 Karma

woodcock
Esteemed Legend

Drop this part:

| stats count by errorMsg
0 Karma

Lucas_Henry_
New Member

That produces a table, which doesn't seem to be picking up the counts of the strings. Each string in the table says 0, but that's not accurate

0 Karma

woodcock
Esteemed Legend

OK, try the updated answer (the top one of the 2).

0 Karma

Lucas_Henry_
New Member

Making progress. The search produces a table with counts for the frequency of each literal string, but the search itself does not seem to produce the errorMsg field itself when searching in Verbose mode. I would like to be able to produce the errorMsg field so I can add it to tables, or grab the errorMsg value for alerts, etc

The full search query is below

index=its_akana* source="/apps/logs/*" host=ent5*ll5app ("at the below stack trace. Not closed in the same method" OR
"Cannot get a connection, pool exhausted" OR
"com.digev.fw.exception.GException: Execution of a DB command failed" OR
"com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERROR" OR
"com.mongodb.MongoSocketOpenException" OR
"com.mongodb.MongoTimeoutException" OR
"Data truncation" OR
"ERROR [DBStatementAndResultSetTracker] PreparedStatementTracker" OR
"Error encountered in WS-Security engine" OR
"Error in creating Prepared statement for the query" OR
"federation member auth token cannot be refreshed" OR
"GC overhead limit exceeded" OR
"Illegal character" OR
"java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer" OR
"java.lang.StackOverflowError" OR
"Log block not closed correctly. Enable log block tracking to see diagnostic information" OR
"Log frame is closed at the below stack trace" OR
"No corresponding startTraceBlock() is seen" OR
"No key found in WSDL for service" OR
"No process found" OR
"No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error" OR
"OutOfMemoryError" OR
"Timeout waiting for idle object" OR
"Unable to initialize SiteMinder agent" OR
"UsageJDBCWriter.writeUsage" OR
"Wsdl does not conform to wsdl schema" OR
"org.elasticsearch.action.UnavailableShardsException" OR
"None of the configured nodes are available" 
) | stats
 [| makeresults | eval errorMsg="at the below stack trace. Not closed in the same method::
com.digev.fw.exception.GException: Execution of a DB command failed::
com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERR::
com.mongodb.MongoSocketOpenException::
com.mongodb.MongoTimeoutException::
Data truncation::
ERR [DBStatementAndResultSetTracker] PreparedStatementTracker::
Error encountered in WS-Security engine::
Error in creating Prepared statement for the query::
federation member auth token cannot be refreshed::
GC overhead limit exceeded::
Illegal character::
java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer::
java.lang.StackOverflowError::
Log block not closed correctly. Enable log block tracking to see diagnostic information::
Log frame is closed at the below stack trace::
No corresponding startTraceBlock() is seen::
No key found in WSDL for service::
No process found::
No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error::
OutOfMemoryError::
Timeout waiting for idle object::
Unable to initialize SiteMinder agent::
UsageJDBCWriter.writeUsage::
Wsdl does not conform to wsdl schema::
org.elasticsearch.action.UnavailableShardsException::
None of the configured nodes are available::
Cannot get a connection, pool exhausted" | makemv delim="::" errorMsg
  | format "" "" "" "" "" "" 
  | rex field=search mode=sed "s/\( errorMsg=| OR errorMsg=/ count(eval(searchmatch(/g s/\" count\(/\")) count(/g s/\s*\)  $/))/ s/\"([^\"]+)\"\)\)/\"\1\"))) AS \"\1\"/g"]
0 Karma

woodcock
Esteemed Legend

Yes, the main problem for you is that the field errorMsg does not exist. If you post a single sample event, then we can help you with that part.

0 Karma

Lucas_Henry_
New Member

2017-03-02 15:14:55,923 ERROR [ScheduleJobThread-1] ManagedEndpointsListener - Start [ManagedEndpointListener.addReference]
[0] Start [HttpClientTransport.accept()]
[1] issueHttpRequest [GET][https://pmruntime-lab.svc.ny.gov:443/rest/services/uddi:f8d50d0b-3143-11e5-8ec9-9133ad19fc08/wsdl?In...]
[1] Message does not have transport security subject associated with it
[206] Request [GET https://pmruntime-lab.svc.ny.gov:443/rest/services/uddi:f8d50d0b-3143-11e5-8ec9-9133ad19fc08/wsdl?In... HTTP/1.1] completed in [205] mS, status [HTTP/1.1 200 OK]
[206] End [HttpClientTransport.accept()]
[209] Releasing client connection [https://pmruntime-lab.svc.ny.gov:443/rest/services/uddi:f8d50d0b-3143-11e5-8ec9-9133ad19fc08/wsdl?In...]
[210] Updating WSDL reference [/rest/services/uddi:f8d50d0b-3143-11e5-8ec9-9133ad19fc08/wsdl]
[210] Sending add notification to [com.soa.mp.dos.ServiceMapper@47bef41c]
[210] ServiceMapper.registrationAdded()
[210] No key found in WSDL for service {http://service.virusscan.soa.coe.its.ny.gov}VirusScan1.0, DoS rules will not be found
[210] Sending add notification to [com.soa.process.bpel.configuration.VSCapabilityConfigurationListener@48ce5035]
[210] VSCapabilityCOnfigurationListener.registrationAdded(/rest/services/uddi:f8d50d0b-3143-11e5-8ec9-9133ad19fc08/wsdl)
[210] {http://service.virusscan.soa.coe.its.ny.gov}VirusScan1.0 was registered with VS engine but no VS config found
[210] Sending add notification to [com.soa.service.identity.runtime.ServiceSubjectMasterBuilder@41be673f]
[233] Sending add notification to [com.soa.jbi.component.wsf.WSFSE@2656eed0]
[234] End [ManagedEndpointListener.addReference]

0 Karma

Lucas_Henry_
New Member

Unfortunately, the strings I'm concerned about appear in different places in each event

0 Karma

Lucas_Henry_
New Member

Still throwing the same error

0 Karma

woodcock
Esteemed Legend

Answer updated to account for spacing variances in format command.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...