How to use ServiceNow data to link back to other Splunk data?

[thefuzz4]

So we have our Splunk instance ingesting everything from ServiceNow. I have some dashboards that track when a service such as tomcat/weblogic etc are shut down. Now I want to be able to accomplish the following things:

I need to be able to link the log data back to the ServiceNow data for that event to show tickets that were created as a result of this, as well as show how much time was spent to fix this problem.

I wrote this search

index=* NOT sourcetype=wls9_managedserver NOT sourcetype=weblogic_access host=*prod* CASE("Server state changed to FAILED") OR CASE("Fatal Error at WriteHandler") OR CASE("Server state changed to SHUTDOWN") OR CASE("Destroying ProtocolHandler") OR CASE("Server startup in*")  sourcetype=weblogic_domain_out
| join type=inner max=0 domain
[
search index=app eventtype=snow_incident domain]

I replaced the actual domain name with domain 🙂

So I'm hoping to be able to get this rolling and enhance my splunkfu in the process. Thank you all in advance for your help with this.

[koshyk]

hi,
Hope you are familiar with Common Information Model (CIM) ? If yes, please download "Service Now Addon" . This will normalise all SNOW fields to CIM standards and will map to "Ticket Management" model. (All_ticket_management -> dest would be your affected CI for example) Also you can use "ServiceNow" cmdb_ci for asset management. Would strongly advise once your environment grows to use CIM standards for all logs if possible and mapping will be a quite easy task

[thefuzz4]

Yes we are using the addon to pump all of the data from service now into splunk already. So it is all CIM compliant. Thanks.

[koshyk]

Cool. I'm not that familiar with web logic, but in websphere add-on you can use "dest" to match "dest" of service now.

[thefuzz4]

So yeah how do I match one portion of a search to another portion? The other thing I don't understand is that the event_type from service now only seems to work inside of the service now app inside of splunk.

[koshyk]

You can use join, something like..

index= xyz sourcetype=abc | join type=left dest [ search index=snow index dest=* | dedup dest ]

[thefuzz4]

Excellent thank you @koshyk Going to give it a whirl now.

[thefuzz4]

Ok so I apologize about my delay in getting back to you been tied up with other issues.

So here is what I have for a search but its only returning from one index and not both.

index=weblogic sourcetype=weblogic_out  | join type=left host [ search index=app eventtype=snow_incident short_description=* | dedup host ]

trying to join on the host name because the host name is in the short_description field that is returned from snow

[koshyk]

You r putting only one index. You could put index=web logic OR index=another index

[Jeremiah]

Are you asking for help with improving the search? Maybe you can give an example of each type of event you are planning to correlate.

[thefuzz4]

Yeah I guess I'm trying to figure out how to match say the hostname to the service now ticket description. Thats where I'm hung up with this I was hoping that someone else has used servicenow and splunk events together. Sorry for the delayed response, was working on other items on Fri and then ya know weekend :). Thank you Jeremiah for your help with this.

So like

From splunk event
host=host-prod

From servicenow event in splunk
hort_description="host-prod"

Let me know if that makes sense or what other information you'll need. Thanks.