Help with Custom Search Please

[vwilson3]

I would really appreciate any help as I am not very experienced with SPL. I am learning every day, literally.

I need to find a way to show the host name and the tag::host the event occurred on that tripped the alert, the date/time, Alert Manager (AM) incident_id, alert name, Splunk/AM user, AM action, AM comment.

Here is what I have so far... there may be better ways to do this... and this doesn't show the host and host tag for the system where the event occurred. I have tried several different things, but had no success getting the host names.

sourcetype=alert_metadata
| join incident_id 
    [search sourcetype=incident_change]
| table _time incident_id alert user action comment

Any suggestions? Any assistance is greatly appreciated.

[vwilson3]

In my haste to post my question, I forgot to say that the events that tripped the alert are in a different index and are different sourcetype. I was thinking I could try using the time field as a common field to try to find the host, but this particular alert is scheduled to run every hour, so the time it shows in AM is not the time of the actual event that tripped the alert. Still stumped.

[nickhills]

Are you able to share some of the event data for both sourcetypes?
If at all possible you want to avoid 'join', and seeing the format of the data may help identify a more efficient approach.

[vwilson3]

I had to make it as generic as possible. I hope it is still helpful.

sourcetype alert_metadata:
{"alert": "myalertname", "alert_time": "2019-03-19T17:15:11.892+00:00", "app": "search", "entry": [{"content": {"earliestTime": "2019-03-19T16:15:00.000+00:00", "eventCount": 1, "eventSearch": "search (myalertsearchstring)", "keywords": "index::myappindex myapp_msgarea::RRR myapp_msgname::SSS", "latestTime": "2019-03-19T17:15:00.000+00:00" remote_mysplunkserver_scheduler_wilsonvsearchRMD54ae8a1459b8e343f_at_1553015700_10843. "optimizedSearch": "| search (mysearchstring", "resultCount": 1, "searchEarliestTime": 1553012100, "searchLatestTime": 1553015700, "searchProviders": ["mySplunkservers]}, "links": {"alternate": "/services/search/jobs/schedulerwilsonvsearchRMD54ae8a1459b8e343f_at_1553015700_10843"}, "name": "search mysearchstring}], "impact": "high", "incident_id": "23464663-4af1-4073-bf84-6edebfb6c837", "job_id": "schedulerwilsonvsearch_RMD54ae8a1459b8e343f_at_1553015700_10843", "name": "myalertname", "owner": "unassigned", "priority": "critical", "result_id": "0", "title": "myalertname", "ttl": 86400, "urgency": "high"}

sourcetype incident_change:
time=2019-03-19T17:15:12.245662 event_id="21794f2b4d134d36359eca1115a478b7" severity=INFO origin="alert_handler" user="splunk-system-user" action="create" alert="myalertname" incident_id="23464663-4af1-4073-bf84-6edebfb6c837" job_id="scheduler_wilsonvsearch_RMD54ae8a1459b8e343f_at_1553015700_10843" result_id="0" owner="unassigned" status="new" urgency="high" ttl="86400" alert_time="2019-03-19T17:15:11.892+00:00"

example of the actual event that tripped the alert:
2019-03-19 16:44:43 myapp_logversion="2" myapp_msgarea="AU" myapp_msgname="L" myapp_pid="37301" myapp_taskno="00031" myapp_slgttyp="CC" myapp_termname="myterm" myapp_username="userhere"

Splunk fields: host, index, sourcetype, source, tag::host

I'm trying to figure out how to get the host name where the event occurred, based on the info in Alert Manager (the top two examples).

[shaileshmali]

try

sourcetype=alert_metadata | fields incident_id ( select fields in main search incident_id date/time, Alert Manager (AM) incident_id, alert name, Splunk/AM user, AM action, AM comment)
| join incident_id
[search sourcetype=incident_change | table incident_id ( select additional fields in subsearch ) ]
| table _time incident_id alert user action comment

[vwilson3]

Thank you for the quick response. I will keep this info for a different report.

In my haste to post my question, I forgot to say that the events that tripped the alert are in a different index and are different sourcetype. I was thinking I could try using the time field as a common field to try to find the host, but this particular alert is scheduled to run every hour, so the time it shows in AM is not the time of the actual event that tripped the alert. Still stumped.