Knowledge Management

Workflow - Anchor secondary search time ranges on event time

nclarkau
Path Finder

I have created a workflow through the GUI (the corresponding workflow_actions.conf is below).

The intention is to provuide the user with a similar action to "show source" which instead displays the application transaction that the event belongs to.

We have a macro search that can perform this search using two parameters (thread and host). And the workflow action triggers this search correctly.

The problem we have is narrowing the secondary search so only the transaction in question is displayed (at least a few extraneous are shown as possible). If the users original search is over 24 hours and they click on "Show transaction" then they get every transaction for 24 hours that matches the host and thread, which can be a very large number. Specifying a time range in the workflow setup does not work as it is relative to now.

Can we make the time range relative to the event that the workflow is triggered from? Or is there another way we could solve this?

[view_myapplication_transaction]
display_location = event_menu
eventtypes = myapplication_log_event
fields = *
label = Show transaction
search.app = MYAPP
search.earliest = -5m
search.latest = +5m
search.preserve_timerange = 0
search.search_string = `full_myapplication_transaction(thread=$thread$,host=$host$)`
search.target = blank
type = search
Tags (1)
1 Solution

nclarkau
Path Finder

Based on the answer from nick it seems that for now (4.1.3) there is no straight forward way to do this.

The workaround we found was to use nick's suggestion of adding a now= clause to the search string in the workflow action. This would be perfect if the format of the _time field was not being changed somewhere in the workflow action code. Since now can only take epochtime format the change of format (away from epochtime) was breaking the solution.

So the extra trick was to add an additional search command to some of the savedsearches/macros that we commonly use.

| convert mktime(_time) as start_time

The downside is that now the action is only available to events generated by these searches/macros. A limitation we hope can be fixed by some improvments to the workflow actions.

The resulting search string is

sourcetype=myapp* host=$host$ thread=$thread$ now=$start_time$ earliest=-10min latest=+10m | `make_myapplication_transactions`

The workflow_actions.conf entry is:

[view_myapplication_transaction]
display_location = event_menu
fields = transaction_time
label = Show transaction
search.app = MYAPP
search.preserve_timerange = 0
search.search_string = sourcetype=myapp* host=$host$ thread=$thread$ now=$start_time$ earliest=-10min latest=+10m | `make_myapplication_transactions`
search.target = blank
type = search

Notice that the earliest/latest modifiers are in the search string.

View solution in original post

rayaivy
Explorer

Any thoughts on this now?

0 Karma

Anam
Community Manager
Community Manager

Hi rayaivy

This question was posted almost 8 years ago and the answer was accepted. If you have a question related to this and the currently accepted answer is not helping you solve your problem, please post a new question. It will give you maximum exposure and help.

Thanks

0 Karma

nclarkau
Path Finder

Based on the answer from nick it seems that for now (4.1.3) there is no straight forward way to do this.

The workaround we found was to use nick's suggestion of adding a now= clause to the search string in the workflow action. This would be perfect if the format of the _time field was not being changed somewhere in the workflow action code. Since now can only take epochtime format the change of format (away from epochtime) was breaking the solution.

So the extra trick was to add an additional search command to some of the savedsearches/macros that we commonly use.

| convert mktime(_time) as start_time

The downside is that now the action is only available to events generated by these searches/macros. A limitation we hope can be fixed by some improvments to the workflow actions.

The resulting search string is

sourcetype=myapp* host=$host$ thread=$thread$ now=$start_time$ earliest=-10min latest=+10m | `make_myapplication_transactions`

The workflow_actions.conf entry is:

[view_myapplication_transaction]
display_location = event_menu
fields = transaction_time
label = Show transaction
search.app = MYAPP
search.preserve_timerange = 0
search.search_string = sourcetype=myapp* host=$host$ thread=$thread$ now=$start_time$ earliest=-10min latest=+10m | `make_myapplication_transactions`
search.target = blank
type = search

Notice that the earliest/latest modifiers are in the search string.

gkanapathy
Splunk Employee
Splunk Employee

You can take the time out of a subsearch and apply it to a main search like this:

"main search term" [ search "subsearch term" | eval earliest=_time-2 | eval latest=_time+2 | fields earliest,latest | format "(" "(" "" ")" "OR" ")" ]

The format is required because Splunk search queries won't accept AND between time modifiers the way it will between other search terms. Personally, I think it's a bug. Anyway, since AND is implicit, we can just remove it from the format default.

Not sure if this helps you, but if you can get _time out of the event in question, you might be able to use the above.

nclarkau
Path Finder

see my comments to nick's answer for further details. i managed to get this working as the search string... sourcetype=myapp* host=$host$ thread=$thread$ now=$transaction_time$ earliest=-10min latest=+10m | make_myapplication_transactions... the transaction_time field was added to a macro as per my other comments below to force the epoch time into an independent field. The workflow action is now only applied to events that contain the transaction_time field.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

i.e., ... | eval tm=strptime("_$time$","%Y-%m-%dT%H:%M:%S%Z") | eval earliest=tm-1 | ...

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

this only happens when it is displayed in events or results, or at least it should only happen when it is displayed. _time is stored internally and usually passed around in epoch format. how exactly are you running the subsearch, or how are you getting the results to pass up. Note that if you have no other choice, you can use the strptime() eval function or the convert command's mktime option.

0 Karma

nclarkau
Path Finder

Everything seems to be coming together except using _time. The problem is that _time is being converted automatically when substituted. So instead of being in an epoch format it is in a date format... 2010-07-02T10:13:46+1000

0 Karma

sideview
SplunkTrust
SplunkTrust

You need to use the now term. now= tells splunkd to interpret any relative time range arguments as though the current time was the specified epochtime.

Ideally you could do search.now=$_time$ and it would work as an API argument. However Im pretty sure that will not work.

So you'll have to put the now="$_time$" into the search_string itself. (make sure it ends up in the initial search clause)

The other problem is that unfortunately splunkd doesnt take the time terms sent to the API and merge them with the time terms specified in the search language. Instead if it sees anything in the search itself, it'll ignore the official API args entirely. So to workaround that problem you'll have to take your search.earliest and search.latest OUT of the config and instead put something like earliest="-10min" latest="+10min" into the search_string itself.

NOTE: Specifying time terms in the search string is normally not recommended cause a) its messy, b) it results in the UI nagging you about how 'your timerange was substituted using values in the search string'. However this is one case where I think it's the only way.

sideview
SplunkTrust
SplunkTrust

From an outside developer's perspective its definitely a bug. Go for it. Since you cant get the epochTime value it makes it really hard to use $_time$ in workflow actions in any meaningful way.

0 Karma

nclarkau
Path Finder

could this be considered a fault in workflow actions? if it is then perhaps a defect can be raised. otherwise a enhancement request to ensure workflow actions have a means to set now to the time of the event.

0 Karma

nclarkau
Path Finder

I think that is what is happening. As a workaround I added a convert statement to a macro that is commonly used. "| convert mktime(_time) as transaction_time". so instead of using _time i use time in the workflow action. the downside is that the action is only available to events generated from the macro that has been enhanced instead of all events of a particulat eventtype.

0 Karma

sideview
SplunkTrust
SplunkTrust

_time is always in epochtime format although in the UI it might appear that it's a string-formatted value. As for the error message, that sounds (very unfortunately) like the workflow actions code for some reason uses the converted ISO value instead of the raw epochTime value like it should.

0 Karma

nclarkau
Path Finder

I think I need to convert the _time format as I am getting this error... "Invalid value "2010-07-02T14:32:33+1000" for time term 'now'". The problem is how do I get _time in epoch seconds in the initial search clause?

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...