Splunk Search

changing `host` and persisting the result

mushkevych
Explorer

I am trying to make this query work:
index="main" | eval host=asset_id | collect index="scanned_app"
where asset_id is a field, not a static value.

Two observations regarding the query:
- without | collect ..., the search shows data as i expect it - with the meta-field host changed
- with | collect ..., the resulting index carries host unchanged from the main index

Q: how do i change the host, so that it can be persisted in another index ?
index="main" | eval *magic_here* | collect index="scanned_app"

Tags (3)
0 Karma

jbjerke_splunk
Splunk Employee
Splunk Employee

You can try and wrap you search in the map command that dynamically let's you generate another search.

This generates an event in the summary index with host=hello set from the outer search.

|makeresults count=1
| eval asset_id="hello"
| map search="
search
index=\"main\"
| collect index=scanned_app host=$asset_id$
"

j

0 Karma

pwild_splunk
Splunk Employee
Splunk Employee

This will work. Remember that the map command, by default, is limited to only 10 sub-search iterations. Use the option maxsearches=10000 or something more appropriate for your data set.

Converting the above to your actual search, see below. You probably don't need "_time=$orig_time$," in the eval.

index="main"
| eval orig_time=_time, orig_raw=_raw
| map maxsearches=10000 search="|makeresults=1
|eval _time=$orig_time$, _raw=$orig_raw$
| collect index="scanned_app" sourcetype=X host=$asset_id$"

0 Karma

mushkevych
Explorer

@jbjerke_splunk and @pwild_splunk thank you for comments. could you perhaps help me understand why the SPL index="main" | eval host=asset_id | collect index="scanned_app" works without |collect... and does not work with |collect... ? What is happening during |collect...?

0 Karma

pwild_splunk
Splunk Employee
Splunk Employee

To understand why this doesn't work as you're expecting you have to understand how the collect command works. When you pipe a search result into collect, it dumps the output of the command into a text file on your splunk server, which is then picked up by a monitor input for indexing in the same way as any other input. Just like when configuring a monitor input, you can specify the host field once for the input, you can't set it on an event by event basis. With collect, when you define fields like index=A, sourcetype=B, host=C you are defining them in the same way you would in an inputs.conf. Those fields are applied to the output for processing by Splunk's data pipeline.

0 Karma

Vijeta
Influencer

@mushkevych -Since host is a default field , and collect command will look for default fields for source sourcetype host unless you override it in collect command

0 Karma

mushkevych
Explorer

@Vijeta thank you for reply. Perhaps you can advise how to override default field host in collect command?

0 Karma

Vijeta
Influencer

Since you want host value to be assigned to a variable asset_id , you will have to use map command as mentioned by @pwild_splunk

0 Karma

pwild_splunk
Splunk Employee
Splunk Employee

Try this

index="main" | eval _raw=_raw.",host=".asset_id | collect index="scanned_app"

0 Karma

pwild_splunk
Splunk Employee
Splunk Employee

This may not do what you want. The events in the summary index will contain a host field that is multi-valued, containing the indexed host field as well as the auto-extracted host value. If the purpose of this is to create a dashboard or graph, you may be able to work with the data by removing the first value with something like this.

| eval host=mvindex(host,1)

0 Karma

adonio
Ultra Champion

can you elaborate? is asset_id a field or a static value?
also, what is it that you are trying to accomplish? i sense lookup will serve you better here

0 Karma

mushkevych
Explorer

asset_id is a field.
my goal is to transform the data set by changing host value and persist it in another index.
P.S. Updated the question to reflect your comment

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...