changing `host` and persisting the result

[mushkevych]

I am trying to make this query work:
index="main" | eval host=asset_id | collect index="scanned_app"
where asset_id is a field, not a static value.

Two observations regarding the query:
- without | collect ..., the search shows data as i expect it - with the meta-field host changed
- with | collect ..., the resulting index carries host unchanged from the main index

Q: how do i change the host, so that it can be persisted in another index ?
index="main" | eval *magic_here* | collect index="scanned_app"

[jbjerke_splunk]

You can try and wrap you search in the map command that dynamically let's you generate another search.

This generates an event in the summary index with host=hello set from the outer search.

|makeresults count=1
| eval asset_id="hello"
| map search="
search
index=\"main\"
| collect index=scanned_app host=$asset_id$
"

j

[pwild_splunk]

This will work. Remember that the map command, by default, is limited to only 10 sub-search iterations. Use the option maxsearches=10000 or something more appropriate for your data set.

Converting the above to your actual search, see below. You probably don't need "_time=$orig_time$," in the eval.

index="main"
| eval orig_time=_time, orig_raw=_raw
| map maxsearches=10000 search="|makeresults=1
|eval _time=$orig_time$, _raw=$orig_raw$
| collect index="scanned_app" sourcetype=X host=$asset_id$"

[mushkevych]

@jbjerke_splunk and @pwild_splunk thank you for comments. could you perhaps help me understand why the SPL index="main" | eval host=asset_id | collect index="scanned_app" works without |collect... and does not work with |collect... ? What is happening during |collect...?

[pwild_splunk]

To understand why this doesn't work as you're expecting you have to understand how the collect command works. When you pipe a search result into collect, it dumps the output of the command into a text file on your splunk server, which is then picked up by a monitor input for indexing in the same way as any other input. Just like when configuring a monitor input, you can specify the host field once for the input, you can't set it on an event by event basis. With collect, when you define fields like index=A, sourcetype=B, host=C you are defining them in the same way you would in an inputs.conf. Those fields are applied to the output for processing by Splunk's data pipeline.

[Vijeta]

@mushkevych -Since host is a default field , and collect command will look for default fields for source sourcetype host unless you override it in collect command

[mushkevych]

@Vijeta thank you for reply. Perhaps you can advise how to override default field host in collect command?

[Vijeta]

Since you want host value to be assigned to a variable asset_id , you will have to use map command as mentioned by @pwild_splunk

[pwild_splunk]

Try this

index="main" | eval _raw=_raw.",host=".asset_id | collect index="scanned_app"

[pwild_splunk]

This may not do what you want. The events in the summary index will contain a host field that is multi-valued, containing the indexed host field as well as the auto-extracted host value. If the purpose of this is to create a dashboard or graph, you may be able to work with the data by removing the first value with something like this.

| eval host=mvindex(host,1)

[adonio]

can you elaborate? is asset_id a field or a static value?
also, what is it that you are trying to accomplish? i sense lookup will serve you better here

[mushkevych]

asset_id is a field.
my goal is to transform the data set by changing host value and persist it in another index.
P.S. Updated the question to reflect your comment