I have a field in an event that contains a number of separate individual fields. What would be the most efficient way to extract these so I can search them? Can this be done at index time using transforms? I'm aware of the makemv
command, but these are not values of the same field; they are fields within a field.
I'm trying to extract the individual fields within the auth_source field. So I would like something like auth_source_auth, auth_source_level, auth_source_login_id, etc
Example:
2018-10-18 19:36:47.000, date="10/18/2018 19:36:47", id="166450895", action_at="1539891407534", action="CREATE", permission_id="106965896", customer_id="5495063", role_id="270", active="1", created_at="2018-10-18 19:36:47.0", auth_source="auth:shhkey-v1 | auth-level:internal | login-id:firstlast@company.com-1539295510177 | cus-id:651613 | user-id:65161 | origin-ip:123.123.228.35 | correlation-id:2282777c-25c8-4e1e-a823-4e495fa354ad | zk-group:app-users:api:web:user-traffic:us-east-1", hub_id="61861251", produced_at="1539891407829"
Do you want these field extractions to be at index time or search time? (Splunk best practice is at search time, but it can be at index time.)
Given that search time is preferred, here is a rex
that will do what you want, and the auto-field-extraction will be similar (this is a run-anywhere search that shows the concept by using your data and showing that it works, the only line you need to use is the rex
line for the regular expression to use for the field extraction):
| makeresults
| eval auth_source="auth:shhkey-v1 | auth-level:internal | login-id:firstlast@company.com-1539295510177 | cus-id:651613 | user-id:65161 | origin-ip:123.123.228.35 | correlation-id:2282777c-25c8-4e1e-a823-4e495fa354ad | zk-group:app-users:api:web:user-traffic:us-east-1"
| rex field=auth_source "auth:(?<auth>[^|]*) \| auth-level:(?<auth_level>[^|]*) \| login-id:(?<login_id>[^|]*) \| cus-id:(?<cus_id>[^|]*) \| user-id:(?<user_id>[^|]*) \| origin-ip:(?<origin_ip>[^|]*) \| correlation-id:(?<correlation_id>[^|]*) \| zk-group:(?<zk_group>[^|]*)"
If you need more help on this, just add your comments.
If doing this at index-time really is requirement, you can always use a report on the part of the event that starts with auth_source=
the report option in your props file associated with your sourcetype the extracts at search time the pairs separated by the = sign
Check this answer for reference 🙂
https://answers.splunk.com/answers/9853/multivalue-field-regex-question.html#answer-9875
Do you want these field extractions to be at index time or search time? (Splunk best practice is at search time, but it can be at index time.)
Given that search time is preferred, here is a rex
that will do what you want, and the auto-field-extraction will be similar (this is a run-anywhere search that shows the concept by using your data and showing that it works, the only line you need to use is the rex
line for the regular expression to use for the field extraction):
| makeresults
| eval auth_source="auth:shhkey-v1 | auth-level:internal | login-id:firstlast@company.com-1539295510177 | cus-id:651613 | user-id:65161 | origin-ip:123.123.228.35 | correlation-id:2282777c-25c8-4e1e-a823-4e495fa354ad | zk-group:app-users:api:web:user-traffic:us-east-1"
| rex field=auth_source "auth:(?<auth>[^|]*) \| auth-level:(?<auth_level>[^|]*) \| login-id:(?<login_id>[^|]*) \| cus-id:(?<cus_id>[^|]*) \| user-id:(?<user_id>[^|]*) \| origin-ip:(?<origin_ip>[^|]*) \| correlation-id:(?<correlation_id>[^|]*) \| zk-group:(?<zk_group>[^|]*)"
If you need more help on this, just add your comments.
Looks great, thanks! I was thinking index time but search time will work just fine.