After transaction: "Field '_raw' does not exist in the data"

[acdevlin]

I'm using a transaction to group events within 30 minutes of one another. What I want to do after that is "undo" one of the large events created by the transaction command, so I can then view the data in more detail.

With some searching, I found this thread, which talks about how to split a transaction. Unfortunately, this method is not working for me; when I try nick's suggested approach, I get the error "Field '_raw' does not exist in the data" and the transaction event is not split up.

The full search is long and complicated. The section relevant to my question looks something like this:

... | transaction user maxpause=30m mvlist=t maxevents=-1 | search start_time="06/15/2011 17:21:52"| eval _raw = split(_raw, "\n") | mvexpand _raw | stats values(req_time)

Any ideas? Am I using the _raw field incorrectly?

[hexx]

Here's an alternative method using rex.

For testing purposes let's associate events from metrics.log in clumps of 4 with :

index=_internal source=*metrics.log | transaction maxevents=4 source

This yields transaction meta-events that look like this :

12-29-2011 08:51:52.940 -0800 INFO  Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
12-29-2011 08:51:52.940 -0800 INFO  Metrics - group=queue, name=typingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=58, smallest_size=0
12-29-2011 08:51:52.940 -0800 INFO  Metrics - group=realtime_search_data, system total, drop_count=0
12-29-2011 08:51:52.940 -0800 INFO  Metrics - group=search_concurrency, system total, active_hist_searches=0, active_realtime_searches=0

To get the original _raw field back (and therefore, display the original events) I use rex with a "delimiter" regex matching the date at the beginning of my event, and then all characters until a CLRF. This creates one multi-value field per transaction containing the pre-transaction values of _raw, which we can then expand back to a single-value field with mvexpand:

index=_internal source=*metrics.log | transaction maxevents=4 source | eval transaction_raw=_raw | rex max_match=1000 "(?msi)^(?[01]\d-[0-3]\d-2011\s[^\r\n]*?)$" | mvexpand raw | eval _raw=raw

Remarks :

• For the delimiter regex to work, you need to specify max_match with a value equal to or higher than the maximum number of events you expect in your transaction.
• The regex will need to be reworked if your original events span multiple lines.
• A transition field (here raw) is necessary. You cannot restore _raw directly unto itself.
• We conserved the transaction's own _raw in transaction_raw which allows to still report on the transaction results. The transaction special fields are also conserved.

[Lowell · Accepted solution]

Your "_raw" field wouldn't exist anymore after your | stats values(req_time). Not sure if that's relevant to your entire search or not.

Have you tried using a field name other than _raw (since it has a special purpose) while debugging your search? Sometimes renaming the field and then naming it back lets you get around some internal assumptions built in to some of the search commands.

For example, you could try:

... | eval raw=split(_raw, "\n") | mvexpand raw | rename raw as _raw | ...

[carasso]

in 4.3, there's mvraw=true, which keeps _raw as a multivalued field

[Krishna_R]

@acdevlin,

I'm trying the same here (to split a transaction back to original events after filtering on transaction) and I hit the same snag 'split(_raw, "\n")' isn't splitting... Could you please tell which character worked for you?

[acdevlin]

Suddenly realized this morning that the problem was the \n delimiter; changing it to a different character helped break up events from _raw. Kind of embarrassing.

Thanks again Lowell for helping me out!

[acdevlin]

Thanks for the speedy reply Lowell.

Your idea DOES get rid of the error about "_raw"...but it seems the transaction event is not being split up. I still get a single event at the end of the query. Do you have any further suggestions on solving this?

Also, the "| stats values(req_time)" is what I'm trying to get out of the search as a whole: the specific "req_time" values associated with this particular series of events in the transaction. If that at all makes sense.