Does SEDCMD work line-by-line or on the entire event?

[Lowell]

Out of the box, the unix sed command operates on a line-by-line basis. Is this the same for the SEDCMD setting in props.conf? Or does it operate on the entire event?

---

Just for the sake of an example. Say I'm using the following unix command to filter out comment lines (lines starting with a ";"). (Technically, this example only clears the comment lines rather than remove them, "grep" would be more obvious choice of unix commands, but splunk doesn't have a GREPCMD, so it's a mute point.)

cat myfile |  sed -re 's/^\s*;.*$//'

Will that work the same way as the following props.conf entry:

SEDCMD-drop_comments = s/^\*;.*$//

[Lowell]

To get the SEDCMD to work on a line-by-line basis, you can use the following:

SEDCMD-drop_comments = s/(?m)^\*;.*$//g

The difference is:

1. The (?m) enables multi-line mode in regex engine. This causes ^ and $ to now mean start-of-line and end-of-line, rather than start-of-event and end-of-event.
2. The g on the end tells the sed command to be applied as many times as possible; therefore the replacement will occur more than once. So without the g, only the first matching line would be replaced instead of all the lines.

[Lowell]

@gkanapathy, very good point. (Yeah, strait sed-style regex are a pain once you've gotten use to a powerful regex engine. I normally end up using the sed -r which enables extended regexs, which are still pretty weak when compared to PCRE.) Splunk is definitely more flexible in that regard 😉

[gkanapathy]

Note that Splunk's SEDCMD uses PCRE regex, not standard sed regex.

[Ledion_Bitincka · Accepted solution]

splunk's SEDCMD works with one event at a time

[Lowell]

That's what I was suspecting. I found you can make it work line-by-line by using multi-line regex mode. (I added an answer of my own with an example.)