Trying to strip the header out of an incoming http post and only index the json for parsing with spath. Having an issue with the regex I believe.
props.conf:
[akamai_post_json]
TRANSFORMS-t1=nullqueue_httpheader
transforms.conf:
[nullqueue_httpheader]
REGEX=^(?:POST|User-Agent|Accept|Content-Length|Expect|Content-Type|Content-Disposition)gm
DEST_KEY=queue
FORMAT=nullQueue
Here's a sample event:
POST / HTTP/1.1
User-Agent: curl/7.26.0
Host: localhost
Accept: */*
Content-Length: 2552
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----------------------------0b1c32056fc5
------------------------------0b1c32056fc5
Content-Disposition: form-data; name="fileupload"; filename="default_connector_schema_1.0.json"
Content-Type: application/octet-stream
{
"apiType" : "String",
"apiFormat" : "String",
"apiVersion" : 0,
"ID" : "String",
"startTime" : "String",
"eventType" : "String",
"cpCode" : 0,
"message" : {
"protocol" : "0",
"protoVersion" : 0,
"clientIP" : "String",
"reqPort" : 0,
"reqHost" : "String",
"reqMethod" : "String",
"reqPath" : "String",
"reqQuery" : "String",
"reqContType" : "String",
"reqContLen" : 0,
"sslProtocol" : "String",
"sslVersion" : 0,
"respStatus" : 0,
"respRedirURL" : "String",
"respContType" : "String",
"respContLen" : 0,
"respBytesServed" : 0,
"userAgent" : "String",
"originHostname" : "String"
},
"httpHeaders" : {
"reqHeader" : {
"accEnc" : "String",
"accLang" : "String",
"auth" : "String",
"cacheCtl" : "String",
"connection" : "String",
"contMD5" : "String",
"cookie" : "String",
"DNT" : "String",
"ifMatch" : "String",
"ifMod" : "String",
"ifNoMatch" : "String",
"pragma" : "String",
"range" : "String",
"referer" : "String",
"TE" : "String",
"upgrade" : "String",
"via" : "String",
"xFrwdFor" : "String",
"xReqWith" : "String"
},
"respHeader" : {
"cacheCtl" : "String",
"connection" : "String",
"contEnc" : "String",
"contLang" : "String",
"contLen" : "String",
"contMD5" : "String",
"contDisp" : "String",
"contRange" : "String",
"date" : "String",
"eTag" : "String",
"expires" : "String",
"lastMod" : "String",
"p3p" : "String",
"pragma" : "String",
"server" : "String",
"setCookie" : "String",
"trailer" : "String",
"transEnc" : "String",
"vary" : "String",
"warning" : "String",
"wwwAuth" : "String"
}
},
"performance" : {
"reqHeadSize" : 0,
"reqBodySize" : 0,
"respHeadSize" : 0,
"respBodySize" : "String",
"downloadTime" : "String",
"originName" : "String",
"originIP" : "String",
"originInitIP" : "String",
"originRetry" : 0,
"lastMileRTT" : 0,
"lastMileBW" : 0,
"netOriginRTT" : 0,
"cacheStatus" : "String",
"lastByte" : true,
"cliCountry" : "String",
"edgeIP" : "String",
"reqID" : "String"
}
}
------------------------------0b1c32056fc5--
I'm afraid that your regex would never do what you want it to. Sending stuff to the nullQueue works on a per-event basis, not on separate lines in a multiline event.
You're probably better off looking at the SEDCMD-name in props.conf. In the docs there is an example of masking credit card numbers and other sensitive data, but you could easily filter out the unwanted lines with a sed script.
EDIT: typo
/k
I'm afraid that your regex would never do what you want it to. Sending stuff to the nullQueue works on a per-event basis, not on separate lines in a multiline event.
You're probably better off looking at the SEDCMD-name in props.conf. In the docs there is an example of masking credit card numbers and other sensitive data, but you could easily filter out the unwanted lines with a sed script.
EDIT: typo
/k
So instead of the transform in props.conf, I use SEDCMD, but I'm still not seeing ANY change. Is the below correct for remove the POST line?
props.conf:
[akamai_post_json]
SEDCMD-httpheader = s/(?mg)^POST.*$\n//g