I run HAProxy and grab it via a universal forwarder and send it to our receiver/indexer (all on same host).
I modified my props.conf as follows.
props.conf
[source::/var/log/*haproxy.log]
TRANSFORMS-syslogstripper = haproxy_syslog_stripper, haproxyfields, clientinfofields, backendfields, requestinfo, connectioninfo, queueinfo, uriinfo
[sourcetype::HAProxy]
MAX_TIMESTAMP_LOOKAHEAD=40
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TZ = US/Mountain
REPORT-haproxyfieldextract = haproxyfields, clientinfofields, backendfields, requestinfo, connectioninfo, queueinfo, uriinfo
TRANSFORMS-haproxystuff = haproxyfields
Here is my transforms.conf where I listed pertinent HAProxy info
transforms.conf
# This will strip the syslog header (date stamp and host) from a syslog event
[haproxy_syslog_stripper]
REGEX = ^[A-Z][a-z]+\s+\d+\s\d+:\d+:\d+\s[^\s]*\s(.*)$
FORMAT = $1
DEST_KEY = _raw
# Transform for HAProxy
[haproxyfields]
DELIMS = " "
FIELDS = haproxy_id,client_info, date_time,frontend_name,backend,request_info,status_code,response_size,val1,val2,flags,connection_info,queue_info,req_header,resp_header,method,uri_info
CLEAN_KEYS=true
#the following is used to extract values from the previous extraction
[clientinfofields]
SOURCE_KEY=client_info
DELIM = ":"
FIELDS = client_ip,client_port
[backendfields]
SOURCE_KEY=backend
DELIM = "/"
FIELDS = backend_name,server_name
[requestinfo]
SOURCE_KEY=request_info
DELIM= "/"
FIELDS=request_time,queue_time,connection_time,response_time,total_time
[connectioninfo]
SOURCE_KEY=connection_info
DELIM= "/"
FIELDS=process_connections,frontend_connections,backend_connections,server_connections,retries
[queueinfo]
SOURCE_KEY=queue_info
DELIM= "/"
FIELDS=server_queue_size,backend_queue_size
#You can still use regex on those extraction that still need it.
[uriinfo]
SOURCE_KEY=uri_info
REGEX=(? [^"]+?)
I am able to get the fields listed in haproxyfields stanza to extract using this search term:
sourcetype="HAProxy" | extract haproxyfields
That works great and I am super excited by that.
The problem is that I have no idea how I would then get the fields that depend on the previously extracted stuff on my haproxyfields stanza to display as well (i.e. client_ip, client_port, backend_name, server_name, etc.)
Any ideas why those fields wouldn't just be extracted along with the "haproxyfields"?
... View more