hey guys -
i'm having trouble working with the real-time output app. i have specified a search, and it appears to be working / outputting data via CEF, however the field map i specified is being ignored. i am relying on this app based on reading i've done on integrating splunk with 3rd party SIEMs:
http://splunk-base.splunk.com/answers/13795/cef-output-to-arcsight-where-can-i-find-rtoutputpy
http://www.splunk.com/web_assets/pdfs/resources/Integrating_Splunk_with_Arcsight.pdf
i'm note sure its a syntax issue - as i'm not clear on any documentation at all for the app.
i am exporting proxy logs from a cisco WSA into CEF format. i don't see much need to rename the fields, mostly because i don't understand much about CEF or this app.
here is my resulting RT search:
[4bb1c423-0e96-4a27-9680-7cae8bceed2c]
description = export data to arcsight using TCP 514
disabled = 0
file_backups = 5
file_size = 5242880
label = SIEM output
mode = cef
search = index=qa_cisco_wsa sourcetype=wsa_accesslogs | eval cef_field_map="dvc_time:dvc_time,duration:duration,dvc_ip:dvc_ip,http_status:http_status,result:result,bytes_in:bytes_in,http_method:http_method,dest_url:dest_url,user_id:user_id,user_domain:user_domain,hierarchy_domain:hierarchy_domain,mime_type:mime_type,action:action,cause:cause,x_access_policy:x_access_policy,x_identity:x_identity,x_routing_policy:x_routing_policy,user_agent:user_agent"
splunk_port = 8089
syslog_facility = None
syslog_host = <foo>
syslog_port = 514
syslog_proto = tcp
target = syslog
splunk_host = localhost
syslog_level = 5
I found that that the application performs CEF field validation. Please have a look at the following path: /etc/apps/SplunkRealTimeOutput/bin/real_time_output/cef
The file ceftool.py
has the list of available supported CEF fields. I modified the Python and recompiled. This worked for me.
I've submitted a request to the SplunkRealTimeOutput developer to add all CEF fields.
Mark
is this crazy or what? where is the rtoutput.py file?
That is a bit bizarre 🙂
and of course here is also the "new" search i added with a different GUID label in my realtime conf:
[4bb1c423-0e96-4a27-9680-7cae8bceed2c]
description = export data to arcsight using TCP 514
disabled = 0
file_backups = 5
file_size = 5242880
label = arcsight output
mode = cef
search = index=qa_cisco_wsa sourcetype=wsa_accesslogs | eval cef_field_map="dvc_time:dvc_time,duration:duration,dvc_ip:dvc_ip
...
weird... after 2 days or so of adding / re-adding the same search query - it somehow started working! you can see a transition mid-stream:
<29> Jan 23 14:26:35 pxyau101mel0001.globaltest.anz.com CEF:0|Splunk|wsa_accesslogs|1.0|100000|generic event|5|rt=1358911573 dvc_time=1358911595.879 user_id=- user_agent="Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/20100101 Firefox/17.0" bytes_in=0 dvchost=test-cef hierarchy_domain=NONE/- dest_url=http://www.theage.com.au/?reload\=true http_method=GET result=TCP_DENIED http_status=407 duration=0 mime_type=- dvc_ip=10.220.114.143
i'm not sure how to debug this either besides using a packet capture.
when i first installed it, it was exporting like 2-3 fields by default. once i modified the search... now it only appears to export "dvchost" which actually maps to the "host" in splunk (our proxy).
here's what the stream looks like:
<29> Jan 23 12:55:25 test-cef CEF:0|Splunk|wsa_accesslogs|1.0|100000|generic event|5|rt=1358906102 dvchost=test-cef
<29> Jan 23 12:55:25 test-cef CEF:0|Splunk|wsa_accesslogs|1.0|100000|generic event|5|rt=1358906102 dvchost=test-cef