Hi Splunkers, We're using Rsyslog to collect many of our appliance syslog streams, and then bringing them into Splunk on our heavy forwarder. However the issue we have is all "host" entries are using the heavy forwarder hostname, and not the syslog/appliance hostname. We've adjusted our Rsyslog configuration file to use dynamic filenames, with the following configuration: # vi /etc/rsyslog.d/splunk.conf template (name="websense-proxy" type="string" string="/var/log/splunk/websense/%HOSTNAME%.log") if $programname == "vendor=Forcepoint" and $msg contains "product=Security" then { action (type="omfile" dynafile="websense-proxy") stop } This works fine, and the log files are appearing correctly as: /var/log/splunk/websense/proxy01.log /var/log/splunk/websense/proxy02.log However our "host_regex" does not work properly, and should be picking up the filename as the hostname (minus the file extention). [monitor://var/log/splunk/websense/.*log] index = proxy sourcetype = websense host_regex = .*\/(.+)\.log We've gone through the inputs.conf / host_regex documentation and several posts here, but can't get it working with Splunk properly. However it seems to be working fine in Regex101 - https://regex101.com/r/ayZak8/3 In Splunk we're expecting to see host proxy01 and proxy02. ... View more

We have some Websense proxy logs which are being parsed by the Splunk Add-on for Websense (https://splunkbase.splunk.com/app/2966/), however, the "user" field contains the complete LDAP string of Server, OUs, DCs, making it difficult to correlate against other indexes and user activity. Example log event: Sep 26 04:40:19 10.100.100.101 Sep 26 13:40:21 10.100.100.101 vendor=Websense product=Security product_version=7.8.3 action=permitted severity=1 category=101 user=LDAP://10.100.100.100 OU=People,OU=Sydney,OU=APAC,DC=ourcompany,DC=org/Fred Flinstone (Boss) src_host=10.10.10.10 src_port=0 dst_host=m.velocity.ebay.com dst_ip=66.211.187.41 dst_port=80 bytes_out=0 bytes_in=0 http_response=0 http_method=- http_content_type=- http_user_agent=- http_proxy_status_code=0 reason=- disposition=1026 policy=- role=0 duration=0 url=http://m.velocity.ebay.com/clink/6d2f83c8f1d4cb3ed6aeda50d3561e2cd We are only getting the LDAP protocol and servername, due to the spaces in the user field: user=LDAP://10.100.100.100 We need this: user=LDAP://10.100.100.100 OU=People,OU=Sydney,OU=APAC,DC=ourcompany,DC=org/Fred Flinstone (Boss) to be this: user=Fred Flinstone (Boss) This search works perfectly, but we need to make it permanent at index time: index=proxy sourcetype=websense:cg:kv | rex "user=.*DC=.*\/(?<user>.*?)\s+src_host" We've also updated the props and transforms to make this permanent, however it does not extract the values from the user field. props.conf TRANSFORMS-get_usernames = get_usernames transforms.conf [get_usernames] REGEX = user=.*DC=.*\/(.*?)\s+src_host FORMAT = user::$1 I've tried a few similar Websense suggestions from the Answers site, however, they've haven't produced the results we need. ... View more

We run a few Exchange servers and we need to collect logs for our Splunk Enterprise Security Suite, however, there are many webapps running on an Exchange server, we want to trim the logs we collect to only those which are presenting OWA externally. Sample to keep: 2016-01-04 22:41:01 xxx.xxx.xxx.xxx GET /owa/auth/logon.aspx url=https://webmail.company.org/owa/user@company.org We want to RegEx on "\/owa" above and keep these events, however there are multiple other web-directories (below) which we want to drop, as there is too much content using our licenses. Sample to drop: /Microsoft-Server-ActiveSync/default.eas /Microsoft-Server-ActiveSync/Proxy /EWS/Exchange.asmx /EWS/mrsproxy.svc /Autodiscover/Autodiscover.xml /EWS/Exchange.ASMX We are using the following settings for our props and transforms on the receiving indexers, however we are still seeing lots of log events for both OWA and non-OWA web URLs. props.conf [source:://C:\inetpub\logs\LogFiles] TRANSFORMS-filter = clean, save_owa [source:://D:\IISLogs\Logging] TRANSFORMS-filter = clean, save_owa transforms.conf [clean] REGEX = . DEST_KEY = queue FORMAT = nullQueue [save_owa] REGEX = \/owa DEST_KEY = queue FORMAT = indexQueue I'm just wondering if anyone can advise why all our logs events are still being indexed, when we only want to keep the OWA events? Regards ... View more

I've written this RSysLog configuration file, which allows network based devices to send their event logging streams to a centralised RSysLog server(s), and the each incoming event will be placed into a specific log file, depending on the contents of the received message. This in effect should allow splunkers to identify which log file holds Cisco ASA events, and which log file holds CheckPoint events - inturn, this will ease setting sourcetypes on network based devices. Syslog server just needs a universal forwarder to send logs to indexers, and an inputs.conf file matching each filename to each sourcetype. For example, all Cisco ASA firewall events contain the text "%ASA-" and should go into "cisco-firewall.log" file, and Cisco SourceFire uses a "$programname" of "SFIMS" in the message, so these will be forwarded to "cisco-sourcefire.log", etc... Any inbound network event which does not match any of the filtering rules, will be sent to "uncategorised.log" in message debug mode, so they can be assessed, and a new filter written for new message type. So there's nothing wrong with this RSysLog configuration, it works perfectly, but I spent a fair amount of time building and testing it with the new configuration commands, I'm sure there are a few people here who would also benefit from it's use. Additionally, collecting logs on a low maintenance Linux server, allows you to do more changes / reboots of the Splunk indexers, that you don't need to worry about missing network event logs if you have to restart an indexer server / service - RSysLog will just keep doing its job. Further details on RSysLog properties can be found here: http://www.rsyslog.com/doc/master/configuration/properties.html This setup opens listeners on UDP514 and TCP10514 - You can change these to suit your own requirements. Regards. # # File: "splunk.conf" # Location: "/etc/rsyslog.d/" # Author: Miles Brennan # Date: 31 Dec 2015 # Ver: 0.3 # # # Prerequisites: # Splunk is running as Linux user: adduser splunk # Add "splunk" user to syslog groups: usermod -G sys,adm,syslog splunk # Create logging directory for Splunk: mkdir /var/log/splunk # Set permissions for log directory: chown -R syslog:adm /var/log/splunk /home/splunk # # # Splunk Universal Forwarder - Input Configuration (On Deployment Server) # File: cisco-firewall.log Sourcetype: Cisco ASA Firewalls # File: cisco-sourcefire.log Sourcetype: Cisco SourceFire # File: checkpoint-firewall.log Sourcetype: Checkpoint Firewalls # # Set up log file rotation and compression for the new RSyslog files. # # vi /etc/logrotate.d/splunk #/var/log/splunk/*.log { # daily # missingok # rotate 14 # compress # notifempty # nocreate #} # Load the RSyslog modules needed for inbound connections # module (load="imudp") module (load="imtcp") # Define the RSysLog logging format for writing events to syslog. template (name="rsyslog-fmt" type="string" string="%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" ) # Review all incoming message streams and deliver message to # appropriate log file, as per RegEx. # # Save in new logging directory at: /var/log/splunk # ruleset (name="network-logs") { if $msg contains "%ASA-" then { action (type="omfile" file="/var/log/splunk/cisco-firewall.log" template="rsyslog-fmt") stop } if $programname contains "SFIMS" then { action (type="omfile" file="/var/log/splunk/cisco-sourcefire.log" template="rsyslog-fmt") stop } if $msg contains "Check Point" then { action (type="omfile" file="/var/log/splunk/checkpoint-firewall.log" template="rsyslog-fmt") stop } # Catch the remaining log events which passed through our filters. action (type="omfile" file="/var/log/splunk/uncategorised.log" template="RSYSLOG_DebugFormat") stop } # Bind the UDP module for inbound connections on port 514, # and apply "remote" ruleset to all incoming messages. # input (type="imudp" port="514" ruleset="network-logs") # Bind the TCP module for inbound connections on port 10514, # and apply "remote" ruleset to all incoming messages. # input (type="imtcp" port="10514" ruleset="network-logs") ... View more