Getting Data In

On a heavy forwarder, can I forward a subset of data to syslog and drop everything else?

reswob4
Builder

Here is my situation: I have a Windows HF that is collecting a lot of different data. Some via powershell scripts, some via WMI, some via file monitoring locally and over UNC paths. All of that data is being forwarded to two indexes.

A few weeks ago I configured one of the file monitoring inputs to send a copy of the data it collected to a syslog server.

I now need to send that data (collected via file monitoring) to the syslog server and NOT to the indexers. IOW, I want all data collected by this HF to go to the indexers, EXCEPT this data which should be sent to the syslog server ONLY.

How do I do that?

I've read through this which helped me get the current configuration:
http://docs.splunk.com/Documentation/SplunkCloud/6.6.1/Forwarding/Forwarddatatothird-partysystemsd

Here are my config files:

.../etc/apps/myapp/local/props.conf:

[WinDNS]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 1000
EXTRACT-Domain = (?i) .*? \.(?P<Domain>[-a-zA-Z0-9@:%_\+.~#?;//=]{2,256}\.[a-z]{2,6})
EXTRACT-src = (?i) [Rcv|Snd] (?P<source_address>\d+\.\d+\.\d+\.\d+)
EXTRACT-Threat_ID,Context,Int_packet_ID,proto,mode,Xid,type,Opcode,Flags_Hex,char_code,ResponseCode,question_type = .+?[AM|PM]\s+(?<Threat_ID>\w+)\s+(?<Context>\w+)\s+(?<Int_packet_ID>\w+)\s+(?<proto>\w+)\s+(?<mode>\w+)\s+\d+\.\d+\.\d+\.\d+\s+(?<Xid>\w+)\s(?<type>(?:R)?)\s+(?<Opcode>\w+)\s+\[(?<Flags_Hex>\w+)\s(?<char_codes>.+?)(?<ResponseCode>[A-Z]+)\]\s+(?<question_type>\w+)\s
EXTRACT-Authoritative_Answer,TrunCation,Recursion_Desired,Recursion_Available = (?m) .+?Message:\W.+\W.+\W.+\W.+\W.+AA\s+(?<Authoritative_Answer>\d)\W.+TC\s+(?<TrunCation>\d)\W.+RD\s+(?<Recursion_Desired>\d)\W.+RA\s+(?<Recursion_Available>\d)
TRANSFORMS-droplocal2 = droplocal2
TRANSFORMS-dropbach = dropbach
#TRANSFORMS-dropall = dropall
SEDCMD-win_dns = s/\(\d+\)/./g
TRANSFORMS-dns = send_to_syslog

.../etc/apps/myapp/local/transforms.conf

[dropbach]
REGEX = \[.+?\]\s+\w+\s+.+?BACH
DEST_KEY=queue
FORMAT=nullQueue

[droplocal2]
REGEX = \[.+?\]\s+\w+\s+.+?local
DEST_KEY=queue
FORMAT=nullQueue

[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = my_syslog_group

#[dropall]
#REGEX = .
#DEST_KEY=queue
#FORMAT=nullQueue

.../etc/system/local/outputs.conf

[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 0

[tcpout-server://splunk-01:9997]

[tcpout:default-autolb-group]
disabled = false
server = splunk-01:9997,splunk-02:9997

[tcpout-server://splunk-02:9997]  # not sure why this is here....

[syslog:my_syslog_group]
server = 1.1.1.5:514

As you can tell, I tried to add a 'dropall', but that just dropped everything without sending a copy to the syslog server first.

I then found this forum post:

https://answers.splunk.com/answers/4083/can-i-route-some-data-as-syslog-output-to-multiple-destinati...

which seems to imply that to do what I want to do, I need to modify the outputs.conf so the defaultgroup=nothing and then modify all the props.conf and transforms.conf files for all my inputs to point to the "default-autolb-group" in outputs.conf that sends to the indexers, and then for this app have the ONLY output reference pointing to the "my_syslog_group" in outputs.conf.

Is that correct or something else?

0 Karma

reswob4
Builder
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...