We use BlueCat for DNS/DHCP and we are forwarding the DNS/DHCP logs via CEF format to HDFS. I am trying to reverse engineer the Splunk Add-on for ISC BIND for Hunk, specifically the assigning of multiple sourcetypes to one index (assign events to their proper sourcetype - isc:bind:query, isc:bind:lameserver, isc:bind:network, isc:bind:transfer).
How would I go about configuring the props.conf file to assigning multiple sourcetypes to my source of BlueCat?
Thx
You should be able to do this with regular expressions. This page should help:
http://docs.splunk.com/Documentation/Splunk/6.4.3/Data/Advancedsourcetypeoverrides
I have my regexes ready to go, but do I list each sourcetype as follows in props.conf as:
[source::/xyz/dnslogs/...]
sourcetype = isc:bind:query
[isc:bind:query]
REPORT-1_extract_field = isc_bind_query_extract_field_0
EVAL-message_type = "Query"
EVAL-query_type = "Query"
EVAL-vendor_product = "ISC:Bind"
LOOKUP-2_look_up_extract = isc_bind_severities_lookup vendor_severity OUTPUT severity
LOOKUP-3_look_up_extract = isc_bind_category_lookup sourcetype OUTPUT vendor_category
transforms.conf
[isc_bind_query_extract_field_0]
REGEX = \s+client\s+([\w-.:]{1,100})#(\d{1,5})(?:\s+([^)]+))?:(?:\s+view\s+[^:]+:)?\s+query:\s+(?([\w-.:]{1,100}))?\s+([^\s]+)\s+([^\s]+)\s++-\s+(([\w-.:]{1,100}))$
FORMAT = vendor_severity::$1 src::$2 src_port::$3 query::$4 record_class::$5 record_type::$6 flag::$7 dest::$8
and so on?
[isc:bind:lameserver]
REPORT-1_extract_field = isc_bind_lameserver_extract_field_0
EVAL-app = "ISC:Bind"
EVAL-type = "alert"
LOOKUP-2_look_up_extract = isc_bind_severities_lookup vendor_severity OUTPUT severity
LOOKUP-3_look_up_extract = isc_bind_category_lookup sourcetype OUTPUT vendor_category
[source::/xyz/dnslogs/...]
sourcetype = isc:bind:lameserver
transforms.conf
[isc_bind_lameserver_extract_field_0]
REGEX = (?:\s+lame-servers:)?(?:\s+([^:]+):)?\s+(error\s+(([^)]+))\s+resolving\s+'([\w-.:]{1,100})/([^/]+)/([^']+)':\s+([\w-.:]{1,100})#(\d{1,5}))$
FORMAT = vendor_severity::$1 body::$2 error_type::$3 query::$4 record_type::$5 record_class::$6 dest::$7 dest_port::$8
Thx
(Take the following with a big grain of salt, as I haven't had a chance to test it myself)
I think what you want to do is something like the following. Choose one sourcetype to be the default for the index, i.e.:
[source::/xyz/dnslogs/...]
sourcetype = isc:bind:lameserver
Then for every other sourcetype you wish to assign, do the following:
transforms.conf
[disambiguate_isc_bind_query]
REGEX = \s+client\s+([\w-.:]{1,100})#(\d{1,5})(?:\s+([^)]+))?:(?:\s+view\s+[^:]+:)?\s+query:\s+(?([\w-.:]{1,100}))?\s+([^\s]+)\s+([^\s]+)\s++-\s+(([\w-.:]{1,100}))$
FORMAT = sourcetype:isc:bind:query
DEST_KEY = MetaData:Sourcetype
props.conf
[source::/xyz/dnslogs/...]
TRANSFORMS-isc_bind_query_transform = disambiguate_isc_bind_query
After combining all stanzas with the same header, your stanza in props should looks like:
[source::/xyz/dnslogs/...]
TRANSFORMS-isc_bind_query_transform = disambiguate_isc_bind_query
TRANSFORMS-isc_bind_network = disambiguate_isc_bind_network
......
......
In Hunk one assigns the sourcetype
by the association to the file location in the HDFS.
Gives the example of -
[source::/user/xyz/ciscologfiles/...]
sourcetype = cisco_syslog
[source::/user/xyz/iislogfiles/...]
sourcetype = iis
Do you need to associate multiple sourcetypes for one source?
Yes, I would like to associate multiple sourcetypes to one source.
Thx