So I am trying to take a single monitored log, and split sourcetypes based off of the terms SCAN, RECV, SEND. I created my props and transforms first. I made sure that splunk recognized the sourcetype in props, this was successfull. I created inputs stanza for the monitored file and the only sourcetype I see is test_barracuda, which was from the props stanza, I am not getting the split transforms should be doing.
[monitor://C:\\Users\\eap\\Desktop\\security\\Splunk\\DEVELOPMENT\\Test_Ingest\\test_raw_spam.txt]
disabled = false
sourcetype = test_barracuda
index = test
[test_barracuda]
CHARSET=AUTO
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
category=Custom
disabled=false
pulldown_type=true
TRANSFORMS-overridest = set_sourcetype
[set_sourcetype]
REGEX = \d+\s+(SEND|SCAN|RECV)\s
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::test_$1
This is an all in one test instance, so I placed these in SPLUNK_HOME/etc/apps/search/local/
Order of implementation: props.conf --> transforms.conf --> stop splunk --> clear event data from index test --> start splunk --> inputs.conf --> restart splunk.
Splunk gets the data. In the correct index, but only in sourcetype=test_barracuda.
I check to see if the regex in transforms is correct:
index=test sourcetype="test_barracuda" | rex field=_raw "\d+\s+(?P<st>SEND|SCAN|RECV)\s"
Query works and I get exactly as many events in the 3 correct st fields.
So I do btools on props and transforms (note I'm not seeing any errors during /debug/refresh or splunk restart)
C:\Program Files\Splunk\etc\apps\search\local\transforms.conf [set_sourcetype]
C:\Program Files\Splunk\etc\system\default\transforms.conf CAN_OPTIMIZE = True
C:\Program Files\Splunk\etc\system\default\transforms.conf CLEAN_KEYS = True
C:\Program Files\Splunk\etc\system\default\transforms.conf DEFAULT_VALUE =
C:\Program Files\Splunk\etc\apps\search\local\transforms.conf DEST_KEY = MetaData:Sourcetype
C:\Program Files\Splunk\etc\apps\search\local\transforms.conf FORMAT = sourcetype::test_$1
C:\Program Files\Splunk\etc\system\default\transforms.conf KEEP_EMPTY_VALS = False
C:\Program Files\Splunk\etc\system\default\transforms.conf LOOKAHEAD = 4096
C:\Program Files\Splunk\etc\system\default\transforms.conf MV_ADD = False
C:\Program Files\Splunk\etc\apps\search\local\transforms.conf REGEX = \d+\s+(SEND|SCAN|RECV)\s
C:\Program Files\Splunk\etc\system\default\transforms.conf SOURCE_KEY = _raw
C:\Program Files\Splunk\etc\system\default\transforms.conf WRITE_META = False
C:\Program Files\Splunk\etc\system\default\transforms.conf [set_sourcetype_to_stash]
C:\Program Files\Splunk\etc\system\default\transforms.conf CAN_OPTIMIZE = True
C:\Program Files\Splunk\etc\system\default\transforms.conf CLEAN_KEYS = True
C:\Program Files\Splunk\etc\system\default\transforms.conf DEFAULT_VALUE =
C:\Program Files\Splunk\etc\system\default\transforms.conf DEST_KEY = MetaData:Sourcetype
C:\Program Files\Splunk\etc\system\default\transforms.conf FORMAT = sourcetype::stash
C:\Program Files\Splunk\etc\system\default\transforms.conf KEEP_EMPTY_VALS = False
C:\Program Files\Splunk\etc\system\default\transforms.conf LOOKAHEAD = 4096
C:\Program Files\Splunk\etc\system\default\transforms.conf MV_ADD = False
C:\Program Files\Splunk\etc\system\default\transforms.conf REGEX = .
C:\Program Files\Splunk\etc\system\default\transforms.conf SOURCE_KEY = _raw
C:\Program Files\Splunk\etc\system\default\transforms.conf WRITE_META = False
C:\Program Files\Splunk\etc\apps\search\local\props.conf [test_barracuda]
C:\Program Files\Splunk\etc\system\default\props.conf ANNOTATE_PUNCT = True
C:\Program Files\Splunk\etc\system\default\props.conf AUTO_KV_JSON = true
C:\Program Files\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE =
C:\Program Files\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE_DATE = True
C:\Program Files\Splunk\etc\apps\search\local\props.conf CHARSET = AUTO
C:\Program Files\Splunk\etc\system\default\props.conf DATETIME_CONFIG = \etc\datetime.xml
C:\Program Files\Splunk\etc\system\default\props.conf HEADER_MODE =
C:\Program Files\Splunk\etc\system\default\props.conf LEARN_MODEL = true
C:\Program Files\Splunk\etc\system\default\props.conf LEARN_SOURCETYPE = true
C:\Program Files\Splunk\etc\system\default\props.conf LINE_BREAKER_LOOKBEHIND = 100
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DAYS_AGO = 2000
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DAYS_HENCE = 2
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_AGO = 3600
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_HENCE = 604800
C:\Program Files\Splunk\etc\system\default\props.conf MAX_EVENTS = 256
C:\Program Files\Splunk\etc\system\default\props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
C:\Program Files\Splunk\etc\system\default\props.conf MUST_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_BEFORE =
C:\Program Files\Splunk\etc\apps\search\local\props.conf NO_BINARY_CHECK = true
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION = indexing
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-all = full
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-inner = inner
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-outer = outer
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-raw = none
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-standard = standard
C:\Program Files\Splunk\etc\apps\search\local\props.conf SHOULD_LINEMERGE = true
C:\Program Files\Splunk\etc\system\default\props.conf TRANSFORMS =
C:\Program Files\Splunk\etc\apps\search\local\props.conf TRANSFORMS-overridest = set_sourcetype
C:\Program Files\Splunk\etc\system\default\props.conf TRUNCATE = 10000
C:\Program Files\Splunk\etc\apps\search\local\props.conf category = Custom
C:\Program Files\Splunk\etc\system\default\props.conf detect_trailing_nulls = auto
C:\Program Files\Splunk\etc\apps\search\local\props.conf disabled = false
C:\Program Files\Splunk\etc\system\default\props.conf maxDist = 100
C:\Program Files\Splunk\etc\system\default\props.conf priority =
C:\Program Files\Splunk\etc\apps\search\local\props.conf pulldown_type = true
C:\Program Files\Splunk\etc\system\default\props.conf sourcetype =
No conflicts, but I'm not seeing anything for the expected sourcetypes test_SEND, test_RECV, or test_SCAN
Any idea where I messed up?
In case you want to test, here are 3 events that match the criteria for each expected sourcetype
Feb 13 12:14:57 192.168.x.x outbound/smtp: 127.0.0.1 1487013294-09b08c0e0d22d7b0001-vy5CMk 0 0 SEND - 1 0112918C8063 250 2.6.0 <2050829162.21743.1487013293752.JavaMail@dc1prjasszap434.whc> [InternalId=91736206477550, Hostname=host.prod.outlook.com] 11518 bytes in 0.192, 58.365 KB/sec Queued mail for delivery #to#name-com.mail.protection.outlook.com[8.8.8.8]:25
Feb 13 12:14:56 192.168.x.x scan: mail2-3.place.com[8.8.8.8] 1487013294-09b08c0e0d22d7b0001-vy5CMk 1487013294 1487013296 SCAN - prvs=7217d0fa1f=services_noreply@place.com name@otherplace.com - 7 88 corporate SZ:3263 SUBJ:Message: Attempt to retrieve your User ID
Feb 13 12:14:15 192.168.x.x inbound/pass1: name.place1.com[8.8.8.8] 1487013254-09b08c0e0e22d7a0001-VY5SBA 1487013254 1487013255 RECV information=place2.com@thing.com name@thingy1.com 2 3 blacklist.org[8.8.8.8]
I am sure that you read this:
http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Advancedsourcetypeoverrides
Your configurations look fine so:
Have you restarted splunkd on every indexer that is receiving these events?
Have you sent NEW events to the indexers (Indexed data is IMMUTABLE; only NEW events, post restart, will have the new configurations applied; previously indexed events will not be changed)?
Sorry I missed this response, I did take your steps, when initially setting up, restarting splunkd, ingesting new data and all, but still no luck. I'm kind of at a loss haha, I may at this point submit a support ticket to splunk.
I tried the responses on similar questions, but they don't seem to be working.
I'm setting up a universal forwarder to see if I need to do this there first.