Transforms not splitting sourcetypes

[EdgarAllenProse]

So I am trying to take a single monitored log, and split sourcetypes based off of the terms SCAN, RECV, SEND. I created my props and transforms first. I made sure that splunk recognized the sourcetype in props, this was successfull. I created inputs stanza for the monitored file and the only sourcetype I see is test_barracuda, which was from the props stanza, I am not getting the split transforms should be doing.

inputs.conf

[monitor://C:\\Users\\eap\\Desktop\\security\\Splunk\\DEVELOPMENT\\Test_Ingest\\test_raw_spam.txt]
disabled = false
sourcetype = test_barracuda
index = test

props.conf

[test_barracuda]
CHARSET=AUTO
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
category=Custom
disabled=false
pulldown_type=true
TRANSFORMS-overridest = set_sourcetype

transforms.conf

[set_sourcetype]
REGEX = \d+\s+(SEND|SCAN|RECV)\s
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::test_$1

This is an all in one test instance, so I placed these in SPLUNK_HOME/etc/apps/search/local/

Order of implementation: props.conf --> transforms.conf --> stop splunk --> clear event data from index test --> start splunk --> inputs.conf --> restart splunk.

Splunk gets the data. In the correct index, but only in sourcetype=test_barracuda.
I check to see if the regex in transforms is correct:

Query

index=test sourcetype="test_barracuda" | rex field=_raw "\d+\s+(?P<st>SEND|SCAN|RECV)\s"

Query works and I get exactly as many events in the 3 correct st fields.

So I do btools on props and transforms (note I'm not seeing any errors during /debug/refresh or splunk restart)

btools

C:\Program Files\Splunk\bin>splunk cmd btool transforms list set_sourcetype --debug

C:\Program Files\Splunk\etc\apps\search\local\transforms.conf [set_sourcetype]
C:\Program Files\Splunk\etc\system\default\transforms.conf    CAN_OPTIMIZE = True
C:\Program Files\Splunk\etc\system\default\transforms.conf    CLEAN_KEYS = True
C:\Program Files\Splunk\etc\system\default\transforms.conf    DEFAULT_VALUE =
C:\Program Files\Splunk\etc\apps\search\local\transforms.conf DEST_KEY = MetaData:Sourcetype
C:\Program Files\Splunk\etc\apps\search\local\transforms.conf FORMAT = sourcetype::test_$1
C:\Program Files\Splunk\etc\system\default\transforms.conf    KEEP_EMPTY_VALS = False
C:\Program Files\Splunk\etc\system\default\transforms.conf    LOOKAHEAD = 4096
C:\Program Files\Splunk\etc\system\default\transforms.conf    MV_ADD = False
C:\Program Files\Splunk\etc\apps\search\local\transforms.conf REGEX = \d+\s+(SEND|SCAN|RECV)\s
C:\Program Files\Splunk\etc\system\default\transforms.conf    SOURCE_KEY = _raw
C:\Program Files\Splunk\etc\system\default\transforms.conf    WRITE_META = False
C:\Program Files\Splunk\etc\system\default\transforms.conf    [set_sourcetype_to_stash]
C:\Program Files\Splunk\etc\system\default\transforms.conf    CAN_OPTIMIZE = True
C:\Program Files\Splunk\etc\system\default\transforms.conf    CLEAN_KEYS = True
C:\Program Files\Splunk\etc\system\default\transforms.conf    DEFAULT_VALUE =
C:\Program Files\Splunk\etc\system\default\transforms.conf    DEST_KEY = MetaData:Sourcetype
C:\Program Files\Splunk\etc\system\default\transforms.conf    FORMAT = sourcetype::stash
C:\Program Files\Splunk\etc\system\default\transforms.conf    KEEP_EMPTY_VALS = False
C:\Program Files\Splunk\etc\system\default\transforms.conf    LOOKAHEAD = 4096
C:\Program Files\Splunk\etc\system\default\transforms.conf    MV_ADD = False
C:\Program Files\Splunk\etc\system\default\transforms.conf    REGEX = .
C:\Program Files\Splunk\etc\system\default\transforms.conf    SOURCE_KEY = _raw
C:\Program Files\Splunk\etc\system\default\transforms.conf    WRITE_META = False

C:\Program Files\Splunk\bin>splunk cmd btool props list test_barracuda --debug

C:\Program Files\Splunk\etc\apps\search\local\props.conf [test_barracuda]
C:\Program Files\Splunk\etc\system\default\props.conf    ANNOTATE_PUNCT = True
C:\Program Files\Splunk\etc\system\default\props.conf    AUTO_KV_JSON = true
C:\Program Files\Splunk\etc\system\default\props.conf    BREAK_ONLY_BEFORE =
C:\Program Files\Splunk\etc\system\default\props.conf    BREAK_ONLY_BEFORE_DATE = True
C:\Program Files\Splunk\etc\apps\search\local\props.conf CHARSET = AUTO
C:\Program Files\Splunk\etc\system\default\props.conf    DATETIME_CONFIG = \etc\datetime.xml
C:\Program Files\Splunk\etc\system\default\props.conf    HEADER_MODE =
C:\Program Files\Splunk\etc\system\default\props.conf    LEARN_MODEL = true
C:\Program Files\Splunk\etc\system\default\props.conf    LEARN_SOURCETYPE = true
C:\Program Files\Splunk\etc\system\default\props.conf    LINE_BREAKER_LOOKBEHIND = 100
C:\Program Files\Splunk\etc\system\default\props.conf    MAX_DAYS_AGO = 2000
C:\Program Files\Splunk\etc\system\default\props.conf    MAX_DAYS_HENCE = 2
C:\Program Files\Splunk\etc\system\default\props.conf    MAX_DIFF_SECS_AGO = 3600
C:\Program Files\Splunk\etc\system\default\props.conf    MAX_DIFF_SECS_HENCE = 604800
C:\Program Files\Splunk\etc\system\default\props.conf    MAX_EVENTS = 256
C:\Program Files\Splunk\etc\system\default\props.conf    MAX_TIMESTAMP_LOOKAHEAD = 128
C:\Program Files\Splunk\etc\system\default\props.conf    MUST_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf    MUST_NOT_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf    MUST_NOT_BREAK_BEFORE =
C:\Program Files\Splunk\etc\apps\search\local\props.conf NO_BINARY_CHECK = true
C:\Program Files\Splunk\etc\system\default\props.conf    SEGMENTATION = indexing
C:\Program Files\Splunk\etc\system\default\props.conf    SEGMENTATION-all = full
C:\Program Files\Splunk\etc\system\default\props.conf    SEGMENTATION-inner = inner
C:\Program Files\Splunk\etc\system\default\props.conf    SEGMENTATION-outer = outer
C:\Program Files\Splunk\etc\system\default\props.conf    SEGMENTATION-raw = none
C:\Program Files\Splunk\etc\system\default\props.conf    SEGMENTATION-standard = standard
C:\Program Files\Splunk\etc\apps\search\local\props.conf SHOULD_LINEMERGE = true
C:\Program Files\Splunk\etc\system\default\props.conf    TRANSFORMS =
C:\Program Files\Splunk\etc\apps\search\local\props.conf TRANSFORMS-overridest = set_sourcetype
C:\Program Files\Splunk\etc\system\default\props.conf    TRUNCATE = 10000
C:\Program Files\Splunk\etc\apps\search\local\props.conf category = Custom
C:\Program Files\Splunk\etc\system\default\props.conf    detect_trailing_nulls = auto
C:\Program Files\Splunk\etc\apps\search\local\props.conf disabled = false
C:\Program Files\Splunk\etc\system\default\props.conf    maxDist = 100
C:\Program Files\Splunk\etc\system\default\props.conf    priority =
C:\Program Files\Splunk\etc\apps\search\local\props.conf pulldown_type = true
C:\Program Files\Splunk\etc\system\default\props.conf    sourcetype =

No conflicts, but I'm not seeing anything for the expected sourcetypes test_SEND, test_RECV, or test_SCAN

Any idea where I messed up?

events for testing

In case you want to test, here are 3 events that match the criteria for each expected sourcetype

 Feb 13 12:14:57 192.168.x.x outbound/smtp: 127.0.0.1 1487013294-09b08c0e0d22d7b0001-vy5CMk 0 0 SEND - 1 0112918C8063 250 2.6.0 <2050829162.21743.1487013293752.JavaMail@dc1prjasszap434.whc> [InternalId=91736206477550, Hostname=host.prod.outlook.com] 11518 bytes in 0.192, 58.365 KB/sec Queued mail for delivery #to#name-com.mail.protection.outlook.com[8.8.8.8]:25

 Feb 13 12:14:56 192.168.x.x  scan: mail2-3.place.com[8.8.8.8] 1487013294-09b08c0e0d22d7b0001-vy5CMk 1487013294 1487013296 SCAN - prvs=7217d0fa1f=services_noreply@place.com name@otherplace.com - 7 88 corporate SZ:3263 SUBJ:Message: Attempt to retrieve your User ID

 Feb 13 12:14:15 192.168.x.x  inbound/pass1: name.place1.com[8.8.8.8] 1487013254-09b08c0e0e22d7a0001-VY5SBA 1487013254 1487013255 RECV information=place2.com@thing.com name@thingy1.com 2 3 blacklist.org[8.8.8.8]

[woodcock]

I am sure that you read this:
http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Advancedsourcetypeoverrides

Your configurations look fine so:
Have you restarted splunkd on every indexer that is receiving these events?
Have you sent NEW events to the indexers (Indexed data is IMMUTABLE; only NEW events, post restart, will have the new configurations applied; previously indexed events will not be changed)?

[EdgarAllenProse]

Sorry I missed this response, I did take your steps, when initially setting up, restarting splunkd, ingesting new data and all, but still no luck. I'm kind of at a loss haha, I may at this point submit a support ticket to splunk.

[EdgarAllenProse]

I tried the responses on similar questions, but they don't seem to be working.

[EdgarAllenProse]

I'm setting up a universal forwarder to see if I need to do this there first.