How do I change _internal index sourcetype?

[wralph_EPACN]

Some how the _internal index changed its sourcetype. How does one go about changing it back? I am not to worried about the data that has already been indexed, but I need to make sure any new data is under the correct sourcetype.

[wralph_EPACN · Accepted solution]

both look fine to me, but this is the first time i am trying to debug an app so...
the first command i get:
/opt/splunk/etc/system/default/props.conf SHOULD_LINEMERGE = false
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/system/default/props.conf TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf category = Network & Security
/opt/splunk/etc/system/default/props.conf description = Output produced by the Cisco Adaptive Security Appliance (ASA) Firewall
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/system/default/props.conf pulldown_type = 1
/opt/splunk/etc/system/default/props.conf sourcetype =
/opt/splunk/etc/apps/TA-cisco_ios/default/props.conf [cisco:ios]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/apps/TA-cisco_ios/default/props.conf EVAL-app = "cisco:ios"
/opt/splunk/etc/apps/TA-cisco_ios/default/props.conf EVAL-authenticator = coalesce(authenticator, case(facility == "PEM" AND mnemonic == "WEBAUTHFAIL", "webauth", facility == "DOT1X", "dot1x"))
/opt/splunk/etc/apps/TA-cisco_ios/default/props.conf EVAL-bytes = bytes_in + bytes_out
/opt/splunk/etc/apps/TA-cisco_ios/default/props.conf EVAL-dest_mac = case(dest_mac == "Unknown MAC", NULL, isnotnull(dest_mac), lower(replace(dest_mac,"^([0-9a-fA-F]{2})([0-9a-fA-F]{2}).([0-9a-fA-F]{2})([0-9a-fA-F]{2}).([0-9a-fA-F]{2})([0-9a-fA-F]{2})","\1:\2:\3:\4:\5:\6")))
/opt/splunk/etc/apps/TA-cisco_ios/default/props.conf EVAL-dvc = coalesce(dvc, host)
/opt/splunk/etc/apps/TA-cisco_ios/default/props.conf EVAL-product = case(isnotnull(filename) AND isnotnull(filename_line), "WLC", isnotnull(direct_ap_mac), "AP", isnull(filename) AND isnull(filename_line) AND isnull(direct_ap_mac), "IOS")
/opt/splunk/etc/apps/TA-cisco_ios/default/props.conf EVAL-reliable_time = if(reliable_time == "", "false", "true")
/opt/splunk/etc/apps/TA-cisco_ios/default/props.conf EVAL-src_int = replace(src_int, "(\S+)\s(\d+)", "\1\2")
/opt/splunk/etc/apps/TA-cisco_ios/default/props.conf EVAL-src_mac = case(src_mac == "Unknown MAC", NULL, isnotnull(src_mac), lower(replace(src_mac,"^([0-9a-fA-F]{2})([0-9a-fA-F]{2}).([0-9a-fA-F]{2})([0-9a-fA-F]{2}).([0-9a-fA-F]{2})([0-9a-fA-F]{2})","\1:\2:\3:\4:\5:\6")))
/opt/splunk/etc/apps/TA-cisco_ios/default/props.conf EVAL-vendor = "Cisco"
/opt/splunk/etc/apps/TA-cisco_ios/default/props.conf EXTRACT-cisco-ios-BGP-3-IO_INIT = IO_INIT(\s)?:\s+Initialization failed: (?Failed accepting a replicated session) unable to find\s+nbr\s+*?(?\S+)*

and for the second:
/opt/splunk/etc/system/default/transforms.conf MV_ADD = False
/opt/splunk/etc/apps/Splunk_SA_CIM/default/transforms.conf REGEX = ^.\/mod(?:alert|workflow).log$
/opt/splunk/etc/apps/Splunk_SA_CIM/default/transforms.conf SOURCE_KEY = MetaData:Source
/opt/splunk/etc/system/default/transforms.conf WRITE_META = False
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf [force_sourcetype_cisco_traceback]
/opt/splunk/etc/system/default/transforms.conf CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf DEFAULT_VALUE =
/opt/splunk/etc/system/default/transforms.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf DEST_KEY = MetaData:Sourcetype
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf FORMAT = sourcetype::cisco:ios:traceback
/opt/splunk/etc/system/default/transforms.conf KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf MV_ADD = False
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf REGEX = -Traceback=
/opt/splunk/etc/system/default/transforms.conf SOURCE_KEY = _raw
/opt/splunk/etc/system/default/transforms.conf WRITE_META = False
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf [force_sourcetype_for_cisco_ios]
/opt/splunk/etc/system/default/transforms.conf CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf DEFAULT_VALUE =
/opt/splunk/etc/system/default/transforms.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf DEST_KEY = MetaData:Sourcetype
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf FORMAT = sourcetype::cisco:ios
/opt/splunk/etc/system/default/transforms.conf KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf MV_ADD = False
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf REGEX = (?:(?:\S+)\s)?(?:(?:\d+)?:\s(?:.\S+:\s)?(?:[.*])?(?:.+)?)?:\s+(?:%|#)(?:(?!POLICY_ENGINE|UCSM|FWSM|ASA|PIX|ACE)[A-Z0-9]+)-(?:(?:[A-Z012_](?:-?[A-Z_][^-]))-?)?(?:[0-7])-(?:[A-Z0-9_]+):(?:(?:[A-Za-z0-9_]+):)?\s(?:.+)
/opt/splunk/etc/system/default/transforms.conf SOURCE_KEY = raw
/opt/splunk/etc/system/default/transforms.conf WRITE_META = False
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf [force_sourcetype_for_cisco_ios-rfc5424]
/opt/splunk/etc/system/default/transforms.conf CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf DEFAULT_VALUE =
/opt/splunk/etc/system/default/transforms.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf DEST_KEY = MetaData:Sourcetype
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf FORMAT = sourcetype::cisco:ios
/opt/splunk/etc/system/default/transforms.conf KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf MV_ADD = False
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf REGEX = (?:<(?:\d+)>)(?:\d+) (?:\S+) (?:\S+)? (?:\d+)\s+(?:\S+)\s+(?:\S+)(?:.+)?:\s+(?:%|#)(?:(?!POLICY_ENGINE|UCSM|FWSM|ASA|PIX|ACE)[A-Z0-9]+)-(?:(?:[A-Z0-2_](?:-?[A-Z_][^-]))-?)?(?:[0-7])-(?:[A-Z0-9_]+):\s(?:.+)
/opt/splunk/etc/system/default/transforms.conf SOURCE_KEY = raw
/opt/splunk/etc/system/default/transforms.conf WRITE_META = False
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf [force_sourcetype_for_cisco_ios-xe]
/opt/splunk/etc/system/default/transforms.conf CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf DEFAULT_VALUE =
/opt/splunk/etc/system/default/transforms.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf DEST_KEY = MetaData:Sourcetype
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf FORMAT = sourcetype::cisco:ios
/opt/splunk/etc/system/default/transforms.conf KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf MV_ADD = False
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf REGEX = (?:(?:\S+)\s)?(?:(?:\d+)?:\s(?:.\S+:\s)?(?:[.*])?(?:.+)?)?:\s+(?:%|#)(?:(?!POLICY_ENGINE|UCSM|FWSM|ASA|PIX|ACE)[A-Z0-9]+)-(?:(?:[A-Z012_](?:-?[A-Z_][^-]))-?)?(?:[0-7])-(?:[A-Z0-9_]+):(?:(?:[A-Za-z0-9_]+):)?\s(?:.+)
/opt/splunk/etc/system/default/transforms.conf SOURCE_KEY = raw
/opt/splunk/etc/system/default/transforms.conf WRITE_META = False
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf [force_sourcetype_for_cisco_ios-xr]
/opt/splunk/etc/system/default/transforms.conf CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf DEFAULT_VALUE =
/opt/splunk/etc/system/default/transforms.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf DEST_KEY = MetaData:Sourcetype
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf FORMAT = sourcetype::cisco:ios
/opt/splunk/etc/system/default/transforms.conf KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf MV_ADD = False
/opt/splunk/etc/apps/TA-cisco_ios/default/transforms.conf REGEX = (?:(?:\S+)\s)?(?:\d+):\s(?:(?:\S+)\s)?(?:(?:[A-Z]+)\/(?:\d+)\/(?:[A-Z0-9]+)\/(?:[A-Z0-9]+)):(?:.+)\s?:\s?(?:[A-Za-z0-9]+)[(?:\d+)]:\s+%(?:[A-Za-z0-9_]+)-(?:[A-Za-z0-9_]+)-(?:(?:[A-Za-z12_](?:-?[A-Za-z_][^-]))-?)?(?:[0-7])-(?:[A-Z0-9_]+)\s:\s(?:.+)
/opt/splunk/etc/system/default/transforms.conf SOURCE_KEY = _raw
/opt/splunk/etc/system/default/transforms.conf WRITE_META = False
/opt/splunk/etc/apps/Splunk_TA_nix/default/transforms.conf [fs_notification_change_type_lookup]
/opt/splunk/etc/system/default/transforms.conf CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf CLEAN_KEYS = True*

[FrankVl]

Ok, so there are four transforms that set that sourcetype:

force_sourcetype_for_cisco_ios
force_sourcetype_for_cisco_ios-rfc5424
force_sourcetype_for_cisco_ios-xe
force_sourcetype_for_cisco_ios-xr

You'll need to check how these transforms are triggered from props.conf, to see if any of that could accidentally apply to internal logs.

[FrankVl]

Given the sourcetype assigned, check the add-on related to cisco:ios and check for any faulty sourcetype overrides there.

Or in general, do a: /opt/splunk/bin/splunk btool props list --debug | grep "cisco:ios" -B 10 -A 10 and /opt/splunk/bin/splunk btool transforms list --debug | grep "cisco:ios" -B 10 -A 10 to find any props or transforms related to setting that sourcetype.

[tiagofbmm]

Run /opt/splunk/bin/splunk btool inputs list --debug | grep /var/log/splunk/ -B 10 -A 10

Check if these inputs are having a sourcetype different than normal being assigned.

[wralph_EPACN]

I checked this against my prod environment and it besides for the host name everything was the same.

[tiagofbmm]

@wralph_EPACN please accept an answer if it solved/helped it and upvote it. Otherwise let us know how can we help further

[wralph_EPACN]

This is part of the search I am doing, index=_internal source=license_usage.log type="Usage" splunk_server= earliest=-7d@d latest=@d. Some how the source type changed from splunkd to cisco:ios and I am wondering how this happened and how to change it back. As this report runs once a week I did not catch it till this past run where it failed to produce anything.

So the question could possibly be stated, how do i change the the licence_usage.log, and whatever else that uses splunkd as a sourcetype back to its default?

[tiagofbmm]

Nothing has changed. Internal index contains several sourcetypes, and you just need to search for the one you're looking for

[FrankVl]

What do you mean by the index changing its sourcetype?

[lakshman239]

There are number of sourcetypes that are present in _internal index [ all of them related core splunk]. Which sourcetype has changed and what's the change?