10-13-2017 14:04:14.654 INFO dispatchRunner - initing LicenseMgr in search process: nonPro=0
10-13-2017 14:04:14.655 INFO dispatchRunner - registering build time modules, count=1
10-13-2017 14:04:14.655 INFO dispatchRunner - registering search time components of build time module name=vix
10-13-2017 14:04:14.655 INFO dispatchRunner - Splunkd starting (build aa7d4b1ccb80).
10-13-2017 14:04:14.655 INFO dispatchRunner - System info: Linux, splunk-search-head-test-01, 3.10.0-693.2.2.el7.x86_64, #1 SMP Sat Sep 9 03:55:24 EDT 2017, x86_64.
10-13-2017 14:04:14.656 INFO dispatchRunner - Detected 1 (virtual) CPUs, 1 CPU cores, and 975MB RAM
10-13-2017 14:04:14.656 INFO dispatchRunner - Maximum number of threads (approximate): 487
10-13-2017 14:04:14.656 INFO dispatchRunner - Arguments are: "search" "--id=1507917854.41" "--maxbuckets=0" "--ttl=600" "--maxout=500000" "--maxtime=8640000" "--lookups=1" "--reduce_freq=10" "--user=bryn" "--pro" "--roles=admin:user"
10-13-2017 14:04:14.656 INFO dispatchRunner - Getting search configuration data from: /opt/splunk/etc/modules/parsing/config.xml
10-13-2017 14:04:14.662 INFO BundlesSetup - Setup stats for /opt/splunk/etc: wallclock_elapsed_msec=24, cpu_time_used=0.021666, shared_services_generation=2, shared_services_population=1
10-13-2017 14:04:14.665 WARN AuthorizationManager - Capability 'delete_by_keyword' had value 'disable' - only 'enabled' is valid. Ignoring...
10-13-2017 14:04:14.665 WARN AuthorizationManager - Capability 'edit_view_html' had value 'disable' - only 'enabled' is valid. Ignoring...
10-13-2017 14:04:14.665 WARN AuthorizationManager - Capability 'list_httpauths' had value 'disable' - only 'enabled' is valid. Ignoring...
10-13-2017 14:04:14.665 WARN AuthorizationManager - Capability 'rtsearch' had value 'disable' - only 'enabled' is valid. Ignoring...
10-13-2017 14:04:14.666 WARN AuthorizationManager - Capability 'delete_by_keyword' had value 'disable' - only 'enabled' is valid. Ignoring...
10-13-2017 14:04:14.666 WARN AuthorizationManager - Capability 'edit_view_html' had value 'disable' - only 'enabled' is valid. Ignoring...
10-13-2017 14:04:14.666 WARN AuthorizationManager - Capability 'list_httpauths' had value 'disable' - only 'enabled' is valid. Ignoring...
10-13-2017 14:04:14.666 WARN AuthorizationManager - Capability 'rtsearch' had value 'disable' - only 'enabled' is valid. Ignoring...
10-13-2017 14:04:14.666 WARN AuthorizationManager - Capability 'delete_by_keyword' had value 'disable' - only 'enabled' is valid. Ignoring...
10-13-2017 14:04:14.666 WARN AuthorizationManager - Capability 'edit_view_html' had value 'disable' - only 'enabled' is valid. Ignoring...
10-13-2017 14:04:14.666 WARN AuthorizationManager - Capability 'rtsearch' had value 'disable' - only 'enabled' is valid. Ignoring...
10-13-2017 14:04:14.666 WARN AuthorizationManager - Capability 'schedule_search' had value 'disable' - only 'enabled' is valid. Ignoring...
10-13-2017 14:04:14.667 INFO UserManagerPro - Load authentication: forcing roles="admin, user"
10-13-2017 14:04:14.671 INFO SessionManager - auth tokens will be generated with shpooling shared secret
10-13-2017 14:04:14.671 INFO UserManager - Setting user context: splunk-system-user
10-13-2017 14:04:14.671 INFO UserManager - Done setting user context: NULL -> splunk-system-user
10-13-2017 14:04:14.672 INFO UserManager - Unwound user context: splunk-system-user -> NULL
10-13-2017 14:04:14.672 INFO UserManager - Setting user context: bryn
10-13-2017 14:04:14.672 INFO UserManager - Done setting user context: NULL -> bryn
10-13-2017 14:04:14.678 INFO dispatchRunner - search context: user="bryn", app="search", bs-pathname="/opt/splunk/etc"
10-13-2017 14:04:14.685 INFO SearchParser - PARSING: search index=*\n| chart count by splunk_server
10-13-2017 14:04:14.689 INFO ISplunkDispatch - Not running in splunkd. Bundle replication not triggered.
10-13-2017 14:04:14.700 INFO UserManager - Setting user context: bryn
10-13-2017 14:04:14.700 INFO UserManager - Done setting user context: NULL -> bryn
10-13-2017 14:04:14.725 INFO SearchProcessor - Building search filter
10-13-2017 14:04:14.725 INFO SearchProcessor - Final search filter= ( ( splunk_server=splunk-index-test-01* ) )
10-13-2017 14:04:14.733 INFO SearchOperator:kv - name=EXTRACT-GUID, can_use_re2=0, regex: (?i)(?!=\w)(?:objectguid|guid)\s*=\s*(?<guid_lookup>[\w\-]+)
10-13-2017 14:04:14.733 INFO SearchOperator:kv - name=EXTRACT-SID, can_use_re2=0, regex: objectSid\s*=\s*(?<sid_lookup>\S+)
10-13-2017 14:04:14.735 INFO SearchOperator:kv - name=ad-kv, can_use_re2=0, regex: (?<_KEY_1>[\w-]+)=(?<_VAL_1>[^\r\n]*)
10-13-2017 14:04:14.737 INFO SearchOperator:kv - name=access-extractions, can_use_re2=0, regex: ^(?P<clientip>\S+)\s++(?P<ident>\S+)\s++(?P<user>\S+)\s++\[(?<req_time>[^\]]*+)\]\s++"\s*+(?P<method>[^\s"]++)?(?:\s++(?<uri>(?:(?<uri_domain>\w++://[^/\s"]++))?+(?<uri_path>(?:/++(?<root>(?:\\"|[^\s\?/"])++)/++)?(?:(?:\\"|[^\s\?/"])*+/++)*(?<file>[^\s\?/]+)?)(?:\?(?<uri_query>[^\s]*))?)(?:\s++(?P<version>[^\s"]++))*)?\s*+"\s++(?P<status>\S+)\s++(?P<bytes>\S+)(?:\s++"(?<referer>(?:(?<referer_domain>\w++://[^/\s"]++))?+[^"]*+)"(?:\s++"(?<useragent>[^"]*+)"(?:\s++"(?<cookie>[^"]*+)")?+)?+)?(?P<other>.*)
10-13-2017 14:04:14.738 INFO SearchOperator:kv - name=syslog-extractions, can_use_re2=0, regex: \s([^\s\[]+)(?:\[(\d+)\])?:\s
10-13-2017 14:04:14.739 INFO SearchOperator:kv - name=db2, can_use_re2=0, regex: ([A-Z]+) *: (.*?)(?=\n|$| +[A-Z]+ *:)
10-13-2017 14:04:14.739 INFO SearchOperator:kv - name=EXTRACT-extract_spent, can_use_re2=0, regex: (?<spent>\d+)ms$
10-13-2017 14:04:14.740 INFO SearchOperator:kv - name=EXTRACT-1, can_use_re2=0, regex: (?<_KEY_1>\S+)::(?<_VAL_1>\S+)
10-13-2017 14:04:14.742 INFO SearchOperator:kv - name=bracket-space, can_use_re2=0, regex: \[(\S+) (.*?)\]
10-13-2017 14:04:14.744 INFO SearchOperator:kv - name=sendmail-extractions, can_use_re2=0, regex: sendmail\[(\d+)\]: (\w+):
10-13-2017 14:04:14.744 INFO SearchOperator:kv - name=tcpdump-endpoints, can_use_re2=0, regex: (\d+\.\d+\.\d+\.\d+):(\d+) -> (\d+\.\d+\.\d+\.\d+):(\d+)
10-13-2017 14:04:14.744 INFO SearchOperator:kv - name=colon-kv, can_use_re2=0, regex: (?<= )([A-Za-z]+): ?((0x[A-F\d]+)|\d+)(?= |\n|$)
10-13-2017 14:04:14.752 INFO SearchOperator:kv - name=EXTRACT-severity,logger, can_use_re2=0, regex: .*?(?<severity>[A-Z]+) ((?<logger>[^\s]+) \-)*
10-13-2017 14:04:14.753 INFO SearchOperator:kv - name=EXTRACT-collection,category,object, can_use_re2=0, regex: collection=\"?(?P<collection>[^\"\n]+)\"?\ncategory=\"?(?P<category>[^\"\n]+)\"?\nobject=\"?(?P<object>[^\"\n]+)\"?\n
10-13-2017 14:04:14.754 INFO SearchOperator:kv - name=wel-message, can_use_re2=0, regex: (?sm)^(?<_pre_msg>.+)\nMessage=(?<Message>.+)$
10-13-2017 14:04:14.754 INFO SearchOperator:kv - name=wel-col-kv, can_use_re2=0, regex: \n([^:\n\r]+):[ \t]++([^\n]*)
10-13-2017 14:04:14.755 INFO SearchOperator:kv - name=EXTRACT-useragent, can_use_re2=0, regex: userAgent=(?P<browser>[^ (]+)
10-13-2017 14:04:14.755 INFO SearchOperator:kv - name=splunk-service-extractions, can_use_re2=0, regex: (?i)^(?:[^ ]* ){2}(?P<log_level>[^\s]*)\s+\[(?P<requestid>\w+)]\s+(?P<component>[^ ]+):(?P<line>\d+) - (?P<message>.+)
10-13-2017 14:04:14.755 INFO SearchOperator:kv - name=EXTRACT-fields, can_use_re2=0, regex: (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+) - (?P<message>.+)
10-13-2017 14:04:14.755 INFO SearchOperator:kv - name=extract_spent, can_use_re2=0, regex: (?P<spent>\d+)ms$
10-13-2017 14:04:14.756 INFO SearchOperator:kv - name=weblogic-code, can_use_re2=0, regex: <BEA-([0-9]+)>
10-13-2017 14:04:14.756 INFO SearchOperator:kv - name=colon-line, can_use_re2=0, regex: ^(\w+)\s*:[ \t]*(.*?)$
10-13-2017 14:04:14.756 INFO SearchOperator:kv - name=was-trlog-code, can_use_re2=0, regex: ] ([a-fA-F0-9]{8})
10-13-2017 14:04:14.757 INFO UnifiedSearch - base lispy: [ AND index::* splunk_server::splunk-index-test-01* ]
10-13-2017 14:04:14.758 INFO UnifiedSearch - Processed search targeting arguments
10-13-2017 14:04:14.758 INFO SortOperator - maxmem = 209715200
10-13-2017 14:04:14.758 INFO SortOperator - maxmem = 209715200
10-13-2017 14:04:14.758 INFO SearchParser - PARSING: prestats count by splunk_server
10-13-2017 14:04:14.758 INFO SearchParser - PARSING: addinfo type=count label=prereport_events
10-13-2017 14:04:14.758 INFO DispatchThread - BatchMode: allowBatchMode: 1, conf(1): 1, timeline/Status buckets(0):0, realtime(0):0, report pipe empty(0):0, reqTimeOrder(0):0, summarize(0):0, statefulStreaming(0):0
10-13-2017 14:04:14.758 INFO DispatchThread - required fields list to add to remote search = prestats_reserved_*,psrsvd_*,splunk_server
10-13-2017 14:04:14.758 INFO SearchParser - PARSING: fields keepcolorder=t "prestats_reserved_*" "psrsvd_*" "splunk_server"
10-13-2017 14:04:14.763 INFO DispatchThread - Did not find a usable summary_id, setting info._summary_mode=none, not modifying input summary_id=49CAB615-276A-428B-972B-FC67E89AEB46_search_bryn_96102898428831f8
10-13-2017 14:04:14.765 INFO DispatchThread - Did not find a usable summary_id, setting info._summary_mode=none, not modifying input summary_id=49CAB615-276A-428B-972B-FC67E89AEB46_search_bryn_NScf8163cdac44f862
10-13-2017 14:04:14.766 INFO DispatchThread - Allow retry on peer failure
10-13-2017 14:04:14.766 INFO UserManager - Setting user context: bryn
10-13-2017 14:04:14.766 INFO UserManager - Done setting user context: bryn -> bryn
10-13-2017 14:04:14.766 INFO UserManager - Unwound user context: bryn -> bryn
10-13-2017 14:04:14.766 INFO DistributedSearchResultCollectionManager - Stream search: litsearch ( index=* ) ( ( splunk_server=splunk-index-test-01* ) ) | addinfo type=count label=prereport_events | fields keepcolorder=t "prestats_reserved_*" "psrsvd_*" "splunk_server" | prestats count by splunk_server
10-13-2017 14:04:14.766 INFO ExternalResultProvider - No external result providers are configured
10-13-2017 14:04:14.766 INFO DistributedSearchResultCollectionManager - Default search group:*
10-13-2017 14:04:14.766 INFO DistributedSearchResultCollectionManager - Connecting to peer splunk-index-test-01 connectAll 0 connectToSpecificPeer 1
10-13-2017 14:04:14.766 INFO DistributedSearchResultCollectionManager - Connecting to peer splunk-index-test-02 connectAll 0 connectToSpecificPeer 1
10-13-2017 14:04:14.766 INFO DistributedSearchResultCollectionManager - Connecting to peer splunk-index-test-03 connectAll 0 connectToSpecificPeer 1
10-13-2017 14:04:14.766 INFO DistributedSearchResultCollectionManager - Connecting to peer splunk-search-head-test-01 connectAll 0 connectToSpecificPeer 1
10-13-2017 14:04:14.766 INFO ServerConfig - Using REMOTE_SERVER_NAME=splunk-search-head-test-01
10-13-2017 14:04:14.767 INFO KeyManagerLocalhost - Checking for localhost key pair
10-13-2017 14:04:14.767 INFO KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem
10-13-2017 14:04:14.767 INFO KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
10-13-2017 14:04:14.767 INFO KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
10-13-2017 14:04:14.767 INFO KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
10-13-2017 14:04:14.767 INFO KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
10-13-2017 14:04:14.768 INFO DistributedSearchResultCollectionManager - Successfully created search result collector for peer=splunk-index-test-01 in 0.003000 seconds
10-13-2017 14:04:14.770 INFO DistributedSearchResultCollectionManager - Successfully created search result collector for peer=splunk-index-test-02 in 0.002000 seconds
10-13-2017 14:04:14.772 INFO DistributedSearchResultCollectionManager - Successfully created search result collector for peer=splunk-index-test-03 in 0.002000 seconds
10-13-2017 14:04:14.772 INFO DispatchThread - Disk quota = 10485760000
10-13-2017 14:04:14.772 INFO UserManager - Setting user context: bryn
10-13-2017 14:04:14.772 INFO UserManager - Done setting user context: NULL -> bryn
10-13-2017 14:04:14.772 INFO SearchParser - PARSING: litsearch ( index=* ) ( ( splunk_server=splunk-index-test-01* ) ) | addinfo type=count label=prereport_events | fields keepcolorder=t "prestats_reserved_*" "psrsvd_*" "splunk_server" | prestats count by splunk_server
10-13-2017 14:04:14.784 INFO UserManager - Setting user context: bryn
10-13-2017 14:04:14.784 INFO UserManager - Done setting user context: NULL -> bryn
10-13-2017 14:04:14.785 INFO UserManager - Setting user context: bryn
10-13-2017 14:04:14.785 INFO UserManager - Done setting user context: NULL -> bryn
10-13-2017 14:04:14.785 INFO UserManager - Setting user context: bryn
10-13-2017 14:04:14.785 INFO UserManager - Done setting user context: NULL -> bryn
10-13-2017 14:04:14.793 INFO UserManager - Setting user context: bryn
10-13-2017 14:04:14.793 INFO UserManager - Done setting user context: NULL -> bryn
10-13-2017 14:04:14.793 INFO UserManager - Setting user context: bryn
10-13-2017 14:04:14.793 INFO UserManager - Done setting user context: NULL -> bryn
10-13-2017 14:04:14.797 INFO SearchParser - PARSING: typer | tags
10-13-2017 14:04:14.798 INFO FastTyper - found nodes count: comparisons=6, unique_comparisons=5, terms=4, unique_terms=4, phrases=12, unique_phrases=12, total leaves=22
10-13-2017 14:04:14.801 INFO UnifiedSearch - Processed search targeting arguments
10-13-2017 14:04:14.801 INFO LocalCollector - Final required fields list = prestats_reserved_*,psrsvd_*,splunk_server
10-13-2017 14:04:14.801 INFO UserManager - Unwound user context: bryn -> NULL
10-13-2017 14:04:14.801 INFO UserManager - Setting user context: bryn
10-13-2017 14:04:14.801 INFO UserManager - Done setting user context: NULL -> bryn
10-13-2017 14:04:14.801 INFO UserManager - Unwound user context: bryn -> NULL
10-13-2017 14:04:14.801 WARN RetryManager - Peer: splunk-search-head-test-01 not found in offset map.
10-13-2017 14:04:15.108 INFO UserManager - Unwound user context: bryn -> NULL
10-13-2017 14:04:15.109 INFO UserManager - Unwound user context: bryn -> NULL
10-13-2017 14:04:15.109 INFO UserManager - Unwound user context: bryn -> NULL
10-13-2017 14:04:15.109 INFO UserManager - Unwound user context: bryn -> NULL
10-13-2017 14:04:15.109 INFO UserManager - Unwound user context: bryn -> NULL
10-13-2017 14:04:15.125 INFO UserManager - Unwound user context: bryn -> NULL
10-13-2017 14:04:15.128 INFO UserManager - Setting user context: bryn
10-13-2017 14:04:15.128 INFO UserManager - Done setting user context: NULL -> bryn
10-13-2017 14:04:15.128 INFO UserManager - Unwound user context: bryn -> NULL
10-13-2017 14:04:15.133 INFO DispatchThread - Downloading all remote search.log files took 0.005 seconds
10-13-2017 14:04:15.135 INFO DispatchManager - DispatchManager::dispatchHasFinished(id='1507917854.41', username='bryn')
10-13-2017 14:04:15.136 INFO UserManager - Unwound user context: bryn -> NULL
10-13-2017 14:04:15.136 INFO ShutdownHandler - Shutting down splunkd
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_Begin"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_JustBeforeKVStore"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_KVStore"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_Thruput"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpInput1"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpOutput"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_UdpInput"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_FifoInput"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_WinEventLogInput"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_HttpInput"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_Scheduler"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_Tailing"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_SyslogOutput"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_HTTPOutput"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_TailingXP"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_PeerManager"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_ArchiveAndOneshot"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_AuditTrailManager"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_AuditTrailQueueServiceThread"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_FSChangeMonitor"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_FSChangeManagerProcessor"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_HttpClientPollingThread"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_AsyncQueuedMessageDispatcherThread"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_OfflineFlusher"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_Slave"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_SlaveSearch"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_Captain"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_Select"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_IdataDO_Collector"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpOutput2"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_IndexerService"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_Database1"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_LastIndexerLevel"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpInput2"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_LoadLDAPUsers"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_MetricsManager"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_Pipeline"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_Queue"
10-13-2017 14:04:15.136 INFO ShutdownHandler - shutting down level "ShutdownLevel_Exec"
10-13-2017 14:04:15.137 INFO ShutdownHandler - shutting down level "ShutdownLevel_CallbackRunner"
10-13-2017 14:04:15.137 INFO ShutdownHandler - shutting down level "ShutdownLevel_HttpClient"
10-13-2017 14:04:15.137 INFO ShutdownHandler - Shutdown complete in 972 microseconds
... View more