Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About GDustin
GDustin
Path Finder
Member since:
06-29-2017
11-04-2022
Community Statistics
| Posts | 38 |
| Solutions | 0 |
| Karma Given | 14 |
| Karma Received | 1 |
| Member Since | 06-29-2017 |
Activity Feed
- Posted Re: Looking for a Splunk trainer or teacher on San Antonio User Group. 11-04-2022 10:48 AM
- Karma Re: Going to try to reconstitute this group for Atriarc. 11-04-2022 10:43 AM
- Karma Going to try to reconstitute this group for Not_Greg. 11-04-2022 10:42 AM
- Posted Re: Splunk trainer or teacher wanted on San Antonio User Group. 11-04-2022 10:39 AM
- Karma Python errors when upgrading Splunk_TA_New_Relic to 2.2.0 for spavin. 10-05-2022 07:33 AM
- Karma Connection aborted error in Splunk's Microsoft log analytics add on application for khyatim. 11-20-2021 01:43 PM
- Karma Re: Connection aborted error in Splunk's Microsoft log analytics add on application for jwalzerpitt. 11-20-2021 01:42 PM
- Posted Re: Why do we get errors on the REST command in the Investigation Overview dashboard on Splunk ES? on Splunk Enterprise Security. 04-23-2021 01:32 PM
- Karma Splunk Security Essential not loading correctly for promisingbank. 06-05-2020 12:51 AM
- Karma How to configure Splunk App for Web Analytics if there are Multiple subdomains in the source and coming from same host ? for damode. 06-05-2020 12:50 AM
- Karma SEP 15 cloud on Symantec Integrated Cyber Defense Manager for Bentash. 06-05-2020 12:50 AM
- Karma How to collect event log from SEPC? for holm_arsene. 06-05-2020 12:50 AM
- Karma SEP 14.2 RU1 log format change for jtwind_2. 06-05-2020 12:50 AM
- Karma User profile web UI option "Restart background jobs when the Splunk software is restarted" what does it actually do? for Lucas_K. 06-05-2020 12:50 AM
- Karma Symantec Cloud Scripted Input for smaat11. 06-05-2020 12:49 AM
- Karma Re: Symantec Cloud Scripted Input for smaat11. 06-05-2020 12:49 AM
- Karma How do I adjust timezone settings for Cisco WSA data to set up accurate alerting? for kearaspoor. 06-05-2020 12:47 AM
- Got Karma for Re: In a search head cluster, how do I search users that have logged in the last 30 minutes, on what cluster member, and what kind of searches they are running?. 06-05-2020 12:47 AM
- Posted Re: Splunk Security Essential not loading correctly on All Apps and Add-ons. 04-13-2020 11:57 PM
- Posted Re: Analysis Of SplunkBase Apps for Splunk; py appears to run well but get alot of errors on scheduled runs on All Apps and Add-ons. 02-12-2020 09:50 PM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 0 |
11-04-2022
10:48 AM
If you email me we can discuss live assistance with subject matter review or live exercises. I've been running a fairly big completely distributed cluster with community, Splunk Inc., and Premium add-ons for years; 90%security/10%ops focus. DustinMcCoyEE@gmail.com ~NorthEast SATX Update: ...Oh yeah, you got all the certs; LMK if you ever want to brainstorm actuals ~live.
... View more
11-04-2022
10:39 AM
Do you have 2 services in play only or distributed? UF/HF forwarding~inputs.conf->outputs.conf (egress TCP/UDP ethereal) IDX reception~inputs.conf (ingress TCP ~9997 default) Deployment Server /Forwarder Management ~serverclass.conf[for above or distributed multiples of above] (ingress TCP 8089 default)
... View more
04-23-2021
01:32 PM
Having the same problem, we can walk right up to 7 days then it starts bricking, it seemed to work fine right out of the box, at first then I think once we broke 7 days age it went away; 6.0.1 variations trying to inspect: | `investigations` all=true earliest="-8d" latest="now" Failed to fetch REST endpoint uri=https://127.0.0.1:8089/services/storage/investigation/investigation?count=0&all=true&earliest=-8d&latest=now from server https://127.0.0.1:8089. Check that the URI path provided exists in the REST API.
... View more
04-13-2020
11:57 PM
me too; v.3.1.0
... View more
02-12-2020
09:50 PM
I had 5.0.2 no go 7.3.2; upgraded to 5.1.1; no change
... View more
02-12-2020
09:45 PM
Any Ideas how to fix?
The code appears to run ok,
/opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py
[I see lots of json meta for apps start flashing in]
But if i let the schedules run I am finding some info like this every 4 hours;
cat /opt/splunk/var/log/splunk/splunkd.log|grep -i getSplunkAppsV1
02-13-2020 03:34:17.455 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" Traceback (most recent call last):
02-13-2020 03:34:17.456 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" File "/opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py", line 90, in
02-13-2020 03:34:17.456 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" if name == "main": main()
02-13-2020 03:34:17.456 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" File "/opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py", line 61, in main
02-13-2020 03:34:17.456 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" for app_json in iterate_apps(app_func):
02-13-2020 03:34:17.456 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" File "/opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py", line 51, in iterate_apps
02-13-2020 03:34:17.456 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" data = get_apps(limit, offset, app_filter) ### Download initial list of the apps
02-13-2020 03:34:17.456 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" File "/opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py", line 18, in get_apps
02-13-2020 03:34:17.456 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" data = json.load(urllib2.urlopen(url))
02-13-2020 03:34:17.456 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" File "/opt/splunk/lib/python2.7/urllib2.py", line 154, in urlopen
02-13-2020 03:34:17.456 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" return opener.open(url, data, timeout)
02-13-2020 03:34:17.456 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" File "/opt/splunk/lib/python2.7/urllib2.py", line 435, in open
02-13-2020 03:34:17.456 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" response = meth(req, response)
02-13-2020 03:34:17.456 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" File "/opt/splunk/lib/python2.7/urllib2.py", line 548, in http_response
02-13-2020 03:34:17.457 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" 'http', request, response, code, msg, hdrs)
02-13-2020 03:34:17.457 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" File "/opt/splunk/lib/python2.7/urllib2.py", line 473, in error
02-13-2020 03:34:17.457 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" return self._call_chain(*args)
02-13-2020 03:34:17.457 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" File "/opt/splunk/lib/python2.7/urllib2.py", line 407, in _call_chain
02-13-2020 03:34:17.457 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" result = func(*args)
02-13-2020 03:34:17.457 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" File "/opt/splunk/lib/python2.7/urllib2.py", line 556, in http_error_default
02-13-2020 03:34:17.457 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
02-13-2020 03:34:17.457 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/analysis_of_splunkbase_apps/bin/getSplunkAppsV1.py" urllib2.HTTPError: HTTP Error 503: Service Unavailable
... View more
11-25-2019
03:50 PM
*
https://docs.splunk.com/Documentation/Splunk/6.2.4/Troubleshooting/Usebtooltotroubleshootconfigurations
... View more
11-25-2019
03:48 PM
look at the first link again; 3rd paragraph;
http://docs.splunk.com/Documentation/Splunk/6.2.4/Troubleshooting/Usebtooltotroubleshootconfigurations
"To view current in-memory configurations, query the REST endpoint /services/properties/."
FYI: the config file parameter is the stanza name inside square brackets [] inside the conf file;
with the rest call you have to URL encode the config file parameter/stanza name;
example:
/opt/splunkforwarder/bin/splunk show config inputs list monitor:///var/syslog-ng/log/ironport_xxx
Becomes:
/opt/splunkforwarder/bin/splunk _internal call /services/properties/inputs/monitor%3A%2F%2F%2Fvar%2Fsyslog-ng%2Flog%2Fironport_xxx
one way you can reference the url encoding by walking up one up in the REST call and then inspect and copy/paste your target quickly from the parameter; [it appears to come out in json; id]
/opt/splunkforwarder/bin/splunk _internal call /services/properties/inputs
/opt/splunkforwarder/bin/splunk _internal call /services/properties/inputs |grep -i ironport
... View more
11-25-2019
03:36 PM
show says specified conf file, not what is currently running;
Example:
/opt/splunkforwarder/bin/splunk help show |grep -i config
show config show the details of a specified conf file. (NOTE: this command will only work if the file exists in the location specified by $SPLUNK_HOME/etc/system/default/conf.conf)
... View more
11-25-2019
03:31 PM
that first little link in the comments has/appears to have had it;
https://docs.splunk.com/Documentation/Splunk/8.0.0/Troubleshooting/Usebtooltotroubleshootconfigurations
"To view current in-memory configurations, query the REST endpoint /services/properties/"
... View more
10-03-2019
09:42 PM
What is criteria for "the optimal number"; threshold for the highest number then back off?
... View more
08-27-2019
02:24 AM
@csperry; False alarm, yes props method looks good and gets our dashboards populating again; Thanks.
`
symantec:ep:security:file : EXTRACT-security_local_ip
Inline Local\sHost\sIP:\s+(? \d[^\,]+) No owner Splunk_TA_symantec-ep Global
[“Original”] Local:\s+(?\d[^\,]+)
[“No Go”] Local Host IP:\s+(?\d[^\,]+)
[“Go”] Local\sHost\sIP:\s+(?\d[^\,]+)
`
... View more
08-26-2019
10:27 PM
14.2RU1Mp1 (14.2.4814.1101).
https://support.symantec.com/us/en/article.TECH154475.html
Name Version/Build Release Date
(General Availability)
14.2.1.1 (14.2 RU1 MP1) 14.2.4814.1101 August 20, 2019
... View more
08-26-2019
04:53 PM
@csperry my guys just deployed 14.2RU1Mp1 any idea if that one is covered by your props method? Immediately my analysts told me; symantec:ep:security:file / [field_extraction_for_agt_security] went away; the only change I saw was "Local:" went to "Local Host IP:" in the raw logs, so I tried to rig some tests with | rex but it does not seem to fix all of it, but i am only just learning. If you have any insight or an update I'll buy you a cola and conf man. I'll see if I can put your props up on the SHC after hours tonight.
... View more
05-22-2019
04:14 PM
some of my code above is still bad/not displaying correctly; it is not presenting the characters "<" or ">", sorry for the confusion
... View more
05-21-2019
12:25 AM
So far worked in dev with a one-shot; looking good for prod;Thank you for sharing.
... View more
05-20-2019
11:57 PM
@rriegert see below, jt come through;
attempt to reduce confusion:
Paste old code in something like regex101
Take the smallest one as an example:
[field_extraction_for_agt_behavior]
^(?i)(?:[[sep_file_prefix]]),\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]),?\s*(?[[sep_file_field]])?,\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]),\s*(?:Begin:\s*(?[[sep_file_field]]))?,\s*(?:End:\s*(?[[sep_file_field]]))?,\s*(?:Rule:\s*(?[[sep_file_field]])),\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]),\s*(?:User:\s*(?[[sep_file_field]])),\s*(?:Domain:\s*(?[[sep_file_field]]))?,\s*(?:Action\sType:\s*(?[[sep_file_field]]))?(?:,\s*File\ssize\s(bytes):\s*(?[[sep_file_field]]),\s*Device\sID:\s*(?[[sep_file_field]]))?$
Get these errors:
All the errors detected are listed below, from left to right, as they appear in the pattern.
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
? The preceding token is not quantifiable
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
(? Incomplete group structure
) Incomplete group structure
The problem is when you paste a code such as:
?
Without the code option in the answers post, the:
get's filtered or removed;
Now; Take the smallest one as an example below; and go paste that one in regex101 or similar:
[field_extraction_for_agt_behavior]
^(?i)(?:[[sep_file_prefix]]),\s*(?<vendor_severity>[[sep_file_field]]),\s*(?<Host_Name>[[sep_file_field]]),?\s*(?<IP_Address>[[sep_file_field]])?,\s*(?<vendor_action>[[sep_file_field]]),\s*(?<Description>[[sep_file_field]]),\s*(?<API>[[sep_file_field]]),\s*(?:Begin:\s*(?<Begin_Time>[[sep_file_field]]))?,\s*(?:End:\s*(?<End_Time>[[sep_file_field]]))?,\s*(?:Rule:\s*(?<rule>[[sep_file_field]])),\s*(?<Caller_Process_ID>[[sep_file_field]]),\s*(?<Caller_Process_Name>[[sep_file_field]]),\s*(?<Return_Address>[[sep_file_field]]),\s*(?<Return_Module>[[sep_file_field]]),\s*(?<Parameter>[[sep_file_field]]),\s*(?:User:\s*(?<user>[[sep_file_field]])),\s*(?:Domain:\s*(?<Domain_Name>[[sep_file_field]]))?,\s*(?:Action\sType:\s*(?<Action_Type>[[sep_file_field]]))?(?:,\s*File\ssize\s\(bytes\):\s*(?<File_Size>[[sep_file_field]]),\s*Device\sID:\s*(?<Device_ID>[[sep_file_field]]))?$
...no errors
... View more
05-19-2019
10:29 AM
"Local time" where?
You specify your explicit local time in SH/SHC/SPL GUI service; "Account Setting>Time Zone"
Otherwise local time where; the source, sourcetransport, indexer, SH Servicer, etc
I don't care what timezone it is[Yes, I very much do care] but I just want it displayed in Splunk; I am constantly reviewing my account settings and having to sensitize users to review their their Account Setting>Time Zone for situational awareness. ISO standard is where no timezone then UTC-0 is assumed not the case in Splunk GUI; no timezone=Any host of settings; what ever is in the user's "Account Setting>Time Zone"; Splunk ingestion; no timezone=assumed UTC-0 - I want even playing field where Splunk eats it's dog food in the GUI with _time display.
... View more
05-17-2019
04:32 AM
this has a working example I've used a few times;
https://splunkbase.splunk.com/app/2975/
https://github.com/hire-vladimir/SA-syslog_collection/blob/master/default/inputs.conf
https://github.com/hire-vladimir/SA-syslog_collection/tree/master/bin
... View more
05-17-2019
04:14 AM
...Got reemed out by Enterprise "X" Management for breaking their iis "compliance" alerts, but gosh darnit we got SAWA up; ...end-users are just resorting to adhoc dashboards.
... View more
05-17-2019
03:57 AM
@jtwind_2
Can you repost with the Code sample option?
This is from the old 2.3.0 transforms[as an example];
(?i)(?:[[sep_file_prefix]]),\s*(?<Risk_Action>[[sep_file_field]]),\s*(?:IP\sAddress:\s*(?<IP_Address>[[sep_file_field]]))?,\s*(?:Computer\sname:\s*(?<Computer_Name>[[sep_file_field]]))?,?\s*(?:Intensive\sProtection\sLevel:\s*(?<Intensive_Protection_Level>[[sep_file_field]]))?,?\s*(?:Certificate\sissuer:\s*(?<Certificate_Issuer>[[sep_file_field]]))?,?\s*(?:Certificate\ssigner:\s*(?<Certificate_Signer>[[sep_file_field]]))?,?\s*(?:Certificate\sthumbprint:\s*(?<Certificate_Thumbprint>[[sep_file_field]]))?,?\s*(?:Signing\stimestamp:\s*(?<Signing_Timestamp>[[sep_file_field]]))?,?\s*(?:Certificate\sserial\snumber:\s*(?<Certificate_Serial_Number>[[sep_file_field]]))?,\s*(?:Source:\s*(?<Source>[[sep_file_field]]))?,\s*(?:Risk\sname:\s*(?<Risk_Name>[[sep_file_field]]))?,\s*(?:Occurrences:\s*(?<Occurrences>[[sep_file_field]]))?,\s*(?<file_path>[[sep_file_field]]),\s*(?<Description>[[sep_file_field]]),\s*(?:Actual\saction:\s*(?<vendor_action>[[sep_file_field]]))?,\s*(?:Requested\saction:\s*(?<Requested_Action>[[sep_file_field]]))?,\s*(?:Secondary\saction:\s*(?<Secondary_Action>[[sep_file_field]]))?,\s*(?:Event\stime:\s*(?<Event_Time>[[sep_file_field]]))?,\s*(?:Inserted:\s*(?<Event_Insert_Time>[[sep_file_field]]))?,\s*(?:End:\s*(?<End_Time>[[sep_file_field]]))?,\s*(?:Last\supdate\stime:\s*(?<Last_Update_Time>[[sep_file_field]]))?,\s*(?:Domain:\s*(?<Domain_Name>[[sep_file_field]]))?,\s*(?:Group:\s*(?<Group_Name>[[sep_file_field]]))?,\s*(?:Server:\s*(?<Server_Name>[[sep_file_field]]))?,\s*(?<user>[[sep_file_field]]),\s*(?:Source\scomputer:\s*(?<Source_Computer_Name>[[sep_file_field]]))?,\s*(?:Source\sIP:\s*(?<Source_Computer_IP>[[sep_file_field]]))?,\s*(?:Disposition:\s*(?<Disposition>[[sep_file_field]]))?,\s*(?:Download\ssite:\s*(?<Download_Site>[[sep_file_field]]))?,\s*(?:Web\sdomain:\s*(?<Web_Domain>.*))?,\s*(?:Downloaded\sby:\s*(?<Downloaded_By>[[sep_file_field]]))?,\s*(?:Prevalence:\s*(?<Prevalence>[[sep_file_field]]))?,\s*(?:Confidence:\s*(?<Confidence>[[sep_file_field]]))?,\s*(?:URL\sTracking\sStatus:\s*(?<URL_Tracking_Status>[[sep_file_field]]))?,\s*(?<Unknown_Field>[[sep_file_field]]),\s*(?:First\sseen:\s*(?<First_Seen>[[sep_file_field]]))?,\s*(?:Sensitivity:\s*(?<Sensitivity>[[sep_file_field]]))?,\s*(?<Reason_For_White_Listing>[[sep_file_field]]),\s*(?:Application\shash:\s*(?<Application_Hash>[[sep_file_field]]))?,\s*(?:Hash\stype:\s*(?<Hash_Type>[[sep_file_field]]))?,\s*(?:Company\sname:\s*(?<Company_Name>.*))?,\s*(?:Application\sname:\s(?<Application_Name>[[sep_file_field]]))?,\s*(?:Application\sversion:\s*(?P<Application_Version>.*))?,\s*(?:Application\stype:\s*(?<Application_Type>[[sep_file_field]]))?,\s*(?:File\ssize\s\(bytes\):\s*(?<File_Size>[[sep_file_field]]))?(?:,\s*Category\sset:\s*(?<Category_Set>[[sep_file_field]]),\s*Category\stype:\s*(?<Category_Type>[[sep_file_field]]))?,?\s*(?:Location:\s*(?<Location>[[sep_file_field]]))?
... View more
05-17-2019
02:28 AM
what version does this work on?
I am getting this on 7.0.3;
Bad regex value: ... / REGEX; why: unrecognized character after (? or (?-
for all those stanzas
... View more
04-28-2019
11:05 AM
I saw a similar issue;
Key Errors:
[-failing tgt-]# tail /opt/splunk/var/log/splunk/splunkd.log -f
04-28-2019 17:18:50.639 +0000 ERROR S2SFileReceiver - event=onFileClosed replicationType=eJournalReplication bid=unix~564~3980C11F-0463-420B-8584-F58CF055EC0E state=eComplete src=B78B7685-1AEF-477F-B50C-BB65C1633777 bucketType=warm status=failed err="bucket is already registered, registered not as a streaming hot target (SPL-90606)"
04-28-2019 17:18:50.639 +0000 WARN S2SFileReceiver - event=processFileSlice bid=unix~564~3980C11F-0463-420B-8584-F58CF055EC0E msg='aborting on local error'
04-28-2019 17:18:50.699 +0000 WARN CMSlave - event=addTargetDone bid=unix~564~3980C11F-0463-420B-8584-F58CF055EC0E but we no longer have the bucket lets remove it from the master as well
04-28-2019 17:18:50.699 +0000 WARN CMSlave - deleting bucket=unix~564~3980C11F-0463-420B-8584-F58CF055EC0E but failed to delete, reason=unable to find bucket
04-28-2019 17:18:50.882 +0000 INFO CMRepJob - job=CMReplicationErrorJob bid=unix~564~3980C11F-0463-420B-8584-F58CF055EC0E failingGuid=14E1138E-A7E7-499E-A7AD-84BC5797B164 srcGuid=B78B7685-1AEF-477F-B50C-BB65C1633777 tgtGuid=14E1138E-A7E7-499E-A7AD-84BC5797B164 succeeded
[attempting least to most resistance affect]
First try;
enable Maintenance-mode; restart affected splunkd on IDXs
You can correlate the affected GUIDs by Master execution:
/opt/splunk/bin/splunk show cluster-status
[It gives output of hostname/GUID/Site]
disable Maintenance-mode
Second try:
Master>Bucket Status
Resync the non-tgt failing bucket
Third try;
Delete the non-tgt failing bucket
Fourth try;
Delete a copy[I felt ok deleting; “copy” where RF3]
Failing tgt was not an option;
~first did failing source, [bid_GUID]
~second-then there was another option for another IDX at another site, did that, and finally it stopped bouncing and the Bucket status error /fix-up finished; [srcGuid]
Reviewing now;Tailing splunkd on all the peers; initially I was only seeing paying attention to ERRORs WRT [tgtGuid] and [bid_GUID]; now that I am looking at it in review there is an INFO log identifying the [srcGuid]
[I have seen this many times, when IT pulls the plug on my precious IDX peers; but thats why we have 3:2][usually another restart of single components in maintenance mode will bring ‘em back in the game]
This was on a 4 peer Cluster with RF3 SF2 running 7.0.3
In my situation the affected tgt was bouncing in :
/opt/splunk/bin/splunk show cluster-status and MasterGUI>Settings>IndexerClustering
Searchable NO
Status Stopped
After IT Support applied some updates and rebooted my IDXs.
... View more
