Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About gozulin
gozulin
Communicator
Member since:
05-14-2013
06-05-2020
Community Statistics
| Posts | 45 |
| Solutions | 2 |
| Karma Given | 12 |
| Karma Received | 18 |
| Member Since | 05-14-2013 |
Activity Feed
- Got Karma for Re: Search head's authentication credentials rejected by peer. I've re-added peers 4 times and this keeps happening!. 05-11-2022 10:58 AM
- Karma Re: Why does my Splunk indexer keep running out of space with my current indexes.conf? for richgalloway. 06-05-2020 12:48 AM
- Karma Re: Why does my Splunk indexer keep running out of space with my current indexes.conf? for richgalloway. 06-05-2020 12:48 AM
- Karma Re: Why does my Splunk indexer keep running out of space with my current indexes.conf? for Yasaswy. 06-05-2020 12:48 AM
- Karma Re: After upgrading from 5.0 to 6.4, getting a regex error issue at startup related to dashes, i think for skoelpin. 06-05-2020 12:48 AM
- Karma Re: After upgrading from 5.0 to 6.4, getting a regex error issue at startup related to dashes, i think for woodcock. 06-05-2020 12:48 AM
- Karma Re: Search head's authentication credentials rejected by peer. I've re-added peers 4 times and this keeps happening! for woodcock. 06-05-2020 12:48 AM
- Got Karma for Search head's authentication credentials rejected by peer. I've re-added peers 4 times and this keeps happening!. 06-05-2020 12:48 AM
- Got Karma for Can I replace Splunk Universal Forwarder with Apache NiFi?. 06-05-2020 12:48 AM
- Got Karma for Re: Search head's authentication credentials rejected by peer. I've re-added peers 4 times and this keeps happening!. 06-05-2020 12:48 AM
- Got Karma for Re: Search head's authentication credentials rejected by peer. I've re-added peers 4 times and this keeps happening!. 06-05-2020 12:48 AM
- Got Karma for Re: Search head's authentication credentials rejected by peer. I've re-added peers 4 times and this keeps happening!. 06-05-2020 12:48 AM
- Got Karma for Re: Search head's authentication credentials rejected by peer. I've re-added peers 4 times and this keeps happening!. 06-05-2020 12:48 AM
- Got Karma for Re: Search head's authentication credentials rejected by peer. I've re-added peers 4 times and this keeps happening!. 06-05-2020 12:48 AM
- Got Karma for Search head's authentication credentials rejected by peer. I've re-added peers 4 times and this keeps happening!. 06-05-2020 12:48 AM
- Got Karma for How to rename an index in Splunk 6?. 06-05-2020 12:47 AM
- Got Karma for How to rename an index in Splunk 6?. 06-05-2020 12:47 AM
- Got Karma for How to rename an index in Splunk 6?. 06-05-2020 12:47 AM
- Got Karma for How to rename an index in Splunk 6?. 06-05-2020 12:47 AM
- Karma Pass variable to a scripted alert for Valky. 06-05-2020 12:46 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 2 | |||
| 1 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 4 | |||
| 0 | |||
| 0 | |||
| 0 |
12-19-2018
02:32 PM
I have a folder for each day. I would like to know how much data is in there, by day, so I can then export the results to a CSV file and track data growth,
... View more
06-01-2018
03:26 AM
We're working with mapr to identify the problem. They did their own install and had the same issue so at least it's easy to reproduce.
... View more
05-24-2018
10:53 AM
Hi rdagan, we're using splunk 7.10.
:8088/conf doesn't work. that port is closed.
https on port 8089 works as expected but /conf is not found.
:8089/services/configs is the closest thing i could find and i don't see any Yarn stuff.
Running stats count gives us a slightly different (and shorter) error:
2 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors.
[maproly] Error while running external process, return_code=255. See search.log for more info
[maproly] Exception - java.io.IOException: Error while waiting for MapReduce job to complete, job_id=job_1525914386605_0005, state=FAILED, reason=Application application_1525914386605_0005 failed 2 times due to AM Container for appattempt_1525914386605_0005_000002 exited with exitCode: -1000
... View more
Thanks to Woodcock, So I just realized something. We added 2 searchheads to the cluster in a 24h span. splsearch01.domain1.com and splsearch01.domain2.com
As you can see, BOTH searchheads’s hostname is splsearch01 , so these problems arise because they kept overwriting each other’s pem files so only one searchhead could be authenticated at any given time. None of this was logged as an error or warning by splunk…so it was effectively invisible.
In other words, this was all caused by the COINCIDENCE that both searchheads we just installed shared the same name because splunk didn’t go and get an FQDN (fully qualified domain name) by default when it was installed. It just used whatever the output of the command “hostname” was and left it at that. That’s right. It didn’t use a unique identifier. It assumed the output of the command hostname would be unique across all splunk instances….why splunk, why?
I went and manually added the fqdn to server.conf on both searchheads so splunk is identifying them by their UNIQUE fqdn, and I’ve verified that the pem files are now be stored in DIFFERENT directories so no pem files will be overwritten.
Going forward, we’ll have to take this into account and rename each host with the fqdn (hostname bla.xxx.cequintecid.com) before installing splunk. (because splunk won't do this for us...it could have easily done so using facter...) This should prevent similar situations from ever arising again.
... View more
Wow..you answer made me realize something. we added 2 searchheads almost simultaneously, which were named sh1.aqx.com and sh1.oly.com. Splunk in it's wisdom only used the first part to identify them in server.conf...both were named sh1.
So my guess is each time I added the cluster to one of them, its pem overwrote the existing pem under /distServerKeys/sh1/ which obviously caused the re-authentication for the one whose pem was overwritten to fail.
This could all have been avoided if splunk used a unique identifier (such as an IP or a FQDN) when deciding what to call itself, instead of just assuming the output of the command "hostname" would be unique across the entire environment.
... View more
03-24-2017
05:41 AM
I've already performed the steps you describe to clean the config and re-add the cluster masters. When I say I re-added the peers, i mean i did it by adding the CM. Not manually. That didn't help. ntp is enabled and this is not a load issue. the indexers have light workloads. any other ideas?
... View more
03-23-2017
09:14 PM
if i use the wrong password when adding a searchhead to a cluster, it doesn't let me add it to begin with. full stop. I also mentioned it works fine for a while after I restart splunk. This would not happen if the secret key were different, is it not so?
In which files, and which stanzas am I supposed to check if the secret keys are the same?
... View more
We've migrated some search heads, i've deleted the indexer peers, readded the cluster masters and everything looks fine and healthy ...until suddenly it doesn't.
I've read the installation guide and i don't see anything wrong with what I've done.
I've gone inside server.conf in the search heads and uncommented the distributed search servers. I've gone through the gui and added the cluster masters for the 2 clusters to be searched.
In the gui, under distributed search, I see this (for all 4 indexers):
Error [00000010] Instance name "splunk01.domain.com" Search head's authentication credentials rejected by peer. Try re-adding the peer. Last Connect Time:2017-03-24T03:43:41.000+00:00; Failed 10 out of 10 times.
and under indexer clustering, everything looks hunky dory:
Cluster Master Searchable Search Factor Replication Factor Status Actions
https://10.1.2.20:8089 All Data is Searchable Met Met
And here's the kicker: Restarting splunk FIXES it temporarily! but less than an hour later, the problem resurfaces...
I don't know causes this and i'm starting to think it's a bug, because i've triple checked my configs and everything looks fine...
Any clues?
... View more
03-15-2017
12:05 PM
1 Karma
NiFi has a putSplunk processor that should do what I want (send data to an indexer)
BUT it doesn't have any place for me to specify sourcetype, or index, and it only has one "Host" field, whereas I usually use autolb with 2 indexers.
Can I do this? If so, how?
Thanks!
... View more
08-12-2016
05:46 AM
how is it cleaner/easier to upgrade first and then cluster?
... View more
08-11-2016
11:11 AM
We have 2 indexers (one site) that are running on Redhat 6.2 that we want to upgrade to 6.7 for security reasons.
We also want to cluster them.
Should we cluster first, then upgrade the OS on one indexer at a time, or upgrade them, then cluster them?
Which is less risky?
Currently, all our forwarders are configured so they can send to both Indexers, like so:
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = 10.1.1.2:9997 , 10.1.1.3:9997
useACK=true
[tcpout-server://10.1.1.2:9997]
[tcpout-server://10.1.1.3:9997]
So either way, stopping one indexer should make the forwarders send to the other indexers, cluster or no cluster, am I right?
The only difference I can see is if we cluster first, search results would not be impacted during our maintenance window.
So, what say thee?
... View more
07-20-2016
12:07 PM
Don't have text to give because I don't know what it does. You can't tell what it's supposed to do either?
... View more
07-20-2016
12:01 PM
Why are you writing <NAME> instead of <foo-bar> ? I'm confused.
... View more
07-20-2016
11:58 AM
When you say "wrap the entire expression in parenthesis", do you mean like this?
`((?i) .*? (?P <foo-bar> [a-f0-9]+-[a-f0-9]+-[a-f0-9]+-[a-f0-9]+-[a-f0-9]+)(?= ))`
Because that did not work.
I don't know why it's not being extracted at search time, I am not the engineer who wrote this config, i'm just doing the migration and trying to fix any syntactic mistakes in the process. I am not testing my expression by restarting anything, I am testing it in https://regex101.com/ which also complains about it, so I'm assuming that it has the same issue splunk has with it, and if I solve it there, splunk will take it too.
... View more
07-20-2016
11:46 AM
There are no spaces in the actual query. I just don't know enough about how to post code on this website that won't mess with my formatting, as it is, if i write anything wrapped in < and > here without spaces, the entire string disappears from my post! (annoying as hell)
Here is a picture so you can see:
http://imgur.com/a/670j4
There is a parenthesis after the lookahead, right after the last "[a-f0-9]+" string. Isn't there? That's what puzzles me.
... View more
07-20-2016
11:39 AM
The last parenthesis was a type, this is the full query. There are 4 parentheses total:
Remove the spaces before and after the foo-bar string to get my actual text:
EXTRACT-foo-bar = (?i) .*? (?P < foo-bar > [a-f0-9]+-[a-f0-9]+-[a-f0-9]+-[a-f0-9]+-[a-f0-9]+)(?= )
... View more
07-20-2016
11:31 AM
That's exactly how I have it, only instead of the string NAME, it's the string foo-bar .
How are you able to type greater than and less than symbols with a word in between. When I do it, they and the word inside them disappear when I post my comment.
Lemme try this again: Remove the spaces before and after the foo-bar string to get my actual text:
EXTRACT-foo-bar = (?i) .*? (?P< foo-bar >[a-f0-9]+-[a-f0-9]+-[a-f0-9]+-[a-f0-9]+-[a-f0-9]+)(?= )
... View more
07-20-2016
11:26 AM
The actual regex is altered when I copy paste it...There is a [foo-bar] after the P. (replace the square brackets with "Less Than" and "Greater Than" symbols . It's just not showing up because of the way this website parses comments...
Update: figured out how to insert code...
EXTRACT-foo-bar = (?i) .*? (?P<foo-bar>[a-f0-9]+-[a-f0-9]+-[a-f0-9]+-[a-f0-9]+-[a-f0-9]+)(?= )
... View more
07-20-2016
11:17 AM
Bad regex value: '(?i) .*? (?P<foo-bar>\[a-f0-9]+\-[a-f0-9]+\-[a-f0-9]+\-[a-f0-9]+\-[a-f0-9]+)(?= )', of param: props.conf / [wsp-prod] / EXTRACT-foo-bar; why: unrecognized character after (?P
Bad regex value: '(?i) .*? (?P[a-f0-9]+-[a-f0-9]+-[a-f0-9]+-[a-f0-9]+-[a-f0-9]+)(?= )', of param: props.conf / [wsp-prod] / EXTRACT-foo-bar; why: unrecognized character after (?P
the regex is:
EXTRACT-foo-bar = (?i) .*? (?P<foo-bar>[a-f0-9]+-[a-f0-9]+-[a-f0-9]+-[a-f0-9]+-[a-f0-9]+)(?= )
Thanks!
... View more
02-25-2016
09:39 AM
Problem solved thanks to Yasaswy and Rich.
Reducing the maxWarmDBCount was the solution.
... View more
02-25-2016
09:37 AM
maxWarmDBCount is very effective. Thanks! Dialing it down to 50 should do the trick. Thanks.
I don't understand how maxHotSpanSecs would help, I have 8 indexes and the fastest-growing ones roll over at around 6GB. Could you explain?
... View more
02-25-2016
07:49 AM
The indexer pauses indexing when free space goes under 5GB on the main partition. This is caused by too many warm buckets filling up space instead of being moved to cold buckets on the larger, spinning-disk volume.
... View more
02-24-2016
01:04 PM
The indexer pauses indexing when free space goes under 5GB on the main partition. This is caused by too many warm buckets filling up space instead of being moved to cold buckets on the larger, spinning-disk volume.
I have a 3.2TB volume for hot/warm data on SSD and a 12TB volume for cold data on spinning disk. This is my indexes.conf. What am I doing wrong?
#general
maxWarmDBCount = 300
homePath.maxDataSizeMB = 3200000
coldPath.maxDataSizeMB = 12000000
#Volumes
[volume:caliente]
path = /splunkdata
maxVolumeDataSizeMB = 3200000
[volume:frio]
path = /cold
maxVolumeDataSizeMB = 12000000
# indexes
[_audit]
thawedPath = $SPLUNK_DB/audit/thaweddb
tstatsHomePath = volume:_splunk_summaries/audit/datamodel_summary
homePath = volume:caliente/splunk_indexes/audit/db
coldPath = volume:frio/_audit
[shenanigans]
thawedPath = $SPLUNK_DB/shenanigans/thaweddb
tstatsHomePath = volume:_splunk_summaries/shenanigans/datamodel_summary
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxDataSize = auto_high_volume
homePath = volume:caliente/splunk_indexes/shenanigans/db
coldPath = volume:frio/shenanigans
... View more
04-07-2015
04:27 PM
Thank you very much! That's a big help 🙂
... View more
