About christeraustad

christeraustad · ‎06-09-2018

Hi jonaslar, Have you had some success in troubleshooting this? We still have big problems with the CPU usage hitting the roof, and causing some searches to take up to 20-30 minutes. Searches that took maybe a minute to complete before upgrading.

christeraustad · ‎06-06-2018

Here's a snapshot. From what I've seen, iowait has been fine since the upgrade. Just for giggles, I applied the configuration bundle once more. And now the cluster looks more like it should be. Will keep an eye on the load and see if it stabilizes. Snapshot: avg-cpu: %user %nice %system %iowait %steal %idle 91.62 0.00 5.31 0.12 0.00 2.95 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sdf 9.80 0.00 95.20 0 476 sdg 60.20 0.00 534.40 0 2672 sda 0.00 0.00 0.00 0 0 sdb 0.00 0.00 0.00 0 0 sdd 0.00 0.00 0.00 0 0 sde 0.00 0.00 0.00 0 0 sdc 0.00 0.00 0.00 0 0 dm-0 0.00 0.00 0.00 0 0 dm-1 92.00 0.00 534.40 0 2672 EDIT: Nope. After a while, the CPU usage is back up at close to 100%.

christeraustad · ‎06-06-2018

It looks like most of the searches has a longer run time. None that truly stands out. Some of the searches that take up to 10 minutes are simple searches that should return within seconds.

christeraustad · ‎06-06-2018

Well. There is a lot of warnings from the DispatchManager: WARN DispatchManager - enforceQuotas: username=USERNAME, search_id=SID - QUEUED (reason='The maximum number of concurrent historical searches for this user based on their role quota has been reached. concurrency_limit=3') But my guess is that's because of the longer runtime of the searches. So when loading a dashboard with multiple searches that limit will be reached pretty quickly.

christeraustad · ‎06-06-2018

Hi, We upgraded our Splunk cluster last Friday. We have 1 Master node, 3 Search Heads in a search head cluster, 2 Search Peers in an indexer cluster and 2 HFs in front of the Search Peers. All have been upgraded to 7.1.1. The SH and HF were previously running 6.5.0 and the Search Peers were running 6.3.2. After the upgrade, the CPU usage has increased significantly. During office hours the CPU usage peaks at 100% on the Search Peers. This was not the case before the upgrade. In the "READ THIS FIRST" I can't see anything that should relate to our deployment in terms of increasing the CPU load. I know that we have some searches that should be optimized to limit the usage of subsearches. But it was working ok before the upgrade, so I can't see why upgrading should have such a huge impact. From what I can see in the DMC it looks like all searches use more CPU than before. But no rotten apples that I can identify. Can anyone help me out on what I should look for when troubleshooting?

christeraustad · ‎05-24-2018

Hi, You need to prepend sourcetype:: in the FORMAT value. [set:a:b:c] DEST_KEY = MetaData:Sourcetype REGEX = (%VOIP) FORMAT = sourcetype::a:b:c [set:a:b:c:d] DEST_KEY = MetaData:Sourcetype REGEX = (VOIP_CALL_STATISTICS)|(DSP) FORMAT = sourcetype::a:b:c:d

christeraustad · ‎09-26-2017

Hi, Splunk version: Splunk Enterprise 6.4.1 OS: Linux CentOS 7 We have a standalone Splunk enterprise where the license will soon expire. As we also have a distributed Splunk implementation with a lot of spare license on. We want to move one of our 5GB licenses from the distributed depolyment to the standalone server. Here is what we have done. Copied out the license XML from the distributed deployment. Deleted the license on the distributed deployment Installed the license on the standalone server. Here is where the issue arises. When we install the license we get a success reply. But when we go back to the license overview the license hasn't been added. But it is saved in the /opt/splunk/etc/license/enterprise directory. And we can see it on the debug page "All license details". We have of course restarted Splunk after installing the new license. From what I understand, there should not be any problems moving a license from one server to another. Is there anything that I am missing? Or is this a bug?

christeraustad · ‎09-07-2016

Hi, We have a Splunk cluster where we have 1400 hosts with Universal Forwarders installed. These UFs are forwarding to two intermediate Heavy Forwarders using SSL and load balancing. The hosts aren't sending a lot of data. I would guess on average 3-4kbps. The problem we are seeing is that all of the hosts (UF) are experiencing SSL error 10054. Which basically means that the HF has dropped the connection. 09-07-2016 20:29:19.439 +0000 INFO TcpOutputProc - Connection to XX.XX.XX.XX:9997 closed. default Error in SSL_read = 10054, SSL Error = error:00000000:lib(0):func(0):reason(0) Has anyone experienced something similar? I guess I should mention that these hosts are connected to the network through a satellite link. Which means that latency and general network connectivity could also play a part in this.

christeraustad · ‎07-22-2016

But if you have already upgraded to 6.3. How do I regenerate new certificates with new dates?

Posts	9
Solutions	1
Karma Given	1
Karma Received	2
Member Since	‎02-25-2014

Online Status	Offline
Date Last Visited	‎02-08-2024 07:40 AM

CPU load on Search Peers increased significantly a...

Move license from cluster to standalone

Why are universal forwarders reporting "Error in S...

Re: CPU load on Search Peers increased significant...

Re: CPU load on Search Peers increased significant...

Re: CPU load on Search Peers increased significant...

Re: CPU load on Search Peers increased significant...

CPU load on Search Peers increased significantly a...

Re: Why does searching absolute sourcetype name re...

Move license from cluster to standalone

Why are universal forwarders reporting "Error in S...

Re: Upgrading my Splunk Enterprise 6.2.x to 6.3.x ...