Deployment Architecture

One of the Search head showing down in F5 load balancer, but both the search head process where up and running fine in server ?

Hemnaath
Motivator

Hi All, Currently I am facing the above issue, ours is distributed system with search head pooling configuration setup. Before the search head F5 load balancer is configured to balance the User traffic hitting the search head.
Splunk Version 6.0.3.

Issue -

I have validate the splunkd process running in both the search head instances, but still in F5 load balancer it showing down and also when one of the splunkd process were stopped its not switching to the other splunk instances automatically. But not sure what algorithm or configuration has been done in F5 side. But before going to F5 team I need to check from splunk side for this issue. So kindly guide me what are the configuration file should verified in Splunk.

Tags (1)
0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

Unfortunately you need to know exactly how the F5 is configured because it sounds like it isnt properly configured.

You should have a "front end" VIP listening on some port (usually 8000, but could be whatever you desire) with a backend pool of splunk search head instances listening on your splunk web port (usually 8000), and load balancing based on cookie persistence. You should be using the session_id_{splunk web port number} cookie for the application cookie based persistence.

You should also have health checks based on your splunk web port number.

View solution in original post

0 Karma

jkat54
SplunkTrust
SplunkTrust

You need to open another question for your new question. We are out of space here. Do you mind accepting my answer as the correct answer too?

0 Karma

Hemnaath
Motivator

Hi Jkat, could you guide me on this problem How to upgrade the Shared search head polling from 6.0.3 to 6.2.1 version? I have posted this in splunk.answers.com with steps but I did not get any response for this. it will be really helpful if you can throw some lights on this.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Unfortunately you need to know exactly how the F5 is configured because it sounds like it isnt properly configured.

You should have a "front end" VIP listening on some port (usually 8000, but could be whatever you desire) with a backend pool of splunk search head instances listening on your splunk web port (usually 8000), and load balancing based on cookie persistence. You should be using the session_id_{splunk web port number} cookie for the application cookie based persistence.

You should also have health checks based on your splunk web port number.

0 Karma

Hemnaath
Motivator

I have followed these Steps to rectify the issue as guide by jkat.
steps:
1) Stop the splunk service running in the search02 server by executing the command
/opt/splunk/bin
./splunk stop
2) Remove the partially created folder under the mount point /splunk_search_pool from search02 server by executing the command
rm -rf /splunk_search_pool
3) Use the below command to mount back to the correct mount point as it is in search01
mount -t nfs splunfs01:/opt/splunk_shp /splunk_search_pool
4) Then restart the splunk service by executing the below command
/opt/splunk/bin/
./splunk start
5) Need to check whether both the URL is working by hitting the respective URLS
https://search01.com:8443
https://search02.com:8443
Thanks in advance.

0 Karma

Hemnaath
Motivator

Hi Jkat, kindly guide me to know whether the below steps can be followed to fix this issue.
Steps :
1) Stop the splunk service running in the search02 server by executing the command
/opt/splunk/bin
./splunk stop
2) Remove the partially created folder under the mount point /splunk_search_pool from search02 server by executing the command
rm -rf /splunk_search_pool
3) Use the below command to mount back to the correct mount point as it is in search01
mount -t nfs splunfs01:/opt/splunk_shp /splunk_search_pool
4) Then restart the splunk service by executing the below command
/opt/splunk/bin/
./splunk start
5) Need to check whether both the URL is working by hitting the respective URLS
https://search01.com:8443
https://search02.com:8443
Thanks in advance.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Yes as long as you do step 3 on search head 2. I believe this will work.

0 Karma

Hemnaath
Motivator

thanks Jkat, since its in production Environment, I had to make double check before proceeding on this.
let me update once done.

0 Karma

jkat54
SplunkTrust
SplunkTrust

sure, I'm standing by.

0 Karma

Hemnaath
Motivator

Hi jkat, thanks for your effort. Now I could see both the search head are up and running with the same port number. F5 Team confirmed that both the search head are showing up. One more thing I wanted to confirm whether both the search head are insycn with each other and how much of load is hitting both the search head .

thanks in Advance.

0 Karma

Hemnaath
Motivator

sure I will hit the answer option for this problem.

0 Karma

Hemnaath
Motivator

Hi All thanks for your inputs on this issue, I had tried to access the URL individually but it failed, I am able to access one of the Splunk search head URL instance which is showing up in the F5 balancer.

https ://hostname.xxx.com:8443 --- splunk search head 1 (Status showing UP in F5)
https://hostname.xxx.com:8443 --- splunk search head 2 (Status showing down in F5)

I am getting page can not be displayed error

How / where to find the web port /URL configuration details for accessing the search head.

thanks in advance

0 Karma

jkat54
SplunkTrust
SplunkTrust

you can log onto the search head and run a few commands that will shed light.

./splunk list web-port

Will show what the current configured web port is

netstat -an

Will show if the server is indeed listening on that web port

./splunk btool web list --debug

Will help to show which web.conf file is setting the server port

0 Karma

Hemnaath
Motivator

thanks jkat54, after executing the below command on both the server, I got this output

Search head 1 :

./splunk btool web list --debug
/splunk_search_pool/etc/apps/ADMIN-all_search_heads/default/web.conf httpport = 8443
/splunk_search_pool/etc/apps/ADMIN-all_search_heads/default/web.conf trustedIP = 10.140.x.x,10.140.x.x,127.0.0.1,168.133.x.x,168.133.x.x,168.133.x.x,168.133.x.x,168.133.x.x,168.133.x.x

search head 2 :

./splunk btool web list --debug

/opt/splunk/etc/system/default/web.conf httpport = 8000

After executing the above commands, I could see that search head 1 is listening to the port 8443 and where as search head 2 listening to the port 8000.

when executed this url https://search head1.xxxx.com:8443 - able to access and login
where as for search head 2, i am able to access using http://search head2.xxxx.com:8000 but unable to login.

Similarly when execute nestat -an , I could see that search head is listening to the port 8443 where as the search head2 its not listening to any ports mentioned in web.conf file.

Kindly guide me how to fix this issue so that I can able to access both the search head and sync with each other.

thanks in advance.

0 Karma

Hemnaath
Motivator

Hi All, Can you guide us on this issue, as we need to make both the splunk search head up URL link up and running in our environment. Currently only one search head is accessible. thanks in advance.

0 Karma

jkat54
SplunkTrust
SplunkTrust

looks like search head 2 is down

Try these commands as the splunk user
/opt/splunk/bin/splunk status <- might tell you its not running
/opt/splunk/bin/splunk restart <- might tell you an error during the startup

Any reason why sh1 has sso enabled and runs on different port? Shouldnt they be in sync?

0 Karma

Hemnaath
Motivator

Hi Jkat thanks for your effort on this, I have checked both splunk status and found to be up and running fine. Even checked with the F5 team and they told me that its configured with Round Robin method and its listen to 8443.

Status of search head 1 (root user)
./splunk status
splunkd is running (PID: 26667).
splunk helpers are running (PIDs: 26668).
splunkweb is running (PID: 26702).

status of search head 2 (root user)

./splunk status
splunkd is running (PID: 31308).
splunk helpers are running (PIDs: 31309).
splunkweb is running (PID: 31372).

I am not sure why the search head is listening to different port. Can you guide me how to make this sync with the same port number.

thanks in advance.

0 Karma

Hemnaath
Motivator

Hi Jkat, Can you guide me how rectify this issue ? so that both the search head are in sync with each other. thanks in advance

0 Karma

jkat54
SplunkTrust
SplunkTrust

./splunk btool web list --debug

This will tell you the configuration Splunk will follow.

And this will explain why it's different on one search head despite having the same web.config app:

http://docs.splunk.com/Documentation/Splunk/6.4.3/Admin/Wheretofindtheconfigurationfiles

One of you configuration files on sh2 is overriding your web.config that is being deployed.

You should also have 127.0.0.1 in the list of trusted IPs on sh2, but first you got to fix your configuration issues.

0 Karma

Hemnaath
Motivator

Hi Jkat, thanks for sharing your inputs on this problem, I had run the btool command and found few directory where missing in search head 2 server under the below path.

Search head 1 :

/splunk_search_pool/etc/apps - Contain list of all apps that are configured for searching and splunk reads this file.

Search head 2:
/splunk_search_pool/etc/apps - There is no apps found in this path, I could see only one directory called learned and one file name sentinel.txt.

So kindly let me what will be the problem why the splunk is taking the least precedence in search head 2 " /opt/splunk/etc/system/default/web.conf"

0 Karma

jkat54
SplunkTrust
SplunkTrust

The /splunk_search_pool/etc/apps directory should be the same on both servers.

df -k should show a list of mounted volumes. The mount point for /splunk_search_pool/ should be the same NFS share/mount on both servers.

So first you need to remap SH2 to the correct NFS share. You may find that /splunk_search_pool has been created as a directory on SH2 and you may have to delete it before remapping the NFS drive.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...