Issues with thread management session?

[sloshburch]

I saw this issue on multiple browsers.

I have splunk fronted by four reverse proxy servers running IBM HTTP Server 7.0.0.17 (Apache 2.2.8) presenting SiteMinder as a SSO agent.

To debug/verify the setup on each individual web server, I navigate to http://.company.com:. This prompts me to log in as expected. I am then able to use any non search related feature in splunk. By that, I mean that I can navigate to the manager but if I navigate to search I am redirected to the native splunk login page.

I see this activity corresponding in the logs:

2013-02-11 09:10:00,976 INFO [5118fbb8ea4e379d0] cached:77 - memoized decorator used on function <function getEntities at 0x4258cf8> with non hashable arguments
2013-02-11 09:10:01,336 INFO [5118fbb9465007c90] cached:77 - memoized decorator used on function <function getEntities at 0x4258cf8> with non hashable arguments
2013-02-11 09:10:01,774 INFO [5118fbb9465007c90] view:1070 - PERF - viewTime=0.2188s templateTime=0.2196s
2013-02-11 09:10:03,112 ERROR [5118fbbb175115150] utility:125 - [HTTP 401] Client is not authenticated
Traceback (most recent call last):
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/utility.py", line 123, in parse_time
parsed = times.splunktime2Iso(ts)
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/times.py", line 173, in splunktime2Iso
serverStatus, serverResp = splunk.rest.simpleRequest('/search/timeparser', getargs=getargs)
File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 452, in simpleRequest
raise splunk.AuthenticationFailed
AuthenticationFailed: [HTTP 401] Client is not authenticated
2013-02-11 09:10:03,365 WARNING [5118fbbb59563d2d0] util:1255 - CSRF form_key mismatch received=14808273786252083278 expected=None
2013-02-11 09:10:03,367 WARNING [5118fbbb59563d2d0] decorators:87 - CSRF: validation failed because client XHR did not include proper header
2013-02-11 09:10:03,376 WARNING [5118fbbb5c50e0890] util:1255 - CSRF form_key mismatch received=14808273786252083278 expected=None
2013-02-11 09:10:03,378 WARNING [5118fbbb5c50e0890] decorators:87 - CSRF: validation failed because client XHR did not include proper header
2013-02-11 09:10:03,570 WARNING [5118fbbb8f4caa190] util:1255 - CSRF form_key mismatch received=14808273786252083278 expected=None
2013-02-11 09:10:03,570 WARNING [5118fbbb8f4caa190] decorators:87 - CSRF: validation failed because client XHR did not include proper header
2013-02-11 09:10:03,738 WARNING [5118fbbbb750eb310] util:1255 - CSRF form_key mismatch received=14808273786252083278 expected=None
2013-02-11 09:10:03,740 WARNING [5118fbbbb750eb310] decorators:87 - CSRF: validation failed because client XHR did not include proper header
2013-02-11 09:10:03,742 WARNING [5118fbbbb750eb310] decorators:95 - CSRF: skipping 401 redirect response because endpoint did not request protection

This does NOT occur when navigating to splunk with my dns/VIP which load balances amongst the four web servers with a sticky session.

I am aware that there is a domain requirement on my SOS policy that requires *.company.com.

I'm not sure how to phrase my question, or if I'm using the right terms. My SOS engineer suggests I research how the session is passed from thread to thread (given my interpretation that search spawns another thread).

Any idea why I am being redirected back to the login page?

[sloshburch · Accepted solution]

I found the issue - not so much an issue, more a feature.

tools.sessions.secure in web.conf, by default, restricts session to only https. By connecting directly to my reverse proxy, I was not using an https connection, but rather an http.

I proved this by setting tools.sessions.secure = False and was able to connect.

[lakshman237]

Hi - I have a similar issue. did setting the tools.session.secure = False alone was enough?

[sloshburch]

It's been a long time and I honestly don't recall anymore 😞 Give it a shot and see? Let us know what you find out?

[dart]

Try hitting the /debug/sso endpoint and see what you get back. It should tell you what is being passed by the Proxy.

[dart]

So you don't have anything in the Value of Remote-User section or Is the incoming request IP in splunkweb's list of trustedIPs?

[sloshburch]

Thanks. I checked there but I'm not confident with the results I've found. 😞

I did find the following:
- In the working scenario, I have a Cookie with a session_id_39002. This is missing from the broken one. 39002 is my splunk httpport.
- In the working scenario, X-Forwarded-Host is splunk.company.com, hostname.company.com:listenport in the broken scenario.
- As expected, the User listed in Server info: footer of the working scenario is my user ID, but 'UNKNOWN_USER' in the broken scenario.
All other values match.