redirect loop

[isrjo]

Greetings.

We just upgraded our servers from opensuse 11.3 --> 11.4 (X64)
After the upgrade splunk no longer works, or more specificity the web does not. The splunk daemon seem to work as expected.

What happens is when accessing the webinterface a redirect loop occurs and the browser gives up.

Example from log:

85.229.239.20 - - [26/Aug/2011:11:20:53] "GET / HTTP/1.1" 303 108 "" "Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0" - 4e5765758219dad10
85.229.239.20 - - [26/Aug/2011:11:20:53] "GET /en-GB/ HTTP/1.1" 303 127 "" "Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0" - 4e5765758819daa50
85.229.239.20 - - [26/Aug/2011:11:20:53] "GET /en-GB/account/login?return_to=%2Fen-GB%2F HTTP/1.1" 303 148 "" "Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0" - 4e5765758c19dae50
85.229.239.20 - - [26/Aug/2011:11:20:53] "GET /en-GB/account/login?return_to=%2Fen-GB%2Faccount%2Flogin%3Freturn_to%3D%252Fen-GB%252F HTTP/1.1" 303 162 "" "Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0" - 4e5765759219daa50
85.229.239.20 - - [26/Aug/2011:11:20:53] "GET /en-GB/account/login?return_to=%2Fen-GB%2Faccount%2Flogin%3Freturn_to%3D%252Fen-GB%252Faccount%252Flogin%253Freturn_to%253D%25252Fen-GB%25252F HTTP/1.1" 303 171 "" "Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0" - 4e5765759719dae50
85.229.239.20 - - [26/Aug/2011:11:20:53] "GET /en-GB/account/login?return_to=%2Fen-GB%2Faccount%2Flogin%3Freturn_to%3D%252Fen-GB%252Faccount%252Flogin%253Freturn_to%253D%25252Fen-GB%25252Faccount%25252Flogin%25253Freturn_to%25253D%2525252Fen-GB%2525252F HTTP/1.1" 303 184 "" "Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0" - 4e5765759c19e3e90
85.229.239.20 - - [26/Aug/2011:11:20:53] "GET /en-GB/account/login?return_to=%2Fen-GB%2Faccount%2Flogin%3Freturn_to%3D%252Fen-GB%252Faccount%252Flogin%253Freturn_to%253D%25252Fen-GB%25252Faccount%25252Flogin%25253Freturn_to%25253D%2525252Fen-GB%2525252Faccount%2525252Flogin%2525253Freturn_to%2525253D%252525252Fen-GB%252525252F HTTP/1.1" 303 195 "" "Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0" - 4e576575a21a410d0
85.229.239.20 - - [26/Aug/2011:11:20:53] "GET /en-GB/account/login?return_to=%2Fen-GB%2Faccount%2Flogin%3Freturn_to%3D%252Fen-GB%252Faccount%252Flogin%253Freturn_to%253D%25252Fen-GB%25252Faccount%25252Flogin%25253Freturn_to%25253D%2525252Fen-GB%2525252Faccount%2525252Flogin%2525253Freturn_to%2525253D%252525252Fen-GB%252525252Faccount%252525252Flogin%252525253Freturn_to%252525253D%25252525252Fen-GB%25252525252F HTTP/1.1" 303 202 "" "Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0" - 4e576575a919dae50
85.229.239.20 - - [26/Aug/2011:11:20:53] "GET /en-GB/account/login?return_to=%2Fen-GB%2Faccount%2Flogin%3Freturn_to%3D%252Fen-GB%252Faccount%252Flogin%253Freturn_to%253D%25252Fen-GB%25252Faccount%25252Flogin%25253Freturn_to%25253D%2525252Fen-GB%2525252Faccount%2525252Flogin%2525253Freturn_to%2525253D%252525252Fen-GB%252525252Faccount%252525252Flogin%252525253Freturn_to%252525253D%25252525252Fen-GB%25252525252Faccount%25252525252Flogin%25252525253Freturn_to%25252525253D%2525252525252Fen-GB%2525252525252F HTTP/1.1" 303 215 "" "Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0" - 4e576575b019daf90
85.229.239.20 - - [26/Aug/2011:11:20:53] "GET /en-GB/account/login?return_to=%2Fen-GB%2Faccount%2Flogin%3Freturn_to%3D%252Fen-GB%252Faccount%252Flogin%253Freturn_to%253D%25252Fen-GB%25252Faccount%25252Flogin%25253Freturn_to%25253D%2525252Fen-GB%2525252Faccount%2525252Flogin%2525253Freturn_to%2525253D%252525252Fen-GB%252525252Faccount%252525252Flogin%252525253Freturn_to%252525253D%25252525252Fen-GB%25252525252Faccount%25252525252Flogin%25252525253Freturn_to%25252525253D%2525252525252Fen-GB%2525252525252Faccount%2525252525252Flogin%2525252525253Freturn_to%2525252525253D%252525252525252Fen-GB%252525252525252F HTTP/1.1" 303 226 "" "Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0" - 4e576575b519dae50

To test I downloaded the whole splunk instance to my laptop also running opensuse 11.4 and there it works. The difference between the two are that our servers are installed with just a minimum set of packages.

I fail to find anything in the manual about dependencies to the OS that could explain this.

I also just to try installed a (clean) later version of splunk in parallel to our production splunk, there I can not login. A wild guess is that something with session is broken.

Anyone have a clue what to do ?

[gbower333]

I had something similar happen and found that the server had a time skew of over two hours.  I set the time and the redirects stopped.

[isrjo]

Let me first clarify that we use the free license therefor having no login. The clean installation I just tried briefly to make some sort of reference, and since that did not work either I uninstalled it. There where however no errors part from 'invalid credentials'

I have enabled debug on the webservice, here is the output:

2011-08-30 18:15:47,724 INFO [4e5d0cb3b819c0a50] decorators:301 - require_login - no splunkd sessionKey variable set; cherrypy_session=84e231950db4515fc085e65e4fb1cc7d9786e4d0 request_path=/en-GB/
2011-08-30 18:15:47,724 INFO [4e5d0cb3b819c0a50] decorators:308 - require_login - redirecting to login

2011-08-30 18:15:47,735 DEBUG [4e5d0cb3bb19b7a50] init:366 - simpleRequest > GET https://217.75.116.40:8089/services/server/info [] sessionSource=cherrypy

2011-08-30 18:15:47,740 DEBUG [4e5d0cb3bb19b7a50] init:380 - simpleRequest < server responded status=200 responseTime=0.0052s

2011-08-30 18:15:47,742 DEBUG [4e5d0cb3bb19b7a50] init:366 - simpleRequest > POST https://217.75.116.40:8089/services/auth/login [[REDACTED]] sessionSource=cherrypy
2011-08-30 18:15:47,746 DEBUG [4e5d0cb3bb19b7a50] init:380 - simpleRequest < server responded status=401 responseTime=0.0038s

2011-08-30 18:15:47,746 DEBUG [4e5d0cb3bb19b7a50] init:394 - simpleRequest - Authentication failed; sessionKey=None

2011-08-30 18:15:47,757 DEBUG [4e5d0cb3c119c0dd0] init:366 - simpleRequest > GET https://217.75.116.40:8089/services/server/info [] sessionSource=cherrypy

2011-08-30 18:15:47,763 DEBUG [4e5d0cb3c119c0dd0] init:380 - simpleRequest < server responded status=200 responseTime=0.0051s

2011-08-30 18:15:47,764 DEBUG [4e5d0cb3c119c0dd0] init:366 - simpleRequest > POST https://217.75.116.40:8089/services/auth/login [[REDACTED]] sessionSource=cherrypy

2011-08-30 18:15:47,768 DEBUG [4e5d0cb3c119c0dd0] init:380 - simpleRequest < server responded status=401 responseTime=0.0034s

2011-08-30 18:15:47,768 DEBUG [4e5d0cb3c119c0dd0] init:394 - simpleRequest - Authentication failed; sessionKey=None

2011-08-30 18:15:47,779 DEBUG [4e5d0cb3c619c0fd0] init:366 - simpleRequest > GET https://217.75.116.40:8089/services/server/info [] sessionSource=cherrypy

2011-08-30 18:15:47,783 DEBUG [4e5d0cb3c619c0fd0] init:380 - simpleRequest < server responded status=200 responseTime=0.0039s

Not knowing to much about splunk internals, who's giving the 401 ? Is the web interface talking to the splunk daemon, responsible for the authentication ?

There are no errors found in any log.

[melting]

Are you getting any messages above the login prompt? Have you tried going to the base splunk url again, after login in and redirect? Are you using a proxy?

Perhaps you could also look at web_service.log to see what it has to say on the matter.