REST API Modular Input: What to put into custom authentication and response handlers to handle a sequence of https requests?

[swasserroth]

Hello Splunkers,

currently I am struggling with the REST-TA Modular Input. We are trying to access the WLAN Controller of Ubiquity Unifi WLAN Access Points, which offers a REST API. We want to retrieve the event log.

The sequence of actions needed to retrieve the event log is:
1. login into the Unifi Controller with https://unifi.domain.de:7777/login?username=XXX&login=login&password=YYY
2. retrieve the event log with https://unifi.domain.de:7777/api/stat/event, this returns a JSON structure
3. logout with https://unifi.domain.de:7777/logout

The difference to all other questions regarding REST-TA is: Before actually GETting the information requested, a https-Login-packet is required, and afterwards a https-Logout-packet must be sent to the REST-API-provider.

This is the inputs.conf stanza:

[rest://Unifi WLAN Events]
auth_type = custom
custom_auth_handler = MyUnifyAuth
endpoint = https://unifi.domain.de:7777/api/stat/event
http_method = GET
index = test
index_error_response_codes = 1
polling_interval = 120
response_handler = MyUnifyResponse
response_type = json
sourcetype = _json
streaming_request = 0
disabled = 1

And this is the custom authhandlers.py (only additional lines to the shipped file, code is taken mostly from Unifi-API examples at github):

import urllib2
import cookielib
#login for Unifi WLAN controller class MyUnifyAuth(AuthBase):
    def __init__(self,**args):
        cj = cookielib.CookieJar()
        self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
        pass

    def __call__(self, r):
        loginurl = 'https://unifi.domain.de:7777/login?username=XXX&login=login&password=YYY'
        self.opener.open(loginurl).read()
        return r

And the custom responsehandlers.py basically is the default handler, extended with the logout:

class MyUnifyResponse:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        self.opener.open('https://unifi.domain.de:7777/logout').read()
        cookies = response_object.cookies
        if cookies:
            req_args["cookies"] = cookies
        print_xml_stream(raw_response_output)    

Now for the problems/questions:

• Did I really get the basic API flow correct? Or did I misunderstand the methods completely?
• Something must be very wrong: SOS tells me ExecProcessor - message from "python /export/server/splunk/etc/apps/rest_ta/bin/rest.py" HTTP Request error: 401 Client Error: Unauthorized
• Since the Unifi-API isn't documented, the missing parts are difficult to find, I assume that something is requiring cookies, and that the cookie handling is wrong (wild, but educated, guess...)
• I would appreciate hints on how to handle the sequence of https-requests described above, i.e. what to put into authentication and what to put into response handlers.

Thanks in advance for helpful hints!!
Best regards,
Stephan

[Damien_Dallimor]

I'm making a very wild guess here as I know nothing about Unifi. So try the below ideas.

[rest://Unifi WLAN Events]
auth_type = custom
custom_auth_handler = MyUnifyAuth
custom_auth_handler_args = username=foo,password=goo,login_url=https://unifi.domain.de:7777/login
endpoint = https://unifi.domain.de:7777/api/stat/event
http_method = GET
index = test
index_error_response_codes = 1
polling_interval = 120
response_handler = MyUnifyResponse
response_handler_args = logout_url=https://unifi.domain.de:7777/logout
response_type = json
sourcetype = _json
streaming_request = 0
disabled = 1

class MyUnifyAuth(AuthBase):
     def __init__(self,**args):
         self.username = args['username']
         self.password = args['password']
         self.login_url = args['login_url']
         pass

     def __call__(self, r):
         payload = {'username': self.username, 'login': 'login','password':self.password}
         login_response = requests.get(self.login_url,params=payload,verify=false)
         cookies = login_response.cookies
         if cookies:
            r.cookies = cookies
         return r

class MyUnifyResponse:

     def __init__(self,**args):
         self.logout_url = args['logout_url']
         pass

     def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
         cookies = response_object.cookies
         if cookies:
             req_args["cookies"] = cookies
         requests.get(self.logout_url,verify=false,cookies=cookies)

         print_xml_stream(raw_response_output) 

[swasserroth]

Hi Damien,
thanks for the very quick answer!! Now I understand the data flow much better. I had to add verify=False to the requests.get, otherwise SSL errors are thrown, because only self signed certificates are used. Otherwise I am using exactl your code.

I have found out that the login sets a session cookie (destroyed at the end of a session) named "unifises" with a hex string, probably a kind of session id.

I still get an 401 error, client unauthorized. Since the cookie is returned in r.cookies probably the session is not reused inside the rest-ta? Currently I can't imagine a reason for the 401.

If it is a certification validation problem: How do I propagate the verify=False to the get-Request of the endpoint-URL?

Thanks again,
Stephan

[Damien_Dallimor]

I only have this to go off : https://github.com/calmh/unifi-api/blob/master/unifi/controller.py

I see that there is some dynamic URL construction going on , is https://unifi.domain.de:7777/api/stat/event the correct URL ?

Can you trace the requests/responses being sent/received (ie: using a tool like wireshark) , to ensure that the HTTP GET looks correct ?

[swasserroth]

The URLs for login, logout and REST-API-call to get the events have been checked inside a Firefox-browser, the browser-window displays the json-list of events. These are the URLs as they actually are constructed by the controller.py from github, what we have used to understand the required flow of https-packets.

Wireshark is not so easy, everything is ssl-encoded...

[Damien_Dallimor]

Is the 401 occurring on the Login HTTP GET or the Endpoint HTTP GET ?

[swasserroth]

OK, I am pretty much shure, that this is related to the session-only duration of the cookie.

What I did inside the MyUnifiAuth class to test this:
- request.get with Login-String, this returned 200
- request.get with event-API-call, this returned 401, and a json-error "api.err.LoginRequired"
Seems that this are actually two sessions...

Thanks for supporting us!!
I think, we should consider other (outside of Splunk) methods to retrieve the event list using an intermediate file.