How to import data from vCloud Director into Splunk using API?

[davide_talesco]

Hi,

I am trying to get DATA into SPLUNK from my vcloud director environment.
I am using the REST API Modular Input app but I am always getting a HTTP Request error: 403 Client Error: Forbidden.
I am using basic authentication and with same details/credentials I am able to use curl or Mozilla RestClient to query vcloud director API succesful.

Has anyone had any experience in importing data from vcloud director into splunk using API?

Thanks,
Davide.

[Damien_Dallimor]

Try something like this. You don't need to change rest.py.

class vCloudAuth(vCloud):
     def __init__(self,**args):
         # Custom Authentication Handler Arguments I am passing from REST API Modular Input.
         self.username = args['username']
         self.password = args['password']
         self.url = args['authURL']
         pass

     def __call__(self,r):

         if r.headers is None:
           r.headers = {}

         if  not 'x-vcloud-authorization' in r.headers:
             headers = {'Accept': 'application/*+xml;version=1.5'}
             auth_response = requests.post(self.url, headers=headers, auth=(self.username, self.password), verify=False)
             r.headers['x-vcloud-authorization'] = auth_response.headers['x-vcloud-authorization']

         return r

[davide_talesco]

Hi Damien, I still get the same error:

12-31-2014 11:09:00.371 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"     (item.split('=') for item in http_header_propertys_str.split(delimiter))) 12-31-2014 11:09:00.371 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"   File "/opt/splunk/etc/apps/rest_ta/bin/rest.py", line 331, in <genexpr> 12-31-2014 11:09:00.371 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"     http_header_propertys = dict((k.strip(), v.strip()) for k,v in 12-31-2014 11:09:00.372 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" ValueError: too many values to unpack

Maybe is it because it doesn't like the "=" sign within the headers variable we define at line 15?

headers = {'Accept': 'application/*+xml;version=1.5'}

[davide_talesco]

If I remove the "=" sign from the headers variable I get a different error:

01-12-2015 11:37:30.364 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Traceback (most recent call last):
01-12-2015 11:37:30.364 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"   File "/opt/splunk/etc/apps/rest_ta/bin/rest.py", line 696, in <module>
01-12-2015 11:37:30.365 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"     do_run()
01-12-2015 11:37:30.365 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"   File "/opt/splunk/etc/apps/rest_ta/bin/rest.py", line 391, in do_run
01-12-2015 11:37:30.365 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"     module = __import__("authhandlers")
01-12-2015 11:37:30.365 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"   File "/opt/splunk/etc/apps/rest_ta/bin/authhandlers.py", line 10, in <module>
01-12-2015 11:37:30.365 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"     class vCloudAuth(vCloud):
01-12-2015 11:37:30.365 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" NameError: name 'vCloud' is not defined

Thanks,
Davide

[Damien_Dallimor]

Totally guessing here as I can't see your actual code or inputs.conf configuration.

Try this :

class vCloudAuth(AuthBase):

instead of

class vCloudAuth(vCloud):

Please note , I am purely giving you example pseudo code as a general guideline for you. I am not familiar with VCloud.

[davide_talesco]

yes that seems to work better (vCloud uses Basic Authentication) but what about my previous question regarding the "=" sign within the headers variable?

[Damien_Dallimor]

You'll have to show your actual code and inputs.conf stanza. Else , it will be guess work for me.

[davide_talesco]

Here you are the inputs.conf stanza:

[rest://Test Vcloud API]
auth_type = custom
custom_auth_handler = vCloudAuth
custom_auth_handler_args = username=user@system,password=password,authURL=https://x.x.x.x/api/sessions
endpoint = https://x.x.x.x/api/query
http_header_propertys = 'Accept: application/*+xml;version=1.5'
http_method = GET
index_error_response_codes = 0
response_type = xml
sourcetype = API
streaming_request = 0
disabled = 0

[Damien_Dallimor]

http_header_propertys = 'Accept: application/*+xml;version=1.5'

should be (as per docs) :

http_header_propertys = Accept=application/*+xml;version=1.5

but....the "=" (which is the key/value delimter) in the value of the header key "Accept" , is causing grief. I'll need to release a patch for this use case. I'll try to do this later tonight.

[davide_talesco]

Hi Damien, do you have any update?

thanks,
Davide.

[Damien_Dallimor]

Try downloading the latest version , 1.3.6

[davide_talesco]

and the custom handler code:

from requests.auth import AuthBase
import requests
import hmac
import base64
import hashlib
import urlparse
import urllib

#add your custom auth handler class to this module
class vCloudAuth(AuthBase):
        def __init__(self,**args):

                self.username = args['username']
                self.password = args['password']
                self.url = args['authURL']
                pass

        def __call__(self,r):

                if r.headers is None:
                        r.headers = {}

                if not 'x-vcloud-authorization' in r.headers:
                        headers = {'Accept: application/*+xml;version=1.5'}
                        auth_response = requests.post(self.url, headers=headers, auth=(self.username, self.password), verify=False)
                        r.headers['x-vcloud-authorization'] = auth_response.headers['x-vcloud-authorization']
                return r

The error I am getting now is the following:

01-12-2015 15:25:43.494 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request: 'set' object has no attribute 'items'

[davide_talesco]

Hi Damien,

I made some progress and I managed to write down the custom authentication handler which I confirm it is working (I tested it standalone using python.exe).
Basically the authentication handler does a Post to vcloud director and once authenticated saves the authentication token (http x-vcloud-authorization header ) which has to be used for the following GET API requests which I make using the REST API Modular Input UI.
My problem is that I am not too sure how and if I have to pass the x-vcloud-authorization token back to the REST API Modular Input

below the code:

class vCloudAuth(vCloud):
    def __init__(self,**args):
        # Custom Authentication Handler Arguments I am passing from REST API Modular Input.
        self.username = args['username']
        self.password = args['password']
        pass

    def __call__(self,r):
        # do a Post to https://x.x.x.x/api/sessions passing 'Accept:application/*+xml;version=1.5' , Username, Password
        # return x-vcloud-authorization token

        # Set the required Headers
        headers = {'Accept': 'application/*+xml;version=1.5'}

        # Set the URL
        url = 'https://x.x.x.x/api/sessions'

        # Make the API POST
        r = requests.post(url, headers=headers, auth=(self.username, self.password), verify=False)

THIS IS THE STEP I AM NOT TOO SURE ABOUT
        #should I return just the header I need? 
        #return r.headers['x-vcloud-authorization']

       #or should I return the whole Request object?
       #return r

Splunkd.log shows the following error:

12-31-2014 11:09:00.371 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Traceback (most recent call last):
12-31-2014 11:09:00.371 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"   File "/opt/splunk/etc/apps/rest_ta/bin/rest.py", line 696, in <module>
12-31-2014 11:09:00.371 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"     do_run()
12-31-2014 11:09:00.371 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"   File "/opt/splunk/etc/apps/rest_ta/bin/rest.py", line 332, in do_run
12-31-2014 11:09:00.371 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"     (item.split('=') for item in http_header_propertys_str.split(delimiter)))
12-31-2014 11:09:00.371 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"   File "/opt/splunk/etc/apps/rest_ta/bin/rest.py", line 331, in <genexpr>
12-31-2014 11:09:00.371 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py"     http_header_propertys = dict((k.strip(), v.strip()) for k,v in
12-31-2014 11:09:00.372 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" ValueError: too many values to unpack

Do I need to modify opt/splunk/etc/apps/rest_ta/bin/rest.py as well?
Happy new Year!

Thanks again,
Davide.

[ppablo]

Hi @davide_talesco

Please be sure to click on "Add comment" link directly below an answer or type in the "Add your comment" box directly below a comment when responding to another user here on Splunk Answers. You've been pasting brand new answers that are actually responses to Damien and the flow of the discussion is getting very confusing on this post. I can't convert answers to comments that are over a certain character count. If you're exceeding a character limit when posting a comment in response, just break it up into multiple comments. Thanks.

Patrick

[davide_talesco]

Hi Patrick, got it. sorry.

[ppablo]

No problemo 🙂

[Damien_Dallimor]

1) how long does an auth token last for ? If it can last "indefinitely" , then you could just skip the first HTTP request to get the token , and setup your REST stanza with a pre-acquired token (using CURL)

2) or , probably better is to use a custom authentication handler in the REST API Modular Input. This requires adding a custom auth handler class to SPLUNK_HOME/etc/apps/rest_ta/bin/authhandlers.py and then declaring this auth handler class to be applied in your REST stanza setup. As an example , your setup might then look like :

[davide_talesco]

Hi Damien,

sorry for my late reply and thanks a million for your help!
Do you know where I can find any documentation on how to implement a custom authentication handler?

Regards,
Davide.

,Hi Damien,

sorry for my late reply and thanks a lot for the information!
Is there any available documentation on how to create a custom authentication handler?

Thanks in advance,
Davide.

[davide_talesco]

Hi Damien,

below you can find the CURL Command and output.

curl -i -k -H 'Accept:application/*+xml;version=1.5' -u sale1@sales-LAB:password -X POST https://10.0.0.21/api/sessions

HTTP/1.1 200 OK
Date: Tue, 28 Oct 2014 08:41:19 GMT
x-vcloud-authorization: 593G+0kSwr03xOhVRs1x3269LFS0sWZomroUsCQMHq8=
Set-Cookie: vcloud-token=593G+0kSwr03xOhVRs1x3269LFS0sWZomroUsCQMHq8=; Secure; Path=/
Content-Type: application/vnd.vmware.vcloud.session+xml;version=1.5
Date: Tue, 28 Oct 2014 08:41:19 GMT
Content-Length: 725

<Link rel="down" type="application/vnd.vmware.vcloud.orgList+xml" href="https://10.0.0.21/api/org/"/>
<Link rel="down" type="application/vnd.vmware.vcloud.query.queryList+xml" href="https://10.0.0.21/api/query"/>
<Link rel="entityResolver" type="application/vnd.vmware.vcloud.entity+xml" href="https://10.0.0.21/api/entity/"/>

[root@r2-prdcldnfs splunk]# curl -i -k -H 'Accept:application/+xml;version=1.5' -H 'x-vcloud-authorization: 593G+0kSwr03xOhVRs1x3269LFS0sWZomroUsCQMHq8=' -X GET https://10.0.0.21/api/query?type=task&format=records
[1] 16471
[root@r2-prdcldnfs splunk]# HTTP/1.1 200 OK
Date: Tue, 28 Oct 2014 08:42:03 GMT
Content-Type: application/+xml;version=1.5
Date: Tue, 28 Oct 2014 08:42:04 GMT
Content-Length: 12861

<Link rel="nextPage" type="application/vnd.vmware.vcloud.query.records+xml" href="https://10.0.0.21/api/query?type=task&amp;page=2&amp;pageSize=25&amp;format=records"/>
<Link rel="lastPage" type="application/vnd.vmware.vcloud.query.records+xml" href="https://10.0.0.21/api/query?type=task&amp;page=4&amp;pageSize=25&amp;format=records"/>
<Link rel="alternate" type="application/vnd.vmware.vcloud.query.references+xml" href="https://10.0.0.21/api/query?type=task&amp;page=1&amp;pageSize=25&amp;format=references"/>
<Link rel="alternate" type="application/vnd.vmware.vcloud.query.idrecords+xml" href="https://10.0.0.21/api/query?type=task&amp;page=1&amp;pageSize=25&amp;format=idrecords"/>
<TaskRecord status="success" startDate="2014-10-14T11:54:07.163+01:00" ownerName="sale1" orgName="Sales-LAB" org="https://10.0.0.21/api/org/43a35638-4e17-44c6-a763-1116a4e0d832" objectType="vm" objectName="VMware ESXi 5.5 Host 2" object="https://10.0.0.21/api/vApp/vm-c82a170e-2fb2-44c3-9282-22b712878af3" name="vappUpdateVm" endDate="2014-10-14T11:54:15.097+01:00" href="https://10.0.0.21/api/task/062b3b38-1cd3-4781-a57a-51b63d6a8ad7" details=" "/>
<TaskRecord status="success" startDate="2014-10-14T13:24:52.197+01:00" ownerName="sale1" orgName="Sales-LAB" org="https://10.0.0.21/api/org/43a35638-4e17-44c6-a763-1116a4e0d832" objectType="vm" objectName="VCS 5.5 Windows 2008 R2 Ent SP1/VCS 5.5" object="https://10.0.0.21/api/vApp/vm-5c1bfec6-a3a7-418b-a329-4873168a274d" name="vappUpdateVm" endDate="2014-10-14T13:24:55.343+01:00" href="https://10.0.0.21/api/task/0a87850b-6a74-408f-9af5-b833cf89069e" details=" "/>

Sorry but I am new to SPLUNK and I am not too sure which inputs.conf you are talking about. Here below are all I found:

[root@r2-prdcldnfs apps]# pwd
/opt/splunk/etc/apps
[root@r2-prdcldnfs apps]# ls -lart
total 72
drwxr-xr-x 9 root root 4096 Nov 11 2013 splunk_app_for_nix
drwxr-xr-x 4 splunk splunk 4096 Oct 10 12:00 SplunkLightForwarder
drwxr-xr-x 4 splunk splunk 4096 Oct 10 12:00 SplunkForwarder
drwxr-xr-x 4 splunk splunk 4096 Oct 10 12:00 user-prefs
drwxr-xr-x 4 splunk splunk 4096 Oct 10 12:00 splunk_datapreview
drwxr-xr-x 9 splunk splunk 4096 Oct 10 12:00 search
drwxr-xr-x 6 splunk splunk 4096 Oct 10 12:00 sample_app
drwxr-xr-x 3 splunk splunk 4096 Oct 10 12:00 legacy
drwxr-xr-x 4 splunk splunk 4096 Oct 10 12:00 introspection_generator_addon
drwxr-xr-x 6 splunk splunk 4096 Oct 10 12:00 gettingstarted
drwxr-xr-x 5 splunk splunk 4096 Oct 10 12:06 learned
drwxr-xr-x 6 splunk splunk 4096 Oct 10 12:06 framework
drwxr-xr-x 15 splunk splunk 4096 Oct 10 12:08 ..
drwx--x--x 8 root root 4096 Oct 27 11:59 rest_ta
drwx--x--x 8 root root 4096 Oct 27 12:00 Splunk_TA_nix
drwx--x--x 6 root root 4096 Oct 27 12:00 SA-nix
drwxr-xr-x 18 splunk splunk 4096 Oct 27 12:00 .
drwxr-xr-x 7 splunk splunk 4096 Oct 27 12:14 launcher
[root@r2-prdcldnfs apps]# find / -name inputs.conf
/opt/splunk/etc/apps/splunk_app_for_nix/install/Splunk_TA_nix/default/inputs.conf
/opt/splunk/etc/apps/splunk_app_for_nix/install/SA-nix/default/inputs.conf
/opt/splunk/etc/apps/splunk_app_for_nix/default/inputs.conf
/opt/splunk/etc/apps/Splunk_TA_nix/default/inputs.conf
/opt/splunk/etc/apps/introspection_generator_addon/default/inputs.conf
/opt/splunk/etc/apps/SplunkLightForwarder/default/inputs.conf
/opt/splunk/etc/apps/sample_app/default/inputs.conf
/opt/splunk/etc/apps/SA-nix/default/inputs.conf
/opt/splunk/etc/apps/launcher/local/inputs.conf
/opt/splunk/etc/system/default/inputs.conf
/opt/splunk/etc/system/local/inputs.conf
/opt/splunk/etc/modules/distributedDeployment/classes/deployable/inputs.conf
[root@r2-prdcldnfs apps]#

Thanks,
Davide.

[davide_talesco]

I know, but as you can see from the Curl example to get the data I need from Vcloud director I need to run 2 API queries:
1- I call a POST operation to https://10.0.0.21/api/sessions to login to vCloud
This operation will return my vCloud authorization token
(x-vcloud-authorization: 593G+0kSwr03xOhVRs1x3269LFS0sWZomroUsCQMHq8=)
2- To get the actual data I am looking for (a list of all vcloud director tasks) I need asecond API call, this time a GET operation to https://10.0.0.21/api/query?type=task&format=records passing the authorization token.

Is this something possible to achieve?

For a more detailed explanation on what I am trying to do you can have a look here:
http://blogs.vmware.com/vsphere/2012/03/exploring-the-vcloud-rest-api-part-1.html

Thanks,
Davide.

[Damien_Dallimor]

Based on correlating your CURL example with your Splunk REST Setup , you seem to be using the incorrect HTTP method in your Splunk REST setup.

Your CURL example is using POST , and your Splunk REST setup is using GET.

So change this.