How to obtain data through the FreshDesk API in REST API Modular Input?

[gonzaloAgui]

Hi,
I need to obtain data from FreshDesk through the FreshDesk API using a url param named "updated_since" (this is a datetime string with format "%Y-%m-%dT%H:%M:%SZ") like a checkpoint from the last ticket indexed. And adding 1 second to the checkpoint because otherwise the last ticket is doubled.
The problem is (I THINK) that the URL Arguments passed from the Response Handler to the rest.py change the format to this:

URL Arguments:
updated_since=1970-10-10 10:10:10

And this generate a Bad request to the API:
03-26-2018 16:27:32.652 -0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" HTTP Request error: 400 Client Error: Bad Request

My response handler script:

class FreshDeskResponseHandler:
    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):

        if response_type == "json":
            output = json.loads(raw_response_output)
            datetime_format = '%Y-%m-%d %H:%M:%S'
            last_update_at = "1970-10-10 10:10:10"
            for freshdesk_event in output:
                print_xml_stream(json.dumps(freshdesk_event))
                if "updated_at" in freshdesk_event:
                    update_at_date = (freshdesk_event["updated_at"].replace("T", " "))[:-1]
                    if update_at_date > last_update_at:
                        update_one_sec = str(datetime.datetime.strptime(update_at_date, datetime_format) + datetime.timedelta(seconds=1))
                        update_one_sec = (update_one_sec.replace(" ", "T")) + str('Z')
                        last_update_at = update_one_sec
            if not "params" in req_args:
                req_args["params"] = {}

            req_args["params"]["updated_since"] = str(last_update_at.replace(" ", "T") + str('Z'))

        else:
            print_xml_stream(raw_response_output)

My inputs .conf:

[rest://TEST - input freshdeck]
auth_password = PASSWORD
auth_type = basic
auth_user = USERNAME
endpoint = https://DOMAIN.freshdesk.com/api/v2/tickets
http_method = GET
index_error_response_codes = 0
response_type = json
sequential_mode = 0
sourcetype = json_freshdesk
streaming_request = 0
cookies = _x_w=24_5
disabled = 0
host = fresh_desk_api
index = testfresh
polling_interval = 60
response_handler_args =
response_handler = FreshDeskResponseHandler
url_args = updated_since=1970-10-10 10:10:10

Has someone encountered a similar problem or knows a workaround to this?
Thanks in advance!

[Damien_Dallimor]

Are you able to get a trace of the actual HTTP GET request sent out , maybe using wireshark or something similar to capture it.

[gonzaloAgui]

Hi Damien, thank you for your quick response. I am trying to trace the http request but failing in the attempt. Could you guide me please? the instance of splunk is in a virtual machine Centos without graphical interface and I do not have access to the server that hosts it.
I have tried with tcpdump, netstat, lsof, iptraf without success.

[Damien_Dallimor]

Try using Splunk Stream 🙂 https://splunkbase.splunk.com/app/1809/

[gonzaloAgui]

I'm already configure splunk stream, but I still can't see the URL with the parameters or the body of the package.
I see data from this sources:

stream:Splunk_DNSIntegrity
stream:Splunk_DNSRequestResponse

stream:Splunk_SSLActivity

stream:Splunk_HTTPURI

stream:http

Is there something I'm missing? let me know if you need a sample of the events.