All Apps and Add-ons

REST API Modular Input: index zip file error

tamduong16
Contributor

I want to read a file in the zip archives and index it into Splunk with REST modular input app. The following is my code for the handler:
alt text
I have been doing research to get my code to work for a week now with no result. I tried running the code on an actual zip file and it works. But when I make change to the actual responsehandlers.py, it doesn't run the whole class and I believe it got stuck in the line file = zipfile.ZipFile(io.BytesIO(response_object.content)). Which is why I wrote the line print_xml_stream(response_type) and print_xml_stream("test2") to test it and only the first print works. The response_type that got index is "text", which lead me to wonder if rest modular input works for handling zip file since on the drop-down list, I only see json, xml and text. I read a few thing about python sdk that run on rest, would this be a good work-around?

0 Karma
1 Solution

tamduong16
Contributor

So I figure out the answer to my question. The only thing I have to change in this code is the line

file = zipfile.ZipFile(io.BytesIO(response_object.content))

to

file = zipfile.ZipFile(StringIO.StringIO(response_object.content))

This problem is due to the difference between python 2.7 and 3.0.

View solution in original post

0 Karma

tamduong16
Contributor

So I figure out the answer to my question. The only thing I have to change in this code is the line

file = zipfile.ZipFile(io.BytesIO(response_object.content))

to

file = zipfile.ZipFile(StringIO.StringIO(response_object.content))

This problem is due to the difference between python 2.7 and 3.0.

0 Karma

tamduong16
Contributor

HI Damien, Thanks for providing this app to us. There's no doubt this app is very useful. I used it to index my data in json format. In this code, I tried to run it without the for loop and if statement. But it won't index the string "test2" either. Which leads me to think it might be stuck at the line above it. My main concern is that it not grabbing the zip file. I don't have experience in dealing with response object from api so I don't know if the statement is right. Thanks for helping!

0 Karma

Damien_Dallimor
Ultra Champion

response_type is irrelevant for this use case , it is ignored.

This app is frequently used to index compressed/binary content over the years (I wrote it).

Your debugging logic is rather flawed however.

ie: perhaps you have an error in your regex pattern csv_file_to_index parameter that is not matching the file correctly in your zip, ergo , the if statement in the loop would not execute.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...