Splunk Search

How to upload updated lookup CSV to Splunk Cloud using REST API WITHOUT using the UI?

gkiffney
Engager

We're heavy SplunkCloud users and have run into a roadblock. We have a lookup CSV file that needs to be updated daily - slowly changing customer information - but try as I might, I cannot find an automated way to upload these CSVs without using the Web user interface.

The closest thing I can find is

https://<host>:<mPort>/services/data/lookup-table-files/{name}

where the POST method will allow you to "Modify a lookup table file by replacing it with a file from
the upload staging area."

But in SplunkCloud, we don't have access to the upload staging area - we don't have file access at all, as far as I can tell.

How can this be done? I'd like to do this using something simple like curl:

curl -k -u admin:password --form upload=@/home/me/lookup.csv https://mycompany.splunkcloud.com:8089/rest-api-call-to-upload-and-update-existing-csv-lookup

dmarling
Builder

I documented an answer for this that I believe can be used for cloud customer here: https://answers.splunk.com/answers/694345/how-to-upload-csv-data-file-into-splunk-by-using-r.html?ch...

It requires powershell, but if some enterprising soul wants to port the concepts to another language, it should be possible.

If this comment/answer was helpful, please up vote it. Thank you.
0 Karma

micahkemp
Champion

I looked into this and found no way to do this via REST, even via undocumented endpoints. I looked at the splunkd_access logs from a timeperiod during which I uploaded a CSV and saw no reference to the upload going through the API. It seems that splunkweb handles the upload and storing the CSV on disk, and then calls a REST endpoint to create the lookup itself using the uploaded file.

Unfortunately I think the answer to this question is simply that it's not possible.

0 Karma

mbintz
Explorer

+1 To this question. I have a lookup table that I'd like to update on a daily basis from a cron job. It would be great if there was a RESTful way to do this.

0 Karma

gkiffney
Engager

This isn't an answer, more of a workaround. I upload the CSV file, a list of practices, to splunk cloud using the forwarder. In props.conf I have this:

[source::.../lookups/ss_practices.csv]
DATETIME_CONFIG=CURRENT
NO_BINARY_CHECK=true
INDEXED_EXTRACTIONS=CSV
HEADER_FIELD_LINE_NUMBER=1

And I run a scheduled report that looks like this:

index=smrts_maint source="*ss_practices.csv" sourcetype=CSV | table NATIVE_PRACTICE_ID CUSTOMER_NAME STREET CITY STATE_REGION POSTAL_CODE COUNTRY_CODE_A2 COUNTRY_NAME TELEPHONE LAT LON | outputlookup ss_practices.csv

and that updates the lookup table ss_practices.csv. DATETIME_CONFIG = CURRENT is necessary to keep Splunk from trying to guess timestamps for the rows in your csv file if it has no timestamps.

Micheal_S
Explorer

I also took a similar approach based on your answer. I used a scripted input to download the csv I needed and pull it into an index and then your sourcetype for formatting. 

 

../app/bin/download_csv.sh
Ensure that this file has the right permissions for the splunkd user, I also ensured that it was executable. 

 

#!/bin/bash

URL="https://www.somesite.com/myfile.csv"

curl -k -s $URL

 

 

../app/local/inputs.conf

 

[script://./bin/download_csv.sh]
disabled = false
interval = * * * * *
index = myindex
sourcetype = mytype_csv

 

 

../app/local/props.conf

 

[mytype_csv]
DATETIME_CONFIG=CURRENT
NO_BINARY_CHECK=true
INDEXED_EXTRACTIONS=CSV
HEADER_FIELD_LINE_NUMBER=1

 


Then I'll update the csv in a similar fashion to your own with the search. 

0 Karma

simonnallen
Engager

I am using the Java SDK and would like to be able to add watch lists to Spunk i.e publish IOC's as a lookup that I can the use is queries.

Note I have already implemented the API's to allow me to connect/auth and execute queries. I take these results and analyze the results. This leaves me with a list of known threats. I then want to automatically publish these back to Splunk as a Black Watch List.

Does anyone know how to do this ?

0 Karma

bizmate
Engager

I also would like a flexible API to upload data, I ll keep investigating but unfortunately for a project I am working on the uploader or doing it by end is not right
I hope they answer with something workable

0 Karma

clong_
Engager

I also have this question. It doesn't seem to be possible via the API, which is silly considering you can use the GUI to upload lookup tables remotely without having to jump through the staging area hoops. Hopefully the API has or will implement similar capabilities.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...