Splunk Search

Load url via SPL at search line

landen99
Motivator

Is it possible to load data from a url using SPL at the search line? Three uses cases, specifically:
1) Load https://server.domain.com:8000/en-US/search/inspector?sid=[sid]&namespace=search for job inspect data for a particular sid,
2) Load https://server.domain.com:8000/en-US/api/search/jobs/[sid]/search.log?outputMode=raw for search.log for a particular sid,
3) Load https://mypage.mysite.com/myfile.csv

I know this can be done with scripts somehow, and while I am interested in that process as well, this question focuses on using the search bar (even with the assistance of an app if necessary) to load the data.

The concept is extremely simple in general. At the SPL search bar, a query something like the following:

| loadurl https://mypage.mysite.com/myfile.csv

Loads the file from the webpage specified just like a normal lookup file would be loaded. If the urls from 1 or 2 were used, the data from those pages would be displayed in the Splunk search head client web UI.

0 Karma
1 Solution

rharrisssi
Path Finder

We're using the Getwatchlist add-on and it's working very well.

https://splunkbase.splunk.com/app/635/

View solution in original post

rharrisssi
Path Finder

We're using the Getwatchlist add-on and it's working very well.

https://splunkbase.splunk.com/app/635/

landen99
Motivator

When I try the following SPL example (found at http://blogs.splunk.com/2011/08/16/getwatchlist-getting-watchlists-into-splunk-quickly-and-easily-wi...), I receive an error:

| getwatchlist http://data.phishtank.com/data/online-valid.csv delimiter=”,” relevantFieldName=url relevantFieldCol=2 referenceCol=3 dateCol=4 categoryCol=8 ignoreFirstLine=true isbad=true | outputlookup phishtank.csv

error:

command="getwatchlist", Error getting settings: need more than 1 value to unpack

What does that mean?

0 Karma

landen99
Motivator

Issue resolved:
It turns out that copying text from an online source sometimes brings the wrong double quotes. In this case, the double quotes around the comma were wrong and this error is resolved by replacing them with normal double quotes.

Resolution:

 | getwatchlist http://data.phishtank.com/data/online-valid.csv delimiter="," relevantFieldName=url relevantFieldCol=2 referenceCol=3 dateCol=4 categoryCol=8 ignoreFirstLine=true isbad=true | outputlookup phishtank.csv
0 Karma

landen99
Motivator

This solution seems designed to pull files served through urls, but can it also be made to work for pulling tables on a webpage?

0 Karma

landen99
Motivator

It seems like people are thinking that I am asking about loading the results from a previously run job, but I am not. Regarding the first two use cases, I am asking how to load the results if the Job Inspect from a previous job and how to load the results of the search.log results from a previous job.
Procedure for #1 in the OP (original post) above:
1) Run job, 2) Click Job Inspect, 3) Right click the new window top bar and choose show as tab (Chrome), 4) Compare url of new window to #1 in the OP above.

Procedure for #2 in the OP above:
1) Run job, 2) Click Job Inspect, 3) Right click the new window top bar and choose show as tab (Chrome),
4) Click search.log, 5) Compare url of window to #2 in the OP above.

Procedure for #3 in the OP above:
1) Go to any file summary page on Virustotal, 2) Look at the url.

For any of the three procedures above, load the results via the Splunk search bar (SPL).

0 Karma

woodcock
Esteemed Legend

If you are on a Search Head's CLI and you need to get search results back from another search head and then SPL on that data, you will have to save the output from the other search head (using curl or similar to run the job and pull the output back) into a directory where you can then use inputcsv on your search head to pull it in as data for a search.

Or am I misunderstanding what you are trying to do?

0 Karma

landen99
Motivator

See my post above for clarification (posted under the OP). I want to load more than just the events from a search.

0 Karma

aljohnson_splun
Splunk Employee
Splunk Employee

There is a load job command.

| loadjob 1447695249.272

where 1447695249.272 is the SID.

Additionally, you can specify that you'd like to load the events with events=t.

0 Karma

landen99
Motivator

See my post above for clarification (posted under the OP). I want to load more than just the events from a search.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...