Splunk Search

Hit URL grab data and index it

tb5821
Communicator

The concept seems simply yet there doesn't seem to be a straightforward way of doing it. I have URL which I want splunk to hit and index all the data off the page ever X seconds. Thats it. I don't want to only have it grab certain field just all the data on the page.

I initially tried the add-on feedparser but I'm not having any luck with it.

Tags (2)
0 Karma

mloven_splunk
Splunk Employee
Splunk Employee

tb5821,

Seems like this is two problems in one.

The first thing you need to solve is grabbing the contents of the web page. Splunk doesn't have anything built in to do this, but it should be easily scriptable via a wget or curl script.

Secondly, how to index that data. Once your script is outputting an html file, just input it into Splunk as you would any other file. It won't look pretty because it's not really a log message, but Splunk will index it fine.

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...