Website Input: How do I monitor a forum that requires authentication to retrieve new content?

[LuiesCui]

Hi guys, I got a problem with the app Website Input and I really need ur help!
Well, I wanted to monitor a forum and get the list of the themes. So I tried the app Website Input (https://splunkbase.splunk.com/app/1818/) and it works well! Well not perfect.
If the content can be shown in the monitored website without logging in, the app works, but the app can't receive the content which only will be displayed after I log in. I filled out the authentication section like this (which really is the account of the monitored website) :

And it just doesn't work. What splunk gets is the content of what I can see before I log in. For Example, the website shows "a" before I log in and shows "b" after I log in. No matter if I fill out the authentication section or not or what I fill out, Splunk only gets "a".
Does anyone know how to solve this? And is there any other app to do that?

[LukeMurphey · Accepted solution]

The forum likely users "forms authentication" not "HTTP authentication". With "HTTP authentication", the browser provides a form something like this.

The reason that the app doesn't support "forms authentication" is that it is very difficult to support.

Update:

I'm looking into supporting forms authentication (which would support this). If you want to follow development of this feature, see this ticket.

Update 2:
Form authentication is supported as of version 4.4. There are some cases the app won't be able to support (such as forms that use two-factor authentication, forms with captchas, etc.) but many forms work.

[LuiesCui]

Thank you for your reply. So we can't do it with Website Input right? Any suggestion to do that?

[LukeMurphey]

It isn't supported yet. However, I'm looking into it.

[LuiesCui]

Thank you!

[LuiesCui]

need help...

[nnmiller]

The author of the application has information on trouble-shooting the app here: http://lukemurphey.net/projects/splunk-web-input/wiki/Troubleshooting

It could be that the website is refusing to show content because it doesn't think the app is a web browser. Websites often block scrapers. You could try changing the user agent string (see https://github.com/LukeMurphey/splunk-web-input/tree/master/src/README ). You should verify, though, that using an automated method to retrieve web content from the website doesn't violate the terms of use--they may block your account if you abuse it.

An simple way to test to see if the website is blocking unknown user agents is to attempt to authenticate by using wget or curl.

[Richfez]

I'm not sure, but I think that Authentication section is for something different than logins for sites like Facebook, forums, or indeed practically anywhere that uses passwords now.

Historically, there is a way not commonly seen any more of setting a password on a particular site that has nothing to do with a login on a web page - it's a login before you get a web page which gives you access to the folder's worth of HTML files. It is very basic, simple, and frankly a terrible way to do it, but it has its use cases - blocking robots, or doing some trivial protection on a home web site you want to keep the kids out of (but don't really care if they figure out a way around it). It was more useful in 1997.

Here's an example of setting that up.

From your screen shot, I'd guess that is what this is for - logging into http authentication protected sites and not logging into Facebook or anything else that may use a username and password.

I am willing to be proven wrong, though.