Website Monitoring: When monitoring a URL that has 'browsing' disabled, we get "HTTP/1.1 401 Unauthorized". Is there a workaround?

[edwinmae]

All our Websites have 'Directory Browsing' disabled -- For good reason that is

When trying to monitor a URL that has 'browsing' disabled, we get a '401' of course

You do not have permission to view this directory or page
HTTP/1.1 401 Unauthorized

Is there a workaround for this?
Will there be some kind of 'authentication' mechanism available in Future versions?

[BRally00]

@LukeMurphey how do we go about using HTTPS authentication?

[luisfernandez]

I install the application yesterday and it was working fine but for some reason, my dashboard only shows the data from the last 18hrs ago. But when I pull data from my index is actively collecting data with 200 OK.

Please, can somebody give me a clue what is wrong?

PS. I install the app twice but it didn't fix anything.

[LukeMurphey]

I updated Website Monitoring tonight to support HTTP authentication in version 1.4.0. This version isn't the default yet; you will need to select version 1.4.0 from the version drop-down to get it. Everything is working but I want to do a bit more testing before I make it the defaull.

Please let me know if it works for you.

[edwinmae]

Website Monitoring package for 1.4.0 retrieves the (old) 1.3.2 package

[LukeMurphey]

You need to manually select version 1.4.0; I haven't made this version the default yet.

[edwinmae]

I did manually select it -- Now it's 1.4.0
MD5 checksum (website-monitoring_140.tgz) b59fcf9d78457eb49720355f25be9681 - 1.46MB

Installed it -- Seems no different than the former 1.4.0 -- Input fields are the same
End result is also the same: 401

The password credentials show up in clear text [Website Availability Check] -- which is not a good thing as other users access the Splunk server as well (so they will see my credentials)

name URL Interval Title Configuration Client Certificate Path Client Certificate Key Path Username Password Source type Index Status Actions

[LukeMurphey]

I added another version of 1.4.0 that adds NTLM support. Hopefully, this with authenticate properly.

[edwinmae]

Unfortunately it didn't work for me; I am still unable to poll the Application Web Sites (in the different domains) for the '200' Response
I tried HTTP Authentication with domain'backslash'user, (domain) user, server'backslash'admin -- Always getting '401' Response (Unauthorized)
When I use wget command directly from the Splunk server, I was able to retrieve file(s) from the Application Web Sites (in the end)

Connecting to xxx.xxx.xxx.xxx:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Reusing existing connection to xxx.xxx.xxx.xxx:80.
HTTP request sent, awaiting response... 401 Unauthorized
Reusing existing connection to xxx.xxx.xxx.xxx:80.
HTTP request sent, awaiting response... 200 OK
Length: 3063 (3.0K) [text/html]
Saving to: “xxxxx”

100%[=============================================================================================================>] 3,063       --.-K/s   in 0s

Of course 'your' Application and HTTP Authentication might work fine and this problem 'we' have here is something on our side that I am not able to resolve at this point.

---

Some other remarks:
Credentials were visible in Data Inputs -- After disabling/enabling the App, only the 'Name', 'Status' and 'Actions' were visible
Are the 'time_exceeded' and 'avg_time_exceeded' (in the status overview) still valid as they always show 0 instead of the 'real times'?

[LukeMurphey]

I think I know why the update didn't work for you. I added support for HTTP authentication but your web-servers are using NTLM. This is a little more work to add but I got most of the work done last night. The only problem I have is that I don't have a site that uses NTLM for testing purposes. I'll provide another version soon (hopefully tonight).

[jkat54]

Question 1: Is there a workaround for this?

1. Dont browse the directory... specify index.html at the end of the URI or index.php, photo0001.png, etc.
2. Set the default document for the site, and include one in the directory

Question 2: Support for auth in future?

No need for authentication mechanism when you can specify in the URI. Can be a bit tricky with special characters... use escape characters and quotes if so.

1. Authenticate with the server: http://user@domain.com:password@yoursplunkserver:8000/en-US/...

[edwinmae]

Scenario:
1. The Web servers (=Web sites (=URL's)) that I want to monitor are located in different domains.

2. Our Splunk server is located in a 'central' location with HTTP(S) access to the different domains (Web servers).
3. Authentication = Windows Authentication (HTTP 401 Challenge) with Kernel-mode authentication enabled
4. I basically don't want to add any credentials straight to the http://user: part as e.g. the Monitor tool is accessible by others as well, etc.

Unfortunately I didn't succeed what jkat54 suggested here -- tried to curl the URL's using whatever combination -- always getting HTTP/1.1 401 Unauthorized. Of course I can (and will) use any file inside the directory -- as long as I am able to access that file ('200 Response) -- All is fine

I believe adding HTTP Authentication to the App would be extremely helpful ---- e.g. the App could include the option where e.g. Domain credentials could be applied when creating a new URL check ---

[LukeMurphey]

@jka54 & @edwinmae:
Is HTTP authentication something that would be helpful? If it is, I am willing to add it to the app natively.