Website Monitoring: Website monitoring not showing any results when all the URLs trying to invoke at the same time and CPU utilization is too high.

[Vamshireddy]

All the URLs (around 90 URLs including WSDLs) are having interval of 1h and it will work fine when we added. But we will shutdown the server from 19:00 to next day morning 4:00 and when the server start, all the URLs are try to invoke at the same time and it will hang the server.
I need to remove the URLs from the input file and need to restart the splunk to bring the server stable.

If i add around 20 to 30 URLs it will work fine, but when we add more it is causing issue.

Do i need to increase the Server configuration?
Can i check WSDLs as well using Website monitoring?
Do i need to do any other changes to make it work?

[LukeMurphey · Accepted solution]

I should be able to improve this with a new feature in Website Monitoring that allows you to change the number of input threads that run at the same time.

I'm planning on doing the work under this ticket: https://lukemurphey.net/issues/1812.

Details:
This happens due to the new multi-threaded input that was introduced in Website Monitoring 2.0. The multi-threaded input allows you to run many times more inputs that the previous input but it can overwhelm the CPU.

My plan is to make it possible for users to decrease this limit in order to prevent overwhelming the CPU.

Update:
I updated version 2.1 to include the ability to control how many concurrent threads are executing. To use this:

1. Install version 2.1 (https://splunkbase.splunk.com/app/1493/)
2. Open the setup page (click "Configuration" in the navigation)
3. Edit the "Maximum number of concurrent threads" value to something lower. I would starting with something very low, like five.
4. Click save. The configuration will be applied immediately.

[Vamshireddy]

Thanks Luke,
Now it is started working without any hung after reducing the concurrent threads.

Thanks,
Vamshi

[LukeMurphey]

@Vamshireddy: awesome! Thanks for the feedback. Could you be so kind to accept this answer? That way others in the future will recognize that this issue has a solution.

[Vamshireddy]

Thanks LukeMurphey for your response.

In my case, when server restart every day morning, all the URLs (around 90 URLs including WSDL) are trying to invoke at the same time and it will hang the server and i can't any search result.
If am not wrong, even in multi-threading each thread will invoke URL one by one and it should complete invoking all the URL in less than 10 Sec unless if there is any Timeout cases.

To make it work, do i need to do any changes in the setting or input config file? OR do we need to wait until new version of website monitoring App?

[Vamshireddy]

Hello,
Could you please let me know if there is any workaround to make it work?

Thanks,
Vamshi