TRUNCATE for sourcetype="web_ping:response", for "website monitoring" app?

[mitag]

Apologies if this has been asked before - or if the answer is all too obvious...

Where is the TRUNCATE setting for sourcetype="web_ping:response", for "website monitoring" app? Just updated the app to the latest release (2.9.1) which adds "Save the response body" option and it seems the response is truncated at 1,000 characters.

Thanks!

[jwiedow · Accepted solution]

@mitag Website Monitoring actually has a separate parameter that can be set to determine the amount of http response to be included when the response body is enabled. The script that performs the ping will then capture up to that amount and store it in the web_ping:response sourcetype.

The default is set in the website_monitoring.conf file with the the parameter named max_response_body_length. The README incorrectly lists the default as -1 (unlimited). The bin/web_ping.py script (see line 727 and line 758) actually sets the default to 1000 if it is not set in the configuration file. Right now, this is a hidden configuration item that you need to manually set in your local/website_monitoring.conf file if you want it greater than 1000 or at -1 (disabled).

The default truncate for the heavy forwarder where you have Website Monitoring or the TRUNCATE= value in your props.conf for default or the web_ping stanza may override the max_response_body_length parameter. You may need to adjust that setting in props.conf as well.

[nickhills]

Truncation settings apply at the parsing stage, so you need to make sure it’s set in the appropriate server for your deployment.

If the website monitoring app is collecting on a heavy forwarder you should set it on the HF.

If instead you run it on your search head, you should configure it there.

In both cases, the path you need to set it in is $SPLUNK_HOME/etc/apps/app name/local/props.conf

[nickhills]

Truncate is set in props.conf

The default value for truncate is 10,000 bytes .
If you wish to disable truncation set 'TRUNCATE = 0`

[mitag]

The default value for truncate is 10,000 bytes

That's correct for my environment. Why does it truncate at 1K then?

Truncate is set in props.conf

You're saying, it's in on a master node in /opt/splunk/etc/master-apps/_cluster/local/props.conf (assuming default locations) and not in any other file that is specific to the app (e.g. somewhere in /opt/splunk/etc/apps/website_monitoring/ on the search head) and should look something like,

[web_ping:response]
TRUNCATE = 0   # or some other number

?

[mitag]

In both cases, the path you need to set it in is $SPLUNK_HOME/etc/apps/app name/local/props.conf

It's on a search head and creating /opt/splunk/etc/apps/website_monitoring/local/props.conf (there wasn't such file there before) with the following stanza and restarting Splunk didn't change the behavior.

[web_ping:response]
TRUNCATE = 0

... it's still truncating at 1K characters.

If you want to check if the app is limiting its own truncation check $SPLUNK_HOME/etc/apps/app name/ldefault/props.conf

Here is what's in /opt/splunk/etc/apps/website_monitoring/default/props.conf:

[source::...web_availability_modular_input.log]
sourcetype=web_availability_modular_input

[source::...website_monitoring_rest_handler.log]
sourcetype=website_monitoring_rest_handler

...