I had the same issue. I finally figured it out. To find out if you're having the same issue look splunkd_access.log in $SPLUNK_HOME/var/log/splunk.
user@server:/opt/splunk/var/log/splunk# grep Cisco splunkd_access.log
127.0.0.1 - splunk-system-user [17/May/2016:15:26:34.142 -0400] "POST /servicesNS/nobody/search/saved/searches/Audit%20Cisco%20Changes/notify?trigger.condition_state=1 HTTP/1.0" 200 1928 - - - 10ms
127.0.0.1 - splunk-system-user [17/May/2016:15:26:34.339 -0400] "GET /servicesNS/user/search/saved/searches/Audit%20Cisco%20Changes?output_mode=json HTTP/1.0" 200 34334 - - - 15ms
127.0.0.1 - splunk-system-user [17/May/2016:15:26:34.730 -0400] "GET /servicesNS/user/search/saved/searches/Audit%20Cisco%20Changes HTTP/1.0" 200 57101 - - - 11ms
127.0.0.1 - splunk-system-user [17/May/2016:15:26:34.498 -0400] "GET /services/pdfgen/render?paper-size=legal-landscape&sid=scheduler_cm9uLm9nbGU__search__RMD5ec6c9fa239e4c78b_at_1463513100_76&input-report=Audit%20Cisco%20Changes&now=1463513100&owner=user&namespace=search HTTP/1.0" **400** 272 - - - **60452ms**
In the log above, you can see the last line has a "400" status code and a time value of "60452ms". The status code will show up in the python.log.
user@server:/opt/splunk/var/log/splunk# grep Cisco python.log
2016-05-17 15:26:34.498 -0400 ERROR sendemail:996 - An error occurred while generating a PDF: Failed to fetch PDF (**status = 400**): Unable to render PDF.<br/><ul><li>Exception raised while trying to prepare "Audit Cisco Changes" for rendering to PDF. Splunkd daemon is not responding: ("Error connecting to /servicesNS/user/search/search/parser: ('The read operation timed out',)",)</li></ul>
2016-05-17 15:26:34,979 -0400 INFO sendemail:112 - Sending email. subject="Splunk Report: Audit Cisco Changes", results_link="https://server.admin.local:8000/app/search/@go?sid=scheduler_cm9uLm9nbGU__search__RMD5ec6c9fa239e4c78b_at_1463462400_63889", recipients="[u'hosted@service-now.com']", server="smtp.admin.local"
The error will also show up in the pdfgen.log:
user@server:/opt/splunk/var/log/splunk# grep -i cisco pdfgen.log
2016-05-17 15:26:34,885 -**0400** ERROR pdfgen_utils:481 - Exception raised while trying to prepare "Audit PCI Cisco Changes New" for rendering to PDF. Splunkd daemon is not responding: ("Error connecting to /servicesNS/user/search/search/parser: ('The read operation timed out',)",)
So that is how to verify that you have the same problem that I had. To fix this problem, you have to change the splunkdConnectionTimeout value in web.conf to something bigger than the time found in the splunkd_access.log file. In my case, I changed the value to 90 (seconds). The default value is 30 seconds.
To make the change, copy or create the web.conf file from etc/system/default/web.conf to etc/system/local/web.conf. Change the splunkdConnectionTimeout= where is greater than the number is seconds from the splunkd_access.log. Note that the splunkd_access.log value is in milliseconds.
Restart the Splunkd service.
... View more