All Apps and Add-ons

Why is Microsoft Log Analytics Add-on(Formerly Know as OMS) data getting stopped after 502 Response code?

ips_mandar
Builder

Hi ,

I have a Microsoft Log Analytics Add-on on a heavy forwarder with interval as 60 sec and lag time as 15 min.

Everything works fine till I get below errors-
Query:- index=_internal ERROR sourcetype="ta:ms:loganalytics:log"
Output:-

2018-10-10 08:13:27,405 ERROR pid=10992 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
 Traceback (most recent call last):
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\modinput_wrapper\base_modinput.py", line 127, in stream_events
     self.collect_events(ew)
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py", line 96, in collect_events
     input_module.collect_events(self, ew)
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\input_module_log_analytics.py", line 72, in collect_events
     response = requests.post(uri,json=search_params,headers=headers)
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\api.py", line 110, in post
     return request('post', url, data=data, json=json, **kwargs)
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\api.py", line 56, in request
     return session.request(method=method, url=url, **kwargs)
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\sessions.py", line 488, in request
     resp = self.send(prep, **send_kwargs)
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\sessions.py", line 641, in send
     r.content
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\models.py", line 781, in content
     self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\models.py", line 706, in generate
     raise ChunkedEncodingError(e)
 ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
 2018-10-10 08:19:04,789 ERROR pid=7208 tid=MainThread file=base_modinput.py:log_error:307 | OMSInputName="omslog" status="502" step="Post Query" response="<html>
 <head><title>502 Bad Gateway</title></head>
 <body bgcolor="white">
 <center><h1>502 Bad Gateway</h1></center>
 <hr><center>nginx</center>
 </body>
 </html>
 "

Once this error comes, OMS data flow gets stopped until I re-enable input. and when I re-enable input it again starts flowing.
Can any one help me? What will be the issue causing data to stopped and not reconnecting again once issue is resolved?

shalinisrinivas
New Member

Hi @ips_mandar, Is your issue resolved yet? I am facing a similar issue with the splunk reporting Add-on for Office365

0 Karma

jkat54
SplunkTrust
SplunkTrust

It appears you have a proxy server or load balancer (nginx) configured for this splunk devices outbound connections and it's causing the issue:

 <head><title>502 Bad Gateway</title></head>
 <body bgcolor="white">
 <center><h1>502 Bad Gateway</h1></center>
 <hr><center>nginx</center>
 </body>
0 Karma

nrduren1115
Explorer

This may not directly answer your question, but I have noticed this behavior in the past with another Microsoft Add-On that uses modinputs. It seems that if there is an error in the execution, it seems to be removed from the scheduling, which would seem to be a bug in Splunk itself. My guess is that if the current input is still running, it will skip until the next run and the failure causes Splunk not to register that there was a failure. Next time there is a failure, try going to the modinputs api URL below and see if it still thinks it is running or not:

https://localhost:8089/services/admin/inputstatus

0 Karma

mstjohn_splunk
Splunk Employee
Splunk Employee

hi @ips_mandar,

did this help you answer you question? If so, please approve it so other users can learn from it. Thanks for posting!

0 Karma

ips_mandar
Builder

Hi @mstjohn_splunk ,
My issue is not yet resolved..

0 Karma

shalinisrinivas
New Member

Hi @ips_mandar,

Is your issue resolved yet? I am facing a similar issue with the Splunk reporting Add-on for Office 365 and looking for help.

0 Karma

ips_mandar
Builder

Currently it is running and it shows-
exit status description exited with code 0
time opened 2018-10-16T11:04:20+0200
total bytes 28587367
I will keep watching once data flow gets stopped...
but is there any solution to avoid this problem?

0 Karma

nrduren1115
Explorer

Not sure why the comment isn't showing up, but I saw your reply that the input was now gone. This definitely seems like a bug, either with the modinputs or with the way this app is designed. I would contact support and file a bug report.

0 Karma

ips_mandar
Builder

actually i checked on my search head regarding input so might be it won't show up then i got to know i need to check on HF but till that time I have re-enabled input ...so I need to check again on HF when data will get stopped.

0 Karma

ips_mandar
Builder

Now I checked after data get stop but it will not conclude if input is stopped or not because it looks same -
exit status description exited with code 0
time opened 2018-10-16T19:10:05+0200
total bytes 17842283

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...