Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

[phularah]

Are there any specific ports or specific permissions this add-on requires/uses, so that I can inform the team, so if any modifications are made data flow is not interrupted.

I have configured Microsoft Log Analytics Add-on in Heavy Forwarder and forwarding the logs received to indexer. There is no clustering. I would like to hear from @jkat54 and @dpanych. Any ideas, why this keep on happening.

I used

index=_internal log_level=err* OR log_level=warn loganalytics*

The latest event I am getting some results using this query is

09-05-2018 18:24:24.168 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" ERROR('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

[jkat54]

Can you try changing the following in input_module_log_analytics.py?

Please change:

    #Delta
    state = now_dt.strftime("%d/%m/%Y %H:%M:%S")

To:

    #Delta
    state = start_datetime.strftime("%d/%m/%Y %H:%M:%S")

And let me know if that fixes this bug please.

[phularah]

Hi @jkat54. I was just waiting for the problem to arise again but looks like we are not getting same problem of data stoppage now in almost a week. I did not made this change

state = start_datetime.strftime("%d/%m/%Y %H:%M:%S")

The only change that I had made after last problem was to copy props.conf from Heavy forwarder and pasting it in apps\search\local local in Search Head. No changes were made in OMS as per my knowledge. I am still not sure what was the problem initially, but everything is working fine now.

Thanks for writing this app. Keep up the good work.

[jkat54]

Ok, thanks and you’re welcome!

[jkat54 · Accepted solution]

It connects to the log analytics API on TCP port 443 aka HTTPS.

Nothing else is needed.

[phularah]

I am not sure why is it happening then. I tried making a new input but still I am unable to see any data. Any idea why it might be happening?

[phularah]

After making new inputs and deleting old inputs, I am getting data now. But, I don't know why it stopped in the first place and now after first making a new input and disabling previous input, data didn't come. I again made a new input after deleting all disabled inputs, now I am getting data. It is really frustrating, and I am unable to pinpoint the source.

Now, new results are coming where log_level=warn,

09-06-2018 10:50:34.492 +0200 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 14261 - data_source="log_analytics://analytics", data_host="Hostname", data_sourcetype="loganalytics"

but it is of no concern, my concern is this error.
Could you please tell why this error might show up.

09-05-2018 18:24:24.168 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" ERROR('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

[jkat54]

Sounds like the checkpoints were messed up. Did you upgrade from a previous version of the app or install fresh/new?

[phularah]

Yes, I had upgraded the add-on. I was using the latest add-on before this problem surfaced.

[jkat54]

Ok looks like the process for upgrading should have been to delete the inputs, upgrade the app, add the inputs.

this is due to the way the checkpoints changed between versions. My apologies for the inconvenience.

Is every input you’ve re-added working now?

[jkat54]

Was everything working after removing and adding the inputs after the upgrade?

[493669]

@jkat54,
is there any retry attempts for input in OMS?
like due to some reason in oms, splunk unable to collect data for 5 min. but after 5 min. everything fine at OMS side...but then splunk unable to receive any kind of data ..it get stopped so is there any retry attempt like it will try to connect with OMS for few attempts and then it will stop attemptting to connect with OMS?

[jkat54]

The “connection broken” error suggests a proxy or firewall or other network issue.

No one else is reporting this error so I believe it to be something with your environment only.

As for if the add on retries connections, no it only attempts one connection per interval.

[jkat54]

Wait... this question wasn’t by you. @493669 do you work with phularah?

If not, you should create your own question.

If yes, then see this link for how to resolve the error you have: https://docs.microsoft.com/en-us/azure/active-directory/application-sign-in-problem-federated-sso-ga...

[phularah]

Yes, @jkat me and @493669 are working together. My comments were sent to moderator, so I asked my colleague @493669 to post these comments. Now, I can see my posted comments.

[phularah]

@jkat54 , but if we have any error at oms side then data will not come... but after I deleted previous input and created new input , data started flowing again...so it doesn't seems to be issue at OMS side..isn't it?

[jkat54]

When you recreate do you recreate in splunk only?

[phularah]

Yes, we create new inputs in Splunk only.

[phularah]

can you please share your take on this, why it might be happening? I had a chat with Azure guys in my team, they say everything is working fine at their side and they are not making any changes.

[phularah]

Also, we are getting these messages running this query, index=_internal log_level=err* loganalytics*. Again after working for a few hours data has again stopped coming.

These are the error messages, we are getting:

9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" UnboundLocalError: local variable 'data' referenced before assignment

    host =  *****   
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"     for i in range(len(data["tables"][0]["rows"])):

    host =  ********    
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\input_module_log_analytics.py", line 86, in collect_events

    host =  ******  
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"     input_module.collect_events(self, ew)

    host =  *****
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py", line 96, in collect_events

    host =  *****
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"     self.collect_events(ew)

    host =  *******
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

[jkat54]

This means the query didn’t return any results.

Any idea how long it takes before it breaks?