All Apps and Add-ons

Why is Splunk eNcore not receiving any Data?

fwbagusf
New Member

Hi All,

Currently we noticed that splunk eNcore not receiving any Data, the error log on cisco:estreamer:log said ERROR EncoreException: PID file already exists. So we assumed that there is conflicted eNcore process, so we try to ./splencore.sh status but found other error said "/etc/apps/TA-eStreamer/bin/encore/" does not exist

After searching for any info, we thought that there is problem with the pkc12 file, so we generated new file from FMC and put the new one into /TA-eStreamer/bin/encore and put the password from setup Page. Unfortunately while we try to save this configuration, another error occurred that said Encountered the following error while trying to update: Splunkd daemon is not responding: (u"Error connecting to /servicesNS/nobody/TA-eStreamer/apps/local/TA-eStreamer/setup: ('The read operation timed out',)",)

Any idea how to fix this?

Thank you for all your assistance.

alt text

Labels (1)
0 Karma

vinz2020
Engager

same issue for me with the last app release 3.6.8
https://splunkbase.splunk.com/app/3662/

and FMC v6.4.0.7

I received some data events during a few minutes and then the estreamer process goes down (and looping for a while)

0 Karma

mkemp
Observer

Per the deployment guide we have three options: https://www.cisco.com/c/en/us/td/docs/security/firepower/630/api/eStreamer_enCore/eStreamereNcoreSpl...

 

■        0: Send all events from the earliest point available on the Firepower Management Center

■        1: Send all events that occur after receiving the client request

■        2: Use a bookmark to pick up where we left off. First run is from 0

 

So, first modify the file to use option 0. Restart the encore and leave it running some time and verify if you see events.

After that you can modify the file to option 1 and restart the encore again and verify if events are seen in encore.

0 Karma

jbrinkman
Explorer

I would take a look within $SPLUNK_HOME/etc/apps/TA-eStreamer/bin/encore. Should have a file referencing the FMC you had configured to reach out to. Mine was IP-port_proc.pid. Rename or delete that file. Restart Splunk. I began ingesting immediately following the restart.

shahrukhvp
Explorer

This resolved my issue. 

0 Karma

Mesa_Splunkr
Loves-to-Learn

Anything recent on this issue? I just discovered I am having the same. Just digging in now. Thanks for posting and your comments.

0 Karma

mhessel
Path Finder

I had to back out to the older version 3.0 and the events came back. I don't know why, but the estreamer process will start, and I get a short burst of events, but stops and nothing happens from that point on.

0 Karma

xtrjx
Explorer

I am experiencing the same issue. The 3.5 version will retrieve a burst of events but then stop. Every time I restart the estreamer processes, either from the Splunk UI by enabling/disabling eNcore, or from the shell using kill -HUP on the estreamer processes, or by restarting splunk entirely, I always get a burst of events when eNcore starts but then it will stop. I can wait hours and still not get a new event until I restart. Then I will get a burst of events from the last "bookmark". I've tried changing many of the values in estreamer.conf including connectTimeout, responseTimeout, workerProcesses, with no luck. I am running on a server with 16 CPUs and 64GB RAM so have plenty of power. estreamer events get processed just fine with the old 2.2.1 version of the estreamer app/addon.

0 Karma

mhessel
Path Finder

This might be related to a bug on the FMC, where the estreamer service will peg a cpu at 100 percent if FireAMP or File events are enabled. Cisco tracks this as bug #CSCvj07843

I was able to get this working by disabling the event types in the FMC, only allowing intrusion and malware events, and now I am getting events in splunk, but that is still running 30 minutes to an hour behind. This does not seem to be related to the estreamer usage on the splunk/client side, but on the FMC instead.

I'm guessing with the additional protocol support added in 3.5.0, the estreamer service on the server side is slowing down too much.

From the bug report, fixed versions of the FMC are 6.2.2.4, and 6.2.3.2.

0 Karma

xtrjx
Explorer

Additional details: I enabled DEBUG estreamer logging. Now I noticed the following log messages estreamer.log that correspond with the data flow stopping:

2018-07-28 06:07:29,722 Decorator    DEBUG    Stashing sequence 80874; buffer: 1

Those messages will repeat with the stashing sequence incrementing by about 100 at a time and buffer number incrementing from 1 to n until I restart the estreamer processes. For example, this is the last log entry after letting estreamer run all weekend:

2018-07-30 15:36:41,905 Decorator    DEBUG    Stashing sequence 167072; buffer: 857
0 Karma

Blackburn2413
New Member

Same boat here. Same message runs all day now. estreamer shows as running in the Cisco Security plugin, but no logs, alerts, results... Nothing.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...