SEP parsing is broken ? Unable to get the SEP data from the host machine to Splunk HF instances?

[Hemnaath]

Hi All, Currently we are not getting the Symantec data into Splunk?
Here is the process for sending logs from SEP to Splunk:

1. Host machine (Symantec Servers) forward logs to splunkxxx.com via UDP/514.
2. splunkthis.xxx.com load balances to Splunk heavy forwarders (splunkh01 and splunkh02)
3. The logs are stored on each heavy forwarder under this path /opt/syslogs/symantec/SymantecServer/SEP01/SEP01.log --> Source file where splunk monitors the data and currently there is no data

Configuration details -
Heavy Forwarder inputs.conf stanza :
[monitor:///opt/syslogs/symantec/SymantecServer/...]
whitelist = .log
index = sec_sym_av
sourcetype = sep
host_segment = 5

HF outputs.conf stanza :

[tcpout]
indexAndForward = false
forwardedindex.filter.disable = true
forceTimebasedAutoLB = true

syslog-ng.conf details ( Only partial details are shared )

INPUTS

source s_net { tcp( max-connections(1024) so_keepalive(yes) ); udp(); };

FILTERS

filter f_syn_vm { host("sep*" type(glob) flags(ignore-case) ); };
filter f_syn_ee_vm { host("semp*" type(glob) flags(ignore-case) ); };
filter f_syn_ee_grm { host("gmsemp*" type(glob) flags(ignore-case) ); };

COMBINATION FILTERS

filter f_syn { filter(f_syn_dc) or filter(f_syn_vm) or filter(f_syn_gen) or filter(f_syn_ee_gm) or filter(f_syn_ee_vm); };

DESTINATIONS

WHEN 3.5 is available use: file("/opt/syslogs/$(lowercase "${HOST}")/$(lowercase "${PROGRAM.log})"");

destination d_symdlp {
file("/opt/syslogs/symantec/${HOST}/${PROGRAM}.log");
};

destination d_syn {
file("/opt/syslogs/symantec/${HOST}/${PROGRAM}/${PROGRAM}.log");
};
log { source(s_net); filter(f_syn); destination(d_syn); };
log { source(s_net); filter(f_sym_dlp); destination(d_symdlp); };

When searched using the below query, got no result found and at same time there is no error found in the splunkd.log

index = sec_sym_av* sourcetype = sep* host= host=sep01*

Note - We are able to ping and retrieve system related information when we queried with the host name = and index= win_srv for the same server.
index = win_svrs* host=sep01*

Kindly guide us in fixing this issue as we need to get the Symantec data report.
thanks in advance

[esix_splunk]

Check the file system on the server and see if those files are being written to disk by syslog.

[Hemnaath]

You mean to check the Symantec server from where the data being generated and forwarded to syslog server ? and Symantec application is running in the Windows server, for which we do not have access.

Data are not written in this folder from syslog opt/syslogs/symantec/SymantecServer/SEP01/SEP01.log

[esix_splunk]

If the logs are not being written in the /opt/syslogs/symantec/SymantecServer/... folders, then the problem is that your Symantec Servers are not sending the logs, not Splunk.

I'd consult with your Symantec Admin.

[Hemnaath]

thanks esix, let me check with the Symantec Admin on this, and at the same do we need to check for any port number on the Symantec server side which is a windows server. How to find which port is being used ?

thanks in advance.

[esix_splunk]

They send to your syslog server (Splunk server). Default is 512 UDP, but your environment could be different.

[Hemnaath]

thanks, I have checked for the port 514 in Symantec server, but its not listening to it. I am sorry to ask could you guide me how and where to get that information.

[esix_splunk]

They send to your server. There is no open port on their side.

[Hemnaath]

thanks, but then how to fix this issue, as we need the Symantec reports for this servers.
kindly guide me what configuration should I need to check on the Symantec server side.

[esix_splunk]

Ask your Symantec Admin to validate log sending to your syslog host.

[Hemnaath]

thank esix, let me check with Symantec Admin.

[Hemnaath]

Hi esix, logs are being sent to the syslog host server from the Host machine (Symantec Server). when searched with the host name we are getting the result but with different source and source type from the same server.

source=/opt/syslogs/generic/SEP01/SymantecServer.log
sourcetype=syslog

Previously the data was being injected to this directory
Source = opt/syslogs/symantec/SymantecServer/SEP01/SEP01.log and source type
sep:risk, sep:traffic, sep, sep:agent

Kindly guide us in identifying the issue and how to fix this problem.

[Hemnaath]

Hi esix, I have doubt in editing the inputs.conf file. How to blacklist a particular log file from a particular host which is being monitored under two different stanza.

As you suggested for editing the inputs.conf to the new file location and to match the source type.

Currently under this stanza we are getting the Symantec information

[monitor:///opt/syslogs/generic/.../*.log]
sourcetype = syslog
host_segment = 4
blacklist = csc0*ltm*
index=unix_svrs

I have added this stanza in the existing inputs.conf, so that splunk can read the data which is being injected from Symantec server.

[monitor:////opt/syslogs/generic/SEP01/*.log]
whitelist = .log
sourcetype = sep
index=sec_sym_av
host_segment = 4

Now how to black list the same file which is being monitored by splunk in two different stanza in the same inputs.conf files.
Will this stanza can fix the Symantec issue for the client.

Kindly guide me on this points. thanks in advance.

[Hemnaath]

thanks Esix, There is no syslog administrator to whom I can check with regarding this, but will try to change the inputs.conf file like this

input stanza ( Currently set like this)
[monitor:///opt/syslogs/symantec/SymantecServer/...]
whitelist = .log
index = sec_sym_av
sourcetype = sep
host_segment = 5

But splunk monitors this path for the SEP01 server
[monitor:///opt/syslogs/generic/.../.log]
sourcetype = syslog
host_segment = 4
blacklist = dcsc0*ltm
index=unix_svrs

so, shall I modify or add the input stanza like this to get the sourcetype as sep?
[monitor:///opt/syslogs/generic/SEP01*/*.log]
whitelist = .log
index = sec_sym_av
sourcetype = sep
host_segment = 5

thanks in advance.

[esix_splunk]

You need to find out why syslog is writing the files as a different name the before. Splunk didnt change it, syslog or the Symantec server changed.

You can update your inputs.conf to reflect the new file names and adjust the sourcetypes to match. However, you really need to know what and why it changed.

[Hemnaath]

thanks esix, I had validated the syslog-ng.conf file detail it was lost modified on last year.

ls -ltr
total 36
-rw-r--r-- 1 root root 2146 Jan 15 2012 syslog-ng.conf.ORIGINAL
lrwxrwxrwx 1 root root 62 Nov 5 2013 splunk_logrotate -> /opt/splunk/etc/apps/ADMIN-hvy_forwarders/splunk_logrotate
drwxr-xr-x 2 root root 4096 Jul 25 2014 patterndb.d
-rw-r--r-- 1 root root 1513 Jul 25 2014 scl.conf
-rw-r--r-- 1 root root 1038 Jul 25 2014 modules.conf
-rw-r--r-- 1 root root 14992 Oct 28 2015 syslog-ng.conf

user wants the following source type "sep:risk, sep:traffic, sep, sep:agent", while searching the data, currently user fetching the sourcetype as syslog but he wants the above sourcetype.

thanks in advance.

[esix_splunk]

You need to have your syslog admin server troubleshoot what configuration changed. Or you can update your Splunk searches.

[Hemnaath]

Hi Esix, can you guide us on fixing this issue.
thanks in advance.

[esix_splunk]

Your monitor is looking for /opt/syslogs/symantec/SymantecServer/....

Is "SymantecServer" a replaced name you used in this config?

I would first trouble shoot this by looking on the actual filesystem and confirm the files are being written to disk. My guess is that your inputs are not matching with what syslog is writing.

[Hemnaath]

Hi esix, thanks for your effort on this issue, Both the syslog server and HF forwarder instance running in the same server and we were getting the data under the source path " /opt/syslogs/symantec/SymantecServer/SEPP01/SEPP01.log". we have verified by executing the below query for the time period set as two months.

host=sep01* sourcetype=sep* index=sec_sym_av* source="/opt/syslogs/symantec/SymantecServer/SEP01/SEP01.log"

3/2/16
11:59:58.000 PM
Mar 2 23:59:58 SymantecServer SEP01: W2X9KVZ1,Category: 0,Smc,Disconnected from Symantec Endpoint Protection Manager (SEP02.xxxx.com)
host = SEP01 source = /opt/syslogs/symantec/SymantecServer/SEP01/SEP01.log sourcetype = sep:agt_system
3/2/16
11:59:58.000 PM
Mar 2 23:59:58 SymantecServer SEP01: W2X9KVZ1,Category: 0,Smc,Connected to Symantec Endpoint Protection Manager (SEP02.xxxx.com)
host = SEP01 source = /opt/syslogs/symantec/SymantecServer/SEP01/SEP01.log sourcetype = sep:agt_system

But currently when we execute the above query we are not getting any data and more over there is no data being injected under the source path in the HF server instance.

source="/opt/syslogs/symantec/SymantecServer/SEP01/SEP01.log" - Empty file

I would first trouble shoot this by looking on the actual filesystem and confirm the files are being written to disk

I am not sure what you mean by actual file system ?

Kindly guide me on how to troubleshoot this issue, as we need to generate Symantec reports.