Deployment Architecture

How to find the log duplicate reasons?

xsstest
Communicator

UF(on the log source server)——> HF——> cluster indexer

UF also a deployment-client, HF also a deployment-server Yesterday, I found a lot of repetition in the logs.

Each event repeat 34 to 55 times,I searched for the relevant host in the _internal index, but no error or warning.

How can I find out why the log is repeated? I need a way of troubleshooting.

Under normal circumstances, the log duplication several possibilities?

index=<your index> sourcetype=<sourcetype> source="<source>" host="<host>" | eval bucket=_bkt | eval indextime=_indextime |table _time, indextime, bucket splunk_server _raw | convert ctime(indextime) | stats count list(*) as * by _raw | where count>1 | fields * _raw

above search get (Returning 1417172 results, the following is just an example.):

alt text

thank you !

Tags (1)
0 Karma

Elsurion
Communicator

I hardly doubt that splunk will send the data to both tcp destinations but to ensure that he's taking only one add

_TCP_ROUTING=<tcpout_destination>

to your inputs.conf

With this way you are going to tell which path the data has to take.

Also why are you using the Syslogport for Splunk> Data Indexing?

0 Karma

xsstest
Communicator

Because there are some problems with the default 9997 port

0 Karma

lguinn2
Legend

I understand your question, but the search that you ran isn't really helpful.

First, take a look at the actual source file on the Universal Forwarder - does the log file (or whatever it is) have repeated data?

Second, take a look at the splunkd.log from the Universal Forwarder. You can do this by searching the _internal index ( index=_internal host=ufname sourcetype=splunkd ) or by simply browsing the SPLUNK_HOME/var/log/splunk/splunkd.log on the Universal Forwarder. The best place to look in the log is during the Splunk startup process; what does Splunk do when it starts to read the files?

Finally, Splunk tracks its status for each file that it is monitoring in the internal "fishbucket." Splunk must be able to maintain this state information. If log files are not appended, but have random updates throughout their records, this will confuse Splunk and it may duplicate data. There are a few other reasons that Splunk might think a file has changed when it has not, but you should get hints from the splunkd.log - when Splunk starts reading at the beginning of a monitored file (for whatever reason), it will log that fact.

Once you look at all this, you could post back with any specific messages that you find in the Splunk logs.

0 Karma

xsstest
Communicator

thank you for your answer.

first, ctual source file no repeated data.
second. check the log when uf starts to read the file ,no problems have been found
third.The program or application does not write to the random line.

I am distributing to client through deployement-server. Each app contains inputs. conf and outputs.conf, all the outputs. conf server is same HF. Is this the cause of duplication?

deployment-client (use deployment-server distributing to them)

app_name: linux_syslog

/opt/splunkforwarder/etc/apps/linux_syslog/inputs.conf

[monitor:///var/log/audit/audit.log]
index=linux_syslog
sourcetype=zx_syslog

[monitor:///var/log/secure]
index=linux_syslog
sourcetype=zx_syslog

/opt/splunkforwarder/etc/apps/linux_syslog/outputs.conf

[tcpout]
defaultGroup=zx_syslog

[tcpout:zx_syslog]
server=172.25.94.71:514

app_name:tomcat_log

/opt/splunkforwarder/etc/apps/tomcat_log/inputs.conf

[monitor:///data/www/log/register-app.log]
index=tomcat_log
sourcetype=zx_tomcat_login-app

[monitor:///data/www/log/register-app.log]
index=tomcat
sourcetype=zx_tomcat_register-app

/opt/splunkforwarder/etc/apps/tomcat_log/outputs.conf

[tcpout]
defaultGroup=tomcat_log

[tcpout:tomcat_log]
server=172.25.94.71:514
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...