Getting Data In

Events getting distorted in splunk production

swamysanjanaput
Explorer

Hi Splunkers,

I am trying to ingest os_metrics logs from one of our prod server to splunk. In QA and dev instance, events are breaking correctly. I pushed the same configs(see below) to production server however i see distorted events when searching the data in prod SH for e.g Thu 10/10/2019 0:43:56.32 Checking "ABC" as one event and ping results as another event. Similarly Thu 10/10/2019 0:44:18.12 Get MAC Address for "PQR" as one event and physical address details as another event(below is the sample data)

Splunk is reading old data files from production server and i am able to see old data breaking into events correctly but when new data started to ingest, i see them all getting distorted So, Do i have to place props in our SH cluster or is it something to do with props?

Can someone please help me to resolve this issue? Thanks in advance.

Sample data:
Thu 10/10/2019 0:43:56.32 Checking "ABC"

Pinging ABC [ip] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 0.0.0.0:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

Thu 10/10/2019 0:44:18.12 Get MAC Address for "PQR"

Physical Address Transport Name

=================== ==========================================================
\Device\Tcpip_{}

N/A Media disconnected

N/A Media disconnected

N/A Media disconnected

props.conf
[xyz]
NO_BINARY_CHECK=true
CHARSET=UTF-8
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE = \w+\s+\d+\/\d+\/\d+\s+\d+:\d+:\d+
disabled=false

inputs.conf
[monitor://abc*.log]
disabled = 0
index = xxxxx
sourcetype = xyz

Tags (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi swamysanjanaputta,
have you in production environment also Heavy Forwarders between sources and Indexers?
If yes, put the props.conf also on Heavy Forwarders (and restart Splunk on them).

Ciao.
Giuseppe

0 Karma

swamysanjanaput
Explorer

Hi, Yes i had initially deployed props to HFs, not sure why data is getting distorted, i see 50% events distorted and other 50% breaking into events correctly. so should i place props on Search Head cluster?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Change your sourcetype definition in props.conf with this

[xyz]
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=\w+\s+\d+\/\d+\/\d+\s+\d+:\d+:\d+)
TIME_PREFIX = ^
TIME_FORMAT = %a %m/%d/%Y %H:%M:%S.%N
0 Karma

swamysanjanaput
Explorer

Thanks for the props, still facing the same issue. I had placed props in HFs aswell but not sure why data is getting distorted. So, do i have to place the props in SH cluster? Please advise..

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...