Getting Data In

Very long log events coming over syslog 514/udp are cut- How would I resolve this?

jrodriguezap
Contributor

Hello
Someone will have happened that the logs come with a length of 1000 characters at most, and these are indexed incompletely?
How could I do to be stored completely all characters?
The source I use is syslog 514/udp.

I would appreciate your support.
greetings

Labels (1)
Tags (4)
0 Karma

jrodman
Splunk Employee
Splunk Employee

syslog-udp cannot transport 1500 character events, beause that exceeds ethernet MTU, and syslog-over-udp is a single packet.
The data is likely being truncated before splunk receives it.

Consider a more reliable transport, such as syslog over tcp or splunk forwarders.

jrodriguezap
Contributor

that's the detail. The same happens to me with an F5 ASM

0 Karma

jrodman
Splunk Employee
Splunk Employee

For syslog-udp there is a maximum possible size that the messages can be in the way they are transmitted over the network. Splunk cannot help here.
For syslog-tcp, splunk does not even know that it is accepting syslog when accepting the data. There is nothing in splunk that truncates lines other than the TRUNCATE setting described by MuS, which defaults to 10KB.

jrodriguezap
Contributor

Hi jrodman
I tried also with syslog via TCP / 514, and also truncated, that's normal?
I am what I am doing from ironport wsa

0 Karma

nitsud
New Member

You can only choose syslog for text-based logs.

The Syslog Push method sends log messages to a remote syslog server on port 514. This method conforms to RFC 3164.

Maximum message size is configurable on the WSA

You can increase the maximum message size on a log subscription in the WSA. It defaults to 1024. 

When you choose this method, you must enter the following information:

  • Syslog server hostname
  • Protocol to use for transmission, either UDP or TCP
  • Maximum message size

    Valid values for UDP are 1024 to 9216.

    Valid values for TCP are 1024 to 65535.

    Maximum message size depends on the syslog server configuration.

  • Facility to use with the log
0 Karma

jrodman
Splunk Employee
Splunk Employee

I think this really is a question for Cisco/Ironport. How can the device be configured to produce its data in a complete way?

0 Karma

sgailey_splunk
Splunk Employee
Splunk Employee

Syslog-ng can transport much longer events; up to 8K I believe but the syslog shipped with most linux and UNIX distributions can't, even when using a tcp transport.

0 Karma

jrodman
Splunk Employee
Splunk Employee

Yeah i think the typical syslog follows the recommended max length specification. Oops, I forgot.

0 Karma

pradeepkumarg
Influencer

set below property in props.conf

TRUNCATE =
* Change the default maximum line length (in bytes).
* Although this is in bytes, line length is rounded down when this would
otherwise land mid-character for multi-byte characters.
* Set to 0 if you never want truncation (very long lines are, however, often a sign of
garbage data).
* Defaults to 10000 bytes.

More details here
http://docs.splunk.com/Documentation/Splunk/6.1.4/Admin/Propsconf

0 Karma

jrodriguezap
Contributor

Hi gpradeepkumarreddy
Thanks for your answer, I have understood that 1 character = 1 byte approx.
However, the lines are approximately 1500 characters, yet are cut, switch to 0 and continue slashing. Have something to do it syslog 514/udp?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...