Getting Data In

filter data that are more than hundred K bytes

SplunkCSIT
Communicator

<!--wewe-->1212341234gfgdfggsgsdf-sgsdgsdgreg-3333

props.conf

[test]
BREAK_ONLY_BEFORE =<xml>
KV_MODE = xml
TRANSFORMS-test1 = content
transforms.conf

[content]
SOURCE_KEY=_raw
REGEX=(.?)<body>.?</body>(.*)
DEST_KEY=_raw
FORMAT=$1$2

it works for above xml but if the content in the are more than few hundred K bytes, it will not work. Any suggestion to resolve this? thks

Tags (1)
0 Karma

lguinn2
Legend

I believe that your problem is not that the .conf files are wrong, but that your events are larger than the Splunk defaults.
You can fix this by adding the following lines to your stanza in props.conf

TRUNCATE = 0
MAX_EVENTS = 10000

TRUNCATE is the maximum number of bytes in an event. Setting it to 0 means "no limit." You might want to set it to an actual maximum size.

MAX_EVENTS is the maximum number of lines in an event. The default is 256. I arbitrarily set it to 10,000 in the example.

Also, see the answer to your original question:

http://answers.splunk.com/answers/132023/propsconf-and-transformsconf-does-not-work

SplunkCSIT
Communicator

it does not work, any other alternative? thks

0 Karma

Lowell
Super Champion

Something else to keep in mind is the LOOKAHEAD value in transforms.conf. If you expect your regex to go more than 4K into the event, then the value will need to be increased. (LOOKAHEAD defaults to 4096 bytes.) Also be careful with the performance implications of changing LOOKAHEAD or the TRUCATE/MAX_EVENTS settings. It's fine to test with arbitrarily high numbers, but sometimes "tests" leak into production when they can kill performance. 😉

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...