Splunk Search

Splunk DB Connect 1: Why are large events > 10000 bytes being truncated to only 10K?

simonelias
New Member

Hi,

I have a DBX input as follow:

[dbmon-tail://HPNA-DB/HPNA-Configs]
host = HPNA-DB
index = hpnaconfigs
output.format = mkv
output.timestamp = 1
output.timestamp.column = LastSnapshotSuccessDate
output.timestamp.format = yyyy-MM-dd HH:mm:ss.SSS
query = with Configs as (\r\n select p.PrimaryIPAddress DeviceIP\r\n ,p.hostname DeviceName\r\n ,p.LastSnapshotSuccessDate\r\n ,ConfigTextId = (select top 1 dd.DeviceDataId from RN_Device d inner join rn_device_data dd ON   dd.DeviceID = d.DeviceID\r\n and d.DeviceID = p.deviceid \r\n and dd.BlockType = 'configuration' \r\n and dd.blockformat = 1\r\n    order by dd.LastModifiedDate desc\r\n                         )\r\n from RN_DEVICE p\r\n)\r\nselect   LastSnapshotSuccessDate\r\n ,DeviceName\r\n ,DeviceIP\r\n
,convert(varchar(50), dd.LastModifiedDate, 21) as LastModifiedDate\r\n  
,substring(DataBlock,1,100) as ConfigTextStart\r\n
,substring(DataBlock,datalength(DataBlock)-100,100) as ConfigTextEnd\r\n
,datalength(DataBlock) as ConfigTextLen1\r\n ,DataBlock as ConfigText\r\n
,datalength(DataBlock) as ConfigTextLen2\r\nfrom Configs c inner join rn_device_data dd on dd.DeviceDataId = c.ConfigTextId\r\n{{WHERE $rising_column$ > ?}}
sourcetype = dbmon:mkv
tail.rising.column = LastSnapshotSuccessDate
disabled = 0
interval = auto
table = HPNA-Configs

and the following props.conf stanzas in system/local, apps/dbx/local and apps/search/local:

[dbmon:mkv]
LINE_BREAKER_LOOKBEHIND = 100000
TRUNCATE = 0
MAX_EVENTS = 100000

However, when searching, events are being truncated after 10K.

Any idea?

0 Karma

jwelsh_splunk
Splunk Employee
Splunk Employee

I ran into a similar issue, was as if Splunk failed to honor the settings in props.conf. I ran across an answer (sorry can't find it now) that suggested using the tpl_*.dbmonevt source. It's solved my issue, could you try adding the following to your props.conf?

[source::...tpl_*.dbmonevt]
LINE_BREAKER_LOOKBEHIND = 100000
TRUNCATE = 0
MAX_EVENTS = 100000

0 Karma

simonelias
New Member

Thanks for the suggestion, however it had no affect, the events are still capped at 10K exactly.

Note: the last column "ConfigTextLen2" in the query is never visible...

0 Karma

simonelias
New Member

this is what the event is tag with as well:

host = HPNA-DB source = dbmon-tail://HPNA-DB/HPNA-Configs sourcetype = dbmon:mkv

0 Karma

ndoshi
Splunk Employee
Splunk Employee

A suggestion was made that if you are using the JDBC drivers that ship with DB Connect and this is MS SQL Server, to swap them out and use Drivers that are shipped directly from Microsoft.

0 Karma

simonelias
New Member

Splunk indexer is running on Linux, i don't believe MS made an SQL driver for this OS.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

yeah, they do make a Linux version -- you can get it here. http://www.microsoft.com/en-us/download/details.aspx?id=11774

I'm not positive that it's relevant to your problem, but we've found that it has fewer weirdnesses.

0 Karma

simonelias
New Member

wow, that is a surprise. Maybe i should try it.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

what type of database and which driver are you using?

0 Karma

simonelias
New Member

I am using MS SQL Server and the Java driver that comes with Splunk

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...