Splunk Search

Splunk DB Connect 1: Why are large events > 10000 bytes being truncated to only 10K?

simonelias
New Member

Hi,

I have a DBX input as follow:

[dbmon-tail://HPNA-DB/HPNA-Configs]
host = HPNA-DB
index = hpnaconfigs
output.format = mkv
output.timestamp = 1
output.timestamp.column = LastSnapshotSuccessDate
output.timestamp.format = yyyy-MM-dd HH:mm:ss.SSS
query = with Configs as (\r\n select p.PrimaryIPAddress DeviceIP\r\n ,p.hostname DeviceName\r\n ,p.LastSnapshotSuccessDate\r\n ,ConfigTextId = (select top 1 dd.DeviceDataId from RN_Device d inner join rn_device_data dd ON   dd.DeviceID = d.DeviceID\r\n and d.DeviceID = p.deviceid \r\n and dd.BlockType = 'configuration' \r\n and dd.blockformat = 1\r\n    order by dd.LastModifiedDate desc\r\n                         )\r\n from RN_DEVICE p\r\n)\r\nselect   LastSnapshotSuccessDate\r\n ,DeviceName\r\n ,DeviceIP\r\n
,convert(varchar(50), dd.LastModifiedDate, 21) as LastModifiedDate\r\n  
,substring(DataBlock,1,100) as ConfigTextStart\r\n
,substring(DataBlock,datalength(DataBlock)-100,100) as ConfigTextEnd\r\n
,datalength(DataBlock) as ConfigTextLen1\r\n ,DataBlock as ConfigText\r\n
,datalength(DataBlock) as ConfigTextLen2\r\nfrom Configs c inner join rn_device_data dd on dd.DeviceDataId = c.ConfigTextId\r\n{{WHERE $rising_column$ > ?}}
sourcetype = dbmon:mkv
tail.rising.column = LastSnapshotSuccessDate
disabled = 0
interval = auto
table = HPNA-Configs

and the following props.conf stanzas in system/local, apps/dbx/local and apps/search/local:

[dbmon:mkv]
LINE_BREAKER_LOOKBEHIND = 100000
TRUNCATE = 0
MAX_EVENTS = 100000

However, when searching, events are being truncated after 10K.

Any idea?

0 Karma

jwelsh_splunk
Splunk Employee
Splunk Employee

I ran into a similar issue, was as if Splunk failed to honor the settings in props.conf. I ran across an answer (sorry can't find it now) that suggested using the tpl_*.dbmonevt source. It's solved my issue, could you try adding the following to your props.conf?

[source::...tpl_*.dbmonevt]
LINE_BREAKER_LOOKBEHIND = 100000
TRUNCATE = 0
MAX_EVENTS = 100000

0 Karma

simonelias
New Member

Thanks for the suggestion, however it had no affect, the events are still capped at 10K exactly.

Note: the last column "ConfigTextLen2" in the query is never visible...

0 Karma

simonelias
New Member

this is what the event is tag with as well:

host = HPNA-DB source = dbmon-tail://HPNA-DB/HPNA-Configs sourcetype = dbmon:mkv

0 Karma

ndoshi
Splunk Employee
Splunk Employee

A suggestion was made that if you are using the JDBC drivers that ship with DB Connect and this is MS SQL Server, to swap them out and use Drivers that are shipped directly from Microsoft.

0 Karma

simonelias
New Member

Splunk indexer is running on Linux, i don't believe MS made an SQL driver for this OS.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

yeah, they do make a Linux version -- you can get it here. http://www.microsoft.com/en-us/download/details.aspx?id=11774

I'm not positive that it's relevant to your problem, but we've found that it has fewer weirdnesses.

0 Karma

simonelias
New Member

wow, that is a surprise. Maybe i should try it.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

what type of database and which driver are you using?

0 Karma

simonelias
New Member

I am using MS SQL Server and the Java driver that comes with Splunk

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...