Qualys App for Splunk Enterprise: After changing python scripts to add more fields than the defaults, why are fields being truncated?

[Ovi]

Hi all

I need to pull into SPlunk more fields from Qualys than the defaults.
I changed the detectionpupulator.py and kppopulator.py scripts to add these fields and they do work - however I have a few challenges with field truncation

Especially the RESULTS fields is severely truncated reduced to just a few characcters from the first line

I am pasting below the details, if anyone had experience with these and can give me hint I would greatly appreciate it

1. Adding RESULTS field = Main Issue = severely truncated I changed detectionpopulator.py and added RESULTS as a detection fields to log detection_fields_to_log = ["QID", "TYPE", "PORT", "PROTOCOL", "SSL", "STATUS", "LAST_UPDATE_DATETIME", "LAST_FOUND_DATETIME","FIRST_FOUND_DATETIME", "LAST_TEST_DATETIME", "RESULTS"]

This works fine but the RESULTS field is severely truncated – it only contains the first few characters up to 10-20, always only form the first line.(I tried to change self.truncation_limit but that doesn’t seem to make any difference)
Question: how do I change this so that it will parse the full RESULTS field?

1. Similar question for the SOLUTION field I added in kbpopulator.py three additional fields. They are getting parsed fine but the SOLUTION field is still being truncated after 2000-3000 characters....and they want it all 🙂

QID_EXTRA_FIELDS_TO_LOG = ["VULN_TYPE", "PATCHABLE", "PCI_FLAG", "TITLE", "CATEGORY", "DIAGNOSIS", "CONSEQUENCE", "SOLUTION", "PUBLISHED_DATETIME"]
Question: basically same question for the SOLUTION field – how can I make sure the field is not truncated

1. HTML tags in SOLUTION field – any way to get rid of them? The SOLUTION field will have a lot of hyperlinks, the data comes in with all the HTML tags so it’s obviously hard to read and much lengthier Any way these tags can be stripped during the file parsing?

Thank you in advance

[muralianup]

I got rid of the HTML Tags using 4SED statements to get rid of all the HTML tags & formatting:

rex field=SOLUTION mode=sed "s/&q u o t //g" ->(without spaces)
rex field=SOLUTION mode=sed "s/<[^>]*>//g
rex field=SOLUTION mode=sed "s/- - & g t//g --> (without spaces)
rex field=SOLUTION mode=sed "s/& g t//g --> (without spaces)

[ltrand]

Is truncation happening when you run the script and output to a local file? Or is truncation happening after the script output? I would suggest looking in props.conf for the app and verify that TRUNCATE=0 is set. This will ensure that Splunk isn't truncating the script output. If the truncation is happening before that, I will defer to those that have a Qualys system to test with.

[srinathd]

In kbpopulator.py file i have added SOLUTION as additional field. I restarted it after the modification. But the solution values are not coming. How to reload this file?

[muralianup]

Did you try append this to the search ?

| lookup qualys_kb_lookup QID OUTPUT TITLE SEVERITY CATEGORY SOLUTION