Getting Data In

How to resolve timestamp and line processing issues in pdfgen.log ?

damode
Motivator

I am getting the below two warning messages,
1. 11-27-2017 06:00:22.902 +1100 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Mon Nov 27 06:00:20 2017). Context: source::C:\Program Files\Splunk\var\log\splunk\pdfgen.log|host::INDEXER|splunk_pdfgen|20662

11-27-2017 06:00:16.835 +1100 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 17586 - data_source="C:\Program Files\Splunk\var\log\splunk\pdfgen.log", data_host="INDEXER", data_sourcetype="splunk_pdfgen"

  1. Sample timestamp in pdfgen.log looks like this
    2017-11-27 06:01:00,206 +1100 INFO pdfgen_table:1041 - renderTable> headerRow: ['host', 'src_interface', 'port_status', 'count']
    2017-11-27 06:01:09,519 +1100 INFO pdfgen_endpoint:271 - Generated pdf, filename = overview-2017-11-27.pdf

  2. props.conf
    [splunk_pdfgen]
    TIME_FORMAT = %m-%d-%Y %H:%M%S,%l
    SHOULD_LINEMERGE = False
    MAX_TIMESTAMP_LOOKAHEAD = 40

arekdabrowski
Explorer

I have the same problem on version 7.3.1
When I have the default props.conf file in the pdfgen file, my data quality displays problems with timestamp analysis, here are the details:
01-15-2020 11:56:18.641 +0100 WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (40) characters of event. Defaulting to timestamp of previous event (Wed Jan 15 11:56:15 2020). Context: source=/opt/splunk/var/log/splunk/pdfgen.log|host=xxxxxxxxxxxxx|splunk_pdfgen|2557
When I add to my props.conf on the system / local / props.conf TIME_FORMAT =% Y-% m-% d% H:% M:% S,% 3N% z
I also have the same problem.
Do you have any ideas?

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi damode,

the TIME_FORMAT = %m-%d-%Y %H:%M%S,%l should be TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N.
Regarding the truncating add TRUNCATE = 20000 to the props.conf

Hope this helps ...

cheers, MuS

damode
Motivator

Hi @MuS,

Thanks for your prompt reply.

I have applied the suggested settings. Will let you know the outcome.

Regards,
Dev

0 Karma

damode
Motivator

Hi @MuS,

I am not getting Truncating line issue anymore. Thanks for that! I am still, however, getting the timestamp issues.

  1. 11-28-2017 06:00:16.854 +1100 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Tue Nov 28 06:00:14 2017). Context: source::C:\Program Files\Splunk\var\log\splunk\pdfgen.log|host::INDEXER|splunk_pdfgen|126
  2. props.conf [splunk_pdfgen] TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N SHOULD_LINEMERGE = False MAX_TIMESTAMP_LOOKAHEAD = 40 TRUNCATE = 20000
0 Karma

MuS
SplunkTrust
SplunkTrust

I just checked the default settings for [splunk_pdfgen] and it actually has this option set:

 TIME_FORMAT = %m-%d-%Y %H:%M%S,%l

So, please remove the TIME_FORMAT you added and try again - really wired...

Can you run this command /opt/splunk/bin/splunk btool props list splunk_pdfgen --debug and compare to this list of options please:

/opt/splunk/etc/system/default/props.conf                  [splunk_pdfgen]
/opt/splunk/etc/system/default/props.conf                  ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf                  ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf                  AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf                  BREAK_ONLY_BEFORE = 
/opt/splunk/etc/system/default/props.conf                  BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf                  CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf                  DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf                  HEADER_MODE = 
/opt/splunk/etc/system/default/props.conf                  LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf                  LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf                  LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf                  MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf                  MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf                  MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf                  MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf                  MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf                  MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf                  MAX_TIMESTAMP_LOOKAHEAD = 40
/opt/splunk/etc/system/default/props.conf                  MUST_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                  MUST_NOT_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                  MUST_NOT_BREAK_BEFORE = 
/opt/splunk/etc/system/default/props.conf                  SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf                  SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf                  SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf                  SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf                  SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf                  SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf                  SHOULD_LINEMERGE = False
/opt/splunk/etc/system/default/props.conf                  TIME_FORMAT = %m-%d-%Y %H:%M%S,%l
/opt/splunk/etc/system/default/props.conf                  TRANSFORMS = 
/opt/splunk/etc/system/default/props.conf                  TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf                  detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf                  maxDist = 100
/opt/splunk/etc/system/default/props.conf                  priority = 
/opt/splunk/etc/system/default/props.conf                  sourcetype = 
0 Karma

damode
Motivator

Hi @MuS,

Upon comparing with the above list of options, I found the below fields having different value in comparison to yours. Everything else is same.
CHARSET = AUTO
TRUNCATE = 20000
detect_trailing_nulls = auto

0 Karma

damode
Motivator

Hi @MuS, I had changed back to default TIME_FORMAT, but that still gave the same issue.
Based on the above observation, do you recommend setting the [splunk_pdfgen] attributes exactly same as yours ?

0 Karma

MuS
SplunkTrust
SplunkTrust

Well, the above settings are the Splunk default settings so they really should work.

0 Karma

damode
Motivator

Now I am getting the same error from datasourcetype = licensealert-5 as well, in addition to splunk_pdfgen.

0 Karma

MuS
SplunkTrust
SplunkTrust

That sounds like a bigger problem here .... also reading all you other questions.

Random question: have you done a FS check lately on your Splunk server to see if everything is healthy?

0 Karma

damode
Motivator

If you mean health check on DMC, then yes.
On Search head, I have license warning and scheduled searches skipped messages. On Indexer, I am getting these event processing issue about which I have posted here.

0 Karma

MuS
SplunkTrust
SplunkTrust

No I meant an actual file system check from the operating system.

0 Karma

damode
Motivator

Hi @MuS, for some reason, the Search Head had the same hostname as the Indexer. Not sure how and when I did that. Once I changed it to its correct username, I stopped getting time parsing warning messages. I believe, that’s probably what was causing the issue.

0 Karma

damode
Motivator

I just did a file system check from the operating system using SFC.EXE /scannow and did not find any integrity violations.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...