Getting Data In

Stack trace data truncated when indexed. Search shows some data or punctuations missing.

Ellen
Splunk Employee
Splunk Employee

We are running Splunk 4.3.4.

Here is a sample stack trace from the server along with the results from a search for the same source.

To sum up what we are seeing, at line 70, Splunk starts removing periods and underscores.
It also removed common parts of the stack trace.

This is making the stack trace very hard to read and use for troubleshooting issues.

Below are sample lines from the actual stack trace:

at org.apache.yoyo.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) ~[yoyo.jar:6.0.29]
at com.abc.common.filter.VisitPageEventPublishFilter.doFilterInternal(VisitPageEventPublishFilter.java:44) ~[xyz_web_app.jar:13.11]
at com.zzz.xyz.common.web.filter.SpringManagedFilter.doFilter(SpringManagedFilter.java:97) ~[hcom-websupport-11.5.jar:11.5]

Below are the search results of the same data set for the stack trace:

at org.apache.yoyo.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) ~[:.yoyojar6.0.29]
at ...() [..comabccommonfilterVisitPageEventPublishFilterdoFilterInternal:.VisitPageEventPublishFilterjava44~.:.]
__xyzwebappjar1311at.....() [..comzzzxyzcommonwebfilterSpringManagedFilterdoFilter:.SpringManagedFilterjava97~..:.]

You can see, it starts removing the periods and underscores along with common parts of the line.

We have indexed the attached stack trace into a local instance of Splunk and it is not showing the same problems when we run a search against it.

We have also confirmed that the only entry we have in the prop.conf or transform.conf files is for the max events setting which is copied below. We setup the same setting on our local instance and re-index the stack trace without any issues, so we do no think it is the cause of the issues.

[hwa_app]
MAX_EVENTS=1024 

How can we troubleshoot?

Tags (3)
1 Solution

Ellen
Splunk Employee
Splunk Employee

First verify what you see in the browser is really how Splunk indexed the data.

A quick check to ensure this was not a browser issue is to run the same search under another browser.

Verify the actual data events indexed are the same as the raw by either
1) selecting 'Export' in any browser or
2) run the search via CLI to view the results.

This was a browser issue as it was confirmed the data was indexed properly.

Under Safari Version 6.0.2 (7536.26.17) the issue was reproducible and the results appear to be mangled.
There were some earlier versions of Chrome where this was also reproduced.

This is a known bug related to Webkit and no longer an issue in Splunk 5.0 since the event viewer has changed. (SPL-55380/SPL-55354)

So check what browsers and version are in use and apply the workarounds below.

The following browser versions could not reproduce the issue and displayed as expected.

IE 9.0.8112.16421
FF 17.0.1
Chrome Version 23.0.1271.95

Workarounds:
1) try using another browser or a more recent version of Firefox, Chrome, or IE.
or
2) you can try to work in the search results UI, select the 'Option' > this will display a 'Results Display Options' window and one of the display options is to select the event segmentation. By default it is 'full' but if you change it to outer or inner, you will not see the odd results.
Don't confuse this with index time segmentation. This workaround is for the search time display.

http://docs.splunk.com/Documentation/Splunk/4.3.4/Data/SetsegmentationinSplunkWeb

View solution in original post

Ellen
Splunk Employee
Splunk Employee

First verify what you see in the browser is really how Splunk indexed the data.

A quick check to ensure this was not a browser issue is to run the same search under another browser.

Verify the actual data events indexed are the same as the raw by either
1) selecting 'Export' in any browser or
2) run the search via CLI to view the results.

This was a browser issue as it was confirmed the data was indexed properly.

Under Safari Version 6.0.2 (7536.26.17) the issue was reproducible and the results appear to be mangled.
There were some earlier versions of Chrome where this was also reproduced.

This is a known bug related to Webkit and no longer an issue in Splunk 5.0 since the event viewer has changed. (SPL-55380/SPL-55354)

So check what browsers and version are in use and apply the workarounds below.

The following browser versions could not reproduce the issue and displayed as expected.

IE 9.0.8112.16421
FF 17.0.1
Chrome Version 23.0.1271.95

Workarounds:
1) try using another browser or a more recent version of Firefox, Chrome, or IE.
or
2) you can try to work in the search results UI, select the 'Option' > this will display a 'Results Display Options' window and one of the display options is to select the event segmentation. By default it is 'full' but if you change it to outer or inner, you will not see the odd results.
Don't confuse this with index time segmentation. This workaround is for the search time display.

http://docs.splunk.com/Documentation/Splunk/4.3.4/Data/SetsegmentationinSplunkWeb

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...