Splunk Search

Splunk indexes text file in binary format ?

sieutruc
Contributor

Hello,

When i monitored a file , at first its content is forwarded from forwarder to indexer in text format, so i can make a table with that content.

But after the system has updated that file by deleting it and creating a new same-name file with different content, I see that Splunk indexes its new data in binary format

5:21:47.000 PM

\x001\x002\x00/\x001\x002\x00/\x001\x002\x00 \x001\x008\x00:\x000\x005\x00:\x001\x007\x00,\x00 \x000\x000\x003\x002\x009\x00 \x00[\x000\x00x\x000\x007\x00E\x004\x00]\x00 \x00=\x00>\x00[\x00T\x00R\x00A\x00N\x00S\x00A\x00C\x00T\x00I\x00O\x00N\x00I\x00N\x00F\x00O\x00

However, i can open this file in notepad and view its content without any issue. So can you tell me what is the problem i have got ?

Tags (1)
0 Karma
1 Solution

sieutruc
Contributor

Yeah, it's worked. This charset has to be set on HF. Thanks

View solution in original post

0 Karma

sieutruc
Contributor

Yeah, it's worked. This charset has to be set on HF. Thanks

0 Karma

srinathd
Contributor

I think, If you open a file in the forwarder, it will create .swp file in the same folder, as and when .swp file is created it will be forwarded to indexer for indexing. Thats why you will see that binary format data and also you can set charset as HF.

0 Karma

sideview
SplunkTrust
SplunkTrust

I'd guess you have to set charset in the HF as well. That's where the real "cooking" part of the indexing process is occurring.

0 Karma

sieutruc
Contributor

Yeah, data is forwarded by the following order:UniversalForwarder -> HeavyForwarder -> Indexer , i think it would be right setting charset in indexer, but iam still getting that issue

0 Karma

sideview
SplunkTrust
SplunkTrust

I'd contact Splunk Support. I think some data input config somewhere needs to be configured to tell Splunk that the incoming data is UTF-16, otherwise it always assumes everything is UTF-8, which explains what you're seeing. Possibly Firefox is doing some overly clever detection and "fixing" the situation at the browser level, but there's still a fundamental problem in the middle layers.

0 Karma

sieutruc
Contributor

It's so strange. When i viewed search results in indexer on Firefox, the data seemed to be well displayed, but with Chrome it showed like "x001x002x00/x001x002x00/x0.....", but in 2 cases, i cannot use any report command to create table from them. The data's encoding is UTF-16LE.

0 Karma

sideview
SplunkTrust
SplunkTrust

Interesting that if you remove all the x00's, that sample ends with "[TRANSACTION]". I've seen something like this before. Is it possible this is a single-byte vs double-byte issue, or a Unicode/UTF8/UTF16 issue?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...