Monitoring Splunk

Why do some local.meta files have \x00 errors in splunkd.log on my heavy forwarder?

lycollicott
Motivator

I find these messages in splunkd.log:

02-15-2017 13:34:04.437 -0500 WARN  IniFile - C:\Splunk\var\run\searchpeers\my_dmc_server-1487183641\apps\fire_brigade\metadata\local.meta, line 4: Cannot parse into key-value pair: \x00

\x00 is a HEX 0, so why is it in my search bundle's local.meta?

NOTE: There is not \x00 in the app's original local.meta file on the DMC.

1 Solution

jcrabb_splunk
Splunk Employee
Splunk Employee

I am not sure what the cause is. I had seen this on a few cases recently where we saw those types of errors and had speculated on what may be happening. As it wasn't the reason for the case, we did not deep dive and chalked it up to a fluke. When I saw your answers post it made me consider that it may be a larger problem. Going through the current cases, I am seeing a few with the same local.meta errors. As I have data in hand, I am going to file a bug with our dev team to see if they understand what is happening. As soon as I get more information, I will share that with you.

## EDIT ##

This is fixed in:

  • 6.6.11 (SPL-136970)
  • 7.0.6 (SPL-158503)
  • 7.1.3 (SPL-158504)
Jacob
Sr. Technical Support Engineer

View solution in original post

timpacl
Path Finder

I have this issue across thousands of local.meta files in v8.0.3. Was the issue reintroduced.

0 Karma

jcrabb_splunk
Splunk Employee
Splunk Employee

Just a quick follow up. We suspect that it is caused by the tar format change introduced in a previous version. We are not aware of any problems this is causing outside of the WARN messages. I'll follow up again once more information has been provided.

Jacob
Sr. Technical Support Engineer

jcrabb_splunk
Splunk Employee
Splunk Employee

I am not sure what the cause is. I had seen this on a few cases recently where we saw those types of errors and had speculated on what may be happening. As it wasn't the reason for the case, we did not deep dive and chalked it up to a fluke. When I saw your answers post it made me consider that it may be a larger problem. Going through the current cases, I am seeing a few with the same local.meta errors. As I have data in hand, I am going to file a bug with our dev team to see if they understand what is happening. As soon as I get more information, I will share that with you.

## EDIT ##

This is fixed in:

  • 6.6.11 (SPL-136970)
  • 7.0.6 (SPL-158503)
  • 7.1.3 (SPL-158504)
Jacob
Sr. Technical Support Engineer

goelli
Communicator

Is there any news on this?
I have the same errors since updating from 6.4.4 to 6.5.3

0 Karma

moesaidi
Path Finder

Similar issue in 7.0.0:
08-16-2018 10:35:01.688 -0300 WARN IniFile - C:\PATH\searchpeers\NAME-ID\system\metadata\local.meta, line 22: Cannot parse into key-value pair:

(it shows "NULL")
Is this the same bug? similar bug? new bug?

0 Karma

jcrabb_splunk
Splunk Employee
Splunk Employee

Appears to be the same. For 7.0.x it is fixed in 7.0.6 (SPL-158503).

Jacob
Sr. Technical Support Engineer
0 Karma

splunkitsicherh
New Member

Don´t know exactly how it was fixed but we run SPLUNK 7.1.1 and there´s the Key-value ERROR in a lot of local.meta files. splunk enterprise runs on linux system(s)

OK I find the SPL-136970 and SPL-158504 in latest-version (7.1.3) on Website. so perhaps with this version the bug is fixed.

Is there a way to only get this bugfix?

0 Karma

jcrabb_splunk
Splunk Employee
Splunk Employee

We do not offer single issue patches. You will want to upgrade when its convenient.

Jacob
Sr. Technical Support Engineer
0 Karma

jcrabb_splunk
Splunk Employee
Splunk Employee

Good news, this finally been tracked down and is going through Q/A. We expect the bug to be addressed in 6.6.10 and I would assume it will be ported to the other, newer versions.

Jacob
Sr. Technical Support Engineer
0 Karma

jralston
Explorer

I am also seeing similar errors. I assume it is also causing of my bundle replication issues. Hope for an update on this.

0 Karma

jcrabb_splunk
Splunk Employee
Splunk Employee

It does not cause any problems other than this WARN message.

Jacob
Sr. Technical Support Engineer
0 Karma

jcrabb_splunk
Splunk Employee
Splunk Employee

No new info just yet. There is a bug and it's being investigated. If you want to suppress the message, edit $SPLUNK_HOME/etc/log-cmdline.cfg, locate the following line:

category.IniFile=INFO

and change it to ERROR:

category.IniFile=ERROR
Jacob
Sr. Technical Support Engineer
0 Karma

martinho
Explorer

Why have you recommended using log-cmdline.cfg instead of log-local.cfg (persists over Splunk upgrade) or log.cfg? I'm just curious - I have used log-local.cfg and it seems to work.

0 Karma

goelli
Communicator

Thanks. Can you keep us informed about the bug status here or should I open a case?

You workaround is a good idea, but it's not working for us. If I switch log level to ERROR, I will not receive correct warnings anymore.
We need the warnings to see, if someone forgets the line-breaking marker "\" if a search or any other value is using multiple lines. In worst case we have a saved search which is missing some lines, because of missing "\", which might result in a security event not recognized...

0 Karma

ChrisBell04
Communicator

I went ahead and created a case (#507968 local.meta - Cannot parse into key-value pair) and was told engineering already aware.

SPL-136970 - default and local meta files getting corrupt or being altered in such a way as to cause warnings.
for those who want to track it.

0 Karma

jcrabb_splunk
Splunk Employee
Splunk Employee

Yes, I will keep you updated.

Jacob
Sr. Technical Support Engineer
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...