Getting Data In

JSON Parsing error

ellothere
Explorer

Setup Splunk monitoring to watch a directory. Files started coming in but with the timestamp not being parsed correctly. I adjusted by Settings > Data > Source Type then I cloned the default json and clicked Advanced and set the timestamp to this `%d-%m-%Y%H:%M:%S` for the field systemTime. (I even tried adding surrounding quotes at one point)

Example dataset:
[{
"systemTime" : "22-01-2019_15:05:01",
"fieldType" : "XXX-XXX",
"fieldLocation" : "XXX1",
"fieldCommand" : "XXXXXX",
"kernalName" : "Linux",
"nodeName" : "x86_64",
"kernalRelease" : "4.15.0-43-generic",
"kernalVersion" : "#46~16.04.1-Ubuntu SMP Fri Dec 7 13:31:08 UTC 2018",
"machine" : "x86_64",
"processor" : "x86_64",
"hardwarePlatform" : "x86_64",
"operatingSystem" : "GNU/Linux",
"timeup" : " 15:05:01 up 8 days, 4:48, 2 users, load average: 0.35, 0.40, 0.31",
"soft1Version" : "XXXXX",
"soft2Version" : "XXXXXXXX"
}]

I noticed the files stopped coming in so I checked index=_internal source=*/splunkd.log OR source=*\\splunkd.log | search *system* log_level=ERROR and found errors like ERROR JsonLineBreaker - JSON StreamId:3524616290329204733 had parsing error:Unexpected character while looking for value: '\\'.

Despite the files not being ingested, when I go to Settings > Data Inputs > Files & Directories the file count for that directory continues to rise.
It seems to be that if I remove the timestamp part, the file does get correctly processed but _time becomes 1979...

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Please try with below configuration in props.conf for your new sourcetype.

props.conf

[yourSourcetype]
INDEXED_EXTRACTIONS=JSON
KV_MODE = none
TIMESTAMP_FIELDS=systemTime
TIME_FORMAT=%d-%m-%Y_%H:%M:%S
0 Karma

ellothere
Explorer

Where do I place the props.conf file? I tried making one in $SPLUNK_HOME/etc/system/local and it wants me to be root to create the file. Will this cause any permissions problems? Thank you!

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Does your splunk instance running as root ? If not then it should not prompt you to create file as root. You need to create file with same user as splunk is running.

You can create this props.conf in $SPLUNK_HOME/etc/system/local or if you have any custom app then $SPLUNK_HOME/etc/apps/<CUSTOM_APP>/local

0 Karma

ellothere
Explorer

None of the events are showing. I created props.conf in /opt/splunk/etc/system/local as root and saved it as system_json1. I made sure to restart Splunk after this. However, I do not see this new source type via the GUI. Even the old events that had the time incorrectly processed disappeared.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

How are you ingesting data into Splunk ? And configuration which you recently created will apply to new data only, it will not apply to data which is already ingested.

0 Karma

ellothere
Explorer

I am ingesting the data into Splunk by using Settings > Add Data > Monitor > Files & Directories. I can change the source type for that data by Settings > Data Inputs > Files & Directories. I clicked on the directory that is causing me problems and changed the source type to props.conf file and the previous data also disappeared with that.
To be clear, Splunk is still showing that the number of files for that directory increment. For whatever reason, the files are not being processed.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

So are you using same sourcetype for previous data and new data ? If you have test instance then I highly recommend to test this in Test Instance.

0 Karma

ellothere
Explorer

Yes, I am currently on the Test Instance. The data source was new and that is why I am just now addressing the incorrect time formatting. I do not know of a way to change the sourcetype using Monitoring without it affecting both the old and new data as it only allows you to specify one.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Try to remove KV_MODE = none from props.conf and then try again.

0 Karma

ellothere
Explorer

Still not happening.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...