Getting Data In

Can you help me extract a timestamp from JSON?

francoisternois
Path Finder

Hi,

I'm having trouble extracting timestamps from JSON on a production environment: Timestamp field is not used by Splunk as timestamp (and Splunk used the _indextime as timestamp). I tried to configure props.conf (see below) on both indexers and heavy forwarder (data are pulled from a data store on Azure). I have to say that on a local instance, it works fine.
alt text

Here is my raw data :

{"odata.etag": "W/\"datetime'2019-01-09T08%3A29%3A09.2933828Z'\"", "PartitionKey": "201901090829", "levelno": "10", "hostname": "5c6744d1d32c", "levelname": "DEBUG", "version": "v1.2.1-741-g19422ce", "component": "Watcher", "Timestamp": "2019-01-09T08:29:09.2933828Z", "RowKey": "20190109082909284-5c6744d1d32c-22-00", "message": "Watcher running...", "pathname": "/app/ccc/watcher/perimeter.py", "lineno": "61"}

Different props.conf that I tried without any success :

[sourcetype]
CHARSET = AUTO
KV_MODE = JSON
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIME_PREFIX = Timestamp": "
category = Structured
disabled = false
pulldown_type = true

[sourcetype]
CHARSET = AUTO
INDEXED_EXTRACTIONS = json
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = Timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIME_PREFIX = Timestamp": "
category = Structured
disabled = false
pulldown_type = true

Any help is very welcome 🙂

Francois

0 Karma
1 Solution

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

If you are receiving data from Azure on Heavy Forwarder then try below props.conf on Heavy Forwarder.

[yoursourcetype]
INDEXED_EXTRACTIONS=JSON
TIMESTAMP_FIELDS=Timestamp

View solution in original post

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

If you are receiving data from Azure on Heavy Forwarder then try below props.conf on Heavy Forwarder.

[yoursourcetype]
INDEXED_EXTRACTIONS=JSON
TIMESTAMP_FIELDS=Timestamp

francoisternois
Path Finder

alt text
Thank again for your quick answer. Here is an example of what I'm facing. And I've no props.conf for this sourcetype. Maybe I should configure one indicating that it's cooked data ?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

On search head can you please configure below props.conf

[yoursourcetype]
KV_MODE = none
0 Karma

francoisternois
Path Finder

That's perfect ! Thank you !

0 Karma

francoisternois
Path Finder

Thank you ! It works very well ! But...
Now I have multivalue fields 😕
I guess both extracted on HF and indexer. Any ideas ?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Only first full instance of splunk is parsing the data so in your case only HF is parsing data. Can you please post your multivalue field sample data and what problem are you facing with it ? Also I'll suggest to remove config from Indexer if HF is parsing data.

EDIT: It maybe due to config on Indexer, so first remove config from Indexer and then check.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...