Getting Data In

Splunk not logging the large json events properly

hvaithia
Path Finder

[my sourcetype]
INDEXED_EXTRACTIONS = json
TIME_FORMAT = %s%3N
TIME_PREFIX = jenkinsTimestamp
SHOULD_LINEMERGE = false
description = JavaScript Object Notation
category = Structured
disabled = false
pulldown_type = true
TRUNCATE=200000
MAX_EVENTS=200000

some of my events are 110,000+ characters long. I tried TRUNCATE=0 and MAX_EVENTS=0 to no effect. some of my events log from the middle and others do not log in the end. Some are logged just fine. Examples below,
----not logging in the start event----
5/22/15
2:08:39.000 AM

ckground-color: #F0F8FF;} </style><h4>Global Patterns</h4><table><tr ><td><b>Previous Job:</b></td><td><b>Started by <a href='/yhudson/job/searchproducts_gsm_upload_assembly/481/'>searchproducts_gsm_upload_assembly #481</a></b></td></tr><tr class=\"aliceblue\"><td><b>Previous Job:</b></td><td><b>Started by <a href='/yhudson/job/searchproducts_gsm_fr_map_component/499/'>searchproducts_gsm_fr_map_component #499</a></b></td></tr><tr ><td><b>Build host:</b></td><td><b>Built on <a href='/yhudson/computer/rhel6-gq1-slave-03.adcd.infra/'>rhel6-gq1-slave-03.adcd.infra</a></b></td></tr></table>"}], "id": "2015-05-22_00-34-30", "keepLog": false, "url": "http://thefactory.corp.xxx.com:9999/jenkins/view/Search_GSM/job/searchproducts_gsm_upload_deploy_int...", "culprits": [{"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/by-tortuga", "fullName": "by-tortuga"}], "result": "SUCCESS", "executor": null, "duration": 98065, "fullDisplayName": "http://thefactory.corp.xxx.com:9999/jenkins/view/Search_GSM/job/searchproducts_gsm_upload_deploy_int", "estimatedDuration": 180575}

---------not logging in the end------
5/22/15
2:08:03.000 AM

{"building": true, "changeSet": {"items": [{"comment": "[GSM-1393] Support 'style' attribute in the GSM CardModule structure.\n", "paths": [{"editType": "edit", "file": "components/common/src/test/java/com/xxx/search/middleware/cosmos/template/CardModuleTest.java"}, {"editType": "edit", "file": "components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java"}], "author": {"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/whsu", "fullName": "whsu"}, "timestamp": 1432159954000, "id": "86b19401707141f58cd3d561158fee38f69f648e", "commitId": "86b19401707141f58cd3d561158fee38f69f648e", "msg": "[GSM-1393] Support 'style' attribute in the GSM CardModule structure.", "date": "2015-05-20T22:12:34+0000 -0700", "affectedPaths": ["components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java", "components/common/src/test/java/com/xxx/search/middleware/cosmos/template/CardModuleTest.java"]}, {"comment": "[GSM-1393] Support 'url' attribute in the GSM CardModule structure.\n", "paths": [{"editType": "edit", "file": "components/common/src/test/java/com/xxx/search/middleware/cosmos/template/CardModuleTest.java"}, {"editType": "edit", "file": "components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java"}], "author": {"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/whsu", "fullName": "whsu"}, "timestamp": 1432181780000, "id": "828dd4633fd5ecc565b1395c59586889fb2f57c9", "commitId": "828dd4633fd5ecc565b1395c59586889fb2f57c9", "msg": "[GSM-1393] Support 'url' attribute in the GSM CardModule structure.", "date": "2015-05-21T04:16:20+0000 -0700", "affectedPaths": ["components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java", "components/common/src/test/java/com/xxx/search/middleware/cosmos/template/CardModuleTest.java"]}, {"comment": "[GSM-1393] Fix url variable in CardModule.java.\n", "paths": [{"editType": "edit", "file": "components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java"}], "author": {"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/whsu", "fullName": "whsu"}, "timestamp": 1432182356000, "id": "a0d1e6e7d2b65d5020cc2292d2bcc2699b8c5470", "commitId": "a0d1e6

Any help is greatly appreciated

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Im not sure indexed extractions are what you want here... Try adding the following and test this..

SHOULD_LINEMERGE = true
KV_MODE = json
BREAK_ONLY_BEFORE = ^\d\d\-\d\d\-\d\d
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 25

This is assuming your events starts with the date and time

0 Karma

hvaithia
Path Finder

Hi @esix_splunk, the timestamp is not the start of the event. My event is typically starts with a """ {"building": """ like this,

{"building": false, "changeSet": {"items": [{"comment": "Revert \"set pos field can be overrided by logGroup dynamically \"\n", "paths": [{"editType": "edit", "file": "src/datamodel/ComponentData.php" }], "author": {"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/prajakta", "fullName": "prajakta"}, "timestamp": 1432177477000, "id": "5f428d3cde806ec296538023dae94ae694bca747", "com mitId": "5f428d3cde806ec296538023dae94ae694bca747", "msg": "Revert \"set pos field can be overrided by logGroup dynamically \"", "date": "2015-05-21T03:04:37+0000 -0700", "affectedPaths": ["src/datamo del/ComponentData.php"]}], "kind": "git"}, "builtOn": "rhel6-gq1-factory-slave-029.infra", "description": null, ...


And jenkinsTimestamp is the timestamp I want splunk to pick up. and hence the
TIME_FORMAT = %s%3N
TIME_PREFIX = jenkinsTimestamp
I tried KV_MODE=json earlier and somehow that doesn't seem to work for me. I read that KV_MODE is only for search-time extractions and not index-time. Do you have any suggestions for this?

0 Karma

hvaithia
Path Finder

Another weird thing that I am noticing is that, it works fine when I analyze the logs when uploading it and using the same sourcetype, but it appears to not work after I upload it and start the search. This is not the 1st time I am seeing this, it happened to me once before. Is the log analyzer in splunk 6.2 not working properly?

0 Karma

hvaithia
Path Finder

I just tested the same sourcetype with splunk 6.1 and it works perfectly! Safe to assume that something is wrong with Splunk 6.2 indexing

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...