I've checked a number of threads about breaking JSON files and I've tried a number of offered solutions and none seem to work. I'm running 8.1.0 and I don't remember seeing this as much of an issue in previous versions. The snort (ids-u2json) JSON is lint-valid as follows:   {"type": "event", "event": {"msg": "ET INFO Microsoft Connection Test", "classification": "Potentially Bad Traffic", "sensor-id": 0, "event-id": 581, "event-second": 1607588446, "event-microsecond": 790456, "signature-id": 2031071, "generator-id": 1, "signature-revision": 2, "classification-id": 3, "priority": 2, "sport-itype": 63591, "dport-icode": 80, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "192.168.1.125", "destination-ip": "13.107.4.52"}} {"type": "event", "event": {"msg": "ET POLICY PE EXE or DLL Windows file download HTTP", "classification": "Potential Corporate Privacy Violation", "sensor-id": 0, "event-id": 582, "event-second": 1607588467, "event-microsecond": 769440, "signature-id": 2018959, "generator-id": 1, "signature-revision": 4, "classification-id": 33, "priority": 1, "sport-itype": 80, "dport-icode": 63676, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "205.185.216.10", "destination-ip": "192.168.1.125"}} {"type": "event", "event": {"msg": "ET INFO Packed Executable Download", "classification": "Misc activity", "sensor-id": 0, "event-id": 583, "event-second": 1607588467, "event-microsecond": 769340, "signature-id": 2014819, "generator-id": 1, "signature-revision": 1, "classification-id": 29, "priority": 3, "sport-itype": 80, "dport-icode": 63676, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "205.185.216.10", "destination-ip": "192.168.1.125"}}   props.conf on the UF is as follows:   [sourcetype=json] KV_MODE=json AUTO_KV_JSON=true NO_BINARY_CHECK = true disabled = false SHOULD_LINEMERGE = false TIME_FORMAT = "event-second": %s, "event-microsecond": %6N LINE_BREAKER = }}(^s)    and props.conf on the indexer/search head as follows:   [stanza] TZ = UTC SHOULD_LINEMERGE = false [_json] DATETIME_CONFIG = LINE_BREAKER = }} NO_BINARY_CHECK = true disabled = false KV_MODE = json [json_no_timestamp] NO_BINARY_CHECK = true SHOULD_LINEMERGE = false disabled = false   According to what I've told the UF to do in props.conf, the JSON events should be splitting up the JSON events using the double braces LINE_BREAKER }} as follows:   {"type": "event", "event": {"msg": "ET INFO Microsoft Connection Test", "classification": "Potentially Bad Traffic", "sensor-id": 0, "event-id": 581, "event-second": 1607588446, "event-microsecond": 790456, "signature-id": 2031071, "generator-id": 1, "signature-revision": 2, "classification-id": 3, "priority": 2, "sport-itype": 63591, "dport-icode": 80, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "192.168.1.125", "destination-ip": "13.107.4.52"}} {"type": "event", "event": {"msg": "ET POLICY PE EXE or DLL Windows file download HTTP", "classification": "Potential Corporate Privacy Violation", "sensor-id": 0, "event-id": 582, "event-second": 1607588467, "event-microsecond": 769440, "signature-id": 2018959, "generator-id": 1, "signature-revision": 4, "classification-id": 33, "priority": 1, "sport-itype": 80, "dport-icode": 63676, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "205.185.216.10", "destination-ip": "192.168.1.125"}} {"type": "event", "event": {"msg": "ET INFO Packed Executable Download", "classification": "Misc activity", "sensor-id": 0, "event-id": 583, "event-second": 1607588467, "event-microsecond": 769340, "signature-id": 2014819, "generator-id": 1, "signature-revision": 1, "classification-id": 29, "priority": 3, "sport-itype": 80, "dport-icode": 63676, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "205.185.216.10", "destination-ip": "192.168.1.125"}}   but it doesn't. Instead, the UF clumps them together as a single event and only reports on the first JSON stanza. Nothing I've tried for LINE_BREAKER seems to work - the UF seems to ignore it. Many thanks ... View more

On a Raspberry Pi 3 armv7l GNU/Linux, INDEXED_EXTRACTIONS=JSON in the props.conf file results in unrecoverable JSON StreamId processing errors: 05-06-2020 17:52:07.836 +0100 ERROR JsonLineBreaker - JSON StreamId:8017092045127549753 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/metrics.log", data_host="rpi3", data_sourcetype="json" 05-06-2020 17:52:07.836 +0100 ERROR JsonLineBreaker - JSON StreamId:8017092045127549753 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/metrics.log", data_host="rpi3", data_sourcetype="json" 05-06-2020 17:52:07.836 +0100 ERROR JsonLineBreaker - JSON StreamId:8017092045127549753 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/metrics.log", data_host="rpi3", data_sourcetype="json" with the log expanding so quickly, it fills up the /opt/splunkforwarder/var/log/splunk/splunkd.log to maximum logrotate capacity. Steps to duplicate bug: Install splunkforwarder-8.0.3-a6754d8441bf-Linux-arm.tgz onto a Raspberry Pi 3. Edit the /opt/splunkforwarder/etc/system/local/props.conf and add the following code: [default] SHOULD_LINEMERGE = false KV_MODE = none INDEXED_EXTRACTIONS=JSON NO_BINARY_CHECK = true TRUNCATE = 0 Add a local JSON file to the splunk file monitor with $SPLUNKHOME/bin/splunk add monitor /var/log/myvalidjsonfile.json -sourcetype json -host myhost -index myindex Restart splunk. Check the file tail -f $SPLUNKHOME/var/log/splunk/splunkd.log Watch it scroll away off the screen! The errors above are reported for both metrics.log and the splunkd.log itself(!) Stop splunk. Edit props.conf again and remove the line INDEXED_EXTRACTIONS=JSON . Restart splunk. Your splunkd.log is back to normal again. ... View more

I've gone through the answers here and tried the following: Unlocking stale PIDs clean locks chown -R <user>:<group> /opt/splunk But nothing seems to work. The last message: Please login as an administrator and correct issue. When I'm root can only mean that something is hardcoded that really shouldn't be. Here's the complete output (which is the same if I run it under the splunk user, or the user set in the /etc/init.d/splunk script which is irrelevant here I think) : root@ubuntu:/opt/splunk/bin# ./splunk start Splunk> All batbelt. No tights. Checking prerequisites... Checking http port [8000]: open Checking mgmt port [8089]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _blocksignature _internal _introspection _thefishbucket history main snort_test summary test Done Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied Error opening username mapping file: /opt/splunk/etc/users/users.ini Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied New certs have been generated in '/opt/splunk/etc/auth'. ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied Your active group is invalid. Please login as an administrator and correct issue. ERROR IniFile - Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied ERROR UsernameMapper - Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied ERROR IniFile - Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied ERROR UsernameMapper - Error opening username mapping file: /opt/splunk/etc/users/users.ini ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/splunk_app_for_nix/metadata/local.meta: Permission denied ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied ERROR IniFile - Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied ERROR UsernameMapper - Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied ERROR IniFile - Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied ERROR UsernameMapper - Error opening username mapping file: /opt/splunk/etc/users/users.ini ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/splunk_app_for_nix/metadata/local.meta: Permission denied ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied The SPLUNK_DB environment variable was defined but the test file ("/opt/splunk/var/lib/splunk/test.kMgOmj") could not be created by the current user: Permission denied Locking test failed on filesystem in path /opt/splunk/var/lib/splunk with code '7'. Please file a case online at http://www.splunk.com/page/submit_issue Checking filesystem compatibility... root@ubuntu:/opt/splunk/bin# Any ideas anyone? Thanks and regards ... View more