Hello, My architecture is the following: cluster of indexers (3 indexers), 1 Master , 1 Search Head. I removed by mistake the SErach head instance form the cluster and I re add it and now I am receiving this error: Failed to start KV Store process. See mongod.log and splunkd.log for details. KV Store changed status to failed. KVStore process terminated. KV Store process terminated abnormally (exit code 6, status PID 3339006 killed by signal 6: Aborted). See mongod.log and splunkd.log for details.   mongod.log : 2023-04-27T14:41:33.317Z I CONTROL [initandlisten] MongoDB starting : pid=3339006 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=sv-na-splhead 2023-04-27T14:41:33.317Z I CONTROL [initandlisten] db version v3.6.17-linux-splunk-v4 2023-04-27T14:41:33.317Z I CONTROL [initandlisten] git version: 226949cc252af265483afbf859b446590b09b098 2023-04-27T14:41:33.317Z I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2zf-fips 21 Jun 2022 2023-04-27T14:41:33.317Z I CONTROL [initandlisten] allocator: tcmalloc 2023-04-27T14:41:33.317Z I CONTROL [initandlisten] modules: none 2023-04-27T14:41:33.317Z I CONTROL [initandlisten] build environment: 2023-04-27T14:41:33.317Z I CONTROL [initandlisten] distarch: x86_64 2023-04-27T14:41:33.317Z I CONTROL [initandlisten] target_arch: x86_64 2023-04-27T14:41:33.317Z I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 8191, ssl: { PEMKeyFile: "/opt/splunk/etc/auth/server.pem", PEMKeyPassword: "<password>", allowInvalidHostnames: true, disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireSSL", sslCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." }, unixDomainSocket: { enabled: false } }, replication: { oplogSizeMB: 200, replSet: "4F365B7A-622E-4121-82B0-CF2933199661" }, security: { javascriptEnabled: false, keyFile: "/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0", oplogFetcherSteadyStateMaxFetcherRestarts: "0" }, storage: { dbPath: "/opt/splunk/var/lib/splunk/kvstore/mongo", engine: "mmapv1", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } } 2023-04-27T14:41:33.320Z I STORAGE [initandlisten] exception in initAndListen: Location28662: Cannot start server. Detected data files in /opt/splunk/var/lib/splunk/kvstore/mongo created by the 'wiredTiger' storage engine, but the specified storage engine was 'mmapv1'., terminating 2023-04-27T14:41:33.320Z F - [initandlisten] Invariant failure globalStorageEngine src/mongo/db/service_context_d.cpp 272 2023-04-27T14:41:33.320Z F - [initandlisten] ***aborting after invariant() failure 2023-04-27T14:41:33.331Z F - [initandlisten] Got signal: 6 (Aborted). 0x55fed413bde1 0x55fed413aff9 0x55fed413b4dd 0x7fb5cae3d140 0x7fb5cac8dce1 0x7fb5cac77537 0x55fed27f5330 0x55fed2aa2d18 0x55fed3fe07f1 0x55fed3fdc8d7 0x55fed286c93c 0x55fed4137195 0x55fed27f6513 0x55fed28762f0 0x55fed287431a 0x55fed27f7159 0x7fb5cac78d0a 0x55fed285b665 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"55FED1E7B000","o":"22C0DE1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55FED1E7B000","o":"22BFFF9"},{"b":"55FED1E7B000","o":"22C04DD"},{"b":"7FB5CAE2A000","o":"13140"},{"b":"7FB5CAC55000","o":"38CE1","s":"gsignal"},{"b":"7FB5CAC55000","o":"22537","s":"abort"},{"b":"55FED1E7B000","o":"97A330","s":"_ZN5mongo22invariantFailedWithMsgEPKcS1_S1_j"},{"b":"55FED1E7B000","o":"C27D18","s":"_ZN5mongo20ServiceContextMongoD9_newOpCtxEPNS_6ClientEj"},{"b":"55FED1E7B000","o":"21657F1","s":"_ZN5mongo14ServiceContext20makeOperationContextEPNS_6ClientE"},{"b":"55FED1E7B000","o":"21618D7","s":"_ZN5mongo6Client20makeOperationContextEv"},{"b":"55FED1E7B000","o":"9F193C"},{"b":"55FED1E7B000","o":"22BC195"},{"b":"55FED1E7B000","o":"97B513","s":"_ZN5mongo8shutdownENS_8ExitCodeERKNS_16ShutdownTaskArgsE"},{"b":"55FED1E7B000","o":"9FB2F0","s":"_ZN5mongo10StatusWithISt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_ESaIS9_EEEC1ERKSC_"},{"b":"55FED1E7B000","o":"9F931A","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"55FED1E7B000","o":"97C159","s":"main"},{"b":"7FB5CAC55000","o":"23D0A","s":"__libc_start_main"},{"b":"55FED1E7B000","o":"9E0665"}],"processInfo":{ "mongodbVersion" : "3.6.17-linux-splunk-v4", "gitVersion" : "226949cc252af265483afbf859b446590b09b098", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "5.10.0-18-amd64", "version" : "#1 SMP Debian 5.10.140-1 (2022-09-02)", "machine" : "x86_64" }, "somap" : [ { "b" : "55FED1E7B000", "elfType" : 3 }, { "b" : "7FFF14ABA000", "path" : "linux-vdso.so.1", "elfType" : 3 }, { "b" : "7FB5CB31C000", "path" : "/opt/splunk/lib/libdlwrapper.so", "elfType" : 3 }, { "b" : "7FB5CB2FE000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3 }, { "b" : "7FB5CB019000", "path" : "/opt/splunk/lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "7FB5CAFA2000", "path" : "/opt/splunk/lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "7FB5CAF9C000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FB5CAF92000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FB5CAE4C000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FB5CAE2A000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FB5CAC55000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FB5CB321000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 }, { "b" : "7FB5CAC38000", "path" : "/opt/splunk/lib/libz.so.1", "elfType" : 3 } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55fed413bde1] mongod(+0x22BFFF9) [0x55fed413aff9] mongod(+0x22C04DD) [0x55fed413b4dd] libpthread.so.0(+0x13140) [0x7fb5cae3d140] libc.so.6(gsignal+0x141) [0x7fb5cac8dce1] libc.so.6(abort+0x123) [0x7fb5cac77537] mongod(_ZN5mongo22invariantFailedWithMsgEPKcS1_S1_j+0x0) [0x55fed27f5330] mongod(_ZN5mongo20ServiceContextMongoD9_newOpCtxEPNS_6ClientEj+0x158) [0x55fed2aa2d18] mongod(_ZN5mongo14ServiceContext20makeOperationContextEPNS_6ClientE+0x41) [0x55fed3fe07f1] mongod(_ZN5mongo6Client20makeOperationContextEv+0x27) [0x55fed3fdc8d7] mongod(+0x9F193C) [0x55fed286c93c] mongod(+0x22BC195) [0x55fed4137195] mongod(_ZN5mongo8shutdownENS_8ExitCodeERKNS_16ShutdownTaskArgsE+0x36E) [0x55fed27f6513] mongod(_ZN5mongo10StatusWithISt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_ESaIS9_EEEC1ERKSC_+0x0) [0x55fed28762f0] mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x87A) [0x55fed287431a] mongod(main+0x9) [0x55fed27f7159] libc.so.6(__libc_start_main+0xEA) [0x7fb5cac78d0a] mongod(+0x9E0665) [0x55fed285b665] ----- END BACKTRACE ----   Can I have any advise on how to resolve it ?   Thank you!       ... View more

Hello, I have a question regarding the TA-Exchnage-Mailbox in splunk app for microsoft exchange. I am using this app on my deployment server to parse the exchange logs but the logs are not parsed on the search head. I copied the default conf files to the local one and I made the changes to receive the logs but they are not parsed especially for the message tracking ones. Any idea on how to configure it?   Thank you in advance! ... View more

Hello , I have  sign&go as SSo tool and I have integrated it with splunk. I configured the chain of certificates "/opt/splunk/etc/auth/idpCerts/idpCertChain_1" in the order needed, but I a, still recieving the sam error . 04-03-2023 16:01:21.658 +0200 ERROR UiSAML [2402186 webui] - Verification of SAML assertion using the IDP's certificate provided failed. Error: failed to verify signature with cert 04-03-2023 16:01:21.658 +0200 ERROR Saml [2402186 webui] - Unable to verify Saml document 04-03-2023 16:01:21.658 +0200 ERROR Saml [2402186 webui] - Error: failed to verify signature with cert :/opt/splunk/etc/auth/idpCerts/idpCertChain_1; How to solve this error? Thank you.   ... View more

Hello, I have an issue regarding the creation of a new index. I want to create a new index to receive logs form NPS servers.  First, I created a new app for NPS in the deployment-apps in the master server but I created the new folder for the app and edited everytihng with a root user. I assigned the app to a new server class in the master server. I installed the UF on NPS servers and they are successfully connected to the deployment server. I added the, the new server class. Now I am getting the error that index= radius (defined in the new app created) dos not exist. So, I went to the master-apps in the master server , the previous defined indexes are present in the  directory  X other than _cluster, I added the index radius in the local file of the Directory X . I validated  the conf and push and all the indexes in the indexer cluster have the same config but the index radius is not present. I did all the modification using the root user. Can anyone advise  me to know the issue ? Thank you.     ... View more

Hello,  I need a search query to detect http outboun irect traffic. Thank  you. ... View more