Quick update: with this approach I was able to bypass opsec_pull_cert issue; however, we failed to collect any logs and received following errors: ERROR: Session end reason: SIC ERROR 147 - SIC Error for lea: Authentication error This may be an issue with either entity_sic_name and/or password. Password worked before and we double checked it. We also checked with checkpoint admins and tried pretty much all possible combinations for various opsec_entity_sic_name entries within the opseclea_connection.conf file. So far no luck 😞 ... View more

Hi, We have been facing the exact same issue. Interestingly enough, we were able to replicate the issue by simply opening up a dashboard and separated the search head and indexer to figure out where the problem was. Search Head was crashing with the existing configuration. Short story: We found a savedsearch within a user's context (private) that was named as a single character "a". Once this saved search was renamed to something longer, the problem went away. $SPLUNK_HOME/etc/users/mary/search/local/savedsearches.conf [a] ... rename the search name to be something longer: [some_longer_name_a] ... For this we had to edit the file, you can not do this from the web interface. Long story: The problem occurred when one of the available dashboards opened (or tried to open the link). This also happened when we create a very simple dashboard with one simple search panel. We were not able to replicate it with concurrent searches so this very much seemed like an issue with web instance. Splunk crashed within the same place all the time and the issue was replicated easily. Here's a portion of the crash log: [build aaff59bb082c] 2016-01-29 21:18:31 Received fatal signal 11 (Segmentation fault). Cause: No memory mapped at address [0x0000000000000008]. Crashing thread: TcpChannelThread Registers: RIP: [0x0000000000DA1D78] _ZNK9Paginator3cmpEP10ConfigItemS1_m + 104 (splunkd) ... OLDMASK: [0x0000000000000000] OS: Linux Arch: x86-64 On a brand new search head, we added apps ( $SPLUNK_HOME/etc/apps ) and local config ( $SPLUNK_HOME/etc/system/local ) and user config ( $SPLUNK_HOME/etc/users ) one by one to figure out where the problem may be. It boiled down to one specific user configuration, say "mary" ( $SPLUNK_HOME/etc/users/mary ). So we one by one removed existing configuration for that user: dashboards, panels, and configuration files and tested the search head crash (opening up a dashboard). It turned out to be the savedsearches.conf file as mentioned in the short version of this story above. The other interesting finding is that when we logon as "mary" and open up this private dashboard, nothing bad happens, no crashes. Conclusion: There's a ticket opened up and we still do not have a fix for this issue yet, but we were able to find out that some users were not following the naming conventions 🙂 ... View more

I wanted to add another quick comment on the solution. The one I put up there did not properly sort the results. I wanted the results to be grouped by websites (top websites by count or another field value, eg. bandwidth). I tried something like the following (2 subsearches): mysearchstring [ mysearchstring | top limit=2 website | table website ] [search [ mysearchstring | top limit=2 website | table website ] | stats count by user | sort 2 -count| table user] | stats count by website,user But this also does not group properly and seems costly. In order to sort, I had to add a sort_field to each event and then use that. For this, I used eventstats to compute a new field and use the average of that when evaluating sort_field . mysearchstring [ mysearchstring | top limit=2 website | table website ] | eventstats count AS sortcount by website | stats count, avg(sortcount) AS sort_field by website,user | dedup 2 website | sort -sort_field | fields website, user ... View more

Thank you javiergn. This helped me find out the solution as the following: mysearchstring [ mysearchstring | top limit=2 website | table website ] | stats count by website,user | sort +website,-count | dedup 2 website first limit is for top websites and limiting the dedup is for top users per website. ... View more

Hi, Did you ever get a response regarding this problem? I'm facing the same problem for the data that I put in thaweddb folder. I was able to search just fine with 6.2.5 but I receive this error message for some of the old data. Also, I noticed that this issue occurred when I selected the earliest to be @y but when I put in 01/01/2015 (beginning of the year) it seems to work. So far, I'm facing inconsistencies and I suspect maybe this is due to background acceleration/summary indexing processes. However, I really do not have a clue at this point 🙂 ... View more

Did you get a solution for this? I'm facing the same error and can not figure out what's going on. thanks ... View more

Hello, did anyone got a solution for this? I'm facing the same issue. ... View more

Thanks for the tip. How do you setup a script that runs only once at startup? What would be the value for "interval" parameter for the script input? ... View more

Thank you. That works for finding the events. So, how do display more fields (e.g. field1, field2, field3 from search#1 and field4, field5 from search#2) to the results so that I can display them in a table (or chart)? I tried fields command but was not successful. ... View more