Ok. I'm a little confused now. Initial answer above said if I move a forwarder from one deployment server to another that both need to have the apps named the same for it to work. Your answer indicates a distinct set of apps is required. ... View more

I was having a similar issue but I think I figured it out. So, my setup was a license server/DMC server. My last 30 days license reports weren't working. I did a bunch of digging and found your notes as well. It was working in a similar setup in non-prod. By chance I looked at my distsearch.conf and noticed that my DMC/license server was categorized as an indexer (it was not setup to forward the data to the other indexers) while in prod that was not the case. I went into the DMC roles in prod and made my DMC/License server also have the indexer role and my reports started working again. ... View more

I totally forgot about being able to set permissions there. That worked exactly as I wanted. Can you please resubmit as an Answer so I can accept it and give you credit? ... View more

Thanks for the clarification. I don't think the timezone data is in the event (as in, it isn't specified in the text of the log file line). Forwarders are running 6.0+ and do not have the outputs.conf config you mentioned. Also, I don't believe any of the props.conf on the indexers have the configs you mention. When does the timezone get obtained on the forwarder? At startup? If the timezone on the machine changes while the forwarder is running, does it know that? Here's some info from my virtual desktop administrator (we are running these forwarders on virtual desktops) about how these machines behave: There were approx. 11 desktops that were logging the wrong time. I do wonder if the splunk forwarder is getting confused when the view agent changes the timezone with the logon/logoff activity. The view agent on the virtual desktop will change the timezone to match what's on the physical desktop. The timezone is changed back to be in sync with the ESXi host when no one is logged on. ... View more

That would definitely work. I was thinking of a way to do it in Perl and I would probably create an output like you just described. ... View more

Thanks! This is pretty close to what I am looking to do. This creates by user, while my actual goal (which I may not have explained well enough) is to compare groups to see how similar they are. I actually have a list of hundreds of groups. I flipped the group and user fields in the search which causes the match not to work anymore. I'm assuming that is because the eventstats count(eval(group_count==2)) syntax is assuming 2 groups. Any ideas how I might do something similar to what you've created where it's a bunch of groups? ... View more

Thanks for the update. I stumbled into the same copy/paste issue you mentioned but later realized it. That fix appeared to approve my uptime but recycles still were required, just less frequently. Since I updated this, I got more information on my case I originally opened and my sales engineer mentioned that there is definitely a bug opened for this specific issue that will be addressed in a future release. My resolution was to migrate my indexers to Linux to restore stability to my environment. ... View more

What OS are you running on? We are having a similar problem on Windows virtual servers running Splunk following our upgrade to 6.3. 6.3.1 didn't seem to help either. Recycling Splunk solves the issue temporarily but it comes back. We first had the issue on our Windows search head, once we rolled back to a 6.2 release the issue went away. Now we are also seeing the issue our on indexers that are 6.3 but rolling back isn't really an option there. Our Linux search head hasn't seen the issue. The system appears to be running into it's own self-imposed max HTTP threads limit. When the server is in this condition it returns 500 errors and eventually fails health checks. However, logging onto the server shows that actual CPU, memory and other system vitals barely being used at all. My first thought was that by overriding maxThreads in server.conf we could escape the issue but that doesn't solve it either. I'm thinking maybe something in 6.3 is either using more of these threads or not cleaning them up as fast. The problem is I don't see any way to measure it other than the messages that come up when you've hit the limit. ... View more

What OS are you running on? I have seem similar issues and log messages on my Windows search head after upgrading to 6.3 and 6.3.1. ... View more

Did this start happening after a recent 6.3 upgrade? What platform are you running? I've seen this message recently too following my 6.3 and some new app installs. ... View more

Seeing this too. Started after I updated to 6.3. I run a distributed but not clustered environment. I'm running on Windows. ... View more

Not sure how to give maciep the credit but that hit the nail on the head. I am using DMC and by going in to the Setup screen and hitting Apply, my search is now able to correctly pull results from all indexers. ... View more

Yes. I configured the indexers as Search Peers using Splunk web. ... View more