What OS is your indexer running?

[skoelpin]

We're in the process of buying another indexing server and my company is set on installing Windows OS on this server. I'm strongly opposed to this as Linux is much easier to use and more reliable. What OS are your indexer(s) running on? Any augments to using one or the other?

[woodcock]

PLEASE do yourself a favor and DO NOT use Windows for your Indexers. It is a HEAP of extra trouble that nobody needs to deal with. Use *NIX instead. Hardly anybody uses Windows for Indexers so it gets FAR less attention and much less "real-world" shakeout.

[lycollicott]

Amen to that!

[woodcock]

There are ALWAYS memory problems with Windows Indexers (just look at any older release and the number of Windows-only bugs with memory there are). These problems FREQUENTLY lead to crashing hard, sometimes the entire box. Your uptime will be SERIOUSLY compromised if you go with Windows.

[woodcock]

Is anybody laughing at my "heap" joke?

[lycollicott]

Yes, but we're not showy about it. LOL

[lycollicott]

The only real drawback to Windows for us is %$!& file permissions that seem to have a mind of their own. If you run as local system it is not so problematic, but I run it as a Managed Service Account (MSA) which should be fine - *famous last words - but due to reasons beyond comprehension of either myself or my Windows admin there are too many moments when BucketMover cannot move buckets due to permissions.

Linux would be chown, chmod and done.

[skoelpin]

I had a good laugh at your famous last words.. That's my biggest argument is the file permissions. I think I have a good shot at getting this on a Linux environment and could use all the ammo I could get

[Richfez]

Managed service accounts are fantastic. Sometimes.

We have had our difficulties with them as well...

[lycollicott]

One of the really weird things is that we have inheritance turned on and full control is applied to "this folder, subfolders and files", but when new buckets get created their rawdata subfolder has "this folder only" and no one can read it - not even Splunk (I have a PowerShell input to calculate and track disk usage...it runs as the bloody MSA, but cannot read rawdata).

Also, if we create a new index and if one of us ever needs to look inside its db folder for any reason then we have to do the "yes/ok" to grant ourselves access to continue. Sounds harmless, right? Well, the next time a bucket is created it gets created with ONLY rights for the team member who did the "yes/ok" thing. That results in the BucketMover errors when it is time to roll it from warm to cold, so we have to go in and manually fix permissions on that bucket. The growling on those days sounds like a pack of wolves.

[Richfez]

There's an answer here (I love the alternate naming conventions used!), or at least my comment. For most use cases Windows is just fine. The only real exception I am aware of is Splunk Enterprise Security which will run just fine but somewhat slower on Windows due to Python issues. I believe there have been improvements in that, and I'd love some official feedback if someone at Splunk would care to comment ... ? 🙂

I think far more important is to use the OS that you have better support for, with a nod to *nix if they're reasonably equivalent or if you are doing Enterprise Security. You say "Linux is much easier to use and more reliable". Leaving aside the actual accuracy of that, if you are the only person there who is comfortable with Linux, then it's NOT easier and more reliable for the company, only for you. If you are out on vacation, who fixes the Splunk problems? Yeah, a lot of basic OS admin'ing is pretty easy to do, but a lot isn't - on either platform - and having no one else who understands the OS can be a very big problem.

So, I'd start with why your company is leaning toward Windows. If it's because they know and understand Windows and have a team of folks who can beat it into submission when it's misbehaving, then perhaps they've got a point.

Frankly, we were Splunk on Windows for years without any problems. It was fast, worked well, etc... We switched to Linux for two reasons during a hardware update/switch a year or two ago: A) We had plans on implementing Splunk Enterprise Security and B) we now have 3 or 4 people (out of 25ish) that are comfortable enough with Linux that they could do OS and Splunk maintenance type tasks in there when I wasn't around, if necessary.

[skoelpin]

Great points @rich7177

I would say there are A LOT more Windows guys than Linux guys at my company, even though we are primarily a Linux environment. We have 1 of our indexers on a Windows machine and I've had 2 major issues with it so far. The first issue is that the server has to be rebooted approx every month for security patches and takes about 3-5 minutes to reboot. The second issue I have is upgrading Splunk on Windows.. We had a bug in our Windows 2012 R2 OS which took 8 hours for Splunk to upgrade and lead to massive outage time. The other thing I like about Linux is the file permissions.. We are building this indexer for Enterprise Security so I'm glad you told me about the performance issues now! Thanks for your input, I really appreciate it!

[Richfez]

So, if you are using the UF to get your data in from places that support a UF, they'll cache waiting for the indexer(s) to come back.

That goes double for using an outboard syslog-ng/rsyslog server with a UF on it to collect those logs you can't get a direct Device-with-UF-> indexer connection from. The UF (and syslog itself) will cache/buffer it until the indexer is available.

If you do an indexer cluster, you can get past this problem, too. They're not real hard, either...

Honestly, I reboot my Linux boxes about monthly too, and neither them nor Windows now takes more than 30 seconds plus BIOS time (though that latter can be considerable on some boxes). Why? Well, I'll ask, why not? I like knowing they WILL restart, but this way I'm more (and perhaps more easily) assured all the patches and things are applied and working properly.

In fact, Server 2012 reboots FAST. I think it's got the edge now over Ubuntu 16, though when VM they're both ready to log into again about as quickly as I can restart whichever utility/connection I'm doing. When they're not VM, nearly the entire wait is BIOS type stuff.

[Richfez]

BTW, what a great thread. Too often these discussions elsewhere end up being a shouting match. It's a testimony to the culture and community surrounding Splunk that we can have a pleasant discourse on the pros and cons of each OS without it resulting in a flamewar.

In this case, *nix probably wins unless there's compelling business reasons for doing Windows. In other circumstances it could go either way depending on a variety of factors.

[muebel · Accepted solution]

Hi skoelpin, This is largely a preference argument, there aren't any clear technical arguments other than that Splunk is built on nix type systems, and had a nix type mindset, but it really shouldn't matter. One point for Windows is that, if you have a need to input remote WMI calls, it has to come from a Windows system. Also, some Windows related apps require windows. But you can always setup a VM based Heavy-forwarder instance for that sort of thing.

I prefer linux for the Splunk Infrastructure.

[dwaddle]

[1] Use what you're most comfortable with. If you're an MSCE with 20 years experience running Windows, and no practical Linux experience except that Raspberry Pi you got for Christmas ... use Windows. If you / your team is equally capable on both Linux and Windows then your decision is harder.

[2] Understand that running Splunk on Windows puts you statistically in the minority. Some things work substantially worse there from a performance standpoint - like things that depend on lots of custom REST endpoints (like DB Connect). (This is something Splunk is trying to improve). Also, a mixed Windows/Linux environment with Windows as the deployment server may have issues with things like scripted inputs.

Your community of peers running Splunk on Windows is much smaller than your community of peers would be on Linux. As such, the overall community (the people on Answers, or in IRC or Slack) will be much less equipped to help you meaning you will have to rely more on support.

In my opinion, running your core Splunk infrastructure on Windows puts you at a substantial disadvantage. But, there are people who do it highly successfully.

[skoelpin]

Awesome idea about installing a heavy forwarder on a VM! I currently have a production indexer on a Windows machine, so what if I install Linux on this second indexer, this should be able to get around this issue of sending remote WMI calls right?

[Richfez]

I would think so.

Keep in mind that ES can share indexers with other data with only minor concessions to "taking extra care", but the ES search heads are really special critters that want their own playground.

Sounds like Linux is the way to go for you and is what I would recommend.

Also, if you can swing it, I HEARTILY recommend have Splunk Professional Services help with standing up ES.

[muebel]

yup, thats right

[Richfez]

Nice answer, muebel. I thought of the WMI point right after I clicked Save. 🙂

To expand: if your remaining infrastructure is Windows, and if you aren't rolling the UF out to everything you have, with a Windows Splunk server you can just suck Event Logs off random boxes with just a couple of clicks via WMI. So if you have minor issues on server X and just want to review its logs, it's 30 seconds to pulling that in via WMI for those one-off cases, vs. quite a bit more work if all your servers are nix. But again, this can be done with *one windows Splunk HF, too.