suggestions for HAProxy dashboard ?

[brianpreston]

We need some kind of dashboard, with green/red indicators, showing issues our HAproxy knows about.

I'd love to implement this on Splunk!

Does this already exist somewhere?
If not, does anyone have suggestions on dashboards to crib? Or other tricks?

Thanks!

[Richfez]

This sounds like a great use for Splunk, but I have to say it appears like it might be a bit of work to get it set up so that you can make the pretty parts.

I didn't see anything in apps that would parse your log files or do many interesting things with this particular set of logs out of the box. I can't tell your experience level, but this could be a bit of a challenge if you are a bit new to Splunk yet. Still, we can help! For more information specifically on any of the below pieces, I would (if I were you) create new questions as required so that each question is relatively small and focused - it gets way better responses that way!

I also can't tell from your question if you are already ingesting the logs for this or not. If you are, you can safely just skim through the first bits here.

First, you have to get data from the HAProxy into Splunk. HAproxy (hereafter I'm gonna do it in lowercase 'cause I'm lazy) appears to want to use syslog. This is perfect - you can have it syslog to the haproxy box as local files (Google can help you with this), then use the universal forwarder to send the files that it is creating in to Splunk. You can also install a syslog server elsewhere, or even install syslog-ng or rsyslog on your Splunk box and let it listen for the traffic and write it to a local file, then pick that file up from there. Google can help you again with that. I only recommend that you do NOT use Splunk to listen for syslog directly on port 514. It can work for low volume test stuff, but really, the small effort required to set up syslog-ng will pay you back tenfold in ease of troubleshooting and use! Some information here on setting up the UF and you can find other places that will help or give examples for the monitor stanzas you will need. It's highly likely there's a clicky-GUI way to get those in, too. 🙂

Now that you have data coming in, you'll want to have Splunk make sense of that data. The idea would be to let you use fields like srv_conn and status_code to search on and write your dashboards against. Unluckily, as I mentioned above, there doesn't appear to be anything pre-written to "interprete" those logs for Splunk, so that means work on your part. Luckily for you, it appears haproxy might actually have their logging format specified reasonably well. Unfortunately, you may need some help writing REGEXes for that stuff. This is a bit of a topic all on its own, but we can totally help. If you get stuck at on this part, get what you can done then create a new question with perhaps a link to that document, an actual sample of a couple of lines of the log file you are receiving and kindly ask for some help. Someone should be able to help or at least give you a good start on it. Remember, smaller more focused questions get the best responses, but I suspect sorting out the entire log line in one question should be totally fine.

A secondary step to that would be trying to make the extractions and whatnot be CIM compliant. If they are done that way then you could probably use the App for Web Proxies. This will be somewhat more work than just ripping fields out, but the payoff is even greater. Still, this might not be a beginner topic.

Now comes the fun part! Once you have data coming in and looking all fancy-pants and nice with fields and things, you can just ... click around and do stuff. You can start at the Splunk Apps page, search for web or perhaps "proxy" (which should give you some examples of things to do).

There are several things you can really do with proxies and I think to some extent your precise usage may tell you better what you can do with the data. At this early stage and without the information of exactly what's on which end of the proxy, I can only say have some fun with it and as you play with it ideas will probably come - then ask if you get stuck implementing one!

So, I apologize if you just needed that last paragraph (or if that's the paragraph you wanted but I didn't put enough in it!), and doubly-apologize for the length of this. If it gets you started, great!

[Lowell]

Great answer! I'd also be sure to check out Splunk Add-on for HAProxy. I haven't used the app but did a quick code review and it looks very solid.