Having an OVZ hosted server for what seems like forever is great up to a point. But their containerised nature ended up biting me in the ass. Being stuck on an ancient kernel for over 2+years, was the last straw. Enough was enough. It was time to move to a new hosting company; one that was local, one that wouldn’t break the bank when the some global pandemic crashes my exchange rate. But I was hesitent for a long time with moving as I was a bit dismayed about leaving my old IP address. I’d worked so hard to build up a nice reputation surrounded by some bad actors. But I just could not handle having a v2.6.32 kernel any longer. It was time for a change, and time to redesign how I hosted my applications.
For the last decade I’ve been hosting my own mail, through a few domains and spend a large chunk of my time still investigating mail issues for $dayjob that I wanted something solid that could be fun enough to tinker with, but without having to worry about double dkim signing because I added the milter on the wrong ingestion point. Yes, that unfortunately actually happened for probably a year… or three. It’s fine though, all emails sent were still technically valid emails as proof by the double-dkim. :oops:
What got me fed up with my OVZ container was finding about about rspamd, via this youtube video back in 2019-05, and then again in 2019-12 by this adonis post over at c0ffee.net. (If you want to learn everything about email hosting. I warn you, your appreciation about how your hosted email system may become distorted). I was super curious about a machine learning model which categorises spam & ham, and has a delightfully beautiful interface. After following their install guide for what was probably 20 seconds, I realised my kernel, v2.6.32 would not, and would never run rspamd. While annoyed, I was complacent paying less than one coffee a month for my server, this is what you get. So why not compile an older version? The long and short of it is, it didn’t go so well. The first image that shows up in the c0ffee post is a delightful overview of how all the systems work together.
Look at it. This is the basis of all the moving parts your very basic email hosting platform has, so why am I hammering this point so hard? Rspamd showed its face again. Immediately I restarted my researching. I think I saved half of rspamd’s documentation to my read-it-later application while still not reading-it-later, but getting stuck into the documentation. But something was familiar. At the end of the day it was just another spam classifier. Spamassassin and Amavis have been my go to for what was close to a decade, and honestly, people other than I have probably been using a bayesian-based spam classifier for as long as it’s been invented. Over the years I taught it what I liked (ham) and what I definitely did not like (spam). We were the closest of best friends for the first honeymoon period. But then, out of seemingly nowhere, we were just two different people. Our relationship was not sustainable anymore. Others, were tricking my carefully curated filters. A little miss-hap now and again is fine, but emails with double email addresses in the
to field without a comma. That’s the oldest, new trick in the book. As I was stuck inside my own server, I never had known about what else was out there. So, I kept training old baisy, but it just didn’t seem to get me anymore. It was time I started looking elsewhere.
Over the horizon, looking through the vastness of blogs, the audio of tech-cons, there it was. Before putting “DevOps” in your about me section the “new hostness” plastering the top of HN posts, slashdot, and reddit, was Neural Networks, and Rspamd had it all. Options for multiple Neural Networks for training emails. Now, not only can you respond to new emerging spam techniques faster, you don’t adversely effect your “safer” network. How cool is that? Why was this not a thing earlier? Am I just old now and not keeping up with the kids and their new fancy toys?
Where to host it?
Now, I love building out servers as much as the next human, and if I had stable, fast upstream pipe at home, I’d probably setup a tiny nginx cache server setup somewhere, and have a rack heating my house. Ahh, the dreams of a tech head… but I lament. Having a VPS allows me to still have control over my data whilst being able to tinker with new tech. This hobby, of running a server in Australia is something that has always been on the semi-expensive side, compared. Where some non-cloud companies charging a standard rate of $70 per month! When the exchange rate was favourable, the same server in Italy was costing me $5! Now, when the world is in a panic, it was costing me around $12. After a weekend of search through threads, duck searches, and what seemed like 30 tabs of nearly all unique hosts in Australia, I found my new home and immediately got to experience one of my favourite past times. Setting up a new server.
Previously in another life, each server was given unique names, and cared for like a beautiful book. Giving it all the care in the world for the first read through. Until the years start showing through, and its tattered beyond believe and your shoving it in your bag whilst late for work. Now with an ever changing landscape of threats, patches, and node dependencies (take that node), we treat servers like we are growing a punnet of tomatoes. Caring for their health when we need it. Getting what we need from it. Before making it redundant for the next years batch. This time I wanted to use configuration management.
I’ve been coding using Ansible for $dayjob for a couple years now, and really enjoy using it. Ansible uses the old tried and true method of connecting via ssh for deployments and is great to bootstrap a system to a desired state. It’s simplicity reminds me of old instruction manuals of installing a system for the first time. Specifically I’m fondly reminiscing over the fun times installing Windows 95 (or Windows NT 4.0) with what felt like 100 floppy disks, and then getting a Zip disk drive, and marveling about the amazing capacity of these little things. Following each step of the manual carefully and methodically to a point where if you didn’t, you’d boot your system up and nothing would work. Then, methodically reading out each step, you find where you went wrong. This also reminds me of nearly every blog tutorial ever written, and all my documentation ever for $dayjob-v0.1. In Ansible, these are called “playbooks”, and they contain all the information in a step-by-step nature to get your system up and running. Automatically and exactly the same, everytime. I love configuration management and could talk about it for ages, but we’re getting off topic. If you have one take away from this, configuration management is amazing.
So the plan was to utilise Ansible to rebuild my server for the future, so rebuilding would be a sinch. I looked at what was installed and saw a terrible mess of packages, and configuration strewn across the system. I couldn’t not use containers after documenting what I needed to move across. I figured, out of the services I have running, email would be the biggest time consumer so i decided to tackle it first. The plan was to setup a docker container for: postfix, dovecot, rspamd & (redis), clamav, opendkim, openspf, and mysql. Being late to the game does have its benefits, so I logged onto the good ol’ internet machine, loaded up a decentralised search index (searx) “docker rspamd mailserver”, and you’ll find our newest friend: Mailcow. A completely email solution platform running in docker containers. Completely open source for everyone to use. This was everything I wanted. Not only did it completely handle the email hosting aspect, but it has integrated caldav, carddav, and even some form of exchange!
After spinning up mailcow on a local box, I knew I was already going to run with it forever until email gets superseeded by some other form of decentralised communication service. It had everything a person that wants to set and forget and just enjoy a service that works. Except rspamd has greylisting enabled by default, and if you’ve read my other post, you know how much I do not enjoy greylisting from a user’s perspective. So I started hacking up a storm and building my ansible playbook, as well as researching how others have setup their playbooks. Once it was already way too late I found out, Mailcow even has their own an ansible role. So I borrowed some inspiration from them as well: mailcow/mailcow-ansiblerole
Two minutes later, DNS updated, LetsEncrypt certificates issued, and my new mailserver is up and running. It even has inbuilt “sync jobs” which is just imapsync, but they really have thought of everything. I could go on-and-on about mailcow. Autodiscovery for email clients, autoconfig files for Apple iDevices, caldav, carddav, automatic emailing for calendar invites. A very, very nice user interface, but would probably only be nice when you have <20 domains. I can forsee issues in their UI, but for a IT geek running only a handful of domains, its pretty damn good! It also has the best thing imaginable: pretty graphs (as seen below). I even ported across my DKIM keys, and everything went to plan. Passing ARC/DMARC like I’ve been doing this for years. Years I tell you!
Rspamd has already proven its worth in mail classification with next to zero training. I have no doubt there will be a follow up post regarding how much I hate neural networks when legitimate mail starts getting flagged as spam, but until then, #teamRspamd. While this probably would have been a daunting task if I had not decided to containerise my email setup. Ansible, Docker, & Mailcow make it trivially easy to configure a server and start receiving emails. The configuration management, UI layout, how the system is designed and configured out of the box, (with the pretty graphs). Goes to show you how much love has gone into the Mailcow product. With what I thought would be the hardest part of my server migration turned out to be very, very pleasant. My friends already hate me for sending them calendar invites for our weekly games nights. Next up will be adding traefik into the mix for reverse proxying ontop of the containers. Setting up Isso in a docker container or maybe some other comment system.
How are you hosting your emails? Has anyone tried scaling out mailcow? ie: multiple containers of the anti-virus scanner for high throughput servers?
PS. Associated picture originally sent via a Toot on the mastodon network. Regarding me finding out about rspamd: link