I was about to write an article on the surreal experience of hosting your own mail server in 2022, but this guy has already written all the things that I wanted to write.
I find it surreal that email is one of the oldest protocols on the Internet, one of the most openly documented, and yet a very small group of actors (with Google and Microsoft on the front line) have managed to make it impossible to self-host. I've also bounced through a lot of the holes mentioned by the author of the article, and I've also got the same impression: everything in the world of email services is designed to only benefit a small subset of actors and discard emails from anybody else. Spam checks are very robust and effective nowadays, but why bother to run them when you can just blacklist the whole IP subnet of a VPS provider with no explanation, and make sure that your friends in your illegal cartel do the same?
My current solution is to use ProtonMail with my own domains linked to it, run the bridge on my VPS, tunnel the IMAP and SMTP ports over SSH on my VPN, and use that as my private mail server. But it's a workaround, and I'm not entirely happy with it. I wish I could just run my Postfix server to manage my domains and send emails like it's 2005. Unfortunately, that's not possible: if you want to use email today either you pay someone for the service, or you accept that your private communications are stored on Google's or Microsoft's servers. Even if you have the skills to run your own server, you no longer have that choice. And it's time for @FRA to break this mafia.
i self host mail for > 10 years and have few problems. then, i don't write many mails to people on google or microsoft addresses. i don't even have setup complicated shenanigans like DNSSEC.
if nobody uses gmail etc. there is no problem with mail. that's the beauty with federation. if people have to expect that they have to deal with other providers than the big few, things will work. i think we need to route around them, not give in.
also, don't scream for more government regulation, enable others to break free of the shackles of those corporations. host mail servers for others, etc.
maybe i'm just lucky that in germany many people use the email provided by their ISP or even pay for a mail service. i haven't seriously dealt with a gmail address for a long time.
@bonifartius @FourOh-LLC @FRA using a Google or MS address just to email people with a Google or MS address doesn't sound like a good trade-off. Most of the people in my "real life" circle use a Gmail address (even though Microsoft is probably the main offender in my experience when it comes to blocking), so my private email would only be used to write to the few people in my "privacy-aware" circle. I used to encrypt all my emails with PGP until a few years ago when sending them to Gmail addresses, but nowadays even finding somebody who knows what PGP is (let alone having a keypair and configuring mobile clients to use it) is quite rare.
And it's not only a problem with email addresses that end with gmail.com or outlook.com: many businesses and public entities nowadays use mail services powered by Google or Microsoft, even if the domain is different, and those would apply the same restrictions to incoming emails.
And then there's a problem with email addresses associated to services. I run a Mastodon instance, a Gitea server, a Matrix homeserver and an SSO service - all things that offer user registration and things like confirmation and notification emails. When I used to run my Postfix server for the platypush.tech domains, most of the registration emails sent to Google or Microsoft addresses would bounce back, basically preventing those users from registering to my services - and, even if I hate Google and Microsoft, I don't want to create such barriers for users.
I'm calling for public intervention because that's IMHO the only way to solve this problem. We can't expect the industry to self-regulate: they have created their own little cartel, and they are happy to ban whole subnets instead of performing granular spam checks, especially when the nice side effect of this behaviour is that they manage to keep any competitors out. And we can't expect people like us to move the needle either: even if we build the perfect solution, we're a tiny minority in a world of billions of Google and Microsoft addresses. And, even if the alternative solution gets enough traction (like Tutanota did, at least on a small scale), Microsoft will just do what it did a couple of days ago - blacklist the domain, or prevent users from using that email address to register to Microsoft services.
We need public intervention because big fines imposed on these giants are the only way to get them to change - the language of profit is the only one they understand. The solution is also straightforward: you can't ban a whole IP subnet just because a spam email came from one of its servers, you are compelled to run granular checks on individual emails instead of applying draconian measures, and the rules to get emails from a new mail server accepted should be transparent, not the current kafkaesque bureaucratic nightmare that we have today.
> [...] and, even if I hate Google and Microsoft, I don't want to create such barriers for users.
that's the wrong line of thought imo. it just normalizes that it's ok to use these things which are the equivalent to shitting in the streets ignoring everyone who says that that's not ok. gmail etc. are big spammers themselves. just because everyone you know is shitting in the streets doesn't make it ok.
it's appeasing to people who don't care about these things. that's completely fine choice for them, but _they_ should face the consequences, not people who host things. this hand-holding and accepting bullshit has been slowly fucking up everything in IT. from linux on the desktop which brought so many bad things to linux it's a disaster, to the weird thing browsers are now, to "mail is now centralized because google is so comfy". it's beyond me why i should cater to those who don't fucking care.
i don't see that any governmental action would help because of things like this:
> More than a year after being asked by the European Union to standardize their Office 2003 XML formats, Microsoft submitted 2,000 pages of documentation for a new file format to the Ecma International consortium for it to be made into an open standard.
that's exactly what would happen to mail - or has happened already, only that google and ms don't have really good complicated standards to show and can't say "we did what you wanted us to" as nobody has asked. all the spam protection standards that got pushed and are "required" now are bullshit: DKIM, DMARC, SPF (which is the sanest one of the bunch). that's what happens if committees get involved, only that the plain google and ms solutions would be much more worse. they would lament about their business and how unfair it would be to drop these practices that politicians would ask them to specify how others should behave.
also those rules you proposed would equally be applied to everyone else. you know what happens next? ms and google start to send spam from single random addresses with plausible deniability because "it was a spammer!!1" and "we deleted the account!11". have fun complying to that self made sword of damocles of fearing a lawsuit or whatever the bureaucrats think of.
i want the government and regulation as far away from the net as possible. everyone hosting things is living in a legal twilight already.
@bonifartius @FourOh-LLC @FRA my point is that I want my services to be as inclusive as possible, because if entry barriers are lower then more people are likely to jump to more privacy-aware alternatives. Many people taking some small steps in the right direction usually have more impact than a smaller group taking bigger steps in the same direction.
If somebody registers to my Mastodon/Pixelfed instance or my Matrix server with a Gmail address, I still want them to be able to participate. Because they may like what they find here and stay, and, even if they use a shitty mail provider, at least we've got a user who spends less time on Facebook, Twitter or Instagram, or has one more use-case for using a private messaging app. If instead we require higher entry barriers (like having to use another email address, or another browser, or not being able to consume some content), then we're likely to lose momentum and only attract people that are already privacy-aware and ready to accept these trade-offs.
About committees - I agree with you on how dysfunctional they are today because of big tech derailing all the discussions to bring water to their mill. But they are the only tool we have, together with legislation, to change things. If regulation doesn't work because it's too slow, then we need to think of how to improve it, not dismiss it. If committees and open standards don't work because big tech throws too much weight at the table, then we need to think of how to enlarge the table so their weight gets diluted, not dismiss the process of open standards.
i understand your intentions but i think it doesn't work. free software "replacements" for closed services will never be able to compete with those, but they shouldn't try to in the first place. they bring so many other features to the table which the closed counterparts never will be able to have, but some of these features outright frighten people. like that there is no higher instance to appeal to. it's learned helplessness and most people like it this way.
i don't see that catering to those who aren't caring will help any more than it has helped in the last decade: we got ubuntu etc. only for them to include things like amazon search. we had a good firefox, only to get it dumbed down and features nobody in the userbase asked for (like aquiring pocket) being added. meanwhile never publishing a non shitty sync server to self-host. redhat added so many bespoke parts for "desktop linux" shit, that almost every distribution is now the same as it is so hard to fight the systemd bullshit layer.
i really don't know why this watering down and compromising is so popular. the strength of free software is precisely that it isn't like the closed source parts and often completely different.
regarding committees, the "write an RFC and publish it" method has served well for decades now. the net is very good at self organizing, many now follow the lure of "doing things the way big tech does", but i think this will be the source of much pain in the future.
@bonifartius @FourOh-LLC @FRA Ubuntu and Firefox are only products. They got funded by someone, they built something, then that somebody came back and said "looks pretty cool, but now how do you make money out of it?". This has happened countless times and it will keep happening. I personally don't care of what Ubuntu or Mozilla specifically do, as long as I have other distros to pick and an open Linux kernel underneath that I can customize and package however I like. Or as long as I have an open-source Chromium/Firefox codebase that I can fork to remove the stuff that I don't like.
I also don't believe in compromise built around mere *inclusion*. I believe in *extension* - the same bitter pill that big tech made us swallow for 40 years. We don't just include some of their things through paid partnerships. We force them to open up their APIs and protocols, and when they don't do that we scrape, reverse, mock and hack the shit out of them, until our containers contain all of their stuff, plus ours. We force the level playing field whatever the means, we force them to fight the competition around openness, not closeness, and that is a battle that we know much better than them how to win.
We should never forget that they don't *own* users, nor any form of content. They only own the container - the infrastructure that holds and processes a bunch of files and database records. We should pick our battle against the container, not against the content.
@bortzmeyer @FRA it's not even the amount of difficulty that scares me. I'm talking as someone who used to run an SMTP server 20 years ago, when dealing with the alien-written Sendmail configuration file was the only way to get emails from A to B.
It's more about the lack of transparency. Do we want a world where your server needs to abide to some conventions and needs to have some kind of "reputation score" in order to forward emails? Fine, then tell me clearly what are those rules, how the score is calculated, and let's make sure that everybody abides to the same rules.
I do hope things aren't as gloomy as painted in the blog post. We've run our own email service from home for 20+ years, and rarely have issues, though when google started making noises about it, we immediately set up SPF and DMARC. But I was toying with the idea of pushing it out to my VPS. Maybe that's not such a good idea if things are this bad. But I do agree it's not just a quick $MTA_OF_YOUR_CHOICE setup and off you go any more.
@fu @FRA DKIM/DMARC/SPF DNS records are the new bare minimum to get an SMTP server to work in 2022. But, as someone who has been hosting for years, I can say that these are not sufficient. Even though I configured all the DNS records correctly for my platypush.tech domain, most of the emails to Microsoft accounts used to still get bounced back (Google, surprisingly, didn't give any trouble).
In my experience Google used to be the main offender, but that role has been recently taken by Microsoft. They are really the ones who are trying to block anything that doesn't come from the established oligopoly. And, even when a provider manages to tick all the boxes (like Tutanota), and they can't come up with technical excuses for blocking them, they just block them directly with non-falsifiable arguments like "We've noticed a lot of spammers using this domain".
So the problem is that these guidelines for self-hosting are subject to change, because some actors out there are actively trying to prevent self-hosted from gaining any momentum, whatever it takes.
I'm very happy to see people continuing to play this cat-and-mouse game with big tech, it's because of them that we are not in a complete oligopoly yet, but in my case I've decided that I don't have unlimited resources to fight all the battles, nor to host all the services. For now the workaround that I've found (ProtonMail as the manager of my mail domains, the Proton bridge for IMAP/SMTP support, and an SSH tunnel exported over my VPN to expose my own "private server") is good enough for what I need, and it makes sure that my emails won't be randomly blocked.
I run my own mailserver for an organization (it's the cheapest way to have a non-Google-backed mailing list, aka what us old folks call a "listserv" or a "reflector"), and I really haven't found it to be all that bad.
You have to use a non-shitty VPS host, though; the real bottom-tier ones tend to have IPs with the worst reputation. And you need to ensure they won't change your IP address, like *ever*.
But good VPS + DKIM + SPF (DMARC is nice too) has given me good results.
My company (ThinkPenguin.con) still runs its own mail server. It's not impossible, but I'll give you this...challenging to jump through all the hoops ... absolutely. The solution to the problem is for more people to make others aware that they are losing email and business as a result of the providers they choose who have a draconian anti spam system. The big bad wolf is mostly Microsoft although there are some other small older providers that suck too.
A platform about automation, open-source, software development, data science, science and tech.