How to set up your own mail server on a VPS
Your own mail on a VPS means mailboxes on your own domain with no per-user fees, full control over your messages and the ability to send newsletters and notifications from your own applications. In exchange you'll have to get to grips with DNS records and look after your IP's reputation: mail providers are strict about checking who sends messages.
The key requirement: an open port 25
Mail servers pass messages to each other over SMTP on port 25. Many hosts block outbound port 25 by default to keep spam from leaving their networks. Without it your server can receive mail but can't send it. That's why this selection includes only plans from hosts where the port is open or can be opened on request to support. Even if it's blocked at first, it's usually enough to write to support and explain that you're setting up mail for your own domain.
What to install
- Mailcow — a ready-made Docker bundle: Postfix, Dovecot, SOGo webmail, Rspamd anti-spam and a convenient admin panel. Needs 4 GB of memory and 2 cores or more, and is the easiest to set up.
- Mail-in-a-Box and iRedMail — all-in-one installers that configure the server and DNS step by step. They run on 1–2 GB of memory.
- Postfix + Dovecot by hand — minimal resources (1 GB is enough), but more manual configuration work.
If the server will run Docker with Mailcow and the ClamAV antivirus, choose 4 GB of RAM or more: ClamAV alone takes about a gigabyte.
DNS: without it your mail ends up in spam
- PTR (reverse record) — the server's IP should point to your mail host's name, for example
mail.example.com. The host sets it up in the control panel or on request — check that this option is available. - SPF — a TXT record listing the servers allowed to send mail on behalf of your domain.
- DKIM — a digital signature for messages; the mail server generates the key, and you publish the public part in DNS.
- DMARC — a policy for what to do with messages that fail the checks; start with
p=nonemode and reports.
IP reputation
Before launch, check the server's IP against blacklists (Spamhaus, Barracuda and others) — the address may have been inherited from a previous owner who sent spam. If it's listed, ask the host for a different one. Start with small volumes: a new address takes weeks to "warm up", and a sudden mass mailing from it will almost certainly land in spam. For large campaigns it's better to use a dedicated service and keep your own server for work mail and notifications.
Storage and backups
Mailboxes grow for years: plan for 20–40 GB or more for a small team and keep an eye on quotas. Store backups of messages and DKIM keys off the server, for example on a separate backup VPS.