Step-by-Step Guide: Building a Production-Ready Mail Server with Postfix, Dovecot, and Rspamd on Ubuntu 22.04

Running your own mail server is very different from hosting a website.
Here you are responsible not only for availability, but also for trust. Other mail servers must believe that your server is legitimate, secure, and not sending spam. That trust is built through DNS, encryption, authentication, and reputation over time.

This guide walks through building a real, production-capable mail server on Ubuntu 22.04, using Postfix, Dovecot, and Rspamd, explaining how each part works and why it is configured the way it is.

1. How Email Delivery Works in Practice

When someone sends an email to user@example.com,  their mail server does not establish a direct connection with your server immediately. Email delivery is a multi-stage process that critically depends on the public verification of identity and trust.

The Step-by-Step Process:

1. Initial DNS Check: The sending server first queries the DNS records for the domain
2. example.com
3. . This is like checking a passport and address before sending a valuable package.
   • MX Record (Mail Exchanger): Specifies which particular server (e.g., mail.example.com ) accepts mail for this domain.
   • A Record: Translates the MX server’s name into its IP address.
   • PTR (Reverse DNS): Verifies that the IP address from which the request is originating matches the claimed server name ( mail.example.com ).
4. Trust Verification (Identity & Anti-Spam): The sending server uses key DNS records to assess whether your server is reputable and not sending spam.
   • SPF (Sender Policy Framework): Checks if the sender’s IP address is authorized to send mail on behalf of example.com .
   • DKIM (DomainKeys Identified Mail): Checks for a digital signature that guarantees the message has not been modified in transit.
   • DMARC (Domain-based Message Authentication, Reporting, and Conformance): Specifies what to do if SPF or DKIM checks fail (e.g., quarantine or reject the email).
5. Establishing the SMTP Connection: Only after successfully passing these DNS checks does the sending server open a secure SMTP (Simple Mail Transfer Protocol) connection with your server.
6. The Receiving Server’s Decision: Your server (Postfix) accepts the connection, validates the user, and decides what to do with the message:
   • Reject: If checks (such as Rspamd) detect a high spam score or a mismatch in DNS records.
   • Accept and Process: The email is passed to internal filters (e.g., Rspamd for spam scanning) and then on to Dovecot.
   • Store: Dovecot saves the message in the user’s mail directory ( ~/Maildir ) and makes it available for reading via IMAP/POP3.

Because of this multi-layered system of public record-based checks, DNS and server identity matter more than the mail software itself. Without correct DNS configuration, your emails will be rejected by major mail providers.

2. DNS Configuration (Must Be Done First)

The correct setup of your Domain Name System (DNS) records is the single most critical step in self-hosting. Before you install any mail software, your DNS must be correctly configured. Mail software cannot compensate for missing or broken DNS records. Without a proper identity established in the public DNS, most major mail providers will automatically reject your emails.Mail Server Hostname

For this guide, we assume the mail server hostname will be:

mail.example.comThis Fully Qualified Domain Name (FQDN) must be used consistently across all components:

• In your public DNS records (A, MX, PTR).
• As the system hostname on the Ubuntu server.
• In your TLS/SSL certificates (e.g., Let’s Encrypt).
• Throughout your Postfix and Dovecot configuration files.

Critical DNS Records

3. Preparing the Ubuntu Server

The foundation of a reliable mail server is a properly secured and configured operating system. Before installing any mail-specific software, you must ensure your Ubuntu server is up-to-date and its core identity is correctly established to maintain system integrity and email deliverability.

Initial System Update and Security Baseline

Mail servers are high-value targets for attackers and spammers. It is critical to start with a system that has all known security patches applied to mitigate vulnerabilities.

Start by updating your package lists and upgrading all installed packages:

apt update && apt upgrade -y

Rationale: Running outdated packages significantly increases the risk of compromise. This command ensures you are beginning the deployment with a secure, stable baseline of the Ubuntu 22.04 operating system.

Setting the Hostname Consistently

The system’s hostname is not just an internal label; it is the identity your mail server presents to every other mail system on the internet during the SMTP connection. This identity must perfectly match the hostname defined in your public DNS records (A and PTR).

The system hostname must be set to the Fully Qualified Domain Name (FQDN) that you have prepared for your mail service. In this guide, that is:

mail.example.com

Set the hostname using the following command:

hostnamectl set-hostname mail.example.com

Verification:

Immediately verify that the change was applied correctly, as the FQDN will be used in subsequent configuration files and by external servers for identity verification:

hostname -f

The output must be exactly:

mail.example.com

Rationale: This FQDN is presented during the initial SMTP handshake. If it does not match the PTR (Reverse DNS) record that external servers query, receiving mail servers will immediately treat the connection as suspicious, often resulting in mail being rejected or sent straight to the spam folder. Consistent hostname configuration is a prerequisite for establishing trust.

4. Installing and Configuring Postfix

Postfix is the powerful and flexible Mail Transfer Agent (MTA) responsible for the core functions of your mail server: securely receiving mail from other mail systems (inbound) and reliably sending mail to external recipients (outbound).

Installation

Begin by installing the Postfix package.

apt install postfix -y

During the installation process, a configuration dialog will appear. Select the following options:

• General type of mail configuration: Select Internet Site.
• System mail name: Enter your primary domain name, e.g., example.com.
• Rationale: Choosing ‘Internet Site’ configures Postfix for direct delivery and receipt of mail on the public internet, which is necessary for a self-hosted server. The ‘System mail name’ sets the default domain for email addresses generated internally by the system.
  

Core Configuration: main.cf

The primary configuration file for Postfix is `/etc/postfix/main.cf`. We will edit this file to establish the server’s identity and control mail flow.

nano /etc/postfix/main.cf

Add or ensure the following minimal production-safe configuration parameters are set:

myhostname = mail.example.com  
mydomain = example.com 
myorigin = $mydomain 
inet_interfaces = all 
inet_protocols = ipv4 
mydestination = localhost

• myhostname: Postfix uses this FQDN to identify itself during the SMTP handshake. It must match your public DNS records (A and PTR) for maximum deliverability.
• mydomain: Specifies the primary domain name.
• myorigin: Determines the domain appended to mail sent by local users. Setting it to
  $mydomain
  ensures local emails are correctly addressed as
  user@example.com
  
• inet_interfaces: Set to
  all
  to instruct Postfix to listen on all network interfaces (public and private).
• inet_protocols: Set to
  ipv4
  to ensure it only binds to IPv4 addresses.
• mydestination: Crucially, set this to
  localhost
  . This prevents Postfix from accepting mail for the
  example.com
  domain directly, which is necessary because actual mailbox delivery (storage) will be delegated to the Dovecot service. This configuration helps prevent the server from becoming an open mail relay.

Enabling Secure Mail Submission (Port 587)

Mail sent by users or applications (outbound mail) should never use the standard port 25, which is intended for server-to-server transfers. Instead, we use port 587, the designated Submission port, which requires encryption and authentication.

Edit the Postfix master configuration file, `/etc/postfix/master.cf`:

nano /etc/postfix/master.cf

Locate the 

submission

 entry (it may be commented out) and add or uncomment the following lines immediately below it:

submission inet n – y – – smtpd 

  -o smtpd_tls_security_level=encrypt 

  -o smtpd_sasl_auth_enable=yes 

  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject

Rationale:

• smtpd_tls_security_level=encrypt: Forces the use of TLS/SSL encryption (STARTTLS) for all mail submission on this port, securing user credentials and message content.
• smtpd_sasl_auth_enable=yes: Enables the use of SASL (Simple Authentication and Security Layer) for requiring a username and password.
• smtpd_recipient_restrictions=permit_sasl_authenticated,reject: This is the primary security measure. It instructs Postfix to only allow authenticated users (
  permit_sasl_authenticated
  ) to send mail to external recipients. Any unauthenticated sender attempting to use this port to send external mail will be rejected. Without this restriction, the server would quickly be abused for spam.

5. Installing and Configuring Dovecot

Dovecot provides the IMAP/POP3 service, allowing users to access and read their mail. It also handles the user authentication process for both incoming mail (IMAP) and outgoing mail (SMTP via Postfix).

Installation

Install Dovecot core and the IMAP daemon:

apt install dovecot-core dovecot-imapd -y

Mail Storage Format

By default, we will configure Dovecot to use the Maildir format, which is more robust than the traditional mbox format, as it stores each message as a separate file.

Edit the configuration file:

nano /etc/dovecot/conf.d/10-mail.conf

Set the mail location:

mail_location = maildir:~/Maildir

Authentication Configuration

Secure authentication is critical. We will enforce the use of encrypted connections for password transmission.

Edit the configuration file:

nano /etc/dovecot/conf.d/10-auth.conf

Set the security options:

disable_plaintext_auth = yes 
auth_mechanisms = plain login

• disable_plaintext_auth = yes ensures that passwords are only accepted over an encrypted (SSL/TLS) connection.
• auth_mechanisms defines the standard methods Postfix will use when forwarding authentication requests to Dovecot.

Connecting to Your Mailbox (Client Settings)

Once Postfix and Dovecot are fully configured, you can connect to your mailbox using any standard email client (e.g., Thunderbird, Outlook, mobile mail app). Your client must be configured with the following settings to ensure a secure, authenticated connection:

Note: Port 993 is the standard port for secure IMAP access (IMAPS), provided by Dovecot. Port 587 is the secure submission port for sending mail (SMTP Submission), handled by Postfix.

6. Connecting Postfix and Dovecot

To handle user authentication for outbound mail sent via the secure submission port (587), Postfix must be configured to delegate this task to Dovecot. Dovecot is already configured to manage user credentials and secure IMAP/POP3 access, making it the central authentication authority for the entire mail system. This separation of duties is a standard security practice, as Postfix should focus on mail transfer (MTA) and Dovecot on mailbox access/user authentication (MDA/Authentication).

Delegating SASL Authentication

We need to instruct Postfix to use Dovecot’s authentication mechanism (SASL – Simple Authentication and Security Layer). This is achieved by telling Postfix where to find the Dovecot authentication service socket.

Edit the primary Postfix configuration file:

nano /etc/postfix/main.cf

Add the following three directives to establish the connection:

smtpd_sasl_type = dovecot 
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes

Rationale:

• smtpd_sasl_type = dovecot
  : Explicitly tells Postfix that its authentication requests should be handled by the Dovecot SASL implementation.
• smtpd_sasl_path = private/auth
  : Specifies the location of the Unix domain socket (
  private/auth
  ) where Dovecot listens for authentication requests from Postfix.
• smtpd_sasl_auth_enable = yes
  : Globally enables SASL authentication on the Postfix SMTP server process.

Applying the Changes

The new configuration requires both services to be restarted to reload the settings and establish the connection between them:

systemctl restart postfix dovecot

Once restarted, Postfix will no longer attempt to store or verify user passwords itself. Instead, it will securely forward all authentication attempts for port 587 to Dovecot, which handles the verification against the system user database.

7. Creating the First Mail User

Before testing the mail flow, you must create a dedicated system user account on the server. This user will serve as the actual mailbox owner, holding all emails delivered to it. For security, this user must be configured to possess a mailbox but be strictly restricted from having direct shell access to the server’s command line. This minimizes the attack surface should an authentication vulnerability be exploited.

Creating the User and Setting the Shell

Use the useradd command with the following critical flags to create the mailbox user securely. Remember to replace username with the desired email address prefix (e.g., info, vlad, or support).

sudo useradd -m -s /usr/sbin/nologin username

• -m : This flag ensures the user’s home directory (/home/username) is created. This directory is where Dovecot will store the user’s mailbox (Maildir).
• -s /usr/sbin/nologin : This is the key security measure. It sets the user’s default shell to /usr/sbin/nologin, which prevents the user from logging into the server directly via SSH or a console. This user can only authenticate for mail services (SMTP and IMAP) via Postfix/Dovecot.

Setting the Password

Immediately set a strong, unique password for the new user. This password will be used by mail clients for both sending (SMTP) and receiving (IMAP) mail.

sudo passwd username 

Rationale: The combination of creating a dedicated home directory and setting the shell to /usr/sbin/nologin establishes a secure environment. The user has a place to store mail, but no ability to execute arbitrary commands on the operating system, ensuring a clean separation between the mail system and system administration access.

8. TLS Encryption with Let’s Encrypt

Encrypted connections are no longer optional—they are an absolute mandate for modern email deliverability and security. Without a valid TLS/SSL certificate, major mail providers will reject your server’s connection, and mail clients will refuse to connect securely, making authentication impossible. We will use Certbot to obtain a free certificate from the trusted Let’s Encrypt authority.

Installing Certbot

Install the Certbot client, which manages the process of obtaining and renewing certificates:

apt install certbot -y

Obtaining the Certificate

We will use the standalone plugin to temporarily run a web server on port 443 to prove domain ownership. Ensure your firewall is temporarily opened or the ports are available. The certificate must be issued for your mail server’s FQDN (mail.example.com).

certbot certonly --standalone -d mail.example.com

Certbot will ask for an email address for urgent renewal and security notices. The certificates and private keys are stored securely in the directory /etc/letsencrypt/live/mail.example.com/.

Applying TLS to Postfix

Postfix requires the paths to the certificate files to enable TLS on its SMTP listener (Port 25) and submission port (Port 587).

Edit the main configuration file: 

nano /etc/postfix/main.cf

Add the following directives:

Applying TLS to Dovecot

Dovecot provides the IMAPS service on port 993, which requires a secured connection immediately upon connection.

Edit the SSL configuration file: 

nano /etc/dovecot/conf.d/10-ssl.conf

Set the following options:

If TLS/SSL configuration fails on either Postfix or Dovecot, mail clients will refuse to connect, and external mail servers will be unable to communicate with your server, causing authentication and delivery to break.

Note: Do not forget to restart both services after applying these changes: systemctl restart postfix dovecot

9. Installing Rspamd (Spam Filtering and DKIM)

Rspamd is an advanced spam filtering system that is essential for a production-ready mail server. It evaluates every incoming and outgoing message using a variety of checks, assigning a spam score. Crucially, it is also responsible for automatically generating the DKIM signature for all outgoing mail, which is mandatory for good deliverability.

Installation

Install the Rspamd package on your Ubuntu server:

apt install rspamd -yConnecting Rspamd to Postfix

To ensure all email passes through Rspamd for scanning and signing, Postfix must be configured to use Rspamd as a Milter (Mail Filter). Rspamd listens for milter connections on port 11332.

Edit the Postfix primary configuration file:

nano /etc/postfix/main.cf

Add the following directives to the end of the file:

smtpd_milters = inet:localhost:11332
non_smtpd_milters = $smtpd_milters
milter_default_action = accept

Rationale: The smtpd_milters parameter directs incoming mail streams to the Rspamd listener. The non_smtpd_milters ensures that locally generated mail also goes through Rspamd. Setting the milter_default_action to accept means that if Rspamd were to crash or be unavailable, Postfix would still deliver mail, preventing a complete outage (though spam filtering would temporarily stop).

Finally, restart Postfix to apply the Milter connection:

systemctl restart postfix

DKIM Key Generation

DomainKeys Identified Mail (DKIM) provides a cryptographic signature for every outgoing email, allowing receiving mail servers to verify that the message was genuinely sent by your server and was not tampered with in transit. Rspamd includes a tool to generate the necessary keys.

Run the following command to generate the DKIM key pair, replacing example.com with your actual domain:

rspamadm dkim_keygen -d example.com -s default

The command will output two parts: the private key (which Rspamd will use to sign messages) and the public key that you must add to your public DNS records. The output will look something like this (the key is shortened for this example):

dkim_private_key: /var/lib/rspamd/dkim/default.private

DNS record: default._domainkey IN TXT ( "v=DKIM1; k=rsa; p=MIGfMA0GCSq...IDAQAB" )

Action Required: This DNS record must be added exactly as shown to your domain’s DNS zone as a TXT record.

Functionality: Once the key is in DNS, Rspamd will automatically use the private key to sign all outgoing mail. Receiving servers will query your DNS for the public key to verify the signature, proving the authenticity of your outgoing mail and significantly improving your deliverability score.

10. Firewall Configuration

The firewall is the first line of defense for your mail server. Since mail servers are high-value targets, you must strictly control which ports are open to the public internet. By default, all ports should be closed, and only the essential mail service ports should be explicitly opened. This guide uses 

Uncomplicated Firewall (UFW), which is the standard firewall management tool on Ubuntu.

You must ensure the following three critical mail ports are open for the server to function correctly.

Required Ports and Services

• Port 25 (SMTP): This port is the global standard for server-to-server mail transfer (inbound mail). You must open this port to receive mail from other mail servers.
• Port 587 (Submission): This is the designated secure port for mail clients (like Thunderbird or Outlook) and applications to send outbound mail. It is mandated to require encryption (STARTTLS) and authentication.
• Port 993 (IMAPS): This is the standard port for secure IMAP access, which allows users to read and manage their mailboxes using an encrypted connection (SSL/TLS).

UFW Commands to Open Ports

Use the following commands to open the required ports:

1. Open the standard inbound SMTP port (25):
   ufw allow 25
2. Open the secure Submission port for outbound mail (587):
   ufw allow 587
3. Open the secure IMAPS port for mail client access (993):
   ufw allow 993
4. Optionally, if you want a basic layer of webmail or need to obtain the certificate via the standalone method:
   ufw allow 80
   ufw allow 443

Enabling the Firewall

Once you have opened the necessary ports, enable the firewall. WARNING: The moment you run this command, the firewall will become active, and any port not explicitly allowed will be blocked. Ensure you have allowed SSH access (Port 22, if applicable) before enabling UFW, or you will lose access to your server.

Enable the firewall:

ufw enable

Verify the firewall status to ensure the ports are correctly listed as ALLOW:

ufw status

11. Testing Mail Delivery

A mail server is not fully configured until both inbound and outbound mail flow are verified, along with critical security headers. This section uses the swaks tool and log analysis to confirm full functionality.

Outbound Mail Test

The first step is to confirm your server can successfully send mail to an external mailbox and that all DNS records are passing.

Install a testing tool:

apt install swaks -y

Send a test message:

swaks --to test@gmail.com --server mail.example.com --from username@example.com --auth-user username

(Replace test@gmail.com with a real external email and username with your created mail user. The command will prompt you for the user’s password.)

Verification (Outbound):

1. Check the logs: Monitor /var/log/mail.log during the test to see the successful SMTP transaction.
2. Inspect the received email: Check the message headers in the external client (e.g., Gmail’s “Show original” feature). SPF, DKIM, and DMARC should all show a PASS status. If any fail, you must review the corresponding DNS records in Section 2

Inbound Mail Test

Next, ensure your Postfix and Dovecot setup is correctly accepting and storing mail addressed to your server.

Send an external message:

Send a new email from any third-party address (e.g., your personal Gmail) to the user you created on your server: username@example.com.

Verification (Inbound):

1. Check Postfix logs for acceptance: Immediately after sending, monitor your server logs. A successful delivery will show the message being handed over to Dovecot.

tail -f /var/log/mail.log | grep 'username@example.com'

Look for a log entry similar to:

status=sent (delivered to mail-box)

or

status=delivered (mailbox /home/username/Maildir/...)

2. Verify Maildir storage: Confirm that Dovecot correctly saved the message as a file in the user’s Maildir folder.

ls -l /home/username/Maildir/new/

The presence of a new file confirms the message was accepted, scanned, and stored for the user to access via IMAP.

11.3 Accessing the Mailbox

With both inbound and outbound mail confirmed, the final step is connecting a mail client. Use the following settings in a client like Thunderbird or Outlook:

Final Notes

Self-hosting your own email server is a powerful decision that grants you complete control and privacy over your communication, but it is fundamentally different from managing a standard website. The success of your mail server is based not just on its uptime, but on the long-term trust and reputation it maintains with the global email ecosystem.

For this reason, your mail server is never “finished”; it requires ongoing, disciplined maintenance to ensure deliverability and security:

• Continuous Security Monitoring: Email servers are frequent targets for attacks. You must establish a routine for applying system updates (apt update && apt upgrade -y), particularly for Postfix, Dovecot, and the OS, to mitigate vulnerabilities.
• Reputation Management: You are solely responsible for preventing your server from being blacklisted. This means constant log monitoring (/var/log/mail.log) to detect unusual activity, adjusting spam rules in Rspamd, and immediately addressing any issues that could lead to your IP being flagged as a source of spam.
• Essential Renewals: TLS certificates (from Let’s Encrypt) are temporary and must be renewed regularly to avoid connection failures for mail clients and other mail servers.
• DNS Health Check: Periodically re-verify your critical DNS records (MX, A, PTR, SPF, DKIM, DMARC). Misconfiguration or expiration of any of these records is the quickest way to have your mail instantly rejected or placed in the spam folder by major providers.

Embrace self-hosting for the control it offers, but be prepared to invest the necessary discipline and vigilance for its long-term health.

1. Open the secure Submission port for outbound mail (587):
   ufw allow 587
2. Open the secure IMAPS port for mail client access (993):
   ufw allow 993
3. Optionally, if you want a basic layer of webmail or need to obtain the certificate via the standalone method:
   ufw allow 80
   ufw allow 443

Enabling the Firewall

Once you have opened the necessary ports, enable the firewall. WARNING: The moment you run this command, the firewall will become active, and any port not explicitly allowed will be blocked. Ensure you have allowed SSH access (Port 22, if applicable) before enabling UFW, or you will lose access to your server.

Enable the firewall:

ufw enable

Verify the firewall status to ensure the ports are correctly listed as ALLOW:

ufw status

FAQ