Tagged in: email, projects, securitywashing, tls

§ intro

While working on an internal email (in)security presentation, I thought: “why not my own? It seriously can’t be that hard.” Even the Internet says it’s not.

There are many guides for the usual postfix & dovecot stack. But they are unnecessarily complex for a simple, stupid self-hosted single-user email. Sure, postfix can handle email for big corps, with aliases, and tens of thousands of users and mailboxes - but that’s clearly not my scope.

Though, I should probably run an LDAP server of my own now… with nextcloud, SSH and email having each their own passphrases, it’s getting a bit difficult.

§ bits and bytes

So, what is needed for modern email?

certificate
DNS
Postfix, dovecot, rspamd (that’s my choice, pick whichever you prefer)

§ Certificate

Pick a new subdomain for mail (I picked car.: carpophobie is the fear of fruits in French; and carpophobia is the fear of wrists in English). Get a certificate for that subdomain from Let’s Encrypt, or whatever CA you favor. We will need 2 files that will be used by both postfix and dovecot:

The key file (/usr/local/etc/ssl/car.popho.be/rsa.key)
The fullchain certificate (/usr/local/etc/ssl/car.popho.be/rsa.fullchain.cer)

acme.sh can generate both files.

/usr/local/sbin/acme.sh --issue --dns dns_ovh -d car.popho.be \
  --cert-file      /usr/local/etc/ssl/car.popho.be/rsa.crt \
  --key-file       /usr/local/etc/ssl/car.popho.be/rsa.key \
  --ca-file        /usr/local/etc/ssl/car.popho.be/rsa.ca.crt \
  --fullchain-file /usr/local/etc/ssl/car.popho.be/rsa.fullchain.cer

After the initial setup, I add the following to my crontab(5), and job done:

@weekly /usr/local/sbin/acme.sh --renew -d car.popho.be

It’s also possible to use an ECC certificate, with acme.sh’s --keylength. I did that, so my mail server has 2 certificates (RSA and ECC), which you can see on the CT logs at crt.sh.

Two certificates can be used at the same time for postfix, too.

§ DNS

There are three main topics here:

MX and PTR: what are my mail servers? My reverse must match.
SPF, DMARC, DKIM: who is allowed to send mail on my domain’s behalf? Will email be signed? What should the recipient do with broken email?
TLSA and DANE

§ MX and PTR

Easy, create an MX record with the hostname of the machine handling incoming. IPv4 and IPv6 should both resolve. And now my domain can receive mail.

The reverse lookup should also resolve back correctly to the MX record. If the reverse does not resolve correctly, there is a very high chance my outgoing mail is going to be flagged as spam (random machines shouldn’t be sending email to SMTP servers).

% drill popho.be MX
;; ANSWER SECTION:
popho.be.   86400   IN  MX  10 car.popho.be.
% drill car.popho.be AAAA
;; ANSWER SECTION:
car.popho.be.   3600    IN  AAAA    2001:470:7a83:7765::6d61:696c
% drill -x 2001:470:7a83:7765::6d61:696c
;; ANSWER SECTION:
c.6.9.6.1.6.d.6.0.0.0.0.0.0.0.0.5.6.7.7.3.8.a.7.0.7.4.0.1.0.0.2.ip6.arpa.   52883   IN  PTR car.popho.be.
% drill car.popho.be A
;; ANSWER SECTION:
car.popho.be.   3598    IN  A   151.80.43.167
% drill -x 151.80.43.167
;; ANSWER SECTION:
167.43.80.151.in-addr.arpa. 52904   IN  PTR car.popho.be.

§ SPF, DKIM, DMARC

We know what we’re doing, and we’ll be using one single host for both incoming and outgoing email. We write the most simple and strict SPF record: only allow our own MX machine to send mail, no one else will be allowed to (-all). It’s mandatory to have at least that in place before testing.

% drill popho.be. TXT
;; ANSWER SECTION:
popho.be.   86400   IN  TXT "v=spf1 mx -all"

DKIM will be put in place with rpsamd, and we don’t yet have a public key to publish on our DNS. In the end though, it looks like this:

% drill dkim._domainkey.popho.be TXT
;; ANSWER SECTION:
dkim._domainkey.popho.be.   3600    IN  TXT "v=DKIM1;s=email;p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDGP7z8fgplfAyBPAaINUEOggSN+ErJGx9L9BP+uD3y8hreG70N8GlGkjIXdItXoB5Ntx6K0iQQ5+hs6hntRgp5ocbozQs/9APgcdZ80vZctBG1LKE8nyoXC2uPryamz16BO5PCFKMW8ake0M97fp76LXtsIfTN5SbqGibY9ThFiwIDAQAB;t=s;"

DMARC will also be very strict; however it depends on both SPF and DKIM being in place, so we’ll let it sit and come back to it after rspamd is setup. FWIW, only GMail ever sent me DMARC reports.

% drill _dmarc.popho.be. TXT
;; ANSWER SECTION:
_dmarc.popho.be.    3600    IN  TXT "v=DMARC1;p=reject;rua=mailto:postmaster@popho.be;sp=reject;aspf=s;"

§ TLSA and DANE

See the Wikipedia article about it: long story short is to put a checksum of the public key (among others) in the DNS. This is of course feasible only if it doesn’t change too regularly. acme.sh doesn’t change the private key, so that’s cool.

openssl(1) can create our TLSA records. Thanks Tykling.

% openssl x509 -noout -pubkey -in rsa.crt | openssl rsa -pubin -outform DER 2>/dev/null | sha512
4b6d8c4ccf8d8539cad36de484b3c463a48914a0ad90fbfda0ba092337fa972a77cd92fb0ab81d4146ee38187dff421b052fb82759a6a3a1f94978910e1ddd0f

Now we can publish that SHA512 sum on the DNS. 25 and TCP are respectively the port and protocols of SMTP.

% drill _25._tcp.car.popho.be. TLSA
;; ANSWER SECTION:
_25._tcp.car.popho.be.  60  IN  TLSA    3 1 2 4b6d8c4ccf8d8539cad36de484b3c463a48914a0ad90fbfda0ba092337fa972a77cd92fb0ab81d4146ee38187dff421b052fb82759a6a3a1f94978910e1ddd0f

Of course, with the second certificate (the ECC one), it’s possible too: rsa should be replaced by ec in the second openssl(1) command.

% openssl x509 -noout -pubkey -in ecc.crt | openssl ec -pubin -outform DER 2>/dev/null | sha512

I chose TLSA 3 1 2 records because I won’t be changing my private keys. Other TLSA records could be used, and might even be more secure (because rollover reduces the utility of a compromised key). However, I have not yet found a tool to manage and automate my DNS zone (hosted at OVH). A tool based on acme.sh’s DNS libs to manipulate the DNS would be awesome.
Maybe I should run my own authoritive servers though? That’ll be another project, that will also require adequate security to avoid monstruous mistakes.

§ Wait, is DNS secure though?

Ha ha, no.

It is if one uses DNSSEC and the clients actually do check. Does your own resolver?

§ Postfix

Postfix handles incoming mail connections, outgoing mail (though it must first pass through rspamd), and even spam filtering (thanks to zen.spamhaus.org and some reject_rbl_client magic)

Postfix has many READMEs, which are much clearer than any other third-party guide (Basic, TLS,…).

With no further ado, here is my config file, as of 2020-01-12:

## LOTS OF DEFAULT CONFIG HERE
compatibility_level = 2
queue_directory = /var/spool/postfix
command_directory = /usr/local/sbin
daemon_directory = /usr/local/libexec/postfix
data_directory = /var/db/postfix
mail_owner = postfix
myhostname = car.popho.be
myorigin = $mydomain
inet_interfaces = all
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
unknown_local_recipient_reject_code = 550
mynetworks_style = host
alias_maps = hash:/etc/mail/aliases
mail_spool_directory = /var/mail/
debug_peer_level = 3
debugger_command =
         PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
         ddd $daemon_directory/$process_name $process_id & sleep 5
sendmail_path = /usr/local/sbin/sendmail
newaliases_path = /usr/local/bin/newaliases
mailq_path = /usr/local/bin/mailq
setgid_group = maildrop
html_directory = /usr/local/share/doc/postfix
manpage_directory = /usr/local/man
sample_directory = /usr/local/etc/postfix
readme_directory = /usr/local/share/doc/postfix
inet_protocols = all
meta_directory = /usr/local/libexec/postfix
shlib_directory = /usr/local/lib/postfix
## END OF DEFAULTS

# TLS and others
smtp_tls_CAfile = /usr/local/etc/ssl/cert.pem

# Force modern TLS on outgoing - this WILL result in undeliverable email
smtp_tls_security_level = encrypt
smtp_tls_mandatory_ciphers = high
smtp_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

# Force modern TLS on incoming - this WILL result in lost email
smtpd_tls_security_level = encrypt
smtpd_tls_mandatory_ciphers = high
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

# ECC keys with EC PARAMETERS object MUST use postfix >= 3.4.8
smtpd_tls_chain_files =
  /usr/local/etc/ssl/car.popho.be/ecc.key,
  /usr/local/etc/ssl/car.popho.be/ecc.fullchain.cer,
  /usr/local/etc/ssl/car.popho.be/rsa.key,
  /usr/local/etc/ssl/car.popho.be/rsa.fullchain.cer

# NAME AND SHAME
smtpd_tls_received_header = yes

# aliases and stuff
recipient_delimiter = +-.

# FROM http://www.postfix.org/SASL_README.html
smtpd_sasl_type = dovecot
# below must correspond in dovecot's conf
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
#smtpd_relay_restrictions (default: permit_mynetworks, permit_sasl_authenticated, defer_unauth_destination)
smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
smtpd_recipient_restrictions =
  permit_sasl_authenticated,
  permit_mynetworks,
  reject_unauth_destination,
  reject_unknown_reverse_client_hostname,
  reject_rbl_client zen.spamhaus.org=127.0.0.[2..11],
  reject_rbl_client b.barracudacentral.org=127.0.0.2
#  reject_unauth_destination,
#  check_policy_service unix:private/policy-spf

# A more sophisticated policy allows plaintext mechanisms, but only over a TLS-encrypted connection:
# Enable after TLS is working...
smtpd_sasl_security_options = noanonymous, noplaintext
smtpd_sasl_tls_security_options = noanonymous

# HARDENING
disable_vrfy_command = yes
smtpd_tls_loglevel = 2
smtp_tls_loglevel = 2

# SPF check
policy-spf_time_limit = 3600s

# Anti-spam (rspamd)
smtpd_milters = inet:localhost:11332

# skip mail without checks if something goes wrong
milter_default_action = accept

Of course, I had to find a usability bug in postfix, which was fixed in less than three weeks by upstream!

I added the reject_rbl_client blindly at first, seeing it around quite a bit. However, Wikipedia does a good job of explaining how it works

We can now add UNIX user accounts (pw(8) or adduser(8)) for our users, and work on aliases(5).

§ dovecot

dovecot is in charge of putting the email into the good mailbox. And that’s it. Most of the configuration is here.

The only real setting that had me tear my hair off my head was mail_location. In conf.d/10-mail.conf, it looks like this, and I had to create (by hand) some folders. That was weird.

mail_location = maildir:/var/mail/%u:INDEX=/var/indexes/%u

# ls -l /var/mail
total xx
drwx------  9 moviuro  wheel   16 Jan 13 19:38 moviuro
-rw-------  1 redis    redis    0 Nov  7 21:57 redis
-rw-------  1 rspamd   rspamd   0 Oct 31 16:14 rspamd

I have no idea why it works, how it works nor how to do it again. This will probably bite me later.

Also, we enable TLS, as we already have all necessary files.

We’ll test dovecot now, just to be sure.

% openssl s_client -connect car.popho.be:993
[...]
a LOGIN foo password!
a NO [AUTHENTICATIONFAILED] Authentication failed.
a LOGIN valid CoRReCTp@$$w0rD
a OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT
  SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND
  URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED
  I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH
  LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY LITERAL+ NOTIFY SPECIAL-USE]
  Logged in

Now, we can setup our mail clients to sync email; and we can also try sending some tests to our own GMail address (not too many though! We only have SPF working at the moment, no DKIM, etc.).

§ rspamd

That was unexpectedly easy. The configuration wizard was straightforward, and seems to have sane defaults. Turning on the redis server also was painless.

The quickstart guide is here. Rspamd also ships a configuration wizard, which makes it all easy (rspamadm configwizard).

rspamd can even take care of DKIM signature! Let’s follow the wiki.

Checking the log showed… a metric ton of errors, because of weird name resolution issues; such as localhost not resolving to the jail’s IP (10.10.10.25) or similar. That in turn caused the DKIM signing to not always take place, and mail getting sent without signature.

§ tests

After setting everything up, of course I had to run tests.

Am I correctly authenticated on my dovecot when I try to send mail (smtp) or read my email (imap)?
Does my email pass SPF (shouldn’t be an issue)? DKIM? Is my DMARC policy strict enough?
Can I receive email? Does rspamd filter out spam correctly?

Do to that, I’d simply try out broken passwords; try sending emails to my GMail account, see the original headers; send an email to my server with the following corpus: XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X, which is the GTUBE string (it must end up as spam).

There was one test that failed: sending email to my current client’s server, and that’s what prompted the following chapter.

§ Securitywashing

If you or your customers use GMail in any way, you have most probably seen the red lock. Already in 2016 Google planned on shaming bad administrators: if your corporation’s email was deemed insecure by Google, people (i.e. your revenue stream) might get scared.

In an ideal world, this would lead to administrators switching on TLS for mail, and tada, Google made the Internet a bit safer.

However, that was assuming that Corporate^TM cared. Spoiler: it doesn’t. It will even go to stupid lengths to not secure anything beyond the minimum (in that case: GMail). Here are some very interesting snippets:

# Bad Corporate -> Google
Received: from mail.corporate.bad (..... [x.x.x.x])
        by mx.google.com with ESMTPS id ...
        for <moviuro@gmail.com>
        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
        Day, 00 Jan 2020 time

This is TLS1.2 0x0c2f, which appears to be deemed sufficient by Mozilla.

# Bad Corporate -> Employer
Received: from mail.corporate.bad (.... [x.x.x.x])
 by moviuros.employer.mail (ESMTP service) with ESMTP id
 .... for <moviuro@employer.mail>; Day, 00 Jan 2020 date

Okay… that’s weird. No encryption?…

# Bad Corporate -> Personal domain
# Mail wasn't delivered, because I set up postfix to enforce use of TLS
[151.80.43.167] #<[151.80.43.167] #5.0.0 smtp; 5.1.0 -
 Unknown address error 530-'5.7.0 Must issue a STARTTLS command first'
 (delivery attempts: 0)> #SMTP#

So far, I can see that this corporate’s mail server actually only uses STARTTLS when communicating with GMail. Why it doesn’t even try to do TLS with my server (or that of my employer) is beyond me. We’re talking about a near-zero cost measure that is being actively held back to diminish the security level of communications.

That was for outgoing email only. What’s it like when I try to send mail to this corporate address?

# Personal email -> Bad Corporate
# /var/log/maillog
Jan 00 time car postfix/smtp[21660]: xxxxxxxxx: to=<moviuro@corporate.bad>,
 relay=mail.corporate.bad[x.x.x.x]:25, delay=1085, delays=1085/0.04/0.38/0,
 dsn=4.7.4, status=deferred (TLS is required, but was not offered by host
 mail.corporate.bad[x.x.x.x])

Not surprising, given my previous finding. However…

# Google -> Bad Corporate
Received: from mail-io1-f41.google.com ([209.85.166.41])  by
 mail.corporate.bad with ESMTP/TLS/AES128-GCM-SHA256; 00 Jan 2020 time

That is the TLS1.2 cipher 0x009c (or 0x00,0x9c), only used for “old” compatibility (Mozilla wiki).

# Employer -> Bad Corporate
Received: from moviuros.employer.mail (HELO moviuros.employer.mail)
 ([x.x.x.x])  by mail.corporate.bad with
 ESMTP/TLS/DHE-RSA-AES128-SHA; 00 Jan 2020 time

This looks like 0x0033 (0x00,0x33). That’s TLSv1.0. In 2020. This specific cipher was deemed barely tolerated by ANSSI (page 48, table A.10).

Quality of Encryption when sending and receiving email on select servers
v From/To >	corporate.bad	employer.mail	popho.be	gmail.com	orange.fr
corporate.bad	x	none	not delivered (none)	TLSv1.2 `0xc02f`	?
employer.mail	TLSv1.0 `0x0033`	x	TLSv1.2 `0xc019`	TLSv1.2 `0xc02f`	?
popho.be	not delivered (none)	TLSv1.2 `0xa7`	x	TLSv1.3 `0x1302`	TLSv1.0 `0x0039`
gmail.com	TLSv1.2 `0x009c`	TLSv1.2 `0xc02f`	TLSv1.3 `0x1301`	x	?
orange.fr	TLSv1.0 `0x0033`	x	TLSv1.0 `0x0033`	TLSv1.0 `0x002f`	x

Hexcodes from testssl.sh; the table was filled with information available in the headers of the received mail, or from the sending server’s maillog (@popho.be). orange.fr strips TLS information from its stored email. Watchful readers will have noted that I ran my tests before disabling TLS < 1.2 on popho.be.

According to my own experiment, the STARTTLS option on mail.corporate.bad is not being offered to everyone. That’s madness. And even when it is offered, it doesn’t present the same options (if it did, employer.mail would use TLSv1.2)! That in turn, is absolutely insane.
Furthermore, that same server will do tolerated TLSv1.2 only to GMail! This means that even if it can do TLSv1.2, it was disabled for most destinations!

orange.fr looks like it can only do barely tolerated TLSv1.0 connections (both for incoming and outgoing mail). Test it out yourself (note that on a domestic or enterprise network, this will probably be blocked by resp. your ISP or your adminsitrator):

% openssl s_client -starttls smtp -connect smtp-in.orange.fr.:25

employer.mail seems to be decently setup: it does some decent TLSv1.2, but won’t complain about the absence of encryption. Why it was presented with a STARTTLS option by Bad Corporation though (and popho.be not), I don’t quite understand.

GMail is the most secure of the bunch, doing fancy TLSv1.3 and stuff, also clearly displaying whether mail was sent encrypted or not. Clearly, it’s magic. There should be some on-by-default option for all mail clients that warns in big fat red letters when mail was sent/received either unencrypted, or “encrypted” with a 1999 technique.

The most disturbing thought is that Bad Corporation did everything possible to fly under the radar. It is actively undermining email security for a reason I don’t even want to hear. It’s clear that there are administrators there who understand what they’re doing (if not, Bad Corporate would have been publicly mocked and shamed by Google and its GMail customers already): if you’re one of those admins, make the securitywashing stop. Either stop pretending that your company cares, and slap that fat red broken lock on your emails; or fix your shit.

GDPR was announced in 2016, and even with threatening fines (up to 10% of global revenue), it transpires that security clearly is off the tables. We’re in 2020 now and I can only imagine that Bad Corporation is plagued by complacent admins and negligent management.

§ limits

Of course, I know that Google crunches your email to target advertizing. Still, they name and shame bad actors.

Of course, Bad Corporation is not in the business of handling email. Though at their current scale, and given their mass, it would be like presenting their website over http. No company would do that today, and no one would use it… right?

Mail is not hard, but it’s horrible

Or Securitywashing