Bot releases are visible (Hide)

mox - v0.0.10 Latest Release

Published by mjl- 7 months ago

New features

Implement SMTP "FUTURERELEASE" extension in SMTP server. For submitting a
message for delivery from the queue at a time in the future, e.g. tomorrow
morning 9:00. (93c52b01a02d)
Delivery from the queue to multiple recipients in a single SMTP transaction,
transferring data only once. This is only done for recipients with the same
recipient domain. The maximum recipients limit from the RFC 9422 SMTP LIMITS is
honored and mox now announces its own limit. (47ebfa81526c)
Allow configuring DNS blocklists (DNSBLs) only for monitoring, without using
them for incoming email. Previously, mox would only monitor DNSBLs that are
used for incoming connections. But it is useful to know if your IPs are on a
blocklist, unrelated to whether you're using the blocklist. (15e450df6117)

Improvements

Mox has a separate website now, https://www.xmox.nl. It should be more
friendly for first-time visitors than the github page. It also has an initial
video, with more to come. Content, except images/videos, is in the mox git
repository, so functionality and documentation can be changed together.
(0bc30729440c)
Fix interpreting a per-account negative total size quota as "no limit",
overriding the global/default setting. (issue #115 by pmarini-nc)
Webmail: When Q/B-word-decoding attachment filenames, recognize more
character set encodings. (issue #113 by jsfan3)
Webmail: show unicode for internationalized email addresses by default. Not
the xn-- names, which are not user-friendly. A hover still shows the xn--
names. After talking to arnt at FOSDEM.
Localparts of addresses are now normalized to Unicode NFC throughout mox,
including for incoming deliveries. Previously, incoming SMTPUTF8 deliveries
with non-NFC-normalized unicode localparts (e.g. with separate code points for
accents and such (NFD)) would not be accepted. (8e6fe7459b6f)
The PRECIS (RFC 8265) password profile is now applied when setting/using
passwords. It prevents confusing users who use unusual unicode whitespace
codepoints in their passwords. (c57aeac7f09e)
Webmail: Implement registering "mailto:" links. Click the new button in the
Help popup to register. From Hans-Jörg. (ee1db2dde705)
In DSNs, show the full (multiline) SMTP response, not just the first line. It
often has helpful details. (50c13965a72b)
More DSN improvements: put the full SMTP reply in field Diagnostic-Code
(1c934f0103d5), when delivering a local DSN add Delivered-To header
(79da4faaa1e7), when parsing an incoming DSN set the Action field (e.g. failed,
delayed) (f6497b1aaf65), match DSN to threads based on Referenced/In-Reply-To
only, not subject (13923e4b7bfe).
In quickstart and self-check, improve the check whether resolver verifies
DSNSEC. We were looking up NS on ".", but some DNSSEC-verifying resolvers
respond to that with unauthentic data. (issue #139 by triatic)
More helpful instructions about setting up a DNSSEC-verifying resolver, and
how to test it. (issue #131 by romner-set)
Relevant for reusable components: the "slog" package from golang.org/x/exp is
now replaced with slog from the standard library. Function signatures have
changed. (d1b87cdb0da1)

Bug fixes

smtpserver: Spurious \r were sometimes injected when fixing up crlf line
endings for incoming messages. The check could look at wrong a buffer,
seemingly randomly incorrectly concluding a \r was missing before a \n.
Messages with the extra \r added shouldn't cause any trouble.
You can find these messages with grep -rn $'\r\r$' data/accounts/$youraccount/msg/,
remove one \r manually and run mox fixmsgsize $youraccount to
reparse the message. (issue #117, by haraldrudell)
Deliveries could seemingly randomly but consistencly fail with an incorrect
diagnostic about the message containing a bare newline. It would happy due to
crlf handling around buffer starts/ends. (issue #129 by x8x)
When adding a message to the queue for delivery, set the correct local
account, so DSNs about delivery failures go to the correct user, not the
postmaster account. (dc83ad1df559)
imapserver: In a sequence/uid pattern, the "*" would in some cases be interpreted
as the first message, but it should always be the last message in a mailbox.
(14aa85482e0e).
Fix displaying DMARC reports with empty values for some fields. The
TypeScript checks would reject them for being invalid enums, empty strings
are now part of the enum. (20812dcf62af)
In outgoing TLS reports, always use DNS ASCII A-labels, not unicode U-labels.
(62be829df078)
For domains configured only for TLS/DMARC reporting, don't reject messages to
that domain during submission, but deliver them as normal. (1d9e80fd7059)
smtpclient: Treat server's size limit of SIZE=0 as "no limit", instead of
failing to deliver. (39bfa4338a12)
Accept TLS reports that reference multiple domains, and that mix reports
about hosts and recipient domains. (e0c36edb8f1b)
Admin: Prevent writing out an invalid domains.conf that cannot be parsed
again. Happened when the last address of an account was removed through the
admin web interface. (issue #133 by ally9335)
Webmail: sending to invalid addresses could result in (failing) attempts to an
empty address instead of returning an error message to the user. (63cef8e3a59b)

Update instructions

Before upgrading, do a dry-run first.

Make a temporary backup with the old mox version:
mox-v0.0.9 backup data/tmp/testupgrade
Verify that all is well with the old version:
mox-v0.0.9 verifydata data/tmp/testupgrade
Verify the state with the new version:
mox-v0.0.10 verifydata data/tmp/testupgrade

With a successful dry-run, the upgrade should go smoothly. Make a new backup
with mox-v0.0.9 backup data/tmp/backup (the previous backup was modified by
the dry-run, so couldn't be used to restore!), replace the binary and restart.
For further details, see
https://www.xmox.nl/faq/#hdr-how-do-i-upgrade-my-mox-installation

If you run into any problems, please create a bug report.

Thanks

Thanks for contributions and/or feedback from: haraldrudell, x8x, romner-set,
triatic, mteege, Hans-Jörg, arnt, jsfan3, pmarini-nc, ArnoSen, andreasheil,
theduke, daluntw, lmeunier, ally9335, p-rintz, daftaupe (and everyone at the
FOSDEM email devroom, and all those I missed).

Feedback, feature requests, bug reports, contributions (start small!) are all
welcome. An easy way to help mox is to use it and spread the word!

Development on mox is funded through the NLnet NGI0 Entrust Fund,
https://nlnet.nl/entrust/, with financial support from the European
Commission's Next Generation Internet programme.

Download or compiling

See https://www.xmox.nl/install/#hdr-download.

mox - v0.0.9

Published by mjl- 9 months ago

New features

Per-account disk space quota. Mox now tracks the total size of messages in an
account (overhead from the message index database or file system is not
included). A maximum disk usage can be configured globally and/or per account.
If configured, the maximum allowed disk usage is enforced. By default, no
maximum is configured. Setting quota for accounts can prevent a single account
from filling up the disks. (d73bda751119)
When suggesting CAA DNS records (specifying which Certificate Authorities are
allowed to sign certificates for a domain), suggest variants that bind to the
ACME account ID, and restricts the validation methods to those used by mox.
Should prevent MitM close to a machine from requesting TLS certificates through
ACME as seen on the internet recently. (db3fef4981f2)
Add config file fields for ACME external account binding (EAB). Some ACME
providers require EAB to link an ACME account with a non-ACME account at the
provider. With EAB, more ACME providers can be used with mox. (ee1094e1cb7c)
Implement the PLUS-variants of the SCRAM authentication mechanisms:
SCRAM-SHA-256-PLUS and SCRAM-SHA-1-PLUS. The PLUS variants add TLS channel
binding: Authentication only succeeds if the client and server are on the same
TLS connection. Authentication will fail if there is a MitM (that has a valid
TLS certificate). (e7478ed6acb9)
Use a mail.<domain> CNAME for the SMTP (submission) and IMAP servers of a
domain, pointing to the mail server host name. Before, clients were instructed
to configure the mail server host name directly, but that makes it harder to
migrate the domain to another mail server in the future: All clients would need
to update their settings. A CNAME can be pointed to a new server without
requiring changes to client settings. (da3ed38a5cb0)

Improvements

The admin, account and mail web interfaces now use session cookie-based
authentication (with csrf) instead of HTTP authentication. These interfaces
now have a "logout" button (not possible with HTTP authentication). (#58,
0f8bf2f22088)
Webmail: Don't automatically mark unclassified messages in the Rejects mailbox
as non-junk when reading them. For all other mailboxes the behaviour is
unchanged. (416113af723a)
Webmail: Ask user to reload the application when the server version has
changed. (8e37fadc1310)
Webmail: In the message view, show the DMARC status of the domain of the
message "From" address. (fb81effe45cf)
Webmail: When composing, leave out our own address when replying. (7c1879da828c)
Junk filtering: Make content-based filtering for first-time senders more
strict for messages delivered over non-TLS connections, or when the addressee
isn't in a To/Cc message header. Common for junk, uncommon for non-junk.
(2ff87a0f9c84)
SMTP server: after "MAIL FROM:" and "RCPT TO:" commands, allow a space (which
is invalid syntax) also for delivery. We only allowed it for submission,
assuming only (submitting) mail clients or spammers had sloppy SMTP
implementations. In practice, also legitimate delivering mail servers have
sloppy implementations. (#101, af5da176239b)
When generating a Authentication-Results message header (with results for
SPF/DKIM/DMARC/etc), put each result on a new line for better readability.
(2710a5b97122)
Make many non-server Go packages more easily reusable. Package imports were
changed so more packages can be imported without pulling in mox internals. See
https://github.com/mjl-/moxtools for a tool that reuses Go packages. It is
deployed publicly at https://tools.xmox.nl/. As part of this change, mox now
uses Go's slog package for logging. Changes in the API's between releases are
tracked at https://github.com/mjl-/mox/tree/main/apidiff. (5b20cba50aa1,
72ac1fde29fb, f3a35a67663e)
SMTP server: for submission, if a message has a Return-Path header, only fail
in pedantic mode. (#103, 57fc37af2296)
Webmail: For messages in the Sent mailbox, show To/Cc/Bcc in italic, and show
all correspondents in collapsed threads. (#104, 802dcef192c0)
The admin and account web interfaces were changed from JavaScript to
TypeScript, making it easier to maintain. (a9940f9855d4).
Implement IMAP-UTF-7 more fully, and allow creating mailboxes with "special"
characters: "&" (the IMAP-UTF-7 escape character), "#" (the IMAP namespace
character), "*" and "%" (matching characters). These were not allowed out of
caution, but occur in real-world mailbox names. Mox now uses IMAP-UTF-7 when
sending mailbox names for clients that did not enable IMAP4rev2 or UTF8=ACCEPT.
Before, mox would always send UTF-8, but not all clients understand that, and it
can cause confusion with IMAP-UTF-7 and "&" escaping. (#110, d84c96eca585)
IMAP server: Add STATUS=SIZE as capability. It was already implemented as
part of IMAP4rev2, but older clients won't recognize that. (59bffa470153)
And more smaller improvements.

Bug fixes

SMTP server and SMTP smuggling: Mox was itself not vulnerable to SMTP
smuggling, treating only "\r\n.\r\n" as end of transaction. But two
improvements have been made: (1f9b640d9ae8)
1. Bare carriage returns are no longer accepted during SMTP transactions. Bugs
  in other mail servers can lead them to accept other sequences as
  end-of-transaction, notably "\r.\r". Mox would accept submitted messages with
  that sequence for delivery. Such messages could trigger bugs in other mail
  servers causing them to materialize non-existent messages. By no longer
  accepting bare carriage returns in submitted messages, mox can no longer be used
  to trigger the "\r.\r"-bug in other mail servers. SMTP transactions with a bare
  carriage return now result in an error mentioning SMTP smuggling. Mox can still
  store messages with bare carriage returns, e.g. from imports. Mox already added
  missing carriage returns to bare newlines.
2. A bug in mox caused sequences of "\nX\n" for any X (including "\n.\n") to
  result in a temporary processing error. For "\n.\n" this accidentially was fine
  behaviour, for other characters the bug has been fixed. Any sequence of
  "\r\n.\r\n" where one or both carriage returns are missing now result in an
  error mentioning SMTP smuggling.
IMAP server: The on-disk message size was not correctly calculated for messages
added with the APPEND command (typically used for imported messages and when a
mail client sends a message) when bare newlines ("\n") got a missing carriage
return added ("\r\n"). This would cause errors when attempting to read the
message. If you are affected by this, run "mox fixmsgsize " to fix up
incorrect message sizes. Reported by daftaupe. (02eb7b50330e)
SMTP server: When writing "slow responses" (when a message is deemed junk),
ensure the total response time isn't too long (slightly less than 30 seconds).
Slow responses were writing 1 byte per second. With a long response (e.g. long
error message), a sending mail server may not consume a full response. If mox
was the sending server, it would report a timeout after 30 seconds. Report by
naturalethic. (fbc18d522d29)
IMAP server: Only send "OLDNAME" in a response to the LIST command when
IMAP4rev2 is enabled. IMAP4rev1 clients (most common) don't understand it. From
duesee with imap-flow. (41e3d1af1057)

Update instructions

Before upgrading, do a dry-run first.

Make a temporary backup with the old mox version:
mox-v0.0.8 backup data/tmp/testupgrade
Verify that all is well with the old version:
mox-v0.0.8 verifydata data/tmp/testupgrade
Verify the state with the new version:
mox-v0.0.9 verifydata data/tmp/testupgrade

With a successful dry-run, the upgrade should go smoothly. Make a new backup
with mox-v0.0.8 backup data/tmp/backup (the previous backup was modified by
the dry-run, so couldn't be used to restore!), replace the binary and restart.
For further details, see
https://github.com/mjl-/mox#how-do-i-upgrade-my-mox-installation

If you run into any problems, please create a bug report.

After upgrading, you may want to:

Run "mox fixmsg " if you've imported messages over IMAP that have
bare newlines ("\n" instead of "\r\n").
Configure your email clients to use authentication mechanism
SCRAM-SHA-256-PLUS for SMTP (submission) and IMAP, if they support it. If mail
clients have trouble logging in after upgrading, they may be picking a SCRAM
PLUS variant without properly supporting it. Explicitly configuring the non-PLUS
authentication mechanism should fix the problem and ensures a MitM cannot
downgrade the chosen authentication mechanism by altering the list of supported
authentication mechanisms.
Change the CAA records for your domains to include the ACME account id and
allowed validation methods. See the suggested DNS records for each configured
domain.
Set disk usage quota, either globally for all accounts or per account. See
QuotaMessageSize in mox.conf, https://pkg.go.dev/github.com/mjl-/mox/config.
Add ClientSettingsDomain: mail.<yourdomain> to each domain in domains.conf,
add the CNAME record as afterwards suggested in the DNS records page, and
update client account settings to use the new host name.
For mox setups configured behind an existing webserver, add "Forwarded: true"
to the (Admin|Account|Webmail)HTTP(S) sections in mox.conf. It causes them to
use X-Forwarded-* headers for determining if HTTPS was active (for secure
cookies), and for the IP used for rate limiting.

Thanks

Thanks for contributions and/or feedback from: Fell, duesee (and
https://github.com/duesee/imap-flow/), daftaupe, naturalethic, jsfan3, Halyul,
mattfbacon, jsaponara, pmarini (and those I missed).

Feedback, feature requests, bug reports, contributions (start small!) are all
welcome. An easy way to help mox is to use it and spread the word!

Development on mox is funded through the NLnet NGI0 Entrust Fund,
https://nlnet.nl/entrust/, with financial support from the European
Commission's Next Generation Internet programme.

mox - v0.0.8

Published by mjl- 11 months ago

New features:

DNSSEC-awareness throughout the code base, based on
https://github.com/mjl-/adns, a fork of Go's DNS resolver. DNSSEC
is a requirement for DANE (see below). If you don't have a
DNSSEC-verifying stub resolver configured, DNS lookups are regarded
as unverified. Installing unbound and and is still the recommended
action.
DANE for incoming and outgoing delivery (RFCs 7672, 6698 and 7671).
DANE is a mechanism to require verified TLS (with STARTTLS) for delivery
over SMTP. Verification with DANE does not use the global WebPKI/PKIX
pool of Certificate Authorities. With DANE, verification is done based
on DNS records of type TLSA. These records specify (hashes of) public
keys to allow (DANE-EE), ignoring expiration/hostname-match/issuing
party, and/or they specify (hashes of) certificates of allowed
certificates authorities (DANE-TA), regardless of whether those
authorities are in the globally trusted WebPKI/PKIX CA pool.
DANE requires that DNS records are DNSSEC-protected, both to protect
the MX records and the TLSA records. MTA-STS (already implemented)
has similar goals, but does use the WebPKI/PKIX Certificate Authorities
pool, both to verify TLS certificates and to protect MX records.
DANE and MTA-STS can coexist: In the default configuration, mox
generates private keys, then retrieves certificates from Let's Encrypt
for these private keys (through https://github.com/mjl-/autocert, a
fork of golang.org/x/crypto/acme/autocert). These certificates are
valid for MTA-STS, and TLSA records are generated for the keys for
verification with DANE. For inbound delivery with DANE protection,
your DNS records must be DNSSEC-protected. For outbound delivery with
DANE protection, a trusted DNSSEC-verifying stub resolver is required.
Mox now compiles on Windows, so "mox localserve" and most other
commands to work, but "mox serve" (the actual mail server) does not
yet work.
"SMTP Require TLS Option" (RFC 8689), consisting of two mechanisms:
1. A REQUIRETLS SMTP extension to require verified TLS along each hop
  in message delivery, either through MTA-STS or DANE.
2. A message header "TLS-Required: No", that overrides any TLS
  requirement along the way as specified by any MTA-STS or DANE
  policy.
  These mechanisms can be used to ensure secure delivery, or to work
  around delivery issues due to TLS requirements. Mox remembers whether
  an SMTP server offered the REQUIRETLS extension. Webmail automatically
  selects it if all recipients support it. Webmail also lets the user
  select the "TLS-Required: No" header.
Outgoing DMARC reports (RFC 7489). Mox now stores the results of DMARC
evaluations for inbound messages. These results can be viewed in the
admin web pages. Reports are typically sent every 24 hours (covering a
24 UTC day), but will be sent for up to 1 hour intervals if requested
by a domain. Sending DMARC reports is enabled by default, but can
be disabled through new option NoOutgoingDMARCReports in mox.conf.
Reporting addresses can be added to a suppression list, to reduce
noise due to deliverability issues. Incoming DMARC reports were
already implemented.
Outgoing SMTP TLS reporting (RFC 8460). When delivering outbound
messages, the SMTP client will look up MTA-STS and/or DANE policies
for TLS requirements, with a fallback to opportunistic TLS.
The evaluated security policies, (TLS) connection success/failure
counts, and any failure details, are stored. Reports are sent once
per day to reporting addresses in the TLSRPT DNS record of a domain,
over a 24 hour UTC day period. By default, reports are only sent
if there was a failure. The pending results can be viewed in the
admin web pages. Sending reports can be disabled with new option
NoOutgoingTLSReports in mox.conf. Reports with only successes can be
enabled through OutgoingTLSReportsForAllSuccess. Reporting addresses
can be added to a suppression list to reduce noise due to delivery
failures.

Improvements:

Webmail: Recognize encoded file names in message attachments. Either with
RFC2231-encoding (as specified) or Q/B-word encoding (as used in practice).
(#82)
Webmail: For portait images, don't let image extend beyond window height.
Webmail: Wrap long header lines, instead of showing horizontal scrollbar.
Webmail: Replying without having text selected now starts a top-post
with an "On ... wrote:"-line. Replying with text selected still starts
a bottom-post containing only the selected text, quoted. (#83)
Webmail: In the compose window, autoresize address input fields to
match the content.
Webmail: When composing a message, show security properties of recipient
addresses: Whether STARTTLS is known to be offered by the SMTP server
(historically), whether MTA-STS is implemented, whether MX records are
DNSSEC-signed, whether DANE is implemented, and whether REQUIRETLS is
offered by the SMTP server (historically).
Webmail: Add clear marker between message header and body, so an
HTML message cannot fake being part of the UI.
Webmail: If a "display name" of an address contains address-like
characters ("@" or "<" or ">"), only display the actual email address
in the message listing, not the display name. Should prevent confusion
attacks with messages specifying an unrelated email address in the
display name.
The suggested SRV DNS record for autodiscovery now points directly to
the host name, not to a CNAME (which is technically invalid, but seems
to work in practice).
When ACME-validation for a new TLS certificate fails, log error messages that
may explain the reason. E.g. "your CAA record forbids Let's Encrypt from
issuing certificates".
SMTP server: workaround for Windows Mail that has invalid additional space in
its "AUTH PLAIN" command.
Fix delivery to recipient domains with an MX host containing an underscore,
such as "_dc-mx.." as apparently used by cloudflare. From
richard g.
When generating a DSN message (for delivery failure), try harder to DKIM-sign
it: With a configured domain, also when sending from
postmaster@mailhost..
For incoming messages, track whether TLS and REQUIRETLS was used during
delivery, and whether the message matched a forwarding or mailing list rule,
and show it in the webmail.
In logging, change "fatal io error" to just "io error". The "fatal" sounds
too serious, it's just the connection that will be closed. (#39)
Add rfc/xr.go to generate HTML pages with cross-referenced code and
RFC. These HTML pages are published at https://www.xmox.nl/xr/dev/
Webmail: In case of long lists of addresses in To/Cc/Bcc headers, only show
the first 4 addresses along with a "More" button. (#98)
Clarify documentation on importing messages from the command-line,
which can be unintuitive due to systemd service file mount points. (#79)
Implement obsolete SASL LOGIN for submission, for interoperability with the
new cloud Outlook.
Fix IMAP ESEARCH response for clients before IMAP4rev2, notably cloud
Outlook.
Many small improvements.

Bug fixes:

Security: When looking up MTA-STS policies, don't follow CNAME records
for the recipient domain. A single unauthenticated CNAME response
could redirect policy lookup to another domain.
Webmail: When replying to selected text consisting of characters in multiple
unicode blocks, don't loose some of the selected text in the reply.
Don't parse DKIM "selectors" as IDNA domains. They are just DNS
labels. Based on email from richard g.
Update to latest bstore (database library) to fix a bug with
deleting/updating records. Problem found during development of new
features, behaviour not seen in any committed version.
Webmail: Fix the date shown in the message headers. It was off by the timezone.
Fix concurrency bug with accessing a math/rand PRNG with Read. Mostly
replaced with crypto/rand. Found during development and tests.
The queue page on the webadmin would fail with a JS error when a message was
in the queue and no transport was configured (which is the default).
For domains configured only to accept DMARC reports, don't request an
autoconfig TLS certificate through ACME at startup.
For incoming messages, convert bare newlines to carriage
return+newline. The import code already did this. Having bare newlines
could cause imapserver's fetch command to fail with a (connection)
panic in some cases.

Update instructions:

Before upgrading, you should do a dry-run first:

Make a temporary backup with the old mox version:
mox-v0.0.7 backup data/tmp/testupgrade
Verify that all is well with the old version:
mox-v0.0.7 verifydata data/tmp/testupgrade
Verify the state with the new version:
mox-v0.0.8 verifydata data/tmp/testupgrade

With a successful dry-run, the upgrade should go smoothly. Make a new backup
with mox-v0.0.7 backup data/tmp/backup (the previous backup used for the
dry-run has been modified, so couldn't be used to restore!), replace the binary
and restart.

If you are upgrading from v0.0.6, see its upgrade instructions for commands to
execute. It's better to immediately upgrade to v0.0.8 (see issue #71).

If you run into any problems, please create an issue.

After upgrading, you may want to configure DANE:

To make use of DANE for outbound deliveries, make sure you have a
trusted DNSSEC-verifying stub resolver. Unbound is recommended. Don't
use systemd-resolved, its DNSSEC support is not ready for use.

To make use of DANE for inbound deliveries, first make sure your
DNS records are DNSSEC signed, and your DNS operator supports TLSA
records. The SMTP TLS private keys ("host keys) should be added to
the TLS section of the "public" listener in mox.conf. If you use ACME
(e.g. with Let's Encrypt), you will want to use the private keys of
existing certificates. Run "mox config ensureacmehostprivatekeys"
to find existing or generate new private keys, and print the config
snippets you'll have to apply to mox.conf.

You may want to update your autodiscovery DNS record. See the "DNS check"
admin page or run "mox config dnscheck ".

Thanks:

Thanks for contributions and/or feedback from: taavi, naturalethic,
mattfbacon, duesee, mpldr, richard g, ArnoSen (and those I missed).

Feedback, requests, bug reports, contributions (start small!) are all welcome.

Development on mox is funded through the NLnet NGI0 Entrust Fund,
https://nlnet.nl/entrust/, with financial support from the European
Commission's Next Generation Internet programme.

To download, see https://github.com/mjl-/mox#download

mox -

Published by mjl- about 1 year ago

version: v0.0.7
date: 2023-09-24

Update instructions:

Due to a bug with "expunging" (deleting) messages that were
junk-filter-trained, messages that were removed from disk could be resurrected,
causing errors when mox would later try to open such messages again. Before
upgrading, you should first check and resolve this problem:

Find missing files by running: mox-v0.0.6 fixmsgsize
Create empty replacements for the missing files (use "touch data/accounts/...").
Update the message metadata in the database for the newly created files:
mox-v0.0.6 fixmsgsize
(this will fix the previously missing files, but should no longer find new
missing files)
Fix per-mailbox message/unread counts for each affected account, run:
mox-v0.0.6 recalculatemailboxcounts
("account" is the name as it appears in the data/accounts/ directory,
not necessarily an email address)
The affected accounts will see the recreated empty messages, which
can be deleted.

Now with the storage consistency resolved, you can start with the upgrade. Mox
v0.0.7 adds message threading, and all messages will be read and assigned a
message thread. This is done in the background, in two steps, and may take a
while. The first step adds Message-ID and a "thread base subject" to each
message in the database. The second step reads through all messages and uses
their References/In-Reply-To/Subject message headers to match threads. In
testing, upgrading took approximately 1 minute per 100k messages, but it will
depend on the hardware. Accounts are available for reading and delivery during
the upgrade, but the webmail may tell you that threading is not yet available.

You should do a dry-run of the upgrade first:

Make a temporary backup with the old mox version:
mox-v0.0.6 backup data/tmp/testupgrade
Verify that all is well with the old version:
mox-v0.0.6 verifydata data/tmp/testupgrade
Trigger the threading upgrade using the new version:
mox-v0.0.7 openaccounts data/tmp/testupgrade
(it prints nothing until done; the live upgrade prints progress)
Verify the new state:
mox-v0.0.7 verifydata data/tmp/testupgrade

With a successful dry-run, the upgrade should go smoothly. Make a new backup
with mox-v0.0.6 backup data/tmp/backup (the previous backup used for the
dry-run has been modified, so couldn't be used to restore!), replace the binary
and restart.

After the upgrade, any accounts affected by the "missing file" problem should
get their "uid validity" increased, so IMAP clients will resynchronize. Run
"mox-v0.0.7 bumpuidvalidity " for each affected account. The change is
made directly in the database file, so no IMAP/webmail sessions should be
active for the account. If any sessions are active, the database file is locked
and "bumpuidvalidity" will print a timeout error. Briefly shutting down mox is
an option, the bumpuidvalidity command operates quickly.

Apologies for the inconvenience, mox aims to make administrating a mail server
easier than this.

If you run into any problems, please create an issue.

New features:

Keep track of message threading, and add a threaded view to the webmail. Newly
delivered messages are matched against threads, as are imported messages. The
message threading is currently only exposed through the webmail client, not yet
through IMAP. The webmail has new keyboard shortcuts to navigate between
threads, collapse/expand them (with state remembered), or mute them (so new
deliveries to the thread are marked as read).
Add option to accept DMARC and TLS reports for other domains. You would add
the other domain in mox, and specify an alternative domain (of your regular
dmarc/tls reporting address) for the recipient of the reports. The webadmin
DNS-check page will show the required DNS records for delegated reporting.
Incoming reports will be delivered to the configured reporting address, and
processed by mox.
Easier account setup on Apple devices without device management profiles
(.mobileconfig files). Download a profile from the account web interface, at an
email address. Or open the QR-code to easily get the profile on a mobile device.
For issue #65 by x8x.
Transparent gzip compression when webserving files for selected content-types
that are likely compressible. Both for static files (with a cache of compressed
files, max 512MB stored in $datadir/tmp/httpstaticcompresscache), and for
forwarded requests (gzip level "fast"). Compression can be enabled per handler.

Improvements:

In SMTP server, for submission (with authenticated clients), don't fail on a
bad domain/IP address in the EHLO command. With submission the domain/IP is
irrelevant, and clients often fill in something that isn't strictly correct. No
need to prevent those users from submitting email (except in pedantic mode).
For issue #55 reported by gimpf.
At top of config files, mention the config file format, and hints to prevent
likely mistakes (given how sconf is different from what admins may be used to),
including a pointer to the sconf documentation page. For issue #56 reported by
kikoreis.
Recognize when quickstart is probably run behind a NAT, possibly a container,
and set the "NATIPs" field of the "public" listener in mox.conf accordingly.
This triggers when you set up mox for a public domain, but only have
private/loopback IPs on the machine. For issue #59 reported by pmarini.
When moving a message out of Rejects mailbox, mark it as unread. For issue #63
by x8x.
The "mox setaccountpassword" subcommand now takes an account name as
parameter instead of an email address. The email address could be confusing
in the face of wildcard addresses and a typo/non-explicitly-created address:
The account holding the wildcard address would get a new password. For issue
#68 by x8x.
Make Mac OS X Mail use the special-use mailbox attributes so it finds the
correct "Sent" and "Trash" mailboxes (instead of creating its own "Sent
Messages" and "Deleted Messages" mailboxes). The IMAP server now
unconditionally sends the special-use flags, even if the mail client doesn't
request it. For issue #66 by x8x.

Bug fixes:

Fix expunging of messages marked junk/nonjunk. The messages would be marked
as expunged, then the junkfilter would retrain and clear the expunged field
again. This would cause the message to be resurrected while the on-disk message
file was already removed. Trying to read such messages would fail. The update
instructions should help fix the problem.
Fix "mox sendmail" when submitting over a TLS connection by setting the remote
host name to verify the certificate of. Due to a logic bug the name wasn't set
and the connection would fail due to the missing setting.
Don't generate duplicate suggested SPF record if hostname is equal to domain
name, e.g. [email protected]. Fixes issue #46 reported by x8x.
Fix showing attachments of type text/plain in webmail, they weren't shown at
all because they were skipped when parsing the message during webmail message
processing.
Fix parsing the List-Post header in messages, for use in webmail when replying
to a mailing list message.
When moving a message to the mailbox that has the special-use "junk" flag,
mark the message as junk and retrain. This should have already worked just like
the "AutomaticJunkFlags" config option, and the default account config already
handles marking messages as junk based on that option, but the special-use flag
should be recognized independently and now also takes precedence.
Set the correct special-use mailbox flag "\Drafts" instead of "\Draft" on the
draft mailbox. Mail clients may have not found the correct drafts mailbox
before. For issue #66 by x8x.
Fix "mox bumpuidvalidity" to not create a (mostly harmless) uidvalidity
inconsistency that "mox verifydata" will warn about. For issue #61 by x8x.
And quite a few smaller tweaks/improvements/fixes.

Special thanks for contributions and/or feedback from: x8x, gimpf, kikoreis,
pmarini, fairking, gedw99, hmfaysal (and those I missed).

Feedback, requests, bug reports, contributions (start small!) are all welcome.

Good news: Mox is now being funded for a year of continued development through
the NGI0 Entrust Fund, a fund established by NLnet with financial support from
the European Commission's Next Generation Internet programme,
https://nlnet.nl/project/Mox/!

mox - v0.0.6

Published by mjl- about 1 year ago

Update instructions:

Make a backup, replace the binary and restart.

After the upgrade, the first time an account is opened with this new version,
new message indexes are created and mailbox message count statistics are
calculated. For large mailboxes, the time and memory this takes can be
noticable. The upgrade tests take about 15 seconds on 570k messages on a
thinkpad x1 from 2018, and run with a max memory data size of 768MB.

Don't forget to make a backup of the data directory with your currently running
mox before upgrading (e.g. "mox-v0.0.5 backup data/tmp/backup"). You can
dry-run the upgrade by making a separate backup ("mox-v0.0.5 backup
data/tmp/testupgrade") and running the "verifydata" command with the new mox
version ("mox-v0.0.6 verifydata data/tmp/testupgrade"). Running "verifydata"
with a newer mox will make changes to the database files, so don't run it on a
backup you may need to restore.

It is recommended to run the new "mox reparse" command after upgrading. It will
reparse all messages with the improved message parsing code.

For existing installations, the new webmail must be enabled manually in mox.conf
with config options "WebmailHTTP" and/or "WebmailHTTPS", similar to
"AccountHTTP(s)". See the example config printed by "mox config
describe-static".

If you are forwarding email to an address hosted with mox, you may want to
configure the new "IsForward" and possibly "AcceptRejectsToMailbox" options in
a delivery ruleset for better junk handling/analysis.

If your mox is behind a NAT, and your mox.conf uses config option IPsNATed, you
should switch to new config option NATIPs, and specify the public IPs that are
NATed.

New features:

Webmail, for reading/writing messages. It is similar to other regular and
webmail clients, with a decent set of features for a first version (with more to
come in the future). Webmail is enabled for new installs created with the
quickstart. For more details about the implementation, see
https://github.com/mjl-/mox/commit/849b4ec9e9f9bf7cdd2862de9e2a2dd096858d5c.
IMAP extensions CONDSTORE and QRESYNC have been implemented. With CONDSTORE,
changes to messages are tracked with a modification sequence, "modseq".
This is used by mail clients both to efficiently find changes when reconnecting,
and for conditionally storing updated message flags (only if modseq is still as
expected). QRESYNC is an additional extension for faster full mailbox
synchronization. These are useful by themselves, but keeping track of the
"modseq" will also help with JMAP.

Improvements:

Faster IMAP "STATUS" command (which checks mailbox status), because mox now
keeps track of total/unseen/deleted number of messages in mailboxes. Noticable
in larger mailbox (>50k messages).
Be less strict by default when parsing messages, and handle non-ascii/utf-8
encodings in message subjects. We already accepted messages with problems, but
stored them mostly unparsed. IMAP clients would parse the message themselves so
users wouldn't notice anything, but now that we have webmail it is more
important to have parsed forms of messages for problematic messages.
Properly decode character encodings other than ASCII and UTF-8 when
returning/searching text in messages.
Be more lenient for (authenticated) submission of email over SMTP (but not
during regular SMTP delivery). Before, we were strict about certain protocol
violations, but that wasn't helpful behaviour for legitimate users. In pedantic
mode, mox still rejects these violating commands.
New config options "AcceptRejectsToMailbox" and "IsForward" in Rulesets, for
handling incoming messages that are forwards from another address. By setting
"AcceptRejectsToMailbox", if the junk analysis says a message is junk, the
message is accepted during SMTP instead of rejected, but delivered to the
configured mailbox. It isn't always a good idea to reject forwarded messages
that are junk. By setting "IsForward", the junk analysis and future
classification based on such a message is done differently. See the config
option for details. (PR #50 by bobobo1618)
Add a KeepRejects options to account configurations, so the Rejects mailbox
isn't automatically cleaned up. (PR #49 by bobobo1618)
Add option NATIPs to a listener in mox.conf for better handling when mox is
configured behind a NAT.
"mox verifydata" now checks for more potential issues, and mox gained a few
subcommands to fix those issues.

Bug fixes:

Fix showing the progress while importing messages. Messages were still
imported, but the SSE connection with progress updates was broken.
Fix potential delays in propagating changes to IMAP IDLE connections.
IMAP compliance: add missing space after continuation line ("+").
IMAP compliance: add missing empty untagged response for SEARCH in case of no
matches.
Fix the -existing-webserver flag for "mox quickstart". Its TLS key/cert check
tripped it up.
And many more small fixes and improvements.

Special thanks for contributions and/or feedback from: Mendel, bobobo1618,
hmfaysal, x8x, kikoreis, gerben, andrii, liesbeth, morki, gedw99 and everyone I
forgot. More feedback/bug reports welcome!

To download, see https://github.com/mjl-/mox#download

mox - v0.0.5

Published by mjl- over 1 year ago

v0.0.5

Update instructions: No special steps required other than updating the binary.

Fixes/Improvements:

Fix "mox localserve", and some commands that talk to a running mox instance
over the ctl socket. Broken in previous release.
During quickstart, warn if the host name resolves to a loopback IP, as it
will likely prevent email delivery to local accounts from working. (#37 by
dmikushin)
In case of DNS resolve errors on Linux likely due to missing name server in
/etc/resolv.conf, point user to systemd-resolved manual page, the likely
cause. (#38 by ArnoSen)
Redirect requests for /admin to /admin/, to prevent confusing users about
failed credentials when they try to open the admin web interface. (#43 by
FieryCod, lemos1235)
Fix race condition when handling forwarded websocket connections. Could show
up as calling Close on a nil connection. (found by existing tests)
Fix parsing bugs in the IMAP server. One of them may cause correct commands
to be rejected, but this probably isn't commonly triggered. (found by new
tests)
Fix bug interpreting UID sets in the IMAP server, potentially causing a
SEARCH not to return all matching messages, and potentially not EXPUNGE-ing
(removing) all messages marked as deleted. (found by new tests)
Fix bug with assigning UIDs when renaming Inbox. Renaming Inbox is a special
case and its special handling wasn't renumbering UIDs of moved messages,
potentially leading to errors on later operations on the destination mailbox.
Command "mox verifydata" has been updated to check for this inconsistency. Two
new unlisted commands are introduced that can correct these inconsistencies:
"mox reassignuids" and "mox fixuidmeta". This will only affect you if you have
renamed your Inbox, not a common operation. (found by new tests)
Fix importing messages with a date with a timezone >= 24 hours. (found while
importing old email archives)

New features:

Implement user-defined flags/keywords in the IMAP server. Previously, only
the standard keywords like \Seen, \Draft, etc were supported.
Configurable routes for outgoing email, through configurable transports. The
default transport is regular direct delivery to MX hosts. Other transport
options: 1. delivery through another mail server with SMTP (with various
TLS/STARTTLS/authentication options), to a relay/smarthost or 3rd party
account; 2. direct delivery but making connections through a SOCKS proxy.
Routes can be configured for accounts, domains, or globally. A matching route
selects the transport. If no route matches (the default), the default transport
(direct delivery) is selected. Routes are evaluated for each delivery attempt.
They are typically matched on destination address, and can also start matching
after a configurable number of failed delivery attempts (with another
transport). (#36 by dmikushin, and earlier discussions on HN)
Support more authentication mechanisms in sendmail: SCRAM-SHA-256,
SCRAM-SHA-1, CRAM-MD5.

Thanks to all who provided feedback, bug reports, patches. You've helped mox
become a better mail server!

To download, see https://github.com/mjl-/mox#download

mox - v0.0.4

Published by mjl- over 1 year ago

Update instructions: No special steps required other than updating the binary.

Changes:

Bugfix: When DKIM-signing messages, take the Date and Message-Id
header into account if we added them just before. If you submit a message
without these headers, they are added automatically, but the signature was
calculated over the message as if those headers were not added. The receiving
side would evaluate such DKIM signature as invalid.
Bugfix: SMTP transactions with SMTP from/to addresses with double quotes
would fail.
Bugfix: Delivery of messages to addresses where the domain does not have an MX
record would fail. In these cases, e.g. where CNAME records need to be followed
or where we need to connect directly to the host, we were aborting immediately
after starting because of a reused canceled Go context.
Bugfix: Use MTASTSHTTPS.NonTLS from the config file for the MTASTS config,
not the similar field for Autoconfig. (#29)
Bugfix: If we encountered an error fetching an mta-sts policy as part of a
delivery attempt, we weren't trying to continue in strict mode, which is safe to
do in this case.
Bugfix: Dropping a message from the queue (with "mox queue drop " or
through the admin web interface) would be removed from the database, but left in
the file system. If you still have old message files in data/queue/, they can be
removed manually.
Update to latest bstore database library and add a new index on the Message
database table, for fast indexed lookups of DKIM reputation for incoming
messages.
Add "mox backup" and "mox verifydata" subcommands. Backup makes a consistent
snapshot of the databases, message files, and the other files in the data
directory. Simply copying the DB files while they are in use will lead to
corrupted files sooner or later, so a consistent snapshot is important. The
message files are hardlinked when possible, so a snapshot is fast and disk
space consumption is limited to the database files. The verifydata subcommand
verifies the database files and messages, useful to verify backups and
restores. See documentation for these new commands for details.
Add support for websockets to the reverse proxy webserver. (#25)
After "logout" of an IMAP connection, close the connection. (#30)
Open pre existing TLS certificates/keys as the root user, not as the
unprivileged user. Makes it easier to use these files when set up by other
tools. (#30)
Various smaller technical and user-experience improvements.

Thanks for the feedback, issues and PR's you sent in.
naturalethic, 154pinkchairs, cuu508, inigoserna, kou029w, belst, and those I
forgot. Keep it coming!

To download, see https://github.com/mjl-/mox#download

mox - v0.0.3

Published by mjl- over 1 year ago

Update instructions: No special steps required other than updating the binary.

Important bug fixes:

When dkim-signing submitted messages, use the domain from the "message from
header" instead of "smtp mail from". If you had an account with multiple
addresses in different domains, then with some mail clients the DKIM signature
on outgoing messages could be for the wrong domain.

Changes:

Add "mox localserve", a subcommand for running mox locally for email-related
testing/developing. Localserve creates a config for listening on localhost
for imap/smtp/submission/http, on port numbers 1000 + the common service port
numbers. All incoming email is accepted (if checks pass), and a few pattern in
localparts are recognized and result in delivery errors.
Implement configurable (rate) limits on outgoing messages, per account. By
default an account can send 1000 messages per day, to max 200 first-time
receivers. The goal is to limit damage spammers can do with compromised
accounts.
In domains.conf, for an account, the Destinations mapping will now always use
full email addresses, no longer localparts relative to the Domain configured
for the account. The old form with just a localpart is still accepted. When
writing domains.conf through the cli commands or admin web pages, the
destinations will automatically be written with full email addresses. In the
future, support for the localpart-only form will be removed.
Implement a catchall address for a domain, an address without localpart (i.e.
starting with "@"). If an incoming message does not match a
configured address, it is delivered to the account configured that has the
catchall address configured.
ACME TLS certificate validation can also be done with method http-01
(previously only with tls-alpn-01).
If you run mox behind a NAT, you can now specify "IPsNATed: true" in the SMTP
listener to skip a few DNS checks that previously would always fail due to the
IPs being NATed.
Add subcommand "mox dkim sign", useful for testing signatures.
For a "WebRedirect" rule, don't match a request if the destination URL would
result in the same URL, causing a redirect. By not matching, the next
configuration rules are evaluated for a match.
Various smaller technical and user-experience improvements.

Thanks to everyone who sent in feedback (e.g. by creating an issue on github).
It's much appreciated and essential for improving mox!

mox - v0.0.2

Published by mjl- over 1 year ago

Update instructions

Follow these instructions if you are updating an existing installation.

Mox now starts as root to bind to network sockets, then starts a less
privileged process to run mox. Update the mox systemd service file and restart,
as root:

# update mox binary to v0.0.2
./mox config printservice >mox.service
systemctl daemon-reload
journalctl -f -u mox &
systemctl restart mox
# If you configured backups, keep in mind files are now owned by
# mox:root, your backup tools need access to the files.

Changes

Start up as root, bind network sockets, then drop privileges to a process as
user mox to do the actual work. Makes it easier to run mox on BSDs, and paves
the way for future privilege separation work.
Add a webserver, with support for reverse proxying, serving static files,
configurable redirects. With ACME of course. Takes away need to configure mox
behind an existing webserver/reverse proxy, greatly simplifying
configuration/operation.
Add docker images and script to build images.
Make it possible to run mox behind an existing webserver/reverse proxy too.
Also add quickstart flag to make it easier to generate a config file for this
case.
Quickstart improvements: Add flag to generate a config for a specified host
name (instead of guessing). If we found public IPs, check them with popular DNS
blocklists and warn user if they are listed.
Add encrypted cid to SMTP error response lines, for easier debugging.
Fix importing of junk messages that the junk filter cannot parse.
Fix bug verifying SPF. For IP lookups (e.g. for mechanism "a"), we would only
look up with the same address family as the incoming connection. If there was no
record, we counted a void lookup. This could lead to incorrect SPF failures, we
reached the void lookup limit when we shouldn't. If the record exists for
another address family (a vs aaaa), we shouldn't count a void lookup.
Fix reputation learning of messages that were initially delivered to the
Rejects mailbox and then moved out. If you marked them as junk or nonjunk (e.g.
by moving to Archive or Junk), we should use the message for reputation
calculation of new incoming messages, but we weren't.
Lots of smaller improvements, bug fixes.

Thanks for reporting issues: mteege, hismailbulut, belst, idnovic, Jens,
the-solipsist, pexarkh, sherief, gedw99, agucova, lormayna, and everyone I
forgot. Thanks for all your feedback, discussions, suggestions! It shaped these
changes, and changes to come.

Download

See https://github.com/mjl-/mox#download

Package Rankings

Top 4.94% on Proxy.golang.org

Related Projects

mailgun-go

Go library for sending mail with the Mailgun API.

28 Feb 2014 695

MailHog

Web and API based SMTP testing

16 Apr 2014 13,950