Just wanted to ask, is there a way to specify that Maddy would serve the right cert based on SNI when using TLS?

RCPT TO:<postmaster>

Should be accepted as valid.
There are several places around the code that assume user@domain in recipients addresses, need to check them and probably move some code around.

See https://tools.ietf.org/html/rfc5321#page-14 and https://tools.ietf.org/html/rfc5321#section-4.5.1.

Steps to reproduce:

1. create user on a fresh, newly-created postgresql db
2. start foxcpp/maddy/master using said db
3. send a email to it with a subject
4. open thunderbird and observe no subject being displayed

Perhaps something like that:

smtp ... {
  aliases path_to_file_with_mappings
}

Format same as postfix's text representation (http://www.postfix.org/aliases.5.html). Except probably we will leave |command, /file, :include support out for now.

After emersion/go-imap#232 gets resolved we want to update maddy to log IP addresses for failed login attempts.

We already do that for SMTP so no changes required here.

Also, write an article (perhaps on wiki pages since it is not related directly to maddy) about how to configure fail2ban for it. Before that, we need to do #26.

We can have other means to submit messages for processing: IMAP submission extensions, JMAP, etc.

This will also simplify the code of the SMTP endpoint module a lot.
One module - one job. The job of the SMTP endpoint module is to accept messages using SMTP and push them to the pipeline. The job of the pipeline module is to connect modules to each other.

Hi,

the current code does not compile. Most certainly because of an updated underlying library.

If I understand what is going on, it seems the go-imap-proxy code tries to use (go-imap-proxy/mailbox.go:16) the m.u.c.Mailbox function as a struct. So go-imap-proxy seems to be the culprit here.

That is the first error. Did not look at the others though.

Are you still interested by that code ? (ie: would you be interested in some help ?)

When I try to compile from a fresh clone I get the following errors:

# github.com/emersion/go-imap-proxy
../../go-imap-proxy/mailbox.go:16: m.u.c.Mailbox.Name undefined (type func() *imap.MailboxStatus has no field or method Name)
../../go-imap-proxy/mailbox.go:33: m.u.c.Mailbox.Name undefined (type func() *imap.MailboxStatus has no field or method Name)
../../go-imap-proxy/mailbox.go:34: invalid indirect of m.u.c.Mailbox (type func() *imap.MailboxStatus)
../../go-imap-proxy/mailbox.go:38: cannot use items (type []string) as type []imap.StatusItem in argument to m.u.c.Status
../../go-imap-proxy/mailbox.go:68: cannot use items (type []string) as type []imap.FetchItem in argument to m.u.c.UidFetch
../../go-imap-proxy/mailbox.go:70: cannot use items (type []string) as type []imap.FetchItem in argument to m.u.c.Fetch
../../go-imap-proxy/mailbox.go:103: cannot use string(operation) (type string) as type imap.StoreItem in argument to m.u.c.UidStore
../../go-imap-proxy/mailbox.go:105: cannot use string(operation) (type string) as type imap.StoreItem in argument to m.u.c.Store
../../go-imap-proxy/user.go:34: cannot use mailbox literal (type *mailbox) as type backend.Mailbox in append:
	*mailbox does not implement backend.Mailbox (wrong type for ListMessages method)
		have ListMessages(bool, *imap.SeqSet, []string, chan<- *imap.Message) error
		want ListMessages(bool, *imap.SeqSet, []imap.FetchItem, chan<- *imap.Message) error
../../go-imap-proxy/user.go:54: impossible type assertion:
	*mailbox does not implement backend.Mailbox (wrong type for ListMessages method)
		have ListMessages(bool, *imap.SeqSet, []string, chan<- *imap.Message) error
		want ListMessages(bool, *imap.SeqSet, []imap.FetchItem, chan<- *imap.Message) error
../../go-imap-proxy/user.go:54: too many errors
# github.com/emersion/go-pgpmail/imap
../../go-pgpmail/imap/mailbox.go:26: cannot use item (type string) as type imap.FetchItem in argument to imap.ParseBodySectionName
../../go-pgpmail/imap/mailbox.go:32: cannot use items (type []string) as type []imap.FetchItem in argument to m.Mailbox.ListMessages
../../go-pgpmail/imap/mailbox.go:64: cannot use items (type []string) as type []imap.FetchItem in argument to m.Mailbox.ListMessages
../../go-pgpmail/imap/user.go:24: cannot use u.getMailbox(m) (type *mailbox) as type backend.Mailbox in assignment:
	*mailbox does not implement backend.Mailbox (wrong type for ListMessages method)
		have ListMessages(bool, *imap.SeqSet, []string, chan<- *imap.Message) error
		want ListMessages(bool, *imap.SeqSet, []imap.FetchItem, chan<- *imap.Message) error
../../go-pgpmail/imap/user.go:34: cannot use u.getMailbox(m) (type *mailbox) as type backend.Mailbox in return argument:
	*mailbox does not implement backend.Mailbox (wrong type for ListMessages method)
		have ListMessages(bool, *imap.SeqSet, []string, chan<- *imap.Message) error
		want ListMessages(bool, *imap.SeqSet, []imap.FetchItem, chan<- *imap.Message) error

Allow writing module definitions inside other module definitions, omitting names.

This feature would allow writing less verbose configuration files with fewer entities (so you don't have to look around a lot to understand what is happening). The key point is to require "named entities" only when they are shared between server components (modules).

For example, the following configuration:

hostname example.org

sql {
  driver sqlite3
  dsn /var/lib/maddy/all.db
}

smtp smtp://0.0.0.0:25 {
  pipeline incoming_smtp
}
smtp smtp://0.0.0.0:587 {
  pipeline outgoing_smtp
}

# let's pretend that we have two modules for DKIM signing and verification, dkim_sign and dkim_verify
dkim_sign {
  public_key pubkey_file_path
  private_key privkey_file_path
}

pipeline incoming_smtp {
    filter dkim_verify
    filter milter /var/run/spamassasin.sock
    deliver sql 
}

pipeline outgoing_smtp {
    filter dkim_sign
    deliver incoming_smtp local_only  # let's pretend that pipeline implements DeliveryTarget
    deliver queue remote_only
}

queue {
  max_tries 1
  target remote
}

could be written like that:

hostname example.org

sql {
  driver sqlite3
  dsn /var/lib/maddy/msgs.db
}

smtp smtp://0.0.0.0:25 {
  pipeline {
    filter dkim_verify require # refering to singleton module 'dkim_verify'
    filter milter /var/run/spamassasin.sock # refering to singleton module 'milter'
    deliver sql # refering to 'sql' module instance
  }
}

smtp smtp://0.0.0.0:587 {
  auth pam # refering to singleton module 'pam'
  pipeline {
    filter dkim_sign { # inline-defined instance of module dkim_sign
      public_key pubkey_file_path
      private_key privkey_file_path
    }
    deliver queue { # inline-defined instance of module queue
      max_tries 1
      target remote # refering to singleton module 'remote' here
    }
    deliver smtp
  }
}

Generic syntax change is from

{
  operation module_instance_name module_options
}

to

{
  operation module_name {
    module_config_directives
  }
}

Here "inner" pipeline instance gets some artificial name like parentname_N so it can use it as a key to store persistent data.
!!! Reordering of configuration directives here may change "artificial names". Can we do something about it?

!!! We need a different way to specify per-call options
Perhaps

{
  operation module_name {
    module_config_directives
    opts options_go_here
  }
}

Note: a { dir } b c is parsed as

a {
  dir
}
b c

But I'm not very sure about the whole idea. Any thougnts?

https://jmap.io/

Before that, we need to get DKIM verification working.

• Reject/mark incoming messages that violate DMARC policy of the source server
• Report generation

Just wanted to ask, is every component compatible with internationalized domain names? I'd really like to leave Postfix because it does database connections LATIN-1 only which is rather unusable.

I guess it might make sense to implement server-side encryption for mail storage to protect past messages in case of database compromise. PGP is a ready-to-use public key[1] infrastructure[2], though we might want to use keys generated by the server for each user to avoid all trust issues.

The symmetric key used to protect private key can be derived from user password using the computation-intensive hash algorithm such as bcrypt.

[1] Public key encryption is necessary to ensure that we can deliver new messages without knowledge of the decryption key.
[2] Tho I think we might want to pick something simpler for this purpose, ideas?

There are also some problems that need to be solved:

• Can it be implemented as a storage-agnostic overlay? We need to design something to store and retrieve generated keys. Users of go-imap-sql may expect everything to be in a single database.
• We should not try to decrypt messages that are already PGP-encrypted by. Underlying storage may provide some indication of whether the message is encrypted so it boils down to the first problem.
• We should try to encrypt the meta-data too, this includes envelope and body structure.

Originally posted by @foxcpp in #10 (comment)

where can I find the test mail I sent?
how to set up an account?

Probably maildir

Rationale and possible alternative: #58 (comment)

Guidance for contributors

Make it possible to specify multiple cert-key pairs in the tls directive:

tls <cert> <key> <cert> <key>

Load them all then call tls.Config.BuildNameToCertificate.

The relevant code is in internal/config/tls_server.go in the readTLSBlock function.
Documentation to update: docs/man/maddy-imap.5.scd, docs/man/maddy-smtp.5.scd and probably docs/man/maddy-tls.5.scd.

Original post

Perhaps something like that:

tls perdomain {
  <domain> <cert_file> <key_file>
  _default cert_file key_file
}

Possible using GetCertificate or GetConfigForClient callbacks in tls.Config.

It would be nice to support DANE in addition to MTA-STS to increase interoperability.

Again, there are two sides in DANE support:

• Check DANE for remote MTAs
• Check if we have DANE record and reject unencrypted MTA connections.

https://tools.ietf.org/html/rfc7672

This allows for multiple workers and makes sure no message gets lost.

Look-up MTA-STS policy for remote MTAs before connecting to them, enforce TLS and cache policy.

We want to run certain checks (#41) on the envelope before the message body is sent to us to not waste traffic. The current pipeline implementation doesn't allow to run steps before the message body is received.

Also, SMTP doesn't allow us to "abort" transaction once we sent 354 intermediate reply to the DATA command.

I see two possible solutions for this problem:

Define a special form of checks to be run before the body is sent

Defined somehow like that:

smtp ... {
  envelope_checks {
    verify_source_hostname
    verify_source_mx
    ...
  }
}

The checks are executed when RCPT/MAIL FROM command received and use the following interface:

type EnvelopeCheck interface {
  CheckFrom(ctx *DeliveryContext, from string) error
  CheckRcpt(ctx *DeliveryContext, rcpt string) error
}

Corresponding method is called for each recipient and for source.

Advantages

• Ability to report failures per recipient as a status for RCPT TO command.

Disadvantages

• See the Advantages of the next method.

Start pipeline before the body is received

Once the client issues the DATA command, start executing pipeline steps. If any of the steps need the message body - send 354 reply and pass body reader to the pipeline step (with some tricks it can be made fully transparent for step implementation).

Advantages

• Uniform interface for all checks
• Conditional checks using pipeline dispatching mechanisms (such as destination directive)

Disadvantages

• Requires changes to go-smtp (pre-body callback for DATA command)
• It is not possible to report errors for individual recipients or even MAIL FROM command.
  Quoting RFC 5321:

  Using a "550 mailbox not found" (or
  equivalent) reply code after the data are accepted makes it difficult
  or impossible for the client to determine which recipients failed.

Question regarding standards conformance/interoperability: Are we allowed to return arbitrary failures in the DATA command before the client sends us the body?

This is necessary when upstream rejects messages.

Configuring MUAs is made much easier with support for automatic configuration. Mozilla's de-facto standard for this is quite widely used: https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration

I'm currently manually maintaining one of these at https://autoconfig.ur.gs/.well-known/autoconfig/mail/config-v1.1.xml

There are DNS-based standards too, but I don't think they're as widely used.

We'll need a HTTP(S) listener for automatic TLS, activesync, webmail, etc, support too, so I don't think it's a big deal to add one.

Following conditions should be met for verification to pass:

• rDNS domain of source server's IP should be equal to the hostname value presented in EHLO/HELO command
• Hostname domain presented in EHLO/HELO command should resolve to source server's IP address
• Domain in MAIL FROM should have MX record pointing to the source server

These checks are built on the assumption that we have a DNSSEC-enabled resolver and the source server does have DNSSEC enabled.

Generally, we should try to not introduce a lot of C dependencies (#20). We can implement basic support for local authentication on Unix systems using /etc/passwd and /etc/shadow databases.

https://tools.ietf.org/html/draft-ietf-dmarc-arc-protocol-23

scdoc is probably the way to go.

Hi,
I'd like to use this project for my ajaj ajaj

comment je jajjjaj

Things like DKIM, MTA-STS are tightly coupled with DNS and HTTPS.

Should we become tiny DNS/HTTPS servers too?

Generic conditions syntax

value operator value

Value can be either variable prefixed with $ or

Variable can be all fields from DeliveryContext and also any key from DeliveryContext.Ctx. If the variable doesn't exist - its value assumed to be an empty string.

The operator is one of the following:

• = Variable value must be exactly equal to the value in condition.
• =* Similar to the = but is case-insensitive.
• != Inverted =.
• !=* Inverted =*.
• ~ Value is a regexp that should match variable value.
• ~* Case-insensitive version of ~.
• !~ Inverted ~.
• !~* Inverted ~*

Inspired by the nginx's if. I highlighted operations currently implemented for the match block.

Variable value is converted into string before comparsion.

Examples

$rcpt_domain = "emersion.fr"
$src_ip ~ "^192\.168\..+"

match => if

Rename match block into if and make it use conditions syntax described above.

Examples

if $rcpt_domain = "emersion.fr" {
  proxy smtp://0.0.0.0
}

My initial idea is to inspect Received fields in the headers of the messages.
If it contains value mentioning the recipient address we are handling and our hostname - we bounce the message because it was processed by our server already.

It is important to check hostname because a single recipient may be processed by multiple servers and each will include its own Received field.
Another consideration to take into account: We should skip Received fields we can't process as some MTAs include information using non-standard formats (qmail, for example). Actually, SMTP RFC requires MTA to not fail delivery if an invalid Received field is present in the header.

Postfix relies on Delivered-To (non-standard AFAIK) fields for this purpose. We might want to include this field when the message leaves the pipeline. Currently, it is included only when the message is stored in an IMAP mailbox.

Multiple copies of the message body (see #87)

Pipeline and filters code copies messages multiple times, resulting in memory usage spikes when processing messages with attachment

One of the possible solutions: Add type-assertions where necessary and reuse existing copy instead of ioutil.ReadAll'ing it.
Type-assert on an interface with Bytes() method [1] when byte slice is needed (go-imap-sql with in-DB storage, e.g. SQLite)
Type-assert on io.ReadSeeker when the message body needs to be sent to multiple locations.

[1] bytes.Reader does not allow to obtain underlying byte slice, we might need to wrap it to add Bytes() method.

Entire message body rewriting when only a small change is required (unnecessary copy) (see #87)

Some filters only read the message and prepend headers, but the current design forces them to rewrite and copy around the entire message body

One of the possible solutions: Special-case prepend operation and optimize it.
One of the possible solutions: Store header in parsed form and allow filters to manipulate it. Serialize when the message exits pipeline (e.g. gets delivered to storage or milter[2]).

[2] Tho it seems like the milter protocol allows us to avoid reserialization of headers.

In-memory message buffering

We need to submit message content to multiple sources (including milters and other out-of-process filters). This basically forces us to buffer the entire message. Currently, we do this in memory. This leads to increased memory usage, especially when delivering messages with attachments.

One of the possible solutions: Allow to buffer messages on disk instead of RAM.
This variant might be incompatible with the second solution for the second problem.

See https://linux.die.net/man/8/local

Main Use-Case: Use different databases for different domains.

Proposed Solution: Introduce module perdomain that calls different auth. providers depending on the source domain and implement auth. provider interface itself.

auth perdomain {
  domain example.org {
    auth virtual { file /etc/maddy/example.org-users }
  }
  domain example.com {
    dont_strip_domain # by default perdomain removes domain from username before passing it to the "nested" provider
    auth virtual { file /etc/maddy/example.com-users }
  }

  # Both make authentication fail by default.
  unknown { # domain not matched by any directives.
    ...
  }
  nodomain { # no domain present in username
    ...
  }
}

mais on lanse comen XD

Currently it is basically line-oriented and thus will not handle the following cases as expected:

• Closing brace will be ignored, leading to incorrect parsing of following statements.

  name args { directive }
  

  name args {
  directive }
  

• This will fail with parse error.

  name args 
  {
    directive
  }
  

As an SMTP pipeline step:

proxy address0 address1 address2

Addresses use the same syntax as config definitions for endpoint modules. Remotes are tried in specified order, no more remotes will be tried after a successful transaction.

By default configuration should be read from /etc/maddy/Maddyfile.
All state (queues, databases) should be stored in $MADDYPATH with the default value of /var/lib/maddy.

The current implementation is just a placeholder using the current working directory for everything.

https://tools.ietf.org/html/rfc5228

I remember having a use-case where I wanted to give email accounts to all PAM users but use passwords from a separate database. And also people on IRC channels for dovecot and postfix often ask questions regarding weird mixes of authentication data sources. So I guess we should have a generic tool to handle such cases.

We create the multi module that implements authentication provider interface (so it can be used in SMTP, IMAP, etc):

multi instance_name {
  user pam
  user virtual { file /etc/maddy/userlist }
  pass virtual { file /etc/maddy/passwd }
}

user directive here refers to a module implementing the following interface:

type UserDB interface {
  HasUser(name string) bool
}

pass directive refers to an authentication provider.

If there is at least one user directive - then at least one "userdb" module should say that the user exists.
Then the user password is also checked against providers listed using pass directives, at least one provider should accept the password.

multi module itself also implements the UserDB interface, this allows mixing things together in more complicated use-cases to get the right results.

E.g. [email protected] and [email protected] are the same.
Messages not getting delivered to the local storage should not have recipient addresses case-folded tho.

Messages are delivered multiple times to local mailboxes.

Investigation revealed that sometimes maddy fails to acknowledge message delivery and instead just doesn't reponds with anything despite successfull delivery. Remote MTA timeouts and retries delivery later resulting in duplicates.

Server restart fixes this problem, but only for some time. It happens only for some messages (needs to be additionally checked) and always reproduced when corresponding SMTP session is replayed as-is.

Configuration

maddy with experimental queue implementation (824fbdc). Based on this commit from this repository: 2355cd9
Using go-imap-sql with PostgreSQL driver.

CC @NamedKitten

Global directive?

log <targets...>

log stderr

Write to stderr (default).

log file_name

Write to file.

log syslog

Send messages to local syslog daemon.

Can be combined:

log stderr syslog /var/log/maddy/maddy.log

username+anything@domain should be rewritten to username@domain during local message delivery.

--

Once sieve filtering has been added, is the possibility of having a [email protected] email that gets emails forwarded into spam folder a reality?
Also +1 on having ManageSieve support too.

Sorry about my English, it's my native language.

Originally posted by @NamedKitten in #53 (comment)

Just like Caddy.

#15 (comment)

--
Original post:

Let's say I configured IMAP endpoint as follows (where first line creates IMAP backend):

imap://127.0.0.1:1993 {
    sql sqlite3 maddy.db
}

Then I want to have SMTP endpoint that will deliver mail to same storage. How would I do this?

1. Assuming that backend also provides implementation of SMTP upstream (perhaps as separate object).

smtp://127.0.0.1:1025 {
	sql sqlite3 maddy.db
}

This approach creates another set of problems, because it now requires two separate backend/upstream objects to coordinate access to the same storage (think of IMAP unilateral updates).
Global variables? External IPC sockets? All this seems to be dirty solution.

What is we can create one "storage" object and associate it with multiple IMAP/SMTP endpoints?
This will transform "another set of problems" into just serialization of access to storage object. Which is easily solved by throwing some mutexes into it (or even without them, I haven't tested that but go-sqlmail backend object should be safe for concurrent use by multiple goroutines).

It also reduces resources usage (we will have only one SQLite "connection" page cache, for example)

Now I can imagine something like this:

backend sql arbitrary_name {
  driver sqlite3
  dsn maddy.db
}

imap://127.0.0.1:1993 {
  backend arbitrary_name 
  # of course this requires storage object to implement go-imap's Backend interface
}

smtp://127.0.0.1:1025 {
  backend arbitrary_name 
  # and also go-smtp's Backend here now.
}

What do you think?

This will allow to reuse a lot of components developed for PAM.

Notes

• CGo dependency: libpam
• libpam does have a callback-based interface, it needs to be used with extreme care to not introduce memory vulnerabilities.

Problems to solve

• maddy needs to access the shadow database (/etc/shadow) on configurations using the local database. Should we just require maddy to be running as root (or with CAP_DAC_READ_SEARCH) or somehow isolate code working with this file?

Some example code can be found here: https://stackoverflow.com/questions/10910193/how-to-authenticate-username-password-using-pam-w-o-root-privileges
Except probably we want to define our own PAM service instead of using "su".

There could be several filter backends. One of them could be https://github.com/emersion/go-milter