saltstack-formulas / postfix-formula Goto Github PK

View Code? Open in Web Editor NEW

25.0 43.0 131.0 438 KB

Home Page: http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html

License: Other

SaltStack 36.19% Ruby 36.99% JavaScript 16.31% Shell 4.94% Jinja 5.57%

postfix-formula's Introduction

postfix-formula

A SaltStack formula to install and configure Postfix mail server.

Table of Contents

General notes

See the full SaltStack Formulas installation and usage instructions.

If you are interested in writing or contributing to formulas, please pay attention to the Writing Formula Section.

If you want to use this formula, please pay attention to the FORMULA file and/or git tag, which contains the currently released version. This formula is versioned according to Semantic Versioning.

See Formula Versioning Section for more details.

Contributing to this repo

Commit message formatting is significant!!

Please see How to contribute for more details.

Available states

`postfix`

Installs and starts postfix SMTP server

`postfix.config`

Manages postfix main.cf and optionally the master.cf configuration file. Generates mappings.

`postfix.policyd-spf`

Installs and configures policyd-spf

`postfix.postgrey`

Installs and starts Postgrey service

`postfix.mysql`

Installs postfix mysql package ( Debian only)

`postfix.pcre`

Installs postfix pcre package ( Debian only)

`postfix.postsrsd`

Installs postfix postsrsd package

Testing

Linux testing is done with kitchen-salt.

Requirements

Ruby
Docker

$ gem install bundler
$ bundle install
$ bin/kitchen test [platform]

Where [platform] is the platform name defined in kitchen.yml, e.g. debian-9-2019-2-py3.

`bin/kitchen converge`

Creates the docker instance and runs the main state(s), ready for testing.

`bin/kitchen verify`

Runs the inspec tests on the actual instance.

`bin/kitchen destroy`

Removes the docker instance.

`bin/kitchen test`

Runs all of the stages above in one go: i.e. destroy + converge + verify + destroy.

`bin/kitchen login`

Gives you SSH access to the instance for manual testing.

postfix-formula's People

Contributors

Stargazers

Watchers

Forkers

trexglobal libermentix crucialcloudhosting q1x skylerberg polliard keithhub openenvi-formulas egarbi packetloop rhertzog doeners kkoppel sys4 ugns omencat jdsieci uoitsci fei-lithicon martinhoefling imran1008 borutmrak zajk alkivi-sas timjones jbeigh tuksik julienba fastmarkets-devops-archive octavhendra greytip evasdk bt-dschleich kerkhofftechnologies feiming aldor007 sunpowered-salt-formulas move-sysadmin amzani suaustralia infra-github rapideinternet dwbn getprodigy simonwgill wazinit rodeofx leifcr titilambert nilsglow bradthurber elfayafifarah hwhesselink rocket-labs-sysadmins technoronin daks aquto dxclabs nypublicradio gerhro virtualkitchen scjcoder jkur nginxinc sqrteh jeroen92 neuhalje kossmac 1and3cr01x netmanagers gillarkod koebi001 kgeorgiy marty30 jason-klein 0xf10e ghtyrant bawuenet jfrederickson vcsi captainstatic cdawg908 polarfox blade2005 mthibaut kti-nicole phliphop cliffordhc poil perceptyx nixm0nk3y totalserversolutions mgomersbach alxwr fhe3 lysiszegerman getsurreal roskens keke8273 cdo-acd

postfix-formula's Issues

Jinja variable odict_values object has no element 0 python2 to python3 error

Your setup

Versions reports (master & minion)

salt3004.1

Pillar / config used

Our postfix.sls update file

#!jinja|yaml|gpg
postfix:
config:
relayhost: '[email-smtp.eu-west-2.amazonaws.com]:587'
smtp_sasl_auth_enable: 'yes'
smtp_sasl_security_options: noanonymous
smtp_sasl_password_maps: hash:/etc/postfix/sasl_passwd
sender_canonical_maps: hash:/etc/postfix/canonical
smtp_use_tls: 'yes'
smtp_tls_security_level: encrypt
smtp_tls_note_starttls_offer: 'yes'
smtp_tls_CAfile: /etc/pki/tls/certs/ca-bundle.crt
mapping:
smtp_sasl_password_maps:
- '[email-smtp.eu-west-2.amazonaws.com]:25': xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
- '[email-smtp.eu-west-2.amazonaws.com]:465': xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
- '[email-smtp.eu-west-2.amazonaws.com]:587': xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Bug details

Describe the bug

In Python 2, calling the "values" method of a dictionary returned a list that then could be accessed using an index, but in Python 3 the "values" method returns a iterable object that can't be indexed.

Steps to reproduce the bug

Salt-3004.1 server
Salt-minion linux 7.9

github postfix formulas

Expected behaviour

You will get the same output as I did

      ID: postfix_smtp_sasl_password_maps
Function: file.managed
    Name: /etc/postfix/sasl_passwd
  Result: False
 Comment: An exception occurred in this state: Traceback (most recent call last):
            File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
              output = template.render(**decoded_context)
            File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1090, in render
              self.environment.handle_exception()
            File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 832, in handle_exception
              reraise(*rewrite_traceback_stack(source=source))
            File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 28, in reraise
              raise value.with_traceback(tb)
            File "<template>", line 10, in top-level template code
            File "/usr/lib/python3.6/site-packages/jinja2/sandbox.py", line 462, in call
              return __context.call(__obj, *args, **kwargs)
            File "/usr/lib/python3.6/site-packages/jinja2/runtime.py", line 679, in _invoke
              rv = self._func(*arguments)
            File "<template>", line 2, in template
            File "/usr/lib/python3.6/site-packages/jinja2/tests.py", line 158, in test_iterable
              iter(value)
          jinja2.exceptions.UndefinedError: odict_values object has no element 0

          During handling of the above exception, another exception occurred:

          Traceback (most recent call last):
            File "/usr/lib/python3.6/site-packages/salt/state.py", line 2180, in call
              *cdata["args"], **cdata["kwargs"]
            File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
              return self.loader.run(run_func, *args, **kwargs)
            File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
              return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
            File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
              return callable(*args, **kwargs)
            File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
              return _func_or_method(*args, **kwargs)
            File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in wrapper
              return f(*args, **kwargs)
            File "/usr/lib/python3.6/site-packages/salt/states/file.py", line 3026, in managed
              **kwargs
            File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
              return self.loader.run(run_func, *args, **kwargs)
            File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
              return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
            File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
              return callable(*args, **kwargs)
            File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
              return _func_or_method(*args, **kwargs)
            File "/usr/lib/python3.6/site-packages/salt/modules/file.py", line 5409, in check_managed_changes
              **kwargs
            File "/usr/lib/python3.6/site-packages/salt/modules/file.py", line 4661, in get_managed
              **kwargs
            File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 261, in render_tmpl
              output = render_str(tmplstr, context, tmplpath)
            File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 509, in render_jinja_tmpl
              raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
          salt.exceptions.SaltRenderError: Jinja variable odict_values object has no element 0

Attempts to fix the bug

No attempts to fix yet

Additional context

N/A

[BUG] smtpd_sasl_auth_enable: no not working

Bug details

Describe the bug

If you want to disable sasl auth for example on a mx only server, this will result in a not working configuration.

If smtpd_sasl_auth_enable: no is set in pillar, this will result in smtpd_sasl_auth_enable = False in main.cf

postfix/smtpd[333]: fatal: bad boolean configuration: smtpd_sasl_auth_enable = False

Steps to reproduce the bug

Pillar:

postfix:
  config:
    smtpd_sasl_auth_enable: no

Expected behaviour

It should write "smtpd_sasl_auth_enable = no" in the main.cf config file.

/usr/sbin/postconf: warning: /etc/postfix/main.cf: unused parameter: compatibility_level=2

Hello,

on Debian 7.11 (oldstable) with Postfix 2.9.6-2, this formula sets this parameter in main.cf

 compatibility_level = 2

and when restarting the service, I got this warning

/usr/sbin/postconf: warning: /etc/postfix/main.cf: unused parameter: compatibility_level=2

grains['fqdn'] lookup in pillar

Please do not promote this, using grain lookup in pillar is BAD idea. Pillar is compiled on master, cashed on minion, so your pillar will feature master's 'fqdn' not the the targeted server. (maybe the same in your test case)

[BUG] resources IDs are not unique and collide with other formulas

Your setup

Formula commit hash / release tag

Tested using commit 19617a0 (tag v1.2.2)

Versions reports (master & minion)

# salt --versions-report
Salt Version:
          Salt: 3002.5
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: unknown
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.5
     gitpython: 2.1.11
        Jinja2: 2.10
       libgit2: 0.27.7
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: 0.27.4
        Python: 3.7.7 (default, Apr  1 2020, 13:48:52)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.1.2
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2
 
System Versions:
          dist: debian testing bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-1-amd64
        system: Linux
       version: Debian GNU/Linux testing bullseye

Pillar / config used

No config needed, just using the followin top.sls

base:
  mail:
    - mysql.client
    - postfix
    - postfix.config
    - postfix.mysql
    - postfix.pcre

Bug details

Describe the bug

There are conflicting resource IDs between the mysql.client and postfix.mysql state files.

Steps to reproduce the bug

~# salt 'mail' state.apply
ERROR: Minions returned with non-zero exit code
mail:
    Data failed to compile:
----------
    Detected conflicting IDs, SLS IDs need to be globally unique.
    The conflicting ID is 'mysql' and is found in SLS 'base:mysql.client' and SLS 'base:postfix.mysql'

Expected behaviour

ID's should not conflict.

Attempts to fix the bug

See PR #120

Additional context

postfix service on specific port fails

Describe the bug

I get a SaltRenderError when I try to use this forumla to make postfix listen on localhost:10025.

Setup

I'm using a github fork of this repository. It's on version 077a6a6.

Steps to reproduce the bug

I have a pillar that reads (in part, snipped for brevity):

 16 postfix:
 17   manage_master_config: True
 18   master_config:
 19     enable_dovecot: True
 20     enable_submission: True
 21     services:
 22         # Make postfix listen on a port for emails to return from amavisd.
 23         localhost:10025:
 24           command: smtpd
 25           type: inet
 26           private: n
 27           args:
 28              - 'content_filter='
 29              - 'smtpd_delay_reject=no'

(Edit 2019-10-06, fixed example pillar to show "localhost:10025" instead of "localhost_10025", as the former actually causes the SaltRenderError, the later is what I had committed to my pillar to avoid the exception being thrown)

When I run salt, I see an error.

$ sudo salt 'mcconnell*' --state-output=changes state.apply postfix.config  saltenv=
dev pillarenv=dev test=True
mcconnell.lan:
  Name: postfix - Function: pkg.installed - Result: Clean Started: - 22:26:29.418995 Duration: 52.299 
ms
  Name: /etc/postfix/main.cf - Function: file.managed - Result: Clean Started: - 22:26:29.476628 Durat
ion: 95.722 ms
----------
          ID: /etc/postfix/master.cf
    Function: file.managed
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1933, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1951, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/states/file.py", line 2769, in managed
                  **kwargs
                File "/usr/lib/python2.7/dist-packages/salt/modules/file.py", line 4824, in check_mana
ged_changes
                  **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/modules/file.py", line 4265, in get_manage
d
                  **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 169, in render_t
mpl
                  output = render_str(tmplstr, context, tmplpath)
                File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 406, in render_j
inja_tmpl
                  buf=tmplstr)
              SaltRenderError: Jinja variable 'unicode object' has no attribute 'get': "<Template memory:7f00b17cded0>", "0"
     Started: 22:26:29.572605
    Duration: 133.425 ms
     Changes:   
----------
…

Expected behaviour

I would expect salt to write out a /etc/postfix/master.cf file that includes an entry for "localhost:10025", which has the arguments listed above (such as content_filter, smtpd_delay_reject, etc). I would expect salt to restart postfix, and then postfix to be listening on port 10025.

Versions report

$ salt --versions-report
Salt Version:
           Salt: 2019.2.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1

System Versions:
           dist: debian 9.11
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-9-amd64
         system: Linux
        version: debian 9.11

Additional context

I've done some investigation. I suspect in postfix/files/master.cf, in the "Handle custom services" section at the end, the "pillar.get" on 'postfix:master_config:services:%s' is causing a lookup of 'postfix:master_config:services:localhost:10025', which is not a key that exists in my pillar.

I don't know how to define a pillar that will result in the formula writing out a postfix service entry for "localhost:10025", and will also satisfy that pillar.get lookup.

master.cf does not support custom arguments with custom services

I made a change to postfix/files/master.cf. I don't know how to make a pull request here.

I replaced lines 139 - 149 of the current version with the below. It allows me to setup amivisd on port 10025 with all the arguments that I need.

{%-   if wrap %}
{{      parameter_str | wordwrap(width=wrap, break_long_words=False, wrapstring='\n%s  ' | format(comment)) }}
{%-   else %}
{{      parameter_str }}
{%-   endif -%}
{%-   elif 'user' in service or 'argv' in postfix_master_services.defaults[service_name] -%}
{%-      set parameter_str = "%s  user=%s argv=%s %s" | format(comment,
                                            service_param(service, service_name, 'user'),
                                            service_param(service, service_name, 'argv'),
                                            service_param(service, service_name, 'extras', '')) -%}
{%-   if wrap %}
{{      parameter_str | wordwrap(width=wrap, break_long_words=False, wrapstring='\n%s  ' | format(comment)) }}
{%-   else %}
{{      parameter_str }}
{%-   endif -%}
{%-   endif -%}
{%-   if service.args is not none -%}
{%-     for option in service.get('args', postfix_master_services.defaults[
                                            service_name].get('args', [])) -%}
{%-       if option.startswith('#') %}
{{ option }}
{%-       else %}
{{ comment }}  {{ option }}
{%-       endif %}
{%-     endfor %}
{%-   endif %}

Don't call 'newaliases' if the 'alias_maps' postfix setting has 'regexp' as its dictionary type

Here's an excerpt from my Pillar:

postfix:
  config:
    alias_maps: regexp:/etc/aliases
    alias_database: regexp:/etc/aliases

  aliases: |
    # Forward all local *nix users mail to our admins (via greedy regexp)
    /.+/    [email protected]
# ...

Running state.apply results in a rather cryptic error:

...
          ID: run-newaliases
    Function: cmd.wait
        Name: newaliases
      Result: False
     Comment: Command "newaliases" run
     Started: 08:29:38.848092
    Duration: 1018.418 ms
     Changes:
              ----------
              pid:
                  24312
              retcode:
                  1
              stderr:
                  postalias: fatal: unsupported dictionary type: regexp. Is the postfix-regexp package installed?
              stdout:
...

We shouldn't presume that everyone is using hash as their dictionary type, as 'newaliases' should not be called if the 'alias_maps' postfix setting has regexp as its dictionary type.

References:

http://www.postfix.org/postmap.1.html see description of the file_type argument
http://postfix.1071664.n5.nabble.com/first-timer-using-postmap-help-please-td14908.html

Add postfix mysql support

I would like to see postfix mysql support added to this formula,

More information:
http://www.rackspace.com/knowledge_center/article/mail-server-configuring-postfix-to-use-mysql

dovecot_deliver not correct/set for Gentoo

Currently the osmapping omits the dovecot_deliver for Gentoo.
This lives at /usr/libexec/dovecot/deliver

[BUG] Jinja variable 'default_database_type' is undefined local

Your setup

Formula commit hash / release tag

postfix-formula: 42afb34 refs/tags/v1.2.2

Versions reports (master & minion)

Salt Version:
           Salt: 3000.6
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Apr  2 2020, 13:16:51)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.9.2009 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1127.el7.x86_64
         system: Linux
        version: CentOS Linux 7.9.2009 Core

Pillar / config used

# vim: ft=yaml
---
postfix:
  manage_master_config: true

  enable_service: true
  reload_service: true

  config:
    smtpd_banner: $myhostname ESMTP $mail_name
    smtp_tls_CApath: /etc/ssl/certs
    biff: 'no'
    append_dot_mydomain: 'no'
    readme_directory: 'no'
    myhostname: localhost
    mydestination: localhost, localhost.localdomain
    relayhost: ''
    mynetworks: 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
    mailbox_size_limit: 0
    recipient_delimiter: +
    inet_interfaces: all
    inet_protocols: all

  mapping:

    smtp_sasl_password_maps:
      - email-smtp.us-east-1.amazonaws.com: AWS_USER:AWS_PASS

    sender_canonical_maps:
      - root: [email protected]
      - nagios: [email protected]

    relay_recipient_maps:
      - example.com: OK

Bug details

Describe the bug

I'm getting the following error when postfix.config is called

[root@minion1 etc]# salt-call state.apply roles.postfix -l info
[INFO    ] Loading fresh modules for state activity
[INFO    ] Fetching file from saltenv 'base', ** done ** 'postfix/config.sls'
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 169, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 404, in render_jinja_tmpl
    buf=tmplstr)
SaltRenderError: Jinja variable 'default_database_type' is undefined
[CRITICAL] Rendering SLS 'base:postfix.config' failed: Jinja variable 'default_database_type' is undefined
local:
    Data failed to compile:
----------
    Rendering SLS 'base:postfix.config' failed: Jinja variable 'default_database_type' is undefined

Steps to reproduce the bug

It looks like the config.sls needs code similar to line 44 in init.sls
{%- set default_database_type = salt['pillar.get']('postfix:config:default_database_type', 'hash') %}

pillar example

I believe line 29 in the pillar example should have a colon rather than =

smtp_tls_CApath = /etc/ssl/certs
smtp_tls_CApath: '/etc/ssl/certs'

aliases may be broken

I am trying to use this formula on CentOS 7. I have a pillar like this:

postfix:
  relayhost: relay.example.com
  inet_interfaces: localhost
  aliases: |
    # Basic system aliases -- these MUST be present.
    mailer-daemon:  postmaster
    postmaster:     root
    root: [email protected]

Due to the arbitrary parameter change (78f3a47) recently accepted, the content of pillar['postfix']['aliases'] is included in the main.cf file. Since this is not valid main.cf content, both postfix and run-newaliases fail.

Any solution to customise master.cf keeping facilities of this formula

Hi,

I'm successfully using this formula, but I extend it to modify master.cf to enable/change maildrop configuration (see http://www.postfix.org/MAILDROP_README.html) with something like this:

postfix_master:
  file.managed:
    - name: '/etc/postfix/master.cf'
    - source: salt://mail_srv/files/master.cf
    - require:
      - pkg: postfix

include:
  - postfix.config
  - postfix

extend:
  postfix:
    service:
      - watch:
        - file: postfix_master

It works well, but it have a major caveat: I can't use this formula 'facilities' like the parameter enable_submission: True because I override master.cf with mine.
It's logical but it complexifies management of different setups (with or without submission for example) as I need to manage several master.cf sources.

I wonder if there should have any possibility to manage master.cf but with the ability to customise specific lines?
Example I want to change maildrop line, my pillar looks something like

master_config:
  maildrop: unix  -       n       n       -       -       pipe
      flags=ODRhu user=vmail argv=/path/to/maildrop -d ${recipient}

and the formula manage master.cf but change the line for maildrop.

I have no idea if this possible (in Salt, in this formula) but it could be great :)

thanks for your job.

Factorize calls to postmap

Number of cmd.wait to process map files is starting to grow. I think it is time to write a macro for handling this in a more concise way.

Mapping features offered by the formula not sufficient to mirror postfix reqs

Turns out, the mapping features offered by the formula are not sufficient to mirror real-life postfix deployments.

This issue serves as a discussion point to consider how to improve the current situation. It would probably also make a lot of sense to have a chat on IRC and discuss our options...

(NB: When I say mapping this can be read as lookup tables as well)

Postfix offers basically three ways of using mapping tables:

Static, in-memory or network based lookup tables which are set in main.cf:
- transport_maps = static:relay_example.com:25
- sender_canonical_maps = tcp:127.0.0.1:10001
External files which can optionally be hashed and generally follow a newline separated "key: value" format.
transport_maps = hash:/etc/postfix/transports
Databases backends, e.g. ldap, mysql etc.
- main.cf config item points at an external configuration file:
  transport_maps = mysql:/etc/postfix/transports.cf
- external configuration file contains e.g. mysql query and connection parameters:
```
dbname = mail
hosts = db.example.com
user = dbuser
password = dbsecret
query = SELECT "relay_example.com:25";
```

Detailed descriptions of the various types can be found at http://www.postfix.org/DATABASE_README.html.

The formula currently has a handful of different ways of configuring maps:

Configured as a static or in-memory lookup table in main.cf:
virtual_uid_maps: static:5000
This is handled as a regular parameter in main.cf
External hash-table:
- Lookup table configured in main.cf:
  relay_recipient_maps: hash:/etc/postfix/relay_domains
- Content configured via the pillar:
```
 postfix:
  mapping:
    relay_recipient_maps:
       - example.com: OK
```
VirtualUser support introduced in 65f86e6 which adds MySQL support to the lookup tables for virtual_alias_maps, virtual_mailbox_domains and virtual_mailbox_maps configured in the vmail pillar as well as a regular hashed transport lookup table configured in the transport pillar. The resulting configuration file uses a syntax that is found under the heading "OBSOLETE QUERY INTERFACE" in http://www.postfix.org/mysql_table.5.html.
An additional change layered on top in f8514b3 that does not really add new functionality but just adds a new pillar named mysql and uses the data included in that pillar to build a mysql query using the newer syntax described in http://www.postfix.org/mysql_table.5.html.

There are multiple shortcomings of the current design.

Only a single parameter is supported per configuration item. Postfix has no trouble consulting multiple files per map item: e.g. transport_maps = hash:/etc/postfix/transport, mysql:/etc/postfix/transports.cf configured in main.cf would have postfix first try a lookup in the on-disk hashed file and if nothing found try a mysql query to a database. This is not supported by the formula.
The VirtualUser related commits should probably never have been merged as is in the first place.
The first commit adding VirtualUser support is written in a very specific way that is probably only usable for a single site as it encodes too many assumptions about the SQL database layout in the formula. Furthermore, the transport pillar should not have been added at all but instead the regular mapping pillar should have been used.
The second commit repeats the mistake of the first commit by encoding site-specifics db schema assumptions in the SQL query and thus the formula.
Furthermore, creating a new pillar entry to replicate the same functionality as already exists is kinda useless as well.
A generic mapping configuration is not provided.
- We'd need something that is able to do multiple entries per lookup table configuration item.
- We'd need to both support traditional hashed key:value files as well as lookup configuration files for database backends.

I am not sure however, how to do these changes without breaking backwards compatibility.
I am not really worried about the virtualuser changes to be honest as I consider them broken already: They are that limited that most likely nobody other than the original contributors are using them.
I am unclear however, how to best allow for multiple entries per lookup table without breaking existing configurations as shown in the pillar.example file:

Currently supported:

postfix:
  config:
    smtp_sasl_password_maps: hash:/etc/postfix/sasl_passwd
  mapping:
    smtp_sasl_password_maps:
      - smtp.example.com: myaccount:somepassword

Future version, which might break backwards compatibility in case we cannot convert internally using jinja:

postfix:
  config:
    smtp_sasl_password_maps:
      - hash:/etc/postfix/sasl_passwd
      - mysql:/etc/postfix/sasl_passwd.cf
  mapping:
    smtp_sasl_password_maps:
      /etc/postfix/sasl_passwd:
        - smtp.example.com: myaccount:somepassword
      /etc/postfix/sasl_passwd.cf:
        dbname = mail
        hosts = db.example.com
        user = dbuser
        password = dbsecret
        query = SELECT password FROM sasl_passwd WHERE user == '%s';

This would basically move the existing mapping entries one level down and add a dict keyed off the filename to separate these. Alternatively, instead of a dict we could use a list and key it off the position, mapping the main.cf entry with the mapping entry.
It might be possible to offer backwards compatibility for existing setups but I am not 100% sure.

Before I start off and actually do the work, I'd like some feedback about the viability. What do other committers think? Worth it? Not mergeable because it breaks backwards compat?

Add support for SPF

See https://help.ubuntu.com/community/Postfix/SPF
I'd be happy to submit a pull request for adding postfix-policyd-spf-python into the formula.
I think it will be hard to parameterize/make optional (the way pillar is used at the moment) as it requires handling master.cf and putting check_policy_service unix:private/policy-spf into main.cf in the correct place - but it seems sensible to activate SPF checks by default?

[salt-lint] Switch tabs to spaces in `postfix/files/mapping.j2`

Introducing salt-lint, got a violation triggered here:

postfix-formula/postfix/files/mapping.j2

Lines 11 to 13 in 561d094

 {{ key }}{% if colon %}:{% endif %} {{ value|join(", ") }} 

 {%- else -%} 

 {{ key }}{% if colon %}:{% endif %} {{ value }}

Examining postfix/files/mapping.j2 of type state
[203] Most files should not contain tabs
postfix/files/mapping.j2:11
{{ key }}{% if colon %}:{% endif %}     {{ value|join(", ") }}

[203] Most files should not contain tabs
postfix/files/mapping.j2:13
{{ key }}{% if colon %}:{% endif %}     {{ value }}

@0xf10e @alxwr Are the tabs absolutely necessary here or can they be changed to spaces?

virtual_alias_maps parameter is ignored.

I am trying to configure the virtual_alias_maps parameter against two maps in my pillar data. In #83, it was suggested something like this might be possible in the future:

postfix:
  config:
    virtual_alias_maps:
      - hash:/etc/postfix/virtual
      - pcre:/etc/postfix/virtual.pcre
  mapping:
    hash:/etc/postfix/virtual:
      root:
        - me
    pcre:/etc/postfix/virtual.pcre:
      - '/(\S+)_(devel|preprod|prod)@sub.example.com$/': '$(1)@$(2).sub.example.com'

But that isn't supported yet, so I was trying to do this instead:

postfix:
  config:
    virtual_alias_maps: $virtual_alias_1_maps $virtual_alias_2_maps
    virtual_alias_1_maps: hash:/etc/postfix/virtual
    virtual_alias_2_maps: pcre:/etc/postfix/virtual.pcre
  mapping:
    virtual_alias_1_maps:
      root:
        - me
    virtual_alias_2_maps:
      - '/(\S+)_(devel|preprod|prod)@sub.example.com$/': '$(1)@$(2).sub.example.com'

The two files /etc/postfix/virtual and /etc/postfix/virtual.pcre get created, however, no parameter entry is added for virtual_alias_maps!

myhostname parameter: is fqdn or myhostname pillar data used?

Hello,

I just discovered this formula which I try to use. I see in pillar.example there is a parameter called

config:
  myhostname: localhost

I thought it will be used to manage the 'myhostname' Postfix parameter, which seems to not be the case: in https://github.com/saltstack-formulas/postfix-formula/blob/master/postfix/files/main.cf#L74 I see

{{ set_parameter('myhostname', grains['fqdn']) }}

It this normal, did I miss something? (I'm quite new to salt)

Thanks.

[BUG][FEATURE] service management needs improvement

Your setup

Formula commit hash / release tag

c0485fb 26/10/2020

Versions reports (master & minion)

not relevant

Pillar / config used

neither

Bug details

Describe the bug

Different problems about service management in this formula

no state to make a systemctl reload postfix, it's only possible to make a restart, see https://github.com/saltstack-formulas/postfix-formula/blob/master/postfix/init.sls#L36
two states in init.sls to manage the postfix service, one here https://github.com/saltstack-formulas/postfix-formula/blob/master/postfix/init.sls#L25 another https://github.com/saltstack-formulas/postfix-formula/blob/master/postfix/init.sls#L36
the state L25 uses a reload parameter which, according to doc and also salt code, does not exist

Steps to reproduce the bug

Expected behaviour

I want to be able to make a reload not a restart of the service. It's useful for example when changing a file for mynetworks parameter. A restart introduces a small service interruption when it's not necessary.

Attempts to fix the bug

None

Additional context

Does this formula completely manage SASL use?

Hi,

I started using the sasl parameters for this formula, my pillar looks like

postfix:
  config:
    smtpd_sasl_auth_enable: yes
    smtpd_sasl_security_options: noanonymous, noplaintext
    smtpd_sasl_tls_security_options: $smtpd_sasl_security_options
    smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd

  mapping:
    smtp_sasl_password_maps:
      - mail.example.com: user1:pass1
      - mail.example.com: user2:pass2

It creates /etc/postfix/sasl/smtpd.conf

pwcheck_method: auxprop
auxprop_plugin: sasldb
mech_list: PLAIN LOGIN CRAM-MD5 DIGEST-MD5

and also /etc/postfix/sasl_passwd and its postmap .db

But I got errors in mail.log like this

warning: SASL authentication problem: unable to open Berkeley db /etc/sasldb2: Permission denied

I fix it changing /etc/sasldb2 user to postfix.

I then got more authentication errors that I solved using saslpasswd2, so iit seems that /etc/sasldb2 is not managed by this formula.

Is this normal (I thought cyrus sasl config will be changed to use /etc/postfix/sasl_passwd), do I need to manage it by myself, or did I miss something?

Thanks

Add support for graylisting

Hi again @nmadhok .
I'm thinking on adding support for greylisting as well (using postgrey), see: http://rimuhosting.com/knowledgebase/linux/mail/greylisting%20with%20postgrey for the concept.
Again it will be hard to parameterize due to how main.cf is generated by using all the pillar values as parameters.
Should I create a pull request for it like for spf?

[BUG] openSUSE Leap overwrites salt config on first start

Your setup

Formula commit hash / release tag

1.0.1

Versions reports (master & minion)

015895a1c19:~ # salt -V
Salt Version:
           Salt: 3000.3
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.11.2
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.10 (default, Jan 16 2020, 09:12:04) [GCC]
   python-gnupg: Not Installed
         PyYAML: 5.3.1
          PyZMQ: 19.0.1
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2
 
System Versions:
           dist: SuSE NAME="openSUSE Leap"
         locale: UTF-8
        machine: x86_64
        release: 5.6.8-300.fc32.x86_64
         system: Linux
        version: SuSE NAME="openSUSE Leap"

Pillar / config used

test/salt/pillar/default.sls

Bug details

Describe the bug

Suse resets salt configuration if file /var/adm/postfix.configured in not present.

Before starting the service, systemd startup script will:

ExecStartPre=/etc/postfix/system/config_postfix
ExecStartPre=/etc/postfix/system/update_chroot
ExecStartPre=/etc/postfix/system/update_postmaps

And /etc/postfix/system/config_postfix will execute all defaults for OpenSuse, effectively rewriting salt written config.

Steps to reproduce the bug

Problem started when including OpenSuse as test platform.

Expected behaviour

Config properly written, and tests verify configuration.

Attempts to fix the bug

Additional context

This is happening while adding inspec tests in PR #108 .

os equals RedHat does not account for variants such as CentOS

The 'os' grain is NOT 'RedHat' on a 'CentOS' system, and likely different on others. This causes the policyd-spf state to break as it does not know which package to install.

The formula needs to map on os_family grain before os; or simply add CentOS to osmap.yaml

e.g.

[root@box ~]# salt 'box' grains.get os_family
box:
RedHat
[root@box ~]# salt 'box' grains.get os
box:
CentOS
[root@box ~]#

Yes, this is with 2017.7.4

[root@box ~]# salt --versions
Salt Version:
Salt: 2017.7.4

Dependency Versions:
cffi: 1.6.0
cherrypy: unknown
dateutil: 1.5
docker-py: 1.10.6
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: 1.3.8
Jinja2: 2.7.2
libgit2: 0.24.6
libnacl: 1.4.3
M2Crypto: 0.21.1
Mako: 0.8.1
msgpack-pure: 0.1.3
msgpack-python: 0.5.1
mysql-python: 1.2.5
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: 3.4.3
pygit2: 0.24.2
Python: 2.7.5 (default, Aug 4 2017, 00:39:18)
python-gnupg: 0.3.8
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: 0.6.5
smmap: 0.9.0
timelib: 0.2.4
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.4.1708 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-693.21.1.el7.x86_64
system: Linux
version: CentOS Linux 7.4.1708 Core

[root@box ~]#

	{{ key }}{% if colon %}:{% endif %} {{ value\|join(", ") }}
	{%- else -%}
	{{ key }}{% if colon %}:{% endif %} {{ value }}