The server software Mail Piler is an open source software for archiving emails. Since 2017, all companies and tradespeople in Germany have been obliged to archive emails. Freelancers and small business owners are an exception. The legal basis for email archiving is based on the Commercial Code, the Tax Code and the GoBD, among others.
Mail Piler offers a good archiving system and is developed by the Hungarian Janos Suto. There is a lot of documentation on his website, but there is still no clearly structured installation guide. There are also very few useful instructions on the Internet. This makes installing Mail Piler a real pain.
The developer responds relatively quickly on his Github account, but the quality of his answers leaves a lot to be desired. A big problem is the change from Sphinxsearch to Manticore Search as of version 1.4.1. This is documented, but only in passing and there is no information about how it changes the installation. In addition, the documentation regarding the installation is a joke. You have to painstakingly piece it together from the puzzle pieces on the Internet. You would think that this Janos Suto thinks that everyone else already knows what he had thought up. In addition, there are two domains. mailpiler.org as a community version with rather rudimentary documentation and mailpiler.com as a commercial site offering archiving. Finally, there is the hidden subdomain „docs.mailpiler.com/piler-ee/“, au, which can only be found via Google search, where there is a user manual, although this is more aimed at the Piler Enterprise Edition, so that at least something can be derived for the functions of the OSS version.
As I said, installation instructions for Mailpiler are poor and in some cases completely outdated. For this reason, the Mailpiler installation on a current Debian 12 Linux will be presented here. I use a VM under KVM for this. Proxmox would be even better for some people, as it is very easy to create a backup/snapshot of the VM via WebGui. Believe me, you will need it, the MailPiler installation is that tricky.
I don't think it's a good idea to make the whole thing available as a Docker container in production mode. If a company uses an application with high regulatory security standards, for example, it doesn't necessarily make sense to use a Docker container, as the VM will always have a higher level of isolation security than the container. And the isolation security should be sufficient for archived emails. Incidentally, this would also be an aspect to consider against commercial email archiving, as you are putting your data in the hands of an external provider.
DNS record, for example piler.yourdomain.tld, should already be created in your DNS beforehand. In this example, we are working with IPv6, as IPv4 addresses have now reached exorbitant prices. MailPiler should also be more accessible in your network than on the Internet.
In case your employees want to access your archived emails from outside, the configuration with IPv6 is shown here.
Your mail server must have access to Piler via SMTP (TCP/25) in order to send emails to the archive via BCC.
Port 25 should only be allowed for your mail server. All others should be denied port 25.
Mail-Piler offers "Smtp Acl" for this, but since this cannot handle IPv6, we will do it with IPTables and with the help of UFW.
Your Piler installation requires access to your mail server via either SMTP (TCP/25) or IMAPS (TCP/993) to restore email.
In addition to the packages that are already installed and are important to you on your system, here are the packages that we will need.
root@piler ~ # apt-get update
root@piler ~ # apt -y install sysstat build-essential libwrap0-dev libpst-dev tnef libytnef0-dev unrtf catdoc libtre-dev tre-agrep poppler-utils libzip-dev unixodbc libpq5 software-properties-common libpoppler-dev openssl libssl-dev mariadb-server mariadb-client python3-mysqldb memcached pwgen telnet git gnupg2 php8.3-cli php8.3-cgi php8.3-mysql php8.3-fpm php8.3-zip php8.3-ldap php8.3-gd php8.3-curl php8.3-xml php8.3-memcached apache2 libapache2-mod-php8.3 certbot python3-certbot python3-certbot-apache
If your initial installation of Debian Linux includes an exim mail server, remove it with
root@piler ~ # apt remove --yes exim
since Piler comes with its own SMTP server and it would not start if another SMTP server was already listening on port 25.
Next we need Manticore Search, as the developer has sworn off Sphinxsearch since Piler version 1.4.1 and has relied on Manticore Search ever since. However, he remains silent about what exactly changes in terms of installation and configuration, especially which installation sources/versions of Manticore he uses.
So we get from Manticore
root@piler ~ # wget repo.manticoresearch.com/manticore-repo.noarch.deb
and install it. The package installs the following files
/etc/apt/trusted.gpg.d/manticore-keyring.gpg
/usr/bin/manticoresearch-release
Then we create the repository for Manticore
root@piler ~ # echo „deb repo.manticoresearch.com/repository/manticoresearch_bookworm_dev bookworm main >> /etc/apt/sources.list.d/manticoresearch.list
root@piler ~ # apt-get update
Manticore can now be installed:
root@piler ~ # apt install manticore manticore-backup manticore-language-packs manticore-buddy manticore-common manticore-dev manticore-icudata-65l manticore-repo manticore-server manticore-server-core manticore-tools manticore-tzdata
It is important that Manticore Search is installed up to this point. The Manticore service does not need to be activated in the systemd,
root@piler ~ # systemctl status manticore.service
○ manticore.service - Manticore Search Engine
Loaded: loaded (/lib/systemd/system/manticore.service; disabled; preset: enabled)
Active: inactive (dead)
Docs: manual.manticoresearch.com,
man:searchd(1)
because that would prevent the Pilersearch service from starting later. So we will limit ourselves here to providing Manticore Search and its functions without starting the service directly. Another aspect that causes the developer to remain silent.
First, we get the MailPiler code from the GitHub repository with
root@piler ~ # git clone github.com/jsuto/piler.git
root@piler ~ # cd piler
Please note that we are working with version 1.4.7 here.
We will need some variables frequently in the installation, which is why we work with variables. Please insert your own values here:
root@piler ~ # PILER_DOMAIN="piler.yourdomain.tld"
root@piler ~ # MAILSERVER_DOMAIN="mx.domain.tld"
We optimize the MySQL settings as recommended in the FAQs
root@piler ~ # cat > /etc/mysql/conf.d/mailpiler.conf <<EOF
innodb_buffer_pool_size=256M
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=64M
innodb_log_file_size=16M
query_cache_size=0
query_cache_type=0
query_cache_limit=2M
EOF
root@piler ~ # systemctl restart mariadb
Now we need a dedicated user piler, the associated group piler and his working or home directory.
root@piler ~ # groupadd piler
root@piler ~ # useradd -g piler -m -s /bin/bash -d /var/piler piler
root@piler ~ # usermod -L piler
root@piler ~ # chmod 755 /var/piler
At this point we start with the actual compilation, installation and updating of the links to the libraries. To do this we change back to the directory that we created with the Git clone.
root@piler ~# cd ~/piler
root@piler ~/piler # ./configure --localstatedir=/var --with-database=mysql --enable-memcached
root@piler ~/piler # make
root@piler ~/piler # make install
root@piler ~/piler # ldconfig
Once MailPiler is installed in the directories "/usr/local/bin/piler", "/usr/local/etc/piler", "/usr/local/lib/piler", "/usr/local/libexec/piler", "/usr/local/sbin/piler", "/usr/local/share/piler", the "postinstall" function of the MailPiler Makefile must be executed. This sets up the database for MailPiler and a few other things.
First, we need to generate a password for the Piler database and insert our previously created variables at a suitable place in the postinsatll.sh script.
root@piler ~/piler # PILER_MYSQL_USER_PW="$(pwgen -cnsB 32 1)"
root@piler ~/piler # echo; echo "---"; echo "MYSQL PILER PASSWORD: $PILER_MYSQL_USER_PW"; echo "---"; echo
root@piler ~/piler # cp util/postinstall.sh util/postinstall.sh.bak
root@piler ~/piler # sed -i "s/ SMARTHOST=.*/ SMARTHOST="\"$MAILSERVER_DOMAIN\""/" util/postinstall.sh
root@piler ~/piler # sed -i 's/ WWWGROUP=.*/ WWWGROUP="www-data"/' util/postinstall.sh
Now you can start the installation of MailPiler:
make postinstall
The default values specified in the installation script should be fine.
The MySQL password you just created will be required via copy&paste during the subsequent installation routine. Here you will also have to answer the following questions:
www-data.mysql password for piler is the password you just generated.mysql root password you can enter anything here, since MariaDB works via sockets anyway.smtp relay gets the same value as was passed to the variable $MAILSERVER_DOMAIN.If MailPiler should listen on the IPv6 interface, we insert our IPv6 address as follows. In the sed command, we use the pipe character instead of the slashes, because this way we avoid the cumbersome masking of the special characters.
root@piler ~ # sed -i "s|listen_addr=0.0.0.0|listen_addr=IPv6-Adresse|" /usr/local/etc/piler/piler.conf
Further adjustments are as follows.
root@piler ~ # sed -i "s/hostid=.*/hostid=$PILER_DOMAIN/" /usr/local/etc/piler/piler.conf
root@piler ~ # sed -i "s/update_counters_to_memcached=.*/update_counters_to_memcached=1/" /usr/local/etc/piler/piler.conf
root@piler ~ # sed -i "s/spam_header_line=.*/spam_header_line=X-Spam-Flag: YES/" /usr/local/etc/piler/piler.conf
Manticore requires this file /usr/local/etc/piler/manticore.conf , but it is not generated by the installation routine. There is still no way to access this file. However, a look at the Docker installation of this repository gives hope. Because that is where the file is located:
Well, how did I come across this file, where did I find it? Actually, I can't find it in the source code provided in the Github repository. But it is included in the Docker container. The developer also provides his MailPiler as a Docker container that can be run immediately. So you change to the Docker directory of the downloaded repository and start the Docker container - assuming you have a current Docker installed. The basic idea is that he only wants to use Manticore, so there should also be an executable configuration file.
root@piler ~ # ls -l ~/piler/docker
-rwxr-xr-x 1 root root 239 2. Aug 10:11 build.sh
-rw-r--r-- 1 root root 1798 2. Aug 10:11 docker-compose.yaml
-rw-r--r-- 1 root root 1520 2. Aug 10:11 Dockerfile
-rw-r--r-- 1 root root 2511 2. Aug 10:11 manticore.conf
-rw-r--r-- 1 root root 229 2. Aug 10:11 piler.cnf
-rw-r--r-- 1 root root 409 2. Aug 10:11 README.md
-rwxr-xr-x 1 root root 6411 2. Aug 10:11 start.sh
This file can now be easily copied to the right location. Make sure to replace "sql_pass" with the password from $PILER_MYSQL_USER_PW.
root@piler ~ # cp ~/piler/docker/manticore.conf /usr/local/etc/piler/
root@piler ~ # sed -i "s|sql_pass = piler123|sql_pass = $PILER_MYSQL_USER_PW|" /usr/local/etc/piler/manticore.conf
Now we need the web server including SSL certification. For this we choose Apache2 and Let'sEncrypt via certbot. The necessary components are already installed. The Let'sEncrypt certificate can be installed as follows:
root@piler ~ # certbot certonly --apache --debug --rsa-key-size 4096 -d piler.yourdomain.tld
If you find connecting the web server directly to the external IP address too risky, there is another solution where the web server listens on localhost. This only requires changing the file /etc/apache2/sites-enabled/000-default.conf.
<VirtualHost *:80>
ServerAdmin
webmaster@localhost
DocumentRoot /var/www/piler
<Directory /var/www/piler>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
allow from all
</Directory>
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
In addition, we change the interface on which the web server should listen, to "localhost only":
root@piler ~ # sed -i "s|Listen 80|Listen 127.0.0.1:80|" /etc/apache2/ports.conf
root@piler ~ # systemctl restart apache2
The MailPiler GUI can now be accessed via the web browser if you first set up SSH port forwarding in your shell.
~$ ssh -L 9000:localhost:80 piler.yourdomain.tld http: //localhost:9000/
Whether you want to do it this way or restrict external access to the MailPiler web interface through firewall rules is up to you.
In order for the owners of a mailbox to also be able to log in to MailPiler and also restore emails from MailPiler, MailPiler can authenticate itself against an IMAP server. To do this, the following content must be added to the file /usr/local/etc/piler/config-site.php and adapted accordingly.
$config['ENABLE_IMAP_AUTH'] = 1;
$config['RESTORE_OVER_IMAP'] = 1;
$config['IMAP_RESTORE_FOLDER_INBOX'] = 'INBOX';
$config['IMAP_RESTORE_FOLDER_SENT'] = 'Sent';
$config['IMAP_HOST'] = '<HOST DES E-MAIL SERVERS>';
MailPiler has had an SMTP Acl list since version 1.5.0, which can be used to specify which IP addresses are allowed to send to MailPiler, but this is more of a cosmetic feature in the context of a functioning firewall. If I always protect my server with a firewall - in this case UFW - then I can already specify here which mail server is allowed to access port 25 on MailPiler. With UFW it looks like this.
root@piler ~ # ufw allow from MAILSERVER-IP proto tcp to any port 25
root@piler ~ # ufw allow from any proto tcp to any port 80
root@piler ~ # ufw allow from any proto tcp to any port 443
root@piler ~ # ufw enable
Anyone who says that it is unsafe to offer web access to the mail archive over the Internet, even with SSL, should be reassured. Every user, including administrators, can additionally protect themselves using Google's built-in two-factor authentication (2FA).
At this point, all MailPiler services should be set up and activated.
root@piler ~ # ln -s /usr/local/sbin/piler /usr/sbin/piler
root@piler ~ # ln -s /usr/local/sbin/piler-smtp /usr/sbin/piler-smtp
root@piler ~ # ln -s /usr/local/etc/piler /etc/piler
root@piler ~ # ln -s /usr/local/libexec/piler/pilersearch.service /etc/systemd/system/pilersearch.service
root@piler ~ # ln -s /usr/local/libexec/piler/piler.service /etc/systemd/system/piler.service
root@piler ~ # ln -s /usr/local/libexec/piler/piler-smtp.service /etc/systemd/system/piler-smtp.service
root@piler ~ # systemctl daemon-reload
root@piler ~ # systemctl enable --now piler
root@piler ~ # systemctl enable --now pilersearch
root@piler ~ # systemctl enable --now piler-smtp
If the services are now running, the administrator interface should now be accessible. The standard login data is for the user name "admin@local" with the password "pilerrocks".
The first thing you should do is assign a new password under administration => users. I would delete this user immediately after creating a new main administrator under their own domain.
Then there is an"auditor@local" with the role "Auditor", which I would also replace directly for security reasons, and which can see and delete absolutely all emails.
MailPiler is absolutely useless as long as the mail server - in this case Postfix - does not know about MailPiler. We have already allowed our mail server to use port 25 via IPtables on MailPiler. Now Postfix just needs to know what else it should do with the mails - incoming and outgoing. To do this, we set up an email address, for example "archive@piler.yourdomain.tld" and change the main.cf with
root@piler ~ # postconf always_bcc=archive@piler.yourdomain.tld
root@piler ~ # postfix reload
This means that the mail server should also send all emails to MailPiler via BCC. If you use Rspamd to filter out both SPAM and viruses or reject them completely, this happens after distribution using BCC, so that SPAM and virus emails are in the mail archive. These must be sorted out using the archiving rules or manually.
Now it is worth mentioning that MailPiler only indexes the emails every 30 minutes using a cron job and makes them visible in the WebGui. If you now want to test the reception and its rules, you have to speed up the process as the piler user.
root@piler ~ # su piler
piler@piler:~$ /usr/bin/indexer --quiet tag1 --rotate --config /usr/local/etc/piler/manticore.conf
piler@piler:~$ /usr/bin/indexer --quiet note1 --rotate --config /usr/local/etc/piler/manticore.conf
piler@piler:~$ /usr/local/libexec/piler/indexer.delta.sh
Especially at the beginning of the test phase, you may want to reset the archive to zero, and this is also possible. First, you should back up the working Piler database:
root@piler ~ # mysqldump --opt -u root --databases > piler.sql
Then the services are stopped and the files in the following directories are deleted.
root@piler ~ # systemctl stop piler
root@piler ~ # systemctl stop pilersearch
root@piler ~ # systemctl stop piler-smtp
root@piler ~ # rm -rf /var/piler/store/00/*
root@piler ~ # rm -rf /var/piler/manticore/*
In addition, the database is completely deleted and then restored from the backup.
root@piler ~ # mysql --defaults-file=/etc/mysql/debian.cnf -e "DROP DATABASE piler;";
root@piler ~ # mysql --defaults-file=/etc/mysql/debian.cnf -e "CREATE DATABASE piler CHARACTER SET utf8;";
root@piler ~ # mysql --defaults-file=/etc/mysql/debian.cnf piler < piler.sql
Afterwards, all services are restarted and you have an empty archive again.
root@piler ~ # systemctl start piler
root@piler ~ # systemctl start pilersearch
root@piler ~ # systemctl start piler-smtp
Please note that the mail archive is not a backup, but only an archive. The emails should be backed up regularly using the backup method of your choice. With MailPiler, the following areas should always be backed up regularly:
/usr/local/etc/piler - Directory with the piler and manticore configuration files. In particular the piler.key file, otherwise decryption would no longer be possible./var/piler/store/00 - This is where the emails are located. Please note that only the last directory changes to '00'./var/piler/manticore/*mysqldump./etc/apache2 and /etc/letsencrypt.MailPiler also offers an export of emails, which can be done daily via cronjob.
root@piler ~ # pilerexport -a 2024.08.12 -b 2024.08.12
root@piler ~ # tar cfz /path/to/backup-2024.08.12.tar.gz *
If you use Proxmox, you can set up regular backups or snapshots of the VM. And in the case of recovery, you have the state of the last backup, for example from last night at 0:00.
If you use KVM, there is now a great backup tool called virtnbdbackup, which can also be used to back up the Piler VM both as a full backup and as an incremental backup.