Piler + Manticore + PMilter: Enterprise Email Archiving, Full-Text Search (Chinese Included), and Access Control

Posted on by Rico

Mail Server Series — Part 8

In the previous articles, we completed:

  • Postfix (SMTP routing & mail transfer)
  • Dovecot (IMAP/POP3 mailbox service)
  • Amavis / ClamAV / SpamAssassin (security filtering)
  • MariaDB + PostfixAdmin (virtual domains & mailbox management)

In this chapter, we focus on a major enterprise requirement:

Email Archiving System — Piler

An enterprise mail system must do more than just “send & receive.”
It must preserve, index, audit, and search emails across the organization.

This article explains:

  • Why email archiving is essential
  • Full Piler architecture & Docker deployment
  • Integrating Postfix → Piler → Manticore Search
  • Enabling full-text search for Chinese
  • Implementing PMilter to inject X-Envelope-To headers
  • Ensuring role-based access control to archived emails
  • Complete configs, compose files, and testing steps

1. Why Do Enterprises Need an Email Archiving System?

It’s not enough to store emails inside IMAP mailboxes.
Enterprises require long-term searchable email records for:

RequirementDescription
Employee off-boardingEmail records must remain accessible
ComplianceCertain emails must be retained for years
AuditsSupervisors/legal teams must access archives
Large-scale SearchSearch across all emails, all accounts
Immutable StorageUsers must not be able to delete archives

Piler provides:

  • A permanent copy of all inbound/outbound mail
  • Full-text indexing
  • Role-based search access (user only sees their own email)
  • Supervisor/audit views with proper security
  • A modern, simple web UI

2. Piler Architecture Overview

Postfix (always_bcc → piler@archive.local)
    ↓
Piler (Port 25)
    ↓  → Store (raw message storage)
    ↓  → MySQL (metadata index)
    ↓  → Manticore Search (full-text search)
    ↓  → Web Interface (user login)

Components:

ComponentPurpose
Piler daemonReceives, stores, indexes messages
MySQLHolds message metadata
ManticoreFull-text search engine (supports Chinese via ICU)
MemcachedCache acceleration
Web UIEnd-user portal
PMilter (custom)Injects X-Envelope-To for permission control

3. Why PMilter Is Necessary (and Why Piler Alone Isn’t Enough)

Piler does not know the actual envelope recipients unless they are explicitly provided.

Without X-Envelope-To:

  • Users may not see emails they actually received
  • Permissions become incomplete
  • Audit logs become inaccurate
  • Group/alias emails will be mis-assigned

Therefore we deploy a lightweight custom PMilter to inject:

X-Envelope-From:
X-Envelope-To:

Postfix config:

smtpd_milters = inet:pilermilter:33333
non_smtpd_milters = inet:pilermilter:33333

Piler config:

extra_to_field = X-Envelope-To:

Now Piler can correctly determine:

  • Who sent the message
  • Which individual users received it
  • Which accounts should have access

This greatly improves enterprise audit accuracy.

4. Full-Text Search for Chinese — Why Manticore Is Required

Default Sphinx/Manticore does not support Chinese segmentation.
We enable:

ngram_len = 2
morphology = icu_chinese

This allows proper search for:

  • 中文主旨（Subject）
  • 中文內文（Body）
  • 中文附件 OCR（if provided）

The table definition:

CREATE TABLE piler1 (
  id bigint,
  sender text indexed,
  rcpt text indexed,
  subject text indexed,
  body text indexed,
  ...
) ngram_len='2' ngram_chars='cjk' morphology='icu_chinese';

Now employees can search:

  • “採購”
  • “請款”
  • “合約”
  • “通知”
  • “報價”

And get accurate results.

5. Docker Architecture

(1) Manticore Search Container

Provides:

  • Port 9306 (SQL access)
  • Port 9307 (readonly SQL)
  • Chinese ICU segmentation
  • Auto-bootstrap of schema (first run)

(2) Piler Container

Environment example:

MANTICORE_HOSTNAME=manticore
MYSQL_HOSTNAME=maildb
MYSQL_DATABASE=piler
MYSQL_USER=piler
MYSQL_PASSWORD=piler8409

Volumes:

  • /var/piler/store — message files
  • /etc/piler — configuration (critical)

(3) Postfix → Piler Integration

Send copies of all emails:

always_bcc = piler@archive.local

Transport rule:

archive.local   smtp:[172.18.0.1]:2525

Piler listens on port 25 internally.

6. Piler config-site.php: Key Options

IMAP Authentication:

$config['ENABLE_IMAP_AUTH'] = 1;
$config['IMAP_HOST'] = 'dovecot';
$config['IMAP_PORT'] = 993;
$config['IMAP_SSL'] = 'SSL';

Full-text search:

$config['SPHINX_MAIN_INDEX'] = 'piler1';
$config['SPHINX_HOSTNAME'] = 'manticore:9306';

Permission control:

$extra_to_field = 'X-Envelope-To:';

7. piler.conf Critical Settings

Storage:

queuedir=/var/piler/store

Index:

sphxhost=manticore
sphxport=9306

Listener:

listen_port=25
listen_addr=0.0.0.0

8. Web Access via Apache Reverse Proxy

Example:

https://archive.it.demo.tw/

Apache configuration:

ProxyPass        / http://piler:80/
ProxyPassReverse / http://piler:80/
ProxyPassReverseCookieDomain piler archive.it.demo.tw

This keeps HTTPS termination on the web proxy.

9. Testing and Verification

✔️ Test email ingestion

Send an email, then search in Piler UI.

✔️ Test Chinese search

Search terms like “請款”, “採購”, “出貨”.

✔️ Test permission model

User A:

  • Should only see mail sent/received by A

Admin:

  • Can access all messages (if privilege enabled)

10. Summary — Enterprise-Grade Archive Completed

Your system now provides:

✔ Permanent email archiving
✔ Immutable storage
✔ Chinese full-text search
✔ Envelope-aware permission control
✔ Fast indexing via Manticore
✔ Comprehensive audit capabilities
✔ A modern UI for employees and admins

This is a highly robust enterprise archive solution, comparable to many commercial systems—but fully open source and customizable.