exmdb_provider

Name

exmdb_provider — http(8gx) service plugin for exmdb connections

Description

exmdb_provider is a service plugin for http(8gx). It offers a plethora of individual functions (124 of them) for operating on mailbox stores. This functionality is also exposed by way of a Gromox-specific network protocol on port 5000. The aforementioned RPC functions transparently operate over the network and may connect to a remote exmdb. In other words, this plugin contains a process-local function API, a network client, and a network server.

Configuration directives

The usual config file location is /etc/gromox/exmdb_provider.cfg.

cache_interval
Default: 2 hours
dbg_synthesize_content
When this directive is set to 1, missing content files will not be regarded as an error and the respective attachment or property is delivered with a replacement string. If set to 2, a replacement string is always delivered; mode 2 is useful for reducing the amount of data downloaded when debugging ICS.
Default: 0
enable_dam
When set to on, inbox rule processing is allowed to create Deferred Action Messages (DAM). Furthermore, the “Deferred Actions” folder will have its contents shown. / Conversely, if this directive is off, no DAMs will be created, and the DAM folder in inboxes is presented as empty to clients (even if it has content from earlier).
Outlook’s DAM handling is poor and if you experience a crash with a primary mailbox that is in non-cached/online mode a few seconds after Outlook has opened it, turn this option off for mitigation.
Default: on
exmdb_body_autosynthesis
When a client requests either PR_BODY, PR_HTML or PR_RTF_COMPRESSED, but that property does not exist on a particular message, automatically synthesize the data on-the-fly from another of the available formats.
Default: on
exmdb_listen_port
The TCP port number for exposing the timer service on.
Default: 5000
exmdb_pf_read_per_user
Keep public folder read states per user (1) or keep one state for all users (0).
Default: 1
exmdb_pf_read_states
When set to 0, messages in public stores/folders will always be shown as read and the folder summary will reflect that.
When set to 1, messages will have new/read markings but the folder summary will indicate 0 new messages at all times.
When set to 2, folder summaries will indicate the number of new messages.
Default: 2
exmdb_schema_upgrades
This directive controls whether database schemas are automatically upgraded when a mailbox is loaded. During this time, the mailbox is unavailable and operations on it will be delayed. Connection aborts, if any, would be due to timeouts in components other than exmdb_provider. (The procedure takes roughly 36sec per gigabyte of exchange.sqlite3 worth of data, or about 110k messages, on a 3700X CPU, single-thread. The file can also temporarily grow to double its size, so ample disk space may be required.)
Default: no
exmdb_search_pacing
When initially populating a search folder (static or dynamic), yield the lock on the sqlite database (file descriptor) after so many messages to give other clients a chance to perform an action.
Default: 2500
exrpc_debug
Log every incoming exmdb network RPC and the return code of the operation in a minimal fashion to stderr. Level 1 emits RPCs with a failure return code, level 2 emits all RPCs. Note that direct function calls from within the process image are not logged this way, so this will not show exmdb_provider invocations from exchange_emsmdb(4gx).
Default: 0
listen_ip
An IPv6 address (or v4-mapped address) for exposing the timer service on.
Default: ::1
max_ext_rule_number
Default: 20
max_router_connections
As a exmdb server, permit at most this many inbound connections for the purpose of sending notifications on these channels. Note that every incoming TCP connection starts as a data connection and only becomes re-classified as “notification” once the LISTEN_NOTIFICATION RPC has been issued by the client.
Default: unlimited (only limited by ulimits)
max_rpc_stub_threads
As a exmdb server, permit at most this many inbound connections for commands.
Default: unlimited (only limited by ulimits)
max_rule_number
Default: 1000
max_store_message_count
Default: 200000
mbox_contention_warning
If there are more than this many threads waiting to use a mailbox, emit a warning log message. Use 0 to disable.
Default: 5
mbox_contention_reject
If there are more than this many threads waiting to use a mailbox, refuse the particular exmdb RPC that is executing. Use 0 to disable.
Default: 5
notify_stub_threads_num
For every remote exmdb server in exmdb_list.txt, establish and keep this many number of outbound connections for receiving notification RPCs.
Default: 4
populating_threads_num
Default: 4
rpc_proxy_connection_num
For every remote exmdb server in exmdb_list.txt, establish and keep this many number of outbound connections for sending commands.
Default: 10
separator_for_bounce
Default: ;
sqlite_debug
If set to 1, every query given to SQLite prepare/execute is logged. If set to 0, only failed queries are logged. (It cannot be made completely silent, since our queries ought to never fail.)
Default: 0
sqlite_mmap_size
Default: 0 (use SQLite default)
sqlite_synchronous
Enables/disables synchronous mode for SQLite databases. See https://www.sqlite.org/pragma.html#pragma_synchronous for details.
Default: off
sqlite_wal_mode
Selects the particular journal mode for SQLite databases; off selects DELETE mode, on selects WAL mode. See https://www.sqlite.org/pragma.html#pragma_journal_mode for details.
Default: on
table_size
Default: 5000
x500_org_name

Default: (unspecified)

Multiserver selection map

The SQL column users.homedir specifies a home directory location in an abstract namespace. This abstract namespace is shared between all Gromox programs, and can be used to divide users into custom subsets and steer connections to different servers.

exmdb_list.txt specifies how to map from this namespace to exmdb servers. The file is used by exmdb clients to select the right server to connect to, and the file is used by exmdb_provider to set up its own data structures.

Each line in this file consists of 4 columns separated by whitespace:

  • Initial prefix to match a user’s exmdb home directory on. The pattern should almost always end in a ‘/’ character, otherwise a prefix of “/home” is able to match a userdir of “/home2/username” as well, which may be undesired.
  • The type of mail stores that are served beneath the prefix. This must either be “private” or “public”.
  • The IPv6 (or v4-mapped) address of the midb server to use for this prefix.
  • The port number.

In the absence of exmdb_list.txt, two implicit default entries are used:

/var/lib/gromox/user/ private ::1 5000
/var/lib/gromox/domain/ public ::1 5000

Network protocol

The transmissions on the socket are simple concatenations of protocol data units built using the NDR format. The PDU length is present within the PDU itself near the start.

{
     leuint32_t length;
     char pdu[];
}
pdu := {
     uint8_t call_id;
     string directory;
     switch (call_id) {
             ...
     }
}

Files

  • config_file_path/exmdb_acl.txt: A file with one address (IPv6 or v4-mapped) per line of allowed clients. In its absence, ::1 is default-whitelisted.
  • config_file_path/exmdb_list.txt: exmdb multiserver selection map.
  • data_file_path/mail_bounce/

config_file_path and data_file_path is determined by the configuration of the program that loaded the exmdb_provider plugin.

See also

gromox(7), http(8gx)