Message Transfer Format

The Gromox Mailbox Transfer format (GXMT) is our streaming-capable serialization format for conveying a bundle of MAPI objects for the purpose of interprocess communication.

Whereas a TNEF file only records one object, GXMT can record many, folders included. GXMT is streamable; there are no forward references, so the stream need not be buffered by a reader to resolve refs. Backreferences are typically named properties and folders that were created as part of processing earlier parts of a stream. Out of convenience of implementation, the gromox-import program buffers one entire object at a time before acting on it.

Spec

All integers are to be in little-endian form.

• char magic[8] = "GXMT0005"; Magic fixed value to indicate the MT stream revision.

• uint8_t splice_flag; Indicates whether the root object of this MT stream is to become a new folder in the target mailbox (0), or whether the objects in this stream are to be spliced (1) into preexisting folders of a mailbox.

• uint8_t public_store_flag; Indicates whether this MT stream was generated from a private (0) or public store (1) and whether folder IDs appearing in target_nid use the magic values from the PRIVATE_FID_ or PUBLIC_FID_ sets for built-in folders.

• uint64_t fm_size; Size in bytes for the folder map section that follows. This must be non-zero, i.e. fm_entires needs to be present, even if fm_entries is 0.

• Folder map:

  • uint64_t fm_entries; The number of entries in the folder map

  • Repeat fm_entries times:

    • uint64_t nid; Numeric identifier for the folder within the MT stream.
    • uint8_t create; Indicates that a new folder is to be created (1), or that a mailbox's existing folder should be reused (0).
    • uint64_t target_nid; If create is 0, then target_nid specifies the matching folder ID in the target mailbox to use. Magic values to denote built-in folders exist, e.g. PRIVATE_FID_INBOX (13); cf. include/gromox/mapi_types.hpp for a list.
    • char name[]; NUL-terminated folder name string for newly created folders.

• uint64_t np_size; Size in bytes for the named property map section that follows. This must be non-zero, i.e. np_entries needs to be present, even if np_entries is 0.

• Named Property map:

  • uint64_t np_entries; Number of entries in the NP map

  • Repeat np_entries times:

    • uint32_t proptag; The named property's assigned proptag for the MT stream. The high 16 bits convey the propid, the low 16 bits the proptype. An MT writer may emit PT_UNSPECIFIED for the proptype to signal that an MT reader shall mask/disregard proptypes during propid-to-propid remapping. If an MT writer emits any other proptype, the MT reader should only do propid-to-propid translations if the proptype matches.
    • PROPERTY_NAME serialized struct

The remainder of the stream is a set of "instructions" (so to speak) to gromox-import to create folders/messages in the target mailbox. Each packet/frame is:

• uint64_t obj_size; Size in bytes for this frame. The obj_size value does not include obj_size's own field size.

• Frame:

  • uint32_t mapi_objtype; MAPI_FOLDER (3), MAPI_MESSAGE (5), or a named property (250).

  • uint64_t nid; A unique identifier for this object in the MT stream. Value 0 is reserved and must not be used. Value 0xffffffffffffffff is reserved and should not be used.

  • uint32_t parent_type;

    • If mapi_objtype is MAPI_FOLDER, parent_type must be MAPI_FOLDER or zero (0 only allowed if parent_fid does not indicate a real parent).
    • If mapi_objtype is MAPI_MESSAGE, parent_type must be MAPI_FOLDER or zero (0 only allowed if parent_fid does not indicate a real parent)
    • If mapi_objtype is namedprop/250, parent_type is ignored.
    • Reader implementations are free to ignore this field if they kept track of the parent object's type in another way.
    • Writer implementations must be truthful about the value.

  • uint64_t parent_fid;

    • Parent object of a folder or message.
    • The value 0 is reserved.
    • The value 0xffffffffffffffff indicates an "unanchored" message that does not belong to any particular folder.

  • For objtype MAPI_FOLDER, more fields:

    • TAGGED_PROPVAL serialized struct for properties

    • uint64_t acl_count;

    • repeat for acl_count:

      • PERMISSION_DATA serialized struct

  • For objtype MAPI_MESSAGE:

    • MESSAGE_CONTENT serialized struct
    • NUL-terminated string for the RFC5322 representation
    • NUL-terminated string (reserved)

  • For objtype 250 (named property):

    • PROPERTY_NAME serialized struct

  • For other object types: Illegal frame. A diagnostic message should be emitted. Discard this frame. A reader implementation may choose to continue parsing at the next frame (the obj_size field is helpful in knowing where the next frame starts), or abort parsing altogether.

Attachments should be tied to a MESSAGE_CONTENT struct and encoded as such. GXMT decoders do not expect to see packets with objtype MAPI_ATTACH (7) or packets with parent_type MAPI_ATTACH.

ATTACHMENT_CONTENT serialization

• TPROPVAL_ARRAY serialized struct indicating the attachment properties

• uint8_t embedded; Indicates whether a file attachment (0) or an embedded message (1) follows.

• If embedded != 0:

  • MESSAGE_CONTENT serialized struct for the message

ATTACHMENT_LIST serialization

• uint16_t atcount; Number of attachments.

• Repeat atcount times:

  • ATTACHMENT_CONTENT serialized struct

MESSAGE_CONTENT serialization

• TPROPVAL_ARRAY serialized struct

• uint8_t have_rcpts;

• if have_rcpts != 0:

  • TARRAY_SET serialized struct

• uint8_t have_attachments;

• if have_attachments != 0:

  • ATTACHMENT_LIST serialized struct

PERMISSION_DATA serialization

• uint8_t flags; For GXMT, this is always ROW_ADD (0).
• TAGGED_PROPVAL serialized struct usually containing PR_SMTP_ADDRESS and PR_MEMBER_RIGHTS properties

PROPERTY_NAME serialization

For the concept of Named Properties, see https://learn.microsoft.com/en-us/office/client-developer/outlook/mapi/mapi-named-properties .

• uint8_t kind;: MNID_ID (0) or MNID_STRING (1).

• FLATUID guid;

• if the kind is MNID_ID, more fields:

  • uint32_t lid;
    The NP's LID= portion.

• if the kind is MNID_STRING:

  • uint8_t name_size;

    • Allocation hint for parsers. Specifies the number of subsequent bytes that make up the name, including the trailing U+0000 codepoint.
    • Reader implementations are free to ignore this field.
    • Writer implementations must not underspecify the size.

  • char16_t name[]; A run of UTF-16 codepoints that make up the name. A U+0000 codepoint marks the authoritative end of the string.

• On any other kind: Illegal namedprop, consider aborting the parse.

TAGGED_PROPVAL serialization

• uint32_t proptag; Property tag, consisting of the property ID (propid) in the high 16 bits and the property type (proptype) in the low 16 bits.

• switch on proptype:

  • PT_UNSPECIFIED (0): a TYPED_PROPVAL serialized struct follows
  • PT_NULL (0x1): (no value)
  • PT_SHORT (0x2): a s16LE integer follows
  • PT_LONG (0x3): a s32LE integer follows
  • PT_FLOAT (0x4): a IEEE754 binary32 (LE) value follows
  • PT_DOUBLE (0x5): a IEEE754 binary64 (LE) value follows
  • PT_CURRENCY (0x6): a s64LE integer indicating a quantity in units of 1/10000.
  • PT_APPTIME (0x7): a IEEE754 binary64 (LE) value follows
  • PT_ERROR (0xa): a u32LE value indicating a MAPI error code; doesn't normally occur in GXMT streams
  • PT_BOOLEAN (0xb): a uint8_t indicating false (0) or true (1). Writers must not emit any other value.
  • PT_OBJECT (0xd): ...
  • PT_I8 (0x14): a s64LE integer
  • PT_STRING8 (0x1e): a C string terminated by NUL. Character set encoding is not conveyed, so it should only use US-ASCII.
  • PT_UNICODE (0x1f): a UTF-16 string terminated by a U+0000 codepoint.
  • PT_SYSTIME (0x40): a s64LE integer indicating time since the NT epoch (1601-01-01) in units of 100 nanoseconds
  • PT_CLSID (0x48): 16 bytes specifying a GUID
  • PT_SVREID (0xfb): SVREID serialized struct
  • PT_SRESTRICTION (0xfd): RESTRICTION serialized struct
  • PT_ACTIONS (0xfe): ACTIONS serialized struct
  • PT_BINARY (0x0102): BINARY serialized struct
  • PT_MV_SHORT (0x1002): SHORT_ARRAY serialized struct
  • PT_MV_LONG (0x1003): LONG_ARRAY serialized struct
  • PT_MV_FLOAT (0x1004): FLOAT_ARRAY serialized struct
  • PT_MV_DOUBLE (0x1005): DOUBLE_ARRAY serialized struct
  • PT_MV_CURRENCY (0x1006): LONGLONG_ARRAY serialized struct
  • PT_MV_APPTIME (0x1007): DOUBLE_ARRAY serialized struct
  • PT_MV_I8 (0x1014): LONGLONG_ARRAY serialized struct
  • PT_MV_STRING8 (0x101e): STRING_ARRAY serialized struct
  • PT_MV_UNICODE (0x101f): WSTRING_ARRAY serialized struct
  • PT_MV_SYSTIME (0x1040): LONGLONG_ARRAY serialized struct
  • PT_MV_CLSID (0x1048): GUID_ARRAY serialized struct
  • PT_MV_BINARY (0x1102): BINARY_ARRAY serialized struct

TARRAY_SET serialization

TARRAY_SET is basically a MAPI table (properties x rows).

• uint32_t count; Number of rows

• Repeat count times:

  • TPROPVAL_ARRAY serialized struct specifying the properties in this row

TPROPVAL_ARRAY serialization

• uint16_t propcount;

• Repeat propcount times:

  • TAGGED_PROPVAL serialized struct specifying the proptag and value.

TYPED_PROPVAL serialization

TYPED_PROPVALs are normally used by MAPI tables to respond to columns with a PT_UNSPECIFIED type. TYPED_PROPVALs are not believed to appear in GXMT streams in practice, as any GXMT writer wishing to write out a TYPED_PROPVAL object could just write a properly formatted TAGGED_PROPVAL object with merged proptag value. Nevertheless, TYPED_PROPVALs are part of the current specification.

• uint16_t proptype;
• TAGGED_PROPVAL serialized struct specifying the propid and value.