Hoodiecrow

Image by yours truly, see the original in Instagram

Hoodiecrow is a scriptable IMAP server for client integration testing. It offers IMAP4ver1 support and some optional plugins that can be turned on and off. Nothing is ever written to disk, so when you restart the server, the original state is restored.

STARTTLS requires Node 0.12 or iojs as it uses tls.TLSSocket API.

Usage

Run as a standalone server

To run Hoodiecrow you need Node.js in your machine. Node should work on almost any platform, so Hoodiecrow should too.

If you have Node.js installed, install Hoodiecrow with the npm command and run it:

npm install -g hoodiecrow-imap
sudo hoodiecrow

Sudo is needed to bind to port 143. If you choose to use a higher port, say 1143 (hoodiecrow -p 1143), you do not need to use sudo.

hoodiecrow command also provides an incoming SMTP server which appends all incoming messages automatically to INBOX. To use it, use smtpPort option (hoodiecrow --smtpPort=1025).

Protip Running hoodiecrow --help displays useful information about command line options for Hoodiecrow and some sample configuration data.

After you have started Hoodiecrow server, you can point your IMAP client to localhost:143. Use "testuser" as user name and "testpass" as password to log in to the server.

Include as a Node.js module

Add hoodiecrow-imap dependency

npm install hoodiecrow-imap

Create and start an IMAP server

var hoodiecrow = require("hoodiecrow-imap"),
    server = hoodiecrow(options);
server.listen(143);

See complete.js for an example.

Scope

Hoodiecrow is a single user / multiple connections IMAP server that uses a JSON object as its directory and messages structure. Nothing is read from or written to disk and the entire directory structure is instantiated every time the server is started, eg. changes made through the IMAP protocol (adding/removing messages/flags etc) are not saved permanently. This should ensure that you can write integration tests for clients in a way where a new fresh server with unmodified data is started for every test.

Several clients can connect to the server simultanously but all the clients share the same user account, even if login credentials are different.

Hoodiecrow is extendable, any command can be overwritten, plugins can be added etc (see command folder for built in command examples and plugin folder for plugin examples).

Authentication

An user can always login with username "testuser" and password "testpass". Any other credentials can be added as needed.

Status

IMAP4rev1

All commands are supported but might be a bit buggy

Supported Plugins

Plugins can be enabled when starting the server but can not be unloaded or loaded when the server is already running. All plugins are self contained and not tied to core. If you do not enable a plugin, no trace of it is left to the system. For example, if you do not enable CONDSTORE, messages do not have a MODSEQ value set.

AUTH-PLAIN Adds AUTH=PLAIN capability. Supports SASL-IR [RFC4959] as well
CONDSTORE Partially implemented CONDSTORE [RFC4551] support
CREATE-SPECIAL-USE Enables CREATE-SPECIAL-USE [RFC6154] capability. Allowed special flags can be set with server option "special-use"
ENABLE Adds ENABLE capability [RFC5161]. Must be loaded before any plugin that requires ENABLE support (eg. CONDSTORE)
ID Adds ID [RFC2971] capability
IDLE Adds IDLE [RFC2177] capability
LITERALPLUS Enables LITERAL+ [RFC2088] capability
LOGINDISABLED Disables LOGIN support for unencrypted connections
NAMESPACE Adds NAMESPACE [RFC2342] capability
SASL-IR Enables SASL-IR [RFC4959] capability
SPECIAL-USE Enables SPECIAL-USE [RFC6154] capability Mailboxes need to have a "special-use" property (String or Array) that will be used as extra flag for LIST and LSUB responses
STARTTLS Adds STARTTLS command
UNSELECT Adds UNSELECT [RFC3691] capability
X-GM-EXT-1 Adds partial support for Gmail specific options. X-GM-MSGID is fully supported, X-GM-LABELS is partially supported (labels can be STOREd and FETCHed but setting a label does not change message behavior, for example the message does not get copied to another mailbox). X-GM-THRID is not supported as I haven't figured threading out yet.
XOAUTH2 GMail XOAUTH2 login. Only works with SALS-IR, if you need non SASL-IR support as well, let me know. Use "testuser" as the username and "testtoken" as Access Token to log in.
XTOYBIRD Custom plugin to allow programmatic control of the server. Login not required to use XTOYBIRD commands

Planned but not yet implemented

MOVE
UIDPLUS
QUOTA

Authentication

An user can always login with username "testuser" and password "testpass". Any other credentials can be added as needed.

Existing XTOYBIRD commands

To use these functions, XTOYBIRD plugin needs to be enabled

Available commands:

XTOYBIRD SERVER dumps server internals
XTOYBIRD CONNECTION dumps connection internals
XTOYBIRD STORAGE dumps storage as JSON
XTOYBIRD USERADD "username" "password" adds or updates user
XTOYBIRD USERDEL "username" removes an user
XTOYBIRD SHUTDOWN Closes the server after the last client disconnects. New connections are rejected.

Example usage for XTOYBIRD STORAGE:

S: * Hoodiecrow ready for rumble
C: A1 XTOYBIRD STORAGE
S: * XTOYBIRD [XJSONDUMP] {3224}
S: {
S:     "INBOX": {
S:         "messages": [
S:             {
S:                 "raw": "Subject: hello 1\r\n\r\nWorld 1!",
S:                 ...
S: A1 OK XTOYBIRD Completed

Useful features for Hoodiecrow I'd like to see

An ability to change UIDVALIDITY at runtime (eg. A1 XTOYBIRD UIDVALIDITY INBOX 123 where 123 is the new UIDVALIDITY for INBOX)
An ability to change available disk space (eg. A1 XTOYBIRD DISKSPACE 100 50 where 100 is total disk space in bytes and 50 is available space)
An ability to restart the server to return initial state (A1 XTOYBIRD RESET)
An ability to change storage runtime by sending a JSON string describing the entire storage (A1 XTOYBIRD UPDATE {123}\r\n{"INBOX":{...}}))

CONDSTORE support

All messages have MODSEQ value
CONDSTORE can be ENABLEd
SELECT/EXAMINE show HIGHESTMODSEQ
SELECT/EXAMINE support (CONDSTORE) option
Updating flags increments MODSEQ value
FETCH (MODSEQ) works
FETCH (CHANGEDSINCE modseq) works
STORE (UNCHANGEDSINCE modseq) partially works (edge cases are not covered)

SEARCH MODSEQ is not supported

Known issues

INBOX* as a separate namespace and managing INBOX subfolders is a mess. CREATE seems to work, DELETE is buggy and RENAME doesn't work with INBOX subfolders (unless the default namespace is "INBOX.", not ""). I need to rethink how this works.

Not sure if these should be fixed or not

STORE does not emit notifications to other clients
MODSEQ updates are not notified

These issues are probably not going to get fixed

Session flags are not supported (this means that \Recent flag is also not supported)
addr-adl (at-domain-list) values are not supported, NIL is always used
anonymous namespaces are not supported
STORE returns NO and nothing is updated if there are pending EXPUNGE messages
CHARSET argument is ignored

Running tests

Running tests requires you to have grunt-cli installed

npm install -g grunt-cli

After which you can run

grunt

npm test

Example configs

Cyrus

config.json:

{
    "INBOX":{},
    "INBOX.":{},
    "user.":{
        "type":"user"
    },
    "":{
        "type":"shared"
    }
}

Gmail

config.json:

{
    "INBOX":{},
    "":{
        "separator": "/",
        "folders":{
            "[Gmail]":{
                "flags": ["\\Noselect"],
                "folders": {
                    "All Mail":{
                        "special-use": "\\All"
                    },
                    "Drafts":{
                        "special-use": "\\Drafts"
                    },
                    "Important":{
                        "special-use": "\\Important"
                    },
                    "Sent Mail":{
                        "special-use": "\\Sent"
                    },
                    "Spam":{
                        "special-use": "\\Junk"
                    },
                    "Starred":{
                        "special-use": "\\Flagged"
                    },
                    "Trash":{
                        "special-use": "\\Trash"
                    }
                }
            }
        }
    }
}

Use Hoodiecrow for testing your client

Creating your tests in Node.js is a piece of cake, you do not even need to run the hoodiecrow command. Here is a sample [nodeunit] test.

var hoodiecrow = require("hoodiecrow-imap"),
    myIMAPCLient = require("../my-imap-client");

module.exports["IMAP tests"] = {

    // Executed before every test
    // creates a new blank IMAP server
    setUp: function(callback){
        this.server = hoodiecrow();
        this.server.listen(1143);
        callback();
    },

    // Executed after every test
    // Closes the IMAP server created for the test
    tearDown: function(callback){
        this.server.close(callback);
    },

    /**
     * In this test a new IMAP client is instantiated that tries to connect
     * to the IMAP server. If client is connected the test is considered
     * as passed.
     */
    "Connect to the server": function(test){
        var client = myIMAPCLient.connect("localhost", 1143);
        client.on("ready", function(){
            client.disconnect();
            test.done();
        });
    }
}

Creating custom plugins

A plugin can be a string as a pointer to a built in plugin or a function. Plugin function is run when the server is created and gets server instance object as an argument.

hoodiecrow({
    // Add two plugins, built in "IDLE" and custom function
    plugin: ["IDLE", myAwesomePlugin]
});

// Plugin handler
function myAwesomePlugin(server){

    // Add a string to the capability listing
    server.registerCapability("XSUM");

    /**
     * Add a new command XSUM
     * If client runs this command, the response is a sum of all
     * numeric arguments provided
     *
     * A1 XSUM 1 2 3 4 5
     * * XSUM 15
     * A1 OK SUM completed
     *
     * @param {Object} connection - Session instance
     * @param {Object} parsed - Input from the client in structured form
     * @param {String} data - Input command as a binary string
     * @param {Function} callback - callback function to run
     */
    server.setCommandHandler("XSUM", function(connection, parsed, data, callback){

        // Send untagged XSUM response
        connection.send({
            tag: "*",
            command: "XSUM",
            attributes:[
                [].concat(parsed.attributes || []).reduce(function(prev, cur){
                    return prev + Number(cur.value);
                }, 0)
            ]
        }, "XSUM", parsed, data);

        // Send tagged OK response
        connection.send({
            tag: parsed.tag,
            command: "OK",
            attributes:[
                // TEXT allows to send unquoted
                {type: "TEXT", value: "XSUM completed"}
            ]
        }, "XSUM", parsed, data);
        callback();
    });
}

Plugin mehtods

Add a capability

server.registerCapability(name[, availabilty])

Where

name a string displayed in the capability response
availability a function which returns boolean value. Executed before displaying the capability response. If the function returns true, the capability is displayed, if false then not.

Example

// Display in CAPABILITY only in Not Authenticated state
server.registerCapability("XAUTH", function(connection){
    return connection.state == "Not Authenticated";
});

Define a command

server.setCommandHandler(name, handler)

Where

name is the command name
handler (connection, parsed, data, callback) is the handler function for the command

Handler arguments

connection - Session instance
parsed - Input from the client in structured form (see imap-handler for reference)
data - Input command as a binary string
callback - callback function to run (does not take any arguments)

The command should send data to the client with connection.send()

connection.send(response, description, parsed, data, /* any additional data */)

Where

response is a imap-handler compatible object. To get the correct tag for responsing OK, NO or BAD, look into parsed.tag
description is a string identifying the response to be used by other plugins
parsed is the parsed argument passed to the handler
data is the data argument passed to the handler
additional arguments can be used to provide input for other plugins

Retrieve an existing handler

To override existing commands you should first cache the existing command, so you can use it in your own command handler.

server.getCommandHandler(name) -> Function

Where

name is the function name

Example

var list = server.getCommandHandler("LIST");
server.setCommandHandler("LIST", function(connection, parsed, data, callback){
    // do something
    console.log("Received LIST request");
    // run the cached command
    list(connection, parsed, data, callback);
});

Reroute input from the client

If your plugin needs to get direct input from the client, you can reroute the incoming data by defining a connection.inputHandler function. The function gets input data as complete lines (without the linebreaks). Once you want to reroute the input back to the command handler, just clear the function.

connection.inputHandler = function(line){
    console.log(line);
    connection.inputHandler = false;
}

See idle.js for an example

Override output

Any response sent to the client can be overriden or cancelled by other handlers. You should append your handler to server.outputHandlers array. If something is being sent to the client, the response object is passed through all handlers in this array.

server.outputHandlers.push(function(connection, /* arguments from connection.send */){})

response arguments from connection.send is an object and thus any modifications will be passed on. If skipResponse property is added to the response object, the data is not sent to the client.

// All untagged responses are ignored and not passed to the client
server.outputHandlers.push(function(connection, response, description){
    if(response.tag == "*"){
        response.skipResponse = true;
        console.log("Ignoring untagged response for %s", description);
    }
});

Other possbile operations

It is possible to append messages to a mailbox; create, delete and rename mailboxes; change authentication state and so on through the server and connection methods and properties. See existing command handlers and plugins for examples.

License

MIT

Hoodiecrow IMAP server

Scriptable IMAP server for client integration testing

Hoodiecrow

Usage

Run as a standalone server

Include as a Node.js module

Scope

Authentication

Status

IMAP4rev1

Supported Plugins

Authentication

Existing XTOYBIRD commands

Useful features for Hoodiecrow I'd like to see

CONDSTORE support

Known issues

Running tests

Example configs

Cyrus

Gmail

Use Hoodiecrow for testing your client

Creating custom plugins

Plugin mehtods

Add a capability

Define a command

Retrieve an existing handler

Reroute input from the client

Override output

Other possbile operations

License