Skip to content
Home » Documentation » Configuration

Configuration

Introduction

Darlean applications can be configured by means of a configuration file, command-line arguments and/or environment variables.

Configuration file

A configuration file can be specified by means of the --darlean-config command-line argument, or by means of the DARLEAN_CONFIG environment variable.

Configuration files can be JSON or JSON5 files. The advantage of JSON5 is that it support comments, and that keys do not have to be quoted.

The layout (schema) of the configuration file is described in our API documentation for IApplicationCfg.

Environment variables

Environment variables start with DARLEAN_, and they override the corresponding values of the configuration file.

Variables that expect an array as value must be comma-separated: a,b,c. The special value none can be used to indicate an empty array.

Values that expect a boolean as value must be provided as true or false.

The naming of environment variables mimics the hierarchy of the corresponding configuration file setting. Hierarchical keys are separated by an underscore (_), and key names are uppercased without any underscores in them.

Example: { runtime: { persistence: { enabled: true } } } is equivalent to DARLEAN_RUNTIME_PERSISTENCE_ENABLED=true.

Note that keys must be uppercased, but values aretypically in lowercase (they are handled in a case-sensitive way).

Command-line arguments

Command-line arguments start with --darlean-. They overrule the corresponding values that are set in environment variables or the configuration file.

All command-line arguments expect a value. Values must be provided as part of the same argument after the = sign.
For example, --darlean-app-id=my-app.

Array values must be comma-separated: a,b,c. The special value none can be used to indicate an empty array.

Boolean values must be provided as true or false.

The naming of command line arguments mimics the hierarchy of the corresponding configuration file setting. Hierarchical keys are separated by a dash (-), and key names are lowercased without any dashes in them.

Example: { runtime: { persistence: { enabled: true } } } is equivalent to --darlean-runtime-persistence-enabled=true.

Application id

Every application in the cluster must have a unique application id (commonly abbreviated as app-id). This id is (amongst others) used to send messages to each other.

The app-id can be specified in the configuration file (see the below snippet), via the --darlean-appid command-line argument, or the DARLEAN_APPID environment variable.

{
    appId: 'my-app-02'
}

Operation modes and runtime apps

Darlean can run in two operation modes: all-in-one and distributed. The modes are described in Operation modes.

By default, Darlean runs in all-in-one mode.

To enable distributed mode, provide the same list of runtime-apps to all applications, either via the configuration file (see the below snippet), --darlean-runtimeapps command-line argument or DARLEAN_RUNTIMEAPPS environment variable:

{
    runtimeApps: ['runtime01', 'runtime02', 'runtime03']
}

Darlean Message Bus (DMB)

The Darlean Message Bus (DMB) connects the applications in the cluster. It currently uses the high-performance NATS message bus, and provides a thin configuration layer around this, but this may change in the future.

For clusters where all applications run on the same host (like a development machine), no additional configuration is required.

Host names

When applications run on multiple hosts, it is necessary to configure the hostnames (or IP addresses) that applications can use to connect to the message bus.

For every runtime application in the list of runtime-apps, a corresponding hostname or IP adress should be configured in the settings file (see below snippet), via the --darlean-messaging-dmb-hosts command-line argument, or via the DARLEAN_MESSAGING_DMB_HOSTS environment variable.

{
    runtimeApps: ['runtime01', 'runtime02', 'runtime03'],
    messaging: {
        dmb: {
            hosts: ['runtime01.example.com', 'runtime02.example.com', 'runtime03.example.com']
        }
    }
}

Important: Every application in the cluster (including the applications that are not runtime applications) must have the above settings configured identically, otherwise applications may not be able to properly connect to the DMB.

Every runtime application for which a host is configured launches a DBM server. Multiple DMB servers together provide a redundant messgae bus. As long as one instance is running, the message bus is available.

Port numbers

By default, DMB servers listen at ports 4500 and 5500. Port 4500 is the client port on which applications connect. Port 5500 is the cluster port on which the DMB servers mutually connect.

When multiple DMB servers run on the same host, the second server listens on port 4501 and 5501, the third on 4502 and 5502, and so on. The rank (first, second, third) is determined by the order in which the runtime apps are listed in the list of runtime-apps.

It is possible to adjust the base port numbers (4500 and 5500) by means of configuration (see below snippet), the --darlean-messaging-dmb-base-port and --darlean-messaging-dmb-cluster-base-port command line arguments and the DARLEAN_MESSAGING_DMB_BASE_PORT and DARLEAN_MESSAGING_DMB_CLUSTER_BASE_PORT environment variables.

{
    runtimeApps: ['runtime01', 'runtime02', 'runtime03'],
    messaging: {
        dmb: {
            hosts: ['runtime01.example.com', 'runtime02.example.com', 'runtime03.example.com'],
            basePort: 1234,
            baseClusterPort: 2345  
        }
    }
}

Important: When you override the default settings for the port numbers, all applications in the cluster (including the applications that are not a runtime application) must be configured identically. They all must receive the same base port numbers. Non-runtime applications only need the client base port, as they do not use the cluster port.

Persistence

The runtime applications provide persistence to the applications in the cluster. When applications want to load or store data, they provide a specifier that indicates the functional role of the data. For example, shoppingcart.state to indicate that the internal state of the shopping cart is to be stored or retrieved.

The runtime then determines which persistence provider should handle the request, and the persistence provider determines in which compartment the data should be stored or loaded from.

A persistence provider is one implementation of persistence. Darlean comes out-of-the-box with a persistence provider that stores data on disk (local or shared, physical or virtual, that does not matter). But other providers (like SQL or cloud storage) can be added as well.

A compartment represents a relatively independent part of the storage. For file system storage, a compartment could be a separate folder in which data is stored. For a SQL provider, it could be a separate table or even a separate database. Compartments provide separation of data, and allow management tasks (like backing up or restoring or moving data to another persistence provider) to take place without affecting other parts of the system.

Because of the complexity of persistence, it is usually configured via configuration files only.

Here is an example of a persistence configuration that:

  • Maps shopping cart related data to a dedicated compartment.
  • Overrides the shard count for that dedicated compartment.
  • Registers the FileSystem Persistence Service as provider for fs.* compartments.

{
    runtime: {
        persistence: {
            // Mapping from specifiers to compartment.
            // Let's map our shoppingcart related data to a dedicated compartment.
            specifiers: [{ specifier: 'shoppingcart.*', compartment: 'fs.shoppingcart' }],

            // Mapping from compartment mask to which actor type implements the persistence service
            // Note: This default mapping is already provided by default, we just show it here for illustration
            handlers: [{ compartment: 'fs.*', actorType: 'io.darlean.fspersistenceservice' }],
            
            // Configuration of file-system persistence
            fs: {
                compartments: [
                    // Default settings for all compartments
                    { compartment: '*', basePath: './persistence', shardCount: 8 },
                    // Settings for the compartment where shopping cart info is stored. We choose here
                    // to increase the shard count to allow even more throughput than Darlean already provides.
                    { compartment: 'fs.shoppingcart', shardCount: 32 }
                ]
            }
        },
    }
}

Handler configuration

The optional handler configuration determines which persistence provider handles the requests for a certain compartment. The configuration consists of a list of records, where each record maps one or more compartments to one handler. The record has the following fields:

  • compartment – A wildcard filter expression that determines to which compartments this record applies.
  • actorType – The type of the actor that implements the persistence provider.

All requests of which the compartment matches with the compartment filter expression are forwarded to an actor with the specified actorType and with an id consisting of one element, namely the full compartment name from the request (not the compartment wildcard filter expression).

By default, Darlean already provides the listed mapping from fs.* to io.darlean.fspersistenceservice, so it is not necessary to include this in your configuration file unless you want to use a different actor type for compartments that start with fs..

Specifier mapping

The optional specifier mapping determines in which compartment requests with a certain specifier end up. In other words, it maps the specifiers that the persistence service receives from applications that want to load or store data on compartment names.

The specifier mapping configuration consists of a list of records, where each record maps one or more specifiers to one or more compartments. A record has the following fields:

  • specifier – A wildcard filter exression that determines to which specifier(s) this record applies
  • compartment – The compartment in which data for the specifier(s) must be stored.

The specifier filter expression can contain wildcards, like a.* or a.*.b. That makes it possible to map multiple specifiers on a certain compartment.

The compartment name can contain wildcard placeholders. Wildcard placeholders look like ${*} or ${**}. The placeholders are replaced with the value from the specifier that matches with the first, respectively second, wildcard match.

Example.

The record { specifier: 'foo.*.z', compartment: 'fs.bar.${*}' } applied to specifier foo.abc.z results in compartment fs.bar.abc.

The first record for which the specifier filter matches applies, all remaining records are ignored.

Two extreme configurations are:

  • All specifiers map to the same compartment. { specifier: '*', compartment: 'fs.default' }.
  • Each specifier has its own compartment. { specifier: '*', compartment: 'fs.${*}' }.

By default, all specifiers map to one single comparment fs.default.

File system persistence configuration

File system persistence is a way of persistence that stores data in files on disk. In principle, it does not matter whether this is a physical disk like a hard drive or SSD or a virtual disk (like a network disk), nor does it matter whether one runtime application has private (exclusive) access to (its part of) the file system or whether all applications share the same file system. The file system persistence supports all.

Files are by default located in the ./persistence/ folder, but the location can be overruled by means of a base-path. Within that folder, a hierarchy is created that consists of a sub-path, the compartment name, the total number of shards, the current shard number, and the app-id of the runtime node when the shard is explicitly bound to a certain app-id.

Data is stored in sqlite files within that folder hierarchy. Sqlite is a powerful, in-process and zero-configuration database engine that is embedded in the Darlean runtime.

File system persistence is configured by means of a list of records. The records are processed top to bottom, and all records for which the compartment filter expression matches with the current compartment are taken into account. That is, the fields that are configured within that record override the same fields that are configured in previous matching records.

A record has the following fields:

  • compartment – A wildcard filter expression that determines to which compartment names this record applies.
  • basePath – The path to where the persistence data is to be stored on the file system. Defaults to ./persistence/.
  • subPath – An additional path appended to basePath to form the full location on the file system.
  • shardCount – The number of shards for this compartment. Every shard is implemented as a separate folder with a separate sqlite database. Shards can live on different hosts, and can be invoked in parallel to increase throughput. The default value is 8.
  • partitionKeyLen – The maximum number of fields in a partition key. Default is 8.
  • sortKeyLen – The maximum number of fields in a sort key. Default is 8.
  • nodes – An optional list of app-id’s to which the shards are pinned. When configured, it is always the same runtime app-id that controls a given shard. Advantage is that this works when there is no shared file system available (that is, when each application has its own private file system). Disadvantage is that when the application is not available, that shard is not accessible. We are still in the process of implementing redundancy, so that every shard is replicated on multiple places. That would solve the mentioned disadvantage.
    The default value is no nodes, which works well when all runtime applications have access to the same shared file system.

By default, all compartments that start with fs. are stored in ./persistence/ using 8 shards and no list of nodes.

Some settings can be overriden via command-line arguments and/or environment variables:

  • The default number of shards can be set via the --darlean-runtime-persistence-fs-shardcount command-line parameter or the DARLEAN_RUNTIME_PERSISTENCE_FS_SHARDCOUNT environment variable.
  • The maximum number of shards for any compartment can be set via the --darlean-runtime-persistence-fs-maxshardcount command-line parameter or the DARLEAN_RUNTIME_PERSISTENCE_FS_MAXSHARDCOUNT environment variable. When individual compartment configurations have a higher shard count configured, they are topped off at the max shard count.
  • The default base path for any compartment can be set via the --darlean-runtime-persistence-fs-basepath command-line parameter or the DARLEAN_RUNTIME_PERSISTENCE_FS_BASEPATH environment variable.