Connecting the Controller¶

Shardmonster relies on a connection to a controller. This is where all the shard metadata is stored. It is recommended that metadata is stored on a dedicated cluster. This helps ensure that reads/writes to metadata are very fast.

import shardmonster
shardmonster.connect_to_controller(
    "mongodb://localhost:27017/?replicaset=cluster-1",
    "sharding_db"

Activate Caching¶

Shardmonster makes use of caching of metadata to ensure that queries are as fast as possible. The cache length is set in seconds. All migrations are paused for at least as long as the cache length to ensure caches are clear before writes are moved to new locations. Having a high cache time will make migrations slower. Having a low cache time will result in more hits to the metadata cluster.

shardmonster.activate_caching(5)

Describe Clusters¶

A cluster describes a replica set that is to store data. Typically, you will have multiple clusters with data spread across them.

shardmonster.ensure_cluster_exists(
    'cluster-1', 'mongodb://localhost:27017/?replicaset=cluster-1')

Create Realms¶

A realm is a sharded collection. To create a realm you need to have defined a field to use as a shard field - this will be used to determine on which shard a document belongs.

The shard field must only contain strings or integers. No other data type is currently supported here.

# Create a realm called messages that is a sharded version of the
# messages_coll collection. By default shards will go to some_db on
# cluster-1. The data will be sharded by the field called "account".
shardmonster.ensure_realm_exists(
    'messages', 'account', 'messages_coll', 'cluster-1/some_db')

Preparing for Queries¶

To actually do sharded queries and inserts you will need a handle to a shard aware collection.

sharded_collection = \
    shardmonster.make_collection_shard_aware("messages")
sharded_collection.insert({"text": "Hello!", "account": 5})

Move some data around¶

Before a shard can be moved to a new cluster it must be first placed at rest at its current location.

# Following on from the previous block...
# Set all data in the messages realm with an account value of 5 to be at
# rest on cluster-1. As the data is already there this does do any movement
# of data.
shardmonster.set_shard_at_rest('messages', 5, 'cluster-1/some_db')

Once this is done the data can be migrated to a new location:

# This moves data from the messages collection with an account value of 5
# to a different cluster. The method returns when it is completed.
shardmonster.do_migration('messages', 5, 'cluster-2/some_other_db')

Where is my data?¶

After you’ve been using shardmonster for some time you might want some help interrogating your data and finding out where it is.

>>> shardmonster.where_is('messages', 5)
'cluster-2/some_other_db'

Realms¶

A realm is a mapping describing a collection’s sharding strategy. It consists of documents of the form:

{
    name: 'some-realm',
    shard_field: 'some-field',
    collection: 'collection-name',
    default_location: 'cluster/database'
}

Shard data¶

There is a shards collection that stores documents of the form:

{
    realm: 'some-realm',
    shard_key: 'some-key',
    status: 'migrating-copy'
            | 'migrating-sync'
            | 'post-migration-paused-at-source'
            | 'post-migration-paused-at-destination'
            | 'post-migration-delete'
            | 'at-rest',
    location: 'cluster/database'
}

This describes the position of the data where some-field is equal to some-key. In the event that the data is in a migrating phase then there will be additional fields:

new_location: 'cluster2/database',

Migration phases¶

Copy

The copy phase is copying all data that matches the shard query from the current location to the new location. Once this phase is finished, the sync phase begins.

During this phase all reads and writes will go to the original location of the data.

Sync

The sync phase is where the oplog is replayed since the start of the migration.

Once the oplog is replayed and we are close to realtime the post migration pause begins.

During this phase all reads and writes will go to the original location of the data.

Post migration pause at source

Once data has been migrated the shard will enter a pause state. This allows for someone (or something) to verify that the data has been migrated successfully and everything is OK. This state shouldn’t really be needed, but, it’s a nice safety valve during testing.

During this phase all reads and writes will go to the original location of the data. The oplog continues to be synced during this period.

Post migration pause at destination

Once any data validation has been performed the shard will be moved to “paused at destination”. During this phase all reads will go to the new location of the data. Writes will be suspended. The oplog continues to be synced during this period to catch any stragglers.

Due to the suspension of writes during this period it is expected that this phase will be very short.

Post migration delete

Once a shard has been migrated the original location should have the data removed. Doing this deletion helps with reducing the amount of query customisation that has to happen to cope with data being in two places at once.

Crucially, the oplog is NOT synced during this phase. That would copy the deletes and lead to all kinds of sadness. Specifically, the removal of most of the data.

Only once this phase is completed is the shard moved to at rest and the location field is updated.

At rest

Data that is at rest is no longer being migrated in any form. Lookups for shard information will be cached for a short length of time.

Running tests¶

> cp sample_test_settings.py test_settings.py

Update test_settings.py to reflect your local environment.

> python setup.py nosetests

Building Docs¶

> python setup.py build_sphinx