Center Replication¶
Overview¶
Data replication across multiple centers is supported by AMI for horizontal scaling. This can either be realtime replication -- data subscription from one center to another -- or for transfering some historical data across centers.
Center replication is done via the AMIDB Shell Tool and requires the following:
- At least 2 AMI centers running, either locally or accessible via browser (see this guide to configure two instances of AMI on the same machine)
- the table schema of interest predefined on the source and target center(s).
Realtime Replication¶
Realtime replication allows for subscription to data from a different (source) center which is then replicated on the target center. The source table must be stored as a Public table with Broadcast
enabled (set to true
) in order to be successfully replicated into the target center.
See the following documentation to create and configure Public tables in a center. Configuring the RefreshPeriodMs
of the source table will allow you to adjust for throughput, performance, and how often updates are pushed:
- Lower
RefreshPeriodMs
means updates are pushed more frequently which potentially means lower throughput. - Higher
RefreshPeriodMs
could mean higher throughput and better performance but fewer updates.
Adjust your values as per your requirements.
Replication Procedures¶
Replication is done in the AMIDB Shell Tool using 'procedures' which are defined below.
__ADD_CENTER(CenterName String, Url String, Certfile String, Password String)
-
Add a center with the given properties:
CenterName
(Required): Specified name for center (source)Url
(Required): Url of center (host: ami.center.port)Certfile
: Path to the certfile (optional)Password
: Password if a certfile is given (optional)
__REMOVE_CENTER(CenterName String)
- Remove center by name
__ADD_REPLICATION(Definition String, Name String, Mapping String, Options String)
-
Add a replication (target table) of a source table with the following properties:
Definition
:Target_TableName
=Source_CenterName.Source_TableName
orSource_CenterName.TableName
(if both source and target have the same table name)Name
: Name for the replicationMapping
(Optional): Mappings to be applied for the tables, (key value delimited by comma) e.g:"target_col_name=source_col_name"
or"(act=account,symbol=symbol,value=(px\*qty))"
. Pass in null to skip mapping.Options
(Optional): options to clear the replication on"Clear=onConnect"
or"Clear=onDisconnect"
or"Clear=Off"
- If configured, the replicated table will clear when the center connects or disconnect
__REMOVE_REPLICATION(Name String)
- Remove a named replication table
SHOW CENTERS
-
Shows properties of all replication sources. Properties shown include:
CenterName
(String)URL
(String)CertFile
(String)Password
(String)Status
(String): shows whether the replication source is connected or notReplicatedTables
(int): Number of replicated tablesReplicatedRows
(int): Number of replicated rows
SHOW REPLICATIONS
-
Shows properties of all replications. Properties shown include:
ReplicationName
(String)TargetTable
(String)SourceCenter
(String)SourceTable
(String)Mapping
(String): shows the mapping for the replicationOptions
(String): shows options to clear replications onReplicatedRows
(int): Number of replicated rowsStatus
(String): shows whether a successful connection is established to the target table
Note
Removing the replication will clear all the copied entries on the target side's table, regardless of the options.
Realtime Replication Example¶
The following is a simple guide for setting up a realtime replication from one center to another, both hosted on the same local machine. Enter the following command in the target center's AMIDB shell tool:
This creates a new callable center in the target center named source
which is hosted on port 3270
. This assumes the target AMI instance is hosted on a different port, for example 4270
. It is important that you configure your AMI instances correctly to avoid port conflicts, especially if using multiple instances from the same machine.
To create a replicated table, do the following:
This copies the table mytable
from the source center into the target table dest_table
.
AMI Center Persist Files¶
There are two persist files responsible for maintaining the replications, __REPLICATION.dat
and __CENTER.dat
. We do not recommend modifying the contents of these files manually, these files may have strict formatting and could cause serious issues if altered incorrectly.
Persist File Location¶
By default, these files are located in the amione/persist
directory. This may not be the case if the persist directory has been changed by the property ami.db.persist.dir
or the system tables directory has been changed by the property ami.db.persist.dir.system.tables
.
To change only the __REPLICATION.dat
and __CENTER.dat
locations, add the following property to your local.properties
file:
__REPLICATION.dat
Format:
__CENTER.dat
Format:
Historical Replication¶
To transfer historical data from one center to another requires the following:
- a source table in some center
- a target centert containing the historical table
Unlike realtime replication, the historical table does not need to subscribe to a realtime feed from the source center. This can be done manually or executed at some given timeframe e.g daily.
Historical Replication Example:¶
In this example, the source center contains a Public table with some the following schema:
Similarly, in the historical center, we have a table with the same name but a different schema:
NB: D is an auto-generated incrementing unique id for the row which unique across all tables (see Reserved Columns on Public Tables).
Next we will use the below procedure to transfer the historical data. The arguments for the procedure are as follows:
tableName
- this is the table you will be transferring data from and to (ie. DataTable)histCenter
- this is the name of the datasource where the historical data will sent towhereClause
- argument that can be used to get the data you want to transfer (ie. you may want to send across data where Date == 20220101)batchSize
- argument to specify the number of data rows to send across in each go
Note that this procedure uses column D to decide which data to send.