MySQL Adventures: GTID Replication In AWS RDS

You all heard about that today AWS announced that RDS is started to support GTID Transactions. I’m a great fan of RDS but not for GTID. Since RDS has better settings and configurations to perform well. Many of you people read about the AWS What’s new page regarding GTID. But here we are going to talk about the actual benefits and drawbacks.

RDS supports GTID on MySQL 5.7.23 or later. But AWS released this version on Oct10 (two days before). So, for now, this is the only version which supports GTID.

NOTE: GTID supports only for RDS, its not available for Aurora. It may support in future)

Before configuring the GTID, lets have a look at what is GTID?

• GTID stands for Global Transaction Identifier.
• It’ll generate a unique ID for each committed transaction.
• The GTID referred as server_UUID:transaction_id
• GTID replication is a better solution in a multi-master environment.
• To learn more about GTID, hit here.

GTID in RDS:

1. You can use GTID only on RDS, not in Aurora.

2. There are 4 types of GTID modes in RDS.

From AWS Docs,

• OFF — No GTID. Anonymous transactions are replicated.
• OFF_PREMISSIVE — New transactions are anonymous transactions, but all transactions can be replicated.
• ON_PERMISSIVE specifies that new transactions are GTID transactions, but all transactions can be replicated.
• ON specifies that new transactions are GTID transactions, and a transaction must be a GTID transaction to be replicated.

3. The default GTID mode in RDS is OFF_PREMISSIVE.

4. RDS support 3 Consistency levels for GTID.

• OFF allows transactions to violate GTID consistency.
• ON prevents transactions from violating GTID consistency.
• WARN allows transactions to violate GTID consistency but generates a warning when a violation occurs.

Replication between RDS to EC2 with GTID:

I have launched an RDS and enabled the below parameters in the Parameter group.

gtid-mode = ONenforce_gtid_consistency = ON

#From RDS ConsoleBackup Retention Period = 2 Days (you can set this as you need)

Create a database with some data:

CREATE DATABASE searcedb; USE searcedb; CREATE TABLE dba_profile   (      id     INT auto_increment PRIMARY KEY,      name   VARCHAR(10),      fav_db VARCHAR(10)   ); INSERT INTO dba_profile (name, fav_db) VALUES ('sqladmin', 'MSSQL');INSERT INTO dba_profile (name, fav_db) VALUES ('mac', 'MySQL');

Create the user for replication:

CREATE USER 'rep_user'@'%' IDENTIFIED BY 'rep_user';  GRANT REPLICATION slave ON *.* TO 'rep_user'@'%' IDENTIFIED BY 'rep_user';  

FLUSH PRIVILEGES;

Take DUMP on RDS:

mysqldump \-h sqladmin-mysql-rds.xxxxx.rds.amazonaws.com \-u sqladmin -p \--routines \--events \--triggers  \--databases searcedb > dbdump.sql

The above command will dump the searcedb with stored procedures, triggers, and events. If you have multiple databases the use --databases db1 db2 db3. Generally, for replicating the database we use --master-data=2 to get the binlog file and position. But this is GTID replication. So it has the last executed GTID information in the dump file.

$ grep PURGED dbdump.sql

SET @@GLOBAL.GTID_PURGED='eac87cf0-cdfe-11e8-9275-0aecd3b2835c:1-13';

You may get this warning message during the dump. It just saying that the dump file contains the Purge GTID command.

Warning: A partial dump from a server that has GTIDs will by default include the GTIDs of all transactions, even those that changed suppressed parts of the database. If you don't want to restore GTIDs, pass --set-gtid-purged=OFF. To make a complete dump, pass --all-databases --triggers --routines --events.

We can’t restore the dump on the MySQL where GTID is not enabled.

Restore the dump on EC2:

Enable the GTID before restoring the DB. RDS will replicate all the databases. In MySQL there some RDS related tables which are not available in EC2 MySQL. There is a table called heartbeat will keep inserting data about the RDS health. This statements also will be replicated to Slave. replicate-ignore-db=mysql will not work here. Because the statements are specifically mentioned the DB name. So we need to ignore these tables on Slave by replicate-ignore-tables.

Enable GTID on EC2:

vi /etc/mysql/mysql.conf.d/mysqld.cnf

server-id                = 1234gtid_mode                = ONenforce_gtid_consistency = ONlog-binlog-slave-updates

#Ignore tablesreplicate_ignore_table = mysql.rds_configuration,mysql.rds_global_status_history_old,mysql.rds_heartbeat2,mysql.rds_history,mysql.rds_replication_status,mysql.rds_sysinfo

#Restart MySQLservice mysql restart

Restore the DB:

mysql -u root -p < dbdump.sql

Establish the replication:

CHANGE MASTER TO MASTER_HOST="sqladmin-mysql-rds.xxxxx.rds.amazonaws.com", MASTER_USER="rep_user", MASTER_PASSWORD="rep_user", MASTER_PORT=3306, MASTER_AUTO_POSITION = 1;

START SLAVE;

Check the Replication:

show slave status\G

Slave_IO_State: Waiting for master to send event

Master_Host: sqladmin-mysql-rds.xxxx.rds.amazonaws.com

Master_Log_File: mysql-bin-changelog.000030

Read_Master_Log_Pos: 551

Relay_Log_File: ip-172-31-29-127-relay-bin.000002

Relay_Log_Pos: 444

Relay_Master_Log_File: mysql-bin-changelog.000030

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

...........

Executed_Gtid_Set: eac87cf0-cdfe-11e8-9275-0aecd3b2835c:1-22

Auto_Position: 1

Lets insert some rows:

On Master:

INSERT INTO dba_profile (name, fav_db) VALUES ('surface', 'PostgresqlSQL');

On Slave:

mysql -u root -pEnter Password:

mysql> select * from searcedb.dba_profile;+----+----------+------------+

| id | name     | fav_db     |

+----+----------+------------+

|  1 | sqladmin | MSSQL      |

|  2 | mac      | MySQL      |

|  3 | surface  | PostgreSQL |

+----+----------+------------+

3 rows in set (0.00 sec)

Enable GTID Replication on existing RDS master and slave/RDS Read Replica:

On Mater RDS:

1. Make sure we are running MySQL 5.7.23 or later.

2. Use the custom parameter group.

3. In the Parameter group,

gtid-mode = ONenforce_gtid_consistency = ON

#From RDS ConsoleBackup Retention Period = 2 Days (you can set this as you need)

4.Need to reboot the RDS to apply these changes.

On Slave:

1. Use the custom parameter group (it's a good practice to have separate parameter group for Mater and Slave)
2. If you are using RDS Read Replica, then in the Parameter group,

gtid-mode = ONenforce_gtid_consistency = ON

3. If you are using EC2 as a Replica then in my.cnf

gtid_mode                = ONenforce_gtid_consistency = ON

4. Reboot the Read Replica.

5. Still, your read replica will use Binlog Position based replication. Run the below command to Start Replication with GTID.

CALL mysql.rds_set_master_auto_position(1);

How to Disable GTID in RDS:

Caution: You need to follow these step as it is. Else it’ll break your replication and you may lose some transactions.

Caution: You need to follow these step as it is. Else it’ll break your replication and you may lose some transactions.

1. Disable the get auto position for replication.

CALL mysql.rds_set_master_auto_position(0);

2. In the parameter group, set gtid-mode = ON_PREMISSIVE

3. Reboot the Replica.

4. Again in the parameter group, set gtid-mode = OFF_PREMISSIVE

5. Make sure all GTID transactions are applied on the Replica. To check this follow the below steps.

• On Master, get the current binlog file name and its position.

show master status\G;

*************************** 1. row ***************************

File: mysql-bin-changelog.000039

Position: 827

Binlog_Do_DB:

Binlog_Ignore_DB:

Executed_Gtid_Set: 2f5e8f57-ce2a-11e8-8874-0a1b0ee9b48e:1-27

• Make a note of the output.
• On each Read Replica, Run the below command. (replace the binlog filename and position with your output)

SELECT MASTER_POS_WAIT('mysql-bin-changelog.000039', 827);

• If you see the result is 0, then we are fine. Else the need to wait for some time and again the run the same command.

+----------------------------------------------------+

| MASTER_POS_WAIT('mysql-bin-changelog.000039', 827) |

+----------------------------------------------------+

|                                                  0 |

+----------------------------------------------------+

• Once we confirmed that all GTID replications are applied then, in the Read Replica Parameter group we can disable the GTID permanently.

gtid_mode – OFFenforce_gtid_consistency – OFF

• Then do the same on the Master RDS parameter group.

Disable GTID on EC2 Slave:

1. Switch replication from auto position to binlog position.

STOP SLAVE;change master to master_auto_position = 0;

2. Verify all the GTID transactions are applied. Run the below command. (replace the binlog filename and position with your output)

SELECT MASTER_POS_WAIT('mysql-bin-changelog.000040', 194);

• If you see the result is 0, then we are fine. Else the need to wait for some time and again the run the same command.

+----------------------------------------------------+

| MASTER_POS_WAIT('mysql-bin-changelog.000040', 194) |

+----------------------------------------------------+

|                                                  0 |

+----------------------------------------------------+

• Now, GTID parameters in my.cnf

vi /etc/mysql/mysql.conf.d/mysqld.cnf

# Remove the below lines:gtid_mode                = ONenforce_gtid_consistency = ON

Best Practice (from my personal thoughts):

• On Master: use ON_PERMISSIVE GTID mode. Since this will replicate both GTID and anonymous transactions.
• On Slave: Use GTID = ON, Because we need strong consistency.
• Finally, use GTID if it is necessary. Because I tried to change the GTID mode frequently on the Master Node, it breaks the replication.
• Don’t try to replicate MariaDB to MySQL. MariaDB has different GTID implementation.
• A few months back I read a blog which is written by Jean-François Gagné. He had done the anonymous transaction replication using a patched version of MySQL.