從mysql 5.6.5版本號開始支持GTID，如今大部分生產環境也開始慢慢切換到GTID模式。為什麽要切換到GTID復制模式？

● Automatic fail-over scripts are easy to implement now.

因為改變GTID_MODE 變量須要重新啟動mysql，所以這讓我們不停庫切換到GTID變得不可能。以下有兩篇文章介紹了怎樣在線切換,booking.com和facebook的兩種方案

In a previous post I was talking about How to Setup MySQL Replication using the classic method (based on binary logs information). In this article I‘ll go through the transaction-based replication implementation using GTID in different scenarios.

The following topics will be covered in this blog:

• What is the concept of GTID protocol?
• GTID Replication Implementation
  • Fresh Installations
  • Adding New Slave
• Migration from classic replication to GTID replication
  • How to perform the migration?

  • Is online migration from classic to GTID replication available?
  • Workaround ?

    ?

• GTID Benefits

What is the concept of GTID protocol?

GTID is a Global Transaction IDentifier which introduced in MySQL 5.6.5. It‘s not only unique on the server it was originated but it‘s unique among all servers in a replication setup.
GTID also guarantee consistency because once a transaction is committed on a server, any other transaction having the same GTID will be ignored, i.e. a committed transaction on a master will be applied only once on the slaves.

GTID consists of two parts separated by a column {source_id:transactions_id}.

WHERE

• source_id: Normally the server‘s UUID on which the transaction originates. e.g. "b9b4712a-df64-11e3-b391-60672090eb04" .
• transaction_id: A sequence number determining the order of the committed transaction.

The following is the GTID for the third transaction on a server having the uuid "b9b4712a-df64-11e3-b391-60672090eb04":
b9b4712a-df64-11e3-b391-60672090eb04:3

As a new protocol in MySQL there is a set of new related variables, the following are the most important ones (IMHO):

• gtid-mode: ON|OFF to enable or disable GTID, this is not a Boolean variable (0 and 1 are not acceptable).
• enforce-gtid-consistency: prevent executing the non transactionally safe statements, like:
  
  • CREATE TABLE .. SELECT.
  • CREATE TEMPORARY TABLE (inside a transaction).
  • Statements that update nontransactional tables inside a transaction.
• gtid_purged: The set of transactions that have been purged from the binary logs.
• gtid_executed: The set of transactions which already executed on that server.
• gtid_next: The GTID which will be used for the next transaction.

GTID Replication Implementation

Fresh Installations

Fresh installation means that there‘s no data yet in the master or in other words, we are building a replication setup from scratch.

The implementation process is divided into two parts:

MASTER‘S SIDE CONFIGURATION:

• Add the following variables to the MySQL configuration file (my.cnf):

  [mysqld]
  server-id=1		
  log-bin=mysql-bin
  binlog_format=ROW
  gtid-mode=on
  enforce-gtid-consistency
  log-slave-updates

• Restart MySQL so that configuration changes take place:

  shell> service mysql restart

• Create a MySQL user to be used by the slave:

  SQL> GRANT REPLICATION SLAVE ON *.* TO [email protected]_ip‘ IDENTIFIED BY ‘s3cret‘;

SLAVE‘S SIDE CONFIGURATION:

• Add the following variables to the my.cnf file:

  [mysqld]
  server-id=2
  log-bin=mysql-bin
  binlog_format=ROW
  relay_log=relay-log
  skip-slave-start
  gtid-mode=on
  enforce-gtid-consistency
  log-slave-updates
  

• Restart MySQL so that configuration changes take place:

  shell> service mysql restart

• Set the master information on the slave‘s:

  Unlike the classic method, we don‘t need the master‘s binary log information and only what we need is to specify MASTER_AUTO_POSITION=1 instead:

  SQL> CHANGE MASTER TO
      -> MASTER_HOST=‘master_ip‘,
      -> MASTER_PORT=3306,
      -> MASTER_USER=‘slave_user_name‘,
      -> MASTER_PASSWORD=‘s3cret‘,
      -> MASTER_AUTO_POSITION=1;

• Start replication:

  SQL> START SLAVE;

• Check the replication status:

  SQL> show slave status\G
                 Slave_IO_State: Waiting for master to send event
                    Master_Host: 127.0.0.1
                    Master_User: gtid_repl
                    Master_Port: 3320
                  Connect_Retry: 60
                Master_Log_File: mysql-bin.000007
            Read_Master_Log_Pos: 191
                 Relay_Log_File: relay-log.000004
                  Relay_Log_Pos: 401
          Relay_Master_Log_File: mysql-bin.000007
               Slave_IO_Running: Yes
              Slave_SQL_Running: Yes
                Replicate_Do_DB: 
  			.
  			.
  			.
  			.
  
             Retrieved_Gtid_Set: b9b4712a-df64-11e3-b391-60672090eb04:1-2
              Executed_Gtid_Set: b9b4712a-df64-11e3-b391-60672090eb04:1-2
                  Auto_Position: 1

Adding New Slave

It‘s a very simple process to add a new slave to a running replication (or setup replication with existing data) where GTID is being used:

• Backup the master server

  shell> mysqldump -u root -p --all-databases --flush-privileges --single-transaction --flush-logs --triggers --routines --events --hex-blob >/path/to/backupdir/full_backup-$TIMESTAMP.sql

• On the new slave, use the same MySQL configuration as described above (except the server id which should be unique) and restart it.
• Restore the backup file taken from the master.
• Use change master to with MASTER_AUTO_POSITION=1
• Start the slave.

Is it so simple like that!! How did the slave know the backup position?

What if some transactions were executed on the master after that backup?

Actually, when GTID is enabled, mysqldump includes the last transaction ID (GTID) at the time of taking the backup:

--
-- GTID state at the beginning of the backup 
--

SET @@GLOBAL.GTID_PURGED=‘b9b4712a-df64-11e3-b391-60672090eb04:1-7‘;

After restoring the backup the variable GTID_EXECUTED will be equal to GTID_PURGED (the above value) and when the slave starts it first sends the range of GTIDs it has executed (GTID_EXECUTED) to the master so that the master can sends back every missing transaction which was not applied yet on the slave.

Migration from classic replication to GTID replication

How to perform the migration?

To migrate an already running replication using the classical method to GTID replication, the following steps should be done:

• Ensure that all servers (master and slaves) are in the same point by setting the master server as read only (SET GLOBAL read_only=ON;) and wait until all slaves catch up the master‘s data.
• Shutdown MySQL on all servers and add the GTID variables to the configuration files.
• Beside the GTID variables, add read-only to the master‘s configuration and skip-slave-start to the slaves configurations.
• Start MySQL service on all servers.
• Issue the change master command with MASTER_AUTO_POSITION=1 on all slaves and then start them.
• Make the master writable again by SET GLOBAL read_only=OFF; (don‘t forget to remove/hash it from the master‘s my.cnf file as well).

Is online migration from classic to GTID replication available?

At the time of writing this article, the online migration is not applicable - as you can see from the above steps - we have to shutdown ALL servers at the same time and that is because of two reasons:

• GTID can NOT be enabled online because GTID_MODE is a read only variable (having this variable to be dynamic is already inOracle‘s plan).
• Replication can NOT be established between two or more servers having different values for GTID_MODE, i.e. either GTID is enabled on ALL servers or disabled on ALL servers.

Workaround ?

?

There‘s a feature request (by MySQL Devs team at Booking.com) to have an extra GTID mode (ANONYMOUS_IN-GTID_OUT) which allows a slave to receives anonymous transactions (transactions from master having GTID_MODE = OFF which do not have GTIDs) and assigns GTIDs for those transactions. In this case, this slave could be used as an intermediate server between master having GTID disabled and slaves having GTID enabled (it will be slave for the master and master for the other slaves)

The online migration steps would be:

• Restart a slave (lets name it slaveA) using the GTID_MODE = ANONYMOUS_IN-GTID_OUT.
• Rolling restart to the other slaves to use the normal GTID_MODE=ON and pointing them to slaveA as a new master.
• Point the application to write to slaveA instead of the old master.
• Restart the old master to use GTID_MODE=ON and having slaveA as a master.

Note: This is not yet available in Oracle binaries

More information on this could be find here.

GTID Benefits

• Simplifies the setup of MySQL replication as master‘s binary logs information is not needed anymore (binary log file name and position).
• Consistency is guaranteed between master and slave as the committed transaction on the master will be applied only once on the slave.
• Simple to determine whether masters and slaves are consistent or not.
• Fail-over process is much easier. When the master fail to operate, no need to calculate a slave‘s binary logs information before promoting it to be new master. MASTER_AUTO_POSITION=1 will do the job as all transactions in all servers inside the replication have the same GTID.
• Automatic fail-over scripts is now much easier to implement.

In a future post, I will write about how to troubleshoot GTID replication.

Online GTID rollout now available in Percona Server 5.6

Global Transaction IDs (GTIDs) are one of my favorite features of MySQL 5.6. The main limitation is that you must stop all the servers at the same time to allow GTID-replication. Not everyone can afford to take a downtime so this requirement has been a showstopper for many people. Starting with Percona Server 5.6.22-72.0 enabling GTID replication can be done without almost no downtime. Let’s see how to do it.

Implementation of the Facebook patch

Finding a solution to migrate to GTIDs with no downtime is not a new idea, and several companies have already developed their own patch. The 2 best known implementations are the one from Facebook and the one from Booking.com.

Both options have pros and cons, and we finally chose to port the Facebook patch and add a new setting (gtid_deployment_step).

Performing the migration

Let’s assume we have a master-slaves setup with 4 servers A, B, C and D. A is the master:

The 1st step is to take each slave out of rotation, one at a time, and set gtid_mode = ON andgtid_deployment_step = ON (and also log_bin, log_slave_updates andenforce_gtid_consistency).

gtid_deployment_step = ON means that a server will not generate GTIDs when it executes writes, but it will record a GTID in its binary log if it gets an event from the replication stream tagged with a GTID.

The 2nd step is to promote one of the slaves to become the new master (for instance C) and to disablegtid_deployment_step. It is a regular slave promotion so you should do it the same way you deal with planned slave promotions (for instance using MHA or your own scripts). Our patch doesn’t help you do this promotion.

At this point replication will break on the old master as it has gtid_mode = OFF andgtid_deployment_step = OFF.

Don’t forget that you need to use CHANGE MASTER TO MASTER_AUTO_POSITION = 1 to enable GTID-based replication.

The 3rd step is to restart the old master to set gtid_mode = ON. Replication will resume automatically, but don’t forget to set MASTER_AUTO_POSITION = 1.

The final step is to disable gtid_deployment_step on all slaves. This can be done dynamically:

and you should remove the setting from the my.cnf file so that it is not set again when the server is restarted.

Optionally, you can promote the old master back to its original role.

That’s it, GTID replication is now available without having restarted all servers at the same time!

Limitations

At some point during the migration, a slave promotion is needed. And at this point, you are still using position-based replication. The patch will not help you with this promotion so use your regular failover scripts. If you have no scripts to deal with that kind of situation, make sure you know how to proceed.

Also be aware that this patch provides a way to migrate to GTIDs with no downtime, but not a way to migrate away from GTIDs with no downtime. So test carefully and make sure you understand all the new stuff that comes with GTIDs, like the new replication protocol, or how to skip transactions.

Other topologies

If you are using master-master replication or multiple tier replication, you can follow the same steps. With multiple tier replication, simply start by setting gtid_mode = ON and gtid_deployment_step = ON for the leaves first.

Conclusion

If you’re interested by the benefits of GTID replication but if taking a downtime has always scared you, you should definitely download the latest Percona Server 5.6 and give it a try!

Migration from classic replication to GTID replication(Online)