What now, I had to dig deep into MySQL Replication
- how it works
- what can probably cause the lag
- an approach that minimizes or eliminates it
How MySQL Replication Works
On the master
On the replica
1. IO thread
This process connects to a master, reads binary log events from the master as they come in and just copies them over to a local log file called relay log.
Even though there’s only one thread reading the binary log from the master and one writing relay log on the slave, very rarely copying of replication events is a slower element of the replication. There could be a network delay, causing a steady delay of a few hundred milliseconds.
If you want to see where IO thread currently is, check the following in “show slave status \G”
Master_Log_File – last file copied from the master (most of the time it would be the same as last binary log written by a master)
Read_Master_Log_Pos – binary log from the master is copied over to the relay log on the slave up until this position.
And then you can compare it to the output of “show master status/G” from the master.
2. SQL thread
The second process – SQL thread – reads events from a relay log stored locally on the replication slave (the file that was written by IO thread) and then applies them as fast as possible.
Going back to “show slave status /G”, you can get the current status of SQL thread from the following variables:
Relay_Master_Log_File – binary log from the master, that SQL thread is “working on” (in reality it is working on relay log, so it’s just a convenient way to display information)
Exec_Master_Log_Pos – which position from the master binary log is being executed by SQL thread.
Why Replication Lag Occurred
Caught The CulpritLet me take you through my journey how I crossed the river.
First, I took the reference to multiple blogs and started gobbling my mind with possible reasons suggesting
- Hardware Faults (getting RAID in degraded mode)
- MySQL Config Updates
- setting sync_binlog=1
- enabling log_slave_updates
- setting innodb_flush_log_at_trx_commit=1
- updating slave_parallel_workers to a higher value
- changing slave_parallel_type to support more parallel workers
- Restarting Replication
And finally, I found one, my DBA friend who suggested me to look for the Binary Log Format that I am using. Let's see what it is
Binary Logging FormatsThe server uses several logging formats to record information in the binary log. The exact format employed depends on the version of MySQL being used. There are three logging formats:
STATEMENT: With statement-based replication, every SQL statement that could modify data is logged on the master. Then those SQL statements are replayed on the slaves against the same dataset and in the same context. There is always less data that is to be transferred between the master and the slave. But, the data inconsistency issue between the master and the slave that creeps up due to the way this kind of replication works.
ROW: With row-based replication, every “row modification” is logged on the master and is then applied to the slave. With row-based replication, each and every change can be replicated and hence this is the safest form of replication. On a system that frequently UPDATE a large number of rows, it produces very large update logs and generates a lot of network traffic between the master and the slave.
MIXED: A third option is also available: mixed logging. With mixed logging, statement-based logging is used by default, but the logging mode switches automatically to row-based in certain cases.
Changing Binary Log FormatThe Binary Log Format is updated on Master MySQL server and requires MySQL service restart to reflect. It can be done for Global, Runtime or Session.
- set at runtime with --binlog-format=format
- setting the global (with the SUPER privilege)
- session value of the binlog_format server variable
So, earlier I was using STATEMENT BinLog Format, which is default one. Since I switched to MIXED BinLog Format, I am very delighted to share the below stats.
Current status of Master Read and Slave Execute position difference and Slave Lag (in sec), both are ZERO.