FEED Validator

for Atom and RSS and KML

Congratulations!

This is a valid RSS feed.

Recommendations

This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

line 178, column 0: description should not contain aria-level attribute (216 occurrences) [help]
```
<li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;" ...
```

Source: http://www.tusacentral.net/joomla/index.php/mysql-blogs?format=feed&type=rss

<?xml version="1.0" encoding="utf-8"?>

<?xml-stylesheet href="/joomla/plugins/system/jce/css/content.css?badb4208be409b1335b815dde676300e" type="text/css"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>TusaCentral - MySQL Blogs</title>
<description><![CDATA[]]></description>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs</link>
<lastBuildDate>Sat, 27 Apr 2024 07:15:02 +0000</lastBuildDate>
<generator>Joomla! - Open Source Content Management</generator>
<atom:link rel="self" type="application/rss+xml" href="http://www.tusacentral.net/joomla/index.php/mysql-blogs?format=feed&type=rss"/>
<language>en-gb</language>
<item>
<title>Is MySQL Router 8.2 Any Better?</title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/255-is-mysql-router-8-2-any-better</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/255-is-mysql-router-8-2-any-better</guid>
<description><![CDATA[In my previous article, <a href="https://www.percona.com/blog/comparisons-of-proxies-for-mysql/">Comparisons of Proxies for MySQL</a>, I showed how MySQL Router was the lesser performing Proxy in the comparison. From that time to now, we had several MySQL releases and, of course, also some new MySQL Router ones.
Most importantly, we also had MySQL Router going back to being a level 7 proxy capable of redirecting traffic in case of R/W operations (<a href="https://dev.mysql.com/doc/mysql-router/8.2/en/router-read-write-splitting.html" rel="nofollow">see this</a>).
All these bring me hope that we will also have some good improvements in what is a basic functionality in a router: routing.
So with these great expectations, I had to repeat the exact same tests I did in my previous tests, plus I tested for MySQL Router only the cost of encapsulating the select inside a transaction.
Just keep in mind that for all the tests, MySQL Router was configured to use the read/write split option.
<h2>The results</h2>
Given this is the continuation of the previous blog, all the explanations about the tests and commands used are in the <a href="https://www.percona.com/blog/comparisons-of-proxies-for-mysql/">first article</a>. If you did not read that, do it now, or it will be difficult for you to follow what is explained later.
As indicated, I was looking to identify when the first proxy would reach a dimension that would not be manageable. The load is all in creating and serving the connections, while the number of operations is capped at 100.
<a href="https://www.percona.com/blog/wp-content/uploads/2024/01/events_rate_82.png"><img src="https://www.percona.com/blog/wp-content/uploads/2024/01/events_rate_82-1024x608.png" alt="" width="600" height="356" class="aligncenter wp-image-93339 size-large" /></a>
As you can see, MySQL Router was reaching the saturation level and was unable to serve traffic at exactly the same time as the previous test.
<h2>Test two</h2>
When the going gets tough, the tough get going reprise ;) 
Let’s remove the –rate limitation and see what will happen. First, let us compare MySQL router versions only:
<a href="https://www.percona.com/blog/wp-content/uploads/2024/01/router_82_80_comparison_events.png"><img src="http://www.tusacentral.net/joomla/images/stories/ismysqlrouter82better/router_82_80_comparison_events.png" alt="router 82 80 comparison events" width="600" height="376" class="size-large wp-image-93348 aligncenter" /></a>
As we can see the MySQL Router version 8.2 is doing better up to 64 concurrent threads.
<a href="https://www.percona.com/blog/wp-content/uploads/2024/01/router_82_80_comparison_latency.png"><img src="https://www.percona.com/blog/wp-content/uploads/2024/01/router_82_80_comparison_latency-1024x680.png" alt="" width="600" height="398" class="size-large wp-image-93349 aligncenter" /></a>
Latency follows the same trend in old and new cases, and we can see that the new version is acting better, up to 1024 threads.
Is this enough to cover the gap with the other proxies? Which, in the end, is what we would like to see. 
<a href="https://www.percona.com/blog/wp-content/uploads/2024/01/events_norate_82.png"><img src="http://www.tusacentral.net/joomla/images/stories/ismysqlrouter82better/events_norate_82.png" alt="events norate 82" width="600" height="326" class="size-large wp-image-93337 aligncenter" /></a>
<a href="https://www.percona.com/blog/wp-content/uploads/2024/01/latency_norate_82.png"><img src="http://www.tusacentral.net/joomla/images/stories/ismysqlrouter82better/latency_norate_82.png" alt="latency norate 82" width="600" height="364" class="size-large wp-image-93342 aligncenter" /></a>
Well, I would say not really; we see a bit of better performance with low concurrent threads, but still not scaling and definitely lower than the other two.
Now let us take a look at the CPU saturation:
<a href="https://www.percona.com/blog/wp-content/uploads/2024/01/cpu_saturation.png"><img src="http://www.tusacentral.net/joomla/images/stories/ismysqlrouter82better/cpu_saturation.png" alt="cpu saturation" width="600" height="300" class="size-full wp-image-93334 aligncenter" /></a>
<a href="https://www.percona.com/blog/wp-content/uploads/2024/01/cpu.png"><img src="http://www.tusacentral.net/joomla/images/stories/ismysqlrouter82better/cpu.png" alt="cpu" width="600" height="300" class="size-full wp-image-93335 aligncenter" /></a>
Here, we can see how MySQL Router hits the top as soon as the rate option is lifted and gets worse with the increase of the running threads.
<h2>Test three</h2>
This simple test was meant to identify the cost of a transaction, or better, what it will cost to include selects inside a transaction.
<a href="https://www.percona.com/blog/wp-content/uploads/2024/01/read_events_trx.png"><img src="http://www.tusacentral.net/joomla/images/stories/ismysqlrouter82better/read_events_trx.png" alt="read events trx" width="600" height="356" class="size-large wp-image-93344 aligncenter" /></a>
<a href="https://www.percona.com/blog/wp-content/uploads/2024/01/latency_events_trx.png"><img src="http://www.tusacentral.net/joomla/images/stories/ismysqlrouter82better/latency_events_trx.png" alt="latency events trx" width="600" height="371" class="size-large wp-image-93340 aligncenter" /></a>
As we can clearly see, MySQL Router, when handling selects inside a transaction, will drop its performance drastically going back to version 8.0 performance.
<h3>Conclusions</h3>
To the initial question — Is MySQL Router 8.2 any better? — we can answer a small (very small) yes.
However, it is still far, far away from being competitive with ProxySQL (same proxy level) or with HAProxy. The fact it is not able to serve efficiently the requests also inside the lower set of concurrent threads, is disappointing.
Even more disappointing because MySQL Router is presented as a critical component in the MySQL InnoDB Cluster solution. How can we use it in the architectures if the product has such limitations?
I know that Oracle suggests to scale out, and I agree with them. When in need to scale with MySQL Router, the only option is to build a forest. However, we must keep in mind that each MySQL Router connects and queries the data nodes constantly and intensively. Given that it requires adding a forest of router nodes to scale, it is not without performance impact, given the increasing noise generated on the data nodes.
Anyhow also, if there is a theoretical option to scale, that is not a good reason to use a poor performing component.
I would prefer to use ProxySQL with Group Replication and add whatever script is needed in mysqlshell to manage it as Oracle is doing for the MySQL InnoDB cluster solution.
What also left me very unhappy is that MySQL InnoDB Cluster is one of the important components of the OCI offer for MySQL. Is Oracle using MySQL Router there as well? I assume so. Can we trust it? I am not feeling like I can.
Finally, what has been done for MySQL Router so far leads me to think that there is no real interest in making it the more robust and performing product that MySQL InnoDB Cluster deserves.
I hope I am wrong and that we will soon see a fully refactored version of MySQL Router. I really hope Oracle will prove me wrong.
Great MySQL to everyone.]]></description>
<category>MySQL</category>
<pubDate>Wed, 10 Jan 2024 18:05:47 +0000</pubDate>
</item>
<item>
<title>Export and import of MySQL passwords using caching_sha2 </title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/254-export-and-import-of-mysql-passwords-using-caching-sha2</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/254-export-and-import-of-mysql-passwords-using-caching-sha2</guid>
<description><![CDATA[Some fun is coming <a href="https://www.percona.com/blog/wp-content/uploads/2023/09/user-migration.jpg"><img src="https://www.percona.com/blog/wp-content/uploads/2023/09/user-migration-300x136.jpg" alt="" width="380" height="172" class="wp-image-91109 alignright" style="float: right;" /></a>
While I was writing the internal guidelines on how to migrate from MariaDB to Percona Server, I had to export the users accounts in a portable way. This given MariaDB uses  some non standard syntax brings me to first test some external tools such as Fred <a href="https://github.com/lefred/mysqlshell-plugins/wiki/user#getusersgrants">https://github.com/lefred/mysqlshell-plugins/wiki/user#getusersgrants</a> and our <a href="https://github.com/percona/percona-toolkit/blob/3.x/bin/pt-show-grants" rel="nofollow">PT-SHOW-GRANTS</a> tool.
Useless to say this had open a can worms, given first I had to fix/convert the specifics for MariaDB (not in the scope of this blog), then while testing I discover another nasty issue, that currently prevent us to easily export the new Passwords in MySQL 8 (and PS 8) when caching_sha2 is used. 
 
So what is the problem I am referring to?
Well the point is that when you generate passwords with caching_sha2 (default in mysql 8) the password generated can (will) contain characters that are not portable, not even between mysql 8. 
Let's see a practical example to understand.
If I use old mysql_native_password and I create a user such as: 
<pre class="lang:mysql decode:true">create user dba@'192.168.1.%' identified with mysql_native_password by 'dba';
</pre>
My authentication_string will be: 
<pre class="lang:mysql decode:true">root@localhost) [(none)]>select user,host,authentication_string,plugin from mysql.user where user ='dba' order by 1,2;
+------+-------------+-------------------------------------------+-----------------------+
| user | host | authentication_string | plugin |
+------+-------------+-------------------------------------------+-----------------------+
| dba | 192.168.1.% | *381AD08BBFA647B14C82AC1094A29AD4D7E4F51D | mysql_native_password |
+------+-------------+-------------------------------------------+-----------------------+
</pre>
At this point if you want to export the user:
<pre class="lang:mysql decode:true">(root@localhost) [(none)]>show create user dba@'192.168.1.%'G
*************************** 1. row ***************************
CREATE USER for dba@192.168.1.%: CREATE USER `dba`@`192.168.1.%` IDENTIFIED WITH 'mysql_native_password' AS '*381AD08BBFA647B14C82AC1094A29AD4D7E4F51D' REQUIRE NONE PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK PASSWORD HISTORY DEFAULT PASSWORD REUSE INTERVAL DEFAULT PASSWORD REQUIRE CURRENT DEFAULT
1 row in set (0.01 sec)
</pre>
You just need to use the text after the semicolon and all will work fine. Remember that when you want to preserve the already converted password you need to use the IDENTIFIED … AS <PW> not the BY or you will re-convert the password ;).
 Anyhow .. this is simple and what we are all used to. 
Now if you instead try to use caching_sha2 things will go differently:
<pre class="lang:mysql decode:true">root@localhost) [(none)]>create user dba@'192.168.4.%' identified with caching_sha2_password by 'dba';
Query OK, 0 rows affected (0.02 sec)
(root@localhost) [(none)]>select user,host,authentication_string,plugin from mysql.user where user ='dba' order by 1,2;
+------+-------------+------------------------------------------------------------------------+-----------------------+
| user | host | authentication_string | plugin |
+------+-------------+------------------------------------------------------------------------+-----------------------+
| dba | 192.168.1.% | *381AD08BBFA647B14C82AC1094A29AD4D7E4F51D | mysql_native_password |
| dba | 192.168.4.% | $A$005$@&%1H5iNQx|.l{N7T/GosA.Lp4EiO0bxLVQp8Zi0WY2nXLr8TkleQPYjaqVxI7 | caching_sha2_password |
+------+-------------+------------------------------------------------------------------------+-----------------------+
2 rows in set (0.00 sec)
</pre>
Probably you will not see it here given that while converted on your screen the special characters will be replaced, but the password contains invalid characters. 
If I try to extract the Create USER text I will get:
<pre class="lang:mysql decode:true"> (root@localhost) [(none)]>show create user dba@'192.168.4.%'G
*************************** 1. row ***************************
CREATE USER for dba@192.168.4.%: CREATE USER `dba`@`192.168.4.%` IDENTIFIED WITH 'caching_sha2_password' AS '$A$005$@&%1H5iNQx|.l{N7T/GosA.Lp4EiO0bxLVQp8Zi0WY2nXLr8TkleQPYjaqVxI7' REQUIRE NONE PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK PASSWORD HISTORY DEFAULT PASSWORD REUSE INTERVAL DEFAULT PASSWORD REQUIRE CURRENT DEFAULT
1 row in set (0.00 sec)</pre>
However if I try to use this text to generate the user after I drop it: 
<pre class="lang:mysql decode:true">(root@localhost) [(none)]>drop user dba@'192.168.4.%';
Query OK, 0 rows affected (0.02 sec)
(root@localhost) [(none)]>create user dba@'192.168.4.%' IDENTIFIED AS 'NQx|.l{N7T/GosA.Lp4EiO0bxLVQp8Zi0WY2nXLr8TkleQPYjaqVxI7';
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AS 'NQx|.l{N7T/GosA.Lp4EiO0bxLVQp8Zi0WY2nXLr8TkleQPYjaqVxI7'' at line 1
</pre>
Don’t waste time, there is nothing wrong in the query, except the simple fact that you CANNOT use the text coming from the authentication_string when you have caching_sha2. 
So? What should we do? 
The answer is easy, we need to convert the password into binary and use/store that. 
Let us try.
First create the user again:
<pre class="lang:mysql decode:true">(root@localhost) [(none)]>select user,host,authentication_string,plugin from mysql.user where user ='dba' order by 1,2;
+------+-------------+------------------------------------------------------------------------+-----------------------+
| user | host | authentication_string | plugin |
+------+-------------+------------------------------------------------------------------------+-----------------------+
| dba | 192.168.1.% | *381AD08BBFA647B14C82AC1094A29AD4D7E4F51D | mysql_native_password |
| dba | 192.168.4.% | $A$005$X>ztS}WfR"k~aH3Hs0hBbF3WmM2FXubKumr/CId182pl2Lj/gEtxLvV0 | caching_sha2_password |
+------+-------------+------------------------------------------------------------------------+-----------------------+
2 rows in set (0.00 sec)
(root@localhost) [(none)]>exit
Bye
[root@master3 ps80]# ./mysql-3307 -udba -pdba -h192.168.4.57 -P3307
...
(dba@192.168.4.57) [(none)]>
</pre>
OK as you can see I create the user and can connect, but as we know the PW is not portable.
Let us convert it and create the user:
<pre class="lang:mysql decode:true">(root@localhost) [(none)]>select user,host,convert(authentication_string using binary),plugin from mysql.user where user ='dba' and host='192.168.4.%' order by 1,2;
+------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
| user | host | convert(authentication_string using binary) | plugin |
+------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
| dba | 192.168.4.% | 0x2441243030352458193E107A74537D0157055C66527F226B7E5C6148334873306842624633576D4D32465875624B756D722F434964313832706C324C6A2F674574784C765630 | caching_sha2_password |
+------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
</pre>
So the password is: 
<pre class="lang:mysql decode:true">0x2441243030352458193E107A74537D0157055C66527F226B7E5C6148334873306842624633576D4D32465875624B756D722F434964313832706C324C6A2F674574784C7656
</pre>
Let us use it:
<pre class="lang:mysql decode:true">(root@localhost) [(none)]>drop user dba@'192.168.4.%';
Query OK, 0 rows affected (0.02 sec)
(root@localhost) [(none)]>create user dba@'192.168.4.%' IDENTIFIED with 'caching_sha2_password' AS 0x2441243030352458193E107A74537D0157055C66527F226B7E5C6148334873306842624633576D4D32465875624B756D722F434964313832706C324C6A2F674574784C765630;
Query OK, 0 rows affected (0.03 sec)
</pre>
Let us check the user now:
<pre class="lang:mysql decode:true">(root@localhost) [(none)]>select user,host, authentication_string,plugin from mysql.user where user ='dba' and host= '192.168.4.%' order by 1,2;
+------+-------------+------------------------------------------------------------------------+-----------------------+
| user | host | authentication_string | plugin |
+------+-------------+------------------------------------------------------------------------+-----------------------+
| dba | 192.168.4.% | $A$005$X>ztS}WfR"k~aH3Hs0hBbF3WmM2FXubKumr/CId182pl2Lj/gEtxLvV0 | caching_sha2_password |
+------+-------------+------------------------------------------------------------------------+-----------------------+
1 row in set (0.00 sec)
[root@master3 ps80]# ./mysql-3307 -udba -pdba -h192.168.4.57 -P3307
(dba@192.168.4.57) [(none)]>select current_user();
+-----------------+
| current_user() |
+-----------------+
| dba@192.168.4.% |
+-----------------+
1 row in set (0.00 sec)
</pre>
As you can see the user has been created correctly and password is again in encrypted format. 
In short what you need to do when in need to export users from MySQL/PS 8 is:
<ol>
<li style="font-weight: 400;" aria-level="1">Read the user information</li>
<li style="font-weight: 400;" aria-level="1">Convert Password to hex format when plugin is caching_sha2</li>
<li style="font-weight: 400;" aria-level="1">Push the AS <password> converted to a file or any other way you were used to</li>
</ol>
 
Another possible solution is to use at session level the parameter print_identified_with_as_hex. If set causes SHOW CREATE USER to display such hash values as hexadecimal strings rather than as regular string literals. Hash values that do not contain unprintable characters still display as regular string literals, even with this variable enabled.
This at the end is exactly what Fred and I have done for our tools:
See:
<ul>
<li style="font-weight: 400;" aria-level="1">Fred: <a href="https://github.com/lefred/mysqlshell-plugins/commit/aa5c6bbe9b9aa689bf7266f5a19a35d0091f6568">https://github.com/lefred/mysqlshell-plugins/commit/aa5c6bbe9b9aa689bf7266f5a19a35d0091f6568</a></li>
<li style="font-weight: 400;" aria-level="1">Pt-show-grants: <a href="https://github.com/percona/percona-toolkit/blob/4a812d4a79c0973bf176105b0d138ad0a2a46b2f/bin/pt-show-grants#L2058" rel="nofollow">https://github.com/percona/percona-toolkit/blob/4a812d4a79c0973bf176105b0d138ad0a2a46b2f/bin/pt-show-grants#L2058</a></li>
</ul>
<h1>Conclusions</h1>
MySQL 8 and Percona server comes with a more secure hashing mechanism caching_sha2_password which is also the default. However if you have the need to migrate users and you use your own tools to export and import the passwords, you must update them as indicated. Or use the Percona Toolkit tools that we keep up to date for you.
 
Have fun with MySQL!!]]></description>
<category>MySQL</category>
<pubDate>Sun, 24 Sep 2023 15:39:08 +0000</pubDate>
</item>
<item>
<title>Proof of Concept: Horizontal Write Scaling for MySQL with Kubernetes Operator</title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/253-horizontal-write-scaling-for-mysql-with-operator-poc</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/253-horizontal-write-scaling-for-mysql-with-operator-poc</guid>
<description><![CDATA[Historically MySQL is great in horizontal READ scale. The scaling in that case is offered by the different number of Replica nodes, no matter if using standard asynchronous replication or synchronous replication. 
However those solutions do not offer the same level of scaling for writes operation. 
Why? Because the solutions still rely on writing in one single node that works as Primary. Also in case of multi-Primary the writes will be distributed by transaction. In both cases, when using virtually-synchronous replication, the process will require certification from each node and local (by node) write, as such the number of writes are NOT distributed across multiple nodes but duplicated. 
The main reason behind this is that MySQL is a relational database system (RDBMS), and any data that is going to be written in it, must respect the RDBMS rules (<a href="https://en.wikipedia.org/wiki/Relational_database">https://en.wikipedia.org/wiki/Relational_database</a>). In short any data that is written must be consistent with the data present. To achieve that the data needs to be checked with the existing through defined relations and constraints. This action is something that can affect very large datasets and be very expensive. Think about updating a table with millions of rows that refer to another table with another million rows.
An image may help:
<a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/schema-db.gif"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/schema-db.gif" alt="schema db" width="478" height="600" class="alignnone size-full wp-image-87678" style="display: block; margin-left: auto; margin-right: auto;" /></a>
Every time I will insert an order, I must be sure that all the related elements are in place and consistent. 
This operation is quite expensive but our database is able to run it in a few milliseconds or less, thanks to several optimizations that allow the node to execute most of them in memory, with none or little access to mass storage. 
The key factor is that the whole data structure resides in the same location (node) facilitating the operations.
Once we have understood that, it will also become clear why we cannot have relational data split in multiple nodes and have to distribute writes by table. If I have a node that manages only the items, another the orders, another the payments, I will need to have my solution able to deal with distributed transactions, each of which needs to certify and verify other nodes data. 
This level of distribution will seriously affect the efficiency of the operation which will increase the response time significantly. This is it, nothing is impossible however the performances will be so impacted that each operation may take seconds instead milliseconds or fraction of it, unless lifting some of the rules breaking the relational model.
MySQL as well as other RDBMS are designed to work respecting the model and cannot scale in any way by fragmenting and distributing a schema, so what can be done to scale?
 
The alternative is to split a consistent set of data into fragments. What is a consistent set of data? It all depends on the kind of information we are dealing with. Keeping in mind the example above where we have a shop online serving multiple customers, we need to identify which is the most effective way to split the data.
For instance if we try to split the data by Product Type (Books, CD/DVD, etc) we will have a huge duplication of data related to customers/orders/shipments and so on, all this data is also quite dynamic given I will have customers constantly ordering things. 
 
Why duplicate the data? Because if I do not duplicate that data I will not know if a customer had already bought or not that specific item, or I will have to ask again about the shipment address and so on. Which means also that any time a customer buys something or puts something in the wish list I have to reconcile the data in all my nodes/clusters.
 
On the other hand if I choose to split my data by country of customer’s residence the only data I will have to duplicate and keep in sync is the one related to the products, of which the most dynamic one will be the number of items in stock. This of course unless I can organize my products by country as well, which is a bit unusual nowadays, but not impossible. 
 
Another possible case is if I am a health organization and I manage several hospitals. As for the example above, it will be easier to split my data by hospital given most of the data related to patients is bound to the hospital itself as well as treatments and any other element related to hospital management. While it will make no sense to split by patient's country of residence.
 
This technique of splitting the data into smaller pieces is called sharding and at the moment is the only way we have to scale RDBM horizontally. 
 
In the MySQL open source ecosystem we have only two consolidated ways to perform sharding, Vitess and ProxySQL. The first one is a complete solution that takes ownership of your database and manages almost any aspect of its operations in a sharded environment, this includes a lot of specific features for DBA to deal with daily operations like table modifications, backup and more. 
 
While this may look great it also comes with some string attached, including the complexity and proprietary environment. That makes Vitess a good fit for “complex” sharding scenarios where other solutions may not be enough.
 
ProxySQL does not have a sharding mechanism “per se” but given the way it works and the features it has, allow us to build simple sharding solutions. 
It is important to note that most of the DBA operations will still be on DBA to be executed with incremented complexity given the sharding environment. 
 
There is a third option which is application aware sharding. 
This solution sees the application be aware of the need to split the data in smaller fragments and internally point the data to different “connectors” who are connected to multiple data sources. 
In this case the application is aware of a customer country and will redirect all the operations related to him to the datasource responsible for the specific fragment.
Normally this solution requires a full code re-design and could be quite difficult to achieve when it is injected after the initial code architecture definition.  
On the other hand, if done at design it is probably the best solution, because it will allow the application to define the sharding rules and can also optimize the different data sources using different technologies for different uses.
One example could be the use of a RDBMS for most of the Online transaction processing (OLTP) data shared by country, and having the products as distributed memory cache with a different technology. At the same time all the data related to orders, payments and customer history can be consolidated in a data warehouse used to generate reporting.    
 
As said the last one is probably the most powerful, scalable and difficult to design and unfortunately it represents probably less than the 5% of the solution currently deployed. 
As well as very few cases are in the need to have a full system/solution to provide scalability with sharding. 
 
By experience, most of the needs for horizontal scaling fell in the simple scenario, where there is the need to achieve sharding and data separation, very often with sharding-nothing architecture. In shared-nothing, each shard can live in a totally separate logical schema instance / physical database server / data center / continent. There is no ongoing need to retain shared access (from between shards) to the other unpartitioned tables in other shards.
 
<h2>The POC</h2>
<h4>Why this POC?</h4>
In the years I have faced a lot of customers that were talking about scaling their database solution and looking at very complex sharding as Vitess as the first and only way to go. 
This without even considering if their needs were driving them there for real. 
<img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/kiss.png" alt="kiss" width="216" height="188" class="wp-image-87680 alignleft" style="margin-right: 10px; float: left;" />
In my experience, and talking with several colleagues, I am not alone, when analyzing the real needs and after discussing with all the parties impacted, only a very small percentage of customers were in the real need of complex solutions. Most of the others were just trying to avoid a project that will implement simple shared-nothing solutions. Why? Because apparently it is simpler to migrate data to a platform that does all for you, than accept a bit of additional work and challenge at the beginning but keep a simple approach. Also going for the last shining things always has its magic.
On top of that, with the rise of Kubernetes and Mysql Operators, a lot of confusion starts to circulate, most of which generate by the total lack of understanding that a Database and a relational database are two separate things. That lack of understanding the difference and the real problems attached to a RDBMS had brought some to talk about horizontal scaling for databases, with a concerning superficiality and without clarifying if they were talking about RDBMS or not. As such some clarification is long due as well as putting back the KISS principle as the main focus.
 
Given that, I thought that refreshing how ProxySQL could help in building a simple sharding solution may help to clarify the issues, reset the expectations and show how we can do things in a simpler way.  (See my old post <a href="https://www.percona.com/blog/mysql-sharding-with-proxysql/">https://www.percona.com/blog/mysql-sharding-with-proxysql/</a>). 
 
To do so I had built a simple POC that illustrates how you can use Percona Operator for MySQL (POM) and ProxySQL to build a sharded environment with a good level of automation for some standard operations like backup/restore software upgrade and resource scaling. 
 
<h4>Why Proxysql?</h4>
In the following example we mimic a case where we need a simple sharding solution, which means we just need to redirect the data to different data containers, keeping the database maintenance operations on us. In this common case we do not need to implement a full sharding system such as Vitess. 

As illustrated above ProxySQL allows us to set up a common entry point for the application and then redirect the traffic on the base of identified sharding keys. It will also allow us to redirect read/write traffic to the primary and read only traffic to all secondaries. 
 
The other interesting thing is that we can have ProxySQL as part of the application pod, or as an independent service. Best practices indicate that having ProxySQL closer to the application will be more efficient especially if we decide to activate the caching feature.  
 
<h4>Why POM</h4>
Percona Operator for MySQL comes with three main solutions, Percona Operator for PXC, Percona Operator for MySQL Group Replication and Percona Operator for Percona server. The first two are based on virtually-synchronous replication, and allow the cluster to keep the data state consistent across all pods, which guarantees that the service will always offer consistent data.  In K8s context we can see POM as a single service with native horizontal scalability for reads, while for writes we will adopt the mentioned sharding approach.
 
The other important aspects of using a POM based solution is the automation it comes with. Deploying POM you will be able to set automation for backups, software updates, monitoring (using PMM) and last but not least the possibility to scale UP or DOWN just changing the needed resources. 
<h3>The elements used</h3>
<a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/horizontal_scaling.jpg"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/horizontal_scaling.jpg" alt="horizontal scaling" width="450" height="422" class="alignnone wp-image-87688" /></a>
In our POC I will use a modified version of sysbench (https://github.com/Tusamarco/sysbench), that has an additional field continent and I will use that as a sharding key. At the moment and for the purpose of this simple POC I will only have 2 Shards.
 
As the diagram above illustrates here we have a simple deployment but good enough to illustrate the sharding approach.
We have:
<ul>
<li style="font-weight: 400;" aria-level="1">The application(s) node(s), it is really up to you if you want to test with one application node or more, nothing will change, as well as for the ProxySQL nodes, just keep in mind that if you use more proxysql nodes is better to activate the internal cluster support or use consul to synchronize them. </li>
<li style="font-weight: 400;" aria-level="1">Shard 1 is based on POM with PXC, it has:</li>
</ul>
<ul>
<li style="font-weight: 400;" aria-level="2">Load balancer for service entry point
<ul>
<li style="font-weight: 400;" aria-level="3">Entry point for r/w</li>
<li style="font-weight: 400;" aria-level="3">Entry point for read only</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="2">3 Pods for Haproxy
<ul>
<li style="font-weight: 400;" aria-level="3">Haproxy container</li>
<li style="font-weight: 400;" aria-level="3">Pmm agent container</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="2">3 Pods with data nodes (PXC)
<ul>
<li style="font-weight: 400;" aria-level="3">PXC cluster node container</li>
<li style="font-weight: 400;" aria-level="3">Log streaming</li>
<li style="font-weight: 400;" aria-level="3">Pmm container </li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="2">Backup/restore service </li>
</ul>
<ul>
<li style="font-weight: 400;" aria-level="1">Shard 2 is based on POM for Percona server and Group Replication (technical preview)  
<ul>
<li style="font-weight: 400;" aria-level="2">Load balancer for service entry point
<ul>
<li style="font-weight: 400;" aria-level="3">Entry point for r/w</li>
<li style="font-weight: 400;" aria-level="3">Entry point for read only</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="2">3 Pods for MySQL Router (testing)
<ul>
<li style="font-weight: 400;" aria-level="3">MySQL router container</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="2">3 Pods with data nodes (PS with GR)
<ul>
<li style="font-weight: 400;" aria-level="3">PS -GR cluster node container</li>
<li style="font-weight: 400;" aria-level="3">Log streaming</li>
<li style="font-weight: 400;" aria-level="3">Pmm container </li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="2">Backup/restore on scheduler</li>
</ul>
</li>
</ul>
 
Now you may have noticed that the representation of the nodes are different in size, this is not a mistake while drawing. It indicates that I have allocated more resources (CPU and Memory) to shard1 than shard2. Why? Because I can and I am simulating a situation where a shard2 gets less traffic, at least temporarily, as such I do not want to give it the same resources as shard1. I will eventually increase them if I see the needs. 
<h2>The settings</h2>
<h3>Data layer</h3>
Let us start with the easy one, the Data layer configuration. Configuring correctly the environment is the key, and to do so I am using a tool that I wrote specifically to calculate the needed configuration in K8s POM environment, you can find it here (<a href="https://github.com/Tusamarco/mysqloperatorcalculator">https://github.com/Tusamarco/mysqloperatorcalculator</a>).
Once you have compiled it and run you can simply ask what “dimensions” are supported, or you can define a custom level of resources, but you will still need to indicate the level of expected load. In any case please refer to the README in the repository which has all the instructions.
The full cr.yaml for PXC shard1 is <a href="https://github.com/Tusamarco/blogs/blob/master/mysql_horizontal_scaling/cr_pxc.yml">here</a>, while the one for PS-GR <a href="https://github.com/Tusamarco/blogs/blob/master/mysql_horizontal_scaling/cr_ps_gr.yml">here</a>. 
For Shard 1: I asked for resources to cover traffic of type 2 (Light OLTP), configuration type 5 (2XLarge) 1000 connections.
For Shard2: I ask for resources to cover traffic of type 2 (Light OLTP), configuration type 2 (Small), 100 connections.     
Once you have the CRs defined you can follow the official guidelines to set the environment up ( PXC (<a href="https://docs.percona.com/percona-operator-for-mysql/pxc/index.html">https://docs.percona.com/percona-operator-for-mysql/pxc/index.html</a>), PS (<a href="https://docs.percona.com/percona-operator-for-mysql/ps/index.html">https://docs.percona.com/percona-operator-for-mysql/ps/index.html</a>)
 
It is time now to see the Proxysql settings.
<h3>ProxySQL and Sharding rules</h3>
As mentioned before we are going to test the load sharding by continent, and as also mentioned before we know that ProxySQL will not provide additional help to automatically manage the sharded environment. 
Given that one way to do it is to create a DBA account per shard, or to inject shard information in the commands while executing. I will use the less comfortable one just to prove if it works, the different DBA accounts. 
 
We will have 2 shards, the sharding key is the continent field, and the continents will be grouped as follows:
<ul>
<li style="font-weight: 400;" aria-level="1">Shard 1:
<ul>
<li style="font-weight: 400;" aria-level="2">Asia</li>
<li style="font-weight: 400;" aria-level="2">Africa</li>
<li style="font-weight: 400;" aria-level="2">Antarctica</li>
<li style="font-weight: 400;" aria-level="2">Europe</li>
<li style="font-weight: 400;" aria-level="2">North America</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">Shard 2:
<ul>
<li style="font-weight: 400;" aria-level="2">Oceania</li>
<li style="font-weight: 400;" aria-level="2">South America</li>
</ul>
</li>
</ul>
 
The DBAs users:
<ul>
<li style="font-weight: 400;" aria-level="1">dba_g1</li>
<li style="font-weight: 400;" aria-level="1">dba_g2</li>
</ul>
 
The application user:
<ul>
<li style="font-weight: 400;" aria-level="1">app_test</li>
</ul>
 
The host groups will be:
<ul>
<li style="font-weight: 400;" aria-level="1">Shard 1
<ul>
<li style="font-weight: 400;" aria-level="2">100 Read and Write</li>
<li style="font-weight: 400;" aria-level="2">101 Read only</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">Shard 2
<ul>
<li style="font-weight: 400;" aria-level="2">200 Read and Write</li>
<li style="font-weight: 400;" aria-level="2">201 Read only</li>
</ul>
</li>
</ul>
 
Once that is defined, we need to identify which query rules will serve us and how.
What we want is to redirect all the incoming queries for:
<ul>
<li style="font-weight: 400;" aria-level="1">Asia, Africa, Antarctica, Europe and North America to shard 1.</li>
<li style="font-weight: 400;" aria-level="1">Oceania and South America to shard 2</li>
<li style="font-weight: 400;" aria-level="1">Split the queries in R/W and Read only</li>
<li style="font-weight: 400;" aria-level="1">Prevent the execution of any query that do not have a shard key</li>
<li style="font-weight: 400;" aria-level="1">Backup data at regular interval and store it in a safe place</li>
</ul>
<a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/query_rukes_sharding.png"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/query_rukes_sharding.png" alt="query rukes sharding" width="800" height="280" class="alignnone size-large wp-image-87681" /></a>
Given the above we first define the rules for the DBAs accounts:
We set the Hostgroup for each DBA and then if the query matches the sharding rule we redirect it to the proper sharding, otherwise the HG will remain as set.
This allows us to execute queries like CREATE/DROP table on our shard without problem, but will allow us to send data where needed. 
For instance the one below is the output of the queries that sysbench will run.
Prepare:
<pre class="lang:mysql decode:true">INSERT INTO windmills_test1 /* continent=Asia */ (uuid,millid,kwatts_s,date,location,continent,active,strrecordtype) VALUES(UUID(), 79, 3949999,NOW(),'mr18n2L9K88eMlGn7CcctT9RwKSB1FebW397','Asia',0,'quq')
</pre>
In this case I have the application simply injecting a comment in the INSERT SQL declaring the shard key, given I am using the account dba_g1 to create/prepare the schemas, rules  32/32 will be used and give I have sett apply=1, ProxySQL will exit the query rules parsing and send the command to relevant hostgroup.
Run:
<pre class="lang:mysql decode:true">SELECT id, millid, date,continent,active,kwatts_s FROM windmills_test1 WHERE id BETWEEN ? AND ? AND continent='South America'
SELECT SUM(kwatts_s) FROM windmills_test1 WHERE id BETWEEN ? AND ? and active=1 AND continent='Asia'
SELECT id, millid, date,continent,active,kwatts_s FROM windmills_test1 WHERE id BETWEEN ? AND ? AND continent='Oceania' ORDER BY millid
SELECT DISTINCT millid,continent,active,kwatts_s FROM windmills_test1 WHERE id BETWEEN ? AND ? AND active =1 AND continent='Oceania' ORDER BY millid
UPDATE windmills_test1 SET active=? WHERE id=? AND continent='Asia'
UPDATE windmills_test1 SET strrecordtype=? WHERE id=? AND continent='North America'
DELETE FROM windmills_test1 WHERE id=? AND continent='Antarctica'
INSERT INTO windmills_test1 /* continent=Antarctica */ (id,uuid,millid,kwatts_s,date,location,continent,active,strrecordtype) VALUES (?, UUID(), ?, ?, NOW(), ?, ?, ?,?) ON DUPLICATE KEY UPDATE kwatts_s=kwatts_s+1
</pre>
The above are executed during the tests. 
In all of them the sharding key is present, either in the WHERE clause OR as comment. 
Of course if I execute one of them without the sharding key,  the firewall rule will stop the query execution, ie: 
<pre class="lang:mysql decode:true">mysql> SELECT id, millid, date,continent,active,kwatts_s FROM windmills_test1 WHERE id BETWEEN ? AND ?;
ERROR 1148 (42000): It is impossible to redirect this command to a defined shard. Please be sure you Have the Continent definition in your query, or that you use a defined DBA account (dba_g{1/2})
</pre>
Check <a href="https://github.com/Tusamarco/blogs/blob/master/mysql_horizontal_scaling/operator_sharding_with_proxysql_public.txt">here </a>for full command list
<h3>Setting up the dataset</h3>
Once we have the rules set it is time to setup the schemas and the data using sysbench (<a href="https://github.com/Tusamarco/sysbench">https://github.com/Tusamarco/sysbench</a>), remember to use windmills_sharding tests.  
The first operation is to build the schema on SHARD2 without filling it with data. This is a DBA action as such we will execute it using the dba_g2 account:
<pre class="lang:sh decode:true">sysbench ./src/lua/windmills_sharding/oltp_read.lua --mysql-host=10.0.1.96 --mysql-port=6033 --mysql-user=dba_g2 --mysql-password=xxx --mysql-db=windmills_large --mysql_storage_engine=innodb --db-driver=mysql --tables=4 --table_size=0 --table_name=windmills --mysql-ignore-errors=all --threads=1 prepare
</pre>
Setting table_size and pointing to the ProxySQL IP/port will do, and I will have: 
<pre class="lang:mysql decode:true">mysql> select current_user(), @@hostname;
+----------------+-------------------+
| current_user() | @@hostname |
+----------------+-------------------+
| dba_g2@% | ps-mysql1-mysql-0 |
+----------------+-------------------+
1 row in set (0.01 sec)
mysql> use windmills_large;
Database changed
mysql> show tables;
+---------------------------+
| Tables_in_windmills_large |
+---------------------------+
| windmills1 |
| windmills2 |
| windmills3 |
| windmills4 |
+---------------------------+
4 rows in set (0.01 sec)
mysql> select count(*) from windmills1;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.09 sec)
</pre>
All set but empty.
Now let us do the same but with the other DBA user:
<pre class="lang:sh decode:true">sysbench ./src/lua/windmills_sharding/oltp_read.lua --mysql-host=10.0.1.96 --mysql-port=6033 --mysql-user=dba_g1 --mysql-password=xxx --mysql-db=windmills_large --mysql_storage_engine=innodb --db-driver=mysql --tables=4 --table_size=400 --table_name=windmills --mysql-ignore-errors=all --threads=1 prepare
</pre>
If I do now the select above with user dba_g2:
<pre class="lang:mysql decode:true">mysql> select current_user(), @@hostname;select count(*) from windmills1;
+----------------+-------------------+
| current_user() | @@hostname |
+----------------+-------------------+
| dba_g2@% | ps-mysql1-mysql-0 |
+----------------+-------------------+
1 row in set (0.00 sec)
+----------+
| count(*) |
+----------+
| 113 |
+----------+
1 row in set (0.00 sec)
</pre>
While If I reconnect and use dba_g1:
<pre class="lang:mysql decode:true">mysql> select current_user(), @@hostname;select count(*) from windmills1;
+----------------+--------------------+
| current_user() | @@hostname |
+----------------+--------------------+
| dba_g1@% | mt-cluster-1-pxc-0 |
+----------------+--------------------+
1 row in set (0.00 sec)
+----------+
| count(*) |
+----------+
| 287 |
+----------+
1 row in set (0.01 sec)
</pre>
I can also check on ProxySQL to see which rules were utilized:
<pre class="lang:mysql decode:true">select active,hits,destination_hostgroup, mysql_query_rules.rule_id, match_digest, match_pattern, replace_pattern, cache_ttl, apply,flagIn,flagOUT FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_id;
+------+-----------------------+---------+---------------------+----------------------------------------------------------------------------+-------+--------+---------+
| hits | destination_hostgroup | rule_id | match_digest | match_pattern | apply | flagIN | flagOUT |
+------+-----------------------+---------+---------------------+----------------------------------------------------------------------------+-------+--------+---------+
| 3261 | 100 | 20 | NULL | NULL | 0 | 0 | 500 |
| 51 | 200 | 21 | NULL | NULL | 0 | 0 | 600 |
| 2320 | 100 | 31 | NULL | scontinents*(=|like)s*'*(Asia|Africa|Antarctica|Europe|North America)'* | 1 | 500 | 0 |
| 880 | 200 | 32 | NULL | scontinents*(=|like)s*'*(Oceania|South America)'* | 1 | 500 | 0 |
| 0 | 100 | 34 | NULL | scontinents*(=|like)s*'*(Asia|Africa|Antarctica|Europe|North America)'* | 1 | 600 | 0 |
| 0 | 200 | 35 | NULL | scontinents*(=|like)s*'*(Oceania|South America)'* | 1 | 600 | 0 |
| 2 | 100 | 51 | NULL | scontinents*(=|like)s*'*(Asia|Africa|Antarctica|Europe|North America)'* | 0 | 0 | 1001 |
| 0 | 200 | 54 | NULL | scontinents*(=|like)s*'*(Oceania|South America)'* | 0 | 0 | 1002 |
| 0 | 100 | 60 | NULL | NULL | 0 | 50 | 1001 |
| 0 | 200 | 62 | NULL | NULL | 0 | 60 | 1002 |
| 7 | NULL | 2000 | . | NULL | 1 | 0 | NULL |
| 0 | 100 | 2040 | ^SELECT.*FOR UPDATE | NULL | 1 | 1001 | NULL |
| 2 | 101 | 2041 | ^SELECT.*$ | NULL | 1 | 1001 | NULL |
| 0 | 200 | 2050 | ^SELECT.*FOR UPDATE | NULL | 1 | 1002 | NULL |
| 0 | 201 | 2051 | ^SELECT.*$ | NULL | 1 | 1002 | NULL |
+------+-----------------------+---------+---------------------+----------------------------------------------------------------------------+-------+--------+---------+
</pre>
<h3>Running the application</h3>
Now that the data load test was successful, let us do the real load following the indication as above but use 80 Tables and just a bit more records like 20000, nothing huge. 
 
Once the data is loaded we will have the 2 shards with different numbers of records, if all went well the shard2 should have ¼ of the total and shard1 ¾ .
 
When load is over I have as expected:
<pre class="lang:mysql decode:true">mysql> select current_user(), @@hostname;select count(*) as shard1 from windmills_large.windmills80;select /* continent=shard2 */ count(*) as shard2 from windmills_large.windmills80;
+----------------+--------------------+
| current_user() | @@hostname |
+----------------+--------------------+
| dba_g1@% | mt-cluster-1-pxc-0 |
+----------------+--------------------+
1 row in set (0.00 sec)
+--------+
| shard1 |
+--------+
| 14272 | ← Table windmills80 in SHARD1
+--------+
+--------+
| shard2 |
+--------+
| 5728 | ← Table windmills80 in SHARD2
+--------+
</pre>
As you may have already noticed, I used a trick to query the other shard using the dba_g1 user, I just passed in the query  the shard2 definition as a comment. That is all we need.
Let us execute the run command for writes in sysbench and see what happens.
The first thing we can notice while doing writes is the query distribution: 
<pre class="lang:mysql decode:true">+--------+-----------+----------------------------------------------------------------------------+----------+--------+----------+----------+--------+---------+-------------+---------+
| weight | hostgroup | srv_host | srv_port | status | ConnUsed | ConnFree | ConnOK | ConnERR | MaxConnUsed | Queries |
+--------+-----------+----------------------------------------------------------------------------+----------+--------+----------+----------+--------+---------+-------------+---------+
| 10000 | 100 | ac966f7d46c04400fb92a3603f0e2634-193113472.eu-central-1.elb.amazonaws.com | 3306 | ONLINE | 24 | 0 | 138 | 66 | 25 | 1309353 |
| 100 | 101 | a5c8836b7c05b41928ca84f2beb48aee-1936458168.eu-central-1.elb.amazonaws.com | 3306 | ONLINE | 0 | 0 | 0 | 0 | 0 | 0 |
| 10000 | 200 | a039ab70e9f564f5e879d5e1374d9ffa-1769267689.eu-central-1.elb.amazonaws.com | 3306 | ONLINE | 24 | 1 | 129 | 66 | 25 | 516407 |
| 10000 | 201 | a039ab70e9f564f5e879d5e1374d9ffa-1769267689.eu-central-1.elb.amazonaws.com | 6447 | ONLINE | 0 | 0 | 0 | 0 | 0 | 0 |
+--------+-----------+----------------------------------------------------------------------------+----------+--------+----------+----------+--------+---------+-------------+---------+
</pre>
Where we can notice that the load in connection is evenly distributed, while the load is mainly going to shard1 as we expected given we have an unbalanced sharding by design.
 
At MySQL level we had:
 
Questions
<a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/pxc_ps_write_questions.png"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/pxc_ps_write_questions.png" alt="pxc ps write questions" width="1024" height="187" class="alignnone size-large wp-image-87684" /></a>
Com Type
<a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/pxc_ps_write_com.png"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/pxc_ps_write_com.png" alt="pxc ps write com" width="1024" height="179" class="alignnone size-large wp-image-87683" /></a>
<a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/number_operation_writes.png"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/number_operation_writes.png" alt="number operation writes" width="473" height="133" class="alignnone size-full wp-image-87687" /></a>
The final point is, what is the gain using this sharding approach?
Well we still need to consider the fact we are testing on a very small set of data, however if we can already identify some benefit here, that will be an interesting result. 
 
Let see the write operations with 24 and 64 threads:
<a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/writes_write.png"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/writes_write.png" alt="writes write" width="450" height="395" class="alignnone wp-image-87693" /></a> <a href="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/latency_write.png"><img src="http://www.tusacentral.net/joomla/images/stories/mysql_horizontal_scaling/latency_write.png" alt="latency write" width="450" height="395" class="alignnone wp-image-87686" /></a>
We get a gain of ~33% just using sharding, while for latency we do not have a cost on the contrary also with small load increase we can see how the sharded solution performs better. Of course we are still talking about low number of rows and running threads but gain is there. 
 
<h3>Backup </h3>
The backup and restore operation when using POM is completely managed by the operator (see instructions in the POM documentation <a href="https://docs.percona.com/percona-operator-for-mysql/pxc/backups.html">https://docs.percona.com/percona-operator-for-mysql/pxc/backups.html</a> and <a href="https://docs.percona.com/percona-operator-for-mysql/ps/backups.html">https://docs.percona.com/percona-operator-for-mysql/ps/backups.html</a> ).
The interesting part is that we can have multiple kind of backup solution, like:
<ul>
<li style="font-weight: 400;" aria-level="1">On demand</li>
<li style="font-weight: 400;" aria-level="1">Scheduled </li>
<li style="font-weight: 400;" aria-level="1">Full Point in time recovery with log streaming</li>
</ul>
Automation will allow us to set schedule as simple as this: 
<pre class="lang:yaml decode:true"> schedule:
- name: "sat-night-backup"
schedule: "0 0 * * 6"
keep: 3
storageName: s3-eu-west
- name: "daily-backup"
schedule: "0 3 * * *"
keep: 7
storageName: s3-eu-west
</pre>
Or if you want to run the on demand:
<pre class="lang:sh decode:true">kubectl apply -f backup.yaml</pre>
Where the backup.yaml file has very simple informations:
<pre class="lang:yaml decode:true">apiVersion: ps.percona.com/v1alpha1
kind: PerconaServerMySQLBackup
metadata:
name: ps-gr-sharding-test-2nd-of-may
# finalizers:
# - delete-backup
spec:
clusterName: ps-mysql1
storageName: s3-ondemand
</pre>
Using both methods we will be able to soon have a good set of backups like:
POM (PXC)
<pre class="lang:sh decode:true">cron-mt-cluster-1-s3-eu-west-20234293010-3vsve mt-cluster-1 s3-eu-west s3://mt-bucket-backup-tl/scheduled/mt-cluster-1-2023-04-29-03:00:10-full Succeeded 3d9h 3d9h
cron-mt-cluster-1-s3-eu-west-20234303010-3vsve mt-cluster-1 s3-eu-west s3://mt-bucket-backup-tl/scheduled/mt-cluster-1-2023-04-30-03:00:10-full Succeeded 2d9h 2d9h
cron-mt-cluster-1-s3-eu-west-2023513010-3vsve mt-cluster-1 s3-eu-west s3://mt-bucket-backup-tl/scheduled/mt-cluster-1-2023-05-01-03:00:10-full Succeeded 33h 33h
cron-mt-cluster-1-s3-eu-west-2023523010-3vsve mt-cluster-1 s3-eu-west s3://mt-bucket-backup-tl/scheduled/mt-cluster-1-2023-05-02-03:00:10-full Succeeded 9h 9h
</pre>
POM (PS) *
<pre class="lang:sh decode:true">NAME STORAGE DESTINATION STATE COMPLETED AGE
ps-gr-sharding-test s3-ondemand s3://mt-bucket-backup-tl/ondemand/ondemand/ps-mysql1-2023-05-01-15:10:04-full Succeeded 21h 21h
ps-gr-sharding-test-2nd-of-may s3-ondemand s3://mt-bucket-backup-tl/ondemand/ondemand/ps-mysql1-2023-05-02-12:22:24-full Succeeded 27m 27m
</pre>
To note that as DBA, we still need to validate the backups with a restore procedure, that part is not automated (yet). 
*Note that Backup for POM PS is available only on demand given the solution is still in technical preview
<h3>When will this solution fit in?</h3>
As mentioned multiple times, this solution can cover simple cases of sharding, better if you have shared-nothing. 
It also requires work from the DBA side in case of DDL operations or resharding. 
You also need to be able to change some SQL code in order to be sure to have present the sharding key/information, in any SQL executed.
 
<h3>When will this solution not fit in?</h3>
There are several things that could prevent you to use this solution, the most common ones are:
<ul>
<li style="font-weight: 400;" aria-level="1">You need to query multiple shards at the same time. This is not possible with ProxySQL</li>
<li style="font-weight: 400;" aria-level="1">You do not have a DBA to perform administrative work and need to rely on automated system</li>
<li style="font-weight: 400;" aria-level="1">Distributed transaction cross shard</li>
<li style="font-weight: 400;" aria-level="1">No access to SQL code.</li>
</ul>
<h2>Conclusions</h2>
We do not have the Amletic dilemma about sharding or not sharding. 
When using a RDBMS like MySQL, if you need horizontal scalability, you need to shard. 
The point is there is no magic wand or solution, moving to sharding is an expensive and impacting operation. If you choose it at the beginning, before doing any application development, the effort can be significantly less. 
Doing sooner will also allow you to test proper solutions, where proper is a KISS solution, always go for the less complex things, in 2 years you will be super happy about your decision.  
If instead you must convert a current solution, then prepare for a bloodshed, or at least for a long journey. 
In any case we need to keep in mind few key points:
<ul>
<li style="font-weight: 400;" aria-level="1">Do not believe most of the articles on the internet that promise you infinite scalability for your database. If there is no distinction in the article about a simple database and a RDBMS, run away. </li>
<li style="font-weight: 400;" aria-level="1">Do not go for the last shiny things just because they shine. Test them and evaluate IF it makes sense for you. Better to spend a quarter testing now a few solutions, than fight for years with something that you do not fully comprehend.  </li>
<li style="font-weight: 400;" aria-level="1">Using containers/operators/kubernetes do not scale per se, you must find a mechanism to have the solution scaling, there is absolutely NO difference with premises. What you may get is a good level of automation, however that will come with a good level of complexity, it is up to you to evaluate if it makes sense or not.  </li>
</ul>
 
As said at the beginning for MYSQL the choice is limited. Vitess is the full complete solution, with a lot of coding to provide you a complete platform to deal with your scaling needs.
However do not be so fast to exclude ProxySQL as possible solutions. There are out there already many using it also for sharding. 
This small POC used a synthetic case, but it also shows that with just 4 rules you can achieve a decent solution. A real scenario could be a bit more complex … or not. 
 
<h2>References</h2>
Vitess (<a href="https://vitess.io/docs/">https://vitess.io/docs/</a>)
ProxySQL (<a href="https://proxysql.com/documentation/">https://proxysql.com/documentation/</a>)
Firewalling with ProxySQL (<a href="https://www.tusacentral.com/joomla/index.php/mysql-blogs/197-proxysql-firewalling">https://www.tusacentral.com/joomla/index.php/mysql-blogs/197-proxysql-firewalling</a>)
Sharding:
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://www.percona.com/blog/mysql-sharding-with-proxysql/">https://www.percona.com/blog/mysql-sharding-with-proxysql/</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://www.percona.com/blog/horizontal-scaling-in-mysql-sharding-followup/">https://https://medium.com/pinterest-engineering/sharding-pinterest-how-we-scaled-our-mysql-fleet-3f341e96ca6fwww.percona.com/blog/horizontal-scaling-in-mysql-sharding-followup/</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://quoraengineering.quora.com/MySQL-sharding-at-Quora">https://quoraengineering.quora.com/MySQL-sharding-at-Quora</a></li>
<li style="font-weight: 400;" aria-level="1">https://skywalking.apache.org/blog/skywalkings-new-storage-feature-based-on-shardingsphere-proxy-mysql-sharding/</li>
</ul>
 ]]></description>
<category>MySQL</category>
<pubDate>Thu, 11 May 2023 12:20:20 +0000</pubDate>
</item>
<item>
<title>Which is the best Proxy for Percona MySQL Operator?</title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/252-comparisons-of-proxies-for-mysql</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/252-comparisons-of-proxies-for-mysql</guid>
<description><![CDATA[<h2>Overview</h2>
HAProxy, ProxySQL, MySQL Router (AKA MySQL Proxy), in the last few years I had to answer multiple times on what proxy to use and in what scenario. When designing an architecture there are many components that need to be considered before deciding what is the best solution.
When deciding what to pick, there are many things to consider like where the proxy needs to be, if it “just” needs to redirect the connections or if more features need to be in, like caching, filtering, or if it needs to be integrated with some MySQL embedded automation.
Given that, there never was a single straight answer, instead an analysis needs to be done. Only after a better understanding of the environment, the needs and the evolution that the platform needs to achieve is it possible to decide what will be the better choice.
However recently we have seen an increase in usage of MySQL on Kubernetes, especially with the adoption of Percona Operator for MySQL. In this case we have a quite well define scenario that can resemble the image below:
<a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-default.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/proxys_gr_comparison-default.png" alt="proxys gr comparison default" width="536" height="614" class="aligncenter wp-image-86219 size-full" style="display: block; margin-left: auto; margin-right: auto;" /></a>
In this scenario the proxies need to sit inside Pods balancing the incoming traffic from the Service LoadBalancer connecting with the active data nodes.
Their role is merely to be sure that any incoming connection is redirected to nodes that are able to serve them, which include having a separation between Read/Write and Read Only traffic, separation that can be achieved, at service level, with automatic recognition or with two separate entry points. 
In this scenario it is also crucial to be efficient in resource utilization and scaling with frugality. In this context features like filtering, firewalling or caching are redundant and may consume resources that could be allocated to scaling. Those are also features that will work better outside the K8s/Operator cluster, given the closer to the application they are located, the better they will serve. 
About that we must always remember the concept that each K8s/Operator cluster needs to be seen as a single service, not as a real cluster. In short each cluster is in reality a single database with High Availability and other functionalities built in. 
<a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-default.png"></a>
Anyhow, we are here to talk about Proxies. Once we have defined that we have one clear mandate in mind, we need to identify which product allow our K8s/Operator solution to:
<ul>
<li style="font-weight: 400;" aria-level="1">Scale at the maximum the number of incoming connections</li>
<li style="font-weight: 400;" aria-level="1">Serve the request with the higher efficiency</li>
<li style="font-weight: 400;" aria-level="1">Consume less resources as possible</li>
</ul>
<a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-default.png"></a>
<h2>The Environment</h2>
To identify the above points I have simulated a possible K8s/Operator environment, creating:
<ul>
<li style="font-weight: 400;" aria-level="1">One powerful application node, where I run sysbench read only tests, scaling from 2 to 4096 threads. (Type c5.4xlarge)</li>
<li style="font-weight: 400;" aria-level="1">3 Mid data nodes with several gigabytes of data in with MySQL and Group Replication (Type m5.xlarge)</li>
<li style="font-weight: 400;" aria-level="1">1 Proxy node running on a resource limited box (Type t2.micro)</li>
</ul>
<a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-default.png"></a>
<h2>The Tests </h2>
We will have very simple test cases. The first one has the scope to define the baseline, identifying the moment when we will have the first level of saturation due to the number of connections. In this case we will increase the number of connections and keep a low number of operations. 
The second test will define how well the increasing load is served inside the range we had previously identified. 
For documentation the sysbench commands are:
Test1
<pre class="lang:sh decode:true">sysbench ./src/lua/windmills/oltp_read.lua --db-driver=mysql --tables=200 --table_size=1000000
--rand-type=zipfian --rand-zipfian-exp=0 --skip_trx=true --report-interval=1 --mysql-ignore-errors=all
--mysql_storage_engine=innodb --auto_inc=off --histogram --stats_format=csv --db-ps-mode=disable --point-selects=50
--reconnect=10 --range-selects=true –rate=100 --threads=<#Threads from 2 to 4096> --time=1200 run
</pre>
Test2
<pre class="lang:sh decode:true">sysbench ./src/lua/windmills/oltp_read.lua --mysql-host=<host> --mysql-port=<port> --mysql-user=<user>
--mysql-password=<pw> --mysql-db=<schema> --db-driver=mysql --tables=200 --table_size=1000000 --rand-type=zipfian
--rand-zipfian-exp=0 --skip_trx=true --report-interval=1 --mysql-ignore-errors=all --mysql_storage_engine=innodb
--auto_inc=off --histogram --table_name=<tablename> --stats_format=csv --db-ps-mode=disable --point-selects=50
--reconnect=10 --range-selects=true --threads=<#Threads from 2 to 4096> --time=1200 run
</pre>
<h2>Results</h2>
<h3>Test 1</h3>
As indicated here I was looking to identify when the first Proxy will reach a dimension that would not be manageable. The load is all in creating and serving the connections, while the number of operations is capped to 100. 
<a href="https://www.percona.com/blog/wp-content/uploads/2023/03/events_rate2.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/events_rate2.png" alt="events rate2" width="600" height="401" class="alignnone size-large wp-image-86207" style="display: block; margin-left: auto; margin-right: auto;" /></a>
As you can see and as I was expecting, the three Proxies were behaving more or less the same, serving the same number of operations (they were capped so why not), until not.
MySQL router after 2048 connection was not able to serve anything more.
NOTE: MySQL Router was actually stopped working at 1024 threads, but using version 8.0.32 I enabled the feature: connection_sharing. That allows it to go a bit further.  
Let us take a look also the the latency:
<a href="https://www.percona.com/blog/wp-content/uploads/2023/03/latency95_rate.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/latency95_rate.png" alt="latency95 rate" width="600" height="424" class="alignnone size-large wp-image-86209" style="display: block; margin-left: auto; margin-right: auto;" /></a>
Here the situation starts to be a little bit more complicated. MySQL Router is the one that has the higher latency no matter what. However HAProxy and ProxySQL have an interesting behavior. HAProxy is performing better with a low number of connections, while ProxySQL is doing better when a high number of connections is in place.  
This is due to the multiplexing and the very efficient way ProxySQL uses to deal with high load.
Everything has a cost:
<a href="https://www.percona.com/blog/wp-content/uploads/2023/03/node-summary-cpu.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/node-summary-cpu.png" alt="node summary cpu" width="600" height="241" class="alignnone size-large wp-image-86215" style="display: block; margin-left: auto; margin-right: auto;" /></a>
HAProxy is definitely using less user’s CPU resources than ProxySQL or MySQL Router …
<a href="https://www.percona.com/blog/wp-content/uploads/2023/03/node-summary-cpu-saturation.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/node-summary-cpu-saturation.png" alt="node summary cpu saturation" width="600" height="238" class="alignnone size-large wp-image-86214" style="display: block; margin-left: auto; margin-right: auto;" /></a>
.. we can also notice that HAProxy barely reaches on average the 1.5 CPU load while ProxySQL is at 2.50 and MySQL Router around 2. 
To be honest I was expecting something like this, given ProxySQL's need to handle the connections and the other basic routing. What was instead a surprise was MySQL Router, why does it have a higher load?
<h4>Brief summary</h4>
This test highlights that HAProxy and ProxySQL are able to reach a level of connection higher than the slowest runner in the game (MySQL Router). It is also clear that traffic is better served under a high number of connections by ProxySQL but it requires more resources. 
<h2>Test 2</h2>
When the going gets tough, the tough gets going
Well let us remove the –rate limitation and see what will happen. 
<a href="https://www.percona.com/blog/wp-content/uploads/2023/03/events_perf.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/events_perf.png" alt="events perf" width="800" height="475" class="alignnone size-large wp-image-86205" style="display: block; margin-left: auto; margin-right: auto;" /></a>
The scenario with load changes drastically. We can see how HAProxy is able to serve the connection and allow the execution of a higher number of operations for the whole test. ProxySQL is immediately after it and is behaving quite well up to 128 threads, then it just collapses. 
MySQL Router never takes off; it always stays below the 1k reads/second while HAProxy was able to serve 8.2k and  ProxySQL 6.6k.
<a href="https://www.percona.com/blog/wp-content/uploads/2023/03/latency95_perf.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/latency95_perf.png" alt="latency95 perf" width="800" height="531" class="alignnone size-large wp-image-86208" style="display: block; margin-left: auto; margin-right: auto;" /></a>
Taking a look at the Latency, we can see that HAProxy had a gradual increase as expected, while ProxySQL and MySQL Router just went up from the 256 threads on. 
To observe that both ProxySQL and MySQL Router were not able to complete the tests with 4096 threads.
<a href="https://www.percona.com/blog/wp-content/uploads/2023/03/node-summary-cpu-perf.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/node-summary-cpu-perf.png" alt="node summary cpu perf" width="800" height="318" class="alignnone size-large wp-image-86212" style="display: block; margin-left: auto; margin-right: auto;" /></a>
Why? HAProxy stays always below 50% cpu, no matter the increasing number of threads/connections, scaling the load very efficiently. MySQL router was almost immediately reaching the saturation point, being affected not only by the number of threads/connections but also by the number of operations, and that was not expected given we do not have a level 7 capability in MySQL Router.
Finally ProxySQL, which was working fine up to a certain limit, then it reached saturation point and was not able to serve the load. I am saying load because ProxySQL is a level 7 proxy and is aware of the content of the load. Given that on top of multiplexing, additional resource consumption was expected.   
<a href="https://www.percona.com/blog/wp-content/uploads/2023/03/node-summary-cpu-saturation-perf.png"><img src="http://www.tusacentral.net/joomla/images/stories/proxies_comparisons_march_2023/node-summary-cpu-saturation-perf.png" alt="node summary cpu saturation perf" width="800" height="319" class="alignnone size-large wp-image-86213" style="display: block; margin-left: auto; margin-right: auto;" /></a>
Here we just have a clear confirmation of what already said above, with 100% cpu utilization reached by MySQL Router with just 16 threads, and ProxySQL way after at 256 threads.
<h2>Brief Summary</h2>
HAProxy comes up as the champion in this test, there is no doubt that it was able to scale the increasing load in connection without being affected significantly by the load generated by the requests. The lower consumption in resources also indicates the possible space for even more scaling.
ProxySQL was penalized by the limited resources, but this was the game, we have to get the most out of the few available. This test indicates that it is not optimal to use ProxySQL inside the Operator, actually it is a wrong choice if low resource and scalability is a must.    
MySQL Router was never in the game. Unless a serious refactoring, MySQL Router is designed for very limited scalability, as such the only way to adopt it is to have many of them at application node level. Utilizing it close to the data nodes in a centralized position is a mistake.  
<h2>Conclusions</h2>
I started showing an image on how the MySQL service is organized and want to close showing the variation that for me is the one to be considered the default approach:
<a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-haproxy.jpg"><img src="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-haproxy.jpg" alt="" width="536" height="651" class="size-full wp-image-86220 aligncenter" style="display: block; margin-left: auto; margin-right: auto;" /></a>
This to highlight that we always need to choose the right tool for the job. 
The Proxy in architectures involving MySQL/PS/PXC is a crucial element for the scalability of the cluster, no matter if using K8s or not. It is important to choose the one that serves us better, which in some cases can be ProxySQL over HAProxy. <a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-haproxy.jpg"> </a>
However when talking of K8s and Operators we must recognize the need to optimize the resources usage for the specific service. In that context there is no discussion about it, HAProxy is the best solution and the one we should go to. <a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-haproxy.jpg"> </a>
My final observation is about MySQL Router (aka MySQL Proxy). 
Unless a significant refactoring of the product at the moment it is not even close to what the other two can do. From the tests done so far, it requires a complete reshaping starting to identify why it is so subject to the load coming from the query, more than the load coming from the connections.   
<a href="https://www.percona.com/blog/wp-content/uploads/2023/03/proxys_gr_comparison-haproxy.jpg"></a>
Great MySQL to everyone. 
<h2>References</h2>
https://www.percona.com/blog/boosting-percona-distribution-for-mysql-operator-efficiency/
https://www.slideshare.net/marcotusa/my-sql-on-kubernetes-demystified
https://docs.haproxy.org/2.7/configuration.html
https://proxysql.com/documentation/
https://dev.mysql.com/doc/mysql-router/8.0/en/
 
 ]]></description>
<category>MySQL</category>
<pubDate>Sun, 19 Mar 2023 17:57:59 +0000</pubDate>
</item>
<item>
<title>Help! I am out of disk space!</title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/251-help-i-am-out-of-disk-space</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/251-help-i-am-out-of-disk-space</guid>
<description><![CDATA[How we could  fix a nasty out of space issue leveraging the flexibility of Percona MySQL operator (PMO)  <img src="http://www.tusacentral.net/joomla/images/stories/DiskSpaceFull.png" alt="DiskSpaceFull" width="350" height="208" style="float: right;" />
When planning a database deployment, one of the most challenging factors to consider is the amount of space we need to dedicate for Data on disk.
This is even more cumbersome when working on bare metal. Given it is definitely more difficult to add space when using this kind of solution in respect to the cloud. 
This is it, when using cloud storage like EBS or similar it is normally easy(er) to extend volumes, which gives us the luxury to plan the space to allocate for data with a good grade of relaxation. 
Is this also true when using a solution based on Kubernetes like Percona Operator for MySQL? Well it depends on where you run it, however if the platform you choose supports the option to extend volumes K8s per se is giving you the possibility to do so as well.
However, if it can go wrong it will, and ending up with a fully filled device with MySQL is not a fun experience. 
As you know, on normal deployments, when mysql has no space left on the device, it simply stops working, ergo it will cause a production down event, which of course is an unfortunate event that we want to avoid at any cost.  
This blog is the story of what happened, what was supposed to happen and why. 
<h2>The story </h2>
Case was on AWS using EKS.
Given all the above, I was quite surprised when we had a case in which a deployed solution based on PMO went out of space. However we start to dig and review what was going on and why.
The first thing we did was to quickly investigate what was really taking space, that could have been an easy win if most of the space was taken by some log, but unfortunately this was not the case, data was really taking all the available space. 
The next step was to check what storage class was used for the PVC
<pre class="lang:sh decode:true">k get pvc
NAME VOLUME CAPACITY ACCESS MODES STORAGECLASS
datadir-mt-cluster-1-pxc-0 pvc-<snip> 233Gi RWO io1
datadir-mt-cluster-1-pxc-1 pvc-<snip> 233Gi RWO io1
datadir-mt-cluster-1-pxc-2 pvc-<snip> 233Gi RWO io1
</pre>
Ok we use the io1 SC, it is now time to check if the SC is supporting volume expansion:
<pre class="lang:sh decode:true">kubectl describe sc io1
Name: io1
IsDefaultClass: No
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"},"name":"io1"},"parameters":{"fsType":"ext4","iopsPerGB":"12","type":"io1"},"provisioner":"kubernetes.io/aws-ebs"}
,storageclass.kubernetes.io/is-default-class=false
Provisioner: kubernetes.io/aws-ebs
Parameters: fsType=ext4,iopsPerGB=12,type=io1
AllowVolumeExpansion: <unset> <------------
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
</pre>
And no is not enabled, in this case we cannot just go and expand the volume, must change the storage class settings first. To enable volume expansion, you need to delete the storage class and enable it again. 
Unfortunately we were unsuccessful in doing that operation, because the storage class kept staying as unset for  ALLOWVOLUMEEXPANSION. 
As said this is a production down event, so we cannot invest too much time in digging why it was not correctly changing the mode, we had to act quickly. 
The only option we had to fix it was:
<ul>
<li style="font-weight: 400;" aria-level="1">Expand the io1 volumes from AWS console (or aws client)</li>
<li style="font-weight: 400;" aria-level="1">Resize the file system </li>
<li style="font-weight: 400;" aria-level="1">Patch any K8 file to allow K8 to correctly see the new volumes dimension  </li>
</ul>
Expanding EBS volumes from the console is trivial, just go to Volumes, select the volume you want to modify, choose modify and change the size of it with the one desired, done. 
Once that is done connect to the Node hosting the pod which has the volume mounted like:
<pre class="lang:sh decode:true"> k get pods -o wide|grep mysql-0
NAME READY STATUS RESTARTS AGE IP NODE
cluster-1-pxc-0 2/2 Running 1 11d 10.1.76.189 <mynode>.eu-central-1.compute.internal</pre>
Then we need to get the id of the pvc to identify it on the node
<pre class="lang:sh decode:true">k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
datadir-cluster-1-pxc-0 Bound pvc-1678c7ee-3e50-4329-a5d8-25cd188dc0df 233Gi RWO io1
</pre>
One note, when doing this kind of recovery with a PXC based solution, always recover node-0 first, then the others.  
So we connect to <mynode> and identify the volume: 
<pre class="lang:sh decode:true">lslbk |grep pvc-1678c7ee-3e50-4329-a5d8-25cd188dc0df
nvme1n1 259:4 0 350G 0 disk /var/lib/kubelet/pods/9724a0f6-fb79-4e6b-be8d-b797062bf716/volumes/kubernetes.io~aws-ebs/pvc-1678c7ee-3e50-4329-a5d8-25cd188dc0df <-----
</pre>
At this point we can resize it:
<pre class="lang:sh decode:true">root@ip-<snip>:/# resize2fs /dev/nvme1n1
resize2fs 1.45.5 (07-Jan-2020)
Filesystem at /dev/nvme1n1 is mounted on /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-central-1a/vol-0ab0db8ecf0293b2f; on-line resizing required
old_desc_blocks = 30, new_desc_blocks = 44
The filesystem on /dev/nvme1n1 is now 91750400 (4k) blocks long.
</pre>
The good thing is that as soon as you do that the MySQL daemon see the space and will restart, however it will happen only on the current pod and K8 will still see the old dimension:
<pre class="lang:sh decode:true">k get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON
pvc-1678c7ee-3e50-4329-a5d8-25cd188dc0df 333Gi RWO Delete Bound pxc/datadir-cluster-1-pxc-0 io1
</pre>
To allow k8 to be align with the real dimension we must patch the information stored, the command is the following:
<pre class="lang:sh decode:true">kubectl patch pvc <pvc-name> -n <pvc-namespace> -p '{ "spec": { "resources": { "requests": { "storage": "NEW STORAGE VALUE" }}}}'
Ie:
kubectl patch pvc datadir-cluster-1-pxc-0 -n pxc -p '{ "spec": { "resources": { "requests": { "storage": "350" }}}}'
</pre>
Remember to use as pvc-name the NAME coming from 
<pre>kubectl get pvc.</pre>
Once this is done k8 will see the new volume dimension correctly.
Just repeat the process for Node-1 and Node-2 and …done the cluster is up again.
Finally do not forget to modify your custom resources file (cr.yaml) to match the new volume size. IE:
<pre class="lang:sh decode:true"> volumeSpec:
persistentVolumeClaim:
storageClassName: "io1"
resources:
requests:
storage: 350G
</pre>
The whole process took just a few minutes, it was time now to investigate why the incident happened and why the storage class was not allowing extension in the first place.  
 
<h2>Why it happened</h2>
Well first and foremost the platform was not correctly monitored. As such there was lack of visibility about the space utilization, and no alert about disk space. 
This was easy to solve just enabling the PMM feature in the cluster cr and set the alert in PMM once the nodes join it (see <a href="https://docs.percona.com/percona-monitoring-and-management/get-started/alerting.html">https://docs.percona.com/percona-monitoring-and-management/get-started/alerting.html</a> for details on how to).
The second issue was the problem with the storage class. Once we had the time to carefully review the configuration files, we identified that there was an additional tab in the SC class, which was causing k8 to ignore the directive. 
Was suppose to be:
<pre class="lang:sh decode:true">kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: io1
annotations:
storageclass.kubernetes.io/is-default-class: "false"
provisioner: kubernetes.io/aws-ebs
parameters:
type: io1
iopsPerGB: "12"
fsType: ext4
allowVolumeExpansion: true <----------
It was:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: io1
annotations:
storageclass.kubernetes.io/is-default-class: "false"
provisioner: kubernetes.io/aws-ebs
parameters:
type: io1
iopsPerGB: "12"
fsType: ext4
allowVolumeExpansion: true. <---------</pre>
What was concerning was the lack of error returned by the Kubernetes API, so in theory the configuration was accepted but not really validated. 
In any case once we had fix the typo and recreated the SC, the setting for volume expansion was correctly accepted:
<pre class="lang:sh decode:true">kubectl describe sc io1
Name: io1
IsDefaultClass: No
Annotations: kubectl.kubernetes.io/last-applied-configuration={"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"},"name":"io1"},"parameters":{"fsType":"ext4","iopsPerGB":"12","type":"io1"},"provisioner":"kubernetes.io/aws-ebs"}
,storageclass.kubernetes.io/is-default-class=false
Provisioner: kubernetes.io/aws-ebs
Parameters: fsType=ext4,iopsPerGB=12,type=io1
AllowVolumeExpansion: True
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
</pre>
<h2>What should have happened instead?</h2>
If a proper monitoring and alerting was in place, the administrators would have the time to act and extend the volumes without downtime. 
However, the procedure for extending volumes on K8 is not complex but also not as straightforward as you may think. My colleague Natalia Marukovich wrote this blog post (<a href="https://www.percona.com/blog/percona-operator-volume-expansion-without-downtime/">https://www.percona.com/blog/percona-operator-volume-expansion-without-downtime/</a>)  that gives you the step by step instructions on how to extend the volumes without downtime.
<h2>Conclusions</h2>
Using the cloud, containers, automation or more complex orchestrators like Kubernetes, do not solve all, do not prevent mistakes from happening, and more importantly do not make the right decisions for you. 
You must set up a proper architecture that includes backup, monitoring and alerting. You must set the right alerts and act on them in time. 
Finally automation is cool, however the devil is in the details and typos are his day to day joy. Be careful and check what you put online, do not rush it. Validate, validate validate… 
Great stateful MySQL to all. ]]></description>
<category>MySQL</category>
<pubDate>Thu, 19 Jan 2023 14:11:24 +0000</pubDate>
</item>
<item>
<title>MySQL Dual password how to manage them programmatically</title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/250-mysql-dual-password-how-to-manage-them-programmatically</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/250-mysql-dual-password-how-to-manage-them-programmatically</guid>
<description><![CDATA[What is dual password in MYSQL and how it works was already covered by my colleague Brian Sumpter here (<a href="https://www.percona.com/blog/using-mysql-8-dual-passwords/">https://www.percona.com/blog/using-mysql-8-dual-passwords/</a>). <img src="http://www.tusacentral.net/joomla/images/stories/rpa-cognitive-blog-img.jpg" alt="rpa cognitive blog img" width="400" height="215" style="float: right;" />
However let me do a brief recap here about it.
Dual password is the MySQL mechanism that allows you to keep two passwords active at the same time. This feature is part of a more extended set of Password management features implemented in MySQL 8 to enforce better security and secrets management, like:
<ul>
<li style="font-weight: 400;" aria-level="1">Internal Versus External Credentials Storage</li>
<li style="font-weight: 400;" aria-level="1">Password Expiration Policy</li>
<li style="font-weight: 400;" aria-level="1">Password Reuse Policy</li>
<li style="font-weight: 400;" aria-level="1">Password Verification-Required Policy</li>
<li style="font-weight: 400;" aria-level="1">Dual Password Support</li>
<li style="font-weight: 400;" aria-level="1">Random Password Generation</li>
<li style="font-weight: 400;" aria-level="1">Failed-Login Tracking and Temporary Account Locking</li>
</ul>
The most important and requested features are the password expiration and verification policy. The problem in implementing them is the complexity of replacing passwords for accounts on very large platforms, like with thousands of applications and hundreds of MySQL servers. 
In fact, while for a single user it is not so complex to change his own password when requested at login, for an application using thousands of sub-services it may require some time. The problem in performing the password change is that while executing the modification some services will have the updated password while others will still use the old one. Without Dual Password a segment of nodes will receive error messages in connecting creating service disruption.
Now let us cover this blog topic. 
With Dual Password it is instead possible to declare a new password keeping the old still active until the whole upgrade has been completed. 
This highlight two very important aspects:
<ul>
<li style="font-weight: 400;" aria-level="1">When automating the password update, it is better to not use a password expiration policy, but base the expiration on the completion of the new password deployment.</li>
<li style="font-weight: 400;" aria-level="1">We need to be sure the account we are changing the password to keeps the password active until we need it, and that is correctly removed when done. </li>
</ul>
As you see I am focusing on the cases when we have automation and not the single interactive user update. 
How dual password works
Let us assume we have create a user like:
<pre class="lang:mysql decode:true">create user dualtest@'192.168.4.%' identified by 'password1';
grant all on test.* to dualtest@'192.168.4.%';
</pre>
This will generate an entry in MySQL mysql.user table as:
<pre class="lang:mysql decode:true">(root@localhost) [(none)]>select user,host, plugin, authentication_string, password_last_changed,User_attributes from mysql.user where user = 'dualtest'\G
*************************** 1. row ***************************
user: dualtest
host: 192.168.4.%
plugin: mysql_native_password
authentication_string: *668425423DB5193AF921380129F465A6425216D0
password_last_changed: 2022-11-17 08:31:37
User_attributes: NULL
1 row in set (0.00 sec)
</pre>
At this point our user will be able to connect from any application located in the correct network and act on the test schema. 
After some time, you as application owner,will be notified by your DBA team that the user dualtest is required to change the password in order to respect the security constraints.
At this point there are two options:
<ol>
<li style="font-weight: 400;" aria-level="1">You have privileges  to use Dual Password (the required dynamic privilege to use dual Password is APPLICATION PASSWORD ADMIN).</li>
<li style="font-weight: 400;" aria-level="1">You do not have the right privileges.</li>
</ol>
In case 2 your DBA team must perform the change for you, and then they will let you know the new password.
In case 1 you can do the operation yourself. 
In the last case what you will do is: 
<pre class="lang:mysql decode:true crayon-selected">ALTER USER 'dualtest'@'192.168.4.%' IDENTIFIED BY 'password2' RETAIN CURRENT PASSWORD;
</pre>
Then check it is done properly:
<pre class="lang:mysql decode:true">select user,host, plugin, authentication_string, password_last_changed,User_attributes from mysql.user where user ='dualtest' order by 1,2\G
*************************** 1. row ***************************
user: dualtest
host: 192.168.4.%
plugin: mysql_native_password
authentication_string: *DC52755F3C09F5923046BD42AFA76BD1D80DF2E9
password_last_changed: 2022-11-17 08:46:28
User_attributes: {"additional_password": "*668425423DB5193AF921380129F465A6425216D0"}
1 row in set (0.00 sec)
</pre>
As you can see here the OLD password has been moved to the User_attributes JSON field that is used in MYSQL8 to store several values. 
At this point you can rollout safely the password change and that change can take an hour or a week, no production impact given the applications will be able to use either of them. 
Once the process is complete, you can ask your DBA team to remove OLD password, or do:
<pre class="lang:mysql decode:true">ALTER USER 'dualtest'@'192.168.4.%' DISCARD OLD PASSWORD;</pre>
Then check if the password has being removed properly:
<pre class="lang:mysql decode:true">(root@localhost) [(none)]>select user,host, plugin, authentication_string, password_last_changed,User_attributes from mysql.user where user ='dualtest' order by 1,2\G
*************************** 1. row ***************************
user: dualtest
host: 192.168.4.%
plugin: mysql_native_password
authentication_string: *DC52755F3C09F5923046BD42AFA76BD1D80DF2E9
password_last_changed: 2022-11-17 08:46:28
User_attributes: NULL
1 row in set (0.00 sec)
</pre>
If all is clean the process can be considered complete. 
Of course all this should be automated and executed by code not by hand high level it should be more or less like this:
<pre class="lang:sh decode:true">{}Input new password
- Check for additional_password in User_attributes in mysql.user
<> If no value you can proceed otherwise exit (another change is in place)
- Read and store authentication_string for the user you need to change password
- Change current password with: Alter user ... RETAIN CURRENT PASSWORD
- Check for additional_password in User_attributes in mysql.user
<> If value is present and match the password stored then you can proceed otherwise exit given there is an error in Dual Password or the passwords are different
- Run update on all application nodes, and verify new password on each application node
<> At regular interval check the number of completed changes and check the additional_password in User_attributes in mysql.user to be sure it is still there
[] When all application nodes are up to date
<> If verification is successful 100%
- Remove OLD password with: ALTER USER ... DISCARD OLD PASSWORD;
- Check for additional_password in User_attributes in mysql.user
<> If no value is present close with OK otherwise report Error for password not removed
() complete
</pre>
<h2>Conclusion</h2>
As also Brian mentioned, those are the small things that could make the difference when in large deployments and enterprise environments. Security is a topic that very often is underestimated in small companies or start-ups, but that is wrong, security operations like password rotation are crucial for your safety. 
It is nice to see that MySQL is finally adopting simple but effective steps to help DBAs to implement proper procedures without causing production impact and without the need to become too creative. 
<h2> </h2>
<h2>References </h2>
<a href="https://www.percona.com/blog/using-mysql-8-dual-passwords/">https://www.percona.com/blog/using-mysql-8-dual-passwords/</a>
<a href="https://dev.mysql.com/doc/refman/8.0/en/password-management.html#dual-passwords">https://dev.mysql.com/doc/refman/8.0/en/password-management.html#dual-passwords</a>]]></description>
<category>MySQL</category>
<pubDate>Thu, 17 Nov 2022 16:26:45 +0000</pubDate>
</item>
<item>
<title>ProxySQL support for MySQL caching_sha2_password</title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/249-proxysql-support-for-mysql-caching-sha2-password</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/249-proxysql-support-for-mysql-caching-sha2-password</guid>
<description><![CDATA[In our time, every day we use dozens if not hundreds of applications connecting to some kind of data repository. This simple step is normally executed over the network and given so, it is subject to possible sniffing with all the possible related consequences.<img src="http://www.tusacentral.net/joomla/images/stories/proxysql/brokenlock.png" alt="brokenlock" style="float: right;" /> 
Given that it is normally better to protect your connection using data encryption like SSL, or at the minimum, make the information you pass to connect less easy to be intercepted. 
At the same time it is best practice to not store connection credential in clear text, not even inside a table in your database. Doing that is the equivalent of writing your password over a sticky note on your desk. Not a good idea.
The main options are instead in either transforming the passwords to be less identifiable like hashing or to store the information in an external centralized vault. 
In MySQL the passwords are transformed in order to not be clear text, and several different plugins are used to authenticate the user. From version 8 MySQL uses caching_sha2_password as default authentication plugin. The caching_sha2_password and sha256_password authentication plugins provide more secure password encryption than the mysql_native_password plugin, and caching_sha2_password provides better performance than sha256_password. Due to these superior security and performance characteristics of caching_sha2_password, it is as of MySQL 8.0 the preferred authentication plugin, and is also the default authentication plugin rather than mysql_native_password.
In this regard recently I got the same question again “Can we use ProxySQL with MySQL 8 authorization mechanism?”, and I decided it was time to write this short blog post.
The short answer is “Yes you can”, however do not expect to have full caching_sha2_password support.
This is because ProxySQL does not fully support the caching_sha2_password mechanism internally and given that a “trick” must be used. 
So, what should we do when using MySQL 8 and ProxySQL? 
In the text below we will see what can be done to continue to use ProxySQL with MySQL and Percona server 8. 
Note that I have used the Percona proxysql_admin tool to manage the users except in the last case. Percona proxysql_admin tool is a nice tool that helps you to manage ProxySQL and in regard to user it also manage and synchronize users from your Percona or MySQL  
In the following examples:
Proxysql is on 192.168.4.191
User name/password is msandbox/msandbox
<h2>Using hashing.</h2>
By default MySQL comes with caching_sha2_password as such if I create a user names msandbox I will have: 
<pre class="lang:mysql decode:true">DC1-1(root@localhost) [(none)]>select user,host, authentication_string,plugin from mysql.user order by 1,2;
+----------------------------+--------------------+------------------------------------------------------------------------+-----------------------+
| user | host | authentication_string | plugin |
+----------------------------+--------------------+------------------------------------------------------------------------+-----------------------+
| msandbox | % | $A$005$Z[z@l'O%[Q5t^ EKJDgxjWXJjDpDEUv91oL7Hoh/0NydTeCzpV.aI06C9. | caching_sha2_password | <---- this user
+----------------------------+--------------------+------------------------------------------------------------------------+-----------------------+
</pre>
Then I use percona_scheduler_admin to sync the users:
<pre class="lang:mysql decode:true">./percona-scheduler-admin --config-file=config.toml --syncusers 
Syncing user accounts from PXC(192.168.4.205:3306) to ProxySQL
Removing existing user from ProxySQL: msandbox
Adding user to ProxySQL: msandbox
Synced PXC users to the ProxySQL database!
mysql> select * from mysql_users ;
+------------+------------------------------------------------------------------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+-----------------------------+
| username   | password                                                               | active | use_ssl | default_hostgroup | default_schema | schema_locked | transaction_persistent | fast_forward | backend | frontend | max_connections | attributes | comment                     |
+------------+------------------------------------------------------------------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+-----------------------------+
| msandbox   | $A$005$Z[z@l'O%[Q5t^ EKJDgxjWXJjDpDEUv91oL7Hoh/0NydTeCzpV.aI06C9 | 1      | 0       | 100               | NULL           | 0             | 1                      | 0            | 1       | 1        | 10000           |            |                             |
+------------+------------------------------------------------------------------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+-----------------------------+</pre>
And set the query rules:
<pre class="lang:mysql decode:true">insert into mysql_query_rules (rule_id,proxy_port,username,destination_hostgroup,active,retries,match_digest,apply) values(1048,6033,'msandbox',100,1,3,'^SELECT.*FOR UPDATE',1);
insert into mysql_query_rules (rule_id,proxy_port,username,destination_hostgroup,active,retries,match_digest,apply) values(1050,6033,'msandbox',101,1,3,'^SELECT.*$',1);
load mysql query rules to run;save mysql query rules to disk;
</pre>
Now I try to connect passing by ProxySQL:
<pre class="lang:sh decode:true"># mysql -h 192.168.4.191 -P6033 -umsandbox -pmsandbox
ERROR 1045 (28000): ProxySQL Error: Access denied for user 'msandbox'@'192.168.4.191' (using password: YES)
</pre>
My account will fail to connect given failed authentication.
To fix this I need to drop the user and recreate it with a different authentication plugin in my MySQL server:
<pre class="lang:mysql decode:true">drop user msandbox@'%';
create user 'msandbox'@'%' identified with mysql_native_password BY 'msandbox';
grant select on *.* to 'msandbox'@'%';
select user,host, authentication_string,plugin from mysql.user order by 1,2;
+----------+--------------------+-------------------------------------------+-----------------------+
| user | host | authentication_string | plugin |
+----------+--------------------+-------------------------------------------+-----------------------+
| msandbox | % | *6C387FC3893DBA1E3BA155E74754DA6682D04747 | mysql_native_password |
+----------+--------------------+-------------------------------------------+-----------------------+
</pre>
At this point I can re-run 
<pre class="lang:sh decode:true">./percona-scheduler-admin --config-file=config.toml --syncusers</pre>
if I try to connect again: 
<pre class="lang:mysql decode:true"># mysql -h 192.168.4.191 -P6033  -umsandbox -pmsandbox
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6708563
Server version: 8.0.28 (ProxySQL). <---------------------------- Connecting to proxysql
Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show global variables like 'version%';
+-------------------------+------------------------------------------------------------------------------------+
| Variable_name           | Value                                                                              |
+-------------------------+------------------------------------------------------------------------------------+
| version                 | 8.0.25-15.1         <--- Percona/MySQL version                                     |
| version_comment         | Percona XtraDB Cluster binary (GPL) 8.0.25, Revision 8638bb0, WSREP version 26.4.3 |
| version_compile_machine | x86_64                                                                             |
| version_compile_os      | Linux                                                                              |
| version_compile_zlib    | 1.2.11                                                                             |
| version_suffix          | .1                                                                                 |
+-------------------------+------------------------------------------------------------------------------------+
6 rows in set (0.02 sec)</pre>
This is the only way to keep the password hashed in MySQL and in ProxySQL.
<h2>Not using Hashing</h2>
What if you cannot use mysql_native_password for the password in your MySQL server?
There is a way to still connect, however I do not recommend it given for me is highly insecure, but for completeness I am going to illustrate it.
First of all disable password hashing in Proxysql:
<pre class="lang:mysql decode:true">update global_variables set Variable_Value='false' where Variable_name='admin-hash_passwords'; </pre>
At this point instead sync the user you can locally create the user like:
<pre class="lang:mysql decode:true">insert into mysql_users (username,password,active,default_hostgroup,default_schema,transaction_persistent,comment) values ('msandbox','msandbox',1,100,'mysql',1,'generic test for security'); 
mysql> select * from runtime_mysql_users where username ='msandbox'; 
+----------+----------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+---------------------------+
| username | password | active | use_ssl | default_hostgroup | default_schema | schema_locked | transaction_persistent | fast_forward | backend | frontend | max_connections | attributes | comment                   |
+----------+----------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+---------------------------+
| msandbox | msandbox | 1      | 0       | 100               | mysql          | 0             | 1                      | 0            | 1       | 0        | 10000           |            | generic test for security |
| msandbox | msandbox | 1      | 0       | 100               | mysql          | 0             | 1                      | 0            | 0       | 1        | 10000           |            | generic test for security |
+----------+----------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+---------------------------+</pre>
As you can see doing that will prevent the password to be hashed and instead it will be clear text.
At this point you will be able to connect to MySQL 8 using the caching_sha2_password, but the password is visible in ProxySQL.
Let me repeat, I DO NOT recommend using it this way, because for me it is highly insecure. 
 
<h1>Conclusion</h1>
While it is still possible to configure your user in MySQL to connect using ProxySQL, it is obvious that we have a gap in the way ProxySQL supports security. 
The hope is that this gap will be filled soon by the ProxySQL development team, also if looking to the past issues this seems pending from years now. 
<h1>References</h1>
<a href="https://proxysql.com/documentation/mysql-8-0/">https://proxysql.com/documentation/mysql-8-0/</a>
<a href="https://github.com/sysown/proxysql/issues/2580">https://github.com/sysown/proxysql/issues/2580</a>
https://www.percona.com/blog/upgrade-your-libraries-authentication-plugin-caching_sha2_password-cannot-be-loaded/]]></description>
<category>MySQL</category>
<pubDate>Thu, 03 Nov 2022 13:47:19 +0000</pubDate>
</item>
<item>
<title>Zero impact on index creation with Aurora 3</title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/248-zero-impact-on-index-creation-with-aurora-3</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/248-zero-impact-on-index-creation-with-aurora-3</guid>
<description><![CDATA[<img src="http://www.tusacentral.net/joomla/images/stories/ddl/aurora_ddl_notes.jpeg" alt="aurora ddl notes" style="float: right;" />Last quarter of 2021 AWS released Aurora version 3. This new version aligns Aurora with the latest MySQL 8 version porting many of the advantages MySQL 8 has over previous versions.
While this brings a lot of new interesting features for Aurora, what we are going to cover here is to see how DDLs behave when using the ONLINE option. With a quick comparison with what happens in MySQL 8 standard and with Group Replication. 
<h2>Tests</h2>
All tests were run on an Aurora instance r6g.large with secondary availability zone. The test was composed by:
        4 connections
<ul>
<li style="list-style-type: none;">
<ul>
<li style="font-weight: 400;" aria-level="2">#1 to perform ddl</li>
<li style="font-weight: 400;" aria-level="2">#2 to perform insert data in the table I am altering</li>
<li style="font-weight: 400;" aria-level="2">#3 to perform insert data on a different table </li>
<li style="font-weight: 400;" aria-level="2">#4 checking the other node operations</li>
</ul>
</li>
</ul>
In the Aurora instance, a sysbench schema with 10 tables and 5 million rows was created, just to get a bit of traffic. While the test table with 5ml rows as well was: 
<pre class="lang:mysql decode:true">CREATE TABLE `windmills_test` (
`id` bigint NOT NULL AUTO_INCREMENT,
`uuid` char(36) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`millid` smallint NOT NULL,
`kwatts_s` int NOT NULL,
`date` date NOT NULL,
`location` varchar(50) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`active` tinyint NOT NULL DEFAULT '1',
`time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`strrecordtype` char(3) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`id`),
KEY `IDX_millid` (`millid`,`active`),
KEY `IDX_active` (`id`,`active`),
KEY `kuuid_x` (`uuid`),
KEY `millid_x` (`millid`),
KEY `active_x` (`active`),
KEY `idx_1` (`uuid`,`active`)
) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8mb3 COLLATE=utf8_bin ROW_FORMAT=DYNAMIC
</pre>
The executed commands:
<pre class="lang:sh decode:true">Connection 1:
ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE, LOCK=NONE;
ALTER TABLE windmills_test drop INDEX idx_1, ALGORITHM=INPLACE;
Connection 2:
while [ 1 = 1 ];do da=$(date +'%s.%3N');mysql --defaults-file=./my.cnf -D windmills_large -e "insert into windmills_test select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmills4 limit 1;" -e "select count(*) from windmills_large.windmills_test;" > /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done
Connection 3:
while [ 1 = 1 ];do da=$(date +'%s.%3N');mysql --defaults-file=./my.cnf -D windmills_large -e "insert into windmills3 select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmills4 limit 1;" -e "select count(*) from windmills_large.windmills_test;" > /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done
Connections 4:
while [ 1 = 1 ];do echo "$(date +'%T.%3N')";mysql --defaults-file=./my.cnf -h <secondary aurora instance> -D windmills_large -e "show full processlist;"|egrep -i -e "(windmills_test|windmills_large)"|grep -i -v localhost;sleep 1;done
</pre>
Operations: 1) start inserts from connections 2) start commands in connections 4 - 5 on the other nodes 3) execute: DC1-1(root@localhost) [windmills_large]>ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE, LOCK=NONE;
With this, what I was looking to capture is the operation impact in doing a common action as creating an Index. My desired expectation is to have no impact when doing operations that are declared “ONLINE” such as creating an index, as well as data consistency between nodes. 
Let us see what happened…
<h2>Results</h2>
While running the insert in the same table, performing the alter:
<pre class="lang:mysql decode:true">mysql> ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE, LOCK=NONE;
Query OK, 0 rows affected (16.51 sec)
Records: 0 Duplicates: 0 Warnings: 0
</pre>
Is NOT stopping the operation in the same table or any other table in the Aurora instance.
We can only identify a minimal performance impact:
<pre class="lang:sh decode:true">[root@ip-10-0-0-11 tmp]# while [ 1 = 1 ];do da=$(date +'%s.%3N');mysql --defaults-file=./my.cnf -D windmills_large -e "insert into windmills_test select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmills4 limit 1;" -e "select count(*) from windmills_large.windmills_test;" > /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done
.347
.283
.278
.297
.291
.317
.686 ← start
<Snip>
.512 ← end
.278
.284
.279
</pre>
The secondary node is not affected at all, and this is because Aurora managed at storage level the data replication. Given that there is no such thing as Apply from Relaylog, as we have in standard MySQL asynchronous or data replicated with Group Replication.  
The result is that in Aurora 3, we can have zero impact index (or any other ONLINE/INSTANT) operation, with this I include the data replicated in the other instances for High Availability. 
If we compare this with Group replication (see <a href="https://www.percona.com/blog/online-ddl-with-group-replication-in-mysql-8-0-27/">blog</a>):
<pre class="lang:sh decode:true"> GR Aurora 3
Time on hold for insert for altering table ~0.217 sec ~0.523 sec
Time on hold for insert for another table ~0.211 sec ~0.205 sec
</pre>
However, keep in mind that MySQL with Group Replication will still need to apply the data on the Secondaries. This means that if your alter was taking 10 hours to build the index, the Secondary nodes will be misaligned with the Source for approximately another 10 hours. 
With Aurora 3 or with PXC, changes will be there when Source has completed the operation.    
What about Percona XtraDB Cluster (PXC)? Well, with PXC we have a different scenario:
<pre class="lang:sh decode:true"> PXC(NBO) Aurora 3
Time on hold for insert for altering table ~120 sec ~0.523 sec
Time on hold for insert for another table ~25 sec ~0.205 sec</pre>
We will have a higher impact while doing the Alter operation, but the data will be on all nodes at the same time maintaining a high level of consistency in the cluster. 
<h2>Conclusion</h2>
Aurora is not for all use, and not for all budgets, however it has some very good aspects like the one we have just seen. The Difference between standard MySQL and Aurora is not in the time of holding/locking (aka operation impact), but on the HA aspects. If I have my data/structure on all my Secondary at the same time of the Source, I will feel much more comfortable, than having to wait an additional T time. 
This is why PXC in that case is a better alternative if you can afford locking time, if not, well Aurora 3 is your solution, just do your math properly and be conservative with the instance resources. 
 ]]></description>
<category>MySQL</category>
<pubDate>Wed, 20 Apr 2022 12:05:56 +0000</pubDate>
</item>
<item>
<title>A face to face with semi-synchronous replication</title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/247-a-face-to-face-with-semi-synchronous-replication</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/247-a-face-to-face-with-semi-synchronous-replication</guid>
<description><![CDATA[<a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/face_to_face.jpeg"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/face_to_face.jpeg" alt="" width="300" height="212" class="size-medium wp-image-80671 alignright" style="float: right;" /></a>
Last month I performed a review of the Percona Operator for MySQL Server (<a href="https://www.percona.com/doc/kubernetes-operator-for-mysql/ps/index.html">https://www.percona.com/doc/kubernetes-operator-for-mysql/ps/index.html</a>) which is still Alpha.  That operator is based on Percona Server and uses standard asynchronous replication, with the option to activate semi-synchronous  replication to gain higher levels of data consistency between nodes.
The whole solution is composed as:
<a href="https://www.percona.com/blog/wp-content/uploads/2022/04/operator.svg"><img src="https://www.percona.com/blog/wp-content/uploads/2022/04/operator.svg" alt="" width="499" height="409" class="alignnone wp-image-80663" /></a>
Additionally, Orchestrator (<a href="https://github.com/openark/orchestrator">https://github.com/openark/orchestrator</a>) is used to manage the topology and the settings to enable on the replica nodes, the semi-synchronous flag if required. While we have not too much to say when using standard Asynchronous replication, I want to spend two words on the needs and expectations on the semi-synchronous (semi-sync) solution.
<h2>A look into semi-synchronous</h2>
Difference between Async and Semi-sync. Asynchronous:
<a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/async-replication-diagram.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/async-replication-diagram.png" alt="" width="725" height="304" class="alignnone size-full wp-image-80662" /></a>
The above diagram represents the standard asynchronous replication. This method is expected by design, to have transactions committed on the Source that are not present on the Replicas. The Replica is supposed to catch-up when possible.   
It is also important to understand that there are two steps in replication:
<ul>
<li style="font-weight: 400;" aria-level="1">Data copy, which is normally very fast. The Data is copied from the binlog of the Source to the relay log on the Replica (IO_Thread).</li>
<li style="font-weight: 400;" aria-level="1">Data apply, where the data is read from the relay log on the Replica node and written inside the database itself (SQL_Thread). This step is normally the bottleneck and while there are some parameters to tune, the efficiency to apply transactions depends on many factors including schema design. </li>
</ul>
Production deployments that utilize the Asynchronous solution are typically designed to manage the possible inconsistent scenario given data on Source is not supposed to be on Replica at commit. At the same time the level of High Availability assigned to this solution is lower than the one we normally obtain with (virtually-)synchronous replication, given we may need to wait for the Replicare to catch-up the gap accumulated in the relay-logs before performing the fail-over.
Semi-sync:
<a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/semisync-replication-diagram.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/semisync-replication-diagram.png" alt="" width="725" height="396" class="alignnone size-full wp-image-80665" /></a>
The above diagram represents the Semi-sync replication method.The introduction of semi-sync adds a checking step on the Source before it returns the acknowledgement to the client. This step happens at the moment of the data-copy, so when the data is copied from the Binary-log on Source to the Relay-log on Replica. 
This is important, there is NO mechanism to ensure a more resilient or efficient data replication, there is only an additional step, that tells the Source to wait a given amount of time for an answer from N replicas, and then return the acknowledgement or timeout and return to the client no matter what. 
This mechanism is introducing a possible significant delay in the service, without giving the 100% guarantee of data consistency. 
In terms of availability of the service, when in presence of high load, this method may lead the Source to stop serving the request while waiting for acknowledgements, significantly reducing the availability of the service itself.   
At the same time only acceptable settings for rpl_semi_sync_source_wait_point is AFTER_SYNC (default) because:  In the event of source failure, all transactions committed on the source have been replicated to the replica (saved to its relay log). An unexpected exit of the source server and failover to the replica is lossless because the replica is up to date.
All clear? No? Let me simplify the thing. 
<ul>
<li style="font-weight: 400;" aria-level="1">In standard replication you have two moments (I am simplifying)
<ul>
<li style="font-weight: 400;" aria-level="2">Copy data from Source to Replica</li>
<li style="font-weight: 400;" aria-level="2">Apply data in the Replica node</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">There is no certification on the data applied about its consistency with the Source</li>
<li style="font-weight: 400;" aria-level="1">With asynchronous the Source task is to write data in the binlog and forget</li>
<li style="font-weight: 400;" aria-level="1">With semi-sync the Source writes the data on binlog and waits T seconds to receive acknowledgement from N servers about them having received the data.</li>
</ul>
To enable semi-sync you follow these steps:  <a href="https://dev.mysql.com/doc/refman/8.0/en/replication-semisync-installation.html">https://dev.mysql.com/doc/refman/8.0/en/replication-semisync-installation.html</a>
In short:
<ul>
<li style="font-weight: 400;" aria-level="1">Register the plugins</li>
<li style="font-weight: 400;" aria-level="1">Enable Source rpl_semi_sync_source_enabled=1</li>
<li style="font-weight: 400;" aria-level="1">Enable Replica rpl_semi_sync_replica_enabled = 1
<ul>
<li style="font-weight: 400;" aria-level="2">If replication is already running STOP/START REPLICA IO_THREAD</li>
</ul>
</li>
</ul>
And here starts the fun, be ready for many “wait whaaat?”. 
What is the T and N I have just mentioned above?
Well the T is a timeout that you can set to avoid having the source wait forever for the Replica acknowledgement. The default is 10 seconds. What happens if the Source waits for more than the timeout?  <img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/wait_whaat.jpeg" alt="" width="115" height="79" class="wp-image-80666 alignright" style="float: right;" /> rpl_semi_sync_source_timeout controls how long the source waits on a commit for acknowledgment from a replica before timing out and reverting to asynchronous replication.
Careful of the wording here! The manual says SOURCE, so it is not that MySQL revert to asynchronous, by transaction or connection, it is for the whole server.
Now analyzing the work-log (see <a href="https://dev.mysql.com/worklog/task/?id=1720">https://dev.mysql.com/worklog/task/?id=1720</a> and more in the references) the Source should revert to semi-synchronous as soon as all involved replicas are aligned again. 
However, checking the code (see  <a href="https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/plugin/semisync/semisync_source.cc#L844">https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/plugin/semisync/semisync_source.cc#L844</a> and following), we can see that we do not have a 100% guarantee that the Source will be able to switch back.
Also in the code: But, it is not that easy to detect that the replica has caught up.  This is caused by the fact that MySQL's replication protocol is  asynchronous, meaning that if thesource does not use the semi-sync protocol, the replica would not send anything to thesource.
In all the runned tests the Source was not able to switch back. In short Source was moving out from semi-sync and that was forever, no rollback. Keep in mind that while we go ahead.
What is the N I mentioned above? It represents the number of Replicas that must provide the acknowledgement back. 
If you have a cluster of 10 nodes you may need to have only 2 of them involved in the semi-sync, no need to include them all. But if you have a cluster of 3 nodes where 1 is the Source, relying on 1 Replica only, is not really secure. What I mean here is that if you choose to be semi-synchronous to ensure the data replicates, having it enabled for one single node is not enough, if that node crashes or whatever, you are doomed, as such you need at least 2 nodes with semi-sync.
Anyhow, the point is that if one of the Replica takes more than T to reply, the whole mechanism stops working, probably forever. 
As we have seen above, to enable semi-sync on Source we manipulate the value of the GLOBAL variable rpl_semi_sync_source_enabled.
However if I check the value of rpl_semi_sync_source_enabled when the Source shift to simple Asynchronous replication because timeout:
select @@rpl_semi_sync_source_enabled;
<pre class="lang:mysql decode:true">select @@rpl_semi_sync_source_enabled;
+--------------------------------+
| @@rpl_semi_sync_source_enabled |
+--------------------------------+
| 1 |
+--------------------------------+
</pre>
As you can see the Global variable reports a value of 1, meaning that semi-sync is active also if not.
In the documentation it is reported that to monitor the semi-sync activity we should check for Rpl_semi_sync_source_status. Which means that you can have <a href="https://www.percona.com/blog/wp-content/uploads/2022/04/wait_whaat.jpeg"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/wait_whaat.jpeg" alt="" width="113" height="77" class="wp-image-80666 alignright" style="float: right;" /></a> Rpl_semi_sync_source_status = 0 and rpl_semi_sync_source_enabled =1 at the same time.
Is this a bug? Well according to documentation: When the source switches between asynchronous or semisynchronous replication due to commit-blocking timeout or a replica catching up, it sets the value of the Rpl_semi_sync_source_status or Rpl_semi_sync_source_status status variable appropriately. Automatic fallback from semisynchronous to asynchronous replication on the source means that it is possible for the rpl_semi_sync_source_enabled or rpl_semi_sync_source_enabled system variable to have a value of 1 on the source side even when semisynchronous replication is in fact not operational at the moment. You can monitor the Rpl_semi_sync_source_status or Rpl_semi_sync_source_status status variable to determine whether the source currently is using asynchronous or semisynchronous replication.
It is not a bug. However, because you documented it, it doesn’t change the fact this is a weird/unfriendly/counterintuitive way of doing, that opens the door to many, many possible issues. Especially given you know the Source may fail to switch semi-synch back.  
Just to close this part, we can summarize as follows:
<ul>
<li style="font-weight: 400;" aria-level="1">You activate semi-sync setting a global variable</li>
<li style="font-weight: 400;" aria-level="1">Server/Source can disable it (silently) without changing that variable </li>
<li style="font-weight: 400;" aria-level="1">Server will never restore semi-sync automatically</li>
<li style="font-weight: 400;" aria-level="1">The way to check if semi-sync works is to use the Status variable</li>
<li style="font-weight: 400;" aria-level="1">When Rpl_semi_sync_source_status = 0 and rpl_semi_sync_source_enabled =1 you had a Timeout and Source is now working in asynchronous replication</li>
<li style="font-weight: 400;" aria-level="1">The way to reactivate semi-sync is to set rpl_semi_sync_source_enabled  to OFF first then rpl_semi_sync_source_enabled = ON. </li>
<li style="font-weight: 400;" aria-level="1">Replicas can be set with semi-sync ON/OFF but unless you do not STOP/START the replica_IO_THREAD the state of the variable can be inconsistent with the state of the Server.</li>
</ul>
What can go wrong?
<h2>Semi-synchronous is not seriously affecting the performance</h2>
Others had already discussed semi-sync performance in better details. However I want to add some color given the recent experience with our operator testing. In the next graphs I will show you the behavior of writes/reads using Asynchronous replication and the same load with Semi-synchronous. For the record the test was a simple Sysbench-tpcc test using 20 tables, 20 warehouses, 256 threads for 600 seconds.
<a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/2.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/2.png" alt="" width="900" height="401" class="alignnone wp-image-80652 size-large" /></a> <a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/3.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/3.png" alt="" width="900" height="421" class="alignnone size-large wp-image-80653" /></a>
The one above indicates a nice and consistent set of load in r/w with minimal fluctuations. This is what we like to have. 
The graphs below, represent the exact same load on the exact same environment but with semi-sync activated and no timeout. 
<a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/9.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/9.png" alt="" width="900" height="421" class="alignnone wp-image-80657 size-large" /></a> <a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/10.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/10.png" alt="" width="900" height="420" class="alignnone wp-image-80658 size-large" /></a>
Aside from the performance loss (we went from Transaction 10k/s to 3k/s), the constant stop/go imposed by the semi-sync mechanism has a very bad effect on the application behavior when you have many concurrent threads and high loads. I challenge any serious production system to work in this way.   
Of course results are inline with this yoyo game:
<a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/1c.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/1c.png" alt="" width="900" height="356" class="alignnone size-large wp-image-80651" /></a>
In the best case, when all was working as expected, and no crazy stuff happening I had something around the 60% loss. I am not oriented to see this as  a minor performance drop. 
<h2>But at least your data is safe</h2>
As already stated at the beginning the scope of semi-synchronous replication is to guarantee that the data in server A reaches server B before returning the OK to the application. 
In short, given a period of 1 second we should have minimal transactions in flight and limited transactions in the apply queue. While for standard replication (asynchronous), we may have … thousands. 
In the graphs below we can see two lines:
<ul>
<li style="font-weight: 400;" aria-level="1">The yellow line represents the number of GTIDs “in flight” from Source to destination, Y2 axes.  In case of Source crash, those transactions are lost and we will have data loss.</li>
<li style="font-weight: 400;" aria-level="1">The blue line represents the number of GTIDs already copied over from Source to Replica but not applied in the database Y1 axes. In case of Source crash we must wait for the Replica to process these entries, before making the node Write active, or we will have data inconsistency.</li>
</ul>
Asynchronous replication:
<a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/4.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/4.png" alt="" width="900" height="526" class="alignnone wp-image-80654 size-large" /></a>
As expected we can see a huge queue in applying the transactions from relay-log, and some spike of transactions in flight. 
Using Semi-synchronous replication:
<a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/8.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/8.png" alt="" width="900" height="421" class="alignnone size-large wp-image-80656" /></a>
Yes, apparently we have reduced the queue and no spikes so no data loss.
But this happens  when all goes as expected, and we know in production this is not the normal. What if we need to enforce the semi-sync but at the same time we cannot set the Timeout to ridiculous values like 1 week? 
Simple, we need to have a check that puts back the semi-sync as soon as it is silently disabled (for any reason). However doing this without waiting for the Replicas to cover the replication gap, cause the following interesting effects:
<a href="http://www.tusacentral.net/joomla/images/stories/semi-sync/5.png"><img src="http://www.tusacentral.net/joomla/images/stories/semi-sync/5.png" alt="" width="900" height="525" class="alignnone size-large wp-image-80655" /></a>
Thousands of transactions queued and shipped with the result of having a significant increase of the possible data loss and still a huge number of data to apply from the relay-log. 
So the only possible alternative is to set the Timeout to a crazy value, However this can cause a full production stop in the case a Replica hangs or for any reason it disables the semi-sync locally. 
 
<h2>Conclusion</h2>
First of all I want to say that the tests on our operator using Asynchronous replication, shows a consistent behavior with the standard deployments in the cloud or premises.  It has the same benefits, like better performance and same possible issues as longer time to failover when it needs to wait a Replica to apply the relay-log queue. 
The semi-synchronous flag in the operator is disabled, and the tests I have done bring me to say “keep it like that!”. At least unless you know very well what you are doing and are able to deal with a semi-sync timeout of days.
I was happy to have the chance to perform these tests, because they gives me a way/time/need to investigate more on the semi-synchronous feature. Personally, I was not convinced about the semi-synchronous replication when it came out, and I am not now. I never saw a less consistent and less trustable feature in MySQL as semi-sync.  
If you need to have a higher level of synchronicity in your database just go for Group Replication, or Percona XtraDB Cluster and stay away from semi-sync. 
Otherwise, stay on Asynchronous replication, which is not perfect but it is predictable.  
<h2>References</h2>
<a href="https://www.percona.com/blog/2012/01/19/how-does-semisynchronous-mysql-replication-work/">https://www.percona.com/blog/2012/01/19/how-does-semisynchronous-mysql-replication-work/</a>
<a href="https://www.percona.com/blog/percona-monitoring-and-management-mysql-semi-sync-summary-dashboard/">https://www.percona.com/blog/percona-monitoring-and-management-mysql-semi-sync-summary-dashboard/</a>
<a href="https://www.percona.com/blog/2012/06/14/comparing-percona-xtradb-cluster-with-semi-sync-replication-cross-wan/">https://www.percona.com/blog/2012/06/14/comparing-percona-xtradb-cluster-with-semi-sync-replication-cross-wan/</a>
<a href="https://datto.engineering/post/lossless-mysql-semi-sync-replication-and-automated-failover">https://datto.engineering/post/lossless-mysql-semi-sync-replication-and-automated-failover</a>
<a href="https://planetscale.com/blog/mysql-semi-sync-replication-durability-consistency-and-split-brains">https://planetscale.com/blog/mysql-semi-sync-replication-durability-consistency-and-split-brains</a>
<a href="https://percona.community/blog/2018/08/23/question-about-semi-synchronous-replication-answer-with-all-the-details/">https://percona.community/blog/2018/08/23/question-about-semi-synchronous-replication-answer-with-all-the-details/</a>
<a href="https://dev.mysql.com/doc/refman/8.0/en/replication-semisync-installation.html">https://dev.mysql.com/doc/refman/8.0/en/replication-semisync-installation.html</a>
<a href="https://dev.mysql.com/worklog/task/?id=1720">https://dev.mysql.com/worklog/task/?id=1720</a>
<a href="https://dev.mysql.com/worklog/task/?id=6630">https://dev.mysql.com/worklog/task/?id=6630</a>
<a href="https://dev.mysql.com/worklog/task/?id=4398">https://dev.mysql.com/worklog/task/?id=4398</a>
<a href="https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/plugin/semisync/semisync_source.cc#L844">https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/plugin/semisync/semisync_source.cc#L844</a>
<a href="https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/plugin/semisync/semisync_source.cc#L881">https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/plugin/semisync/semisync_source.cc#L881</a>]]></description>
<category>MySQL</category>
<pubDate>Tue, 12 Apr 2022 10:00:01 +0000</pubDate>
</item>
<item>
<title>Online DDL with Group Replication In MySQL 8.0.27</title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/246-online-ddl-with-group-replication-in-mysql-8-0-27</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/246-online-ddl-with-group-replication-in-mysql-8-0-27</guid>
<description><![CDATA[Last April 2021 I wrote an <a href="https://www.percona.com/blog/2021/04/15/online-ddl-with-group-replication-in-percona-server-for-mysql-8-0-22/">article</a> about Online DDL and Group Replication. At that time we were dealing with MySQL 8.0.23 and also opened a <a href="https://bugs.mysql.com/bug.php?id=103421">bug</a> report which did not have the right answer to the case presented.
Anyhow, in that article I have shown how an online DDL was de facto locking the whole cluster for a very long time even when using the consistency level set to EVENTUAL.
This article is to give justice to the work done by the MySQL/Oracle engineers to correct that annoying inconvenience. 
Before going ahead, let us remember how an Online DDL was propagated in a group replication cluster, and identify the differences with what happens now, all with the consistency level set to EVENTUAL (<a href="https://dev.mysql.com/doc/refman/8.0/en/group-replication-configuring-consistency-guarantees.html">see</a>).
In MySQL 8.0.23 we were having:
<table>
<tbody>
<tr>
<td><a href="https://www.percona.com/blog/wp-content/uploads/2022/01/1-gr-ddl.png"><img src="http://www.tusacentral.net/joomla/images/stories/gr_ddl_8_0_27/1-gr-ddl.png" alt="1 gr ddl" width="201" height="222" class="alignnone size-full wp-image-79683" /></a></td>
<td><a href="https://www.percona.com/blog/wp-content/uploads/2022/01/gr-ddl-2-old.png"><img src="http://www.tusacentral.net/joomla/images/stories/gr_ddl_8_0_27/gr-ddl-2-old.png" alt="gr ddl 2 old" width="201" height="222" class="alignnone size-full wp-image-79689" /></a></td>
<td><a href="https://www.percona.com/blog/wp-content/uploads/2022/01/gr-ddl-3-old.png"><img src="http://www.tusacentral.net/joomla/images/stories/gr_ddl_8_0_27/gr-ddl-3-old.png" alt="gr ddl 3 old" width="211" height="242" class="alignnone size-full wp-image-79684" /></a></td>
</tr>
</tbody>
</table>
While in MySQL 8.0.27 we have:
<table>
<tbody>
<tr>
<td><a href="https://www.percona.com/blog/wp-content/uploads/2022/01/1-gr-ddl.png"><img src="http://www.tusacentral.net/joomla/images/stories/gr_ddl_8_0_27/1-gr-ddl-new.png" alt="1 gr ddl new" width="201" height="222" class="alignnone size-full wp-image-79683" /></a></td>
<td><a href="https://www.percona.com/blog/wp-content/uploads/2022/01/gr-ddl-2-new.png"><img src="http://www.tusacentral.net/joomla/images/stories/gr_ddl_8_0_27/gr-ddl-2-new.png" alt="gr ddl 2 new" width="231" height="257" class="alignnone size-full wp-image-79686" /></a></td>
<td><a href="https://www.percona.com/blog/wp-content/uploads/2022/01/gr-ddl-3-new.png"><img src="http://www.tusacentral.net/joomla/images/stories/gr_ddl_8_0_27/gr-ddl-3-new.png" alt="gr ddl 3 new" width="201" height="212" class="alignnone size-full wp-image-79687" /></a></td>
</tr>
</tbody>
</table>
 
As you can see from the images we have 3 different phases. Phase 1 is the same between version 8.0.23 and version 8.0.27. 
Phase 2 and 3 instead are quite different. In MySQL 8.0.23 after the DDL is applied on the Primary it is propagated to the other nodes, but a metalock was also acquired and the control was NOT returned. The result was that not only the session executing the DDL was kept on hold, but also all the other sessions performing modifications. 
Only when the operation was over on all secondaries, the DDL was pushed to Binlog and disseminated for Asynchronous replication, lock raised and operation can restart.
Instead, in MySQL 8.0.27,  once the operation is over on the primary the DDL is pushed to binlog, disseminated to the secondaries and control returned. The result is that the write operations on primary have no interruption whatsoever and the DDL is distributed to secondary and Asynchronous replication at the same time. 
This is a fantastic improvement, available only with consistency level EVENTUAL, but still, fantastic.
<h3>Let's see some numbers.</h3>
To test the operation, I have used the same approach used in the previous tests in the article mentioned above.
<pre class="lang:sh decode:true">Connection 1:
ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE, LOCK=NONE;
ALTER TABLE windmills_test drop INDEX idx_1, ALGORITHM=INPLACE;
Connection 2:
while [ 1 = 1 ];do da=$(date +'%s.%3N');/opt/mysql_templates/mysql-8P/bin/mysql --defaults-file=./my.cnf -uroot -D windmills_large -e "insert into windmills_test select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmill7 limit 1;" -e "select count(*) from windmills_large.windmills_test;" > /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done
Connection 3:
while [ 1 = 1 ];do da=$(date +'%s.%3N');/opt/mysql_templates/mysql-8P/bin/mysql --defaults-file=./my.cnf -uroot -D windmills_large -e "insert into windmill8 select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmill7 limit 1;" -e "select count(*) from windmills_large.windmills_test;" > /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done
Connections 4-5:
while [ 1 = 1 ];do echo "$(date +'%T.%3N')";/opt/mysql_templates/mysql-8P/bin/mysql --defaults-file=./my.cnf -uroot -D windmills_large -e "show full processlist;"|egrep -i -e "(windmills_test|windmills_large)"|grep -i -v localhost;sleep 1;done
</pre>
Modifying a table with ~5 million rows:
<pre class="lang:mysql decode:true">node1-DC1 (root@localhost) [windmills_large]>select count(*) from windmills_test;
+----------+
| count(*) |
+----------+
| 5002909 |
+----------+
</pre>
The numbers below represent the time second.milliseconds taken by the operation to complete. While I was also catching the state of the ALTER on the other node I am not reporting it here given it is not relevant. 
<pre class="lang:sh decode:true">EVENTUAL (on the primary only)
-------------------
Node 1 same table:
.184
.186 <--- no locking during alter on the same node
.184
<snip>
.184
.217 <--- moment of commit
.186
.186
.186
.185
Node 1 another table :
.189
.198 <--- no locking during alter on the same node
.188
<snip>
.191
.211 <--- moment of commit
.194
</pre>
As you can see there is just a very small delay at the moment of commit but other impact.
Now if we compare this with the recent tests I have done for PXC Non Blocking operation (see <a href="https://www.percona.com/blog/percona-xtradb-cluster-non-blocking-operation-for-online-schema-upgrade/">here</a>) with same number of rows and same kind of table/data:
<table border="1">
<thead>
<tr><th>Action</th><th align="right">Group Replication</th><th align="right">PXC (NBO)</th></tr>
</thead>
<tbody>
<tr>
<td>Time on hold for insert in altering table</td>
<td align="right">~ 0.217 sec</td>
<td align="right">~ 120 sec</td>
</tr>
<tr>
<td>Time on hold for insert in another table</td>
<td align="right">~ 0.211 sec</td>
<td align="right">~ 25 sec</td>
</tr>
</tbody>
</table>
 
However, yes there is a however, PXC was maintaining consistency between the different nodes during the DDL execution, while MySQL 8.0.27 with Group Replication was postponing consistency on the secondaries, thus Primary and Secondary were not in sync until full DDL finalization on the secondaries.
<h1>Conclusions</h1>
MySQL 8.0.27 comes with this nice fix that significantly reduces the impact of an online DDL operation on a busy server. But we can still observe a significant misalignment of the data between the nodes when a DDL is executing. 
On the other hand PXC with NBO is a bit more “expensive” in time, but nodes remain aligned all the time.
At the end is what is more important for you to choose one or the other solution, consistency vs. operational impact.
Great MySQL to all.]]></description>
<category>MySQL</category>
<pubDate>Tue, 11 Jan 2022 13:04:00 +0000</pubDate>
</item>
<item>
<title>A look into Percona XtraDB Cluster Non Blocking Operation for Online Schema Upgrade</title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/245-a-look-into-percona-xtradb-cluster-non-blocking-operation-for-online-schema-upgrade</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/245-a-look-into-percona-xtradb-cluster-non-blocking-operation-for-online-schema-upgrade</guid>
<description><![CDATA[Percona XtraDB Cluster 8.0.25 has introduced a new option to perform online schema modifications: NBO (<a href="https://www.percona.com/doc/percona-xtradb-cluster/LATEST/features/nbo.html#nbo">Non Blocking Operation</a>).
When using PXC the cluster relies on wsrep_OSU_method parameter to define the Online Schema Upgrade (OSU) method the node uses to replicate DDL statements.  <img src="http://www.tusacentral.net/joomla/images/stories/NBO/breaking_bariers.jpg" alt="breaking bariers" width="400" height="239" style="float: right;" />
Until now we normally have 3 options:
<ul>
<li style="font-weight: 400;" aria-level="1">Use Total Isolation Order (TOI, the default)</li>
<li style="font-weight: 400;" aria-level="1">Use Rolling Schema Upgrade (RSU)</li>
<li style="font-weight: 400;" aria-level="1">Use Percona’s online schema change tool (TOI + <a href="https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html">PTOSC</a>)</li>
</ul>
Each method has some positive and negative aspects. TOI will lock the whole cluster from being able to accept data modifications for the entire time it takes to perform the DDL operation. RSU will misalign the schema definition between the nodes, and in any case the node performing the DDL operation is still locked. Finally TOI+PTOSC will rely on creating triggers and copying data, so in some cases this can be very impactful. 
The new Non Blocking Operation (NBO) method is to help to reduce the impact on the cluster and make it easier to perform some DDL operations.
At the moment we only support a limited set of operations with NBO like:
<ul>
<li style="font-weight: 400;" aria-level="1">ALTER INDEX</li>
<li style="font-weight: 400;" aria-level="1">CREATE INDEX</li>
<li style="font-weight: 400;" aria-level="1">DROP INDEX</li>
</ul>
Any other command will result in an error message ER_NOT_SUPPORTED_YET.
But let us see how it works and what is the impact while we will also compare it with the default method TOI.
What we will do is working with 4 connections:
1 to perform ddl 2 to perform insert data in the table being altered 3 to perform insert data on a different table  4-5 checking the other two nodes operations
PXC must be at least Ver 8.0.25-15.1.
The table we will modify is :
<pre class="lang:mysql decode:true">DC1-1(root@localhost) [windmills_s]>show create table windmills_test\G
*************************** 1. row ***************************
Table: windmills_test
Create Table: CREATE TABLE `windmills_test` (
`id` bigint NOT NULL AUTO_INCREMENT,
`uuid` char(36) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`millid` smallint NOT NULL,
`kwatts_s` int NOT NULL,
`date` date NOT NULL,
`location` varchar(50) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`active` tinyint NOT NULL DEFAULT '1',
`time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`strrecordtype` char(3) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`id`),
KEY `IDX_millid` (`millid`,`active`),
KEY `IDX_active` (`id`,`active`),
KEY `kuuid_x` (`uuid`),
KEY `millid_x` (`millid`),
KEY `active_x` (`active`)
) ENGINE=InnoDB AUTO_INCREMENT=8199260 DEFAULT CHARSET=utf8mb3 COLLATE=utf8_bin ROW_FORMAT=DYNAMIC
1 row in set (0.00 sec)
</pre>
And contains ~5 million rows.
<pre class="lang:sh decode:true">DC1-1(root@localhost) [windmills_s]>select count(*) from windmills_test;
+----------+
| count(*) |
+----------+
| 5002909 |
+----------+
1 row in set (0.44 sec)
</pre>
The commands. Connection 1:
<pre class="lang:sh decode:true"> ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE;
ALTER TABLE windmills_test drop INDEX idx_1, ALGORITHM=INPLACE;
</pre>
 
Connection 2:
<pre class="lang:sh decode:true">while [ 1 = 1 ];do da=$(date +'%s.%3N');/opt/mysql_templates/PXC8P/bin/mysql --defaults-file=./my.cnf -uroot -D windmills_s -e "insert into windmills_test select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmills7 limit 1;" -e "select count(*) from windmills_s.windmills_test;" > /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done</pre>
 
Connection 3:
<pre class="lang:sh decode:true"> while [ 1 = 1 ];do da=$(date +'%s.%3N');/opt/mysql_templates/PXC8P/bin/mysql --defaults-file=./my.cnf -uroot -D windmills_s -e "insert into windmills8 select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmills7 limit 1;" -e "select count(*) from windmills_s.windmills_test;" > /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done</pre>
 
Connections 4-5:
<pre class="lang:sh decode:true">while [ 1 = 1 ];do echo "$(date +'%T.%3N')";/opt/mysql_templates/PXC8P/bin/mysql --defaults-file=./my.cnf -uroot -D windmills_s -e "show full processlist;"|egrep -i -e "(windmills_test|windmills_s)"|grep -i -v localhost;sleep 1;done</pre>
Operations:
<ul>
<li>start inserts from connections</li>
<li>start commands in connections 4 - 5 on the other nodes</li>
<li>execute: 
<ul>
<li style="font-weight: 400;" aria-level="1">for TOI 
<ul>
<li aria-level="2">
<pre class="">DC1-1(root@localhost) [windmills_s]>SET SESSION wsrep_OSU_method=TOI;</pre>
</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">for NBO
<ul>
<li aria-level="2">
<pre class="">DC1-1(root@localhost) [windmills_s]>SET SESSION wsrep_OSU_method=NBO;</pre>
</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">For both
<ul>
<li>
<pre class="">DC1-1(root@localhost) [windmills_s]>ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE, LOCK=shared;</pre>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<h1> </h1>
<h1 style="padding-left: 40px;">Let us run it</h1>
<h2>Altering a table with TOI.</h2>
<pre class="lang:sh decode:true">DC1-1(root@localhost) [windmills_s]>ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE;
Query OK, 0 rows affected (1 min 4.74 sec)
Records: 0 Duplicates: 0 Warnings: 0
</pre>
 
Inserts in the altering table (connection 2):
<pre class="lang:sh decode:true">.450
.492
64.993 <--- Alter blocks all inserts on the table we are altering
.788
.609
</pre>
 
Inserts on the other table (connection 3):
<pre class="lang:sh decode:true">.455
.461
64.161 <--- Alter blocks all inserts on all the other tables as well
.641
.483
</pre>
 
On the other nodes at the same time of the ALTER we can see:
<pre class="lang:sh decode:true">Id User db Command Time State Info Time_ms Rows_sent Rows_examined
15 system user windmills_s Query 102 altering table ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE 102238 0 0 <--- time from start
</pre>
So in short we have the whole cluster locked for ~64 seconds. During this period of time, all the operations to modify data or structure were on hold. 
<h2> </h2>
<h2>Let us now try with NBO</h2>
Inserts in the altering table:
<pre class="lang:sh decode:true">.437
.487
120.758 <---- Execution time increase
.617
.510
</pre>
 
Inserts on the other table:
<pre class="lang:sh decode:true">.468
.485
25.061 <---- still a metalock, but not locking the other tables for the whole duration
.494
.471
</pre>
 
On the other nodes at the same time of the ALTER we can see:
<pre class="lang:sh decode:true">Id User db Command Time State Info Time_ms Rows_sent Rows_examined
110068 system user windmills_s Connect 86 altering table ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE 120420 0 0
</pre>
 
In this case what is also interesting to note is that:
<ol>
<li style="font-weight: 400;" aria-level="1">We have a moment of metalock:<ol>
<li aria-level="2">
<pre class="">110174 pmm 127.0.0.1:42728 NULL Query 2 Waiting for table metadata lock SELECT x FROM information_schema.tables  WHERE TABLE_SCHEMA = 'windmills_s' 1486 10   0</pre>
</li>
<li aria-level="2">
<pre class="">110068 system user connecting host windmills_s Connect 111 closing tables ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE 111000 0 0</pre>
</li>
</ol></li>
<li>The execution time is longer </li>
</ol>
Summarizing:
<pre class="lang:sh decode:true"> TOI NBO
Time on hold for insert for altering table ~64 sec ~120 sec
Time on hold for insert for another table ~64 sec ~25 sec
metalock whole time only at the end
</pre>
<h2> </h2>
<h2>What is happening, what are the differences and why takes longer with NBO?</h2>
Let see at very high level how the two works:
<ul>
<li style="font-weight: 400;" aria-level="1">TOI: when you issue a DDL like ADD INDEX a metadata lock is taken on the table and it will be released only at the end of the operation. During this time, you cannot: 
<ul>
<li style="font-weight: 400;" aria-level="2">Perform DMLs on any cluster node</li>
<li style="font-weight: 400;" aria-level="2">Alter another table in the cluster</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">NBO: the metadata lock is taken at the start and at the end for a very brief period of time. The ADD INDEX operation will then work on each node independently. The lock taken at the end is to have all the nodes agree on the operation and commit or roll back (using cluster error voting). This final phase costs a bit more in time and is what adds a few seconds to the operation execution. But during the operation:
<ul>
<li style="font-weight: 400;" aria-level="2">You can alter another table (using NBO)</li>
<li style="font-weight: 400;" aria-level="2">You can continue to insert data, except in the table(s) you are altering.</li>
<li style="font-weight: 400;" aria-level="2">On node crash the operation will continue on the other nodes, and if successful it will persist.  </li>
</ul>
</li>
</ul>
In short the cluster server behavior changes significantly when using NBO, offering significant flexibility compared to TOI. The cost in time should not linearly increase with the dimension of the table, but more in relation to the single node efficiency in performing the ALTER operation.       
<h1>Conclusion</h1>
NBO can be significantly helpful to reduce the impact of DDL on the cluster, for now limited to the widely used creation/modification/drop of an index. But in the future … we may expand it. 
The feature is still a technology preview, so do not trust in production, but test it and let us know what you think. 
Final comment. Another distribution has introduced NBO, but only if you buy the enterprise version.
Percona, which is truly open source with facts not just words, has implemented NBO in standard PXC, and the code is fully open source. This is not the first one, but just another of the many features Percona is offering for free, while others ask you to buy the enterprise version.
Enjoy the product and let us have your feedback!
Great MySQL to all! 
 ]]></description>
<category>MySQL</category>
<pubDate>Fri, 10 Dec 2021 10:00:29 +0000</pubDate>
</item>
<item>
<title>What if … MySQL’s repeatable reads cause you to lose money?</title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/244-what-if-mysql-s-repeatable-reads-cause-you-to-lose-money</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/244-what-if-mysql-s-repeatable-reads-cause-you-to-lose-money</guid>
<description><![CDATA[Well, let me say if that happens because there is a logic mistake in your application. But you need to know and understand what happens in MySQL to be able to avoid the problem. 
In short the WHY of this article is to inform you about possible pitfalls and how to prevent that to cause you damage. <img src="http://www.tusacentral.net/joomla/images/stories/repeatable_read/pitfalls1.jpg" alt="pitfalls1" style="float: right;" />
Let us start by having a short introduction to what Repeatable reads are about. Given I am extremely lazy, I am going to use (a lot) existing documentation from <a href="https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html">MySQL documentation.</a>
Transaction isolation is one of the foundations of database processing. Isolation is the I in the acronym ACID; the isolation level is the setting that fine-tunes the balance between performance and reliability, consistency, and reproducibility of results when multiple transactions are making changes and performing queries at the same time.
InnoDB offers all four transaction isolation levels described by the SQL:1992 standard: READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ, and SERIALIZABLE. The default isolation level for InnoDB is REPEATABLE READ.
<ul>
<li style="font-weight: 400;" aria-level="1">REPEATABLE READ This is the default isolation level for InnoDB. Consistent reads within the same transaction read the snapshot established by the first read. This means that if you issue several plain (nonlocking) SELECT statements within the same transaction, these SELECT statements are consistent also with respect to each other.</li>
</ul>
<ul>
<li style="font-weight: 400;" aria-level="1">READ COMMITTED Each consistent read, even within the same transaction, sets and reads its own fresh snapshot.</li>
</ul>
And about Consistent Non blocking reads: A consistent read means that InnoDB uses multi-versioning to present to a query a snapshot of the database at a point in time. The query sees the changes made by transactions that committed before that point in time, and no changes made by later or uncommitted transactions. The exception to this rule is that the query sees the changes made by earlier statements within the same transaction. This exception causes the following anomaly: If you update some rows in a table, a SELECT sees the latest version of the updated rows, but it might also see older versions of any rows. If other sessions simultaneously update the same table, the anomaly means that you might see the table in a state that never existed in the database.
Ok, but what does all this mean in practice?
To understand, let us simulate this scenario:
I have a shop and I decide to grant a bonus discount to a selected number of customers that:
<ul>
<li style="font-weight: 400;" aria-level="1">Have an active account to my shop</li>
<li style="font-weight: 400;" aria-level="1">Match my personal criteria to access the bonus</li>
</ul>
My application is set to perform batch operations in a moment of less traffic and unattended. 
This includes reactivating dormant accounts that customers ask to reactivate.  
What we will do is to see what happens, by default, then see what we can do to avoid pitfalls.
<h1>The scenario</h1>
I will use 3 different connections to connect to the same MySQL 8.0.27 instance. The only relevant setting I have modified is the Innodb_timeout that I set to 50 seconds. 
<ul>
<li style="font-weight: 400;" aria-level="1">Session 1 will simulate a process that should activate the bonus feature for the selected customers.</li>
<li style="font-weight: 400;" aria-level="1">Session 2 is an independent process that reactivate the given list of customers</li>
<li style="font-weight: 400;" aria-level="1">Session 3 is used to collect lock information.</li>
</ul>
For this simple test I will use the customer table in the sakila schema modified as below:
<pre class="lang:mysql decode:true">CREATE TABLE `customer` (
`customer_id` smallint unsigned NOT NULL AUTO_INCREMENT,
`store_id` tinyint unsigned NOT NULL,
`first_name` varchar(45) NOT NULL,
`last_name` varchar(45) NOT NULL,
`email` varchar(50) DEFAULT NULL,
`address_id` smallint unsigned NOT NULL,
`active` tinyint(1) NOT NULL DEFAULT '1',
`create_date` datetime NOT NULL,
`last_update` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`bonus` int NOT NULL DEFAULT '0',
`activate_bonus` varchar(45) NOT NULL DEFAULT '0',
PRIMARY KEY (`customer_id`),
KEY `idx_fk_store_id` (`store_id`),
KEY `idx_fk_address_id` (`address_id`),
KEY `idx_last_name` (`last_name`),
KEY `idx_bonus` (`bonus`),
CONSTRAINT `fk_customer_address` FOREIGN KEY (`address_id`) REFERENCES `address` (`address_id`) ON DELETE RESTRICT ON UPDATE CASCADE,
CONSTRAINT `fk_customer_store` FOREIGN KEY (`store_id`) REFERENCES `store` (`store_id`) ON DELETE RESTRICT ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=600 DEFAULT CHARSET=utf8mb4 COLLATE=ut
</pre>
As you can see I have added the bonus and activate_bonus fields plus the idx_bonus index.
To be able to trace the locks these are the threads ids by session:
<pre class="lang:sh decode:true">session 1 17439
session 2 17430
session 3 17443
</pre>
To collect the lock information:
<pre class="lang:mysql decode:true">SELECT
index_name, lock_type, lock_mode, lock_data, thread_id
FROM
performance_schema.data_locks
WHERE
object_schema = 'sakila'
AND object_name = 'customer'
AND lock_type = 'RECORD'
AND thread_id IN (17439 , 17430)
ORDER BY index_name , lock_data DESC;
</pre>
ok , ready? Let us start!
<h1>The run…</h1>
While the following steps can be done in a more compressed way, I prefer to do it in a more verbose way, to make it more human readable.
First let us set the environment:
<pre class="lang:mysql decode:true">session1 >set transaction_isolation = 'REPEATABLE-READ';
Query OK, 0 rows affected (0.07 sec)
session1 >Start Transaction;
Query OK, 0 rows affected (0.07 sec)
</pre>
Then let see the list of the customers we will modify:
<pre class="lang:mysql decode:true">session1 >SELECT * FROM sakila.customer where bonus = 1 and active =1 order by last_name;
+-------------+----------+------------+-----------+-------+----------------+
| customer_id | store_id | first_name | last_name | bonus | activate_bonus |
+-------------+----------+------------+-----------+-------+----------------+
| 383 | 1 | MARTIN | BALES | 1 | 0 |
| 539 | 1 | MATHEW | BOLIN | 1 | 0 |
| 441 | 1 | MARIO | CHEATHAM | 1 | 0 |
| 482 | 1 | MAURICE | CRAWLEY | 1 | 0 |
| 293 | 2 | MAE | FLETCHER | 1 | 0 |
| 38 | 1 | MARTHA | GONZALEZ | 1 | 0 |
| 444 | 2 | MARCUS | HIDALGO | 1 | 0 |
| 252 | 2 | MATTIE | HOFFMAN | 1 | 0 |
| 256 | 2 | MABEL | HOLLAND | 1 | 0 |
| 226 | 2 | MAUREEN | LITTLE | 1 | 0 |
| 588 | 1 | MARION | OCAMPO | 1 | 0 |
| 499 | 2 | MARC | OUTLAW | 1 | 0 |
| 553 | 1 | MAX | PITT | 1 | 0 |
| 312 | 2 | MARK | RINEHART | 1 | 0 |
| 80 | 1 | MARILYN | ROSS | 1 | 0 |
| 583 | 1 | MARSHALL | THORN | 1 | 0 |
| 128 | 1 | MARJORIE | TUCKER | 1 | 0 |
| 44 | 1 | MARIE | TURNER | 1 | 0 |
| 267 | 1 | MARGIE | WADE | 1 | 0 |
| 240 | 1 | MARLENE | WELCH | 1 | 0 |
| 413 | 2 | MARVIN | YEE | 1 | 0 |
+-------------+----------+------------+-----------+-------+----------------+
21 rows in set (0.08 sec)
</pre>
As you can see we have 21 customers fitting our criteria. How much money is involved in this exercise?
<pre class="lang:mysql decode:true">session1 >SELECT
-> SUM(amount) income,
-> SUM(amount) * 0.90 income_with_bonus,
-> (SUM(amount) - (SUM(amount) * 0.90)) loss_because_bonus
-> FROM
-> sakila.customer AS c
-> JOIN
-> sakila.payment AS p ON c.customer_id = p.customer_id
-> where active = 1 and bonus =1 ;
+---------+-------------------+--------------------+
| income | income_with_bonus | loss_because_bonus |
+---------+-------------------+--------------------+
| 2416.23 | 2174.6070 | 241.6230 |
+---------+-------------------+--------------------+
</pre>
This exercise is going to cost me ~242 dollars. Keep this number in mind. What locks do I have at this point?
<pre class="lang:mysql decode:true">session3 >select index_name, lock_type, lock_mode,lock_data from performance_schema.data_locks where object_schema = 'sakila' and object_name = 'customer' and lock_type = 'RECORD' and
thread_id in (17439,17430) order by index_name, lock_data desc;
Empty set (0.00 sec)
</pre>
Answer is none. 
Meanwhile we have the other process that needs to reactivate the customers:
<pre class="lang:mysql decode:true">session2 >set transaction_isolation = 'REPEATABLE-READ';
Query OK, 0 rows affected (0.00 sec)
session2 >Start Transaction;
Query OK, 0 rows affected (0.00 sec)
session2 >SELECT * FROM sakila.customer where bonus = 1 and active =0 ;
+-------------+----------+------------+-----------+-------+----------------+
| customer_id | store_id | first_name | last_name | bonus | activate_bonus |
+-------------+----------+------------+-----------+-------+----------------+
| 1 | 1 | MARY | SMITH | 1 | 0 |
| 7 | 1 | MARIA | MILLER | 1 | 0 |
| 9 | 2 | MARGARET | MOORE | 1 | 0 |
| 178 | 2 | MARION | SNYDER | 1 | 0 |
| 236 | 1 | MARCIA | DEAN | 1 | 0 |
| 246 | 1 | MARIAN | MENDOZA | 1 | 0 |
| 254 | 2 | MAXINE | SILVA | 1 | 0 |
| 257 | 2 | MARSHA | DOUGLAS | 1 | 0 |
| 323 | 2 | MATTHEW | MAHAN | 1 | 0 |
| 408 | 1 | MANUEL | MURRELL | 1 | 0 |
+-------------+----------+------------+-----------+-------+----------------+
10 rows in set (0.00 sec)
</pre>
In this case the process needs to reactivate 10 users.
<pre class="lang:mysql decode:true">session2 >update sakila.customer set active = 1 where bonus = 1 and active =0 ;
Query OK, 10 rows affected (0.00 sec)
Rows matched: 10 Changed: 10 Warnings: 0
session2 >commit;
Query OK, 0 rows affected (0.00 sec)
</pre>
All good, right? But before going ahead let us double check, session 1:
<pre class="lang:mysql decode:true">session1 >SELECT * FROM sakila.customer where bonus = 1 and active =1 order by last_name;
+-------------+----------+------------+-----------+-------+----------------+
| customer_id | store_id | first_name | last_name | bonus | activate_bonus |
+-------------+----------+------------+-----------+-------+----------------+
| 383 | 1 | MARTIN | BALES | 1 | 0 |
| 539 | 1 | MATHEW | BOLIN | 1 | 0 |
| 441 | 1 | MARIO | CHEATHAM | 1 | 0 |
| 482 | 1 | MAURICE | CRAWLEY | 1 | 0 |
| 293 | 2 | MAE | FLETCHER | 1 | 0 |
| 38 | 1 | MARTHA | GONZALEZ | 1 | 0 |
| 444 | 2 | MARCUS | HIDALGO | 1 | 0 |
| 252 | 2 | MATTIE | HOFFMAN | 1 | 0 |
| 256 | 2 | MABEL | HOLLAND | 1 | 0 |
| 226 | 2 | MAUREEN | LITTLE | 1 | 0 |
| 588 | 1 | MARION | OCAMPO | 1 | 0 |
| 499 | 2 | MARC | OUTLAW | 1 | 0 |
| 553 | 1 | MAX | PITT | 1 | 0 |
| 312 | 2 | MARK | RINEHART | 1 | 0 |
| 80 | 1 | MARILYN | ROSS | 1 | 0 |
| 583 | 1 | MARSHALL | THORN | 1 | 0 |
| 128 | 1 | MARJORIE | TUCKER | 1 | 0 |
| 44 | 1 | MARIE | TURNER | 1 | 0 |
| 267 | 1 | MARGIE | WADE | 1 | 0 |
| 240 | 1 | MARLENE | WELCH | 1 | 0 |
| 413 | 2 | MARVIN | YEE | 1 | 0 |
+-------------+----------+------------+-----------+-------+----------------+
21 rows in set (0.08 sec)
</pre>
Perfect! My Repeatable Read still sees the same snapshot. Let me apply the changes:
<pre class="lang:mysql decode:true">session1 >update sakila.customer set activate_bonus=1 where bonus = 1 and active =1 ;
Query OK, 31 rows affected (0.06 sec)
Rows matched: 31 Changed: 31 Warnings: 0
</pre>
Wait, what? My list reports 21 entries, but I have modified 31! And if I check the cost:
<pre class="lang:mysql decode:true">session1 >SELECT
-> SUM(amount) income,
-> SUM(amount) * 0.90 income_with_bonus,
-> (SUM(amount) - (SUM(amount) * 0.90)) loss_because_bonus
-> FROM
-> sakila.customer AS c
-> JOIN
-> sakila.payment AS p ON c.customer_id = p.customer_id
-> where active = 1 and bonus =1 ;
+---------+-------------------+--------------------+
| income | income_with_bonus | loss_because_bonus |
+---------+-------------------+--------------------+
| 3754.01 | 3378.6090 | 375.4010 |
+---------+-------------------+--------------------+
</pre>
Well now the cost of this operation is 375 dollars not 242. In this case we are talking about peanuts, but guess what could be if we do something similar on thousands of users. 
Anyhow let us first:
<pre class="lang:mysql decode:true">session1 >rollback;
Query OK, 0 rows affected (0.08 sec)
</pre>
And cancel the operation.
So what happened, is this a bug?
No it is not! It is how repeatable reads work in MySQL. The snapshot is relevant to the read operation, But if another session is able to write given also the absence of lock at the moment of reads, the next update operation will touch any value in the table matching the conditions. 
As shown above this can be very dangerous. But only if you don’t do the right things in the code. 
<h2>How can I prevent this from happening?</h2>
When coding, hope for the best, plan for the worse, always! Especially when dealing with databases. That approach may save you from spending nights trying to fix the impossible. 
So how can this be prevented? You have two ways, both simple, but both with positive and negative consequences.
<ol>
<li style="font-weight: 400;" aria-level="1">Add this simple clausole to the select statement: for share</li>
<li style="font-weight: 400;" aria-level="1">Add the other simple clausole to the select statement: for update</li>
<li style="font-weight: 400;" aria-level="1">Use Read-committed</li>
</ol>
Solution 1 is easy and clean, no other change in the code. BUT create locks, and if your application is lock sensitive this may be an issue for you.
Solution 2 is also easy, but a bit more encapsulated and locking. 
On the other hand Solution 3 does not add locks, but requires modifications in the code and it still leaves some space for problems.
Let us see them in detail.
<h3>Solution 1</h3>
Let us repeat the same steps
<pre class="lang:mysql decode:true">session1 >set transaction_isolation = 'REPEATABLE-READ';
session1 >Start Transaction;
session1 >SELECT * FROM sakila.customer where bonus = 1 and active =1 order by last_name for share;
</pre>
If now we check the locks for brevity I have cut some entries and kept a few as a sample:
<pre class="lang:mysql decode:true">session3 >select index_name, lock_type, lock_mode,lock_data,thread_id from performance_schema.data_locks where
-> object_schema = 'sakila'
-> and object_name = 'customer'
-> and lock_type = 'RECORD'
-> and thread_id = 17439
-> order by index_name, lock_data desc;
+------------+-----------+---------------+------------------------+-----------+
| index_name | lock_type | lock_mode | lock_data | thread_id |
+------------+-----------+---------------+------------------------+-----------+
| idx_bonus | RECORD | S | supremum pseudo-record | 17439 |
| idx_bonus | RECORD | S | 1, 9 | 17439 |
<snip>
| idx_bonus | RECORD | S | 1, 128 | 17439 |
| idx_bonus | RECORD | S | 1, 1 | 17439 |
| PRIMARY | RECORD | S,REC_NOT_GAP | 9 | 17439 |
<snip>
| PRIMARY | RECORD | S,REC_NOT_GAP | 1 | 17439 |
+------------+-----------+---------------+------------------------+-----------+
63 rows in set (0.00 sec)
</pre>
This time we can see that the select has raised a few locks all in S (shared) mode. 
For brevity I am skipping to report exactly the same results as in the first exercise.
If we go ahead and try with session 2:
<pre class="lang:mysql decode:true">session2 >set transaction_isolation = 'READ-COMMITTED';
session2 >Start Transaction;
session2 >SELECT * FROM sakila.customer where bonus = 1 and active =0 ;
session2 >update sakila.customer set active = 1 where bonus = 1 and active =0 ;
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
</pre>
Here we go! Attempting to change the values in the customer table is now on hold waiting to acquire the lock. If the duration exceeds the Innodb_wait_timeout, then the execution is interrupted and the application must have a try-catch mechanism to retry the operation. This last one is a best practice that you should already have in place in your code. If not, well add it now!
At this point session 1 can proceed and complete the operations. After that and before the final commit, we will be able to observe:
<pre class="lang:mysql decode:true">+------------+-----------+---------------+------------------------+-----------+
| index_name | lock_type | lock_mode | lock_data | thread_id |
+------------+-----------+---------------+------------------------+-----------+
| idx_bonus | RECORD | S | supremum pseudo-record | 17439 |
| idx_bonus | RECORD | X | supremum pseudo-record | 17439 |
| idx_bonus | RECORD | S | 1, 9 | 17439 |
| idx_bonus | RECORD | X | 1, 9 | 17439 |
<snip>
| idx_bonus | RECORD | X | 1, 1 | 17439 |
| idx_bonus | RECORD | S | 1, 1 | 17439 |
| PRIMARY | RECORD | X,REC_NOT_GAP | 9 | 17439 |
| PRIMARY | RECORD | S,REC_NOT_GAP | 9 | 17439 |
<snip>|
| PRIMARY | RECORD | X,REC_NOT_GAP | 1 | 17439 |
| PRIMARY | RECORD | S,REC_NOT_GAP | 1 | 17439 |
+------------+-----------+---------------+------------------------+-----------+
126 rows in set (0.00 sec)
</pre>
As you can see we now have two different lock types, the Shared one from the select, and the exclusive lock (X) from the update.
<h3>Solution 2</h3>
In this case we just change the kind of lock we set on the select. In solution 1 we opt for a shared lock, which allows other sessions to eventually acquire another shared lock. Here we go for an exclusive lock that will put all the other operations on hold. 
<pre class="lang:mysql decode:true">session1 >set transaction_isolation = 'REPEATABLE-READ';
session1 >Start Transaction;
session1 >SELECT * FROM sakila.customer where bonus = 1 and active =1 order by last_name for UPDATE;
</pre>
If now we check the locks for brevity I have cut some entries and kept few as a sample:
<pre class="lang:mysql decode:true">session3 >select index_name, lock_type, lock_mode,lock_data,thread_id from performance_schema.data_locks where
-> object_schema = 'sakila'
-> and object_name = 'customer'
-> and lock_type = 'RECORD'
-> and thread_id = 17439
-> order by index_name, lock_data desc;
+------------+-----------+---------------+------------------------+-----------+
| index_name | lock_type | lock_mode | lock_data | thread_id |
+------------+-----------+---------------+------------------------+-----------+
| idx_bonus | RECORD | X | supremum pseudo-record | 17439 |
| idx_bonus | RECORD | X | 1, 9 | 17439 |
<snip>
| idx_bonus | RECORD | X | 1, 128 | 17439 |
| idx_bonus | RECORD | X | 1, 1 | 17439 |
| PRIMARY | RECORD | X,REC_NOT_GAP | 9 | 17439 |
| PRIMARY | RECORD | X,REC_NOT_GAP | 80 | 17439 |
<snip>
| PRIMARY | RECORD | X,REC_NOT_GAP | 128 | 17439 |
| PRIMARY | RECORD | X,REC_NOT_GAP | 1 | 17439 |
+------------+-----------+---------------+------------------------+-----------+
63 rows in set (0.09 sec)
</pre>
In this case the kind of lock is X as exclusive, so the other operations when requesting a lock must wait for this transaction to complete.
<pre class="lang:mysql decode:true">session2 >#set transaction_isolation = 'READ-COMMITTED';
session2 >
session2 >Start Transaction;
Query OK, 0 rows affected (0.06 sec)
session2 >SELECT * FROM sakila.customer where bonus = 1 and active =0 for update;
</pre>
And it will wait for N time where N is either the Innodb_lock_wait_timeout or the commit time from season 1.
Note the lock request on hold:
<pre class="lang:mysql decode:true crayon-selected">+------------+-----------+---------------+------------------------+-----------+
| index_name | lock_type | lock_mode | lock_data | thread_id |
+------------+-----------+---------------+------------------------+-----------+
| idx_bonus | RECORD | X | supremum pseudo-record | 17439 |
<snip>
| idx_bonus | RECORD | X | 1, 128 | 17439 |
| idx_bonus | RECORD | X | 1, 1 | 17430 |<--
| idx_bonus | RECORD | X | 1, 1 | 17439 |
| PRIMARY | RECORD | X,REC_NOT_GAP | 9 | 17439 |
<snip>
| PRIMARY | RECORD | X,REC_NOT_GAP | 1 | 17439 |
+------------+-----------+---------------+------------------------+-----------+
64 rows in set (0.09 sec)
</pre>
Session 2 is trying to lock exclusively the idx_bonus but cannot proceed.
Once session 1 goes ahead we have:
<pre class="lang:mysql decode:true">session1 >update sakila.customer set activate_bonus=1 where bonus = 1 and active =1 ;
Query OK, 0 rows affected (0.06 sec)
Rows matched: 21 Changed: 0 Warnings: 0
session1 >
session1 >SELECT * FROM sakila.customer where bonus = 1 and active =1 ;
+-------------+----------+------------+-----------+-------+----------------+
| customer_id | store_id | first_name | last_name | bonus | activate_bonus |
+-------------+----------+------------+-----------+-------+----------------+
| 38 | 1 | MARTHA | GONZALEZ | 1 | 1 |
| 44 | 1 | MARIE | TURNER | 1 | 1 |
| 80 | 1 | MARILYN | ROSS | 1 | 1 |
| 128 | 1 | MARJORIE | TUCKER | 1 | 1 |
| 226 | 2 | MAUREEN | LITTLE | 1 | 1 |
| 240 | 1 | MARLENE | WELCH | 1 | 1 |
| 252 | 2 | MATTIE | HOFFMAN | 1 | 1 |
| 256 | 2 | MABEL | HOLLAND | 1 | 1 |
| 267 | 1 | MARGIE | WADE | 1 | 1 |
| 293 | 2 | MAE | FLETCHER | 1 | 1 |
| 312 | 2 | MARK | RINEHART | 1 | 1 |
| 383 | 1 | MARTIN | BALES | 1 | 1 |
| 413 | 2 | MARVIN | YEE | 1 | 1 |
| 441 | 1 | MARIO | CHEATHAM | 1 | 1 |
| 444 | 2 | MARCUS | HIDALGO | 1 | 1 |
| 482 | 1 | MAURICE | CRAWLEY | 1 | 1 |
| 499 | 2 | MARC | OUTLAW | 1 | 1 |
| 539 | 1 | MATHEW | BOLIN | 1 | 1 |
| 553 | 1 | MAX | PITT | 1 | 1 |
| 583 | 1 | MARSHALL | THORN | 1 | 1 |
| 588 | 1 | MARION | OCAMPO | 1 | 1 |
+-------------+----------+------------+-----------+-------+----------------+
21 rows in set (0.08 sec)
session1 >SELECT SUM(amount) income, SUM(amount) * 0.90 income_with_bonus, (SUM(amount) - (SUM(amount) * 0.90)) loss_because_bonus FROM sakila.customer AS c JOIN sakila.payment AS p ON c.customer_id = p.customer_id where active = 1 and bonus =1;
+---------+-------------------+--------------------+
| income | income_with_bonus | loss_because_bonus |
+---------+-------------------+--------------------+
| 2416.23 | 2174.6070 | 241.6230 |
+---------+-------------------+--------------------+
1 row in set (0.06 sec)
</pre>
Now my update matches the expectations and the other operations are on hold.
After session 1 commits.
<pre class="lang:mysql decode:true">session2 >SELECT * FROM sakila.customer where bonus = 1 and active =0 for update;
+-------------+----------+------------+-----------+-------+----------------+
| customer_id | store_id | first_name | last_name | bonus | activate_bonus |
+-------------+----------+------------+-----------+-------+----------------+
| 1 | 1 | MARY | SMITH | 1 | 1 |
| 7 | 1 | MARIA | MILLER | 1 | 1 |
| 9 | 2 | MARGARET | MOORE | 1 | 1 |
| 178 | 2 | MARION | SNYDER | 1 | 1 |
| 236 | 1 | MARCIA | DEAN | 1 | 1 |
| 246 | 1 | MARIAN | MENDOZA | 1 | 1 |
| 254 | 2 | MAXINE | SILVA | 1 | 1 |
| 257 | 2 | MARSHA | DOUGLAS | 1 | 1 |
| 323 | 2 | MATTHEW | MAHAN | 1 | 1 |
| 408 | 1 | MANUEL | MURRELL | 1 | 1 |
+-------------+----------+------------+-----------+-------+----------------+
10 rows in set (52.72 sec) <-- Note the time!
session2 >update sakila.customer set active = 1 where bonus = 1 and active =0 ;
Query OK, 10 rows affected (0.06 sec)
Rows matched: 10 Changed: 10 Warnings: 0
</pre>
Session 2 is able to complete BUT it was on hold for 52 seconds waiting for session 1. As said this solution is a good one if you can afford locks and waiting time.
<h3>Solution 3</h3>
In this case we will use a different isolation model that will allow session 1 to see what session 2 has modified.
<pre class="lang:mysql decode:true">session1 >set transaction_isolation = 'READ-COMMITTED';
session1 >Start Transaction;
session1 >SELECT * FROM sakila.customer where bonus = 1 and active =1 order by last_name;
+-------------+----------+------------+-----------+-------+----------------+
| customer_id | store_id | first_name | last_name | bonus | activate_bonus |
+-------------+----------+------------+-----------+-------+----------------+
| 383 | 1 | MARTIN | BALES | 1 | 0 |
| 539 | 1 | MATHEW | BOLIN | 1 | 0 |
| 441 | 1 | MARIO | CHEATHAM | 1 | 0 |
| 482 | 1 | MAURICE | CRAWLEY | 1 | 0 |
| 293 | 2 | MAE | FLETCHER | 1 | 0 |
| 38 | 1 | MARTHA | GONZALEZ | 1 | 0 |
| 444 | 2 | MARCUS | HIDALGO | 1 | 0 |
| 252 | 2 | MATTIE | HOFFMAN | 1 | 0 |
| 256 | 2 | MABEL | HOLLAND | 1 | 0 |
| 226 | 2 | MAUREEN | LITTLE | 1 | 0 |
| 588 | 1 | MARION | OCAMPO | 1 | 0 |
| 499 | 2 | MARC | OUTLAW | 1 | 0 |
| 553 | 1 | MAX | PITT | 1 | 0 |
| 312 | 2 | MARK | RINEHART | 1 | 0 |
| 80 | 1 | MARILYN | ROSS | 1 | 0 |
| 583 | 1 | MARSHALL | THORN | 1 | 0 |
| 128 | 1 | MARJORIE | TUCKER | 1 | 0 |
| 44 | 1 | MARIE | TURNER | 1 | 0 |
| 267 | 1 | MARGIE | WADE | 1 | 0 |
| 240 | 1 | MARLENE | WELCH | 1 | 0 |
| 413 | 2 | MARVIN | YEE | 1 | 0 |
+-------------+----------+------------+-----------+-------+----------------+
21 rows in set (0.08 sec)
</pre>
Same result as before. Now let us execute commands in session2:
<pre class="lang:mysql decode:true">session2 >set transaction_isolation = 'READ-COMMITTED';
Query OK, 0 rows affected (0.00 sec)
session2 >
session2 >Start Transaction;
Query OK, 0 rows affected (0.00 sec)
session2 >SELECT * FROM sakila.customer where bonus = 1 and active =0 ;
+-------------+----------+------------+-----------+-------+----------------+
| customer_id | store_id | first_name | last_name | bonus | activate_bonus |
+-------------+----------+------------+-----------+-------+----------------+
| 1 | 1 | MARY | SMITH | 1 | 0 |
| 7 | 1 | MARIA | MILLER | 1 | 0 |
| 9 | 2 | MARGARET | MOORE | 1 | 0 |
| 178 | 2 | MARION | SNYDER | 1 | 0 |
| 236 | 1 | MARCIA | DEAN | 1 | 0 |
| 246 | 1 | MARIAN | MENDOZA | 1 | 0 |
| 254 | 2 | MAXINE | SILVA | 1 | 0 |
| 257 | 2 | MARSHA | DOUGLAS | 1 | 0 |
| 323 | 2 | MATTHEW | MAHAN | 1 | 0 |
| 408 | 1 | MANUEL | MURRELL | 1 | 0 |
+-------------+----------+------------+-----------+-------+----------------+
10 rows in set (0.00 sec)
session2 >update sakila.customer set active = 1 where bonus = 1 and active =0 ;
Query OK, 10 rows affected (0.00 sec)
Rows matched: 10 Changed: 10 Warnings: 0
session2 >commit;
Query OK, 0 rows affected (0.00 sec)
</pre>
All seems the same, but if we check again in session 1:
<pre class="lang:mysql decode:true">session1 >SELECT * FROM sakila.customer where bonus = 1 and active =1 order by last_name;
+-------------+----------+------------+-----------+------+----------------+
| customer_id | store_id | first_name | last_name |bonus | activate_bonus |
+-------------+----------+------------+-----------+------+----------------+
| 383 | 1 | MARTIN | BALES | 1 | 1 |
| 539 | 1 | MATHEW | BOLIN | 1 | 1 |
| 441 | 1 | MARIO | CHEATHAM | 1 | 1 |
| 482 | 1 | MAURICE | CRAWLEY | 1 | 1 |
| 236 | 1 | MARCIA | DEAN | 1 | 0 |
| 257 | 2 | MARSHA | DOUGLAS | 1 | 0 |
| 293 | 2 | MAE | FLETCHER | 1 | 1 |
| 38 | 1 | MARTHA | GONZALEZ | 1 | 1 |
| 444 | 2 | MARCUS | HIDALGO | 1 | 1 |
| 252 | 2 | MATTIE | HOFFMAN | 1 | 1 |
| 256 | 2 | MABEL | HOLLAND | 1 | 1 |
| 226 | 2 | MAUREEN | LITTLE | 1 | 1 |
| 323 | 2 | MATTHEW | MAHAN | 1 | 0 |
| 246 | 1 | MARIAN | MENDOZA | 1 | 0 |
| 7 | 1 | MARIA | MILLER | 1 | 0 |
| 9 | 2 | MARGARET | MOORE | 1 | 0 |
| 408 | 1 | MANUEL | MURRELL | 1 | 0 |
| 588 | 1 | MARION | OCAMPO | 1 | 1 |
| 499 | 2 | MARC | OUTLAW | 1 | 1 |
| 553 | 1 | MAX | PITT | 1 | 1 |
| 312 | 2 | MARK | RINEHART | 1 | 1 |
| 80 | 1 | MARILYN | ROSS | 1 | 1 |
| 254 | 2 | MAXINE | SILVA | 1 | 0 |
| 1 | 1 | MARY | SMITH | 1 | 0 |
| 178 | 2 | MARION | SNYDER | 1 | 0 |
| 583 | 1 | MARSHALL | THORN | 1 | 1 |
| 128 | 1 | MARJORIE | TUCKER | 1 | 1 |
| 44 | 1 | MARIE | TURNER | 1 | 1 |
| 267 | 1 | MARGIE | WADE | 1 | 1 |
| 240 | 1 | MARLENE | WELCH | 1 | 1 |
| 413 | 2 | MARVIN | YEE | 1 | 1 |
+-------------+----------+------------+-----------+------+----------------+
31 rows in set (0.09 sec)
</pre>
We now can see all the 31 customers and also the value calculation:
<pre class="lang:mysql decode:true">+---------+-------------------+--------------------+
| income | income_with_bonus | loss_because_bonus |
+---------+-------------------+--------------------+
| 3754.01 | 3378.6090 | 375.4010 |
+---------+-------------------+--------------------+
</pre>
Is reporting the right values.
At this point we can program some logic in the process to check the possible tolerance, and have the process either complete performing the update operations or to stop the process and exit.
This as mentioned requires additional coding and more logic.
To be fully honest, this solution still leaves space to some possible interference, but at least allow the application to be informed about what is happening. Still it should be used only if you cannot afford to have a higher level of locking, but this is another story/article.
<h1>Conclusions</h1>
When writing applications that interact with a RDBMS, you must be very careful in what you do and HOW you do it. While using data facilitation layers like Object Relational Mapping (ORM), seems to make your life easier, in reality you may lose control on a few crucial aspects of the application’s interaction. So be very careful when opting for how to access your data.
About the case reported above we can summarize a few pitfalls:
<ol>
<li style="font-weight: 400;" aria-level="1">First and most important, never ever have processes that may interfere with the one running at the same time. Be very careful when you design them and even more careful when planning their executions.  </li>
<li style="font-weight: 400;" aria-level="1">Use options such as for share or for update in your select statements. Use them carefully to avoid unuseful locks, but use them. </li>
<li style="font-weight: 400;" aria-level="1">In case you have a long process that requires modifying data, and you need to be able to check if other processes have altered the status of the data, but at the same time, cannot have a long wait time for locks. Then use a mix, setting READ-COMMITTED as isolation, to allow your application to check, but also add things like for share or for update in the select statement, immediately before the DML and the commit. That will allow you to prevent writes while you are finalizing, and also to significantly reduce the locking time. </li>
<li style="font-weight: 400;" aria-level="1">Keep in mind that long running processes and long running transactions can be the source of a lot of pain, especially when using REPEATABLE-READ. Try whenever you can to split the operations in small chunks. </li>
</ol>
Finally when developing, remember that DBAs are friends, and are there to help you to do the things at the best. It may seem they are giving you a hard time, but that is because their point of view is different, focusing on data consistency, availability and durability. But they can help you to save a lot of time after the code is released, especially when you try to identify why something has gone wrong. 
So involve them soon in the design process, use them and let them be part of the process. 
<h1>References</h1>
<a href="https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html">https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html</a>
<a href="https://dev.mysql.com/doc/refman/8.0/en/innodb-consistent-read.html">https://dev.mysql.com/doc/refman/8.0/en/innodb-consistent-read.html</a>
<a href="https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-model.html">https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-model.html</a>]]></description>
<category>MySQL</category>
<pubDate>Tue, 09 Nov 2021 10:11:31 +0000</pubDate>
</item>
<item>
<title>MySQL on Kubernetes demystified</title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/243-mysql-on-kubernetes-demystified</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/243-mysql-on-kubernetes-demystified</guid>
<description><![CDATA[<h1>Why<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/stop-befool.jpg" alt="stop befool" style="float: right;" /></h1>
Marco, why did you write this long article?
Yes, it is long, and I know most of the people will not read it in full, but my hope is that at least someone will, and I count on them to make the wave of sanity. 
Why I wrote it is simple. We write articles to share something we discover, or to share new approaches or as in this case to try to demystify and put in the right perspective the “last shining thing” that will save the world.  
The “last shining thing” is the use of containers for relational databases management systems (RDBMS) and all the attached solutions like Kubernetes or similar. 
Why is this a problem? The use of containers for RDBMS is not really a problem per se, but it had become a problem because it was not correctly contextualized and even more important, the model that should be used to properly design the solutions, was not reviewed and modified in respect to the classic one.
One example for all is this image:
<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/db_on_kube_1.png" alt="db on kube 1" width="671" height="386" />
Source (<a href="https://www.cockroachlabs.com/blog/kubernetes-trends/"></a><a href="https://www.cockroachlabs.com/blog/kubernetes-trends/">https://www.cockroachlabs.com/blog/kubernetes-trends/</a>) 
In this report we find the use of the term Database multiple times, and reference how easy it is to adopt and scale a Database on Kubernetes. But the problem is … what Database? Not all the Databases are the same, not all can be adopted so easily, some use more restrictive design to be real RDBMS, others less. 
Some are designed to scale horizontally, others no. The other missed part is that, to get something you need to give something. If I design a system to scale horizontally, I probably have to pay a price on something else. It could be the lack of referential integrity or the Isolation level, it doesn’t matter, but you do not get anything for free. 
Generalizing without mentioning the difference is misleading.
What happens instead is that we have a lot of people, presenting solutions that are so generic that are unusable. The most hilarious thing is that they present that as an inevitable evolution, the step into the future that will solve every problem. But when doing this, they do not clarify what is the “Database” in use, what you get, what you lose. This may lead to misunderstanding and future frustration. 
For instance we constantly see presentations that illustrate how easy it will be to manage hundreds, thousands of pods containing RDBMS, without even understanding the concept of RDBMS, the data they may host and its dimension. 
The more we go ahead the more dangerous this disinformation is becoming, because we can say that on 100 companies currently using RDBMS to manage their crucial data, only 10 have a good team of experts that understand how to really use containers, and probably only 1 or 2 of them have real data experts that are able to redesign the data layer to be correctly utilized in containers or able to see to what proper Database to migrate to achieve the expected results.
Another common expectation is to move to container/kubernetes to reduce the costs, no matter if related to the iron (physical servers) or of the management of them. 
Indeed you can optimise that part, but you need to understand the limitations of your future solution. You must take into account that you may not have the same level of service in a way or another. Honestly I haven't seen that addressed in current blogging or reports. 
This is why this article. What I want to do is to open a door to a discussion, that will lead us to review the original model used to design data inside RDBMS, that will allow anyone to safely approach a different model to use for modern database design.
<h1>When</h1>
(if you know how we get here you can skip this)
To fully understand what we are talking about we need to do a jump back in time, because without history we are nothing, without memory we are lost. Just note that I am going to touch on things at a very high level and for what we are concerned, otherwise this will be a book not an article.
A long time ago in a world far far away where the internet was not, there was: client server approach.
<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/ClientServerArchitecture1.png" alt="ClientServerArchitecture1" width="340" height="223" /> 
At that time we had many clients connecting to a server to provide access to whatever. Most of the clients were performing information rendering and local data validation, then they sent the information back to the application server who processed them and store… where? Well there was a high variety of containers. Some were just proprietary files with custom format, others connected to an external database.  What is relevant here is that we had a very limited number of clients (often less than 1k) and at the end the amount of data in transit and then store was very small (if a database had more than 1GB data in total  it was considered a monster).
 
Then came the internet … and many things started to change. 
<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/clientserver-web.gif" alt="clientserver web" width="431" height="205" />
We still have a client on our PC, but now is called a web Browser, the information rendering is now standardize using SGML standards (HTML tagging), the server is not anymore a simple Application Server, but a web server who is connected to one or many application(s) using common gateway interface (CGI). Each application was then handling the data in/out independently, some using databases some not. 
This new approach add to the previous model many challenges like:
<ul>
<li aria-level="1">Anyone from anywhere can access the Web Server and any application hosted</li>
<li aria-level="1">The number of requests and clients connected jump from well predictable numbers to something impossible to define.</li>
<li aria-level="1">The connection to a web server follows a different approach based on a request issued by a browser and a set of information that will be sent as an answer from the server. (later we will have more interactive/active protocols but let us stay high ok?) </li>
<li aria-level="1">Many application were duplicating the same functions but with different approach</li>
<li aria-level="1">Data received and sent needs to be consistent not only locally on the server, but also between requests. </li>
</ul>
So on top of the problem generated by the number of possible clients connected and the amount of requests per second a web server was in need to process, the initial model was wasting resources in development and when running. There was the need to optimise the interactions between applications and their functions, to cover that the Service Oriented Architecture (SOA) model was largely adopted.
<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/1920px-SOA_Metamodelsvg.png" alt="1920px SOA Metamodelsvg" width="500" height="413" />
The data problem was partially solved identifying RDBMS as the best tool to use to guarantee the level of data consistency and at the same time it was possible to organize data in containers (tables) with validated interactions (Foreign keys and constraints) solving the online transaction processing (OLTP) problem. While for business analysis and reporting the Online analytical processing (OLAP) model was chosen. The OLAP model is based on facts and dimensions, which are elements of the data hypercube which is the base model that is utilized to first transform and then access the data.
During this time, the data as volume starts to increase significantly, most of the systems, when well designed, start to implement the concept of archiving, partitioning, and sharding. 
We are around 2005-2008.
 
Then social media and online shopping explode... 
Well we know that, right? We are still in this huge big bang of nonsense. What changed was not only that now anyone can access whatever is on a web server as certified information, but we can chat with anyone, we can buy almost anything, we can sell anything, we can make photos and share them, videos and ask anyone from anywhere to comment and we can say whatever and that crap will remain forever around. 
And guess what? To do that you need a very, very, very powerful scalable platform and it should be resilient, but for real not as a joke. When we look for something today, we open our phones, we expect to have that now and no matter what. If we do not get it, we may look to a different place, like another shop, or another chat system or whatever.  To achieve this you do not only need to have some specialized bloks, but to split the load in many small parts, then be able to have redundancy for each one of them, and be able to have any functional block able to connect to another functional block. Moreover, if one of the blocks goes down I must be able to retry the operation using another functional block of the same kind, and be sure I will have the same results.
To achieve this the SOA concept was evolved into the Microservice concept. But microservice also brings new challenges. One for all is how to manage these thousands of different functional bloks, and how to efficiently deploy/remove/modify them. 
Interestingly the term DevOps also started to get more relevance, and automation became a crucial part of ANY environment. 
In short the modern application architectures moves from Monolithic or SOA to microservice as shown here:
<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/microservice-draft-3-03-db.png" alt="microservice draft 3 03 db" width="500" height="263" />
Of course reality sees each microservice block be redundant and scalable. Given that for each microservice kind we can have 1 to infinite number of instances. 
It is in this context that the use of Kubernetes comes in help. Kubernetes is designed to help manage the resources as microservices, each element (pod) is seen in function of the service provided, the service will scale up and down  adding/removing the pod, keeping the service resilient. 
Expecting to deal with a structure like this without proper tooling/automation is just not possible:
<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/amazon-death-star.png" alt="amazon death star" width="500" height="332" />
This picture represent the Amazon microservice infrastructure in 2008 <a href="https://twitter.com/Werner/status/741673514567143424">https://twitter.com/Werner/status/741673514567143424</a>
<h2>In conclusion</h2>
In short, we can say that the complex topologies we have to deal with today are the results of a needed evolution from monolithic application approach to a distributed one in services/microservices. The proliferation of which had made absolutely necessary the creation and usage of a tool to facilitate the management, reducing complexity, time to perform the operations and costs. 
<h1>Cool eh? What about the database? </h1>
The first thing to say is that RDBMS as Oracle, Postgres and MySQL did not evolve in the same direction. They remain centralized structures. We have clusters but mostly to answer to High Availability needs and not scaling. 
This is the part of the story, the current evangelists of the “last shining thing” are not telling you. There is no coverage for the RDBMS like Oracle, Postgres or MySQL in the microservice approach. None, zero, nich, nada!
To bypass that problem, many microservices developers implemented complex mechanisms of caching or use noSQL approaches. Using caches to speedup the processing is absolutely ok, trying to use them instead of an ACID solution, is not. Same as the use of noSQL, if it makes sense because the service requires then ok otherwise no.
But what is the problem in seeing a RDBMS as a microservice? Let us start with a positive thing. If you design it as a microservice and reset the expectations, then it is ok.
<h2>What benefits (pros)? </h2>
The first thought is why I should move to kubernetes/containers. What am I expecting to get more that I do not already have?
Well let see the few basic points a microservice solution should help me with in case of application:
<ul>
<li aria-level="1">Modularity: 
<ul>
<li aria-level="2">This makes the application easier to understand, develop, test, and become more resilient to architecture erosion.</li>
</ul>
</li>
<li aria-level="1">Scalability: 
<ul>
<li aria-level="2">Since microservices are implemented and deployed independently of each other, i.e. they run within independent processes, they can be monitored and scaled independently.</li>
</ul>
</li>
<li aria-level="1">Better Fault Isolation for More Resilient Applications.
<ul>
<li aria-level="2">With a microservices architecture, the failure of one service is less likely to negatively impact other parts of the application because each microservice runs autonomously from the others. </li>
<li aria-level="2">Nevertheless, large distributed microservices architectures tend to have many dependencies, so developers need to protect the application from a dependency failure related shut down.</li>
</ul>
</li>
<li aria-level="1">Integration of heterogeneous and legacy systems: 
<ul>
<li aria-level="2">microservices is considered as a viable means for modernizing existing monolithic software applications.</li>
<li aria-level="2">There are experience reports of several companies who have successfully replaced (parts of) their existing software with microservices, or are in the process of doing so.</li>
<li aria-level="2">The process for Software modernization of legacy applications is done using an incremental approach.</li>
</ul>
</li>
<li aria-level="1">Faster Time to Market and “Future-Proofing” 
<ul>
<li aria-level="2">The pluggability of a microservices application architecture allows for easier and faster application development and upgrades. Developers can quickly build or change a microservice, then plug it into the architecture with less risk of coding conflicts and service outages. Moreover, due to the independence of each microservice, teams don’t have to worry about coding conflicts, and they don’t have to wait for slower-moving projects before launching their part of the application.</li>
<li aria-level="2">Microservices also support the CI/CD/CD (Continuous Integration, Continuous Delivery, Continuous Deployment) development philosophy. This means that you quickly deploy the core microservices of an application – as a minimum viable product (MVP) – and continuously update the product over time as you complete additional microservices. It also means that you can respond to security threats and add new technologies as they appear. The result for the user is a “future proof” product that’s constantly evolving with the times.</li>
</ul>
</li>
</ul>
Of course this is for applications let us try to map similar things for a database if it makes any sense. 
<ul>
<li aria-level="1">Modularity. We can have two different level of modularity:
<ul>
<li aria-level="2">By service. When using by service we need to have data modules that serve the microservices who use that segment of data. IE login needs data about user login only, not information about customer orders.</li>
<li aria-level="2">By context. When by context, we need to group as much as possible the data relevant to a specific topic. In a multitenant database we will need to separate the data such that all microservices will be able to scale accessing the data relevant to only one tenant, not all, then filtering. IE in our example above, we will have the whole schema in our service but only for one single tenant  </li>
</ul>
</li>
<li aria-level="1">Scalability. To gain infinite scalability as we do for applications, we should be able to:
<ul>
<li aria-level="2">Either the database service is able to perform non impacting distributed locking. What I mean here is that independently by the type of modularity used, if we need to be able to scale the number of requests/sec duplicating the service who answer the requests, we need to be able to avoid two different pods to overwrite eachother without or with minimal impact on the performance. As for today in the MySQL area we have two solutions that could cover that, PXC with galera and MySQL/PS with group replication. But both solutions have performance impact when using multiple writers. Not only, in these solutions data is not distributed by partitions, but duplicated in full for each pod. This means that unless you have a very good archiving strategy your space utilization will always be S x N (where s is the total space in one pod and N the number of pods). Because of this, if we choose this approach, we need to keep our dataset to the minimum to reduce any locking impact and data utilization.    </li>
<li aria-level="2">Or sharding the data by pods, which means we need to have a controller able to distribute the data. Of course we need to have High Availability so we need to consider at least a copy of each node or distributed partitioning. As far as I know MySQL or Postgres don’t have this functionality, Mongo not sure.</li>
</ul>
</li>
</ul>
In the end, a RDBMS microservice will not be able to scale in the same way an application microservice does. 
<ul>
<li aria-level="1">Better Fault Isolation for More Resilient Applications. Well it comes without saying. If the model used is by context, the fact we will isolate the data service in smaller entities will help the resilience of the whole system.  But the relationship that exists inside a schema can make the by service model more fragile, given that when 1 segment is down it will not be possible to validate the reference .   </li>
<li aria-level="1">Integration of heterogeneous and legacy systems. To be honest I can see this is possible only if modularity by service is used. Otherwise we will still have to deal with the whole structure and multiple services. In the last case I don’t see any benefit and we will have to perform operations like modifying a table definition, that can be as fast as renaming a field, or very impactful.</li>
<li aria-level="1">Faster Time to Market and “Future-Proofing” Indeed the divide and conquer approach especially if using the by service model will help in making each service more flexible to changes, but if by context is used, then we may have a negative effect because an application microservice may read data from N data microservices. If we change one data microservice (ie adding an index to a table), this change must be applied to all of the N microservice by context or the behaviour of the microservices will be different from the one with the change.</li>
<li aria-level="1">Distributed development. Also in this case we need to consider it easier to achieve if modularity by service is used. Otherwise I don’t really see how it will be possible to perform any of the mentioned points that are valid for applications. </li>
</ul>
<h2>What concerns (cons)?</h2>
Now let us discuss the possible concerns.
<ul>
<li aria-level="1">Services form information barriers. Well it comes without saying. </li>
<li aria-level="1">Inter-service calls over a network have a higher cost in terms of network latency and message processing time than in-process calls within a monolithic service process. Just think about some standard maintenance work that needs to be done to consolidate the data from each service to a single location for business intelligence purposes. And keep in mind that all must be encrypted for security reasons.  </li>
<li aria-level="1">Testing and deployment are more complicated. While acting on a single module will be easier, the deployment of the full solution and its testing will become extremely complicated.  </li>
<li aria-level="1">Viewing the size of services as the primary structuring mechanism can lead to too many services when the alternative of internal modularization may lead to a simpler design. This is true when using the by service modularity, we may end up having too many fragments of data, with serious difficulties in consolidating it. And of course, we still need to know and keep an eye on what is happening on the rest of the schema used by other services.</li>
<li aria-level="1">Two-phased commits are regarded as an anti-pattern in microservices-based architectures as this results in a tighter coupling of all the participants within the transaction. However, lack of this technology causes awkward dances which have to be implemented by all the transaction participants in order to maintain data consistency.</li>
<li aria-level="1">Development and support of many services is more challenging if they are built with different tools and technologies. It is not uncommon to have complex applications use different data sources, like RDBMS and a document store. While the use of microservice can help in simplifying how the data is collected and handled from these different sources, the additional fragmentation leads to a significant increase of the complexity to handle the platform. To be honest, solutions as Kubernetes can help a lot here, but still an issue.  </li>
<li aria-level="1">The protocol typically used with microservices (HTTP) was designed for public-facing services, and as such is unsuitable for working internal microservices that often must be impeccably reliable. This issue applies to databases in a different way. Because we will have 2 different aspects. The first one is the application microservice connecting and reading/writing to the data microservice. These operations must use the proprietary client protocol of the RDBMS. Then the second part is when the application microservice is sharing/serving the data to others. Say I am posting a payment and a network glitch happens between application microservices. I must be able to:
<ul>
<li aria-level="2">Retry the data sharing</li>
<li aria-level="2">Rollback the operation on the database if failing</li>
</ul>
</li>
</ul>
The HTTP protocol is not the best/most performant way to that given the post/get approach used.  
<h2>Let me give an example to clarify this. </h2>
We are an organization who does multiple activities. We have shops all over the globe and a huge segment of my sales is online. We also organize travels and host web-stores. To deal with the traffic and load I have moved my large part of the application architecture to microservice. To provide better services to my customers I also have to differentiate my catalog by geographic regions, splitting my market into North America, Europe, Middle-East, Asia and pacific. I have data centers in each geo-zone. Each geo-zone also hosts a local data repository that is focused on the specifics of the area. Finally my sales are consolidated in two steps, first locally by geo-area, then centrally in the North America HQ. We have hundreds of database services for each segment of business, and thousands of pods of the application.
One of mine online shop that use microservices at high level looks like this for each geo-zone:
<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/db-as-service1.png" alt="db as service1" width="349" height="221" />
Where Service-1 can be the login of a user and its authentication, Service-2 the show/retrieval of the items in special-sales, Service-3 dealing with existing order information … and so on. 
So I have specialized microservices that may serve multiple applications, like sign on, but each microservice at the moment needs to retrieve and send data from/to the same repository. 
Each database contains data relevant to a specific business activity. For instance our main store data is separate from the travel data and so on. Because this the “My DB” above just represents one of many databases we have. The detailed design of it may resemble this:
<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/schema-db.gif" alt="schema db" width="693" height="870" />
All credits for this diagram go to Barry Williams
<h2>Let us play the game, what do I need to do to convert this to a microservice?</h2>
Let us now try to apply what we have discussed so far about the expectations (pros/cons) and see if we will be able to make this fit. Given I am a MySQL expert I am going to cover the next step in the MySQL context.
<h3>Case one Modular by context</h3>
The first step is to choose the kind of modularity we want/can use. Let us start with the one which requires less changes, the module by context. 
When using that approach our schema remains the same, what we do is in case of multi-tenant to make it single-tenant, then we need to see how to cover the other aspects. 
 
The next step is scalability, and here we start to have problems. As already mentioned above there is no way to achieve the same level of scalability we have with the application layer. So what can we really do? 
As mentioned in the MySQL ecosystem we have two major solutions, one is Percona XtraDB Cluster (PXC) using galera virtually synchronous replication, the other is MySQL or Percona Server with group replication (GR). We have another player here which may actually fit much better, but at the time of writing there is no container solution for it. I am talking about the MySQL NDB cluster.
Anyhow both PXC and GR allow to have Single primary (one writer) or multiple Primary (multiple writers). But the use of multi Primary is discouraged because of the loss of performance caused by the write conflict resolution. What this means is that at each commit the node receiving the writes must certify the commit against the whole cluster. The more nodes write at the same time, the more chances of conflicts will rise. Each conflict will end up with a conflict resolution and a possible rollback of the operation. All these instead adding scalability is actually impacting it and significantly reducing the performance.
On the other hand, reads can easily scale using multiple nodes. But keep in mind that the two solutions cannot really scale in number of nodes to the infinite. GR has a hard limit of nine nodes, while PXC has no hard limit, but more nodes, more latency at each write commit and more cost in maintaining the cluster, in short when you reach a cluster with 7 nodes you already start to experience significant slow down and latency.    
So what level of scalability do we have here? The answer is easy, none! 
Yes none, as it is you can have and will have only 1 node (pod) writing at a given time, so your level of scalability in writing is the capacity of 1 node, period. 
 
Unless you adopt sharding. In that case you may have multiple microservice each covering one shard with a fixed number of nodes (pods). So the scaling is not internal to the microservice but in the duplication of it. 
Application at this point has three ways to use this approach:
<ul>
<li aria-level="1">Be shard aware and have a list of microservice to use when in the need (a shard catalog)</li>
<li aria-level="1">Be shard aware but refer to a generalized catalog service to access the data.</li>
<li aria-level="1">Not shard aware but pass information to the generalized catalog service to identify the shard.</li>
</ul>
 The first option is not really optimal because anytime we add a shard we need to “inform/modify” the catalog in all relevant application microservice. 
The second and the third assume the presence of another service dedicated to resolving the sharding.  
I am not going into the details here but this can be achieved by adding a ProxySQL microservice. Such that the scenario will be:
<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/db-as-service-proxy-microservice-sharding.png" alt="db as service proxy microservice sharding" width="439" height="402" />
Where each data service shard is a Percona Distribution for MySQL Operator (PDMO).
Scalability is then guaranteed by adding a PDMO shard and just an entry in ProxySQL in the mysql servers and query rules. 
 
Better Fault Isolation for More Resilient Applications. Adopting the above model will allow us to be not only more resilient in case we migrate from a multi-tenant approach, but also in case we do not. No matter what if the application is correctly written if a shard will go down only that segment will be affected. 
 
Integration and changes will become more cumbersome given the scale of the services. This is in the model and will represent a risk in case of mistakes or improper data definition modifications. But it will be easier to roll back by shard than against the monolithic data service. 
Distributed development. There will not be any advantage here, as developer or DBA I still need to consider the whole data definition scenario and not just a small segment. So while a developer can focus only on one aspect (i.e. login), a DBA still needs to take care of the whole schema and the impact of the operations (like locking) on the single table (i.e. users) for that service. 
 
Dimension of the dataset. But here we have another important concern, at least for now. To be so agile each single service (or microservice) needs to be able to be rebuilt quickly. This means that when a pod inside a data microservice crashes the cluster needs to rebuild it as fast as possible, or the QOS (quality of service) will be impacted. As for now both PXC and GR if in need to rebuild a node must actually copy the data over the new pod. This can be very fast in case of small datasets, or take days in case of very large. Given that when we plan to use a data microservice we also need to plan/design a very efficient archiving mechanism that allows us to stay in our microservices only the data that is really needed at the moment. All the data that can be considered not actual should be moved to an OLAP system and have dedicated service to retrieve it.
 
<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/db-as-service-etl.png" alt="db as service etl" width="594" height="632" />
 
<h3>Case two modularity by service. </h3>
What modularity by service means? It means I can split the data schema definition into small segments that correspond to the services. So my data segment will be functional to the use done by the application microservices. As an example I may have microservice dealing with: Login; shipments; invoicing … and so on. 
This means that in case of the invoice I should fragment the schema as follow:
<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_stop_crazy/schema-invoice-order.png" alt="schema invoice order" width="500" height="357" />
But these are the same tables I also need to serve shipments, so I must use the same data microservice, for both. This is ok … but … the question is can we? Can we split in small segments a schema that is built on relationships? Orders are linked to Customers and to products. How do we manage these referential integrity links if we start to split the schema?  
Again a simple answer, we cannot. If you are using a RDBMS the R is for relational and you cannot break the model and hope to have some data integrity. 
If you need that approach, then please use something else like noSQL Cassandra or similar. And then consolidate the data in the OLAP system. But do not use a RDBMS.
Given the above the by service approach, which was the one closer to give us what we need in the microsystem area, is not utilizable when talking of RDBMS. Period. 
<h2>I want to move to containers/Kubernetes to rationalize the resource utilizations and lower management cost</h2>
So far we have talked about moving to containers/kubernetes as an answer to application microservices, but there is another possible need that may lead you to see containers/kubernetes as “the solution”, the need to optimize resources.
 
It goes without saying that if you have hundreds or more small instances of a RDBMS, sparse n many different physical servers, or also some virtualization, the move may help you. The current operators like PDMO offer you a lot of automation, to perform not only installation of the main environment, but you will be able to easily automate and manage:
<ul>
<li aria-level="1">Installation</li>
<li aria-level="1">Geographic distribution</li>
<li aria-level="1">Backup/restore</li>
<li aria-level="1">Point in time recovery</li>
</ul>
All this with simple commands through kubernetes. 
 
As also indicated in my blog <a href="http://www.tusacentral.net/joomla/index.php/mysql-blogs/242-compare-percona-distribution-for-mysql-operator-vs-aws-aurora-and-standard-rds">here</a>, you will also be able to rationalize the resource utilization reducing the costs. 
 
So yes, of course this can be helpful and probably you should seriously look into migrating your environment to it. But it is not for all. 
Again keep in mind that Kubernetes/containers are mainly designed to work with distributed services, and when talking about resiliency, the resilient factor for kubernetes is the service, not the single pod.
What it means is that it could happen that a pod has issues and that it will be destroyed and rebuilt. If you have a pod containing stateless application, this takes seconds and has no impact on the cluster. 
But if you are talking about the node of a cluster with several terabytes of data, that will impact the cluster and the service. 
 
This is it, when dealing with large clusters and kubernetes, you may need to keep into account factors that may trigger actions, like dropping a pod, that instead require more flexibility. 
A good example is when a pod starts to have temporary network glitches. In the case of PDMO with PXC, the cluster will try to heal itself at PXC level. That operation may temporarily exclude a node from the cluster, and then let it join again. This operation has a cost in performance and efficiency of the cluster itself, but normally it resolves in a few seconds.
The same network glitches can be seen from kubernetes as a pod failure, and action may be taken directly on the pod, removing it from the cluster and rebuilding a new node. That operation not only may take hours or days, but will impact on the cluster performance in a significant way until all the data is copied over from a data node to another.
The addition of kubernetes also adds the need to have additional understanding on how effectively tuning the service to prevent wrong management actions like the one described. Tuning that needs to be done in respect to the traffic served and the dimension of the dataset.    
 
The advice here is to start small, adopt the help of experts in setting up the environment and tune it properly, gain good understanding and then start to increase the challenge. And never trust the defaults settings in the configuration files.
<h1>Wrapping up … </h1>
Given all the above.
If we are looking to containers/Kubernetes for our databases as an addition to an existing or new environment based on microservices, then the question is: Are we following evolution and really providing a benefit adopting the kubernetes/microservice approach to RDBMS?
As we have seen the move to microservice was an answer to provide better flexibility, scalability and many other *bility to the application layer. 
An RDBMS is, on the other hand, a system designed to answer to the ACID paradigm, which is the core of a RDBM and the model should never be broken.  
As we have seen, it is possible to create fragments and scale, but this needs to happen in getting consistent sets of data. Sharding is one option, but it requires a lot of investment in re-design the data layer and adapting the application. It will also open a lot of new problems, including data archiving and possible/probable data redundancy. All problems that require a deep analysis and proper design before moving. Not impossible but not something you should do without preparation, care and investment in time and money.
If instead the scope is to reduce the overhead in management and resource optimization, the task is easier. The scaling factor is not there, and you are not bounding your data layer to a microservice concept. But, you still need to be careful and properly analyze the move, considering the side effects not only today, but in the long run. Questions like, how big my dataset will be in 1 year, are pivotal.    
Blindly moving your database stack to containers/kubernetes or similar, without raising the right questions will only be your source of pain and despair for the next few years. It will not be your saviour but your executioner. 
At the same time there are cases where it may fit, as shown previously, but is not a blanket solution. 
Keep in mind this brief list:
<ul>
<li aria-level="1">An RDBMS like MySQL, does not scale writes inside a single service (at the moment of writing)</li>
<li aria-level="1">Bigger is the dataset longer it will be the possible downtime you will suffer (consider review your RTO/RPO)</li>
<li aria-level="1">You cannot have a relation 1:1 with an application microservice, unless your RDBMS contains only non relational schemas.</li>
<li aria-level="1">In case you go for microservice (and shard). You MUST have an OLAP system to serve consolidated data.</li>
<li aria-level="1">In case you are looking to reduce management cost and gain resource optimization. Start small, and keep going like that.    </li>
</ul>
Said that, if you instead of moving foolishly, stop and ask DBA and DATA experts for help, and design a proper system, then the use of RDBMS on containers/Kubernetes could become a nice surprise. 
Last thing, you may discover that you do not need a tight RDBMS, you may discover that something else could serve your needs better like Cassandra or CockroachDB, and then you need to migrate to it as a real solution. In that case your need for data experts is even more urgent and the need to have them working closely with your developers is a mandatory step to success. 
 
<h1>Disclaimer </h1>
My blog, my opinions. 
I am not, in any way or aspects, presenting here the official line of thoughts of the company I am working for.
<h1>Comments</h1>
Comments are welcome but given the topic and the fact this is going to be more probable (hopefully) an open discussion, I advice you to comment on <a href="https://www.linkedin.com/posts/marcotusa_mysql-on-kubernetes-demystified-activity-6854020804426915840-45mW">this linkedin post</a>. ]]></description>
<category>MySQL</category>
<pubDate>Wed, 13 Oct 2021 07:00:09 +0000</pubDate>
</item>
<item>
<title>Compare Percona Distribution for MySQL Operator VS AWS Aurora and standard RDS</title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/242-compare-percona-distribution-for-mysql-operator-vs-aws-aurora-and-standard-rds</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/242-compare-percona-distribution-for-mysql-operator-vs-aws-aurora-and-standard-rds</guid>
<description><![CDATA[I decided to write this article after a discussion with other colleagues.  We were wondering how and if the Percona Distribution for MySQL Operator(PDMO) could help a starting bu<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/performance-best-practices.png" alt="best performance rubber stamp grunge design dust scratches effects can be easily removed clean crisp look color 88158699" width="280" height="300" style="float: right;" />siness or a new project inside an enterprise.  We were also wondering how it behaves in relation to already well established solutions, like Amazon RDS or Google SQL.
In fact, we often see analysis and benchmarking covering huge datasets and instances with very high levels of resources . But we tend to forget that any application or solution started as something small, and it is important to identify a database platform that allows us to scale from zero to a decent amount of load, while keeping the cost as low as possible.  
This also happened while I was trying to get a bit more performance and stability out of Percona Operator for MySQL, and identify the possible minimal entrypoint and its usage. 
Given all the above I decided to perform an investigation comparing the Percona Distribution for MySQL Operatorin the two most used cloud services, AWS and Google cloud. I am working on the Azure AKS and will publish results when available.
The investigation was taking in consideration three possible scenarios:
<ul>
<li aria-level="1">1S - I have a new website and my editorial system is X, my content is mainly static (Mainly Read Only)</li>
<li aria-level="1">2S - I have a new social platform, what should I use (OLTP style)</li>
<li aria-level="1">3S - I have a small application dealing with orders/payments, highly transactional  (more TPC-C</li>
</ul>
And try to identify which is the best solution to use.
Of course, I do not pretend to cover all cases or even at 100% the one mentioned. But I think we can say a good 70% of cases nowaday match one of the three above
<h1>Tests</h1>
I have split the tests in three segment one for each major provider. This to be able to better compare the results, and following the assumption that if you are already registered on a platform you are more interested in comparing data locally than cross providers.
As usual for the tests I have used sysbench adapted version from here (<a href="https://github.com/Tusamarco/sysbench"></a><a href="https://github.com/Tusamarco/sysbench">https://github.com/Tusamarco/sysbench</a>) and sysbench-tpcc style from here (<a href="https://github.com/Tusamarco/sysbench-tpcc"></a><a href="https://github.com/Tusamarco/sysbench-tpcc">https://github.com/Tusamarco/sysbench-tpcc</a>). The customization for standard sysbench is in the table definition, and for tpcc like because I have improved the report layout.
Given we are talking about an entry point for a new product, I was considering two different scenarios:
<ul>
<li aria-level="1">When dataset fits in memory</li>
<li aria-level="1">When dataset doesn’t fits in memory  </li>
</ul>
Data sets dimensions:
<pre class="language-bash language-bash bash"><code>+--------------------+--------+-----------+----------+-----------+-----------+ | TABLE_SCHEMA | TABLES | ROWS | DATA (M) | INDEX (M) | TOTAL(M) | +--------------------+--------+-----------+----------+-----------+-----------+ | tpcc | 180 | 293113897 | 44762.00 | 9033.07 | 53795.07 | | tpcc_small | 90 | 48899475 | 7611.15 | 1542.35 | 9153.51 | | windmills_large | 80 | 317752934 | 35360.00 | 38463.79 | 73823.79 | | windmills_small | 80 | 39731506 | 4447.50 | 6075.82 | 10523.32 | +--------------------+--------+-----------+----------+-----------+-----------+ </code></pre>
All tables are using InnoDB. Keep in mind that scenario 1S and scenario 2S use the same schemas (windmills_xx).
This is because normally when you start your new adventure, you do not have Terabytes of data in your database, but .. you plan to have them in the future. So most of the time you start with something very small and want to have a solution able to scale at least a bit.
I was also considering whether they need to be able to scale (a bit) so I was running tests using 64, 92,128, 256, 512 threads. 
Finally each test was executed for 100 minutes, multiple times in different moments of the day and of the week. When multiple crashes happen the result will be zero. 
Summarizing we have:
<ul>
<li aria-level="1">3 different scenarios
<ul>
<li aria-level="2">Read Only (when write are so scarce that do not represent a variable)</li>
<li aria-level="2">OLTP Read and write some are inside transaction some not (especially reads)</li>
<li aria-level="2">TPC-C like load almost all inside transactions</li>
</ul>
</li>
<li aria-level="1">2 Dimensions of dataset  for each environment dimension 
<ul>
<li aria-level="2">Fits in memory (small) </li>
<li aria-level="2">Large enough to cause more flushing and disk activity (large)</li>
</ul>
</li>
<li aria-level="1">5 increasing number of threads </li>
</ul>
<h2>Assumptions</h2>
Assumptions and clarification before starting the review:
Q: Why are you using only one RDS instance instead of also spinning a Replica, we can also have the PDMO using only one node and reducing the cost impact.
A:  We are testing minimal production solutions. Using only a node for PDMO is delivering a solution that has no High Availability. RDS with multi AZ is providing High Availability, with lower level of nines but still functional. See: <a href="https://aws.amazon.com/blogs/database/amazon-rds-under-the-hood-multi-az/">https://aws.amazon.com/blogs/database/amazon-rds-under-the-hood-multi-az/</a>   <a href="https://aws.amazon.com/blogs/database/amazon-rds-under-the-hood-single-az-instance-recovery/">https://aws.amazon.com/blogs/database/amazon-rds-under-the-hood-single-az-instance-recovery/</a>
Q: Did you tune any of the default settings coming with the service? 
A: No. The assumption was that the provider is already providing optimised setup in relation to the solution offered. 
Q: Did you modify the PDMO settings?
A: Yes. The basic config coming with the PDMO is set for testing on a very minimal/test setup. It is not production ready and should not be used in production as well. The settings for PDMO had been changed to match the resources available and the expected load. Nothing more than what we should expect from the service providers.
<h2>The layout</h2>
<h3>AWS</h3>
The following is a simplified layout of the tested solutions.
In general terms each solution spans across multiple subnets and AZ, each solution has a dedicated Application for testing and application connected directly to the closest endpoint provided. In case of EKS the nodes are distributed over 3 different subnets.
<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/operator_layout_aws.png" alt="operator layout aws" style="display: block; margin-left: auto; margin-right: auto;" />
I am not reporting the whole internal details about how the Percona Distribution for MySQL Operator (PDMO) works, for that please refer to the documentation <a href="https://www.percona.com/doc/kubernetes-operator-for-pxc/index.html">https://www.percona.com/doc/kubernetes-operator-for-pxc/index.html</a>
The table below is reporting the high level description of the solutions tested:
<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/machines.PNG" alt="machines" width="700" height="160" />
*For PDMO we had to use machines with more resources to host additional services like HAProxy and the operator itself. But the resources allocated for the database services were reflecting the ones used AWS for Aurora and standard RDS. Said that, while for this exercise we have used machines with a bit more resources, it is common practice to have one powerful host node with multiple database clusters to optimise cost and resource utilization, I will cover this better later in the cost section.
<h1>Down into the rabbit hole </h1>
A lot of graphs can be found <a href="https://github.com/Tusamarco/blogs/tree/master/comparing_rds_operator/files/aws-files">here</a>: 
The name includes the test and the data set dimension.
In this section we will review the results of the tests by provider.
<h2>AWS</h2>
AWS is the most utilized Cloud provider and is also the one with more solutions when discussing Database services. I have tested and compared PDMO vs Aurora and standard RDS.
For clarity I am reporting two different sets of graphs for details. One is related to CPU, memory etc, the other is more MySQL specific. 
In this context we have the following:
<ul>
<li aria-level="1">aurora-eks-test1-instance-1; Primary node of the Aurora service</li>
<li aria-level="1">aurora-eks-test1-instance-1-eu-central-1a; Secondary node of the Aurora service</li>
<li aria-level="1">Mt-rds-kube-test; RDS instance </li>
<li aria-level="1">pxc-cluster1-pxc-0 ; Pod serving the MySQL traffic for PDMO </li>
</ul>
As you may notice from the graphs, the Aurora service automatically swap/failover from a node to another more than once.  
<h3>Checking Read Only traffic (scenario 1S)</h3>
Let us start with the easiest case, a simple new website with some editorial. As already mentioned this can be considered a Read Only case given the writes should be counted in few by minutes or even less. 
<h4>Small Dataset</h4>
As previously mentioned, this case is the first baby step, when your DB is really at an infant stage.
Y axis reports operations/sec
 <img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Reads-sec-RO-small-_10.png" alt="chart 2021 09 28 Reads sec RO small 10" width="600" height="407" />
In case no matter if the dataset fits in memory or not, we can notice how the Percona MySQL operator is able to serve a higher amount of traffic in respect to the other AWS solutions. If we check what seems to be the limitation factor, we can see that:
 2 CPU
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_small_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_small_OS_cpu.png" alt="2 cpu ro small OS cpu" width="700" height="116" /></a>
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_small_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_small_threads.png" alt="2 cpu ro small threads" width="700" height="157" /></a>
4 CPU
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_small_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_small_OS_cpu.png" alt="4 cpu ro small OS cpu" width="700" height="133" /></a>
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_small_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_small_threads.png" alt="4 cpu ro small threads" width="700" height="176" /></a>
We can see how in both cases the AWs solutions reach 100% of CPU utilization, limiting the number of operations they can execute in memory. Also notice how the load was automatically moved from one Aurora instance to another during the tests (CPUs graph).
<h4>Large Dataset</h4>
Now let's see what happens if I am successful and my website starts to increase in content, such that my dataset does not fit in memory, and as such I will have MySQL evicting old pages from memory
<img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Ops-sec-RO-Large_1.png" alt="chart 2021 09 28 Ops sec RO Large 1" width="700" height="474" />
In case the dataset exceeds the dimension of the memory, PDMO is still performing significantly better than the other two solutions when using 4 CPUs. But it has a significant reduction of performance when using 2 CPUs.
2 CPUs
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_large_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_large_OS_cpu.png" alt="2 cpu ro large OS cpu" width="700" height="116" /></a>
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_large_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_large_threads.png" alt="2 cpu ro large threads" width="700" height="159" /></a>
Disk Latency
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_large_OS_disk_latency.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_ro_large_OS_disk_latency.png" alt="2 cpu ro large OS disk latency" width="700" height="114" /></a>
The other relevant fact to notice is how the disk latency affected the reads for the PDMO when using a large dataset and 2 CPUs. This negative effect doesn't show up in case of 4 CPUs 
4 CPUs 
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_large_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_large_OS_cpu.png" alt="4 cpu ro large OS cpu" width="700" height="133" /></a>
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_large_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_large_threads.png" alt="4 cpu ro large threads" width="700" height="183" /></a>
Also in the case of 4 CPUs we see the same behavior, with AWS solutions not able to scale and failing to serve traffic. 
Disk Latency
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_large_OS_disk_latency.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_ro_large_OS_disk_latency.png" alt="4 cpu ro large OS disk latency" width="700" height="132" /></a>
<h3>Checking OLTP (scenario 2S)</h3>
This case is obviously more complicated given the writes and transactions. The level of reads is still high, around 93% but all operations are enclosed inside transactions. 
In this case we MUST review in parallel the SMALL and LARGE dataset to better understand what is happening.
<h4>Operation/sec</h4>
<table>
<tbody>
<tr>
<td> <a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Reads-sec-OLTP-small-_11.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Reads-sec-OLTP-small-_11.png" alt="chart 2021 09 28 Reads sec OLTP small 11" width="400" height="271" /></a></td>
<td><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Ops-sec-OLTP-large_2.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Ops-sec-OLTP-large_2.png" alt="chart 2021 09 28 Ops sec OLTP large 2" width="400" height="272" /></a> </td>
</tr>
</tbody>
</table>
 
As you can see the solution using the Operator is overperforming in the case of small and large datasets. While AWS solutions are still suffering from the same CPU limitation we have seen in the read only load. 
2 CPUs small
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_small_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_small_OS_cpu.png" alt="2 cpu rw small OS cpu" width="700" height="118" /></a>
 <a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_small_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_small_threads.png" alt="2 cpu rw small threads" width="700" height="156" /></a>
2 CPUs Large
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_large_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_large_OS_cpu.png" alt="2 cpu rw large OS cpu" width="700" height="117" /></a>
We can observe how CPUs resources in the case of AWS solutions are always at 100%, while PDMO can count on resource scaling.
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_large_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_large_threads.png" alt="2 cpu rw large threads" width="700" height="157" /></a>
2 CPUs large Disk Latency
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_large_OS_disk_latency.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/2_cpu_rw_large_OS_disk_latency.png" alt="2 cpu rw large OS disk latency" width="700" height="116" /></a>
4 CPUs small
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_small_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_small_OS_cpu.png" alt="4 cpu rw small OS cpu" width="700" height="132" /></a>
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_small_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_small_threads.png" alt="4 cpu rw small threads" width="700" height="178" /></a> 
4 CPUs Small Disk Latency
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_small_OS_disk_latency.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_small_OS_disk_latency.png" alt="4 cpu rw small OS disk latency" width="700" height="130" /></a>
4 CPUs large
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_large_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_large_OS_cpu.png" alt="4 cpu rw large OS cpu" width="700" height="134" /></a>
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_large_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_large_threads.png" alt="4 cpu rw large threads" width="700" height="183" /></a>
4 CPUs Large Disk Latency
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_large_OS_disk_latency.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_rw_large_OS_disk_latency.png" alt="4 cpu rw large OS disk latency" width="700" height="131" /></a>
With 4 CPUs available the Aurora solution can count in a bit more “space” to scale, but still not enough to result as efficient as PDMO.
As for the read only, we can see the disk latency significantly increase in the case of Large dataset, especially for the PDMO solution.  
It is interesting to see how Aurora is penalized not only by the CPU utilization but also by a very high disk latency. 
 
<h4>Latency 95pct (lower is better)</h4>
<table>
<tbody>
<tr>
<td><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-OLTP-small-_20.png" target="_blank"> <img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-OLTP-small-_20.png" alt="chart 2021 09 28 Latency 95 Pct (y axis in ms) OLTP small 20" width="400" height="272" /></a></td>
<td><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-OLTP-large_19.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-OLTP-large_19.png" alt="chart 2021 09 28 Latency 95 Pct (y axis in ms) OLTP large 19" width="400" height="272" /></a> </td>
</tr>
</tbody>
</table>
 
Latency metrics reflect the operation trend, highlighting how the low CPU-resources/Disk-latency affects all solutions, especially in case of large datasets. 
<h3>Checking TPC-c (inspired) load (scenario 3S)</h3>
Also in this case we have that all the queries are encapsulated in a transaction and in this kind of test the read and write operations are close to being split at the 50% mark, so we expect to see a significant load on disk write operation and for the data replication mechanism.  
<h4>Operation/sec</h4>
<table>
<tbody>
<tr>
<td><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Reads-sec-TPCC-small-_12.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Reads-sec-TPCC-small-_12.png" alt="chart 2021 09 28 Reads sec TPCC small 12" width="400" height="272" /></a></td>
<td><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Writes-sec-TPCC-large_14.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Writes-sec-TPCC-large_14.png" alt="chart 2021 09 28 Writes sec TPCC large 14" width="400" height="271" /></a></td>
</tr>
</tbody>
</table>
 
In this scenario PDMO results to be more penalized when the dataset is larger than the available memory. In some way this is expected, given the level of writing is such that the solution pays the cost of the virtual-synchronous replication mechanism used in PXC.
If we check the details we can see:
4 CPUs Large
 <a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_OS_cpu.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_OS_cpu.png" alt="4 cpu tpcc large OS cpu" width="700" height="135" /></a>
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_threads.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_threads.png" alt="4 cpu tpcc large threads" width="700" height="181" /></a>
4 CPUs Disck Latency
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_OS_disk_latency.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_OS_disk_latency.png" alt="4 cpu tpcc large OS disk latency" width="700" height="131" /></a> 
As we can see the PDMO node receiving the direct load is not crazy loaded, while the disk latency is higher than RDS with an average of 4ms write and 8ms for read operations. But that is not enough to justify the delta we have in performance loss. What is preventing the solution from being as efficient as the other two?
PXC flow control
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_pxc_flow_control_new.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_pxc_flow_control_new.png" alt="4 cpu tpcc large pxc flow control new" width="700" height="218" /></a>
PDMO is using PXC as technology to offer service high availability and full data consistency. This means that the cluster must certify and apply all writes in any node at each commit, with the increase of the write operations this solution also increases the pressure on the replication layer and if any node has a delay in applying the write, that delay will affect the efficiency of the whole cluster. 
In our case we can see that nodes pxc-0 and pxc-2 are on hold for pxc-1 to perform the writes. 
We also have as side effect the increase of the sending queue on the primary node (pxc-0)
<a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_pxc_send_queue.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/graphs/img_ro_2_4_cpu/4_cpu_tpcc_large_pxc_send_queue.png" alt="4 cpu tpcc large pxc send queue" width="700" height="204" /></a>
<h4> </h4>
<h4>Latency</h4>
<table>
<tbody>
<tr>
<td><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-TPCC-small-_22.png" target="_blank"> <img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-TPCC-small-_22.png" alt="chart 2021 09 28 Latency 95 Pct (y axis in ms) TPCC small 22" width="400" height="272" /></a></td>
<td><a href="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-TPCC-large_21.png" target="_blank"><img src="http://www.tusacentral.net/joomla/images/stories/pdmo_comparisons/aws/img/chart_2021_09_28_Latency-95-Pct-(y-axis-in-ms)-TPCC-large_21.png" alt="chart 2021 09 28 Latency 95 Pct (y axis in ms) TPCC large 21" width="400" height="272" /> </a></td>
</tr>
</tbody>
</table>
Latency is just confirming us the picture we had from the operations.  
<h2>What should I pick? </h2>
Before indicating what should be a good solution for what, let us talk about costs.
<h3>Costs</h3>
To discuss the cost, we need to make some assumptions. 
The first assumption is that we need to calculate the cost by solutions and not by node. This is because we cannot consider a solution with a single node for PDMO or Aurora. To be effective both require their production minimum requirements, which for Aurora is two nodes and for PDMO given is based on PXC 3 nodes. Given that calculating by node is a fictitious artifact, while calculating by solution gives you the real cost.   
The second assumption is that we can have more PDMO solutions running on the same Node. Which means we will have a better relation cost/benefit in utilizing larger nodes and use them to run multiple PDMO solutions. In that case, the only cost that will need to be added is the cost of the attached EBS volume per delivered POD. 
 
Let me provide an example of PDMO calculation. 
If I consider the minimal Node resource requirements, as we did for these tests, I will need to have:
<table border="1">
<thead>
<tr style="background-color: #bbbbbb;">
<td>
Solution
</td>
<td>
Node Type
</td>
<td>
Monthly Cost
</td>
</tr>
</thead>
<tbody>
<tr>
<td>
PDMO 2 CPU
</td>
<td>
R5.xlarge
</td>
<td>
1,609 USD
</td>
</tr>
<tr>
<td>
PDMO 4 CPU
</td>
<td>
R5.2xlarge
</td>
<td>
2,274 USD
</td>
</tr>
</tbody>
</table>
 
But if we use a larger node that can host multiple PDMO like a r5.4xlarge (16 CPU 128GB) that can host multiple PDMO :
<table border="1">
<tbody>
<tr style="background-color: #bbbbbb;">
<td>
Solution
</td>
<td>
PDMO instances per node
</td>
<td>
Monthly Cost - full solution for PDMO
</td>
</tr>
<tr>
<td>
PDMO 2 CPU
</td>
<td>
3
</td>
<td>
1,202 USD
</td>
</tr>
<tr>
<td>
PDMO 4 CPU
</td>
<td>
2
</td>
<td>
1,803 USD
</td>
</tr>
</tbody>
</table>
 
As you can see also if a solution based on r5.4xlarge has higher absolute cost per node, the fact you can run multiple PDMO in a node allows you to reduce the cost in a significant way by solution.   
 
Said that let us summarize the cost of the solutions:
<table border="1">
<tbody>
<tr style="background-color: #bbbbbb;">
<td>
Solution
</td>
<td>
Node Type
</td>
<td>
Monthly Cost
</td>
</tr>
<tr>
<td>
Aurora* 2 CPU
</td>
<td>
db.r5.large
</td>
<td>
$6,323.30 
</td>
</tr>
<tr>
<td>
Aurora* 4 CPU
</td>
<td>
db.r5.xlarge
</td>
<td>
$6,834.30 
</td>
</tr>
<tr>
<td>
RDS 2 CPU
</td>
<td>
db.r5.large
</td>
<td>
$1,329.80 
</td>
</tr>
<tr>
<td>
RDS 4 CPU
</td>
<td>
db.r5.xlarge
</td>
<td>
$1,753.20 
</td>
</tr>
<tr>
<td>
PDMO 2 CPU (single instance per delivered node)
</td>
<td>
r5.xlarge
</td>
<td>
$1,609.20 
</td>
</tr>
<tr>
<td>
PDMO 4 CPU (single instance per delivered node)
</td>
<td>
r5.2xlarge
</td>
<td>
$2,274.96 
</td>
</tr>
<tr>
<td>
PDMO 2 CPU (multiple instance per delivered node)
</td>
<td>
r5.4xlarge
</td>
<td>
$1,202
</td>
</tr>
<tr>
<td>
PDMO 4 CPU (single instance per delivered node)
</td>
<td>
r5.4xlarge
</td>
<td>
$1,803
</td>
</tr>
</tbody>
</table>
* For Aurora we had to include an average number of operations based on the test executed. Please refer to: <a href="https://calculator.aws/#/createCalculator/AuroraMySQL"></a><a href="https://calculator.aws/#/createCalculator/AuroraMySQL">https://calculator.aws/#/createCalculator/AuroraMySQL</a>
 
As you can see PDMO is just a little bit more expensive than RDS when deployed in single instance mode. But it also provides better performance in most cases and always has a higher level of High Availability, given PXC is used in the background. 
There is no discussion if you can use multiple PDMO on larger nodes, in that case the cost goes lower than RDS.
<h3>So what to pick</h3>
<h4>Read-only</h4>
Choosing is not only based on performance, but also how efficient the solution we pick is to perform the maintenance of our platform. In this regard both operator and Aurora offer a good level of automation. But PDMO was always performing better than the other two solutions. On top of that Aurora based solutions are more expensive than the others, so in the end, for me, choosing PDMO is the logical thing to do, being ready to eventually do a small scale-up in case I need more resources. This is another advantage of PDMO, I can eventually decide to add fragments of the CPU cycle, without the need to move to a higher and more expensive system.
<h4>OLTP</h4>
In this case there is no doubt that PDMO is the solution to go to. Aurora can be seen as a possible replacement given that the RDS solution has a lower level of HA in respect to the other two. But as we know the Aurora costs are higher than any other compared solutions here.
<h4>TPCC</h4>
RDS is the solution to go to serve the load and save the money, but you will have a lower level of HA. If you are willing to cover the cost Aurora is the solution offering you good performance and high level of High Availability. 
Finally we can say that PDMO was penalized by the replication/certification mechanism in use in PXC but, the significant saving in terms of cost can justify the adoption, especially considering that with a limited scaling up in resources we can gain more traffic served. Scaling will never be as expensive as if we adopt Aurora as a solution.     
<h1>Conclusions</h1>
Trying to identify an entry point solution for a starting activity is harder than choosing a platform for something already well defined. There is a lot of uncertainty, about the traffic, the kind of operations, the frequency of them, the growth (if any). 
In my opinion there are already so many variables and things that can possibly go wrong, that we should try to adopt, if not the safest solution, the one that gives us a good level of confidence in what we will be able to achieve. 
I was not surprised to see products such as Aurora shine when we have Tpcc like load, but I was pleased to see Percona solution with the operator performing in an admirable way. 
Of course I was not using the settings coming from the vanilla installation, given they are for testing. So if you install PDMO in production you MUST change them  (see <a href="https://github.com/Tusamarco/blogs/tree/master/comparing_rds_operator/files/crs">here</a> for my cr.yaml) and I was counting on version 1.9.0 at least, without which PDMO is not even in the game. 
But it is obvious that Percona is on the right path, with a long road ahead, but we it is moving in the right direction.
Percona Distribution for MySQL Operatoris still a baby, but is growing fast, and the more we have people using, testing and providing feedback, the fastest Percona will be able to provide software that not only early adopters implement, but also the more conservative ones can use for their production.
<h2>References</h2>
<a href="https://calculator.aws/#/createCalculator/RDSMySQL"></a><a href="https://calculator.aws/">https://calculator.aws/</a>
<h1>Disclaimer </h1>
My blog, my opinions. 
I am not, in any way or aspects, presenting here the official line of the company I am working for.
<h1>Comments</h1>
If you have comments or suggestions please add them to the <a href="https://www.linkedin.com/posts/marcotusa_compare-percona-distribution-for-mysql-operator-activity-6852136882537545728-jfTe">linkedin post</a>]]></description>
<category>MySQL</category>
<pubDate>Sat, 09 Oct 2021 07:00:45 +0000</pubDate>
</item>
<item>
<title>Boosting Percona MySQL Operator efficiency</title>
<link>http://www.tusacentral.net/joomla/index.php/mysql-blogs/241-boosting-percona-mysql-operator-efficiency</link>
<guid isPermaLink="true">http://www.tusacentral.net/joomla/index.php/mysql-blogs/241-boosting-percona-mysql-operator-efficiency</guid>
<description><![CDATA[<h1>Preface</h1>
Percona is well known for its offer of several outstanding fully open source, free to <a href="https://www.percona.com/software">download</a> software packages. And while Percona started as a MySQL focus company, nowadays it covers different technologies such as MySQL, Postgres and Mongodb. <a href="https://www.percona.com/blog/wp-content/uploads/2018/11/lightspeed.jpg"><img src="https://www.percona.com/blog/wp-content/uploads/2018/11/lightspeed-300x245.jpg" alt="" width="300" height="245" class="size-medium wp-image-53798 alignright" style="float: right;" /></a>
In its constant effort to make life easier for our utilizer Percona had moved from providing single software packages, to Percona Distributions for MySQL, MongoDB and Postgres. Percona Distributions are a set of software packages that Percona has tested and certifies working together. That it easier to deploy architectural solutions requiring the use of multiple components, such as proxy, topology manager, backup software and more. 
But we are going even further, and with the release of Percona Distribution for MySQL/MongoDB <a href="https://www.percona.com/doc/kubernetes-operator-for-pxc/index.html">Operator</a> we are providing a high level of automation to roll out and manage solutions based on Percona Distributions. 
One of my tasks, as MySQL technical leader, is to identify optimal architectures to serve several common cases. Such as Percona Distribution for MySQL: <a href="https://www.percona.com/blog/2021/04/14/percona-distribution-for-mysql-high-availability-with-group-replication-solution/">High Availability with Group Replication Solutio</a>n. Or in the case of  Percona Distribution for MySQL Operator, identify the different dimensions (low/mid/high utilization) and suggest a Vanilla setup with the scope to get the most out of the solution deployed.
This is a long exercise, which started with a lot of internal discussions to identify what can make sense as traffic, then testing, identifying the saturation points, testing again and so on. 
It is during this process that I found a small problem (<a href="https://jira.percona.com/browse/K8SPXC-749">Feature Request</a>). This small issue is preventing us from easily and dynamically modifying some parameters in the checks the Operator  uses. Given that we had to put the testing on hold until the above FR is implemented. As you can see it is a small thing but it will give us better control over the Operator behavior and will help to have a well tuned platform.
This article is to show the level of improvement you can have with small but targeted tuning.To do so I used the smallest solution we have identified. The solution is dimensioned to serve a small website or a simple application with low level of traffic. 
<h1>The environment</h1>
To help identify a balanced setup we were using sysbench and sysbench-tpcc. The whole stack on GCP was composed of Application nodes with sysbench, 2 ProxySQL nodes for R/W split only, 3 VMS 8 CPU 32GB RAM, with the Percona operator managing the MySQL service.
<a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Untitled-Diagram.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Untitled_Diagram.png" alt="Untitled Diagram" width="370" height="396" class="wp-image-76889 aligncenter" style="display: block; margin-left: auto; margin-right: auto;" /></a>
<h2>Tests</h2>
For this specific test we were running 68 - 96 -128 - 256 threads:
<ul>
<li style="font-weight: 400;" aria-level="1">Sysbench read only</li>
<li style="font-weight: 400;" aria-level="1">Sysbench read/write</li>
<li style="font-weight: 400;" aria-level="1">Sysbench Tpc-c like </li>
</ul>
The tests were run multiple times and the data considered is the consolidation of the multiple runs. 
We always run first on basic environments for baseline. Meaning no tuning for the MySQL or Operator, just dimension correctly disk space and BufferPool (and related).
Then we apply some tuning and run the tests multiple times eventually refining when/where needed. 
The code can be found <a href="https://github.com/Tusamarco/sysbench">here</a> and <a href="https://github.com/Tusamarco/sysbench-tpcc">here</a>
<h1>And now the results…</h1>
<h2>Sysbench r/w tests</h2>
I am not going to describe in detail the images that I think are clear enough. Just keep in mind on the left we have the results from our baseline, on the right the same tests on the same platform with the optimization applied.
<h3>Operations</h3>
<a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture6.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture6.png" alt="Picture6" width="800" height="392" class="alignnone size-large wp-image-76891" /></a>
Worth to mention that without tuning, the platform was not able to consistently scale up to 256 threads. While with a bit of adjustment not only it was able to serve 256 threads but we could have gone a bit further.
<h3>Reads</h3>
<a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture7.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture7.png" alt="Picture7" width="800" height="392" class="alignnone size-large wp-image-76892" /></a>
<h3>Writes</h3>
<a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture8.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture8.png" alt="Picture8" width="800" height="392" class="alignnone size-large wp-image-76893" /></a>
<h3>Comments</h3>
As you can see the sysbench tests clearly indicate that the platform with minor adjustment was acting better, and that it was able to serve more and with constant scaling. Let me add that almost all the tests runned on the “basic” platform had incidents, meaning as soon as the traffic was increasing, Sysbench was reporting connection interruptions or errors.
<h2>TPC-C</h2>
<h3>Operations</h3>
<h3><a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture10.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture10.png" alt="Picture10" width="800" height="395" class="alignnone size-large wp-image-76895" /></a></h3>
<h3>Reads</h3>
<a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture11.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture11.png" alt="Picture11" width="800" height="395" class="alignnone size-large wp-image-76896" /></a>
<h3>Writes</h3>
<h3><a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture12.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture12.png" alt="Picture12" width="800" height="395" class="alignnone size-large wp-image-76897" /></a></h3>
<h3>Comments</h3>
Also for Tpc-c like tests we have exactly the same trend. With our “optimised” solution able to serve up to 1516 qps while the “basic” one was able to reach only 322. In this case Also the “optimised” solution was not able to scale up to 256 threads, but that makes sense, given the more intense write workload present in this test and the small dimension of the platform.
 
<h1>Wooha what have you changed?</h1>
You may think we have done crazy things to get this difference, but we did not.
Let us jump back. As indicated at the beginning I had opened a FR (<a href="https://jira.percona.com/browse/K8SPXC-749">https://jira.percona.com/browse/K8SPXC-749</a>) to be able to tune some/most of the timeouts existing in the operator. <a href="https://www.percona.com/blog/wp-content/uploads/2021/06/8e4.jpg"><img src="https://www.percona.com/blog/wp-content/uploads/2021/06/8e4.jpg" alt="" width="287" height="198" class="wp-image-76885 alignright" style="float: right;" /></a>
Why? Think about this, when you install a cluster on iron, you do not set it to be able to work only when the load is low, and all the components of the server are able to answer in nanoseconds. What you do instead is tune the whole system to accommodate the increasing load, and you will give to some elements more space for “flexibility” eventually expecting to have delays in answer. When doing so you also need to correctly align all the parameters that will be affected on cascade. For instance if you know your data nodes will be very busy serving queries, they may also slow down in answering internal health checks, but if you relax the cluster health checks and not the checks used for testing the cluster from operator point of view, the platform will be unbalanced and will not work correctly.
At the same time, if you do not tune the solution at all, you may end up with a platform that is theoretically able to serve the load, but that is crashing for artificial limitations. 
The last one is exactly what was happening with our “basic” solution. As it is the operator comes with parameters that allow it to work well, but that are not designed to scale. Is like having a server where your CPUs are always at 20% and if the applications ask more, a controller will chop them in fear of having too much load. But the fact is that you want to have the CPUs at 80% or the server will be underutilized. 
Anyhow what we have changed was some Innodb parameters, to allow internal operations to work better. Then we force consistent reads in PXC, which actually SLOW down the operations, and finally we tune the PXC cluster to be more flexible in its internal checks, avoiding having it to expel nodes unless really needed to. 
All the above were done using the Operator configuration, but then we had to work manually changing all the timeouts parameters used by the operator checks to be aligned with what we had defined in the cluster. 
In particular what we have changed was:
<pre class="lang:sh decode:true">script line value
/usr/bin/clustercheckcron 33 TIMEOUT=10
liveness-check.sh 23 TIMEOUT=5
readiness-check.sh 21 TIMEOUT=10
/usr/local/bin/check_pxc.sh 15 TIMEOUT=${CUSTOM_TIMEOUT:-10}</pre>
<h2>Wait .. why ProxySQL?</h2>
Ok this is another long discussion and I will cover it better in another article. For now just consider that HAProxy does not allow r/w splitting or other nice functionalities like firewalling etcetera etcetera. So the idea is simple, let us use the operator with what fits it better, and then decouple the special needs, eventually adding proxysql in a separate deployment. 
If you are scared of the cost of adding an additional block to the architecture:
<a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture1-a.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture1-a.png" alt="Picture1 a" width="800" height="401" class="alignnone wp-image-76883" /></a>
<a href="https://www.percona.com/blog/wp-content/uploads/2021/06/Picture1-b.png"><img src="http://www.tusacentral.net/joomla/images/stories/boosting_operator/Picture1-b.png" alt="Picture1 b" width="800" height="264" class="alignnone wp-image-76884" /></a>
Where: ProxySQL means: Application → ProxySQL → HAProxy. HAProxy means: Application → HAProxy. 
Hope this puts your worries at rest, of course this is using the “optimised” solution.
<h1>Conclusions</h1>
Percona Distribution for MySQL Operator, is a constantly growing/improving solution. It also has a lot of interesting features, like being able to manage your backup/restore, point in time recovery, and more. But its adoption is still limited and it is normal to have some drawbacks like this one. It is on us who play with real production environments, or as in this case playing to define certified solutions, to give feedback to improve how the operator works, in order to make it a stronger product able to serve you better day by day.
Now we are going to wait for the FR to be implemented, and then we will recover our dimensioning work. ]]></description>
<category>MySQL</category>
<pubDate>Thu, 24 Jun 2021 08:09:19 +0000</pubDate>
</item>
</channel>
</rss>

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

Download the "valid RSS" banner.
Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)
Add this HTML to your page (change the image src attribute if necessary):

<a href="http://www.feedvalidator.org/check.cgi?url=http%3A//www.tusacentral.net/joomla/index.php/mysql-blogs%3Fformat%3Dfeed%26type%3Drss"><img src="valid-rss-rogers.png" alt="[Valid RSS]" title="Validate my RSS feed" /></a>

If you would like to create a text link instead, here is the URL you can use:

http://www.feedvalidator.org/check.cgi?url=http%3A//www.tusacentral.net/joomla/index.php/mysql-blogs%3Fformat%3Dfeed%26type%3Drss

Home · About · News · Docs · Terms