How ProxySQL deal with schema (and schemaname)

I think that very often we are so focus in analyzing internals, or specific behaviours/situations/anomalies that we tend to forget the simple things.

It happened to me that last week a couple of customers raise the same question: "How ProxySQL manage the default schema, or the ones declared inside a FROM/JOIN?"
I was a bit surprise because I was given that for granted, and my first thought was, 'well read the documentation', but then I realize we do not have a clear section in the documentation about this.

Given that and also because I realize I had not done a full and extensive test on how the SCHEMA is actually managed.
I decide to do a simple set of tests and write down few lines.

This blog is to answer that very simple question:"How ProxySQL manage the default schema, or the ones declared inside a FROM/JOIN?"
The blog is split in two parts, part 1 simple declaration and summary of what happen. Part 2 all the details and tests, in case you want to see them.

Schemaname and ProxySQL

In ProxySQL we can specify the schema in two different places and with different scope.

In the mysql_user table as property of the USER, where it will represent the DEFAULT schema.
In the mysql_query_rules as filter for which a query can be capture. The filter is valid only for the explicitly set default SCHEMA, (ie with -D mysql command line or USE). ProxySQL will NOT analyze the FROM SQL clausole. Given the limitation in the above point 2, it is not 100% safe to trust the SCHEMANAME as filter unless you are 200% sure the code do not contains commands to change default schema like USE.

On the other hand if I want to filter by a schemaname (in general) it is safer and more flexible to use regular expression and query_rules, as I will show later.

For the scope of this article I want to answer these simple sub-questions:

How ProxySQL differes from MySQL in managing explicit default schema declaration? does it respect the -D or USE
How proxy respect/follow security agains schema
How schemaname filter acts in the query rules?
How can I transparently redirect using schema name?

To test the above I have created:

two servers:

Master

Slave

two schemas:

world

City

Country

myworld

CityM

CountryM

three users:

uallworld, can access all the schemas (including test)

uworld, can access world in write/read on Master, read on slave. Can access myworld in read on slave.

umyworld, can access myworld in write/read on Master, read on slave. Can access world in read on slave.

Queries used during the tests:

    select database();
    update world.City set Population=10500000 where ID=1024;
    update world.Country set LifeExpectancy=62.5 where Code='IND';
    update myworld.CityM set Population=10500001 where ID=1024;
    update myworld.CountryM set LifeExpectancy=0 where Code='IND';
    Select * from world.City order by Population DESC limit 5 ;
    Select * from myworld.CityM order by Population DESC limit 5 ;
    Select City.*, Country.name, Country.LifeExpectancy from world.City as City

        join world.Country as Country on City.CountryCode=Country.code order by Population DESC limit 5; 
    Select City.*, Country.name, Country.LifeExpectancy from myworld.CityM as City

        join myworld.CountryM as Country on City.CountryCode=Country.code order by Population DESC limit 5; 
    Select City.*, Country.name, Country.LifeExpectancy from world.City as City

        join myworld.CountryM as Country on City.CountryCode=Country.code order by Population DESC limit 5;

To setup the environment see instructions at Annex 1.

Short story

How ProxySQL differes from MySQL in managing explicit default schema declaration? Does it respect the -D or USE?
MySQL and ProxySQL will behave the same when passing the default schema, setting it as default.
MySQL
mysql -uuallworld -ptest -h192.168.1.107 -P 3306 -D test

ProxySQL
mysql -uuallworld -ptest -h127.0.0.1 -P 6033 -D test

If a default schema is set in ProxySQL the schema coming from command line or connection (like in java:"connUrl=jdbc:mysql://192.168.1.50:6033/test"), will override the ProxySQL default.
In case a default schema is not pass during the connection MySQL and ProxySQL will differs on how the behave:
MySQL will set the current schema to NULL. It is to be noted that MySQL accept a NULL schema when u connect but then once SET it with USE you cannot set it back to NULL.
ProxySQL will set it as the one declared default in the mysql_user table. If no schema is declared as default, ProxySQL will elect information_schema as the default. In short ProxySQL cannot have a default schema set to NULL.
How proxy respect/follow security agains schema
MySQL
mysql -uuworld -ptest -h192.168.1.107 -P 3306 -D test ERROR 1044 (42000):
Access denied for user 'uworld'@'%' to database 'test'

ProxySQL
mysql -uuworld -ptest -h127.0.0.1 -P 6033 -D test
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 483902 Server version: 5.5.30 (ProxySQL)

On connection the behaviour is different between MySQL and ProxySQL.
Why? Because while you directly connect to MySQL, when you connect to ProxySQL you do not actually touch the final server.

ProxySQL is NOT a forward proxy, but a reverse proxy, so its scope is act as an intermediary for its associated servers to be contacted by any client. Backend connection that will serve your client is establish at the monent you actually send a query, which will be comunicated to the relevent host group.

As such when you connect, you only open a connection to the ProxySQL. While issue a query will USE a connection to the backend and if the user do not have the right grants an error will be returned.
But ProxySQL will not known until you submit the query and it can decide where this query should go (which HG to point to).
mysql> select database(); ERROR 1044 (42000): Access denied for user 'uworld'@'%' to database 'test'

Aside from this all the GRANTS defined in MySQL are transparent and followed by ProxySQL
How schemaname filter acts in the query rules?

In MySQL we can easily change the default schema with USE , this action is fully supported by ProxySQL.
But it may have some side effects when using "schemaname" as filter in the query_rules.

If you define a rule that include the default schemaname and the default schema is changed with USE, the rule will not apply, and unpredictable results may happen.
To prevent that ProxySQL has another option in mysql_user "schema_locked" which will prevent the schema to be changed by USE.

This feature is present but not fully implemented, and after a brief conversation with Rene (https://github.com/sysown/proxysql/issues/1133), I hope it will be soon.

Given that, when designing Query rules using the Default schema, you must to take in consideration the possibility to have the application or user changing the default schema and invalidating that rule.
how can I transparently redirect using schema name?

This is not a Schema feature, more one of the things that in ProxySQL are quite easy to set, while close to be impossible if done in plain MySQL.
When connecting directly with MySQL there is no option for you to "play" with GRANTS and schema such that you will transparently allow a user to do an action on a schema/server pair
and another on a different schema/server pair.
When using ProxySQL to filter by schemaname is quite trivial.
For instance assuming we have 3 users one is admin of the platform which include 2 separate schemas (or more), each user can access one schema for write (but that can be table as well),
and a final slave with reporting information, where all the users that needs to read from other schema except their own can read cross schemas. While all the select not cross schema mus still got to the Master;
This is not so uncommon, actually with few variant is exactly what one of the customer I spoke last week needs to do.
Now in MySQL this is impossible while in ProxySQL is just a matter of 3 rules:
insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply) values(10,'uworld',10,1,3,'^SELECT.*FOR UPDATE',1);
insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply,FlagOUT,FlagIN) values(11,'uworld',10,1,3,'^SELECT ',0,50,0);
insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply,FlagOUT,FlagIN) values(12,'uworld',11,1,3,'\smyworld.',1,50,50);
Simply applying the rules above will allow the application to transparently access the data from different servers without errors,
following the GRANTS given on the MySQL side. This for ONE user (uworld), but it can eventually extended to any, and the rule chain is minimal just 2 rules, so minimal overhead also with high traffic.

Summarizing

ProxySQL is following the MySQL model to access the schema, in most parts. There are a couple of differences though.

ProxySQL will require to set a default schema when connecting, implicitly or explicitly.

ProxySQL will not return an error at connection time, if a user is not authorized to connect to the given schema. Error will raise at the first query, moment when ProxySQL will actually establish the connection.

Finally using ProxySQL, will allow administrator to play with GRANTS and HG/servers to provide transparent access to data in a more granular way, choosing an HG where user may have read (or other specific) access, against one where user is not allow at all .

Long Story ... click me

Annex 1

Create environment for test

create schema myworld;
create table myworld.CityM like City;
create table myworld.CountryM like Country;
create table myworld.CountryLanguageM like CountryLanguage;
insert into  myworld.CityM select * from City;
insert into  myworld.CountryM select * from Country;
insert into  myworld.CountryLanguageM select * from CountryLanguage;
 
delete from mysql_users where username like '%world';
insert into mysql_users (username,password,active,default_hostgroup,default_schema,transaction_persistent) values ('uworld','test',1,10,'world',1);
insert into mysql_users (username,password,active,default_hostgroup,default_schema,transaction_persistent) values ('umyworld','test',1,10,'myworld',1);
insert into mysql_users (username,password,active,default_hostgroup,default_schema,transaction_persistent) values ('uallworld','test',1,10,'test',1);
 
LOAD MYSQL USERS TO RUNTIME;SAVE MYSQL USERS TO DISK;
 
delete from mysql_servers where hostgroup_id in (10,11,20,21);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,comment) VALUES ('192.168.1.107',10,3306,100,'master');
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,comment) VALUES ('192.168.1.109',11,3307,100,'slave');
 
LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;
 
delete from mysql_replication_hostgroups;
INSERT INTO mysql_replication_hostgroups VALUES (10,11,'world-myworld replication hgroup');
LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;
 
DO NOT RUN AT THE beginning this is for test 3 !!
delete from mysql_query_rules where rule_id in (10,11,12,13,14,15);
insert into mysql_query_rules (rule_id,username,schemaname,destination_hostgroup,active,retries,match_digest,apply)

  values(10,'uworld','world',10,1,3,'^SELECT.*FOR UPDATE',1);
insert into mysql_query_rules (rule_id,username,schemaname,destination_hostgroup,active,retries,match_digest,apply)

  values(11,'uworld','world',11,1,3,'^SELECT ',1);
LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK;
 
DO NOT RUN AT THE beginning this is for test 4 !!
delete from mysql_query_rules where rule_id in (10,11,12,13,14,15);
Let see what we need and how to do it:
1) user(s) uworld & umyworld need to go to their default schema on Master for Writes.
2) user(s) uworld & umyworld should go to their default schema on  master for direct reads
3) user(s) uworld & umyworld should go to the slave for reads when the other schema is used
 
To do this we will need the following rules:
insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply) values(10,'uworld',10,1,3,'^SELECT.*FOR UPDATE',1);
insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply,FlagOUT,FlagIN) values(11,'uworld',10,1,3,'^SELECT ',0,50,0); 
insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply,FlagOUT,FlagIN) values(12,'uworld',11,1,3,'myworld.',1,50,50);
 
 
insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply) values(13,'umyworld',10,1,3,'^SELECT.*FOR UPDATE',1);
insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply,FlagOUT,FlagIN) values(14,'umyworld',10,1,3,'^SELECT ',0,50,0); 
insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply,FlagOUT,FlagIN) values(15,'umyworld',11,1,3,'\sworld.',1,50,50);
 
LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK;
 
GRANTS
-------
grant all on *.* to uallworld@'%' identified by 'test';
grant all on world.* to uworld@'%' identified by 'test';     
grant all on myworld.* to umyworld@'%' identified by 'test';   
 
on the slave
REVOKE ALL ON *.*  FROM 'uworld'@'%';
REVOKE ALL ON *.*  FROM 'umyworld'@'%';
grant select on myworld.* to uworld@'%' identified by 'test';     
grant select on world.* to umyworld@'%' identified by 'test';
 
 
To monitor what is happening
---------------------------------
watch -n 1 'mysql -h 127.0.0.1 -P 6032 -uadmin -padmin -t -e "select * from stats_mysql_connection_pool where hostgroup

< 30 order by hostgroup,srv_host desc;" -e " select srv_host,command,avg(time_ms), count(ThreadID) from

stats_mysql_processlist group by srv_host,command;" -e "select * from stats_mysql_users;";

mysql -h 127.0.0.1 -P 6032 -uadmin -padmin -t -e "select * from stats_mysql_global "|egrep -i  "(mirror|memory|stmt)"'  
 
select active,hits,destination_hostgroup, mysql_query_rules.rule_id, match_digest, match_pattern,

   replace_pattern, cache_ttl, apply,flagIn,flagOUT FROM mysql_query_rules

   NATURAL JOIN stats.stats_mysql_query_rules where destination_hostgroup < 30  ORDER BY mysql_query_rules.rule_id; 
select * from stats_mysql_query_digest;
select * from stats_mysql_query_digest_reset;

Add new comment

How ProxySQL deal with schema (and schemaname) Long story

This article is the spin-off of the article How ProxySQL deal with schema (and schemaname)

1 How ProxySQL differes from MySQL in managing explicit default schema declaration? does it respect the -D or USE

MySQL and ProxySQL will behave the same when passing the default schema, setting it as default.

      MySQL
        mysql -uuallworld -ptest -h192.168.1.107 -P 3306 -D test
        select database();
        +------------+
        | database() |
        +------------+
        | test       |
        +------------+
 
        SHOW TABLES;
        +-----------------+
        | Tables_in_test  |
        +-----------------+
        | rewrite_tbtest  |
        | rewrite_tbtest1 |
        +-----------------+
 
      ProxySQL
        mysql -uuallworld -ptest -h127.0.0.1 -P 6033 -D test
        select database();
        +------------+
        | database() |
        +------------+
        | test       |
        +------------+
 
        SHOW TABLES;
        +-----------------+
        | Tables_in_test  |
        +-----------------+
        | rewrite_tbtest  |
        | rewrite_tbtest1 |
        +-----------------+

Passing a different schema from the default one in ProxySQL will override the ProxySQL default:

        MySQL
         ./mysql-3306 -uuallworld -ptest -D mysql
        (uallworld@localhost) [mysql]>select database();
        +------------+
        | database() |
        +------------+
        | mysql      |
        +------------+
        ProxySQL
        [root@rms2 server57S]# ./mysql-3306 -uuallworld -ptest -h192.168.1.50 -P6033 -D mysql
 
        (This email address is being protected from spambots. You need JavaScript enabled to view it.) [mysql]>select database();
        +------------+
        | database() |
        +------------+
        | mysql      |
        +------------+

Connecting without DEFAULT Schema

      MySQL 
        mysql -uuallworld -ptest -h192.168.1.107 -P 3306
        mysql> select database();
          +------------+
          | database() |
          +------------+
          | NULL       |
          +------------+
 
      ProxySQL 
        mysql -uuallworld -ptest -h127.0.0.1 -P 6033
        mysql> select database();
        +------------+
        | database() |
        +------------+
        | test       |
        +------------+

The last is different between MySQL and ProxySQL.

By default ProxySQL will connect the user to his default schema.

Not only, if the default schema is not define:

      +-----------+----------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+
      | username  | password | active | use_ssl | default_hostgroup | default_schema | schema_locked | transaction_persistent | fast_forward | backend | frontend | max_connections |
      +-----------+----------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+
      | uallworld | test     | 1      | 0       | 10                | NULL           | 0             | 1                      | 0            | 1       | 1        | 10000           |
 
      ProxySQL will point the connection to information_schema.
          mysql> select database();
          +--------------------+
          | database()         |
          +--------------------+
          | information_schema |
          +--------------------+

2 How proxy respect/follow security agains schema

      MySQL
      mysql -uuworld -ptest -h192.168.1.107 -P 3306 -D test
        mysql: [Warning] Using a password on the command line interface can be insecure.
        ERROR 1044 (42000): Access denied for user 'uworld'@'%' to database 'test'
 
      ProxySQL
      mysql -uuworld -ptest -h127.0.0.1 -P 6033 -D test
        mysql: [Warning] Using a password on the command line interface can be insecure.
        Welcome to the MySQL monitor.  Commands end with ; or \g.
        Your MySQL connection id is 483902
        Server version: 5.5.30 (ProxySQL)

And no warnings.

Why?

Because while you directly connect to MySQL, when you connect to ProxySQL you do not actually touch the final server.

You only open a connection to the ProxySQL, issue a query will open a connection and if the user do not have the right grants an error will be returned.

Let us see.

just issuing the command:

1
2
3

        mysql> select database();
        ERROR 1044 (42000): Access denied for user 'uworld'@'%' to database 'test'

As soon as you try to query the MySQL server you got an error.

3 How schemaname filter acts in the query rules?

In MySQL we can easily change the default schema with USE , this action is fully supported by ProxySQL.
But it may have some side effects when using "schemaname" as filter in the query_rules.
If you define a rule that include the default schemaname and the default schema is changed with USE, the rule will not apply, and unpredictable results may happen.
To prevent that ProxySQL has another option in mysql_user "schema_locked" which will prevent the schema to be changed by USE.
This feature is present but not fully implemented, and after a brief conversation with Rene (https://github.com/sysown/proxysql/issues/1133).
For now my advice is to DO NOT use schemaname as filter in the query_rules.

Below the full test case:

    mysql> select active,hits,destination_hostgroup, mysql_query_rules.rule_id, match_digest, match_pattern, replace_pattern, cache_ttl, apply,flagIn,flagOUT FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules where destination_hostgroup < 30  ORDER BY mysql_query_rules.rule_id;
      +--------+------+-----------------------+---------+---------------------+---------------+-----------------+-----------+-------+--------+---------+
      | active | hits | destination_hostgroup | rule_id | match_digest        | match_pattern | replace_pattern | cache_ttl | apply | flagIN | flagOUT |
      +--------+------+-----------------------+---------+---------------------+---------------+-----------------+-----------+-------+--------+---------+
      | 1      | 0    | 10                    | 10      | ^SELECT.*FOR UPDATE | NULL          | NULL            | NULL      | 1     | 0      | NULL    |
      | 1      | 0    | 11                    | 11      | ^SELECT             | NULL          | NULL            | NULL      | 1     | 0      | NULL    |
      +--------+------+-----------------------+---------+---------------------+---------------+-----------------+-----------+-------+--------+---------+
      2 rows in set (0.00 sec)
 
 
      (This email address is being protected from spambots. You need JavaScript enabled to view it.) [world]>Select * from world.City order by Population DESC limit 5 ;
      +------+-----------------+-------------+--------------+------------+
      | ID   | Name            | CountryCode | District     | Population |
      +------+-----------------+-------------+--------------+------------+
      | 1024 | Mumbai (Bombay) | IND         | Maharashtra  |   10500000 |
      | 2331 | Seoul           | KOR         | Seoul        |    9981619 |
      |  206 | São Paulo       | BRA         | São Paulo    |    9968485 |
      | 1890 | Shanghai        | CHN         | Shanghai     |    9696300 |
      |  939 | Jakarta         | IDN         | Jakarta Raya |    9604900 |
      +------+-----------------+-------------+--------------+------------+
      5 rows in set (0.01 sec)
 
 
 
      mysql> select active,hits,destination_hostgroup, mysql_query_rules.rule_id, match_digest, match_pattern, replace_pattern, cache_ttl, apply,flagIn,flagOUT FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules where destination_hostgroup < 30  ORDER BY mysql_query_rules.rule_id;
      +--------+------+-----------------------+---------+---------------------+---------------+-----------------+-----------+-------+--------+---------+
      | active | hits | destination_hostgroup | rule_id | match_digest        | match_pattern | replace_pattern | cache_ttl | apply | flagIN | flagOUT |
      +--------+------+-----------------------+---------+---------------------+---------------+-----------------+-----------+-------+--------+---------+
      | 1      | 0    | 10                    | 10      | ^SELECT.*FOR UPDATE | NULL          | NULL            | NULL      | 1     | 0      | NULL    |
      | 1      | 1    | 11                    | 11      | ^SELECT             | NULL          | NULL            | NULL      | 1     | 0      | NULL    | <---
      +--------+------+-----------------------+---------+---------------------+---------------+-----------------+-----------+-------+--------+---------+
      2 rows in set (0.00 sec)
 
      mysql> select * from stats_mysql_query_digest;
      +-----------+------------+----------+--------------------+-----------------------------------------------------------+------------+------------+------------+----------+----------+----------+
      | hostgroup | schemaname | username | digest             | digest_text                                               | count_star | first_seen | last_seen  | sum_time | min_time | max_time |
      +-----------+------------+----------+--------------------+-----------------------------------------------------------+------------+------------+------------+----------+----------+----------+
      | 10        | world      | uworld   | 0x594F2C744B698066 | select USER()                                             | 1          | 1502789864 | 1502789864 | 0        | 0        | 0        |
      | 10        | world      | umyworld | 0x134B0F1E837A0B8B | Select * from world.City order by Population DESC limit ? | 1          | 1502789849 | 1502789849 | 10264    | 10264    | 10264    |
      | 10        | world      | uworld   | 0x02033E45904D3DF0 | show databases                                            | 1          | 1502789863 | 1502789863 | 1750     | 1750     | 1750     |
      | 10        | world      | umyworld | 0x594F2C744B698066 | select USER()                                             | 1          | 1502789745 | 1502789745 | 0        | 0        | 0        |
      | 10        | world      | uworld   | 0x226CD90D52A2BA0B | select @@version_comment limit ?                          | 1          | 1502789863 | 1502789863 | 0        | 0        | 0        |
      | 10        | world      | uworld   | 0x99531AEFF718C501 | show tables                                               | 1          | 1502789863 | 1502789863 | 470      | 470      | 470      |
      | 10        | world      | umyworld | 0x226CD90D52A2BA0B | select @@version_comment limit ?                          | 1          | 1502789745 | 1502789745 | 0        | 0        | 0        |
      | 10        | world      | umyworld | 0x99531AEFF718C501 | show tables                                               | 1          | 1502789745 | 1502789745 | 7427     | 7427     | 7427     |
      | 11        | world      | uworld   | 0x134B0F1E837A0B8B | Select * from world.City order by Population DESC limit ? | 1          | 1502789866 | 1502789866 | 3349     | 3349     | 3349     | <---
      | 10        | world      | umyworld | 0x02033E45904D3DF0 | show databases                                            | 1          | 1502789745 | 1502789745 | 615      | 615      | 615      |
      +-----------+------------+----------+--------------------+-----------------------------------------------------------+------------+------------+------------+----------+----------+----------+
      10 rows in set (0.00 sec)
 
      (This email address is being protected from spambots. You need JavaScript enabled to view it.) [world]>USE information_schema;
      Reading table information for completion of table and column names
      You can turn off this feature to get a quicker startup with -A
 
      Database changed
      (This email address is being protected from spambots. You need JavaScript enabled to view it.) [information_schema]>Select * from world.City order by Population DESC limit 5 ;
      +------+-----------------+-------------+--------------+------------+
      | ID   | Name            | CountryCode | District     | Population |
      +------+-----------------+-------------+--------------+------------+
      | 1024 | Mumbai (Bombay) | IND         | Maharashtra  |   10500000 |
      | 2331 | Seoul           | KOR         | Seoul        |    9981619 |
      |  206 | São Paulo       | BRA         | São Paulo    |    9968485 |
      | 1890 | Shanghai        | CHN         | Shanghai     |    9696300 |
      |  939 | Jakarta         | IDN         | Jakarta Raya |    9604900 |
      +------+-----------------+-------------+--------------+------------+
      5 rows in set (0.01 sec)
 
 
 
 
      mysql> select * from stats_mysql_query_digest;
      +-----------+--------------------+----------+--------------------+-----------------------------------------------------------+------------+------------+------------+----------+----------+----------+
      | hostgroup | schemaname         | username | digest             | digest_text                                               | count_star | first_seen | last_seen  | sum_time | min_time | max_time |
      +-----------+--------------------+----------+--------------------+-----------------------------------------------------------+------------+------------+------------+----------+----------+----------+
      | 10        | information_schema | uworld   | 0x99531AEFF718C501 | show tables                                               | 1          | 1502789929 | 1502789929 | 540      | 540      | 540      |
      | 10        | information_schema | uworld   | 0x02033E45904D3DF0 | show databases                                            | 1          | 1502789929 | 1502789929 | 897      | 897      | 897      |
      | 11        | world              | uworld   | 0x620B328FE9D6D71A | SELECT DATABASE()                                         | 1          | 1502789929 | 1502789929 | 537      | 537      | 537      |
      | 10        | world              | uworld   | 0x594F2C744B698066 | select USER()                                             | 1          | 1502789864 | 1502789864 | 0        | 0        | 0        |
      | 10        | world              | umyworld | 0x134B0F1E837A0B8B | Select * from world.City order by Population DESC limit ? | 1          | 1502789849 | 1502789849 | 10264    | 10264    | 10264    |
      | 10        | world              | uworld   | 0x02033E45904D3DF0 | show databases                                            | 1          | 1502789863 | 1502789863 | 1750     | 1750     | 1750     |
      | 10        | world              | umyworld | 0x594F2C744B698066 | select USER()                                             | 1          | 1502789745 | 1502789745 | 0        | 0        | 0        |
      | 10        | world              | uworld   | 0x226CD90D52A2BA0B | select @@version_comment limit ?                          | 1          | 1502789863 | 1502789863 | 0        | 0        | 0        |
      | 10        | world              | uworld   | 0x99531AEFF718C501 | show tables                                               | 1          | 1502789863 | 1502789863 | 470      | 470      | 470      |
      | 10        | world              | umyworld | 0x226CD90D52A2BA0B | select @@version_comment limit ?                          | 1          | 1502789745 | 1502789745 | 0        | 0        | 0        |
      | 10        | world              | umyworld | 0x99531AEFF718C501 | show tables                                               | 1          | 1502789745 | 1502789745 | 7427     | 7427     | 7427     |
      | 10        | information_schema | uworld   | 0x134B0F1E837A0B8B | Select * from world.City order by Population DESC limit ? | 1          | 1502789933 | 1502789933 | 3707     | 3707     | 3707     | <---
      | 11        | world              | uworld   | 0x134B0F1E837A0B8B | Select * from world.City order by Population DESC limit ? | 1          | 1502789866 | 1502789866 | 3349     | 3349     | 3349     | <---
      | 10        | world              | umyworld | 0x02033E45904D3DF0 | show databases                                            | 1          | 1502789745 | 1502789745 | 615      | 615      | 615      |
      +-----------+--------------------+----------+--------------------+-----------------------------------------------------------+------------+------------+------------+----------+----------+----------+
      14 rows in set (0.01 sec)
 
      mysql> select active,hits,destination_hostgroup, mysql_query_rules.rule_id, match_digest, match_pattern, replace_pattern, cache_ttl, apply,flagIn,flagOUT FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules where destination_hostgroup < 30  ORDER BY mysql_query_rules.rule_id;
      +--------+------+-----------------------+---------+---------------------+---------------+-----------------+-----------+-------+--------+---------+
      | active | hits | destination_hostgroup | rule_id | match_digest        | match_pattern | replace_pattern | cache_ttl | apply | flagIN | flagOUT |
      +--------+------+-----------------------+---------+---------------------+---------------+-----------------+-----------+-------+--------+---------+
      | 1      | 0    | 10                    | 10      | ^SELECT.*FOR UPDATE | NULL          | NULL            | NULL      | 1     | 0      | NULL    |
      | 1      | 2    | 11                    | 11      | ^SELECT             | NULL          | NULL            | NULL      | 1     | 0      | NULL    |
      +--------+------+-----------------------+---------+---------------------+---------------+-----------------+-----------+-------+--------+---------+
      2 rows in set (0.00 sec)

4 how can I transparently redirect using schema name?

This is not a Schema feature, more one of the things that in ProxySQL are quite easy to set, while close to be impossible if done in plain MySQL.
When connecting directly with MySQL there is no option for you to "play" with GRANTS and schema such that you will transparently allow a user to do an action on a schema/server pair and another on a different schema/server pair.
When using ProxySQL to filter by schemaname is quite trivial.

For instance assuming we have 3 users one is admin of the platform which include 2 separate schemas (or more), each user can access one schema for write (but that can be table as well), and a final slave with reporting information, where all the users that needs to read from other schema except their own can read cross schemas.
While all the select not cross schema mus still got to the Master.
This is not so uncommon, actually with few variant is exactly what one of the customer I spoke last week needs to do.

Let see what we need and how to do it:

user(s) uworld & umyworld need to go to their default schema on Master for Writes.
user(s) uworld & umyworld should go to their default schema on master for direct reads
user(s) uworld & umyworld should go to the slave for reads when the other schema is used

To do this we will need the following rules:

        insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply) values(10,'uworld',10,1,3,'^SELECT.*FOR UPDATE',1);
        insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply,FlagOUT,FlagIN) values(11,'uworld',10,1,3,'^SELECT ',0,50,0); 
        insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply,FlagOUT,FlagIN) values(12,'uworld',11,1,3,'\smyworld.',1,50,50);
 
 
        insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply) values(13,'umyworld',10,1,3,'^SELECT.*FOR UPDATE',1);
        insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply,FlagOUT,FlagIN) values(14,'umyworld',10,1,3,'^SELECT ',0,50,0); 
        insert into mysql_query_rules (rule_id,username,destination_hostgroup,active,retries,match_digest,apply,FlagOUT,FlagIN) values(15,'umyworld',11,1,3,'\sworld.',1,50,50);

        LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK;

To check the behaviour:

        select database();
        update world.City set Population=10500000 where ID=1024;
        update world.Country set LifeExpectancy=62.5 where Code='IND';
        update myworld.CityM set Population=10500001 where ID=1024;
        update myworld.CountryM set LifeExpectancy=0 where Code='IND';
 
        select database();
        Select * from world.City order by Population DESC limit 5 ;
        Select * from myworld.CityM order by Population DESC limit 5 ;
 
        Select City.*, Country.name, Country.LifeExpectancy from world.City as City join world.Country as Country on City.CountryCode=Country.code order by Population DESC limit 5; 
        Select City.*, Country.name, Country.LifeExpectancy from myworld.CityM as City join myworld.CountryM as Country on City.CountryCode=Country.code order by Population DESC limit 5; 
 
        Select City.*, Country.name, Country.LifeExpectancy from world.City as City join myworld.CountryM as Country on City.CountryCode=Country.code order by Population DESC limit 5;

Once I run the test (queries above):

        mysql> select active,hits,destination_hostgroup, mysql_query_rules.rule_id, match_digest, match_pattern, replace_pattern, cache_ttl, apply,flagIn,flagOUT FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules where destination_hostgroup < 30  ORDER BY mysql_query_rules.rule_id;
        +--------+------+-----------------------+---------+---------------------+---------------+-----------------+-----------+-------+--------+---------+
        | active | hits | destination_hostgroup | rule_id | match_digest        | match_pattern | replace_pattern | cache_ttl | apply | flagIN | flagOUT |
        +--------+------+-----------------------+---------+---------------------+---------------+-----------------+-----------+-------+--------+---------+
        | 1      | 0    | 10                    | 10      | ^SELECT.*FOR UPDATE | NULL          | NULL            | NULL      | 1     | 0      | NULL    |
        | 1      | 5    | 10                    | 11      | ^SELECT             | NULL          | NULL            | NULL      | 0     | 0      | 50      | <-- 5 selects in total OK
        | 1      | 3    | 11                    | 12      | myworld.            | NULL          | NULL            | NULL      | 1     | 50     | 50      | <-- 3 contains term myworld OK
        | 1      | 0    | 10                    | 13      | ^SELECT.*FOR UPDATE | NULL          | NULL            | NULL      | 1     | 0      | NULL    |
        | 1      | 0    | 10                    | 14      | ^SELECT             | NULL          | NULL            | NULL      | 0     | 0      | 50      |
        | 1      | 0    | 11                    | 15      | world.              | NULL          | NULL            | NULL      | 1     | 50     | 50      |
        +--------+------+-----------------------+---------+---------------------+---------------+-----------------+-----------+-------+--------+---------+
        6 rows in set (0.00 sec)
 
 
 
        mysql> select * from stats_mysql_query_digest;                                                                                                                                         
        +-----------+------------+----------+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+------------+------------+----------+----------+----------+
        | hostgroup | schemaname | username | digest             | digest_text                                                                                                                                                                       | count_star | first_seen | last_seen  | sum_time | min_time | max_time |
        +-----------+------------+----------+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+------------+------------+----------+----------+----------+
        | 11        | world      | uworld   | 0x921512ADAF79D0FF | Select City.*, Country.name, Country.LifeExpectancy from world.City as City join myworld.CountryM as Country on City.CountryCode=Country.code order by Population DESC limit ?    | 1          | 1502718358 | 1502718358 | 6531     | 6531     | 6531     |
        | 11        | world      | uworld   | 0xE9D26001526F2618 | Select City.*, Country.name, Country.LifeExpectancy from myworld.CityM as City join myworld.CountryM as Country on City.CountryCode=Country.code order by Population DESC limit ? | 1          | 1502718358 | 1502718358 | 6573     | 6573     | 6573     |
        | 10        | world      | uworld   | 0xE846287B5A6B3945 | Select City.*, Country.name, Country.LifeExpectancy from world.City as City join world.Country as Country on City.CountryCode=Country.code order by Population DESC limit ?       | 1          | 1502718358 | 1502718358 | 3181     | 3181     | 3181     |
        | 11        | world      | uworld   | 0x55FFF888F4642D3A | Select * from myworld.CityM order by Population DESC limit ?                                                                                                                      | 1          | 1502718358 | 1502718358 | 7753     | 7753     | 7753     |
        | 10        | world      | uworld   | 0x26DB674419D1E979 | update myworld.CountryM set LifeExpectancy=? where Code=?                                                                                                                         | 1          | 1502718358 | 1502718358 | 257      | 257      | 257      |
        | 10        | world      | uworld   | 0x056615DE2CFD8C8E | update myworld.CityM set Population=? where ID=?                                                                                                                                  | 1          | 1502718358 | 1502718358 | 235      | 235      | 235      |
        | 10        | world      | uworld   | 0x134B0F1E837A0B8B | Select * from world.City order by Population DESC limit ?                                                                                                                         | 1          | 1502718358 | 1502718358 | 3262     | 3262     | 3262     |
        | 10        | world      | uworld   | 0x7A15CC342D54452D | update world.Country set LifeExpectancy=?.? where Code=?                                                                                                                          | 1          | 1502718358 | 1502718358 | 319      | 319      | 319      |
        | 10        | world      | uworld   | 0x500E6F01B02078B6 | update world.City set Population=? where ID=?                                                                                                                                     | 1          | 1502718358 | 1502718358 | 970      | 970      | 970      |
        +-----------+------------+----------+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+------------+------------+----------+----------+----------+

As we can see with this simple set of rules my uworld user was able to perform exactly as expected and able to access the schema from the designated HG.

All the selects with schema "myworld" were redirect to HG 11.

Results details:

        (This email address is being protected from spambots. You need JavaScript enabled to view it.) [(none)]>update world.City set Population=10500000 where ID=1024;
        Query OK, 1 row affected (0.01 sec)
        Rows matched: 1  Changed: 0  Warnings: 0
 
        (This email address is being protected from spambots. You need JavaScript enabled to view it.) [(none)]>update world.Country set LifeExpectancy=62.5 where Code='IND';
        Query OK, 1 row affected (0.00 sec)
        Rows matched: 1  Changed: 0  Warnings: 0
 
        (This email address is being protected from spambots. You need JavaScript enabled to view it.) [(none)]>update myworld.CityM set Population=10500001 where ID=1024;      <--- I am NOT managing this but I could if needed
        ERROR 1142 (42000): UPDATE command denied to user 'uworld'@'192.168.1.50' for table 'CityM'
        (This email address is being protected from spambots. You need JavaScript enabled to view it.) [(none)]>update myworld.CountryM set LifeExpectancy=0 where Code='IND';   <--- I am NOT managing this but I could if needed
        ERROR 1142 (42000): UPDATE command denied to user 'uworld'@'192.168.1.50' for table 'CountryM'
        (This email address is being protected from spambots. You need JavaScript enabled to view it.) [(none)]>
 
        (This email address is being protected from spambots. You need JavaScript enabled to view it.) [(none)]>Select * from world.City order by Population DESC limit 5 ;
        +------+-----------------+-------------+--------------+------------+
        | ID   | Name            | CountryCode | District     | Population |
        +------+-----------------+-------------+--------------+------------+
        | 1024 | Mumbai (Bombay) | IND         | Maharashtra  |   10500000 |
        | 2331 | Seoul           | KOR         | Seoul        |    9981619 |
        |  206 | São Paulo       | BRA         | São Paulo    |    9968485 |
        | 1890 | Shanghai        | CHN         | Shanghai     |    9696300 |
        |  939 | Jakarta         | IDN         | Jakarta Raya |    9604900 |
        +------+-----------------+-------------+--------------+------------+
        5 rows in set (0.00 sec)
 
        (This email address is being protected from spambots. You need JavaScript enabled to view it.) [(none)]>Select * from myworld.CityM order by Population DESC limit 5 ;
        +------+-----------------+-------------+--------------+------------+
        | ID   | Name            | CountryCode | District     | Population |
        +------+-----------------+-------------+--------------+------------+
        | 1024 | Mumbai (Bombay) | IND         | Maharashtra  |   10500001 |
        | 2331 | Seoul           | KOR         | Seoul        |    9981619 |
        |  206 | São Paulo       | BRA         | São Paulo    |    9968485 |
        | 1890 | Shanghai        | CHN         | Shanghai     |    9696300 |
        |  939 | Jakarta         | IDN         | Jakarta Raya |    9604900 |
        +------+-----------------+-------------+--------------+------------+
        5 rows in set (0.01 sec)
 
        (This email address is being protected from spambots. You need JavaScript enabled to view it.) [(none)]>
        (This email address is being protected from spambots. You need JavaScript enabled to view it.) [(none)]>Select City.*, Country.name, Country.LifeExpectancy from world.City as City join world.Country as Country on City.CountryCode=Country.code order by Population DESC limit 5; 
        +------+-----------------+-------------+--------------+------------+-------------+----------------+
        | ID   | Name            | CountryCode | District     | Population | name        | LifeExpectancy |
        +------+-----------------+-------------+--------------+------------+-------------+----------------+
        | 1024 | Mumbai (Bombay) | IND         | Maharashtra  |   10500000 | India       |           62.5 |
        | 2331 | Seoul           | KOR         | Seoul        |    9981619 | South Korea |           74.4 |
        |  206 | São Paulo       | BRA         | São Paulo    |    9968485 | Brazil      |           62.9 |
        | 1890 | Shanghai        | CHN         | Shanghai     |    9696300 | China       |           71.4 |
        |  939 | Jakarta         | IDN         | Jakarta Raya |    9604900 | Indonesia   |           68.0 |
        +------+-----------------+-------------+--------------+------------+-------------+----------------+
        5 rows in set (0.01 sec)
 
        (This email address is being protected from spambots. You need JavaScript enabled to view it.) [(none)]>Select City.*, Country.name, Country.LifeExpectancy from myworld.CityM as City join myworld.CountryM as Country on City.CountryCode=Country.code order by Population DESC limit 5; 
        +------+-----------------+-------------+--------------+------------+-------------+----------------+
        | ID   | Name            | CountryCode | District     | Population | name        | LifeExpectancy |
        +------+-----------------+-------------+--------------+------------+-------------+----------------+
        | 1024 | Mumbai (Bombay) | IND         | Maharashtra  |   10500001 | India       |            0.0 |
        | 2331 | Seoul           | KOR         | Seoul        |    9981619 | South Korea |           74.4 |
        |  206 | São Paulo       | BRA         | São Paulo    |    9968485 | Brazil      |           62.9 |
        | 1890 | Shanghai        | CHN         | Shanghai     |    9696300 | China       |           71.4 |
        |  939 | Jakarta         | IDN         | Jakarta Raya |    9604900 | Indonesia   |           68.0 |
        +------+-----------------+-------------+--------------+------------+-------------+----------------+
        5 rows in set (0.00 sec)
 
        (This email address is being protected from spambots. You need JavaScript enabled to view it.) [(none)]>
        (This email address is being protected from spambots. You need JavaScript enabled to view it.) [(none)]>Select City.*, Country.name, Country.LifeExpectancy from world.City as City join myworld.CountryM as Country on City.CountryCode=Country.code order by Population DESC limit 5; 
        +------+-----------------+-------------+--------------+------------+-------------+----------------+
        | ID   | Name            | CountryCode | District     | Population | name        | LifeExpectancy |
        +------+-----------------+-------------+--------------+------------+-------------+----------------+
        | 1024 | Mumbai (Bombay) | IND         | Maharashtra  |   10500000 | India       |            0.0 |
        | 2331 | Seoul           | KOR         | Seoul        |    9981619 | South Korea |           74.4 |
        |  206 | São Paulo       | BRA         | São Paulo    |    9968485 | Brazil      |           62.9 |
        | 1890 | Shanghai        | CHN         | Shanghai     |    9696300 | China       |           71.4 |
        |  939 | Jakarta         | IDN         | Jakarta Raya |    9604900 | Indonesia   |           68.0 |
        +------+-----------------+-------------+--------------+------------+-------------+----------------+
        5 rows in set (0.00 sec)

Obviosuly doing it on the other user/schema work exacty the same.

Back to main article

Add new comment

Sweet and sour can become bitter

Recently Fred published a post (http://lefred.be/content/mysql-group-replication-is-sweet-but-can-be-sour-if-you-misunderstand-it) in which he was stating, I had publish my blog (http://www.tusacentral.net/joomla/index.php/mysql-blogs/191-group-replication-sweet-a-sour.html) which contains few “misunderstanding”.

All the people that knows me, also know I have no problem to admit if I do a mistake, at the end doing mistake is a way of learning and be afraid of the ones who don’t do them.

But at the same time, I need to have proof of it. As I provide proof of what I wrote with numbers and tests (all available on github).

Let us put down the basis of the discussion with facts, not personal speculation or assertions.

Facts:

1) From MySQL official documentations (https://dev.mysql.com/doc/refman/5.7/en/group-replication-use-cases.html)

Group Replication enables you to create fault-tolerant systems with redundancy by replicating the system state throughout a set of servers. Consequently, even if some of the servers fail, as long it is not all or a majority, the system is still available, and all it could have degraded performance or scalability, it is still available. Server failures are isolated and independent. They are tracked by a group membership service which relies on a distributed failure detector that is able to signal when any servers leave the group, either voluntarily or due to an unexpected halt. There is a distributed recovery procedure to ensure that when servers join the group they are brought up to date automatically. There is no need for server fail-over, and the multi-master update everywhere nature ensures that not even updates are blocked in the event of a single server failure. Therefore MySQL Group Replication guarantees that the database service is continuously available.

2) Still from MySQL (https://dev.mysql.com/worklog/task/?id=9838) Group Replication introduced flow control to avoid excessive buffering and to maintain group members reasonably close to one another. For several reasons, it's more efficient to keep buffering low but, as mentioned before, it is not a requirement that members are kept in sync for replication to work: once a slave becomes delayed it will just increase the number of pending transactions in the relay log.

The flow control mechanism enters the scene to bound the back-log members can accumulate, in terms of transactions to certify and transactions to apply. Once those bounds are exceeded, and while they remain exceeded, the flow-control mechanism limits the throughput of writer members to adjust to the capacity of the slowest members of the group. By imposing conservative bounds it tries to make sure all members are able to keep-up and can return safely to within the bounds defined by the user.

So given the above, I, as “standard” user, read that as “when using MySQL and GR, and setting (eventually) the Flow Control correctly, I will achieve to have a data platform that is continuously available given the use of GR”.

Cool, right? So what I was doing in my tests? Well two main things, one is to see if I can use GR for scaling reads as I was doing (and hundreds of customers as well) with PXC, the other is to see if in case of crash of my Master, I can safely fail-over another slave.

This doesn’t mean I am comparing the way the two product works. I cannot care less at this stage, as I am sure most of the customer will not care. What they care is what they can do SAFELY, and with what. All the mambo-jambo about the technical details (how much sync or async I can be) is not their concern.

So the point was and is… Given I am used to product X can I move to product Y and if so how and what I should be aware of?

Once more I was trying to answer to the question “If GR and InnoDB cluster has to work as alternative to other (virtually) synchronous replication mechanism, what change or shift our customers must consider if they want move from one to the other.”

The outcome of the tests is pointing to answer that, period.

Let us clarify two things more:

I am perfectly aware (Fred talking to you) that GR use a different mechanism for FC, and that the numbers set in the group_replication_flow_control_certifier_threshold/ group_replication_flow_control_applier_threshold are then use to calculate the Quota. Still they are user threshold, and Specifies the number of waiting transactions in the applier/certifier queue that trigger flow control which are connected to the final statements reported above: By imposing conservative bounds it tries to make sure all members are able to keep-up and can return safely to within the bounds defined by the user.
Bound that as for the manual can go from 0 to 2147483648.
As such setting it to 25 (I did try also 1000 or more with even worse results) is perfectly legit and I have some issues in considering it a mistake.

I was measuring the lag with the only tools MySQL/Oracle had give us, in the article I said I had used:
“select @last_rec:=SUBSTRING_INDEX(SUBSTRING_INDEX(SUBSTRING_INDEX( Received_transaction_set,':',-2),':',1),'-',-1) last_received FROM performance_schema.replication_connection_status WHERE Channel_name = 'group_replication_applier'; select (@last_rec - @last_exec) as real_lag"

Which use the only information available at the moment regarding the incoming and the current executed transactions.
Fred says that is not valid because the certification and This means that a transaction listed on last_received_transaction_set is certified and queued to apply on this member. On the other members it may already be certified/applied or soon be.
I agree that it may not be perfect .. and I should have said: is for sure the last apply or soon to be given the certification on the master a node can know about.
But here we are not talking of 1 or 10 entries, but in most cases lag of hundreds or thousands entries. So also if this is not perfect and I can miss in a way or another a couple or entries because still processing the certification, I still think it is a valid way to evaluate the lag given the dimensions we are talking about.

BTW if this is not … well please help us and provide the right way to calculate the REAL lag between each node in a GR cluster and the writing master, with the most accurate precision.

About the comment of the node dimension, well thanks and yes you are right here I forgot to put the details.

The four nodes are VMS on separated hosts, so the gr1/2 where hosted on Host1 while gr3/4 hosted on the another host. Nothing else running on the hosts while test was running. Applications and monitor where running on a 3 host. The exact dimension is not really relevant given all of them were the same.

I want to say that I can setup the perfect environment for GR or PXC or NDB or whatever you are testing and showing how cool that is.

But I think we should consider that real life is far to be like that, and that we must see/test is how a product is behaving in adverse conditions, or if not adverse challenging.

Given all the above, in my opinion the tests clearly shown that also if the Flow Control mechanism in GR is the coolest thing ever conceptualize, at the end it does not what it is suppose to, no matter what.

Was I setting the thresholds wrong? Well not sure about that, given the results I got.

If I have 4 nodes and 2 of them (so … no majority) are lagging behind of hundreds or even thousands of entries, while my threshold is in the order of hundreds or less, this means that the cool mechanism is not doing his job, period.

One think is write down whatever about this or that but another is doing tests and provide numbers and evidences, not just words.

As said in a comment (not published in the Fred blog) I am more than happy to perform more tests and do them in conjunctions with anyone else.

Then if I am wrong I am wrong… but until I have different evidence, just saying a car can fly because it has very nice and colorful wings, doesn’t make it fly.

chitty-1