Overview

The machine has a single NIC, Storage attach by fibre channel, 8CPU hyper threading, 64GB RAM and finally heavy usage of NFS.

1
2
3
4
5
6
7
8
9

CURRENT TCP buffer setting 
------------------------------
net.ipv4.tcp_mtu_probing = 0
net.core.rmem_max = 131071
net.core.wmem_max = 131071
net.ipv4.tcp_rmem = 4096    87380    4194304
net.ipv4.tcp_wmem = 4096    16384    4194304
------------------------------
 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

Suggested TCP buffer settings
------------------------------
#TCP max buffer size
net.core.rmem_max = 16777216 
net.core.wmem_max = 16777216 
 
#Linux auto-tuning TCP buffer 
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
 
#length of the processor input queue
net.core.netdev_max_backlog = 30000
 
#default congestion control is htcp 
net.ipv4.tcp_congestion_control=htcp
 

1
2
3
4
5
6
7

MySQL change
-----------------------------------------------
back_log                        = 1024
thread_cache_size               = 512
thread_stack                    = 256K
wait_timeout                    = 600
 

Conclusion

Reference

{autotoc enabled=yes}

Introduction

Recently I had some free spare time that I used to read, think, process and analyze few ideas and work on my own projects.

That was great for me because I had the chance to develop new tools and to review few concept related to work. So I had the chance to focus on the ideas behind the procedures or "how-to", including reviewing what I am doing at work, from different angles and prospective.

One of the different or better to say modified prospective, was the outcome of a mental process started with a reading.

Reading that I initially considered a waste of money, time and mind effort.

This because the topic discussed, and the way the topic was presented, is something that I had the chance to study when I was in school starting from secondary. In fact the topic of "Critical thinking" is, or I should say was, included in the school programs in our learning path associate to "Logic", "Grammar" and "Philosophy".

So when I read the books I commit the crime of assumptiveness, feeling also bore while reading, until the moment the book was covering the topic related to "Decision Making". In that chapter the writer was underlining how easy is for us to be caught in trap by our own knowledge and ideas.

I did stop to read, close the book went doing something else, and then I try to empty my mind. Only at that point I realize I was not reading at all, better my eyes were, some part of my brain was, but my mind and my attention were not, this because I had categorize the book from the initial chapters in the erroneous way.

So I take a glass of wine, had some time, good music and open the book again from the start. This time the book was presenting me a different scenario and prospective, it drives my thought through several mind paths and brings me to review some assumptions. At this point I was able to make some parallels with what is our/mine day-to-day activity in life and work. It was funny for me to discover how some personal best practices, fit perfectly in a well categorize universal model.

There was no magic, that is true, but what was good and interesting, was how "School Training" can forge your way of doing in an instinctive way, but also how the instinctive action path, can be transform and express in few, clear, universal and simple to read steps.

In particular I saw a good parallel with two critical areas of our work; the credibility of the source and the process of the decision-making.

The rest of the writing is a summary, a go through few points I have identify as relevant, and that I have see covering some critical grey areas.

I am aware this is just a small part of the picture, and as usual I am open to discussion and comments, and I will be more then happy if that will happen, actually this will mean that I have reach my target.

Credibility of the Source

In our work, as well as in many others, having a good credibility is not a plus but a must. Being credible as a company or as a single person, is not coming from free, and is a process that could takes years to build and days to get destroy.

The credibility is not only the result of "best practices" or "how to", but is also the result of a correct approach and process in what we do, how we do it and how we decide to do things (see after the decision-making section).

Whenever a customer will come to us, for an advice or help, he will ask to himself some questions, questions that we should answer or the customer will redirect his attention to someone else.

Those questions are:

• Do they (us) have the relevant expertise (experience, knowledge and if needed formal qualification)?
• Do they have the ability to observe accurately (eyesight, hearing, proximity to event, absence of distraction, appropriate instruments, skills in using instrument)?
• Does their reputation suggest that they are reliable?
• Do they have any interest or possible bias?
• Are them claiming and providing evidences of knowledge about MY context?
• Are they providing direct expertise?
• Is they level of expertise base on direct experience?
• Is what they say support by evidence and logic pattern?
• Are other sources consistent?

Answer to all the above, as said, is not something you can achieve with limited or superficial effort, it instead require an extensive and constant shift in mentality, and require some well define ideas and behaviour. My interpretation is the following:

• Always be "super partes", also avoid as much as possible to follow ephemeral trends, like the use and abuse of the "magic" term of the year, often used by others to show they capacity to be on the "trend". Unfortunately be there very often means doing without knowing. Be more conservative and analytic is the right things to do when responsible of other people.
• Be under constant training and education, perform extensive tests, and provide public evidence of our conclusion and analysis. Publish few but focus blogs.
• Avoid blog about everything, and avoid generalization, that will create more noise and confusion, yes you will be there, but as chatter not as an expert.
• When claiming about something, provide evidence and a well-documented reasoning path to support your claim.
• Always put the claim in a clear defined context, and if possible and available include the references to others reasoning and/or similar evidence and sources.
• Whenever possible try to be or use a direct source, like provide the test you have done yourself, or review and repeat the tests done by others to validate them.
• Never use other source material as yours, instead document them and contextualize them providing credit to the source. Again double check other source conclusion and provide evidence of your process.
• Whatever evidence or conclusion you will provide, it needs to have an exact match with the discussed topic, avoid generalization. Assumption can be good only if supported by good and documented reasoning.
• Do not rush, this is not a race, do not send out an answer or a comment without having the time to think on it. If possible, review it several times, and cover your reasoning also with others, this to be sure you have cover all the possible areas of uncertainly, and if you still see them, declare them.

I will be more then happy to have discussion on the above points, and if possible to extend them including more helpful suggestions.

Decision-making process

As mention previously the other point is related on how we take our own decisions, and how we evaluate other people conclusion/reasoning/motivation.

In our work we are constantly call to take decisions, some of them are very simple ones, and we can take action with very limited thinking, but others could be much more complex and could require significant effort from our side, more time and processing to efficiently evaluate what will be the right decision.

Unfortunately very often we are affected by at least one of the following bad behaviour:

• We do not give us enough time to think.
• We see a possible fit on a though and we remain there not giving us the space to evolve.
• We do not process all the possible alternatives to/of the problem, and we do not develop more then one solution.
• We do not evolve our solution/action into a clear path of possible consequences.
• We often skip to take more information because the "time" issue (or others) and miss relevant part of the puzzle.
• We sometime forget what is relevant for us and how much this can impact on our judgment.
• We are emotionally involved and it affects the process and the decisions.
• We just do what our Boss is saying to do.
• Other recommendations influence us without applying analytical thinking.

Going through the above points, trying to clarify and to see what we can do to prevent them.

• Time, time is relevant and often we have to take some decision fast, but thinking require time, time to take information, time to analyze them, time for the reasoning. The process should not be compromise by our rush, because results will be affected and our decisions can be imprecise (if lucky) or completely wrong, not only it could happen to take a wrong decision, but when this happen because rush, we do not have a good reasoning to support and justify our mistake, in short there will not be a learning lesson, only the mistake.
• How often have we feel in love with our ideas, and not ready to divorce from them? Too often we must admit it, instead we should be able to go beyond and process all the possible options. We should keep our mind open and listen to other external suggestion but always applying analytical process.
• When I was a kid I learn that "each action include/imply a reaction". Before performing any action, before apply what we think is correct in our decision, we should carefully think "What will happen next?". We should analyze the actions, and have a good level of understanding of what will be the path of events that our actions will generate, and be ready for possible unexpected bad behaviour.
• In our job information about what is going on is everything. We should never stop to dig more, and get better understanding. Never consider the outcome of some tools/script enough for our analysis, taking their results as given without applying an analytical review. We should stop only when we are really confident that we cannot get more relevant information, and if possible we should ask to a trusted source to compare what we got, to see if we have miss anything.
• Sometime we forget that we have personal commitments, those could affect our judgment. For example, if we are fully focus on open source, it could become almost automatic for us to skip the evaluation of a non-open source solution. Or if we are Linux fundamentalist just to have to approach windows server, could drive us to have a not objective approach to the problem. Again we must keep our mind open and process the problem by analytical steps, not considering the preconceptions in our thinking, but be able to filter them out and have an objective mind process.
• How many times we have found that customer so annoying? His reiterate questions where lacking of any sense, and is not some time his behaviour to be so close to be offensive? On the other hand this other customer is really nice, he gives you a lot of credit, he has a good understanding of the effort you are doing to keep his environment in good shape. Can you honestly say that you have always gives to the two the same "time and attention"? This is a fact, it is in the human nature, to be more careful and nice comes easier with the ones that are nice with us. But this is not correct, we should always apply the same time/effort/reasoning independently to the customer behaviour. The reasoning is the point not our feeling. Understanding it and be able to mange it is a matter of be more or less professional.
• Do not follow the boss or others advice, direction blindly. We must listen carefully to anyone, we should evaluate what they have to say and objectively extract whatever is good from their suggestion or recommendation. But never accept it without our own thinking/reasoning; also it will be appropriate to share with them our process step by step, before getting to the conclusion. This will help us in learning from each other work, and will provide advantage to everyone also reducing the chance of mistakes.

Summarizing we should ask ourselves the following before, during and after having done our reasoning for a decision:

1. What make this decision necessary? What is the objective?
2. What I am going to recommend, and on what basis?
3. What other possible alternatives exists, which one is the more realistic feasible, which one the more innovative?
4. What are the possible consequence of my decision, and how likely they are going to happen?
5. If this consequence will happen what will be the relevance and how we can manage them?
6. Comparing different solution, which one will be the best to mitigate negative effect?
7. How I can transform my decision into an action reducing to minimum the risk of bad behaviour or mistakes?

Conclusion

In the above sections, I was just trying to report in a concise and easy way, what is part of a more complex topic. I am aware that most of us do the right thing, just doing it right, but I am also confident that reporting black on white those simple points could help us to avoid mistakes, and if possible to define process and checklists that other people less conscientious then us, can follow to make their work behaviour more trustable.

Reference

Ennis , R.H Critical thinking Prentice Hall 1996

Fisher A The logic of real arguments Cambridge University press 1988

Fisher A Critical Thinking – an Introduction Cambridge University press 2001

{joscommentenable}

or summary  for lazy guys on how to use it...

{autotoc enabled=yes}

I know that a lot has being written around Xtrabackup, and good documentation can be found on the Percona web site.

1
2
3
4
5

CREATE USER 'backup'@'localhost' IDENTIFIED BY 'bckuser123';
REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'backup'@'localhost';
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'backup'@'localhost';
FLUSH PRIVILEGES;
 

root@mysqlt3:/home/mysql/backup/2012_12_21_1300/2012-12-21_14-32-02# ll
total 200088
drwxr-xr-x 15 root root      4096 Dec 21 14:41 ./
drwxr-xr-x  3 root root      4096 Dec 21 14:46 ../
-rw-r--r--  1 root root       263 Dec 21 14:32 backup-my.cnf
-rw-r-----  1 root root 104857600 Dec 21 14:32 ibdata1
drwxr-xr-x  2 root root      4096 Dec 21 14:41 mysql/
drwxr-xr-x  2 root root      4096 Dec 21 14:41 performance_schema/
drwx------  2 root root      4096 Dec 21 14:41 security/
drwx------  2 root root      4096 Dec 21 14:41 test/
drwx------  2 root root      4096 Dec 21 14:41 test_audit/
drwx------  2 root root      4096 Dec 21 14:41 timstaging/
drwx------  2 root root      4096 Dec 21 14:41 timtags/
drwxr-xr-x  2 root root      4096 Dec 21 14:41 world/
-rw-r--r--  1 root root        13 Dec 21 14:41 xtrabackup_binary
-rw-r--r--  1 root root        26 Dec 21 14:41 xtrabackup_binlog_info
-rw-r-----  1 root root        85 Dec 21 14:41 xtrabackup_checkpoints
-rw-r-----  1 root root  99912192 Dec 21 14:41 xtrabackup_logfile
-rw-r--r--  1 root root        53 Dec 21 14:41 xtrabackup_slave_info
backup-my.cnf <--------------- very essential version of the my.cnf with innodb information
ibdata1<---------------------- Main tablespace
mysql/ <----------------------
world/<----------------------- DBs ... with files copy in
xtrabackup_binary <----------- contains the name of the xtrabackup binary used
xtrabackup_binlog_info <------ Binary log information (name/position)
xtrabackup_checkpoints <------ Information regarding the LSN position and range
xtrabackup_logfile <---------- File containing the delta of the modifications
xtrabackup_slave_info <------- Slave information (if slave)
 

121221 15:57:04  InnoDB: Waiting for the background threads to start
121221 15:57:05 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 30312932364

InnoDB: Last MySQL binlog file position 0 213145807, file name /home/mysql/instances/mtest1/binlog.000011
xtrabackup: starting shutdown with innodb_fast_shutdown = 1
121221 15:57:05  InnoDB: Starting shutdown...
121221 15:57:09  InnoDB: Shutdown completed; log sequence number 30312932364
121221 15:57:09  innobackupex: completed OK!

innobackupex: Starting to copy InnoDB system tablespace
innobackupex: in '/home/mysql/backup/2012_12_21_1300/2012-12-21_14-32-02'
innobackupex: back to original InnoDB data directory '/home/mysql/instances/mtest1/data'
innobackupex: Copying '/home/mysql/backup/2012_12_21_1300/2012-12-21_14-32-02/ibdata1' to '/home/mysql/instances/mtest1/data/ibdata1'
innobackupex: Starting to copy InnoDB log files
innobackupex: in '/home/mysql/backup/2012_12_21_1300/2012-12-21_14-32-02'
innobackupex: back to original InnoDB log directory '/home/mysql/logs/mtest1/innodblog'
innobackupex: Finished copying back files.
121221 16:41:38  innobackupex: completed OK!

121221 16:44:08 mysqld_safe Starting mysqld daemon with databases from /home/mysql/instances/mtest1/data
121221 16:44:09 [Note] Plugin 'FEDERATED' is disabled.
121221 16:44:09 InnoDB: The InnoDB memory heap is disabled
121221 16:44:09 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
121221 16:44:09 InnoDB: Compressed tables use zlib 1.2.3
121221 16:44:09 InnoDB: Using Linux native AIO
121221 16:44:09 InnoDB: Initializing buffer pool, size = 1.0G
121221 16:44:09 InnoDB: Completed initialization of buffer pool
121221 16:44:09  InnoDB: Log file /home/mysql/logs/mtest1/innodblog/ib_logfile0 did not exist: new to be created
InnoDB: Setting log file /home/mysql/logs/mtest1/innodblog/ib_logfile0 size to 100 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Progress in MB: 100
...
121221 16:44:15 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
121221 16:44:15  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Last MySQL binlog file position 0 213145807, file name /home/mysql/instances/mtest1/binlog.000011
121221 16:44:17  InnoDB: Waiting for the background threads to start
21221 16:44:18 InnoDB: 1.1.8 started; log sequence number 30312933388
121221 16:44:18 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3310
121221 16:44:18 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
121221 16:44:18 [Note] Server socket created on IP: '0.0.0.0'.
121221 16:44:18 [Note] Event Scheduler: Loaded 0 events
121221 16:44:18 [Note] /home/mysql/templates/mysql-55p/bin/mysqld: ready for connections.
Version: '5.5.27-log'  socket: '/home/mysql/instances/mtest1/mysql.sock'  port: 3310  MySQL Community Server (GPL)

1
2
3
4
5
6
7
8
9

+--------------+--------+--------+----------+----------+-----------+----------+
| TABLE_SCHEMA | ENGINE | TABLES | ROWS     | DATA (M) | INDEX (M) | TOTAL(M) |
+--------------+--------+--------+----------+----------+-----------+----------+
| test         | InnoDB |     51 |  9023205 |  5843.14 |   1314.62 |  7157.76 |
| test         | NULL   |     51 |  9023205 |  5843.14 |   1314.62 |  7157.76 |
| test_audit   | InnoDB |      9 |  1211381 |   658.54 |    230.54 |   889.09 |
| test_audit   | NULL   |      9 |  1211381 |   658.54 |    230.54 |   889.09 |
| NULL         | NULL   |     61 | 10234586 |  6501.68 |   1545.17 |  8046.86 |
+--------------+--------+--------+----------+----------+-----------+----------+

7 rows in set (6.92 sec)

#pigz --version
pigz 2.1.6

drwxr-xr-x  3 root root 4.0K Dec 21 17:08 ./
drwxr-xr-x  3 root root 4.0K Dec 21 14:16 ../
drwxr-xr-x 15 root root 4.0K Dec 21 16:31 2012-12-21_14-32-02/
-rw-r--r--  1 root root 737M Dec 21 17:18 full_mtest1.tar.gz <-------------

121221 17:09:52  innobackupex-1.5.1: Starting mysql with options:  --defaults-file='/home/mysql/instances/mtest1/my.cnf' --password=xxxxxxxx --user='backup' --unbuffered --
121221 17:18:29  innobackupex-1.5.1: completed OK!

-rw-r--r-- 1 root root       269 Dec 21 17:10 backup-my.cnf
-rw-rw---- 1 root root 104857600 Dec 21 17:12 ibdata1
drwxr-xr-x 2 root root      4096 Dec 21 17:40 mysql
drwxr-xr-x 2 root root      4096 Dec 21 17:39 performance_schema
drwxr-xr-x 2 root root      4096 Dec 21 17:39 security
drwxr-xr-x 2 root root      4096 Dec 21 17:39 test
drwxr-xr-x 2 root root      4096 Dec 21 17:39 test_audit
drwxr-xr-x 2 root root      4096 Dec 21 17:39 timstaging
drwxr-xr-x 2 root root      4096 Dec 21 17:39 timtags
drwxr-xr-x 2 root root      4096 Dec 21 17:39 world
-rw-r--r-- 1 root root        13 Dec 21 17:18 xtrabackup_binary
-rw-r--r-- 1 root root        26 Dec 21 17:18 xtrabackup_binlog_info
-rw-rw---- 1 root root        85 Dec 21 17:18 xtrabackup_checkpoints
-rw-rw---- 1 root root 282056704 Dec 21 17:18 xtrabackup_logfile
-rw-r--r-- 1 root root        53 Dec 21 17:18 xtrabackup_slave_info

121221 17:52:17  InnoDB: Starting shutdown...
121221 17:52:21  InnoDB: Shutdown completed; log sequence number 30595333132
121221 17:52:21  innobackupex: completed OK!
 

121221 18:04:48  innobackupex: completed OK!

121221 18:06:38 mysqld_safe Starting mysqld daemon with databases from /home/mysql/instances/mtestslave/data
121221 18:06:39 [Note] Plugin 'FEDERATED' is disabled.
121221 18:06:39 InnoDB: The InnoDB memory heap is disabled
121221 18:06:39 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
121221 18:06:39 InnoDB: Compressed tables use zlib 1.2.3
121221 18:06:39 InnoDB: Using Linux native AIO
121221 18:06:39 InnoDB: Initializing buffer pool, size = 1.0G
121221 18:06:39 InnoDB: Completed initialization of buffer pool
121221 18:06:39  InnoDB: Log file /home/mysql/logs/mtestslave/innodblog/ib_logfile0 did not exist: new to be created
InnoDB: Setting log file /home/mysql/logs/mtestslave/innodblog/ib_logfile0 size to 10 MB
InnoDB: Database physically writes the file full: wait...
121221 18:06:40 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
121221 18:06:40  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Last MySQL binlog file position 0 150497896, file name /home/mysql/instances/mtest1/binlog.000001
121221 18:06:42  InnoDB: Waiting for the background threads to start
121221 18:06:43 InnoDB: 1.1.8 started; log sequence number 30595333644
121221 18:06:43 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3311
121221 18:06:43 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
121221 18:06:43 [Note] Server socket created on IP: '0.0.0.0'.
121221 18:06:43 [Note] Event Scheduler: Loaded 0 events
121221 18:06:43 [Note] /home/mysql/templates/mysql-55p/bin/mysqld: ready for connections.
Version: '5.5.27-log'  socket: '/home/mysql/instances/mtestslave/mysql.sock'  port: 3311  MySQL Community Server (GPL)
 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

root@localhost [(none)]> show schemas;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| security           |
| test               |
| test_audit         |
| world              |
+--------------------+
13 rows in set (0.04 sec)
root@localhost [(none)]> SELECT TABLE_SCHEMA, ENGINE, COUNT(1) as 'TABLES', sum(TABLE_ROWS) as 'ROWS',
TRUNCATE(sum(DATA_LENGTH)/pow(1024,2),2) as 'DATA (M)',
TRUNCATE(sum(INDEX_LENGTH)/pow(1024,2),2) as 'INDEX (M)',
TRUNCATE((sum(DATA_LENGTH)+sum(INDEX_LENGTH))/pow(1024,2),2) AS 'TOTAL(M)'
FROM information_schema.tables       WHERE TABLE_SCHEMA <> 'information_schema'
AND TABLE_SCHEMA <> 'mysql'     AND TABLE_SCHEMA not like 'avail%'
AND TABLE_SCHEMA <> 'maatkit'            AND TABLE_TYPE = 'BASE TABLE'
GROUP BY TABLE_SCHEMA, ENGINE      WITH ROLLUP;
+--------------------+--------------------+--------+----------+----------+-----------+----------+
| TABLE_SCHEMA       | ENGINE             | TABLES | ROWS     | DATA (M) | INDEX (M) | TOTAL(M) |
+--------------------+--------------------+--------+----------+----------+-----------+----------+
| performance_schema | PERFORMANCE_SCHEMA |     17 |    23014 |     0.00 |      0.00 |     0.00 |
| performance_schema | NULL               |     17 |    23014 |     0.00 |      0.00 |     0.00 |
| security           | InnoDB             |      1 |  1454967 |   170.73 |     60.75 |   231.48 |
| security           | NULL               |      1 |  1454967 |   170.73 |     60.75 |   231.48 |
| test               | InnoDB             |     51 |  9298913 |  6058.39 |   1347.78 |  7406.17 |
| test               | NULL               |     51 |  9298913 |  6058.39 |   1347.78 |  7406.17 |
| test_audit         | InnoDB             |      9 |  1189343 |   685.56 |    236.56 |   922.12 |
| test_audit         | NULL               |      9 |  1189343 |   685.56 |    236.56 |   922.12 |
| world              | MyISAM             |      3 |     5302 |     0.35 |      0.06 |     0.42 |
| world              | NULL               |      3 |     5302 |     0.35 |      0.06 |     0.42 |
| NULL               | NULL               |    227 | 11971539 |  6916.70 |   1645.74 |  8562.44 |
+--------------------+--------------------+--------+----------+----------+-----------+----------+
 

1
2
3

root@localhost [(none)]> SHOW slave STATUS\G
Empty SET (0.00 sec)
root@localhost [(none)]>

1
2

cat xtrabackup_binlog_info
binlog.000001    150497896

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

root@localhost [(none)]> SHOW slave STATUS\G
*************************** 1. row ***************************
Slave_IO_State:
Master_Host: 192.168.0.3
Master_User: replica
Master_Port: 3310
Connect_Retry: 60
Master_Log_File: binlog.000001
Read_Master_Log_Pos: 150497896
Relay_Log_File: mysql-relay-bin.000001
Relay_Log_Pos: 4
Relay_Master_Log_File: binlog.000001
Slave_IO_Running: No
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 150497896
Relay_Log_Space: 107
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 0
1 row IN SET (0.00 sec)
root@localhost [(none)]>

Perfect start the slave:

slave start;

AND CHECK again:

root@localhost [(none)]> SHOW slave STATUS\G
*************************** 1. row ***************************
Slave_IO_State: Waiting FOR master TO send event
Master_Host: 192.168.0.3
Master_User: replica
Master_Port: 3310
Connect_Retry: 60
Master_Log_File: binlog.000001
Read_Master_Log_Pos: 206843593
Relay_Log_File: mysql-relay-bin.000002
Relay_Log_Pos: 22872
Relay_Master_Log_File: binlog.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 150520518
Relay_Log_Space: 56346103
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 30
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 3310
1 row IN SET (0.00 sec)

root@tusacentral03:/home/mysql/backup/2013-01-10_13-07-24# cat xtrabackup_checkpoints
backup_type = full-backuped
from_lsn = 0
to_lsn = 32473279827
last_lsn = 32473279827
Last LSN is 32473279827

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

root@localhost [test]> SHOW processlist;
+-----+--------+---------------------------+------+---------+------+--------+------------------------------------------------------------------------------------------------------+
| Id  | User   | Host                      | db   | Command | Time | State  | Info                                                                                                 |
+-----+--------+---------------------------+------+---------+------+--------+------------------------------------------------------------------------------------------------------+
|  87 | root   | localhost                 | test | Query   |    0 | NULL   | SHOW processlist                                                                                     |
|  92 | stress | tusacentral01.LOCAL:37293 | test | Sleep   |    0 |        | NULL                                                                                                 |
|  94 | stress | tusacentral01.LOCAL:37296 | test | Query   |    0 | UPDATE | INSERT INTO tbtest30 (uuid,a,b,c,counter,partitionid,strrecordtype) VALUES(UUID(),731188002,"hd rsg  |
|  95 | root   | localhost:37295           | test | Query   |    0 | update | INSERT INTO test_audit.tbtest4 values(NEW.autoInc,NEW.a,NEW.uuid,NEW.b,NEW.c,NEW.counter,NEW.time,NE |
|  96 | stress | tusacentral01.local:37298 | test | Query   |    0 | NULL   | COMMIT                                                                                               |
|  97 | root   | localhost:37299           | test | Query   |    0 | update | INSERT INTO test_audit.tbtest4 values(NEW.autoInc,NEW.a,NEW.uuid,NEW.b,NEW.c,NEW.counter,NEW.time,NE |
|  98 | stress | tusacentral01.local:37300 | test | Query   |    0 | update | insert INTO tbtest15 (uuid,a,b,c,counter,partitionid,strrecordtype) VALUES(UUID(),598854171,"usfcrgl |
|  99 | root   | localhost:37301           | test | Query   |    0 | UPDATE | INSERT INTO test_audit.tbtest4 VALUES(NEW.autoInc,NEW.a,NEW.uuid,NEW.b,NEW.c,NEW.counter,NEW.time,NE |
| 100 | stress | tusacentral01.LOCAL:37302 | test | Query   |    0 | UPDATE | INSERT INTO tbtest15 (uuid,a,b,c,counter,partitionid,strrecordtype) VALUES(UUID(),22723485,"vno ehhr |
| 101 | stress | tusacentral01.local:37303 | test | Query   |    0 | update | insert INTO tbtest1 (uuid,a,b,c,counter,partitionid,strrecordtype) VALUES(UUID(),991063177,"nqdcogeu |
| 102 | stress | tusacentral01.LOCAL:37304 | test | Query   |    0 | UPDATE | INSERT INTO tbtest1 (uuid,a,b,c,counter,partitionid,strrecordtype) VALUES(UUID(),86481207,"sdfabnogn |
| 103 | stress | tusacentral01.local:37305 | test | Query   |    0 | NULL   | COMMIT                                                                                               |
+-----+--------+---------------------------+------+---------+------+--------+------------------------------------------------------------------------------------------------------+
12 rows in set (0.00 sec)

total 20
drwxr-xr-x  5 root  root  4096 Jan 10 13:30 ./
drwxr-xr-x 18 mysql mysql 4096 Dec 28 12:16 ../
drwxr-xr-x 15 root  root  4096 Jan 10 13:17 2013-01-10_13-07-24/
drwxr-xr-x 15 root  root  4096 Jan 10 13:34 2013-01-10_13-30-43/ <-------- the last one is the Incremental
 

root@tusacentral03:/home/mysql/backup/2013-01-10_13-30-43# du -sh *
4.0K    backup-my.cnf
4.5M    ibdata1.delta
4.0K    ibdata1.meta
1.5M    mysql
212K    performance_schema
18M        security <---------------------------------
1.2G    test <---------------------------------
173M    test_audit <---------------------------
488K    world
4.0K    xtrabackup_binary
4.0K    xtrabackup_binlog_info
4.0K    xtrabackup_checkpoints
4.0K    xtrabackup_logfile
4.0K    xtrabackup_slave_info
 

root@tusacentral03:/home/mysql/backup/2013-01-10_13-30-43/test# cat tbtest1.ibd.meta
page_size = 16384
zip_size = 0
space_id = 1983

root@tusacentral03:/home/mysql/backup/2013-01-10_13-30-43# cat xtrabackup_checkpoints
backup_type = incremental
from_lsn = 32473279827 <------------ starting point
to_lsn = 33215076229   <------------ End point
last_lsn = 33215076229

root@tusacentral03:/opt/percona-xtrabackup-2.0.4/bin# /opt/percona-xtrabackup-2.0.4/bin/innobackupex-1.5.1 --incremental \
--incremental-basedir=/home/mysql/backup/2013-01-10_13-30-43/ \
--defaults-file=/home/mysql/instances/mtest1/my.cnf  \
--slave-info --user=backup --password=bckuser123   /home/mysql/backup/

drwxr-xr-x 15 root  root  4096 Jan 10 13:17 2013-01-10_13-07-24/
drwxr-xr-x 15 root  root  4096 Jan 10 13:34 2013-01-10_13-30-43/ <--------- First incremental
drwxr-xr-x 15 root  root  4096 Jan 10 14:02 2013-01-10_13-57-04/ <--------- Second incremental

root@tusacentral03:/home/mysql/restore/2013-01-10_13-07-24# cat xtrabackup_binlog_info
binlog.000005    275195253     <------------ Original from Base
root@tusacentral03:/home/mysql/restore/2013-01-10_13-07-24# cat ../../backup/2013-01-10_13-07-24/xtrabackup_binlog_info
binlog.000003    322056528     <------------ Up to date from incremental
 

[root@tusacentral07 data]# scp -R This email address is being protected from spambots. You need JavaScript enabled to view it..0.3:/home/mysql/restore/2013-01-10_13-07-24/*  .
[root@tusacentral07 data]# sudo chown -R mysql:mysql .
 

grep last_lsn xtrabackup_checkpoints|awk -F' = ' '{print $2}'
34925032837

/opt/percona-xtrabackup-2.0.4/bin/innobackupex-1.5.1 --incremental \
--incremental-lsn=34925032837 \
--defaults-file=/home/mysql/instances/mtest1/my.cnf \
--slave-info --user=backup --password=bckuser123 \
--extra-lsndir=/home/mysql/backup/ \
--stream=xbstream 
--parallel=4 ./ |pigz -p4 - > /home/mysql/backup/incremental_2013_01_11_11_35.gz

root@tusacentral03:/home/mysql/backup# ll
total 631952
drwxr-xr-x  3 root  root       4096 Jan 11 11:36 ./
drwxr-xr-x 19 mysql mysql      4096 Jan 10 15:01 ../
drwxr-xr-x 15 root  root       4096 Jan 10 17:27 full_2013_01_10_18_54.gz
-rw-r--r--  1 root  root  360874358 Jan 11 11:25 incremental_2013_01_10_19_05.gz
-rw-r--r--  1 root  root  286216063 Jan 11 11:41 incremental_2013_01_11_11_35.gz
-rw-r--r--  1 root  root         93 Jan 11 11:41 xtrabackup_checkpoints
 

[] Check binary version

[] Check binaries are accessible in the PATH and accessible

[] Assign correct user/password in MySQL for backup user

[] Create or check backup data destination folder

[] Check my.cnf for datadir and be sure is pointing in the right place

[] Execute backup

[] Apply logs

Simple restore

[] be sure mysql server is down

[] remove / move data from original directory

[] remove / move ib_logs from original directory

[] run innobackupex --copy-back

[] check file permissions for mysql

[] start mysql

[] check the mysql log for error

[] log in and check for data.

Backup with Stream and compression

[] Check binary version

[] Check binaries are accessible in the PATH and accessible

[] Assign correct user/password in MySQL for backup user

[] Create or check backup data destination folder

[] Check my.cnf for datadir and be sure is pointing in the right place

[] Check for Pigz presence and version

[] Execute backup

Restore from Stream on a different machines (slave)

[] be sure mysql server is down

[] remove / move data from original directory

[] remove / move ib_logs from original directory

[] copy over the compress file

[] expand the backup in a safe directory

[] run innobackupex --copy-back

[] check file permissions for mysql

[] check that server WILL NOT restart the slave process on start

[] start mysql

[] check the mysql log for error

[] log in and check for data.

[] take the master log position

[] check for slave process information

[] apply new binary log position

[] restart slave

[] check slave status

Incremental Backup with Stream and compression

[] Check binary version

[] Check binaries are accessible in the PATH and accessible

[] Assign correct user/password in MySQL for backup user

[] Create or check backup data destination folder

[] Check my.cnf for datadir and be sure is pointing in the right place

[] Check for Pigz presence and version

[] Check for LSN file postion in xtrabackup_checkpoints

[] Assign the LSN to the "incremental-lsn" parameter

[] Be sure that the --extra-lsndir parametr is present and pointing to an existing directory

[] Execute backup

Incremental Restore from Stream on a different machines or slave

[] be sure mysql server is down

[] remove / move data from original directory

[] remove / move ib_logs from original directory

[] copy over the compress files

[] validate the chronological order from the BASE to the last increment

{loop for each file set}

[] expand the backup in a safe directory one a time

[] be sure that you apply log with "--apply-log --redo-only" parameters every time

[] be sure you always have the correct destination directory set (BASE set)

[] remove the incremental once apply

{loop end}

[] run innobackupex --apply-log on the BASE set

[] remove IB_log files

[] copy files to destination directory

[] check file permissions for mysql

[] check that server WILL NOT restart the slave process on start

[] start mysql

[] check the mysql log for error

[] log in and check for data.

[] take the master log position

[] check for slave process information

[] apply new binary log position

[] restart slave

[] check slave status

{joscommentenable}

Overview

In the last few months, we have seen more and more discussion on the use of Galera solution for MySQL Clustering.

I have being one of those that had heavily test and implement Galera solution, actually with quite good results and I have also presented SOME of them at Oracle Connect.

On the other side I have be work with MySQL NDB for years (at least from 2007) at many customers site, from simple to complex setups.

So also if I cannot consider myself as mega expert, I think to have some good experience and insight on both platform.

The point here is that I was not happy in reading some articles comparing the two, not because the kind of tests, or results.

Not because I prefer this or that, but simply because, from my point of view, it does not make any sense to compare the two.

We can spend pages and pages in discussing the point here, but I want try to give a simple generalize idea of WHY it makes no sense in few lines.

NDB brief list

• NDB is not a simple storage engine and can work independently, MySQL is “just “ a client.
• NDB is mainly an in memory database and also if it support table on disk the cost of them not always make sense.
• NDB is fully synchronous, no action can be returned to client until transactions on all nodes are really accepted.
• NDB use horizontal partition to equally distribute data cross node, but none of them has the whole dataset (unless you use one node group only, which happens ONLY when you don’t know how to use it).
• NDB Replicate data for a specific factor, which is the number of replica, and that replication factor will not change with the increase of the nodes number.
• Clients retrieve data from NDB as whole, but internally data is retrieve by node, often use parallel execution. (Not going in the details here on the difference between select methods like match by key, range, IN option and so on).
• NDB scale by node group that means it really scale in the Dataset dimension it can manage and operation it could execute, and it really scale!

Galera brief list

• Galera is an additional layer working inside the MySQL context.
• Galera require InnoDB to work.
• Galera offer “virtually synchronous” replication.
• Galera replicate the full dataset across ALL nodes.
• Galera data replication overhead, increase with the number of nodes present in the cluster.
• Galera replicate data from one node to cluster on the commit, but apply them on each node by a FIFO queue (multi thread).
• Galera do not offers any parallelism between the nodes when retrieving the data; clients rely on the single node they access.

So why they cannot be compare?

It should be quite clear that the two, are very different from starting from the main conceptualization, given NDB is a cluster of many node groups with distribute dataset, while Galera is a very efficient (highly efficient) replication layer.

But just to avoid confusion:

1. NDB does data partition and data distribution with redundant factor.
2. Galera just replicate data all over.
3. NDB apply parallel execution to the incoming request, involving more node groups in data fetch.
4. Galera is not involved at all in the data fetch and clients need to connect to one node or more by themselves, means application require managing parallel request in case of need.
5. In NDB the more node groups you add the more you get in possible operation per second and data archived/retrieved. 
6. In Galera the more nodes you add, the more overhead you generate in the replication, so more data will require to be “locally” commit by the replication layer, until when the number of nodes and operations executed on them will compromise the performance for each node.

Conclusion

NDB Cluster is a real cluster solution, design to internally scale and to perform internally all the required operation to guarantee high availability and synchronous data distribution.

Galera is a very efficient solution to bypass the current inefficient mechanism MySQL has for the replication.

Galera allow to create a cluster of MySQL nodes, in virtually synchronous replication. This with almost zero complexity added on the standard MySQL management.

Never the less the obtaining platform is composed by separate nodes, which for the good or the bad is not a system of distributed data.

Given that, the scenario where we can use Galera or NDB are dramatically different, trying to compare them is like comparing a surfboard with a snowboard.

I love them both, and honestly I am expecting to see Galera deployment to dramatically increase in 2013, but I am still respecting my motto “use the right tool for the job”.

Let us try to make our life easier and avoid confusions.

Happy MySQL to all!!

Ho-ho-ho

{joscommentenable}