mysqld_multi: How to run multiple instances of MySQL

The need to have multiple instances of MySQL (the well-known mysqld process) running in the same server concurrently in a transparent way, instead of having them executed in separate containers/virtual machines, is not very common. Yet from time to time the Percona Support team receives a request from a customer to assist in the configuration of such an environment. MySQL provides a tool to facilitate the execution of multiple instances called mysqld_multi:

“mysqld_multi is designed to manage several mysqld processes that listen for connections on different Unix socket files and TCP/IP ports. It can start or stop servers, or report their current status.”

For tests and development purposes, MySQL Sandbox might be more practical and I personally prefer to use it for my own tests. Both tools work around launching and managing multiple mysqld processes but Sandbox has, as the name suggests, a “sandbox” approach, making it easy to both create and dispose of a new instance (including all data inside it). It is more usual to see mysqld_multi being used in production servers: It’s provided with the server package and uses the same single configuration file that people are used to looking for when setting up MySQL. So, how does it work? How do we configure and manage the instances? And as importantly, how do we backup all the instances we create?

Understanding the concept of groups in my.cnf
You may have noticed already that MySQL’s main configuration file (or “option file“), my.cnf, is arranged under what is called group structures: Sections defining configuration options specific to a given program or purpose. Usually, the program itself gives a name to the group, which appears enclosed by brackets. Here’s a basic my.cnf showing three such groups:

Shell
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
user = john
password = p455w0rd

[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
datadir = /var/lib/mysql

[xtrabackup]
target_dir = /backups/mysql/
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
user = john
password = p455w0rd

[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
datadir = /var/lib/mysql

[xtrabackup]
target_dir = /backups/mysql/
The options defined in the group [client] above are used by the mysql command-line tool. As such, if you don’t specify any other option when executing mysql it will attempt to connect to the local MySQL server through the socket in /var/run/mysqld/mysqld.sock and using the credentials stated in that group. Similarly, mysqld will look for the options defined under its section at startup, and the same happens with Percona XtraBackup when you run a backup with that tool. However, the operating parameters defined by the above groups may also be stated as command-line options during the execution of the program, in which case they replace the ones defined in my.cnf.

Getting started with multiple instances of MySQL
To have multiple instances of MySQL running we must replace the [mysqld] group in my.cnf configuration file by as many [mysqlN] groups as we want instances running, with “N” being a positive integer, also called option group number. This number is used by mysqld_multi to identify each instance, so it must be unique across the server. Apart from the distinct group name, the same options that are valid for [mysqld] applies on [mysqldN] groups, the difference being that while stating them is optional for [mysqld] (it’s possible to start MySQL with an empty my.cnf as default values are used if not explicitly provided) some of them (like socket, port, pid-file, and datadir) are mandatory when defining multiple instances – so they don’t step on each other’s feet. Here’s a simple modified my.cnf showing the original [mysqld] group plus two other instances:

Shell
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
datadir = /var/lib/mysql

[mysqld1]
user = mysql
pid-file = /var/run/mysqld/mysqld1.pid
socket = /var/run/mysqld/mysqld1.sock
port = 3307
datadir = /data/mysql/mysql1

[mysqld7]
user = mysql
pid-file = /var/run/mysqld/mysqld7.pid
socket = /var/run/mysqld/mysqld7.sock
port = 3308
datadir = /data/mysql/mysql7
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
datadir = /var/lib/mysql

[mysqld1]
user = mysql
pid-file = /var/run/mysqld/mysqld1.pid
socket = /var/run/mysqld/mysqld1.sock
port = 3307
datadir = /data/mysql/mysql1

[mysqld7]
user = mysql
pid-file = /var/run/mysqld/mysqld7.pid
socket = /var/run/mysqld/mysqld7.sock
port = 3308
datadir = /data/mysql/mysql7
Besides using different pid files, ports and sockets for the new instances I’ve also defined a different datadir for each – it’s very important that the instances do not share the same datadir. Chances are you’re importing the data from a backup but if that’s not the case you can simply use mysql_install_db to create each additional datadir (but make sure the parent directory exists and that the mysql user has write access on it):

Shell
mysql_install_db --user=mysql --datadir=/data/mysql/mysql7
1
mysql_install_db --user=mysql --datadir=/data/mysql/mysql7
Note that if /data/mysql/mysql7 doesn’t exist and you start this instance anyway then myqld_multi will call mysqld_install_db itself to have the datadir created and the system tables installed inside it. Alternatively, from restoring a backup or having a new datadir created you can make a physical copy of the existing one from the main instance – just make sure to stop it first with a clean shutdown, so any pending changes are flushed to disk first.

Now, you may have noted I wrote above that you need to replace your original MySQL instance group ([mysqld]) by one with an option group number ([mysqlN]). That’s not entirely true, as they can co-exist in harmony. However, the usual start/stop script used to manage MySQL won’t work with the additional instances, nor mysqld_multi really manages [mysqld]. The simple solution here is to have the group [mysqld] renamed with a suffix integer, say [mysqld0] (you don’t need to make any changes to its current options though), and let mysqld_multi manage all instances.

Two commands you might find useful when configuring multiple instances are:

Shell
$ mysqld_multi --example
1
$ mysqld_multi --example
…which provides an example of a my.cnf file configured with multiple instances and showing the use of different options, and:

Shell
$ my_print_defaults --defaults-file=/etc/my.cnf mysqld7
1
$ my_print_defaults --defaults-file=/etc/my.cnf mysqld7
…which shows how a given group (“mysqld7” in the example above) was defined within my.cnf.

Managing multiple instances
mysqld_multi allows you to start, stop, reload (which is effectively a restart) and report the current status of a given instance, all instances or a subset of them. The most important observation here is that the “stop” action is managed through mysqladmin – and internally that happens on an individual basis, with one “mysqladmin … stop” call per instance, even if you have mysqld_multi stop all of them. For this to work properly you need to setup a MySQL account with the SHUTDOWN privilege and defined with the same user name and password in all instances. Yes, it will work out of the box if you run mysqld_multi as root in a freshly installed server where the root user can access MySQL passwordless in all instances. But as the manual suggests, it’s better to have a specific account created for this purpose:

MySQL
mysql> GRANT SHUTDOWN ON . TO ‘multi_admin’@‘localhost’ IDENTIFIED BY ‘multipass’;
mysql> FLUSH PRIVILEGES;
1
2
mysql> GRANT SHUTDOWN ON . TO ‘multi_admin’@‘localhost’ IDENTIFIED BY ‘multipass’;
mysql> FLUSH PRIVILEGES;
If you plan on replicating the datadir of the main server across your other instances you can have that account created before you make copies of it, otherwise you just need to connect to each instance and create a similar account (remember, the privileged account is only needed by mysqld_multi to stop the instances, not to start them). There’s a special group that can be used on my.cnf to define options for mysqld_multi, which should be used to store these credentials. You might also indicate in there the path for the mysqladmin and mysqld (or mysqld_safe) binaries to use, though you might have a specific mysqld binary defined for each instance inside its respective group. Here’s one example:

Shell
[mysqld_multi]
mysqld = /usr/bin/mysqld_safe
mysqladmin = /usr/bin/mysqladmin
user = multi_admin
password = multipass
1
2
3
4
5
[mysqld_multi]
mysqld = /usr/bin/mysqld_safe
mysqladmin = /usr/bin/mysqladmin
user = multi_admin
password = multipass
You can use mysqld_multi to start, stop, restart or report the status of a particular instance, all instances or a subset of them. Here are a few examples that speak for themselves:

Shell
$ mysqld_multi report
Reporting MySQL (Percona Server) servers
MySQL (Percona Server) from group: mysqld0 is not running
MySQL (Percona Server) from group: mysqld1 is not running
MySQL (Percona Server) from group: mysqld7 is not running

$ mysqld_multi start

$ mysqld_multi report
Reporting MySQL (Percona Server) servers
MySQL (Percona Server) from group: mysqld0 is running
MySQL (Percona Server) from group: mysqld1 is running
MySQL (Percona Server) from group: mysqld7 is running

$ mysqld_multi stop 7,0

$ mysqld_multi report 7
Reporting MySQL (Percona Server) servers
MySQL (Percona Server) from group: mysqld7 is not running

$ mysqld_multi report
Reporting MySQL (Percona Server) servers
MySQL (Percona Server) from group: mysqld0 is not running
MySQL (Percona Server) from group: mysqld1 is running
MySQL (Percona Server) from group: mysqld7 is not running
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ mysqld_multi report
Reporting MySQL (Percona Server) servers
MySQL (Percona Server) from group: mysqld0 is not running
MySQL (Percona Server) from group: mysqld1 is not running
MySQL (Percona Server) from group: mysqld7 is not running

$ mysqld_multi start

$ mysqld_multi report
Reporting MySQL (Percona Server) servers
MySQL (Percona Server) from group: mysqld0 is running
MySQL (Percona Server) from group: mysqld1 is running
MySQL (Percona Server) from group: mysqld7 is running

$ mysqld_multi stop 7,0

$ mysqld_multi report 7
Reporting MySQL (Percona Server) servers
MySQL (Percona Server) from group: mysqld7 is not running

$ mysqld_multi report
Reporting MySQL (Percona Server) servers
MySQL (Percona Server) from group: mysqld0 is not running
MySQL (Percona Server) from group: mysqld1 is running
MySQL (Percona Server) from group: mysqld7 is not running
Managing the MySQL daemon
What is missing here is an init script to automate the start/stop of all instances upon server initialization/shutdown; now that we use mysqld_multi to control the instances, the usual /etc/init.d/mysql won’t work anymore. But a similar startup script (though much simpler and less robust) relying on mysqld_multi is provided alongside MySQL/Percona Server, which can be found in /usr/share/<mysql|percona-server>/mysqld_multi.server. You can simply copy it over as /etc/init.d/mysql, effectively replacing the original script while maintaining its name. Please note: You may need to edit it first and modify the first two lines defining “basedir” and “bindir” as this script was not designed to find out the good working values for these variables itself, which the original single-instance /etc/init.d/mysql does. Considering you probably have mysqld_multi installed in /usr/bin, setting these variables as follows is enough:

Shell
basedir=/usr
bindir=/usr/bin
1
2
basedir=/usr
bindir=/usr/bin
Configuring an instance with a different version of MySQL
If you’re planning to have multiple instances of MySQL running concurrently chances are you want to use a mix of different versions for each of them, such as during a development cycle to test application compatibility. This is a common use for mysqld_multi, and simple enough to achieve. To showcase its use I downloaded the latest version of MySQL 5.6 available and extracted the TAR file in /opt:

Shell
$ tar -zxvf mysql-5.6.20-linux-glibc2.5-x86_64.tar.gz -C /opt
1
$ tar -zxvf mysql-5.6.20-linux-glibc2.5-x86_64.tar.gz -C /opt
Then I made a cold copy of the datadir from one of the existing instances to /data/mysql/mysqld574:

Shell
$ mysqld_multi stop 0
$ cp -r /data/mysql/mysql1 /data/mysql/mysql5620
$ chown mysql:mysql -R /data/mysql/mysql5620
1
2
3
$ mysqld_multi stop 0
$ cp -r /data/mysql/mysql1 /data/mysql/mysql5620
$ chown mysql:mysql -R /data/mysql/mysql5620
and added a new group to my.cnf as follows:

Shell
[mysqld5620]
user = mysql
pid-file = /var/run/mysqld/mysqld5620.pid
socket = /var/run/mysqld/mysqld5620.sock
port = 3309
datadir = /data/mysql/mysql5620
basedir = /opt/mysql-5.6.20-linux-glibc2.5-x86_64
mysqld = /opt/mysql-5.6.20-linux-glibc2.5-x86_64/bin/mysqld_safe
1
2
3
4
5
6
7
8
[mysqld5620]
user = mysql
pid-file = /var/run/mysqld/mysqld5620.pid
socket = /var/run/mysqld/mysqld5620.sock
port = 3309
datadir = /data/mysql/mysql5620
basedir = /opt/mysql-5.6.20-linux-glibc2.5-x86_64
mysqld = /opt/mysql-5.6.20-linux-glibc2.5-x86_64/bin/mysqld_safe
Note the use of basedir, pointing to the path were the binaries for MySQL 5.6.20 were extracted, as well as a specific mysqld to be used with this instance. If you have made a copy of the datadir from an instance running a previous version of MySQL/Percona Server you will need to consider the same approach use when upgrading and run mysql_upgrade.

• I did try to use the latest experimental release of MySQL 5.7 (mysql-5.7.4-m14-linux-glibc2.5-x86_64.tar.gz) but it crashed with:

Shell
*** glibc detected *** bin/mysqld: double free or corruption (!prev): 0x0000000003627650 ***
1
*** glibc detected *** bin/mysqld: double free or corruption (!prev): 0x0000000003627650 ***
Using the conventional tools to start and stop an instance
Even though mysqld_multi makes things easier to control in general let’s not forget it is a wrapper; you can still rely (though not always, as shown below) on the conventional tools directly to start and stop an instance: mysqld* and mysqladmin. Just make sure to use the parameter –defaults-group-suffix to identify which instance you want to start:

Shell
mysqld --defaults-group-suffix=5620
1
mysqld --defaults-group-suffix=5620
and –socket to indicate the one you want to stop:

Shell
$mysqladmin -S /var/run/mysqld/mysqld5620.sock shutdown
1
$mysqladmin -S /var/run/mysqld/mysqld5620.sock shutdown

• However, mysqld won’t work to start an instance if you have redefined the option ‘mysqld’ on the configuration group, as I did for [mysqld5620] above, stating:

Shell
[ERROR] mysqld: unknown variable ‘mysqld=/opt/mysql-5.6.20-linux-glibc2.5-x86_64/bin/mysqld_safe’
1
[ERROR] mysqld: unknown variable ‘mysqld=/opt/mysql-5.6.20-linux-glibc2.5-x86_64/bin/mysqld_safe’
I’ve tested using “ledir” to indicate the path to the directory containing the binaries for MySQL 5.6.20 instead of “mysqld” but it also failed with a similar error. If nothing else, that shows you need to stick with mysqld_multi when starting instances in a mixed-version environment.

Backups
The backup of multiple instances must be done on an individual basis like you would if each instance was located in a different server. You just need to provide the appropriate parameters to identify the instance you’re targeting. For example, we can simply use socket with mysqldump when running it locally:

Shell
$ mysqldump --socket=/var/run/mysqld/mysqld7.sock --all-databases > mysqld7.sql
1
$ mysqldump --socket=/var/run/mysqld/mysqld7.sock --all-databases > mysqld7.sql
In Percona XtraBackup there’s an option named –defaults-group that should be used in environments running multiple instances to indicate which one you want to backup :

Shell
$ innobackupex --defaults-file=/etc/my.cnf --defaults-group=mysqld7 --socket=/var/run/mysqld/mysqld7.sock /root/Backup/
1
$ innobackupex --defaults-file=/etc/my.cnf --defaults-group=mysqld7 --socket=/var/run/mysqld/mysqld7.sock /root/Backup/
Yes, you also need to provide a path to the socket (when running the command locally), even though that information is already available in “–defaults-group=mysqld7”; as it turns out, only the Percona XtraBackup tool (which is called by innobackupex during the backup process) makes use of the information available in the group option. You may need to provide credentials as well (“–user” & “–password”), and don’t forget you’ll need to prepare the backup afterward. The option “defaults-group” is not available in all versions of Percona XtraBackup so make sure to use the latest one.

Summary
Running multiple instances of MySQL concurrently in the same server transparently and without any contextualization or a virtualization layer is possible with both mysqld_multi and MySQL Sandbox. We have been using the later at Percona Support to quickly spin on new disposable instances (though you might as easily keep them running indefinitely). In this post though, I’ve looked at mysqld_multi which is provided with MySQL server and remains the official solution for providing an environment with multiple instances.

The key aspect when configuring multiple instances in my.cnf is the notion of group name option, as you replace a single [mysqld] section by as many [mysqldN] sections as you want instances running. It’s important though to pay attention to certain details when defining the options for each one of these groups, especially when mixing instances from different MySQL/Percona Server versions. Different from MySQL Sandbox, where each instance relies on its own configuration file, you should be careful each time you edit the shared my.cnf file as a syntax error when configuring a single group option will prevent all instances from starting upon the server’s (re)initialization.

I hope to have covered the major points about mysqld_multi here but feel free to leave us a note below if you have something else to add or any comment to contribute.