Common Issues With MMM

This articles covers common issues and recommended solutions that come up in a MMM environment. This applies to our Linux Cloud and Dedicated solutions. You can find more information about these solutions at http://www.hosting.com/services/platform-services

Need to performance maintenance on a node

If you need to perform maintenance on a database node in  the MMM environment, you will want to take it offline via the following command: mmm_control set_offline host

This will remove all roles and stop replication on the node.

Once maintenance on the node is complete, run this command: mmm_control set_online host

You should not have application servers read from this node until mmm_control checks host shows OK for the rep_backlog test. This will ensure database reads return up to date data.

Replication Stops Working

You can check the status of replication by running the following command: mmm_control checks all

If any of the four checks performed fails, the role of that node will failover to another (E.g., it's reader IP and writer IPs will be moved).

Typical caues of this are the following:

1. The node is not pingable by the monitoring server (Maybe it's OFFLINE, or their is a firewall issue)
2. The node's MySQL server is not running or is not reachable by the monitoring server (MySQL Server may have crashed, been taken offline, or their is a firewall issue)
3. Replication threads on the database server are no longer running
4. The backlog is too big on the database node. It will automatically be taken OFFLINE to catch up

Issue 4 will typically fix itself. Issues 1 through 3 require manual intervention to resolve typically (Unless you performed a restart of the node and expect it to be back online without human intervention).

The server was restarted and mmm_agentd and mmm_mond aren't running

This is typically verified by running the following command: chkconfig --list | grep mmm

Run level 3 needs to be set to On for the service to start on reboots. mysql-mmm-monitor should have Run level 3 set to On on the monitoring server. mysql-mmm-agent shhould have Run level 3 set to On on the database nodes (Master and Read Only MySQL servers).

To start each service up, you can do the following: