froscon2010 - PREVIEW

Free and Open Source Software Conference

Istvan Podor
Tag Day 1 - 2010-08-21
Raum HS6
Beginn 17:45
Dauer 01:00
ID 536
Veranstaltungstyp Vortrag
Track OpenSQLCamp
Sprache der Veranstaltung englisch

Keep your MySQL backend online no matter what

How to build a multilevel mysql backend to stay online at all the time with MySQL, LVS and MMM

I helped architect and build a system to get 4 nines (99.99) availability for the mysql backend behind a website. It used mysql replication to set up a multi-master environment, MMM to manage it and LVS for mysql backend loadbalancing. This architecture not only provided high availability, but also permitted us to change the schema, upgrade MySQL versions, or even replace hardware without any downtime. This presentation explains how this solution worked.

One of the sites where this solution was deployed was a credit card processing interface for an alexa top100 site. The other one is a famous alexa top500 page ( with huge traffic. At first I would like to give a short overview of how these systems built up, mention the traffic and the goals that each system had. The payment site needed to be as reliable as possible with the highest possible uptime. The data consistency was very important, but downtime can't be tolerated below the 4 nines border. I will cover the scenarios we had to keep the data consistent on failure. On the other site at ustream, 10k visitors were online at all times, every single minute. During the peaks, more than 700k visitors were online at the same time. But the greatest challenge was when some unique live event happened, such as Michael Jackson's funeral, when the smallest mistake would make disappointed visitors who would look for another broadcaster immediately. I'll also discuss the following:

  • The requirements of each service mentioned in the title
  • The limitations of mysql
  • The required configuration of LVS
  • Issues around MMM
  • Monitoring and failover management and how we ensured data consistency
  • The requirements on the application side
  • What support the DBAs require from the developers To attend, you should know the mysql replication, routing and tcp basics and some experience with web based applications would be useful also. This talk won't mention installation of software or configuration of services. It will focus on the principles, rules, and main goals of this solution, and finally the workarounds it requires.