Prepare Your Databases for High Traffic on Black Friday

It’s November, so we all know what that means; it’s peak shopping season, and no date is bigger than Black Friday. But how will your database handle all that new, relentless traffic? Not only does your database have to handle traffic without slowing down, but web servers can sometimes see such sudden traffic as an attack; meaning your site(s) could go down completely.

Every year there are news stories about websites that were unresponsive or disappeared completely right at the exact moment when potential shoppers were coming online. Don’t let this be you! To ensure your databases are prepared for a high traffic event, download our Database Disaster Prevention Checklist and read what we advise you do before, during, and after the big day. Let’s get started.

What You Need To Know About High Traffic Events

1. Your users expect instant response and immediate feedback from your application. If you don’t give it to them, your competition will.
2. Database slowdowns for you can be perceived as downtime for your customers, and they will lose confidence in your ability.
3. A few seconds of downtime costs you not only direct revenue but also lost future business. Use our Cost of Database Downtime Calculator to see just how much downtime could cost you.
4. When you failover a slow system, the slowness follows to the new system. If your slowness alleviates, it may be because your customers went somewhere else when you were down.
5. Time is finite; you can’t get more of it. You have to make the most of it. It can be much more valuable than money.

Before the Event

1. Setup monitoring and tooling before the event. If you can’t see what’s going on, how can you measure your success? Percona Monitoring and Management can help with that.
2. Load test your applications and test how you will scale under normal and peak loads. The best time to find bottlenecks is before the event, as during is costly and impactful.
3. Test failover and understand how quickly you can recover. The worst time to find out your failover doesn’t work is during the busiest day of the year.
4. Put a code and configuration freeze in place in advance of the event. It’s really hard to ensure performance if the application is growing and evolving.
5. Get a second opinion, double-check, and don’t assume. A large portion of your business is tied to this event. Trust but verify things are ready; it’s worth it. Most big outages are caused by easily overlooked things.
6. Check your backups. Make sure you have reliable and consistent backups of your databases. This will make it easier to restore a crashed database. Percona XtraBackup for MySQL can help.

During the Event

1. Realize that failing over to another system is the absolute last resort.  Failing over a busy system moves the traffic to a new server. Additionally, most systems are slower when traffic is added as it takes time to warm up the cache.
2. Have the right people standing by to monitor, tweak, and fix issues before they get out of hand. Too many issues are prolonged by waiting to get the right people in the room to fix issues. Many larger companies make this an all-hands event.
3. Don’t lose sight of the goal! Getting back up and running and allowing customers access to the site is your goal, not developing a permanent fix in the heat of the moment. Time to make an impact is finite, but equally important people under pressure are more prone to make mistakes.  An easier temporary fix to get you through the day can work in your favor, as long as you don’t forget to make the permanent one eventually.
4. Don’t make the problem worse. Some activities can cause cascading slowness… know the impact of the changes before you make them. We get a lot of calls from people who had good intentions to try and fix a minor issue but made a much bigger problem. The road to hell is paved with good intentions.
5. Collect and store the data needed to analyze and improve for the next event. Often when things don’t go right, issues end up being transient and difficult to understand after the fact. Get the data you need, when things are going well or not.

After the Event

1. Analyze and understand your traffic and usage. Use this data to plan, enhance, and tweak your strategy for future events.
2. Don’t leave the quick fixes in place and just forget about them, these could escalate at the worst times. Take the time and expense to fix them during slow periods.
3. Learn from your mistakes and build a plan to mitigate problems and risks in the future.
4. Update your systems to the latest builds and security fixes. Take advantage of the slower load and catch up after a freeze/blackout period around the event.
5. Don’t fall into complacency. Congrats that you survived this year, but each application and user base is a living, breathing entity, and what worked last year may not work this year. You have to analyze, plan, and review on a regular basis.

Conclusion

Now you’re better prepared for the database high-traffic days coming your way soon. For more information, learn how to ensure peak database performance for your event and download our Database Disaster Prevention Checklist, orcontact us today. Percona’s experts can maximize your application performance with our open source database support, managed services or consulting for MySQL, MariaDB, MongoDB, PostgreSQL in on-premises and cloud environments.

Database Disaster Prevention Checklist