It’s been a long night/morning for us here at Dribbble HQ. We profusely apologize for the downtime, and can’t thank you all enough for your patience while we get back on track. Last night at approximately 8pm EDT, our database server went belly up and we’ve been working non-stop since then to bring the site back up.
Here’s what we know now, and what we’re doing to move forward:
- We had corruption in our database. The exact cause is unknown but our leading theory is that we experienced a memory corruption problem where a chunk of memory got wiped and replaced with zeros. We continue to investigate.
- This site is back up and running, but we’re a bit tentative as we are still unsure of the root cause of the problem. For the time being, we’re running much more frequent backups and working toward a more robust database architecture.
- We spent much of the night trying to salvage the latest data, but ultimately had to rollback to a backup copy of our database from roughly 2am EDT May 14, so most of the work that was posted yesterday has been lost. This pains us to no end are we are deeply sorry for that.
- We’ll be restoring lost data regarding pro accounts (orders and status) and looking to see what other information we can retrieve. If you went Pro yesterday, hang in there, we’ll get your status upgraded as soon as possible. Thank you for your continued patience as we work our way through this recovery. We’ll keep you updated as we step through this process.
- This episode has exposed a lot of flaws in our production architecture and our ability to react to crises and we’re going to work hard to correct them.
Thanks again for your understanding, patience and support. Rest assured, we’ll continue to do everything we can to keep Dribbble running smoothly again.
Find more Updates stories on our blog Courtside. Have a suggestion? Contact stories@dribbble.com.
