Server's Back up - Minor Rollback

Any important server news or announcements will be posted here.

Moderators: Lead Developers, Senior Lead Admins

Locked
User avatar
Mmartin
Head of Development
Head of Development
Posts: 7231
Joined: Thu Mar 01, 2012 10:05 pm
Ingame name: Lucas Foreman
Location: Basement

Server's Back up - Minor Rollback

Post by Mmartin » Mon Jul 22, 2019 2:12 am

Hey all,

As I'm sure you've noticed, the server experienced a lot of turbulence starting on July 18th. It was then turned off on the morning of 19th. The downtime was a result of minor misconfiguration that was done about a year ago and took a while to blow up, making our drives unreliable. That corrupted some of our services, most notably our database server.

During the server's downtime, the last two days I've done my best to restore the most recent version of our database. We obviously back up everything daily, but since I wanted to avoid a large rollback (our most recent full backup was from July 18th 3 AM or so), I managed to manually recover everything up to around July 18th afternoon/evening - the evening before the server went down for maintenance.

To clear some confusion - the downtime wasn't a cause of a DDOS, or any other player attack. The misconfiguration was completely my fault, for which I sincerely apologize.

Despite my best efforts, there's still some minor data loss. This could be virtually any changes (player stats. vehicles, or even accounts) which were created around the afternoon of July 18th or later. If you lost any assets gained in this time and have sufficient proof, you're good to create a refund request. If you lost donation perks purchased in this timeframe, please open a support ticket, regardless of evidence you have. If you lost items that you've had since before July 18th, please create a refund request with at least some sort of evidence and I can try to check it, but if the purhcase happened on 18th I can't do anything.

All in all, with the data corruption that happened, this is a successful recovery. I sympathize with anyone that lost an item or two during the few hours of rollback, but given the situation it could've gone much, much worse if we weren't properly prepared for this - data corruption often leads to complete data loss if it's not backed up properly.

In order to avoid a similar situation in the future, I'll increase the frequency of our backups from once a day to multiple times for the most crucial data (player accounts, vehicles, properties etc) so should something similar happen in the future, the loss will be even smaller. This is the first rollback we've had in the last few years (that I remember, anyway) and I hope it's the last one I'm writing about!

Current donation packages have been extended by 7 days (a week) to compensate for the two day downtime.

Thank you for your understanding, I'll try to make up for the mess
Mmartin

User avatar
Mmartin
Head of Development
Head of Development
Posts: 7231
Joined: Thu Mar 01, 2012 10:05 pm
Ingame name: Lucas Foreman
Location: Basement

Re: Server's Back up - Minor Rollback

Post by Mmartin » Mon Jul 22, 2019 2:21 am


Locked

Return to “Announcements”

Who is online

Users browsing this forum: No registered users