Drahan
GM
Posts: 2147
Joined: 02-06-17
Last post: 266 days
Last view: 210 days
Update: The maintenance has been completed at 09:48PM 04/10/17 EST.
All servers are running normally.
Thank you for your patience.
Hey Milletians!
This post is to inform you all about the current state of our currently excessively long maintenance (which could potentially be the longest one to date).
It seems that in this current maintenance, there has been some major...major setbacks...
There has been numerous hardware failures in our back end infrastructure.
Our current production server is set to be brought offline tonight at 12:00 AM for repair analysis, and that is simply to analyze the current state of the hardware- who knows how long it will take to diagnose the issues, and then rebuild the server with repaired hardware.
Right now, we are moving files off of our current back end server to a new production server which will act as a fallback while our main server undergoes hardware maintenance tonight.
However, we have over 120GBs of data to transfer, analyze, verify, and then set up again in our new production server. This type of transfer was completely unexpected and we were not prepared.
Essentially, all of our backups for the production game server are in the form of 60GB .bin files containing the entire hard disk of our servers in a single file.
In order to extract the data we need, we must transfer it to the desktop of a system administrator, who will then mount the disk locally, retrieve the data, and then re-upload the data to our production server.
Due to limitations in network speed, such a transfer could take up to 10 hours - and that's before we can even retrieve the required data to run the game server.
Unfortunately, the game server is not the only server that requires transferring - we have to transfer our web services, database server, and miscellaneous systems too.
Much of these services require complete reconfiguration and re-provisioning. It takes a long time to setup all of these services from scratch, and we only have 1 person who is capable of working on maintenance of this scale, meaning this is a 1 man job.
I'm sorry that this maintenance has been so long, and for however much longer it will take.
I'll try to keep you all updated, but I'll be pretty busy as you might imagine.
Luckily, there will not be any loss of data during this maintenance. As for those who have time-restrained items/events/wine/etc, we will cross that bridge when the time comes however it's certainly not at the front of our minds right now.
That being said, there is no ETA for completion of this maintenance.
We apologize for the inconvenience.