MabiPro: The Authentic Old-School Mabinogi Experience

Update: The maintenance has been completed at 09:48PM 04/10/17 EST.
All servers are running normally.
Thank you for your patience.

Hey Milletians!

This post is to inform you all about the current state of our currently excessively long maintenance (which could potentially be the longest one to date).

It seems that in this current maintenance, there has been some major...major setbacks...

There has been numerous hardware failures in our back end infrastructure.
Our current production server is set to be brought offline tonight at 12:00 AM for repair analysis, and that is simply to analyze the current state of the hardware- who knows how long it will take to diagnose the issues, and then rebuild the server with repaired hardware.

Right now, we are moving files off of our current back end server to a new production server which will act as a fallback while our main server undergoes hardware maintenance tonight.

However, we have over 120GBs of data to transfer, analyze, verify, and then set up again in our new production server. This type of transfer was completely unexpected and we were not prepared.
Essentially, all of our backups for the production game server are in the form of 60GB .bin files containing the entire hard disk of our servers in a single file.
In order to extract the data we need, we must transfer it to the desktop of a system administrator, who will then mount the disk locally, retrieve the data, and then re-upload the data to our production server.
Due to limitations in network speed, such a transfer could take up to 10 hours - and that's before we can even retrieve the required data to run the game server.

Unfortunately, the game server is not the only server that requires transferring - we have to transfer our web services, database server, and miscellaneous systems too.
Much of these services require complete reconfiguration and re-provisioning. It takes a long time to setup all of these services from scratch, and we only have 1 person who is capable of working on maintenance of this scale, meaning this is a 1 man job.
I'm sorry that this maintenance has been so long, and for however much longer it will take.
I'll try to keep you all updated, but I'll be pretty busy as you might imagine.

Luckily, there will not be any loss of data during this maintenance. As for those who have time-restrained items/events/wine/etc, we will cross that bridge when the time comes however it's certainly not at the front of our minds right now.

That being said, there is no ETA for completion of this maintenance.
We apologize for the inconvenience.

In addition to the Maintenance having no ETA of Completion.
As Game Admin and Event Manager, Easter Events will become extended of additional days of lost time of server being offline, and we will plan a compensation event as well.

Sounds like that one person is gonna need some coffee. Do what you gotta do - I'm just glad we've got GMs who are willing to go through all this effort in the first place. Thanks for your hard work, and for keeping us informed.

Not only one... Almost all of the GM team is involved in getting this as soon as we can.
      

Transparency goes a long way. Just gonna say, +1 for the efforts, for all involved.
_________________________
Discord: Tonkatunk#7079
https://github.com/Tonkatunk/Mabi-Pro-Lazyfae-Mods

Hopefully things go smoothly enough with the transfer. Thanks for keeping us up to date.

I really appreciate this post. Keep up the good work, guys.

Of course, I know you're all working hard, hence "GMs". Again, I appreciate it.

Large data ingestion from PROD-OLD to PROD-NEW underway..
Should complete about 10% of the process when it's complete...assuming the ingestion works properly.
Lots more to go..

Well, that's one data ingestion completed.
Probably about 3-5 more to go..

Thanks for keeping us updated, you are doing wonderfully and I'm sure we all appreciate the care for things like event timing! Don't feel rushed, and best of luck getting everything worked out. I'll see ya ingame when I see ya.

Currently, I am ingesting data from our PROD-WEB-OLD to PROD-WEB-NEW, the web services basically.
This handles stuff like guild list, housing board, visual chats, and any other web services that we utilize outside of the forum. This requires manual reconfiguration of each individual service, however.

After this, registration will be possible again and assuming it's all working well, I can start ingesting data from the game servers.

wow talk about bad luck... here's to hoping that it goes atlest somewhat smoothly

Thank you for keeping us up-to-date on what's going on, what went wrong, etc. You all are doing a very great job keeping this site as well as the game alive. You have my +10 points on reputation list. You guys are just simply awesome! Take care and don't rush yourselves. At least I'll be here when the servers come back online. Not right away, but when I have the chance to. lol...
Much care, love, as well as hope,
- Hope

Seems that the web services are functional again; this means you are able to login and register on the forum again after we reflect the changes on the forum side.
More progress...

List of things to do has been narrowed down.

Good luck on the work Drahan, we all are supporting you from afar.

Aw, that's adorable! Thanks!

As for the next update: We've managed to utilize the 1Gbit link between our old and new servers to our advantage, so it should not be necessary for us to download 120GBs to our desktops, manually mount the filesystem, and extract the data that way. We can do a server->server transfer. This is much faster than what we had in mind before.

Right now, data is being moved at rapid rates. However, there's still a lot of data left to go.

You can do eet. You are the only one who can....

(literally)

Provisioning new machine right now - data has been transferred from the old server's data repository to the new server.

Once this machine is provisioned, I will have to reconfigure about every firewall in the chain to point to this new node instead of our old node.

After that's done, the maintenance is over....hopefully.

Huh, maybe it'll be done before I'm back from work, ha. Glad to hear you could transfer server to server directly. And the updates have been great to check in and see. Awesome job, really enjoy the transparency.