Major setbacks in current maintenance (rev. 1 by Drahan on 04-11-17, 01:52 am)
Posted at 04-10-17, 07:22 pm Link | #1
Drahan GM

Posts: 2147
Joined: 02-06-17
Last post: 42 days
Last view: 8 days
Update: The maintenance has been completed at 09:48PM 04/10/17 EST.
All servers are running normally.
Thank you for your patience.


Hey Milletians!

This post is to inform you all about the current state of our currently excessively long maintenance (which could potentially be the longest one to date).

It seems that in this current maintenance, there has been some major...major setbacks...

There has been numerous hardware failures in our back end infrastructure.
Our current production server is set to be brought offline tonight at 12:00 AM for repair analysis, and that is simply to analyze the current state of the hardware- who knows how long it will take to diagnose the issues, and then rebuild the server with repaired hardware.

Right now, we are moving files off of our current back end server to a new production server which will act as a fallback while our main server undergoes hardware maintenance tonight.

However, we have over 120GBs of data to transfer, analyze, verify, and then set up again in our new production server. This type of transfer was completely unexpected and we were not prepared.
Essentially, all of our backups for the production game server are in the form of 60GB .bin files containing the entire hard disk of our servers in a single file.
In order to extract the data we need, we must transfer it to the desktop of a system administrator, who will then mount the disk locally, retrieve the data, and then re-upload the data to our production server.
Due to limitations in network speed, such a transfer could take up to 10 hours - and that's before we can even retrieve the required data to run the game server.

Unfortunately, the game server is not the only server that requires transferring - we have to transfer our web services, database server, and miscellaneous systems too.
Much of these services require complete reconfiguration and re-provisioning. It takes a long time to setup all of these services from scratch, and we only have 1 person who is capable of working on maintenance of this scale, meaning this is a 1 man job.
I'm sorry that this maintenance has been so long, and for however much longer it will take.
I'll try to keep you all updated, but I'll be pretty busy as you might imagine.

Luckily, there will not be any loss of data during this maintenance. As for those who have time-restrained items/events/wine/etc, we will cross that bridge when the time comes however it's certainly not at the front of our minds right now.

That being said, there is no ETA for completion of this maintenance.
We apologize for the inconvenience.
Posted at 04-10-17, 07:32 pm Link | #2
Flipend0 GM

Posts: 987
Joined: 02-06-17
Last post: 10 days
Last view: 10 days
In addition to the Maintenance having no ETA of Completion.
As Game Admin and Event Manager, Easter Events will become extended of additional days of lost time of server being offline, and we will plan a compensation event as well.
Posted at 04-10-17, 07:34 pm Link | #3
Valikonis

Posts: 10
Joined: 03-16-17
Last post: 2151 days
Last view: 1666 days
Sounds like that one person is gonna need some coffee. Do what you gotta do - I'm just glad we've got GMs who are willing to go through all this effort in the first place. Thanks for your hard work, and for keeping us informed.

Posted at 04-10-17, 07:49 pm Link | #4
Hisao GM

Posts: 101
Joined: 02-06-17
Last post: 874 days
Last view: 20 days
Sounds like that one person is gonna need some coffee. Do what you gotta do - I'm just glad we've got GMs who are willing to go through all this effort in the first place. Thanks for your hard work, and for keeping us informed.


Not only one... Almost all of the GM team is involved in getting this as soon as we can.
      
Posted at 04-10-17, 07:49 pm Link | #5
LazyFae
Still a casual gamer

Posts: 306
Joined: 02-23-17
Last post: 723 days
Last view: 718 days
Transparency goes a long way. Just gonna say, +1 for the efforts, for all involved.
_________________________
Discord: Tonkatunk#7079
https://github.com/Tonkatunk/Mabi-Pro-Lazyfae-Mods
Posted at 04-10-17, 07:51 pm Link | #6
Zeno

Posts: 19
Joined: 03-27-17
Last post: 1803 days
Last view: 1798 days
Hopefully things go smoothly enough with the transfer. Thanks for keeping us up to date.
Posted at 04-10-17, 08:07 pm Link | #7
Jean

Posts: 28
Joined: 03-31-17
Last post: 2008 days
Last view: 796 days
I really appreciate this post. Keep up the good work, guys.
Posted at 04-10-17, 08:10 pm Link | #8
Valikonis

Posts: 10
Joined: 03-16-17
Last post: 2151 days
Last view: 1666 days
Not only one... Almost all of the GM team is involved in getting this as soon as we can.

Of course, I know you're all working hard, hence "GMs". Again, I appreciate it.
Posted at 04-10-17, 08:36 pm Link | #9
Drahan GM

Posts: 2147
Joined: 02-06-17
Last post: 42 days
Last view: 8 days
Large data ingestion from PROD-OLD to PROD-NEW underway..
Should complete about 10% of the process when it's complete...assuming the ingestion works properly.
Lots more to go..
Posted at 04-10-17, 09:28 pm Link | #10
Drahan GM

Posts: 2147
Joined: 02-06-17
Last post: 42 days
Last view: 8 days
Large data ingestion from PROD-OLD to PROD-NEW underway..
Should complete about 10% of the process when it's complete...assuming the ingestion works properly.
Lots more to go..
Well, that's one data ingestion completed.
Probably about 3-5 more to go..
Posted at 04-10-17, 09:37 pm Link | #11
lycoris

Posts: 89
Joined: 02-16-17
Last post: 1971 days
Last view: 1967 days
Thanks for keeping us updated, you are doing wonderfully and I'm sure we all appreciate the care for things like event timing! Don't feel rushed, and best of luck getting everything worked out. I'll see ya ingame when I see ya.
Posted at 04-10-17, 09:50 pm Link | #12
Drahan GM

Posts: 2147
Joined: 02-06-17
Last post: 42 days
Last view: 8 days
Currently, I am ingesting data from our PROD-WEB-OLD to PROD-WEB-NEW, the web services basically.
This handles stuff like guild list, housing board, visual chats, and any other web services that we utilize outside of the forum. This requires manual reconfiguration of each individual service, however.

After this, registration will be possible again and assuming it's all working well, I can start ingesting data from the game servers.
Posted at 04-10-17, 09:51 pm Link | #13
Kaytavo Tovani

Posts: 1
Joined: 04-03-17
Last post: 2544 days
Last view: 1894 days
wow talk about bad luck... here's to hoping that it goes atlest somewhat smoothly
Posted at 04-10-17, 10:06 pm Link | #14
Hope

Posts: 46
Joined: 03-29-17
Last post: 808 days
Last view: 4 days
Thank you for keeping us up-to-date on what's going on, what went wrong, etc. You all are doing a very great job keeping this site as well as the game alive. You have my +10 points on reputation list. You guys are just simply awesome! Take care and don't rush yourselves. At least I'll be here when the servers come back online. Not right away, but when I have the chance to. lol...
Much care, love, as well as hope,
- Hope
post rev. 2 by Drahan on 04-10-17, 10:13 pm
Posted at 04-10-17, 10:12 pm Link | #15
Drahan GM

Posts: 2147
Joined: 02-06-17
Last post: 42 days
Last view: 8 days
Seems that the web services are functional again; this means you are able to login and register on the forum again after we reflect the changes on the forum side.
More progress...

List of things to do has been narrowed down.
Posted at 04-10-17, 10:37 pm Link | #16
Meorin

Posts: 67
Joined: 02-19-17
Last post: 478 days
Last view: 459 days
Good luck on the work Drahan, we all are supporting you from afar.

Posted at 04-10-17, 10:45 pm Link | #17
Drahan GM

Posts: 2147
Joined: 02-06-17
Last post: 42 days
Last view: 8 days
Good luck on the work Drahan, we all are supporting you from afar.

Aw, that's adorable! Thanks!

As for the next update: We've managed to utilize the 1Gbit link between our old and new servers to our advantage, so it should not be necessary for us to download 120GBs to our desktops, manually mount the filesystem, and extract the data that way. We can do a server->server transfer. This is much faster than what we had in mind before.

Right now, data is being moved at rapid rates. However, there's still a lot of data left to go.
Posted at 04-10-17, 11:35 pm Link | #18
Glacii

Posts: 31
Joined: 03-11-17
Last post: 2519 days
Last view: 2488 days
You can do eet. You are the only one who can....

(literally)
Posted at 04-11-17, 12:01 am Link | #19
Drahan GM

Posts: 2147
Joined: 02-06-17
Last post: 42 days
Last view: 8 days
Provisioning new machine right now - data has been transferred from the old server's data repository to the new server.

Once this machine is provisioned, I will have to reconfigure about every firewall in the chain to point to this new node instead of our old node.

After that's done, the maintenance is over....hopefully.
Posted at 04-11-17, 12:44 am Link | #20
Zeno

Posts: 19
Joined: 03-27-17
Last post: 1803 days
Last view: 1798 days
Huh, maybe it'll be done before I'm back from work, ha. Glad to hear you could transfer server to server directly. And the updates have been great to check in and see. Awesome job, really enjoy the transparency.
Terms

Powered by mabi.pro v1.0034-arisa (View credits)
MabiPro is not associated with Nexon Co., Ltd. in any way shape or form.