MabiPro: The Authentic Old-School Mabinogi Experience

info Recent Server Instability
Posted on 01-18-18, 02:01 am by Drahan

Hello Milletians!

Recently, you may have noticed that recently the server has had a steep decline in stability.
We take server stability very seriously, and it's a real shame that we aren't able to fulfill our goals for stability.
We know these crashes are extremely frustrating for our players, especially those who lose items during the crashes. Trust me, they are extremely frustrating for us too. It seems like every time I look away for any extended period of time, something has gone wrong.

Today, I am going to write about the each individual crash, why they happened, and how the team is planning to solve these problems in the future.

Server crash explanations
In the past 2 weeks, the server has crashed 5 times. This means, during a working day there is a 50% chance that the server will crash.
A 50% crash rate is certainly not something we are proud of.

Here is a list of the last 5 crashes, and their causes.
1: Unexpected server maintenance (to patch Spectre and Meltdown). Our server host did not notify us in advance about this.
2: Database server stalled due to resource exhaustion.
3: Login server stalled due to resource exhaustion.
4: Database server stalled due to resource exhaustion.
5: Unexpected Windows OS level error, probably caused by resource exhaustion but not verified.

---

Pet lag (or rather, the sinister database stall)
You may have experienced database server stalling first hand - it's what is known as "pet lag". This "pet lag" is much more sinister than it may look at first glance.
The reason people call it pet lag is because summon times take an extremely long time to finish, the cause of this is the database server failing to respond to queries in a timely manner.
This lag also affects logging in, saving bank info, changing channels, and basically anything that touches the database.
The game server has a very aggressive cache on player data, so you don't notice it while playing until you summon a pet usually.
If pet lag is preset, the server is on a downward spiral towards an imminent full crash - where you will not be able to play at all.

Luckily, pet lag takes a long time to fully manifest into a full crash, so you may notice we do not restart the server until it gets extremely bad (and usually right before the full crash).
Unfortunately, this causes players to have a less than desirable experience - but it's our only choice. If we immediately restart the server every time the database starts to stall, we would be restarting the server every few days. We choose the lesser of two evils.

In the event of a full database server crash, you will be unable to login (it will stay on the login screen for a very long time and never finish).
Additionally, all unsaved player data is lost permanently and rolled back to a previous state.

Login server stalling
This is simply caused by extreme resource exhaustion. Mabinogi services do not recover well from failure, and under immense pressure, servers will occasionally lose connectivity with eachother.
The moment the login server loses connectivity with the server coordinator or database server is when it's stalled until manual reboot.

You can tell this has happened when you attempt to authenticate and you're immediately met with a "Unable to connect to server" error. The login server will not accept new connections when stalled.

Windows errors
We're using Windows 2003 for the game server. Not much more to say here. The software is outdated and occasionally misbehaves, especially under high load.
We use this because the Mabinogi server we use (G13 specifically) was designed for Win 2003 - it isn't very stable in other versions of Windows. Just like the Mabinogi client isn't very compatible with Win8/Win10, the server's got the exact same problems (random freezes, etc)!

Again, we choose the lesser of two evils - unstable server software or unstable operating system. It just so happens that the operating system as a whole crashes less than unstable server software.

This particular problem, though rare, is actually relatively simple to solve (in theory) compared to the other problems: just use a more stable windows version.
We are able to manually fix each crash we encounter on updated versions of Windows server, which we have done in beta testing - but we're never fully confident in that work, never really sure that we haven't missed some small case in the hundreds of thousands of lines of assembly code that we have to work with to fix such problems.

---

A common theme

You may have noticed a common theme by now..resource exhaustion. 4/5 of our server crashes were from resource exhaustion.
You're probably thinking, "Drahan, why exactly does MabiPro suffer from resource exhaustion in the first place?".

It's a good question - especially when you know how the Mabinogi server scales. It does not scale by player count or NPC count...it scales by NPC client count.
More players does not scale linearly to more resource usage.
Our actual resource usage, for the most part, is entirely static. It almost never changes.
Under this logic, there is absolutely no reason that we should be suffering from resource exhaustion in the first place..

So why?
It all boils down to one thing: the way we host and finance the server.
MabiPro is actually under an extremely low budget, and as such, we cut corners financially...a LOT.
The biggest corner we cut is our dedicated server hosting.
To cut dedicated server costs, we've got connections with a server hosting coalition that shall not be named; and we cut a deal with them.

Not only do we get an extreme discount for dedicated shares of their servers (much cheaper than what we can buy publicly), we are also allowed to use shares of the server that are unused, as in not being used by paying customers currently, free of charge.

We're using these unused server shares to power almost 80-90% of our server.

(Those of who you are reading who are knowledgeable in the server hosting industry - no, we're not technically using a dedicated server, it's just easier to explain it that way)

What's the catch?
You see, nothing is really free in this world. We save a lot of money from this deal, but it comes at a cost: server stability.

The problem is when a paying customer decides that they want to use the "unused" resources (read as, the resources MabiPro is using) that they deserve.
These resources are ganked away from MabiPro instantly, and allocated to that paying customer until they're done using it.
After they're done using it, we gain those resources again.

(For those with knowledgeable in the server hosting industry - yes, this is a form of overselling. It isn't good.)

However, Mabinogi does not respond kindly to it's resources being forcefully taken away.
This process will instantly make MabiPro crash for one reason or another.

Progression of the problem...
The more paying customers grow, the more often we get our resources (that we aren't paying for) forcefully taken away.
In the recent weeks, this has grown to a point where it may happen every other day due to simple growth of the server host.
In other words: we need to solve this problem, and we need to solve it fast.

---

The Solution

Really, the solution to a large majority of our problems is so simple, any of you could figure it out.
We just need to rent a new dedicated server, really.
This is what we plan to do in near future.

This will increase our operational cost by a significant amount, and for the most part we pay for the server out of pocket.
Fortunately, we have a good sum of Bitcoin donations (thanks to our generous community) that we have kept for a long while - we're going to use these funds to pay for the extra expenses incurred by renting a new server.
If we decide to go through with this plan, we will need continued support from the community in the long run.

---

We will try our best not to let you guys down, as long as you don't let us down.
Hopefully you enjoyed my large wall of text.

-GM Drahan

4 comments (last by Drahan). Log in to post a comment.

content poll Content Poll #4
Posted on 01-01-18, 02:48 am by Arisa

It's time for Content Poll #4! The questions will be posted below in advance of voting beginning. All questions are Yes or No answers, and you will be free to skip any of them that you're not sure about. Voting will begin ingame on January 3. As usual, for a question to pass, at least 75% of voters must have said Yes.

Please note that votes are always audited and that voting with alts is not allowed.

First off, there's some questions about increasing stack sizes! We propose increasing the stack sizes of ores from mining significantly, because currently it doesn't make much sense to mine when you can use metallurgy to get much more compact ores (and gems!). Also, in the last content poll, it was voted that alchemy crystal stack sizes should be doubled. This poll will decide if they get doubled yet again.

Followed by some questions about cosmetic changes to spirit weapons:



Historically the Refining skill has been notorious for being difficult to rank because of the failures required. We have the ability to make failures less important to rank the skill:


Previous content polls only made Composing ranks accessible one at a time. Why don't we just do all the rest and get it over with?


It is a minor change, but it may be convenient for characters starting in places other than Tir Chonaill to have access to skill resets without needing to travel to Duncan. We propose allowing Alexina, Castanea and King Krug to perform skill resets as well as Duncan. Additionally, if you have a character at total level 100 then the game considers you to be a seasoned player - but since on MabiPro many starter skills start at rank E, this can be problematic for assistant characters created after reaching total level 100 with your human. We propose checking if the character itself is at least total level 100, rather than if the account has any character that is at least total level 100.



It has been suggested that ancient medals from the elf/giant trans quests become possible to trade:


For some reason, new players start with the Blaze skill at rank E. None of us can remember why that is, but Blaze at rank E has a higher skill CP than any other starter skill does at rank 1. We propose making it possible to remove Blaze from your character. You would be able to ask Shyla to remove Blaze for free.


This next question is pretty self explanatory, but we may impose a small minimum donation amount to receive the title. Contributions to Patoots' editor fund will count.


Next are some various buffs to combat skills that were requested by players. The changes to Taunt and Sand Burst would fix a problem where it becomes impossible to train the skill after reaching a certain CP.





Some proposals to make items available in shops:



And finally, some cosmetic changes:





Feel free to discuss these questions in the comments of this post! Voting will begin ingame on January 3, and continue for at least a week.

31 comments (last by Shin Taillelaine). Log in to post a comment.

info Regarding event rewards
Posted on 01-01-18, 01:53 am by MabiPro Team

The Jasper's "12 Days of Christmas" event has concluded earlier this week, though we understand many players were upset with how its rewards were distributed.

The premise of the event was that the player collects 12 pieces of a Christmas carol and trades them in to give everyone on the server a gift. The abundance of carol pieces and the gifts that were distributed as a result were much higher than we had anticipated; over 35,000 gifts were distributed during the course of the event.

We received several suggestions on what to do about this. However, our developers had very limited time available to make changes to the event, and only a few changes were able to be made. These changes were controversial and, in hindsight, not ideal; though they were the best that we were able to do given the time and resources that we had available to us. We anticipate that, in the very near future, our developers will have more time to prepare events (and that we will have more new developers working on them!) so that they are better received than this one.

It seems that despite these setbacks, the server still managed to have a great Christmas celebration on the 25th, with everyone socializing and opening gifts together. What a huge crowd that was in Dunbarton! Positive moments like these strongly drive us to continue operating MabiPro, and we promise to do our best for all of you in 2018!

-GMs Arisa, Drahan, Flipend0, Jade

5 comments (last by Selzyr). Log in to post a comment.

cold time Cold Time Winter Dyes
Posted on 12-21-17, 07:48 am by Flipend0

Brr...It's freezing cold outside...It wouldn't make sense if we called this a h-hot time now would it...

Happy Holidays from the MabiPro Team. You can't go into winter fashion without participating in this cold time.
Check this out! If you're in game during these times listed below, you'll be rewarded a free fixed dye or metal dye. All you have to do is stay logged in. Easy right?

Cold Time Winter Dyes Schedule:

• Friday, 12/22 3:00PM EST
  
• Friday, 12/22 7:00PM EST
  
• Saturday, 12/23 3:00PM EST
  
• Saturday, 12/23 7:00PM EST
  
• Monday, 12/25 10:00AM EST
  
• Monday, 12/25 7:00PM EST
  

Don't miss out!
-MabiPro Staff

2 comments (last by Nickel). Log in to post a comment.

game event Jasper's "12 Days of Christmas" Event (12/15~12/26)
Posted on 12-15-17, 11:02 am by Drahan

Jasper Santa wishes to celebrate Christmas in a spectacular way: Playing a song from a different universe, titled "12 Days of Christmas".
However there was an issue! Jasper went to the library to get the song, but it appears that the monsters roaming around Erinn have stolen the music notes!
Jasper Santa requests help from you Milletians to recover the notes in order to save the Christmas spirit!

Event Details:
*This event will be active from 12/15~12/26
* Jasper can be found at Tir Chonaill.
* When defeating monsters, they will have a small chance to drop a page.
* If you wish, you can exchange single page for a small gift to yourself.
* If you're feeling more generous and collect all 12 pages, you can gift the entire server a gift.
* You must have 10 stacks of the gachapon to be able to open 1.
* You must be a total level of 60 to be able to open the gachapon.
* The gift cannot be opened until 12/25/17.

Special Thanks to "Rydian" for providing the MML Code for the "12 days of Christmas" song.

7 comments (last by Flipend0). Log in to post a comment.