Jump to content
wes

A Bumpy Start - A Detailed Explanation of Server/Database Errors, DLC and More

Recommended Posts

Hey campers,
 
Been a rocky few days, to say the least. Few of us have slept since Thursday, but I wanted to take a moment to inform all of you as to what’s going on with everything. We’re having global issues, on all platforms (PC/XB1/PS4). Here’s a breakdown of the most common issues so far. 
 
Known issues. Why they are happening. What we’re doing to fix them.
 
Issue: “I can’t login, it keeps saying something about a Database Error.”
 
Why: Our database was hit really hard when the game went live. We had upwards of 75K players in a short period of time. Due to this, our database was hit hard. We tried spinning up more nodes to help alleviate the stress but ultimately caused some corrupted nodes which are causing people not to be able to log in that were affected by those nodes. In addition, it has caused major slowdowns in our database software which can on and off effect other accounts from logging in.
 
Fix: We are working around the clock to get this solved. We are a small team but brought in outside resources to help. It's our top priority right now and we need to make sure we can get our servers properly scaled and balanced to handle the current demand. Once we can alleviate some stress on the database this should all start working smoothly. We are working hard to properly load balance our servers and identify issues with accounts that are slowing down the database.
 
Issue: “I can sign in (sometimes) but I can’t find a match when I select Quick Play”
 
Why: Due to a high demand of PS4 players there is an issue with Matchmaking when it finds an open session several players try to join that open session at once filling up the room the same time you receive the message this room is still open. You end up not being able to connect to that session and move on but it wastes time taking it a lot longer to find an open session. On Xbox One there also is a high number of players trying to find games and the amount of requests is really adding up which is overloading the Matchmaking service.
 
Fix: We have tuned Matchmaking on PS4 to create more open sessions to spread out the player base. We also have tuned the time it takes to create new sessions. We are currently testing the fix and hope to have a patch out real soon. For Xbox One we have been working with Microsoft who has tuned some settings on their end. This has helped some users but not all. We are still working on tuning our Matchmaking algorithm along throttling requests more in line with LIVE standards and with working with Microsoft on this as well. We hope to have a patch in testing early this week.
 
Issue: “It’s not saving my Jason kills, XP, clothing customization or achievements.”
 
Why: This is a result of the mentioned above database issue. Unfortunately, all of this information is tied to the database as well.
 
Fix: Once we can alleviate some stress on the database this should all start working smoothly. We are working hard to properly load balance our servers and identify issues with accounts that are slowing down the database.
 
Issue: “What happened to my XP? I was level ‘X’ now I’m level 0” or “My XP is stuck and I’m not getting any more.”
 
Why: This is also a result to the database issue. Match statistic reports still seem to be coming into the database correctly so if you had a level at some point, it should be still progressing.
 
Fix: Once we can alleviate some stress on the database this should all start working smoothly. You should still be earning XP but it’s just not showing it properly. When the database is less stressed you should see your proper XP level.
 
Issue: “I see others can play, but I have yet to get past the login screen.”
 
Why: This is also the result of the database issue. We’ve noticed it can randomly affect specific accounts. So while you can’t play someone in the next room with a different account can play. We have identified what causes it for some accounts and fixed it, but there are a few causes at play here.
 
Fix:  Once we can alleviate some stress on the database this should all start working smoothly. We are working hard to properly load balance our servers and identify issues with accounts that are slowing down the database.
 
Issue: “My DLC isn’t working. I downloaded it, where is it?”
 
Why: PC Keys were not working and we had to issue a new set of keys. For PS4 DLC keys we are still waiting for the DLC to be approved and go live.
 
Fix: If you are a PC user please make sure you have received your new keys. Check your email. If not please contact us. PS4 users we will keep you posted when the DLC is live.
 
Issue: “I never got my game/DLC code. I’m a backer, what’s up?”
 
Why: This could be due to multiple reasons. Emails from BackerKit do not contain codes, you must log in to BackerKit to retrieve codes you’ve ordered. You possibly did not complete your BackerKit survey? Or there may be an actual customer service problem, error or glitch. 
 
Fix: The first step is to use your original Kickstarter email address to log in to https://friday-the-13th-the-game.backerkit.com/ and retrieve your codes. Keep in mind that if you ordered multiple times that you will have to check each order survey to look for all your codes as they are associated with each order they were placed on. 
 
If you did not fill out your survey/s, you will need to do so and then we can lock it down and process your codes so they show up in your account. Remember, if you had multiple orders then you will need to go through and fill out each survey. 
 
 
Issue: “I received the wrong platform code via email. Help!”
 
Why: Sometimes mistakes are made and glitches happen. Some players received codes for the wrong platforms, but this can be rectified after verification of the issue. 
 
Fix: If for some reason you received the wrong code for a platform you didn’t order for, please contact BackerKit by logging on to your account at https://friday-the-13th-the-game.backerkit.com/ and then use the contact form to input your issue. You will then only need to wait for a response. You cannot switch to a different platform. You will receive the code that was selected in your original BackerKit survey. We repeat, you cannot swap platforms just because you changed your mind.
 
Issue: “I can play, but i’m experiencing glitches/bugs/crashes.”
 
Why: General gameplay bugs that may have already been fixed but a patch hasn’t been released for that platform yet. In addition, as new players play they are helping us find bugs all the time.
 
Fix: We have an Xbox One patch that addresses a lot of the glitches/bugs/crashes that we are testing now and are shooting to have it out mid next week. We are currently working on another PS4 patch to help with Matchmaking and some gameplay bugs. Our team is constantly working on fixing bugs, optimizations and pushing new patches on all platforms.
 
Issue: “OK, but why weren’t you guys prepared for this?”
 
We thought we were. We looked at average concurrent players during the beta; 5K. We then took PC pre-order numbers, as well as backer numbers and tripled it for three platforms (XB1, PS4 and PC). We then added more cushion just to make sure. We were forecasting somewhere around 30K concurrents and that’s what we setup our database to handle. However, when we flipped the switch, making the game live, 75K players showed up in the first 20 minutes. 50K players got in, but at that point, our database was melting. We immediately started spinning up new database servers to meet demand. But in the process, the original database started getting damaged nodes due to the influx of players. We were moving just as fast as we could to meet the demand. But it wasn’t fast enough. 
 
Due to the damaged node, bad accounts started being recorded. Bad as in, the data associated with the account was taking too long for the database to read it. So we had two issues hitting at once, within a 30-minute time frame; not a robust enough database to meet demand, and corrupted data (bad accounts). Basically, we have accounts that are taking a long time to query and we are investigating why. 
 
If you log in on your bad account and the request hits before someone with a good account, it’s timing out both players. Right now, bad accounts are taking upwards of 120 seconds to pull it down. This slows everyone down that is in line with them on that node. And in some cases, the database says “this is taking too long” and just throws up the error for everyone standing in line. Good or bad account. 
 
The fix to this is a more robust scaling solution in our database. 90% of player accounts are good. The remaining 10% will take more time due to how corrupted they are. Again, this data is salvageable, just tedious and time consuming to repair. 

 

Sorry everyone. We are doing our absolute best. 

-W
 
 
 

  • Like 17

Share this post


Link to post
Share on other sites

Thanks for the update. I've been telling people you're working hard on the issue whenever I post. I'm glad you have taken the time to explain some of the issues. I'll just link them to this thread. :)

Good luck to you, the team, and the consultants!  

  • Like 1

Share this post


Link to post
Share on other sites

Wes, I'm sorry, but this means nothing to us. Seriously. I dont mean that as "we dont care". I mean this as, I have no idea what that actually means for everything. This post, to me, says "Shit's broke. We're trying to fix it. Dont know how long. Could be an hour, or could be a year." Seriously......that's all I can glean from this.

  • Like 2

Share this post


Link to post
Share on other sites

Im really thankful to you guys for constantly updating us on progress with the issue, but it can't stop how heartbroken I am over this whole issue - it's lead to a plethora of bad press for the game and I'm the only person in my group of friends who suffers from the issue, so I've just had to watch on the sidelines as they enjoy the game I've backed and waited 2 years for while I'm unable to even log in.

Share this post


Link to post
Share on other sites
9 minutes ago, BloodOnTheRocks said:

I have no idea what that actually means for everything. This post, to me, says "Shit's broke. We're trying to fix it. Dont know how long. Could be an hour, or could be a year." Seriously......that's all I can glean from this.

To people that don't have technical knowledge of databases, servers, etc. there is little anybody can say for those people to understand. What you believe about this post is a fair assessment and yet other people will just see it as an excuse or poor planning. 

it is really tough to plan for a launch day. No one knows how many people will log in at the same time. Sounds like they had a decent plan in mind but the influx was far more than they could have imagined. 

I, for one, will be patient and wait. As someone who works in IT and with servers and databases, I know this stuff takes time.  For example, you can wait for hours for a database restore, only to have it not work or to fail in the middle.  

its incredibly frustrating for everyone, but I'm sure the team feels terrible for the issues.  Now they can only fix them and hope that the impatient players will stick around.  

  • Like 1

Share this post


Link to post
Share on other sites

Thanks for the update. With a little technical knowledge this kinda makes sense to me. Data conflicts are the worst and can take time to resolve the conflict. Wish you luck and making it a speedy recovery!

Share this post


Link to post
Share on other sites
9 minutes ago, BloodOnTheRocks said:

Wes, I'm sorry, but this means nothing to us. Seriously. I dont mean that as "we dont care". I mean this as, I have no idea what that actually means for everything. This post, to me, says "Shit's broke. We're trying to fix it. Dont know how long. Could be an hour, or could be a year." Seriously......that's all I can glean from this.

Basically they have tried to adapt the database with something new (a fix) and its conflicted with the system they already had and its....well....f***ed it i guess.

Share this post


Link to post
Share on other sites

Having 250% of your estimated audience turn up within the first 20 minutes is a great thing for films, not so great for servers.

Thanks for the update and I hope the stress eases on the system and you guys soon.

  • Like 5

Share this post


Link to post
Share on other sites
1 hour ago, wes said:

Hey campers,
 
Been a rocky few days, to say the least. Few of us have slept since Thursday,
 

You got it in one. For different reasons of course. 

I'm just gonna toss out a little morale. Even though launch has been crazyballs, even when half the time games end midway through, and lobbies empty and we all have to deal with getting back in, when the game works, it's fucking magic. 

So a pat on the back for making the F13 game we've all wanted.

  • Like 8

Share this post


Link to post
Share on other sites

I've been watching streams, takes some time to find a decent streamer who doesn't like the sound of their own voice...but from watching those that play the game how it's meant to be played or have picked up how to play well, I'm very, very impressed. 

Watched a very good Jason earlier, who was also very, very effective as a counselor. Was on a ridiculous winning streak, and played the game so well. There were some counselor moments that really had me on the edge of my seat, as he was. Really captured the thrill and tension of a final chase of Friday film down to a tee.

I can't wait to play this game!! :) I'm yet to play it, but I know I'm going to love it.

 

 

  • Like 2

Share this post


Link to post
Share on other sites
1 hour ago, BloodOnTheRocks said:

Wes, I'm sorry, but this means nothing to us. Seriously. I dont mean that as "we dont care". I mean this as, I have no idea what that actually means for everything. This post, to me, says "Shit's broke. We're trying to fix it. Dont know how long. Could be an hour, or could be a year." Seriously......that's all I can glean from this.

Understood. But we were getting inundated with "Tell us exactly what's going on..." and "Why aren't you giving us details?". The problem was, up until early this morning, we weren't sure what was going on either. Once we could see the issue, I brought that information to the community as fast as I could. I realize the only real information everyone wants is "when will this be fixed?" - The problem is we don't know. We don't have an ETA. I wish we did. But it's not a problem with a simple solution. Nor is it as predictable as troubleshooting, say, a knock coming from a car engine. I wish it was, I truly do. We made good strides in the wee hours this morning. We started seeing more successful login attempts and servers filling up. But there's still a lot that can't. When I know more, you'll know more.

 

-W

  • Like 6

Share this post


Link to post
Share on other sites
5 minutes ago, derrickkolba said:

how many bags of potatoes are you up to now wes?

hqdefault.jpg

 

We're actually using the highest-end servers that Amazon provides, as well as the same database environment LoL uses. So it's not about potato servers. More about a massive tidal wave of players, hitting all at once, that we underestimated.  

  • Like 1

Share this post


Link to post
Share on other sites

well this post doesnt mean anything then you must be some kind of casual gamer who for some reason whatsoever found this thing called message board and then signed up for it.....

the last couple hours workked really well, XP and CP was earned again, i leveld up etc. gameplay was fun, some bugs and glitches happened to all our amusement.
A rocky start but like its said.... the boat is floating again..

 

now the only thing I personally need to be truly happy is my PS4 DLC keys which are in the works and then its all cool for me except of course for patches and updates :)

aaaaand i wanna play you in-game @wes @Randygbk 

Share this post


Link to post
Share on other sites

shut it all down and start it back up one platform at a time. this is i think your best option, best do it now then drag it out longer.

Share this post


Link to post
Share on other sites
Just now, derrickkolba said:

shut it all down and start it back up one platform at a time. this is i think your best option, best do it now then drag it out longer.

It doesn't work like that. Shutting everything down doesn't really fix the problem. We can work on the database while still live. No use in kicking the 50K+ players online right now.

  • Like 1

Share this post


Link to post
Share on other sites
3 minutes ago, wes said:

It doesn't work like that. Shutting everything down doesn't really fix the problem. We can work on the database while still live. No use in kicking the 50K+ players online right now.

ok then how do you plan to relieve the stress on the server if you allow more and more to connect?

Share this post


Link to post
Share on other sites
Just now, derrickkolba said:

ok then how do you plan to relieve the stress on the server if you allow more and more to connect?

There isn't stress on the server. We spun up enough for 250K concurrent players/sign in. As I explained in the post, the pain point are these bad accounts slowing down everyone else.  

Share this post


Link to post
Share on other sites
7 hours ago, derrickkolba said:

shut it all down and start it back up one platform at a time. this is i think your best option, best do it now then drag it out longer.

 

 

 

6 hours ago, wes said:

There isn't stress on the server. We spun up enough for 250K concurrent players/sign in. As I explained in the post, the pain point are these bad accounts slowing down everyone else.  

let me explain it in a way that even a dumbo might understand it:

the gastank of the car isnt big enough for the engine. when it started, the tank was immediatly empty and the engine sucked in air. now there is a much much bigger and easily sufficient gastank attached to that engine, but there is still some air left in the pipes connection the tank to the engine.... causing some left over stutters that have to be resolved over time.

  • Like 4

Share this post


Link to post
Share on other sites
3 minutes ago, IH8YH said:

let me explain it in a way that even a dumbo might understand it:

the gastank of the car isnt big enough for the engine. when it started, the tank was immediatly empty and the engine sucked in air. now there is a much much bigger and easily sufficient gastank attached to that engine, but there is still some air left in the pipes connection the tank to the engine.... causing some left over stutters that have to be resolved over time.

your explanation is worse then wes's leave it to wes to explain.

 

 

Share this post


Link to post
Share on other sites
18 minutes ago, IH8YH said:

well this post doesnt mean anything then you must be some kind of casual gamer who for some reason whatsoever found this thing called message board and then signed up for it.....

the last couple hours workked really well, XP and CP was earned again, i leveld up etc. gameplay was fun, some bugs and glitches happened to all our amusement.
A rocky start but like its said.... the boat is floating again..

 

now the only thing I personally need to be truly happy is my PS4 DLC keys which are in the works and then its all cool for me except of course for patches and updates :)

aaaaand i wanna play you in-game @wes @Randygbk 

Same for me. Everything has been great on PS 4 with Private matches. Basically what it comes down to is THIS GAME IS FUCKEN POPULAR ! and they didn't expect the overload on day one. This is nothing new to any other day one release game. For those that haven't been able to play I feel bad for you but just be patient and it will get worked out and trust me when I say you will have a fucken blast when you get in !...Thanks Wes for the update.

  • Like 1

Share this post


Link to post
Share on other sites
11 minutes ago, wes said:

There isn't stress on the server. We spun up enough for 250K concurrent players/sign in. As I explained in the post, the pain point are these bad accounts slowing down everyone else.  

What happens with the "bad accounts" that are slowing everyone else down? I hosted a game earlier and some players reported  "lagging" but I never did lag at all. Not long after i lost connection and haven't gotten it back since this morning. How do you know if you have a "bad account '   

Share this post


Link to post
Share on other sites
13 minutes ago, wes said:

We're actually using the highest-end servers that Amazon provides, as well as the same database environment LoL uses. So it's not about potato servers. More about a massive tidal wave of players, hitting all at once, that we underestimated.  

Here is what I have a major problem with.You guys had over 12k backers that were getting the game 100% guaranteed. Now giving that it is a holiday weekend maybe 10% of those weren't going to play so that leaves you 10k gamers and that is the number you tripled to as you put it in "forecasting". You all have 90k followers on twitter, you pimped yourselves out to every big streamer which probably averages out to 25k viewers and then you went ahead with the gaming industry media to get even more eyes on the project. And yet the best forecast you all came up with was in the low 30k range. That's astonishing to me and you should be so thankful that people are still on your side. People are not even allowed to criticize this abomination of a release with getting so much shit thrown their way.  With all the analytics at your disposal how you didnt estimate 100k is ridiculous on 3 platforms having one of the most hype indie releases in history. 

 

The game is fantastic you really can create a good product with a limited team, but you need to surround yourself with people that actually know the business. You said you wanted to make this as a passion project and I get that, but when I hear someone use the term passion project it means they have done everything to make the project a reality by themselves i.e. self funded. Why didnt you look to a big publisher to handle this aspect? Respawn seemed to work out a great deal with EA where they handle whats important to them(the game and their IP) and just give them a cut of the game. Going by the 75k number you and your real investors just made 2.5mil in 2 days. 

 

This is why I have a really hard time cutting you slack, minus the xbox launch issues this is all self inflicted and all very much was avoidable

  • Like 1

Share this post


Link to post
Share on other sites
3 minutes ago, urbt said:

Here is what I have a major problem with.You guys had over 12k backers that were getting the game 100% guaranteed. Now giving that it is a holiday weekend maybe 10% of those weren't going to play so that leaves you 10k gamers and that is the number you tripled to as you put it in "forecasting". You all have 90k followers on twitter, you pimped yourselves out to every big streamer which probably averages out to 25k viewers and then you went ahead with the gaming industry media to get even more eyes on the project. And yet the best forecast you all came up with was in the low 30k range. That's astonishing to me and you should be so thankful that people are still on your side. People are not even allowed to criticize this abomination of a release with getting so much shit thrown their way.  With all the analytics at your disposal how you didnt estimate 100k is ridiculous on 3 platforms having one of the most hype indie releases in history. 

 

The game is fantastic you really can create a good product with a limited team, but you need to surround yourself with people that actually know the business. You said you wanted to make this as a passion project and I get that, but when I hear someone use the term passion project it means they have done everything to make the project a reality by themselves i.e. self funded. Why didnt you look to a big publisher to handle this aspect? Respawn seemed to work out a great deal with EA where they handle whats important to them(the game and their IP) and just give them a cut of the game. Going by the 75k number you and your real investors just made 2.5mil in 2 days. 

 

This is why I have a really hard time cutting you slack, minus the xbox launch issues this is all self inflicted and all very much was avoidable

I think you might be confusing 30K concurrent players with 30K units sold. HUGE difference. We prepped for 30K concurrent players. Which means every second, of every day, 24/7, our database can support 30K players.

90K twitter followers does not equate sales. The conversion rate on social platforms to actual paying customers are fractions of percents. 

The conversion rate from a twitch viewer to a paying customer hovers around 2%, given the streamer holds an audience of roughly 2K viewers. Meaning of 2K viewers, about 2% or 40 customers give or take, will purchase the title. 

And we didn't have one of the most hyped indie releases in history. We did have momentum, that is for sure. But looking at one month's worth of PC pre-orders, we made the best calculations we could. 

As for why we didn't look to a "big publisher" - We met with every single one of them 2 years ago. None were interested. They either didn't believe in the concept or had their mind made up that film IP doesn't equate to good games. We did meet with one that was somewhat interested. But they were far from "big".  

  • Like 1

Share this post


Link to post
Share on other sites
7 minutes ago, wes said:

I think you might be confusing 30K concurrent players with 30K units sold. HUGE difference. We prepped for 30K concurrent players. Which means every second, of every day, 24/7, our database can support 30K players.

90K twitter followers does not equate sales. The conversion rate on social platforms to actual paying customers are fractions of percents. 

The conversion rate from a twitch viewer to a paying customer hovers around 2%, given the streamer holds an audience of roughly 2K viewers. Meaning of 2K viewers, about 2% or 40 customers give or take, will purchase the title. 

And we didn't have one of the most hyped indie releases in history. We did have momentum, that is for sure. But looking at one month's worth of PC pre-orders, we made the best calculations we could. 

As for why we didn't look to a "big publisher" - We met with every single one of them 2 years ago. None were interested. They either didn't believe in the concept or had their mind made up that film IP doesn't equate to good games. We did meet with one that was somewhat interested. But they were far from "big".  

This is actually great info.  I mean...its still a shitshow but makes a little more sense reading this.  

Share this post


Link to post
Share on other sites

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now

×