When you load a page on MOBAFire we start a new session, to track you across requests, mainly for the purposes of letting you log in. The session ID that identifies you is stored as a cookie. If you clear your cookies, switch browsers, or switch to incognito mode, that session ID is lost and you will be given a new one.
We have 6 web servers behind a load balancer. So you connect to the load balancer and it forwards your request to one of the 6 web servers (prioritizing the ones with the least number of current connections). The load balancer records your session ID from the response, and then when you load another page, the load balancer sees your session ID, it sees that your previous request went to node 4, say, and so it sends you back to node 4 every time. So as long as your session ID cookie exists, you will always be sent to the same node on subsequent requests. I won't explain why that's necessary, but suffice to say it is necessary :)
SO, when you lose your session ID cookie - by logging out, clearing cookies, switching to incognito or switching to a different browser (or by the cookie timing out and your browser deleting it automatically) - you will be assigned to a node all over again, which has a 5/6 chance of being a different node from the one that you were on before. If one of the nodes is giving you trouble, you can do one of the above things to try and move to a different node that isn't giving you trouble.
If you are in incognito mode, every single request will go to a different node, because you're never storing your session ID cookie.
So, you guys are having 500 errors consistently on one node it seems, and then by performing an action that puts you on a different node you can load the site again. The problem is, I am checking every node individually, directly, throughout the day, and I have never once had a 500 error. So I'm trying to think why that is the case. It could be;
-something wrong with your accounts
-something broken on the specific pages you're trying to load
-something wrong with your session file
-something regional
-something wrong with your browsers
-something else entirely that I haven't even thought of
This is really difficult to troubleshoot :\ Next time you get a 500 error, also note down the page or action you were performing. Additionally, try loading the homepage directly and see if that loads. If so, go back and try the previous page and see if it still gives a 500 error. At least that will help to see if there is specific content or actions causing trouble. In my own frequent checking I'm just checking the homepage, maybe it's some sub pages giving trouble.
Edit: I just found an issue with our 5th web node, and I've fixed it. That MIGHT have been it. There was a connection issue between that node and our database but just for our cache stuff. So ... again, continue to let me know if you have issues - this may have been it. This was an issue with the migration to the new server though so, the outages before last night were probably fixed by the move to the new db, and the outages today were created by the move to the new db. Basically, assuming this ultimately fixes it for everyone... This was a weird circumstantial issue only affecting a single node and only sometimes. Fun. I did promise some chop ;)
We did run an extra database backup a while ago, and that message you pasted is exactly what shows while we're running a database backup. So it's possible you just happened to load the site while that was running. The site will show that message for 10-15 minutes while the database backs up. Normally that happens at midnight PST but I had to run one at an unusual time after the migration.
Let me know if you continue getting that message any time other than 12am-12:15am PST :)
Let me know if you continue getting that message any time other than 12am-12:15am PST :)
Matt wrote:
FYI if you just got that same message again in the last 15 mins or so it was another backup running. Still adjusting things on the new database server :)
Don't the backups usually happen around 3 hours before the current time (11:00 UTC+0)? That's when I always have ~5 minutes of downtime.
Correct, they normally happen at midnight PST. However there was a problem with them tonight and they didn't finish properly so I had to go in and make some adjustments and run it again to make sure it was working right (and to make sure we have a backup for tonight). Issues with the new db server. It ran fine this time so HOPEFULLY that's the end of it and we're back to normal :)
You need to log in before commenting.
<Perfectionist>