Official 4/12 Service Level Explanation and...

[quote=Zeprido]*SERVERS BACK ONLINE AS OF 9:15PM PST* For those who rarely traverse main LoL Forums! As can be found here: [url=http://www.leagueoflegends.com/board/showthread.php?t=654620]http://www.leagueoflegends.com/board/showthread.php?t=654620[/url] Hello summoners, As you all have noticed (or at least heard about) by now, we have been experiencing many service affecting issues concurrently and systematically that have lead to an exceptionally long downtime this patch day. While we are currently working diligently to resolve these issues, we feel you all deserve a better explanation of what happened today and why you have been unable to log in and play. Given how vast this mornings complications have been however, that list is very long and will take a while to compile. As such I will work regressively, starting with frequent updates to what is happening right now and slowly add in the opposing direction with what has already happened and been resolved, and how we handled it. (PST) 7:38pm - Organizing Rioter stream to entertain you all during the downtime. 7:28pm - The Tomcat heap went out of memory. The servers will be down for a minimum of 2 hours. Everyone who can possibly be involved in efficiently resolving this issue is on the case. 7:18pm - Patcher set to Unavailable while we diagnose. 7:13pm - Button pressed. 7:07pm - Reports of massive game lag. 7:03pm - Games not starting. Login failures. Unknown errors. Investigating. 6:44pm - Bear brought me a box of chocolate covered rice crispy treats. Awesome. 6:18pm - Everything still running smooth. Button disabled. 6:04pm - Everything still looking good so far. Everyone who was stuck in champion select will need to restart their clients in order to play again. 5:45pm - All systems are back online. Monitoring as we funnel players back in. Login queue time around 20 minutes. 5:31pm - One of the new boxes that was added to our server cluster failed and promptly removed. Queue time is currently 25 while we filter players back in. 5:01pm - We are currently having login and game start issues. 4:51pm - Chat, platform, and server issues presumably resolved. 4:47pm - Website back online. 4:36pm - Web issue identified as an issue with the firewall. 4:33pm - Chat issues, platform issues, website issues, and server issues all simultaneously being triaged. 4:30pm - Button pressed for North America. 4:23pm - Confirmed reports of platform issues. Summoner disconnects, crashes, and login failures. Began triaging live issues. Website back up. 4:10pm - Still working to remedy web issue. Players are able to login to the North American platform fine. Web issues not affecting PvP.Net. 3:55pm - Site extremely slammed. Put up the maintenance page to alleviate the server workload while we continued triaging. 3:47pm - Still triaging. Cause still elusive as the forums are serving intermittently, but most of the time inaccessible. Apache server began choking up. 3:38pm - Triaging of web servers in progress. Many boxes are responding while some are not. (Which made this very difficult to pinpoint and is also why some people were able to visit the website while others were not.) 3:30pm - North American forums and website went down. Investigation proceeded. 12:30pm - Audit complete. Corrupt data found in several locations. Downtime extended to 2p.m. 9:35am - Patch deploy took much longer than expected. Began patch audit. 3:04am - European servers explode (not literally). Not going to go into too much detail here but the issues that affected EU resulted in a considerable amount of our operations and production staff (the same staff that has been triaging NA all morning as well) were called in to address the issues. I'm going to bring them cookies tomorrow. 3:04am - All users out of games. 1:34am - Autoshutdown initiated for scheduled downtime." *Should be read from bottom up for timeline. *Will update with new information [u] [i][b]Updates since posting on Mobafire:[/b][/i][/u] 9:15pm - Servers back online! Logins may be slow as players funnel back in. 8:10pm - Tune in [url=http://www.own3d.tv/live/8663/Serif]here[/url] for some Rioter livestream (on EU). (Will go live in 5-10 minutes) ([i]has since been removed from posting and does not exist[/i]) 8:06pm - Initiating restart. Estimated time that the servers will become available: 10:00pm PST. 7:49pm - Website experiencing intermittent timeouts due to excessive traffic.[/quote]

[quote=SinDelta]Well at least this subsided my super rage lol. Thanks for the info now i can go tell everyone. Not sarcasm btw lol im being sincere.[/quote]

[quote=JunSupport]Well, at least they're explaining themselves. Not like the money-grubbing retards at Nexon. ( They would extend patches for hours, and even do roll-backs on MapleStory with little to no explanations; terrible hack-shields too. ) 10pm pst... -grabs some popcorn for a movie-[/quote]

[quote=Riggsinator]At first I was disappointed that I couldn't be on at all today because I was doing an endless assignment. Come home at Midnight just now. Try to do the patch and log on for one quick game. [img]http://cdn0.knowyourmeme.com/system/icons/5180/square/Y0UJC.png?1298577746[/img] Not sure why they don't keep doing what they did on the April Fool's week. Have the real patch on patch day, then add the new champion in a few days later. Lets the Server get chuggin' before the zerg of "ZOMG NEW CHAMPION" people flooding the gates for the next two days then going back dormant.[/quote]

[quote=Zeprido]Supposedly the servers are back online for those wondering. "9:15pm - Servers back online! Logins may be slow as players funnel back in."[/quote]