You know what we need? We need a full-time network administrator working for us. Anyone interested?
Why, you ask? It turns out (according to our subnet cheat sheet) that we pigeon-holed ourselves into a very limited number of machines when we set up our production network. We found this out when we tried adding another machine to the network and found that it didn't work. Scrounging around for the answer, we figured out that we needed to change the subnet mask of all of our machines.
Philip and I came up with a plan of attack. He and Randy would head over to our colo and I would remote desktop into the machines to change their subnet masks. Those of you who know networking are probably already groaning but... changing the subnet remotely should not have been considered an option at all. Doing so disconnected me from the machines so I was stuck in some middle state, unable to configure the machines and bring the site back up! Philip and Randy had to deal with it on site.
A bit relieved that that task was off of my shoulder (it's nerve-wracking; you don't know if you're going to bring the entire site down or not), I went back to work, implementing the kick butt features that we're about to roll out. I get a call from Philip after several minutes, asking me to try hitting our site. It didn't work. He called me a few minutes later, asking me to try again. It didn't work. After a few more calls, Philip and Randy finally figured out the problem and we got Twango back up again... except, we can't VPN (remotely connect) into the machines anymore.
That last issue hasn't been fixed yet but we're working hard on it. Fortunately, it doesn't affect Twango for all you Twangons (Twangonese? Twangoeuros?) so you're all good to enjoy the goodness of Twango while we figure out the problem.