Split-Testing and Late-Night Coding

When testing a feature wrapped in a spiffy-keen augmentation of an experiment system live, please remember to take into account that you have a 50% chance to land in the “nothing happens” side of the experiment. This will probably save you a lot of trouble.

Experiments!

Split Tests are a very powerful way to determine if a thing you are doing is in your best interest or not. If you aren’t familiar with A/B tests, the very high level is that you show half of your users one thing, half another thing, and you compare the results of each set of users.

Humanity

One of the more mundane, human problems you can run into as an engineer is tiredness. Tunnel vision sets in, and you focus on specific tasks. If you’re solving a complex math problem, you make an arithmetic error. If you’re a coder not using TDD, often you’re running into syntax errors or the like.

These things happen.

Process

At IMVU, we deploy code fifty times a day. The code you just wrote goes out to the production cluster without waiting for QE people to sign off on it, or for your manager to approve it.

Engineering under TDD acts like a blanket to shield you from a lot of problems. You know from the get-go if you broke something in a more interesting way (assuming said interesting way was protected by a test). The tests exist to protect you before you get near the production systems. A failed test on your buildbot is a good thing, because that’s a broken thing your customers never had to deal with. As someone who had jobs where he was tacitly encouraged to code untested fixes on the fly on the production machines in other jobs, this is a godlike boon.

However, things go wrong, and sandboxes and production environments still can disagree! This is why the final step of pushing code live to a production server is always, always, always verify your change manually out in the live website.

And if it’s not verified (or if your change breaks things in very unexpected ways, causing die spikes or high database load or the such), to revert to a previous known good state and investigate further.

Moral

Now that all of the background for tonight is set up, I’ll skip the actual stupidity, and jump straight to the conclusion:

When testing a feature wrapped in a spiffy-keen augmentation of an experiment system live, please remember to take into account that you have a 50% chance to land in the “nothing happens” side of the experiment. This will probably save you a lot of trouble.

It was a pleasant surprise when I looked up to see a wonderful forest around me. I’d just been staring at this one tree over here.

The other kind of Restful

The fun part is my team’s Quality Engineer, who was familiar with this feature and had tested it in production live before, was also up working. He didn’t catch the obvious either, because it was so obvious.

Say a human has a 10% chance of making a mistake/forgetting something obvious. Say he’s got a pairing partner with the same ratio. Assuming an ideal world, you’ve reduced your net error ratio to 1% by having another dude on the case with you. Yay pairing!

If both dudes are tired, their error rates go up, and even pairing, you still have a higher chance of fucking something up. Boo tired!

…Perhaps there’s a better moral than the one listed above hidden in here?

IMVU is 3d avatar chat; it’s also a pride-inducing piece of software engineering.

Guy whose “About me” page says “I currently work full time on my blog” mocked the place I work for not being a cure for cancer.

Twitter conversing

codinghorror: If I worked on “IMVU: 3D Avatar Chat Instant Messenger & Dress Up Game”, I’d be too embarrassed to blog about it, frankly.

codinghorror: He’s like “our deployment is sweet!” I’m like “dude, you’re deploying a 3d chat game for tweens.” Congratulations, I guess http://is.gd/j4Bh

antumbral: @codinghorror Feel free to ignore the lessons learned by industry leaders like Nexon just because their customers are younger than you.

codinghorror: @antumbral well, let’s just say 3d chat avatar dress-up software was not the cure for cancer I had hoped it would be.

Prestemon: Guy whose “About me” page says “I currently work full time on my blog” mocked the place I work for not being a cure for cancer. Speechless.

What, really?


Jeff Atwood/Coding Horror

What Jeff Atwood is missing the complete point of (in the second tweet) is that the deployment system is sweet, and a successful “3d chat game” requires some amazing pieces of engineering required to pull it off. We have a 3d desktop application, a chat service, a giant catalog, a very high-traffic site, and an always-increasing number of persistent users.

Shame?

As far as what IMVU is: IMVU is a service supplying a need, and making people happy in the process. A lot of people. It’s an MMORP without the rules-based G.

I can speak, however, to being a software engineer at IMVU in much more detail, as I spend many more hours engineering than I do as a user of the service. The number of interesting things to do at IMVU as an engineer is endless: Do you want to work in C++ today and play with the deep guts of 3d implementation and optimization? Would you like to work in python and use a really cool task system? How about some Flash that’ll be used daily by a horde of customers, or the infrastructure to allow those customers to make (and sell) their own flash for other customers to use in the product? Interested in scalability problems and other large-scale optimizations or the infrastructure and strategies needed to allow for making changes to a truly titanic amount of data? Or maybe just work on new and better things that will make customers, real people, happy?

Want to be in a functional business environment yet, as a team of engineers, ship code fifty times a day to a live and heavily used service?

We do these things every day. It’s engineering candyland.

I work at IMVU, and I’m damned proud of it mister Atwood.

Battle A-go-go

Battle A-go-go

 


Woo, technology.

So here’s some of the fancy shenanegans I’ve been up to lately with Gayo on Sully. I’ve been working on putting the render stack under test to hunt down some obscure bugs and mystery crashes, and that’s been going nicely. Lesson learned: FreeImage on a bad index is bad… so I augmented the verge3 engine to noisy-error in that circumstance: although we might eventually just want it to do a noop instead of doing a fragile error or doing a bad operation.

I’ve also made a bit of progress on the http://beta.verge-rpg.com site. I should work out a good stylesheet for the forums and be done with that major third of the site… although really I need to make the forum management UIs. Which are just two screens, as far as I can tell…

Nothing’s been happening on the iPhone front in the last week. I’ve been sorta really busy at week getting IMVU’s Music Portal whipped into shape. And shape has been whipped into it. So all is well now.