Bloodpact Blogging

Chad Austin: Your Version Control and Build Systems Dont Scale

Thu, 04 Mar 2010 08:02:18 +0000

Let’s spend a minute and talk about the performance of a tool you use every day: your build system. For this purpose, they’re all the same, so let’s assume you use GNU Make. For a no-op build, what is Make’s running time? You know, that O(N) stuff.

If you said O(Files), you’re right. Every time you type make and press enter, it scans every file in the project, looking for out-of-date sources. That doesn’t sound so bad. Linear algorithms are good, right?

Oh, but projects grow over time, typically in proportion with the number of engineers (even if some are net negative). Since projects grow:

O(Files) = O(Engineers * Time)

Assuming you’re successful, your team is probably growing too:

O(Engineers) = O(Time)

Applying some substitutions:

O(Make) = O(Files) = O(Engineers * Time) = O(Time^2)

A no-op make is quadratic in the duration of the project?!

Exhibit A, Mozilla:

Chad@laptop /c/xulrunner/mozilla-central/obj-xulrunner
$ time make
rm -f -rf ./dist/sdk
[tons of output snipped]
make[1]: Leaving directory `/c/xulrunner/mozilla-central/obj-xulrunner'

real    21m31.526s
user    4m13.292s
sys     9m15.066s

21.5 MINUTES! This is a no-op build! I changed nothing from the last build!

Mozilla’s build is an especially egregious example, but it’s not uncommon for nontrivial Make, SCons, or Jam no-op builds to exceed 15 seconds. Every time you run SCons, for example, it:

loads Python,
imports a bunch of Python code,
creates an entire dependency graph in memory,
scans all files for changes,
determines what to build,
builds it,
and throws all of those intermediate results away.

Am I the only person who thinks this situation is insane? We’re spending all of our time improving constant factors rather than addressing fundamentally quadratic build algorithms.

No-op builds should be O(1) and instantaneous, and most other builds should be O(WhateverChanged). What if I told you this is possible?

Introducing IBB: I/O-Bound Build

Dusty, one of the best engineers I know, once told me that C compilers are now fast enough that they’re I/O-bound, not CPU-bound as you’d expect. This was inspiration for a ~~build system~~ dependency engine architecture:

A long-running server process that contains a complete dependency graph between files, including the commands required to update targets from sources.
An ultra-tiny C program that communicates build requests to the build server.
A background thread that watches for filesystem updates (via ReadDirectoryChangesW or libfam) and updates the dependency graph in memory.

I have built a prototype of ibb’s architecture, and its code is available on my github.

In ibb, a no-op build takes constant time, no matter the size of the project. I tested with Noel Llopis’s build system stress test and no-op builds are merely 50 milliseconds. Take that, everyone else!

A Faster Grep

Let’s say you wanted to run a regular expression across your entire codebase. On Windows, you’ll spend all of your time in kernel calls, even if the files are in cache. However, remember that modern memory bandwidths are measured in GB/s…

ibb’s file search implementation can run a regular expression across 200 MB of PHP and JavaScript in 100 milliseconds. It copies file data into memory on the first run and rapidly scans across that memory on future runs. It could just as easily mmap and use OS’s disk cache to avoid kernel calls.

Minimal Test Runner

Carl Masak’s inspirational addictive TDD harness is also possible with ibb. If you can determine which unit tests depend on which source files (via an __import__ hook in Python, say), you can run appropriate tests immediately after saving a source file.

Version Control

I’ve watched both git and Subversion fall apart as a codebase grows. git is so faaast… until you import a 20 GB repository. Then it’s O(Files) just like everything else.

$ time git status
# On branch master
nothing to commit (working directory clean)

real    0m9.603s
user    0m0.858s
sys     0m7.378s

10 seconds to see what files I’ve modified. Ouch.

Imagine if ibb could keep track of which files have changed…

What This Means For You

Running tests, grepping code, and building binaries from source are equivalent problems. ibb’s architecture supports lightning-fast implementations of all of the above.

By converting an O(Time^2) build, test runner, or version control system to O(1), we eliminate hours of costly engineer time and encourage good habits like committing often and frequently running tests, no matter the age of the project. Frankly, instant feedback feels great.

It’s crazy to me that this architecture isn’t standard practice across development tools.

Take ibb’s concepts or code, integrate it with your engineering processes, and improve the quality of your team’s life today.

Michael Rooney: Why I Switched from Ubuntu One back to Dropbox

noreply@blogger.com (Michael) — Mon, 22 Feb 2010 23:01:53 +0000

As many of you surely know, Dropbox and Ubuntu One are applications to keep files on your desktop in sync across multiple computers, and backed up in the cloud. After using Dropbox for a year or so on Ubuntu, the Ubuntu One project came out and I thought I'd move over to it. I assumed it would be easier to set up, being pre-installed, and could integrate better with the file manager and other applications, being made specifically for Ubuntu.

After about 6 months of using Ubuntu One, I found it to be too much of a regression compared to Dropbox and switched back. I thought I'd detail why here for the usefulness of others and to hopefully provide some constructive criticism to the Ubuntu One team which overall is doing good work. So here's why I went back to Dropbox:

Better Nautilus integration
The nautilus (file manager) integration in Dropbox feels very mature and polished. Normally I'd say this is to be expected since Ubuntu One is much younger, but since the nautilus aspect of Dropbox is open-source, there didn't seem to be much of an excuse for the Ubuntu One team to not use it as a starting point or at least as inspiration. Whenever I added files to my Ubuntu One folder, no matter how large or how many, they instantly had the checkbox emblem which implies to me they are in sync, even though they couldn't possibly be uploaded that quickly. In Dropbox, the files show an animated progress emblem until they are actually uploaded, and show this again when they are being updated. I wasn't able to trust the status of files with Ubuntu One, and that wasn't a good feeling.

Sharing files in Dropbox is also a lot easier via Nautilus, but that deserves it's own point.

Easier file sharing
In my typical use cases of file sharing, I want to go from having a file in mind to someone else seeing that file as quickly as possible, be it in an instant message, chatroom (IRC/Jabber), or email to a friend or two. Dropbox makes this a breeze; drag a file into the "Public" folder in your Dropbox directory, and right-click on it and select "Dropbox > Copy Public Link". Now you have a publicly accessible link to your file in your clipboard!

I never quite figured out how to do this in Ubuntu One; it seems you have to share files with specific people who also are running Ubuntu One (which really compounds the non-cross-platform issue) via the web interface by typing in an email address. This is a cool idea, but seems way over-engineered as a starting point. This was even brought up at the last UDS in an Ubuntu One session but was brushed off; they sadly seemed more interested in engineering complex sharing UIs than getting feedback to ensure they were solving actual problems.

Sane notifications
I think bug 462747 is what ultimately drove me to drop Ubuntu One. It notifies you not once but twice for every file you change; once to tell you it is syncing the file and another to tell you it is done. Regarding the first notification, I probably already know that I changed that file; regarding the second, if I'm really curious about the status of the synchronization, a quick glance at the applet should tell me what I need to know. This was mildly annoying as is, but it does this for hidden files like vim swap files too. Every time I would open or save a file in vim (and I save early and often), I got two notifications about syncing the hidden swap file!

Dropbox was much more sane regarding notifications; whenever I would turn on or sit down at a computer, it would show one notification telling me how many files it synced from other machines or, if there was just one changed file, the name of it. Beautiful!

Better web UI
The web user interface for Dropbox felt a lot easier to use, and I often had problems where the Ubuntu One web view would show deleted files or not show files I knew were there that were added days ago. Sometimes I had to collapse and expand a folder a time or two to get it to show the right contents. This led to a similar problem that I mentioned with the nautilus integration; it wasn't a UI I could trust to be accurate as was therefore essentially useless. The Dropbox UI was always accurate, and had some nice extras which could prove to be lifesavers like getting past versions of files.

I also experienced some other issues including bug 498444 which caused Ubuntu One applet to start up with the exclamation icon, giving the impression of not syncing files. If I wanted to edit or view a file that might have changed remotely, I had to manually tell the applet to try syncing again after startup to make sure the file was latest version (or else I could silently end up with .u1conflict file, but that's another issue). Additionally, although the transparency is praiseworthy, all these tweets about recent issues or downtime do not inspire confidence.

That being said
Ubuntu One does have a few things going for it, however. It is installed by default which means it is the easiest cloud sync/backup solution to get started with for an Ubuntu user. And thanks to CouchDB and a package manager, Ubuntu can ship applications that make it trivial to sync their data with Ubuntu One out of the box. If I can tell all the applications I care about to sync with Ubuntu One (bookmarks, notes, podcasts, basic OS settings/appearance) from one configuration UI, that's going to be pretty compelling.

While I currently can't in good conscience recommend Ubuntu One to a friend (and plenty of my friends don't even use Ubuntu), I do have high hopes for the project; it is young and if they can work out some of the above issues, many of them being personal show-stoppers, while providing application and OS integration, Ubuntu One (and perhaps Ubuntu itself) could become too compelling to not use for many people. For now however, Dropbox is the solution which stays out of my way, allows me to solve my problems, and just works.

If you are using Ubuntu One or Dropbox, certainly chime in on what the critical features are for you and why you chose one over the other!

Jeff Lindsay: The latest from Wozniak

Wed, 17 Feb 2010 05:33:25 +0000

"My latest idea that I've been expressing is that there's no real value to humanity in creating technology like computers. What really matters and makes a difference is when you do something nice for someone else, like tying the shoe of a child. Or bringing smiles to any stranger."

Chuck Rector: Poisonous devices

noreply@blogger.com (卡车 Chuck) — Sun, 14 Feb 2010 11:46:00 +0000

Phones

Sometimes, it's nice when friends and family call. I've even been pleasantly surprised by strangers before. Other times, people call me for the wrong reasons and at the wrong times. I used to believe that when phones ring they must be answered, but that's no longer the case. A phone is an assistive device. If it's not assisting me then what value does it provide?

I used to be terrified of phones. After a part-time job which required a lot of phone time with strangers, that terror went away. Except that in hindsight I don't think it did. It was always stressful. I simply grew to accept that stress.

Over the years I've been tuning my phone usage. I used to answer every call because it was about facing my fear. Then I answered every call because... that's just what you do. It's a phone! Then I realized that it's not my job to answer the phone. This was an important realization for me, because that part-time job had programmed me to believe that it was. I began answering the phone less and less.

These days, I hardly ever answer my phone. One huge source of stress: gone.

This sometimes means I miss calls from friends and family, but ultimately the people I know and love have many ways to reach me. If a chat on the phone is absolutely necessary, it will happen. Eventually. If it's a call from a stranger about random crap I don't care about, guess what: I win. You are unlikely to call me again.

One of my favorite things to do is to turn off my phone. Especially when I go to sleep. Often I'll turn it off when I get home after work as well. And it's fantastic. Especially the part where I'm never jarred awake in the middle of the night.

Clocks

I used to have a watch and I would look at it all the time. If I'm at the bus stop waiting for the bus, I'm checking my watch. If I'm at work and wishing that lunch would come, I'm checking my watch. If I'm wishing work is over so I can go home, I'm checking my watch. If I'm waiting for a friend to come over, if I'm waiting for a show to come on, if I'm trying not to stay up too late...

I decided that looking at clocks is stressful. How about I just change how I'm living so I don't need to look at them? That's what I do now. I engineer my life to alleviate stress and to optimize fun and learning.

For example:

I don't look at bus schedules anymore. Instead, I get enough sleep, wake up refreshed, and walk over to the bus stop when I'm good and ready. This is usually hours before I need to be at work, so I'm never in a rush. I listen to music while I wait, enjoy the cool morning air, and watch the birds while I munch on an apple. Some mornings are foggy and wet and some are bright and warm. I notice and appreciate them because I'm not late for work, I'm not tapping my toe, and I'm not constantly looking at my watch.
There are dozens of ways to alleviate stress at work, and I ruthlessly identify and crush them. As a result, I'm not wishing lunch would come quickly so I can get a moment's relief. I'm not wishing the day would end so I can go home and get a few more moments of relief. I've made my work incredibly satisfying. When lunch rolls around, it's usually a happy surprise. Same thing for the end of the day. So much so that I come into the office on weekends because it's an enjoyable part of my life that I want to experience daily.
It turns out that once I alleviated so much stress in other parts of my life, I also alleviated boredom. For me, boredom was the end result of living in a constant state of stress. This may not make immediate sense, but when I was in a constant state of stress, I was always doing one of two things: stressing or recovering. In that mode of living, I was never actually spending time on myself, either by having fun or by feeding my mind with fulfilling personal learning. Even when I thought I was doing these things, I really wasn't. In that severe state of stress, I believe there was no fun or personal learning. There was a heavy weight which pushed me down and which permeated everything I did. What I thought was fun or learning was instead "escape". Enter boredom: when escape velocity was not sufficient to outmatch stress.

Whenever I think about clocks, I think about the rabbit from Alice in Wonderland. I don't want to be That Guy, running around, worrying his head off, with precious little time to spend on himself. I'm valuable to myself, so I make plenty of time for me.

Alarms

Alarms are like clocks, plus a punch in the face. If I don't like waking up in the middle of the night when I'm not done sleeping, then I don't like waking up in the morning when I'm not done sleeping either. How about I just get enough sleep so that I don't need an alarm?

I have also learned that caffeine is the catalyst for my own personal hell (another poisonous device!) It kicks off a vicious cycle. I drink caffeine. I am amped. I come down off the high. I drink more. It's late and I'm full of energy and my body's burning calories so I eat. I now have even more energy. I stay up late. I have to get into work early so I set an alarm. Too few hours later, my alarm punches me in the face. I drink caffeine to jump start my system. And away we go...

As I mentioned, I turn my phone off when I go to sleep. It's not entirely a coincidence that my phone is also my alarm (iPhone.) I turn it off with purpose. When I'm sleeping, phone calls and alarms are equivalent. Depriving myself of sleep does not improve my life. I control my phone, therefore I choose to improve my life by turning it off when I'm sleeping.

Alarms should be used for alarming circumstances. Like fires. Not burning to death is a good way to add value to my life. Other alarms, not so much.

Chuck Rector: Right the wrongs

noreply@blogger.com (卡车 Chuck) — Fri, 12 Feb 2010 00:03:00 +0000

It's difficult and painful and time-consuming to switch from a habitual lifestyle of bitching over to one of fixing, but it's possible to change. My first impulse when encountering foreign and confusing code has always been to complain about it to someone, immediately. It's such a dumb and mindless thing to do, but I've done it a million times. On some level it's about camaraderie. Wah-wah, it sucks, let me cry on your shoulder. You understand how awful this is, right?

My first attempts at correcting this behavior focused on the wrong thing: demeanor. It's not bitching if I'm calm, collected, and diplomatic when I say it right? It's okay if I try to trick myself into thinking I made an attempt, but was mostly silently seething to myself, and then came to you for help right? What a passive-agressive waste of someone else's time. There's help and then there's hand-holding. Instead, my focus needed to be on solving problems, not the injustice of hard ones and trying to foist them on others.

This video http://www.youtube.com/watch?v=k2h2lvhzMDc (Ed Catmull, Pixar) reminded me of useless bitching (the kind I envision "mediocre teams" would participate in) and that my programmed response in the face of difficulty should be one of reason and action rather than victimization. It's not that someone intended to punch me in the face with their code, it's that the target has shifted and I need to correct course. It's not about personal vendetta, it's about what works and what doesn't work.

This is my reminder to myself to be intelligent and thoughtful and to right wrongs rather than wallow, sidestep, or avoid. I righted wrongs today, and it's awesome.

Michael Rooney: wxBanker 0.7: simple personal finance

noreply@blogger.com (Michael) — Sun, 07 Feb 2010 18:15:00 +0000

Your favorite personal finance application, wxBanker, has turned 0.7!

This release comes about 2 months after the previous release, and focuses on usability and user experience issues that I obtained from watching people use wxBanker as well as from Launchpad bugs (thanks Arty!). Let's take a look at some of the changes, starting with the account control:

On the right we have the new account chooser in 0.7. The main change is using radio buttons for the accounts instead of links. This is a much better, already understood method for choosing accounts, and will also fit in with themes better. The last item is now "All accounts" and is selectable, making it easy to get a view of all your transactions and search in them.

The "Hide zero-balance accounts" option has moved to the View menu, and now has a keyboard shortcut. I've also removed the total number of accounts from the header, as well as the colons after the account names, to reduce clutter. Finally, everything has been given a bit more padding and the buttons have been slightly rearranged.

Additionally, the previous graphing library has been replaced by cairoplot (thanks Karel!), which looks much more attractive. Let's check it out:

The Summary view allows you to see a graph of your balance over time, and you can view a graph of a specific account or all accounts by using the account chooser on the left (previously the graph had its own account chooser, that was silly!).

And in case you missed the 0.6 announcement, that version brought recurring transactions, XDG directory support, and more intuitive behavior regarding deleting/editing transfer transactions.

For downloads and the full list of features and bug-fixes, check out the release page. You can also add the PPA for easy installation and upgrades.

If you'd like to stay in the loop join the wxBanker Users team (and announcement mailing list) on Launchpad, follow wxBanker on Twitter, or hang out in #wxbanker on irc.freenode.net.

Chuck Rector: I am posting to my blog. Stop.

noreply@blogger.com (卡车 Chuck) — Tue, 02 Feb 2010 17:52:00 +0000

I am kind of amused by people who end their emails with their name. Your name is in my contact list. I can read the From: field. I know who you are.

Even more amusing is when they end it with a single letter. Is that supposed to be a signature? I don't get it. If you're going to use a signature, make it cool.

For example, there's this guy who "signs" all of his CSS with this:

c(~)

It's a mug of beer with a handle and sloshitude inside, see? Awesome.

Jeff Lindsay: On Knowledge

Tue, 02 Feb 2010 17:36:18 +0000

Despite conventional wisdom, knowledge is not just a collected body of facts. This is because that's exactly what information is, so knowledge must be something else. In fact, it's so much more.

First of all, I argue that knowledge must be validated to be knowledge. You might hear about wrong information, but rarely do you hear about wrong knowledge. In fact, you commonly hear knowledge associated with truth. This is because knowledge is acquired through experience, the validation of consistency against reality. As they say, you never know until you try.

I also argue that knowledge is not real until it is learned. In a sense, knowledge doesn't exist outside of a person. We talk about our collected body of knowledge, but we're really talking about people and the knowledge they have, or had when they were alive. Anything else is merely information, potential knowledge that has yet to be validated and internalized. Although we trust certain information to be validated "knowledge" since it was once acquired by someone else, it's only an assumption until you have learned it (experienced it) for yourself.

Whether you buy that or not, we all seem to agree that knowledge is power. This is revealing about knowledge when you consider that power is defined as the capacity to cause change. This implies that the nature of knowledge is biased towards the idea of how-to. Although we consider the declarative what-is-true knowledge of, say, mathematics, to be knowledge, it is only a shorthand for the proof behind it, and the process of deriving a proof is obviously a question of how-to.

The idea that knowledge is concerned with change and *how* things happen is important in contrast to why things happen. The fundamental difference between how and why is that how is concerned with structure and process, while why is concerned with function and meaning. The how of something can be determined without context through analysis. How does it work? Take it apart and find out! The why of something is highly dependent on context and synthesis. Why do we have xyz? You'll *never* know from taking xyz apart. Meaning comes from the outside. It accumulates not as knowledge, but as understanding.

A probably more significant implication of knowledge being about change and process, is its relation to computer science. Not computer science as in the specific field of study, but computer science as the essence of computation. Perhaps, philosophy of computation. What computation and programming are about is the notion of formalizing intuitions about . . . process.

Harold Abelson, co-developer of MIT's introductory computer science course, says computer science is not even so much about the computer. "When a field is just getting started, it's easy to confuse the essence of that field with the tools that you use." I think the big significance of computation is not computation at all . . . it's about a means to formally talk about knowledge itself.

We have general language as a formal expression of information and data, and we have basically programming and computers as a formal expression of knowledge and process. This leads me to wonder: what will we develop that allows us to formally express understanding and meaning? What kind of revolution will that be?

Jeff Lindsay: Learning from disparity, aka “failure”

Tue, 02 Feb 2010 07:32:14 +0000

I’ve long argued that failure is the only way a person can learn. This has been a very reasonating bit of wisdom for me, despite the fact people have often argued against it with rather sound logic. “Yes, you can learn from failure, but you can also learn from success!” Only after reading Jason Fried’s [...]

Michael Rooney: Automating translation template generation and check-ins for Launchpad

noreply@blogger.com (Michael) — Tue, 02 Feb 2010 01:35:52 +0000

I previously wrote about how excited I was for the automated translation import and export features of Launchpad. Launchpad will automatically notice when you commit a translation template and import it, making it available for translation online. Generous translators will then contribute translations, and Launchpad will commit them back to your project.

Okay, so this is pretty good! However for this process to work well, the translation template needs to be generated manually by a developer whenever there are changes to strings. Otherwise, translators are working on potentially outdated strings; some are perhaps not in the application anymore, and there are likely some strings which aren't in the template yet.

After forgetting to generate and commit my template until shortly before a release more than once (and thus having poor translation coverage on newly added strings), I decided to automate this part of the process as well. All it took was a relatively small script to generate the template, and then if there were any changes, commit and push to the branch configured for automatic import in Launchpad. The following script does just that, by searching for any files using gettext calls starting with _(" or _(', and passing them to xgettext.


#!/bin/bash
set -e
ack-grep "_\([\"\']" -l | xargs xgettext --output=wxbanker/po/wxbanker.pot
ACTUALCHANGES="`bzr diff | grep \"^[\-\+]msg\" | wc -l`"
if [[ "$ACTUALCHANGES" != "0" ]]; then
  bzr ci -m "automated generation of translation template"
  bzr push :parent
fi

Note that if you aren't using Python, you may need to tweak the regular expression supplied to ack-grep. Once the template is generated, the diff is piped through grep to grab any changes to actual messages and make sure there was at least one. Otherwise there would always be changes due to the timestamp in the template, causing useless commits.

I then threw this script into a Hudson job, the Continuous Integration server I use for wxBanker. I configured this particular job to run nightly, pulling down the latest bzr branch beforehand, and emailing me on any failures.

It seems to be working quite well and ensures translators are always translating the latest strings and leaves nothing for me to forget, smoothing out the release process.

If this sounds interesting to you but you're not familiar with Launchpad as a translation system, check out http://blog.launchpad.net/general/trying-out-launchpad-translations and feel free to ask any questions here. If you do have experience with translations, how do you handle generating translation templates and then integrating the translations?

Jessy Cowan-Sharp: Culture of Yes

Wed, 06 Jan 2010 21:30:45 +0000

Washington, DC’s recently released open government directive has a lot of us in the open government community stoked about the mandate we are finally being given, collectively and formally, to make government more transparent and accessible. The three tenets of participation, transparency, and collaboration are particularly relevant because, while they are couched in specific deliverables [...]

Chad Austin: Comparing Final Fantasy III to Final Fantasy IV

Tue, 05 Jan 2010 03:57:31 +0000

Having grown up with Final Fantasy IV on SNES, I recently enjoyed playing through Final Fantasy III. For my own edification and those curious, I’d like to note some of the design similarities between the games, providing color on the evolution of the Final Fantasy franchise.

Airships are a primary theme, both as transportation and cultural status.
Final Fantasy 3 introduced the character class system in a big way, setting the stage for the classes used in FF4’s story.
Desch and Yang provide similar service to their respective stories: help the party for a while, followed by dramatic sacrifice.
FF3 introduced the Toad and Mini status effects as first-class gameplay elements. Toad allows you to swim and Mini allows you to fit in tiny passages. While Mini, your attacks are useless but magic still works. These status effects remained in FF4, although vestigial. (I always wondered why “mini-mages” were so common in FF4. Without having played FF3, they didn’t make sense.)
FF3 introduced the Leviathan, Odin, Bahamut hierarchy.
Eureka (FF3) paralleled the Lunar Subterrane (FF4). They’re both extremely hard when you first enter, but they force the party to gain about ten levels. In addition, they contain several smaller bosses and a bunch of powerful equipment. Both provide a sense of progress and accomplishment, without feeling like grind.
The last areas and bosses of both games are unusually difficult. They each took me multiple attempts to complete.
In both final battles, previous allies provide “spiritual assistance”, making it possible to defeat the overpowering final boss.

However incomplete this list, these are the similarities I noted. Final Fantasy 3 had not quite discovered the Final Fantasy “sauce”, but it was an important step along the way.

Chad Austin: The Completionists Guide to Sims 3 iPhone

Tue, 05 Jan 2010 03:56:25 +0000

I play simulation games in two phases: first, I tackle the often-repetitive mechanics, unlocking every option and building up money so that I can creatively decorate my house or farm or whatever.

Sims 3 for iPhone was no different. Having accomplished nearly everything in the game, I will share a strategy for completing all 73 goals and acquiring the best job: $600 per day at the Pawn Shop.

The Objective

To open the pawn shop, you must complete the following 73 goals:

Goals for all Sims (55)

Try fishing
Try cooking
Buy fishing kit
Buy watering can
Buy repair kit
Buy a stove
Buy a bath
Gain a skill point at cooking
Gain a skill point at fishing
Gain a skill point at repairing
Meet a new Sim
Befriend a Sim
Begin a romantic relationship
Make an enemy
Make a Sim laugh
Annoy a Sim
Insult a Sim
Creep-out another Sim
Slap a Sim
Get a job
Buy something
Catch a fish
Catch a trout
Catch a salmon
Catch a catfish
Repair something
Discover a new recipe
Cook something
Cook grilled cheese
Cook steak & veggies
Cook minestrone
Grow something
Grow carrots
Grow corn
Grow tomato
Watch TV
Kick over a Trash Can
Sleep in another Sim’s bed
Use another Sim’s shower
Use another Sim’s toilet
Get a better couch
Get a better TV
Accumulate $1000
Catch 15 fish
Harvest 30 crops
Stay entertained for three days
Stay fed for three days
Stay rested for three days
Stay clean for three days
Meet all the Sims in town
Make a Sim jealous of you
Sleep in three other Sim’s beds
WooHoo with someone
Get a promotion
Reach the top of the career ladder

Maniac Personal Goals (4)

Use everyone’s toilet at least once
Use everyone’s shower at least once
Creep out five people
Watch three people sleeping

Sleaze Personal Goals (2)

Be romantically involved with three Sims
WooHoo eight times in one day

Power Seeker Personal Goals (3)

Accumulate $5000
Own the best house
Own the best TV, stereo and stove

Nice Guy Personal Goals (2)

Get married
Be liked by all the Sims in town

Jerk Personal Goals (4)

Be disliked by all the Sims in town
Slap four people
Insult five people
Kick over all the trash cans in town

Jack of All Trades Personal Goals (3)

Achieve the highest fishing skill level
Achieve the highest repairing skill level
Achieve the highest cooking skill level

There are six Sim classes:

Jack of All Trades
Nice Guy
Jerk
Sleaze
Power Seeker
Maniac

As previously mentioned, you will need to complete every Goal to unlock the Pawn Shop. Of the 73 total Goals, 55 randomly appear as Wishes, no matter which Sim class you choose, and they’re generally easy to complete.

On the other hand, each Sim’s Personal Goals are relatively difficult. You’ll need to create at least six Sims, one of each type. For the optimal path through the game, you should complete their personal goals from hardest to easiest. I will give an efficient play sequence below.

Play Order

Jack of All Trades

Create six Sims, and designate one as your “main” Sim. Your main Sim is the one with which you most identify.

Your main Sim is a Jack of All Trades (you’ll later need its Repair skill at the Pawn Shop). Play him or her first, building up skill levels as rapidly as possible. Learn to fish, because fishing is the best way to make money. As you accumulate money, buy as many of the cheapest stereos you can afford, filling your house. Each time you enter the house to eat or rest, turn them all on. Eventually, they will start breaking down, allowing you to practice repairing.

Practicing cooking is easy – simply buy bread, cheese, and the Grilled Cheese recipe and make it over and over again.

Once fishing, repairing, and cooking are level 5, save, quit and move to the next Sim.

Power Seeker

Your second Sim is your Power Seeker. Fish until you have $5000 and tool around town until you have the option to upgrade your house twice and buy the most expensive TV, stereo, and stove. Furniture and home upgrades simply take time. This is a good opportunity to accumulate general Goals.

At this point, you should have most of the 55 general Goals.

The Other Sim Types

Sleaze, Nice Guy, Jerk, and Maniac have fairly easy Personal Goals. The order in which you complete them does not matter.

At this point, you should have almost all of the Personal Goals. If not, keep playing — they’ll eventually show up.

Goal Tips

If you need the “Buy XXX” goal but you already have XXX, then try selling it. Eventually the Goal should appear as a Wish. Lock it in and repurchase the item.

If you need the “Gain a skill point at XXX” goal and you’re maxed out, switch to a Sim that is not maxed out.

If you’re having trouble Meeting all Sims, double-check all of the houses at different times of the day. I’ve noticed Bernie can be hard to find. Most Sims are sleeping in the middle of the night – try breaking into their houses, waking them up, and meeting them.

If you need to befriend a Sim but you’re already friends with all of the Sims, start a new Sim or insult a Sim until it becomes an enemy, then befriend him or her.

The best way to annoy, insult, and creep-out other Sims is to find a house with two Sims (e.g. Marcell and Theresa), barge in, and start flirting with one of them. Both will be creeped out and annoyed, and the flirting will insult the other and make him or her jealous. If that doesn’t work, start using their toilet, shower, and fridge.

Tips and Tricks

Fishing is the best way to make money. Each tuna sells at QuickMart for $100, and salmon for $65.

Besides the Pawn Shop, working at the Town Hall is the best. It’s easy to get promotions (both Ruth and the Town Hall are nearby) and you just need to be friends with everyone in town. As Vice President, you can make $300 per day.

Creating romantic relationships is easy. It sounds rude, but keep pestering the Sim with Flirt, Tender Embrace, Hot Smooch, and WooHoo. You should be able marry a Sim within two conversations.

Too many WooHoos and your Sim will die! Don’t overdo it in the bedroom!

If you get married, give your spouse a phone call and invite him or her over. Once your spouse arrives, you can invite him or her to move in.

Since the Town Hall is right by your house, it’s the best job until the Pawn Shop is open.

The Sims

Nina
Ruth
Johnny
Kia
Jake
Maggie
Jack
Jill
Walter
Luke
Bernie
Marcell
Theresa

The Jobs

Town Hall (boss: Ruth)

Campaign Intern ($100/day) M-F 8:30-18:00
City Council Member ($150/day) M-F 8:30-18:00
Local Representative ($200/day) M-F 9:00-18:00
Mayor ($250/day) M-F 10:00-18:00
Vice President ($300/day) M-F 10:00-17:00

Corsican Bistro (boss: Marcell)

Kitchen Scullion ($100/day) M-F 10:30-18:30
Ingredient Taster ($150/day) M-F 10:30-18:30
Line Chef ($200/day) Tue-Sat 11:00-17:00
Sous-Chef ($250/day) Tue-Sat 11:00-17:00
Chef de Cuisine ($300/day) Tue-Fri 11:00-17:00

Laboratory (boss: Kia)

Test Subject ($100/day) M-F 9:00-16:00
Lab Tech ($150/day) M-F 09:00-16:00
Fertilizer Analyst ($200/day) M-F 09:00-16:00
Carnivorous Plant Tender ($250/day) M-F 09:00-16:00
Genetic Resequencer ($300/day) M-F 10:00-16:00

Quickmart (boss: Bernie)

Filing Clerk ($100/day) M-F 08:30-18:30
It’s not worth getting promotions at Quickmart. Bernie is a dick and you have to work there FOREVER before he’ll promote you.

Pawn Shop (boss: Nina)

Con Artist ($300/day) M-F 16:30-23:55
Safecracker ($400/day) M-F 16:30-23:55
Cat Burglar ($500/day) M-F 17:00-23:55
Master Thief ($600/day) M-F 17:00-23:00

In the End

With the information given above, you should have no problem achieving 73 goals and as much money as you need to decorate your dream home.

If you get stuck at any point, feel free to leave a comment on this page and I’ll update the document.

Chad Austin: Avoid Manual Initialization

Fri, 01 Jan 2010 07:20:39 +0000

Too many software libraries require that you manually initialize them before use. As illustration:

library.initialize()
window = library.createLibraryObject()

Why require your users to remember initialization busywork? createLibraryObject should initialize the library, refcounting the initialization if necessary.

The primary benefit is cognitive. If a library requires initialization, its objects and functions have an additional, externally-visible behavior: “If not initialized, return error.”

If the library’s internal state is managed implicitly, it does not leak out of the API. createLibraryObject should lazily initialize the library on object construction. Optionally, on object destruction, it can deinitialize the library.

A secondary benefit of lazy initialization is performance: interactive desktop applications rarely need immediate access to all of its subsystems. By deferring as much initialization as possible, application startup time is reduced.

[This post is a test of my new host: prgmr.com]

Benjamin McGraw: Verge Files Directory

Wed, 30 Dec 2009 08:53:06 +0000

If all went well, this will only show up on my gruedorf feed and not on the main site rss feed.

Yesterday, I got the file directory view completed for beta.verge-rpg.com. See it at http://beta.verge-rpg.com/downloads/directory/.

Next up will be the file redirects! Whee.

Hey Kildorf, where’s your post?

Benjamin McGraw: Gruedorf: Sandbox Sitta

Wed, 30 Dec 2009 06:47:26 +0000

This week I completed the autoschema tool that’s been long overdue to maintain a healthy database state across the four sandboxes used for development.

I also cleaned up layout errors and corrected a bug in screenshot uploading. Also improved the search result page.

Three minor files section tasks left: A directory link/listing, build the legacy 301 redirect table, and give the files frontpage some polish.

Next major task after that: the Gallery Controller.

Jeff Lindsay: How to argue the case for webhooks

Tue, 22 Dec 2009 23:28:58 +0000

Push/callbacks are better than polling.

This should be an easy argument. It’s not just more efficient, it’s more real-time. I once heard something like 40% of delicious.com requests returned 304 Not Modified. I imagine similar numbers for other popular sites. The more real-time you try to be with polling, the worse it gets. Don’t call us, we’ll call you. It’s more efficient.

RPC solutions provide more value than messaging solutions.

Although you can argue that RPC can be powered by messaging, and messaging can be used for RPC, the point is that they are different mindsets. RPC is about triggering code that does something. Messaging is about putting a piece of data in another bucket. When you try to call somebody, you usually don’t want to get their voicemail, right? This analogy goes even further in that, like voicemail, it must be checked on the other end. Messaging just pushes polling somewhere else.

More importantly, the focus on triggering code is central to this value proposition. Generative systems are more valuable than sterile systems. When did the web get interesting? When it became about more than just static content, and code was put in the loop to generate dynamic content. The point is that if you prioritize code to receive a message before humans, you open up many more possibilities.

RPC is about making things happen. Messaging stops short short of that by just moving data around.

HTTP is the defacto RPC protocol.

HTTP is everywhere. There are powerful free servers, clients in every major programming environment, and people know it well. It’s proven and it just works. HTTTP’s simple design also allows it to be extremely versatile. It’s basically the TCP of the application layer.

However, the best thing is that HTTP is RPC. This subtle fact has been true ever since CGI was introduced. We’ve gone through building RPC on top of it with XML-RPC and SOAP, but wisely settled on a form of RPC that’s just HTTP and is even aligned with HTTP semantics: REST.

If HTTP is RPC and HTTP is everywhere, it is our defacto RPC protocol. Especially for web applications that breath HTTP, it almost doesn’t make sense to think of any kind of inter-application communication that isn’t HTTP. Turtles all the way down!

HTTP RPC + Indirection = Webhooks

David Wheeler said, “All problems in computer science can be solved by another level of indirection.” Webhooks, and all callbacks, are about taking a procedure call and performing it on a variable function. This is indirection and this is very powerful. This is why Unix pipes work. STDIN and STDOUT are not hardcoded values, they’re variables that you can control.

Now imagine if all the web applications you used had extension points that you could effectively hook together with any other application. Well, that’s what webhooks are about.

Jeff Lindsay: Webhooks finally explained for regular people!

Tue, 22 Dec 2009 23:17:46 +0000

Although it’s not the most compelling story, this blog post is a terribly effective analogy. So effective, non-techies can read it and “get webhooks” … in some cases leading them to rally for webhooks as much as I do! The analogy focuses on a non-computer, real-world analogy based on telephone calls. Then it follows up with a more concrete example that helps explain the possibilities:

A concrete example of a story made possible from webhooks that might be a useful scenario for many of you involves Twitter. Let’s say Twitter supported webhook callbacks for when somebody follows you. Right now you get an email, and from there you can decide what to do manually: follow them back, block them, or do nothing. I used to go out of my way to block users that I knew were spam bots, but now there’s so many it’s not worth the time. And of course I also generally follow back people that I actually know. If Twitter would simply call a script of mine whenever somebody followed me passing along the user ID, I could very easily run this logic in a PHP script or a simple App Engine app. Or perhaps I’d use Scriptlets (ahem, which was made exactly for these kinds of web scripts). It would work like this:

First, use the Twitter API to look up the user from the ID, and grab their name. Then use the Facebook API to check if that name shows up in my list of friends on Facebook. If so, use the Twitter API to follow them back. Otherwise, if they’re following over 1000 users and that number is more than twice the number that’s following them (which is roughly the heuristic I use manually), use the Twitter API to block them. All automatic.

Definitely worth the read, if I do say so myself. It’s also worth pointing people that want a quick understanding of webhooks. What kind of analogies have you come up with?

Jeff Lindsay: Unprecedented State of Busy

Tue, 15 Dec 2009 06:54:56 +0000

Both now and in general. 2009 is probably the most notable year I can recall. It must have something to do with how busy I've been. Too busy to write in my LiveJournal, that's for sure. And there's been growth and development on all fronts: intellectual, financial, personal, interpersonal, professional, physical, etc.

At the same time, there are daily issues to overcome, stress of many projects and upcoming deadlines, and the occasional bout of exhaustion. I still need to pull my finances into better order, get my semi-professional brand out there as an umbrella for all my projects, and figure out at what point will I retire from pure technology. Most of my major projects are so long-term and require a constant stream of smaller projects to support them, I'm not sure I'll know when I've succeeded.

It may be that what I'll ultimately end up doing is moving where there are more people doing what I really want to do. Living here means I'm completely surrounded by startups and technology, so I can't help but spend cycles thinking about that. It frames my thinking and inspiration.

We'll see.

Michael Rooney: Launchpad is now an automatic, magical translation factory!

noreply@blogger.com (Michael) — Mon, 30 Nov 2009 12:55:44 +0000

I've been using Launchpad to host my personal finance application wxBanker for a few years now. The thing I was hearing most often was that it wasn't localized; people wanted currencies to look the way they should in their country, and the application to be in their language. Let me explain how Launchpad helped me provide translations for my application, and how much of an utterly slothful breeze it has recently become.

Normally to handle translations, an application has to wrap strings with gettext, create a template, find translators and give the template to them, collect translation files back, and integrate them into the project. This is painful and is why many applications aren't localized, and shut out most of the world as a result. One of the amazing features of Launchpad however, happens to be Rosetta, which brings translators TO your translations via a simple web interface, and then makes those translations available to the developer. With Rosetta, translators don't need to understand gettext or translation file formats; they just need to know two languages!

So that's what a translator sees. Notice how Launchpad even shows how other applications translated the same string. So once you generate a template and upload it, you can assign a translation group to your project such as "Ubuntu Translators" so that your strings will be translated by volunteers on the Ubuntu project; if your project isn't relevant to the Ubuntu project, you can use the more generic Launchpad Translators group. Now all you have to do is wait for some translations, then download them and check it in to your code. Not too bad, right?

It isn't, but Launchpad has recently made it so much better. They started by adding an option to automatically import translation templates from your project. This means as you are developing, all you have to do is regenerate the template and commit, and new strings will show up for translators in Rosetta and be translated automatically (from the developer's perspective). Then today, they announced the other side of this, which is automatically committing those translations back into your code on a daily basis. This means that all I have to do is commit templates as I change strings, and Launchpad handles everything else. This is a profound tool for developers.

What's the next step? Well, from a developer's perspective the translation template is a tool to give to the translators or in this case Launchpad. In the future Launchpad could eliminate this by generating the template itself from the code (this is what developers are doing themselves, after all), so that truly all you have to do after you set up the initial i18n/l10n framework is commit code as normal, and Launchpad magically commits back translations.

All this work Launchpad is doing gives developers more time to develop while still having localized applications at a very minimal cost. This is continuous translation integration, and boy is it cool!

Jessy Cowan-Sharp: Deriving Organizational Structures from Evolutionary Algorithms

Sun, 22 Nov 2009 07:55:02 +0000

A couple of weeks ago I had the pleasure of sitting in on an Organicities design studio at EPFL, focused on “the digital generation of architecture using biological paradigms.” It was an exploration of the potential these paradigms might hold for generating effective public, corporate, and community spaces in urban environments. Compared to [...]

Jessy Cowan-Sharp: PostGIS Beginner Notes

Sun, 22 Nov 2009 04:58:06 +0000

these are basically my notes as a total n00b coming to postgres and postgis from previous experience with mysql. hopefully, they will be useful to others as well in finding their feet. i am using ubuntu, but except for the package names below these instructions should be rather generic for non-windozz systems.

first thing’s is first.

1. install postgres (sudo apt-get install postgresql)

2. install postgis libraries and postgres postgis support (sudo apt-get install postgis postgresql-8.3-postgis).

the approach to postgis is a little different than something like mysql, which many people (like me) are used to. the default postgres user is called postgres (usually) and this is the default superuser. the first time you connect to the postgres manager, you need to su to the postgres user. it will connect to he default database (also named postgres). in general, if you are connecting as user x, and you do not specify a database name it will default to a database of the same name as the current user.

another subtle difference is that postgres has several distinct commands which you can run from the command line, which arent necessarily obvious. where mysql uses mysqladmin <some command>, postgres just uses separate commands without a common prefix- eg. createdb (create a database, assuming the user you connect as has sufficient privileges), createuser, etc. i find this a little weird since in some cases it’s not really obvious that, for example, createuser has anything to do with postgis.

3. you might want to create a user with the same username as your regular account username to simplify interacting with postgres. to do this you can either connect to postgres and do this from within the DBMS, or you can use the createuser postgres command from the command line:

$ sudo -u postgres createuser -s new_superuser

or better yet:

$ sudo -u postgres createuser -s $USER

where $USER is the environment variable that set on most unix style machines that will expand to your username. some people recommend for maximum convenience to also create a database with the same name, to enable connecting to postgres without specifying a database explicitly.

$ createdb $USER

so now we’re set up, and you can connec to postgres by just typing

$ psql

in general to connect as an arbitrary user to an arbitrary database (for which you have permissions of course):

$ psql -U username -d dbname

Once you’re logged in, these commands will come in handy.

\du lists existing roles (users)
\l list all databases
\d tablename (of current database- like ‘describe’ in mysql)

each database that will use postgis has to explicitly enable the postgis functions and datatypes. yes– even though postgis support has been enabled through the packages you installed, you still have to manually enable the specific functions and data types for each database that will use them. there is a standard procedure for doing this. standard enough, in fact, that you can simplify the process by using postgres’ notion of database templates. think of it like creating templates in an office program– like a template for an invoice, or a newsletter. even the default new database is an instantiation of a template: the basic or raw template, called template1. as you can see from the instructions below, a template database is not anything different or special from a regular database.

to create a spatially enabled template, do the following (ref: clear and simple instructions from this blog here):

$ sudo su postgres #optional, if your current user is not a superuser
$ createdb postgistemplate
$ createlang plpgsql postgistemplate
$ psql -d postgistemplate -f /usr/share/postgresql-8.2-postgis/lwpostgis.sql
$ psql -d postgistemplate -f /usr/share/postgresql-8.2-postgis/spatial_ref_sys.sql

we then have to make sure the correct users have permissions on the template database (important or they wont be able to use it!), optionally create a schema, and then test it!

$ psql -d postgistemplate

# ALTER TABLE geometry_columns OWNER TO youruser;
# ALTER TABLE spatial_ref_sys OWNER TO youruser;
# CREATE SCHEMA gis_schema AUTHORIZATION gis;

now to create a spatially enabled database (note: this can be done from inside or outside psql):

# CREATE DATABASE gis_test TEMPLATE = postgistemplate;

finally, make sure all is good by defining a table for the new database. here’s a simple example which defines a table that maps to the NMEA gps sentence GPGGA. put these commands in a file, or enter them in on the psql command line:

CREATE TABLE gpgga (
       pk serial PRIMARY KEY,
       utc TIME NOT NULL,
       gps_fix int NOT NULL,
       sats_in_view int NOT NULL,
       horiz_error FLOAT NOT NULL,
       alt_asl FLOAT NOT NULL,
       height_above_WGS84 FLOAT NOT NULL,
       DGPS_ref_id int NOT NULL
);
SELECT AddGeometryColumn('gis_schema', 'gpgga', 'latlong', 4326, 'POINT', 2);

Note that you create the table with all its non-spatial columns first, and then add the POINT (or other spatial) column. The arguments to the AddGeometryColumn are, respectively (schema, table_name, column_name, SRID or spatial reference id, data_type, dimensions). the SRID is required by open GIS, and is actually a foreign key into the SPATIAL_REF_SYS table. it tells the table what the reference system or projection of the data is. you can look these up depending on your data type.

To read this in from the command line is pretty standard:

$ psql -d gis_test < table_definition.sql

upon execution, you will likely see output like the following:

$ psql -d gis_test < table_definition.sql
NOTICE:  CREATE TABLE will create implicit sequence "gpgga_pk_seq" for serial column "gpgga.pk"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "gpgga_pkey" for table "gpgga"
CREATE TABLE
                addgeometrycolumn
--------------------------------------------------
 public.gpgga.latlong SRID:4326 TYPE:POINT DIMS:2
 
(1 row)

Congrats! Coming soon– parsing your nmea files.

Jessy Cowan-Sharp: NASA Image of the Day as Gnome background

Sun, 22 Nov 2009 04:57:49 +0000

Lately I’ve really been enjoying the NASA Image of the Day, and so tonight I decided to make a quick and dirty script to update my gnome background daily with the new images. I thought of making a nice python script using the elegant universal feed parser; something able to handle generic image feeds, but as soon as i realized how differently the different image feeds were implemented, i quickly gave up in favour of a much more terse solution in bash.

the script below just pulls the rss entry down using wget, and then uses grep to mine out the necessary components of the RSS entry. once it has the image url, it saves that to a directory ~/.backgrounds in your home directory, and then uses gnome’s gconftool-2 to set the value of the background image. add this to your /etc/cron.daily/ directory and you should be good to go.

#!/bin/bash
 
# grabs the nasa image of the day by RSS feed and updates the gnome
# background. add this to your cron jobs to have this happen daily.
# eg. in your /etc/cron.daily/ directory, add a one line script that
# simply executes this script (dont forget to make it executable):
 
# #!/bin/sh
# exec /path/to/this/file/nasabg
 
# note the nasa image of the day only returns one RSS entry, so we are
# not concerned with parsing multiple entries.
 
# this is, obviously, a hack, that is likely to break at the slightest
# change of NASA's RSS implementation. yay standards!                                       
 
rss=`wget -q -O - http://www.nasa.gov/rss/lg_image_of_the_day.rss`
 
img_url=`echo $rss | grep -o '<enclosure [^>]*>' | grep -o 'http://[^\"]*'`
 
img_name=`echo $img_url | grep -o [^/]*\.\w*$`
 
mkdir -p ~/.backgrounds
 
wget -q -O ~/.backgrounds/$img_name $img_url
 
gconftool-2 -t string --set /desktop/gnome/background/picture_filename ~/.backgrounds/$img_name

From a developer’s standpoint, I think NASA’s RSS feeds are a great step in the right direction. but this is also an example of an area that, by going just a few steps farther, we could expose hundreds of gigabytes of images a day in ways that would allow developers to actually build on this content, and not just view or scrape it. A rest API where you could query based on mission, image size, instrument, image type, planet, year, etc. would be truly amazing!

Jessy Cowan-Sharp: Colourizing Python Print Statements

Sun, 22 Nov 2009 04:57:43 +0000

Lately I find myself writing a lot of python scripts that involves a ton of text flying by, and me inventing increasingly creative combinations of obscure symbols to delineate different portions of the output. Luckily, if you’re not using windows using a sensible operating system, then you probably have an underappreciated tool at your disposal– [...]

Jessy Cowan-Sharp: JSON Encoding and Decoding with Custom Objects in Python

Sun, 22 Nov 2009 04:57:36 +0000

The JSON module supports encoding (aka serializing) for all the basic built-in python types– strings, lists, dictionaries, tuples, etc. but if you have your own user-defined class that you want to store, I found the documentation to be pretty ambiguous. And since I also didnt see any complete examples out there of custom object [...]

Jessy Cowan-Sharp: Ssh-Agent Forwarding

Sun, 22 Nov 2009 04:57:28 +0000

For some reason this seems confusing– but it isn’t! That said, I always forget the specifics. Here’s how you use ssh-agent to do key forwarding to remove machines. On your local machine, execute: eval `ssh-agent` Note the back ticks around ssh-agent. you need to eval this, not run it! Now you need to add the identities you want [...]

Jessy Cowan-Sharp: Direct Data

Sun, 15 Nov 2009 09:03:07 +0000

It is the end of science. Science as hypothesis is giving way to science as statistically derived by large data sets. We’re finding in many cases that data mining algorithms applied to loosely specified models leads to better results than finely tuned models derived from theoretical equations [see the unreasonable effectiveness of data (pdf)]. It’s [...]

Jessy Cowan-Sharp: Cities are Human Settlements

Fri, 06 Nov 2009 07:57:45 +0000

My two passions have always been the quantitative study of social dynamics, and community. How can we detect and understand patterns in human social interactions? What does that say about their role in, or relationship to, natural properties of the universe? If we can use observed patterns in human dynamics to reverse engineer properties of [...]

Jeff Lindsay: Please date food

Fri, 06 Nov 2009 04:33:36 +0000

At the Dojo, somebody left a note on the fridge:

"Please date food."

Somebody crossed out date and wrote "marry."

Later, somebody crossed out marry and wrote, "cheat on."

Somebody else wrote along the side "+1 for marrying food."

Chad Austin: 9 Reasons Why MinTTY is the Best Terminal on Windows

Tue, 20 Oct 2009 08:33:42 +0000

MinTTY is a fantastic piece of software that smoothes some of the edges of the Windows command line experience. You should use it and here’s why:

MinTTY comes with Cygwin

Installing MinTTY is trivial: just select the mintty package in Cygwin’s setup.exe and point your Cygwin shortcut to C:\cygwin\bin\mintty.exe -. The trailing hyphen is important.

Sane copy and paste

By default, MinTTY copies on select and pastes on right-click, just like Linux. However, you can make it behave like a typical Windows application and show a context menu upon right-click.

I won’t bother explaining how difficult copying and pasting is in standard Windows consoles.

Resizing

MinTTY supports arbitrary window sizes, including maximized. Enough said.

Works with less/emacs/ssh

Because MinTTY is based on PuTTY, it doesn’t display output strangely when running emacs over ssh, among other examples.

It’s FAST

Anyone who has done command line work on Windows has surely noticed that, when a program spews output to the console, system performance nosedives. Sometimes, even the mouse cursor skips, making it hard to kill the program responsible.

MinTTY doesn’t have this problem — it uses minimal CPU, even under heavy load.

Doesn’t bypass RSI Guard

Five years ago, I was diagnosed with repetitive stress injuries from programming. To make matters worse, I get obsessive when I work, and nothing can pull me away from the keyboard.

RSI Guard keeps my wrists and elbows pain-free by enforcing short, periodic breaks. However… years of exposure to RSI Guard has caused me to discover its holes. For example, native console windows bypass RSI Guard’s protection, so when RSI Guard blocked keyboard and mouse input, I would quickly switch to typing in a console window and continue to work. Because MinTTY is a standard Windows application, it closes this backdoor.

Closes even when programs are backgrounded

Open a fresh Cygwin and type notepad & followed by exit. The Cygwin console sticks open until you close Notepad.

In MinTTY, you can always close the window, no matter how many background processes you’ve started.

Alt-F2 opens a new terminal window

Pure convenience: Alt-F2 opens a new terminal. No need to reach for the mouse.

Shift-PageUp and Shift-PageDown!

For rapidly paging through previous lines of output, you can press Shift-PageUp and Shift-PageDown, just like the Linux console. Another huge convenience.

Jeff Lindsay: Page One

Sat, 17 Oct 2009 00:21:57 +0000

Hacker Dojo == Hacker Heaven

Also, the city of Mountain View sent us their congratulations. They say the Major wants to meet us.

Jeff Lindsay: Combat Tanks

Fri, 16 Oct 2009 09:45:41 +0000

I'm bringing back the game developer within. I will now develop a series of awesome, simple, but increasingly scoped networked multiplayer games in Flash using Flixel. First on the list is Combat Tanks. Play the alpha preview!

Mostly developed in the 3 days of TIGJam, it's a remake of a favorite Windows 3.1 game. There are still bugs and major features missing, but it will be awesome and it will take the Internet by storm.

Jeff Lindsay: Johnny 5

Sun, 04 Oct 2009 09:02:48 +0000

See it in action.

Benjamin McGraw: Fatty

Thu, 24 Sep 2009 09:21:56 +0000

Fatty. (Noun)

A big creature. Fatness starts around 4/4 and never ends from there.

“I love the fatties.”

Benjamin McGraw: Files, and the editing thereof.

Tue, 22 Sep 2009 07:27:31 +0000

The most pedestrian screenshot ever. To what depths has this once-mighty work-journal fallen?

My gruedorf showings are shameful of late, and so are all of yours. The only one holding the line at this moment is Ustor. Perhaps this should be called “UstorUstor”. Or “TorUs”.

At any rate, my small token to personal projects this week is the file management section’s edit page. Woo, now you can edit your files once they exist. Necessary, and very quotidian to a web developer.

Up next on this mystical journey is the upload section’s frontpage, featuring most downloaded recently, newest files, and staff picks (aka, Demo Alarms). After that, we’re a docs section from release. At this point, I think I should screen-scrape my old site’s documentation and import all of that into a wiki. Anyone have objections?

Michael Rooney: wxBanker 0.6 preview available, now with recurring transactions!

noreply@blogger.com (Michael) — Mon, 21 Sep 2009 01:29:01 +0000

I've just released wxBanker 0.5.9 in the wxbanker-testing PPA, which is a preview release for 0.6. If you aren't familiar with wxBanker already, it is a cross-platform, lightweight personal finance application, and you can find more info at https://launchpad.net/wxbanker.

While there are many improvements and fixes in 0.6, the main feature is recurring transactions, allowing you to automate repetitive transactions. They are functional in 0.5.9 with one caveat and the purpose of this preview release: there's no way (besides sqlitebrowser :) to modify existing recurring transactions. As such I'd love to get feedback on your impressions of recurring transactions, ideas on a nice configuration UI for them, and of course general feedback. With this I can implement the configuration UI and release 0.6.

Here's an example of a simple quarterly transaction:

and here is a more complex one, with specific days chosen, as well as the transaction being a transfer from another account:

When you start up wxBanker and there is a recurring transaction due, you will see something like:

I didn't reinvent recurrence rules but instead (luckily) found the dateutil module for python which includes rrule, an "implementation of the recurrence rules documented in the iCalendar RFC", so it should behave quite well.

As far as the needed configuration UI goes, there are a couple ideas that I've had so far. Below is the current top-left portion of the UI, for reference:

One idea was to have a third "Recurring Transactions" tab after the "Transactions" and "Summary" tabs, which appears if and only if recurring transactions exist. This would provide a list of recurring transactions and allow editing them via the same UI used for creation, as well as changing the account or removing them altogether. A second idea is to add another button next to the Add/Rename/Remove account buttons in the upper-left, for account configuration (this will be necessary for future features anyway), and allow modifying recurring transactions for that account there.

Another perhaps complementary way would be to provide a right-click action on existing transactions which were recurred, for editing. I'd also like to implement functionality similar to Google Calendar where editing a value on an existing transaction caused by a recurring transaction will ask you if you want to apply that change to just this transaction, all existing, or all future.

If you have ideas, please feel free to leave comments here or drop by #wxbanker on irc.freenode.net any time this week after 10AM PDT, to have a more interactive chat about them.

Please do let me know what you think from either just the screenshots here or actually playing around with the application!

Chuck Rector: I don't like IDEs anymore.

noreply@blogger.com (卡车 Chuck) — Sun, 20 Sep 2009 03:41:00 +0000

I spent seven or eight years using Eclipse as my primary development environment. I never really questioned it until I switched jobs and saw some of the mavericks using Emacs, Vim, and command shells. I decided to become fluent in Emacs, and have been using it daily for the past couple years now.

Thinking about it now, it's most interesting to me that I never looked back and said "Gee, life was so much better back in the IDE days." Hell no. In my mind, Eclipse is a sofa and Emacs is a treadmill. I have a goal, I'm on task, I know what I want to do and I do it.

My big fancy IDE mentality was all about victimization, blame, and laziness. "Big fancy IDE crash! No my fault." "Big fancy IDE get confoozed and lie! I do bad thing, oops!!" "Big fancy IDE slow on big project, me play while wait!" "Big fancy IDE big and fancy! It break, I no no why. I confoozed."

My Emacs mentality is scientific, hands-on, and active. I can count on one hand the number of times GNU Emacs has ever crashed on me. Emacs doesn't happily chew on gigs of files in the background, caching data that I could give two shits about. When I want something, I ask for it explicitly. My UX isn't impacted by background tasks that are attempting to read my mind, because there's only ever one task: the one I'm doing right now.

Being born into royalty and pampered for so many years feels nice, but over time it just turned me into a fat lazy slob with a soft mushy brain. It trained me to surrender my critical thinking skills. Emacs and command shells forced me to learn how to tie my own shoes and make my own meals. I enjoy the independence and self-reliance.

I tried Komodo today, at the recommendation of a coworker, and it has a nice polished look, but it reminds me too much of Eclipse. The editor looks as though it is based on Scintilla, which is cool. I am a fan of SciTE. The background churn as it attempts to cache everything under the sun makes the UI sluggish and choppy, which blows my mind. I'm running a quad core with buttloads of memory on an SSD. Come on, man. Seriously?

These are only my first impressions, so I'm going to keep at it for at least a couple weeks to get a real feel for it. Maybe I can right some wrongs, but I get the sense it's never going to feel as light and razor sharp as Emacs.

Jeff Lindsay: Interview on the uses of webhooks

Sat, 19 Sep 2009 00:49:24 +0000

A while back I was interviewed on the Nearsoft blog about webhooks. It goes into more details on the whole push, pipes and plugin use cases. Real-time web is a hot topic these days, so I had to mention how the webhooks movement relates to that trend. Here’s an excerpt about using webhooks for real-time notifications:

Notifications seem to be [a big draw for webhooks] … as in, tell me when something new has been posted, or when something has changed. But with your code on the receiving end of the notification, you can decide exactly how you get notified. For example, tell me about changes over Twitter, not email. In fact, no, use this other hook script that uses cloud telephony to call my cell phone and use text-to-speech to tell me.

Real-time notifications, exactly how you want them.

Jeff Lindsay: HookPress brings webhooks to WordPress!

Sat, 19 Sep 2009 00:40:16 +0000

So much has been going on in the world of webhooks that it’s hard to keep up. From PubSubHubbub developments, to new services supporting webhooks … to this plugin for WordPress called HookPress. HookPress opens the WordPress plugin API over webhooks! It was developed by my new buddy Mitcho. He made an excellent screencast to demonstrate the power of HookPress, and the power of the emerging webhooks ecosystem. Check it out:

Joseph Mathes: Maybe this will get me posting again

Wed, 16 Sep 2009 13:52:23 +0000

3k hits a day’s what this blog struck
down to none now; it’s run by a slack schmuck
so I picked a slick schmuck cure;
strict limerick structure
if it works, I was right. Wish me luck!

Benjamin McGraw: vrpg beta work continues

Mon, 07 Sep 2009 20:59:13 +0000

I’ve mainly been dealing with code concerning the files section. I’ve improved the quality of the upload process, corrected the file lineage graph and made strides towards a saner system, have made it so you can comment upon file entries, and made it so anyone can upload screenshots of any file.

Also I made an schema-application tool to keep track of which .sql files have been applied in what environment. Trust me, this is useful.

So where did I go for two months, far, far away from Gruedorf? I honestly can say I was shocked to crack open this dev folder to find the last svn log from July 10. So wherever I went, it didn’t seem as long as it was. :(

Jeff Lindsay: mp3

Fri, 28 Aug 2009 21:42:23 +0000

This is about ten days old, but still worth mentioning. I was working on PostBin trying to support file uploads. I realized I really wanted to have icons for the files to help differentiate them from other POST params and make it feel more polished. Of course, this would mean I’d have to find icons for the popular file/mime types … and if I were going to do that, I might as well build a service … but I didn’t want to build a service.

I turned to Twitter. In that past, I’d mentioned wanting something and somebody actually built it (using my tools no less). I figured this was a bit more work than last time, but it couldn’t be that much more. So I gave it a shot and tweeted it. Next thing I know, Paul Tarjan is on it. Some hours later: stdicon.com

I’m credited at the bottom, but really, I just had the idea and came up with the domain. Paul wrote all the code and even collected all the icons (and manually uploaded them to the app). We discussed implementation details over IM, but that was it. Pretty rad!

The idea is that given a file extension (”txt”, “gif”, etc) or a mimetype (”text/plain”, “application/zip”, etc) you can get a resizable icon by putting together a simple URL, like http://www.stdicon.com/mp3?size=16 or from a particular icon set http://www.stdicon.com/neu/html?size=64. Here are some examples from various sets:

Nuvola set:
Apache set:
Crystal set:

And that’s it! Paul wrote a post about it. Somebody requested a simple API for just doing the mimetype to file extension conversion, so Paul added it. Strangely, the same day, MIME API was released. That’s fine. Just means stdicon can focus on icons.

Anyway, I pulled this same stunt just the other day, using Twitter to get more cool infrastructure built, but I’ll have to write about it later.

Jeff Lindsay: progrium

Wed, 26 Aug 2009 07:09:40 +0000

Oh, webhooks. What have you become? Just another buzzword for the rising realtime web trend? I admit, without a spec or obvious definition, it seems to lend itself to such a fate. It’s kind of like AJAX. Although, those that know the true meaning of AJAX exists mostly in first letter, and know that it is actually a significant and useful pattern, should know the same of webhooks.

In fact, those that are familiar with the heart of AJAX can even compare webhooks mechanically to AJAX. It’s like an inverted, backend, server-to-server version. Yeah? Okay, that’s a stretch. Maybe I won’t open my SXSW talk with that description.

Actually, I do want to start my SXSW talk backwards. What I usually leave until the end of my webhooks talks, and as such tend to skim over for lack of time, I’m going to make the focus on my next major talk. You see, some people just don’t get it. I think starting with the mechanism and extrapolating doesn’t work if people get stuck on the mechanism.

I questioned both Anil Dash and Mark Cuban’s posts about webhooks and PubSubHubbub. It turns out they both do actually get it, they just realize they need to simplify it for people to understand. The same reason PubSubHubbub is focusing on feeds for now as opposed to general HTTP pubsub.

It’s not that people are stupid. Or even that we’re smarter than others. The magic of webhooks is in the emergence of an event-driven programmable web, something that’s not terribly obvious when looking at what webhooks are in of themselves: callbacks over HTTP. Most people don’t even see that; they see notifications over HTTP.

You can compare it to Go. A game you can learn to play so quickly, but can spend a lifetime to master. More importantly, after you learn the rules and do some simple scenarios, you think you get the game… but as you play more, it becomes something else. What it’s “really about” emerges as you actually experience it.

It reminds me of some people that say “Well, you can do that with XMPP; that’s what you should really use,” when it turns out they’ve never really programmed a system with XMPP, and definitely not with webhooks. They have no idea the convenience of webhooks over XMPP, or what that affords. Even for those that believe webhooks sound nice in theory, they’ll go implement it and come out going, “Wow, this really is quite cool,” as PBworks CTO said after implementing webhooks in PBworks.

Anyway, I haven’t even gotten to answering my question: What are webhooks really about? The real answer is that they’re about something completely different!

I was standing next to my long-time colleague Adam Smith (we built AjaxWar together in 2005, before Comet had a name) as he read Mark Cuban’s post. After, he remarked, “Well it’s good to know you’re still two steps ahead.” Yep. ^_^

You see, the funny thing is that webhooks aren’t even the ultimate goal. They’re a means to an end. That’s what my next talk is about, only I decided against “Beyond WebHooks” … instead I went with “How WebHooks Will Make Us All Programmers.”

Have I been pushing an agenda to subvert the masses and make us all programmers? I’ll admit that’d be nice, but really I just think it’s the most interesting outcome of the progression I see going on. Honestly, if explained properly, I think it’s something we can all get behind. In fact, I explained webhooks to Cheri Renee at the last DevHouse from this perspective and it actually got her excited. She’s not even a programmer.

So that’s the angle for my next talk on webhooks. The real big picture. At least for me, that’s what webhooks are really about. Hopefully I’ll find some time to get into more details before the talk. Until then, please vote for the proposal so I can finally make it to SXSW (and give this talk)!

Chad Austin: OpenCL Rocks: Why You Care

Mon, 24 Aug 2009 05:28:08 +0000

Today I attended the Hot Chips 2009 conference, where I was introduced to OpenCL*. I’d never heard of OpenCL before, but it’s totally hot and I will tell you why.

What is OpenCL?

OpenCL (Open Computing Library) is an open and free API that lets your programs efficiently leverage all of the computing power in your computer, including multiple CPU cores, vector computation units, GPUs, Cell coprocessors, DSPs, etc. OpenCL programs, called kernels, are written in a variant of C and are automatically scheduled across your available hardware. Your customers don’t have a GPU but their processor has 2 or 4 cores? No problem: OpenCL will utilize that spare power.

In addition, OpenCL specifies 4-, 8-, and 16-wide SIMD vector types, meaning it can easily leverage both the vector computation units on GPUs and extended vector instruction sets on CPUs, such as SSE and AVX. (Looking forward to Larrabee.)

With OpenCL, Apple and the Khronos Group are cleanly solving two clear market needs: we have increasing amounts of computational hardware available to us, but each bit of hardware is a bit different, with separate APIs, performance, and instruction sets. As much as possible, OpenCL unifies the interface to this hardware.

It’s true that GPUs and CPUs have different performance characteristics, but the same types of programs run efficiently on both: do way more math than memory loads and split your program into large parallel chunks.

Where do I get it?

The major vendors are starting to provide OpenCL implementations:

Intel’s on the way, I’m sure. I expect we’ll see an OpenCL implementation on Larrabee as well.

How do I use it?

OpenCL programs are called kernels — functions that run in parallel across large amounts of data. You’re given a lot of freedom: OpenCL uses a variant of the C99 programming language, with only a handful of restrictions.

If you find that too onerous, you can use OpenCL with native functions you’ve already written! Just let it schedule your tasks and manage task dependencies to easily get started running code on multiple cores.

I won’t go into the details of the memory model or API; you can look that up on Google. However, here’s an example kernel:

kernel void square(
    global float* input,
    global float* output
) {
    int i = get_global_id(0);
    output[i] = input[i] * input[i];
}

It squares a bunch of numbers in an array, storing the results in an output array. This trivial example will easily consume all of the compute power in your system. (Defining compute to mean ALU + memory, of course…)

Next Steps

I hope that’s enough information to get you excited about OpenCL! OpenCL is a huge step towards transparently leveraging additional cores, vector instructions, and GPUs.

* Full disclaimer: I have never actually used OpenCL, but I’ve definitely struggled with the problems it solves. The IMVU client has some optimized inner loops that we’ve hand-implemented for the GPU, SSE, or C-compiled x87 code. Unfortunately, we have had to manually select between those code paths based on the capabilities of the hardware. This sucks. With even decent OpenCL implementations, we could avoid all of that work and transparently benefit from future hardware improvements.

Jeff Lindsay: progrium

Thu, 13 Aug 2009 09:21:35 +0000

So I recently put this blog’s feed through FeedBurner. If I recall, I think I wanted to get subscriber statistics, as well as to try out their integration of PubSubHubbub publishing. But since I didn’t think I would get new subscribers often and I’m too lazy to check stats, I really wanted a way to be notified when people subscribed. Since the means of subscribing would be a link, it hit me: click-through webhooks.

I wanted a link wrapper that would trigger a URL callback (aka a webhook) when people clicked through the link. Similar to a URL shortener, you’d just give it a URL and it would give you another URL, but instead of being shorter, it would be tied to a callback that would run as it redirected the user to the original URL. What could you do with this, you ask? It’s a webhook; you can do anything with this event.

Coming back to my original use case, I’d probably tie it to a script that sent me an IM that somebody subscribed. Or better yet, with something like Yapper, I could get a Growl notification. Sweet, right?

I knew it would be trivial to build, and even though it was a feature that I really wanted, it was still just a feature. I thought, it really doesn’t need its own app, does it? So I went to my URL shortener of choice, tr.im, and suggested it as a feature. I also happened to mention to all my friends they should vote it up. Well, it ended up being the 6th most requested feature because of that. I thought for sure they’d implement it. I mean it was so much easier than the others, and I gave them code to help do it and everything.

A couple weeks went by and nothing happened. I got no response. I emailed them and got nothing back. They didn’t even mark it as “under review.” It may have had to do with the fact they were then deciding to shut down … which I hear they’ve changed their mind about. Nevertheless, it still hadn’t happened. And I really wanted to be able to do this notification on subscription thing!

Along came my WebHooks and PubSubHubbub meetup where I wanted to demo this bit of plumbing. Of course, I had no way to do it because nobody had implemented it. So I figured I’d just build it that day before the meetup. Twenty minutes later, I had it live: ClickHooks.

Okay, so that part’s done. How do I get it to IM me? Well, Scriptlets was made to write the glue code to use with these kinds of webhooks. And my project Protocol Droid was made to make it easy to use other protocols from HTTP. So I just threw this little baby on Scriptlets and I was done:


import urllib

payload = {
 'username': 'demo@progrium.com',
 'password': 'secret',
 'to': 'progrium@gmail.com',
 'body': "You got a new subscriber on blogrium!"
}

fetch("http://pdroid.progrium.com/xmpp:talk.google.com/send", urllib.urlencode(payload), "POST")

Yes, that pdroid endpoint is live. No, I don’t recommend using it for production. Stand up your own Protocol Droid gateway. ;)

Anyway, simple enough to understand though, right? It should be. And it was simple to get set up. That’s the whole point of a world of webhooks: you can easily do so much cool stuff if this simple infrastructure is there.

As an aside, later I realized I could generalize my IM scriptlet to use a GET param for the body of the message, allowing me to make a personal messaging micro-webservice: http://www.scriptlets.org/abcdef?body=Hello, world! The data in this URL could then be hidden behind a URL shortener, giving me a regular looking URL that I could then give to people that would IM me the message when they clicked it. Kind of silly, but I almost set up a link that would IM me “At the door!” that people could click from their phone browsers as a doorbell for the WebHooks meetup.

Ah, fun with infrastructure.

Benjamin McGraw: PHP, Language of Mystery

Tue, 21 Jul 2009 17:08:48 +0000

PHP seems so arbitrary sometimes.

function test_nine() {
	$$"9" = 22;
}

fails to parse.
(which is good, I suppose)

This passes:

function test_nine() {
	$n = 9;
	$$n = 22;
	$this->assertEqual($$n, 22);
}

-Andy Friesen,
of giant-communist-robots.com.

Benjamin McGraw: Gruedorf: the Swampening

Wed, 15 Jul 2009 20:49:23 +0000

Between putting in incredible hours at work, and a few Magic: the Gathering pre-release events in the last week, I haven’t had much time for gruedorfing.

However, I still got 14 commits in. (All of them Friday morning.)

The work done was all in finalizing the /downloads/complete-new-os/ page, and tracking down and fixing errors that had cropped up in the upload process.

The second item distresses me, presently, since a solid upload system was the main impetus for the rewrite. finding out that uploads are still fragile and very prone to breaking with forward development, combined with the fact that I don’t have a testing strategy for roundtrip uploads, means I’m kinda nervous.

So, anyone out there have a good solution for a round-trip integration test where you upload, via flash and javascript, a unique file, and then verify that the file exists on a unix filesystem, and then verify that a corresponding database entry and further corresponding webpage works?

Moving forward, I’ll be working on an administrative page to allow file owners/admins to associate file instances as older/newer versions of the same file.

Benjamin McGraw: Uploads, Downloads, and the Management Thereof.

Wed, 08 Jul 2009 16:36:57 +0000

After some prodding, I managed to get the dorf back into gruedorf. Let the games resume.

Since last I posted about beta.vrpg, 58 revisions have occurred. In that time:

I got file submissions up.
I got (simple, title-based) file searches up.
I got most of the file details page up (thread spawning, screenshot uploading pending).
I did a lot of database reworking and massaging, mainly to get to a point where…
…single file entries can have multiple physical files attached, so one game can have file links to every OS it’s released on

Oh, and I made a Silly 500 Internal Service Error page.

Definitely the most important part of this work.

Going forward will be finishing the file management side of things (letting you deprecate old versions of files in favor of new ones, showing a browsable history, letting you edit and delete files you own), and cranking out the community-based side of things (ie, image uploading and talkback threads on file pages.) After that, advanced search options and a browsable file index, and the section should be complete.

I’m mainly excited to see a upload system that hasn’t seen a failure yet. This was one of the major failures of the current vrpg that prompted the rewrite. I used swfupload for the actual UI/handling of the upload, and integrating it went very smoothly.

Benjamin McGraw: PHP5, you king of bastards (setting an instance member inside a static class call)

Wed, 08 Jul 2009 13:48:58 +0000

If I’m in an instanced class, and I call another class’s method statically inside of a method in the first class, and that static method should happen to (erroneously) have $this->whatever inside of it…

…it sets $whatever on my outer instanced class.

Here’s some illustrative code:

<?
    class Inner_Static {
        function burrito() {
            $this->_member = ‘A keyboard. How quaint.’;
        }
    }

    class Outer_Instance {
        function taco() {
            Inner_Static::burrito();
        }
    }

    $my_guy = new Outer_Instance();
    $my_guy->taco();

    print ‘$my_guy->_member: ‘ . $my_guy->_member;

This boggles my mind a bit. Tracking it down cost me a bit of time. Is there actually a sane use for this little tidbit, or am I justified in thinking that a good language would warn you that scope shenanigans are going on here?

Michael Rooney: Simple timing of Python code

noreply@blogger.com (Michael) — Tue, 07 Jul 2009 00:12:08 +0000

Often when I am writing in Python, I find that I want to see how long a particular function call or set of statements are taking to execute. Let's say I have the following code that gets executed frequently:


for i in range(10000000):
   x = 934.12 ** 32.61 * i / 453.12 ** 0.23

and I want to know how long it takes to execute to see if it is slowing down my app and should be optimized. Previously I would surround it as such:


import time; x = time.time()
for i in range(10000000):
   x = 934.12 ** 32.61 * i / 453.12 ** 0.23
print time.time() - x

This will print out the duration in seconds of that code segment, but is more work and typing than I want, and more cleaning up later. I realized that the new "with" statement in Python could probably help me out. Let's create a little timer class that cooperates with it:


class Timer():
   def __enter__(self): self.start = time.time()
   def __exit__(self, *args): print time.time() - self.start

Now all we have to do is:


with Timer():
   for i in range(1000000):
       x = 934.12 ** 32.61 * i / 453.12 ** 0.23

You can also try:


with Timer():
   time.sleep(1.5)

For these, 0.28738 and 1.50169 are what I get, respectively. While something like this couldn't really replace application-wide profiling via a module like cProfile, it can be an extremely useful and quick way to see if your prototype is scalable or not. I usually end up having a debug.py or helpers.py file in my larger projects with little tools like this, and I'll probably end up adding this one as well.

Let me know if you are doing something similar, or if I've reinvented something that already exists. I'd also love to hear from people profiling their python code and what techniques they are using, as I am just starting to learn about it.

Benjamin McGraw: Simple scripts always ship.

Wed, 01 Jul 2009 11:18:32 +0000

Simple needs, simple deeds.

A need for a simple one-off occurred Monday evening at work. Some non-engineer coworkers needed to be shown how to make some non-Latin-1 text into html entities.

Basically, they needed a 5 line php script.

Right before sending them to any number of sites that already do this 5-line operation, I decide: what the hell, I’ll just make one of my own. And so, that night after unwinding, I did.

Here’s my HTML Entitizer. Suitable for all your HTML entitizing needs, large or small!

PHP in Escape from L.A.(jokesfornerds)

One of the reasons PHP is so popular is the fact that it’s got so many handy little functions. In this case, htmlentities() is the blade of PHP’s swiss army knife that we’re using. Of course, indiscriminate use of functions like this cause problems like double-escaped characters (&amp;) showing up in databases.

PHP’s solution to this sort of thing is generally to throw new corkscrews and toothpicks onto it’s ever-growing pile of tools on it’s knife. If you look at the documentation page for htmlentities(), you can see that as of version 5.2.3 of PHP, another optional parameter was added: double_encode. Which will

Getting the right escape order can be hard, especially for novices or small teams inheriting code from other small teams. To date I don’t think I’ve inherited work on a webapp whose database wasn’t littered with extraneous &’s and rogue \\’s. It’s only by virtue of verge-rpg.com having a single developer who is really, really annoyed by this problem the the point of neuroticism that &amp; never appears in the database except in posts that actually wanted to display &.

Although even this level of OCD-database-cleansing didn’t prevent escaping-related errors on the first go-round.

Teh (sic) Implementation.

So, the only time-consuming part of getting this script into my wordpress site (other than me taking a bloody half hour to blog about a 2-minute job) was convincing wordpress that it should allow <?php fragments into my posts.

Luckily, other people before me have wanted this very thing. Wordpress’s biggest strength is again one of PHP’s: if you want it, it’s already been made for you. In this case the Exec-PHP module will let admin-level users of your blog post for-reals, actual PHP into your posts! You’d better hope your admins don’t have their accounts compromised! (…one moment. Changing my password.)

In PHP’s case, the code that you want that’ already been written for you is in every single function’s talkback thread. These functions may not be hyper-optimized or the best solution for any given problem, but they generally are what you want to prove a point. You scan the docs, you grab the code, you put it in your site, and you move on to the next problem in your own app.

Quick and dirty, the way PHP likes it.

For the curious, here’s the solution I spat out (copy included):

<h1>Escape html</h1>
<p>This is a simple script to give the escaped codes for some html. Useful
for making foreign languages play nice with html, regardless of how the
server handles string encoding.</p>
<?
if( isset($_POST['escape_me']) ) {
echo('<h2>Your escaped html</h2><div'.
     'style="background-color: #ddd; padding: 8px;">');
echo(str_replace('&','&amp;',htmlentities(stripslashes($_POST['escape_me']),
     ENT_NOQUOTES,'UTF-8')));
echo( '</div>' );
}
?>
<form action='/html-entitizer/' method='POST'>
Text to escape:
<textarea name='escape_me' style='width: 550px; height: 200px;'></textarea>
<input type='submit' value='Create my html entities'>
</form>

(weird formatting so it doesn’t run off the side of the screen; I don’t actually code like that. Mostly. ;)

Coda

It is of note that this tiny amount of effort got me two really sincere “thank yous” from the non-engineers. It’s important to note that things that are mindlessly trivial to a programmer can be tedious tasks to someone without the power to bully the computer into doing labor for them. One of the guys, a computer saavy guy who had access to lists of what each escaped character translated to, insisted that he’d just search/replace the offensive characters by hand.

Ostensibly, this was offered to save me work.

How much time did my 2 minutes of code save him?

(Don’t forget a google for “html entity converter” would’ve saved even that, had I cared to not make the tiny toy script of my own. I’ll ramble on about re-using other people’s work more later, but it’s key to remember even what little work I did end up doing was extraneous and largely an exercise in vanity.)

Benjamin McGraw: beta is the new gamma

Mon, 29 Jun 2009 12:38:44 +0000

Work continues apace on beta.verge-rpg.com.

Actually, it’s quite brisk now. Almost 60 revisions have passed in the last week.

Gayo and crew have banged on the forums, and I patched several oversights. Locally I’ve moved on to the new downloads section, which I’m looking forward greatly to finishing. Mainly so there can be an upload script that works for most users and gives progress feedback.

Are all webapps 98% mundane crap? I’ve been doing them for almost a decade now, and I think I have an answer.

(The answer is “yes”.)

Jeff Lindsay: Yapper, a Jabber/XMPP interface for Growl

Sun, 28 Jun 2009 08:12:02 +0000

Today I released Yapper, a full featured Jabber/XMPP interface for Growl. It was based on the simple Twisted script I wrote while building my notification email to Growl piping. Although it’s limited to OS X users, I still think this is a big deal. Let me explain.

If you haven’t heard of Growl, it’s a global notification system that lets applications notify you of events in a consistent, customizable way. It’s been around for a while and is integrated or has plugins for lots of popular apps like iTunes, Adium, and Tweetie. It has bindings for a bunch of languages and also has a command line tool that you shell scripters can use to pipe notifications into. For example, I used this recently for my reminder system.

I think Growl is great particularly because it very nicely solves passive real-time notifications. That is, you want to be notified, but you don’t want it to interrupt you by requiring action. This is why I hate email notifications.

However, the major problem with Growl is that for the most part, it’s limited to local notifications. Sure, you can get notifications from “out there,” like Twitter or IRC or Gmail … but it requires you to have a local application that pushes them into Growl. One for each, in fact.

It turns out, I don’t actually care about most local notifications. Telling me a file is done downloading or what song is playing in iTunes is not terribly notification worthy. If I’m sitting there to get the notification, I probably already know. The most useful notifications are of events important to me that I’m nowhere near … events from things out on the Internet.

So Growl needs a network interface. Oh, wait! It has one! Well, what’s the problem? Two things: it’s a non-standard protocol, and it requires a direct connection. That not only raises the bar for things “out there” to notify you with Growl, but it literally makes it impossible if you’re constantly changing IPs, sitting behind a firewall or NAT, etc.

Let’s see… who’s solved this problem already? Right, IM! XMPP, the open standard for IM and real-time message passing (also known as Jabber), seems to have everything we need. It’s a popular protocol, fairly accessible from any language, and doesn’t require a direct point to point connection. So it’s a perfect transport for network Growl notifications!

Since I’m not one to wait around for people to implement what I think they should implement, I solved this problem by making Yapper. Yapper is a lightweight Jabber client made specifically to receive Growl notifications. It starts when your machine starts and sits in the background just waiting for whatever you have sent to it so it can pop it up with Growl.

You can now easily have Growl notifications sent to you from anywhere. Websites have less of a reason not to directly notify you with Growl for events that are important to you. More importantly for me, I can more easily set up Growl notifications for whatever I want. With the rise of webhooks, so can you. Anyway, to me this generally makes Growl much more useful.

Give Yapper a look. It’s on GitHub under MIT license. It’s the first release, but it should work fine. If not, drop off an issue and it’ll likely be taken care of.

Jeff Lindsay: progrium

Fri, 26 Jun 2009 09:01:58 +0000

Recently I’ve been evaluating software to help support agile/scrum development on our team, and ideally to roll into our NASA Code product for others to use. We’re already married to Trac, so we’ve been playing with Agilo and are looking at some of the other agile plugins for Trac. Unfortunately they’re all so heavyweight, despite some that claim not to be.

I came back to a realization I’m sure a lot of us have had: most software sucks. Especially software that’s intended to augment some real-life process. When asking Timothy Fitz (my insight into the amazing continuous improvement processes used at IMVU) about recommendations on agile tools, he said: “a board and post-its (seriously)”

This is part of the reason most enterprise software sucks so terribly. Enterprise is about lots of real-life process and workflow, and given that process augmentation software even in small doses generally sucks, large amounts of it will suck exponentially.

A lot of us have learned that less software is more effective. One major attraction of Trac was their goal of staying out of the way through minimalism. The trick with minimalism, in general, is knowing what’s actually important—the essence of the message or design. This is a big part of my design process. Asking, “How can I fold these requirements into fewer features and UI?” instead of directly implementing a feature for every requirement.

The other thing about minimalism is that, like abstraction (another form of compression), everything you leave in the design makes such a huge difference. In programming, when you make abstractions, you’re deciding what you can assume. This means abstractions can go in different directions depending on the assumption requirements of what’s going to use the abstraction. The risk with minimalist software is that a simple design choice can drastically change the direction of the abstraction and make or break whether the software fits your needs.

Luckily, minimalism buys you a sort of abstraction that can enable projection. By this I mean that users can project their actual process and workflow onto the software. If it doesn’t have features that impose a particular process, users are free to do what works for them. This is why wikis are so powerful.

Coming back to Timothy’s “a board and post-its” remark, why do you even need software? If you can do it without software, why would you want to bring software in to slow things down?

Software does have a couple strengths. First, it encodes process in way that means you can automate parts of it. Nobody has to worry about manually typesetting when using a word processor. Second, it persists and organizes information that would normally be lost in handwritten notes, or worse, somebody’s head. The trick is getting these advantages without getting in the way.

A naive approach to software design is thinking that perfectly modeling a system, such as your development process, is the way to good software. I used to think this. It sounds great because then you can programmatically reason about every aspect of the system. But in the real-world, no two systems are exactly alike. In fact, a given system can change quite a bit in its lifetime. So there’s really no point to modeling systems with that kind of precision.

However, I’m seeing a lot of this in agile/scrum software. Requirements have stories, stories have tasks, organized into iterations and releases. CRUD for every noun mentioned in scrum. This on top of abstractions in a direction different than we need. Numbers where it doesn’t really matter. Nice pie chart breakdowns we’ll rarely use. Top it off with horrible UI, since with all these features there isn’t time to make them easy to use.

Honestly, Pivotal Tracker seems to have the best abstraction of agile. It folds requirements, stories and tasks into just stories. It automatically and dynamically creates iterations and calculates velocity. It keeps you on a single page, keeping you focused on what’s important.

Unfortunately, we can’t use Pivotal Tracker since we’d need it on our servers and the licensing they offered doesn’t scale if we want to essentially give it away as part of NASA Code. It’s likely I’ll want to just start nudging Trac in the right direction using Pivotal Tracker as a model reference, pulling together code from Agilo and other plugins. If there’s one thing that complements minimalist design, it’s an extension architecture, and Trac has an excellent plugin system.

Anyway, as far as augmenting process and workflow, I’ve always liked the idea of starting with a wiki and lazily formalizing the process into custom software as needed. As long as you can keep it under control, mind your requirement abstractions, and avoid writing too much software.

Jeff Lindsay: Adam with my laptop and the Growl notice

Mon, 22 Jun 2009 08:31:01 +0000

The other day I realized I hate notification emails. In particular I’m talking about SVN commit notifications. I have them set up to skip my inbox and filter into a label on GMail, but this doesn’t really help. They don’t pollute my inbox, but I still have to go out of the way to read them. Or at the very least, mark them as read, which is still quite annoying!

Growl is supposed to help solve this problem. In fact, when I was retooling the notification system at work, I would have loved to integrate Growl … except that Growl’s network interface requires a direct connection.

Taking a step back, it’s not just work notifications. A lot of us get a lot of notification emails about a lot of things. Most of them are poorly designed, for example, without important details of the notice in the subject. Plus, as email they require disrupting action. Not just to open and read if you have to, but just to archive and/or mark as read.

It’s great to have an archive in email, but email is just not ideal for notifications. If I’m around to receive them, I should get the important information passively and not have to do anything else until I need to reference it later, if ever. That means Growl.

Getting Notification Emails to Growl

I thought to myself, “I should be able to rig this up. And it should be easy.” And if the infrastructure and plumbing I’ve wanted was in place, it would be easy. So I decided to lay it down and make it happen. Here’s what I needed:

An email to webhook bridge, like my old Mailhook service.
An XMPP to Growl bridge. We all want this anyway, right?
An HTTP to XMPP bridge. Why? You’ll see.
Glue code that can run in the cloud. This is what Scriptlets is for.

I’ve since turned off Mailhook, but I’ve been planning to roll its code into my still early Protocol Droid project. This is an HTTP to anything bridge written in Python with Twisted that makes heavy use of webhooks. I’d plan to roll the HTTP to XMPP bridge in there as well. For now, the XMPP to Growl bridge would be separate.

Having all that, here was the pipeline I envisioned:

A notification email would come into Gmail and get filtered
It would forward to an address that points to Protocol Droid
Protocol Droid would post the email to a script on Scriptlets
The script would use Protocol Droid again to send an XMPP message
The message would be sent to a JID for my laptop if online
The XMPP to Growl bridge on my laptop would receive and notify me

Building the Infrastructure

I decided to build all this last Friday night. I got caffeinated and went to work, starting with Protocol Droid. I hadn’t touched it since I was trying to build an HTTP to SMTP module for it. It wasn’t part of this project, but I decided to finish it off. You know, to get warmed up and have an early win.

That happened pretty quickly, so I was excited get started on the next task: an email to webhook bridge. This was a little different. So far everything in Protocol Droid was for outgoing connections. This would be the first listening module. The idea was that it would be an SMTP server that would parse incoming email and post it to a callback URL registered for a recipient domain. I’d written several of these before for mailhook.org, so I ported the code to Twisted for Protocol Droid. Another pretty easy win. I used PostBin to debug this, obviously.

At that point, I decided to take a break and watch an episode of Pushing Daisies. I still haven’t seen all of the second season!

Then came the hard part. Well, I thought it would be easy. The HTTP to XMPP bridge. It turns out the XMPP support in Twisted isn’t very well documented and slightly underdeveloped. I had to build a couple of prototypes to get the hang of it. Even then, building this module and the XMPP to Growl daemon took about 7 hours! Compared to the 3 hours spent doing the last two modules. XMPP is definitely a bit hard to really get into.

By 8am I had all the pieces working. I thought about going the last mile and getting it all online and set up the pipeline … but I decided I deserved some sleep. I’d do it when I wake up.

Getting the Pipeline in Place

When I woke up on Saturday, I was still burnt out from coding. I spent the day hanging out with friends. Then later I invited some more friends over for hacking at my place. There I spent more time than I wanted getting my code to run on my server. Apparently OpenSSL for Python wasn’t installed and the Google Talk servers required TLS. It took a while to figure this out since no error was raised and Google would just drop my connection.

I finally got all the infrastructure I built online and tested. Finally, I could work on the actual pipeline. It was supposed to be easy from that point on. And it was! Except for one bit …

I got almost all the pieces connected and it all came down to the glue code on Scriptlets. The script would parse the email however I like and set up the message that would end up on my laptop via Growl. It turns out Scriptlets is a bit of a pain to use. No edit, no debugging, crappy error messages. Since I built it as a proof of concept, I hadn’t done much development on it, let alone with it. Anyway, I learned a lot and have some things to fix. But! I was still able to get everything working!

I had my friend Adam send the first test email that would go through the entire pipeline I outlined above. He sent it from his phone, just to add a little to the magic. Low and behold, it worked perfectly, quite quickly. It took 4 seconds from him sending to the notice appearing … but about 3 of those seconds were the time it took for the phone to send the email. Nevertheless, brilliant!

Next Steps

I’ve since found out about Wokkel, which might simplify some of my XMPP code. I have some other enhancements to make to each piece to make them more general. And I mentioned some of the things I need to add to Scriptlets.

Functionally, the pipeline is pretty good. The only addition I can think of would require an HTTP to IMAP bridge in Protocol Droid (something I’ve already prototyped). I’d use this to mark the notification email as unread if it wasn’t able to deliver the IM. But for now, I think I’m done on this project. And you can all take advantage of the infrastructure I built out for rigging up your own cool hacks.

Most of the code this weekend went into these two projects, so check them out!

Benjamin McGraw: Gruedorf: boards done.

Mon, 22 Jun 2009 05:45:53 +0000

Even though I’ve been lax in posting, I’ve been working (when I can) on finishing the beta.verge-rpg.com boards.

As of revision 237, the section is good to go for this run-through. I’m going to now move on to the file section

Chad Austin: IMVU Crash Reporting: Stalls and Deadlocks

Thu, 18 Jun 2009 06:52:38 +0000

By mid-2006, we’d primarily focused on access violations and unhandled exceptions, the explosive application failures. After extensive effort, we got our client’s crash rate down to 2% or so, where 2% of all sessions ended in a crash.* Still the customers cried “Fix the crashes!”

It turns out that when a customer says “crash” they mean “it stopped doing what I wanted”, but engineers hear “the program threw an exception or caused an access violation”. Thus, to the customer, crash can mean:

the application was unresponsive for a period of time
the UI failed to load, making the client unusable
the application has been disconnected from the server

In short, any time the customer cannot make progress and it’s not (perceived to be) their fault, the application has crashed.

OK, we’ve got our work cut out for us… Let’s start by considering deadlocks and stalls.

First, some terminology: in computer science, a deadlock is a situation where two threads or processes are waiting for each other, so neither makes progress. That definition is a bit academic for our purposes. Let’s redefine deadlock as any situation where the program becomes unresponsive for an unreasonable length of time. This definition includes livelock, slow operations without progress indication, and network (or disk!) I/O that blocks the program from responding to input.

It actually doesn’t matter whether the program will eventually respond to input. People get impatient quickly. You’ve only got a few seconds to respond to the customer’s commands.

Detecting Deadlocks in C++

The embedded programming world has a “watchdog timer” concept. Your program is responsible for periodically pinging the watchdog, and if for several seconds you don’t, the watchdog restarts your program and reports debugging information.

Implementing this in C++ is straightforward:

Start a watchdog thread that wakes up every few seconds to check that the program is still responding to events.
Add a heartbeat to your main event loop that frequently pings the watchdog.
If the watchdog timer detects the program is unresponsive, record stack traces and log files, then report the failure.

IMVU’s CallStack API allows us to grab the C++ call stack of an arbitrary thread, so, if the main thread is unresponsive, we report its current stack every couple of seconds. This is often all that’s needed to find and fix the deadlock.

Detecting Deadlocks in Python

In Python, we can take the same approach as above:

Start a watchdog thread.
Ping the Python watchdog thread in your main loop.
If the watchdog detects that you’re unresponsive, record the main thread’s Python stack (this time with sys._current_frames) and report it.

Python’s global interpreter lock (GIL) can throw a wrench in this plan. If one thread enters an infinite loop while keeping the GIL held (say, in a native extension), the watchdog thread will never wake and so cannot report a deadlock. In practice, this isn’t a problem, because the C++ deadlock detector will notice and report a deadlock. Plus, most common deadlocks are caused by calls that release the GIL: threading.Lock.acquire, socket.read, file.read, and so on.

It might help to think of the Python deadlock detector as a fallback that, if successful, adds richer information to your deadlock reports. If it failed, whatever. The C++ deadlock detector is probably enough to diagnose and fix the problem.

What did we learn?

It turned out the IMVU client had several bugs where we blocked the main thread on the network, sometimes for up to 30 seconds. By that point, most users just clicked the close box [X] and terminated the process. Oops.

In addition, the deadlock detectors pointed out places where we were doing too much work in between message pumps. For example, loading some assets into the 3D scene might nominally take 200ms. On a computer with 256 MB of RAM, though, the system might start thrashing and loading the same assets would take 5s and report as a “deadlock”. The solution was to reducing the program’s working set and bite off smaller chunks of work in between pumps.

I don’t recall seeing many “computer science” deadlocks, but these watchdogs were invaluable in tracking down important failure conditions in the IMVU client.

Next time, we’ll improve the accuracy of our crash metrics and answer the question “How do you know your metrics are valid?”

* Median session length is a more useful reliability metric. It’s possible to fix crashes and see no change in your percentage of failed sessions, if fixing crashes simply causes sessions to become longer.

Jeff Lindsay: progrium

Thu, 18 Jun 2009 06:35:23 +0000

That’s right. Hide your floppies and cover your Ethernet ports. Virus-laden hackers are coming to take over your computer, steal your passwords, and do terrible things! Like hack your MySpace! Oh noes!

I guess I’m getting over the fact that we probably won’t be able to undo the damage done with the public’s perception of what a “hacker” is. Perhaps, though, we can overload it to the point of ambiguity, so there’s at least some question of context whether it’s the good kind or the bad kind.

The problem is that positive connotations aren’t enough. The people that would venture to understand “hackers” the slightest bit more than what they hear in the headlines are going to pretty quickly find the “good hackers” … they’re white hats, right?

The other day I was questioned by somebody (that should know better) if I hacked somebody’s website that was recently compromised. Seriously? My response was “I barely have enough time to do important things, let alone something that would be a waste of my time.”

I self-identify as a hacker. I invite my friends over to “hack.” I started a party for “hackers and thinkers” (a very intentional choice of words). I’m co-founding a community center called Hacker Dojo. What are we doing at all these functions? Building and learning.

We use the term perhaps too liberally, but always implying tribute to the true hackers that, as Steven Levy put it, “regard computing as the most important thing in the world.”

These people push the envelope of what’s possible through hands-on exploration, driven by relentless curiosity and a desire to challenge the status quo. Steve Wozniak, Lee Felsenstein, Linus Torvalds, Tim Berners-Lee, John Carmack …

Hell, there’s something big behind this idea, why stop at computing? Buckminster Fuller, Nikola Tesla, Richard Feynman, Alfred Kinsey, Ben Franklin …

Perhaps we’re generalizing too far. Perhaps we’re rendering “hacker” meaningless. Or are we giving it more meaning? Getting down to its essence. I wouldn’t be defending this idea so strongly if I didn’t think it had some great significance to humanity.

What upsets me is that many who would identify as hackers in this sense seem to be afraid to claim it. Most likely in fear of confusing the layman that has the media’s myopic view of hackers. You’ve never heard of the canonical conference for real hackers. No, it’s not DefCon. (Just get up and leave.) It’s a conference called Hackers. You’ve never heard of it because they keep it secret!

This conference was started to gather everybody together that was mentioned in Steven Levy’s book Hackers: Heroes of the Computer Revolution (one of the last good publication on hackers, and it came out in 1984!). This conference holds all the values of true hackerism and it’s been happening for 25 years. But they won’t promote it! It goes by a fake name and even has all mentions of “hacker” on their website replaced with an image so it won’t be indexed. Seriously??

Trying to supplant public perception of hackers by just saying they’re something different and providing a better name nobody uses (”crackers”) is not going to work. It hasn’t worked. They need something to replace those visions of crackers with. We need tangible examples and stories. We need heroes. Heroes willing to wear the title.

Luckily, we have a new generation of hackers. One that has started a global movement called hackerspaces, probably one of the biggest things for hackers in years. Our local hackerscene fostered by Silicon Valley culture and events like SuperHappyDevHouse have led to a hackerspace we hope will have a big impact. One that proudly wears the name hacker: Hacker Dojo.

Jeff Lindsay: Tweet in PostBin

Mon, 15 Jun 2009 09:48:51 +0000

When I first gave a public talk on webhooks in 2007, the guys at Twitter were the first ones to come up to me saying they wanted to implement them. Unfortunately it never happened, and this was well before they were as popular as they are now. It wouldn’t scale, they said.

As an alternative, just a few months ago, they released a stream API using long-running HTTP. It not only included a firehose stream, but a stream that allowed you to pull updates from selected users. I knew immediately from this I could build a webhook transformer, effectively giving Twitter webhooks. Unfortunately, the stream for selected users was behind a EULA that prevented that use of their API. Something about not redistributing data.

Then a few weeks ago, they added a selective stream that was publicly available. I had already designed the system and started writing some code before when I found out about the EULA. It took me a while to find some time, but last night and today I did it. I finished a proof of concept service called Twitter Hooks.

It’s very simple and is mostly about the plumbing behind it. You sign in with your Twitter account using OAuth, then you just give it a callback URL. Behind the scenes, it has a stream consumer listening for updates from the users that have a callback URL. When it gets an update from you, it posts it to your callback URL.

Obviously, there is only so much you can do with your own tweets. The next step is to listen to updates from your friends and post them to your callback URL. After that, using some trickery not involving the stream API, it will post direct messages and new followers to your URL. Throw in keyword tracking, and you have hooks for almost every significant event in Twitter. John Herren would be proud.

The problem is… Twitter is still too limiting with their stream API. As it stands, the non-EULA stream only allows following 200 users. That means Twitter Hooks can only support 200 users. It also means there’s no way we can roll out the friend update hook because some people have over 200 friends by themselves. Until that’s resolved, Twitter Hooks will remain a proof of concept demo.

But I have to say, having this is pretty sweet. The stream API is about as real-time as you can get. While experimenting, I was getting updates from the stream quicker than the page reload from posting a new tweet. This is, of course, then posted to your URL immediately for you to do whatever you want with. Here’s a tweet of mine posted to PostBin:

If you want to play with Twitter Hooks, it’s currently limited to users I authorize since it can only have 200 users and it’s terribly early/rough. Just @ me on Twitter and I can get you set up.

Jeff Lindsay: Convention for batching

Sun, 14 Jun 2009 18:55:06 +0000

Last month I talked about webhooks at Glue, a new conference on the “glue” of the web. One of the other speakers was Josh Elman of the Facebook Platform team. I ended up on the flight back with him, so we talked about webhooks. He seemed excited about them as a fan of the callback/hooking pattern in classical programming. In fact, he mentioned how they were experimenting with them already for notifications within the Facebook Platform, but he brought up the issue of batching. Something I hadn’t thought of before, but is important for large-scale implementations that are likely to be posting a lot of events to a target endpoint.

Later, when I read about Google Wave, and how they use webhooks in their API for creating bots, they mentioned that they may batch events. I’ve yet to crack open my Google Wave developer account and play with their implementation, but I’ve since realized a very simple convention for batching: JSON lists as the outer structure.

This works because an event object is ALWAYS going to be a key/pair object. So receiving code just has to check if the payload JSON is an array or an object. If it’s an array, handle each object inside as separate events. If it’s an object, handle as you would.

This doesn’t work for POST variables, since it’s very difficult to do arrays in general, but particularly as the outer data structure. It’s just not made to be able to do that. So this would require you to use at least JSON. It works in XML as well, but because there’s more variety in XML payloads, it’s probably harder to settle on such a simple convention as this.

Let me know what you think. Or if any of you already know how Google Wave batches events in their Robot API.