Fundamentally, I just want to play FF Tactics against and alongside other people. The rest of this project exists to construct a framework in which other people experience the same desire, and possibly give me money for helping them do so.

This is part of my “Do One Thing” philosophy: understand the most basic abstract goal you’re setting out to achieve. In this case, it’s to reproduce the feeling of Final Fantasy Tactics‘ combat and some of its setting in a multiplayer, persistent way. All of the other theorizing and design must in some way contribute to this goal: it must either be part of creating the gameplay itself, or it must be part of encouraging other people to enjoy it, or it must be part of encouraging other people to pay for it.

In another life, I write line-of-business apps for people with far more money than sense with a group of cool people. One of the things this has taught me is the importance of writing something.

Case study: many moons ago, we started migrating all the builds of various Java components from Ant to Maven. One of the things Maven does for us is package up a little bit of metadata in each “artifact” (JAR file, web application, EAR, what have you) identifying what version it is, when it was built, whether it’s a debug build, and so on.

Of course, the first thing the technical management decided they wanted was to be able to query the components, in place, to verify that when we deploy version 4.20 of the frobnitzer module we actually get 4.20 loaded in the application, not a lingering version of 4.19 or someone’s development build. There were discussions and threads on the main mailing list and all sorts of “what is the best way to do this” and “should we use the Maven object model” and general bureaucratic foot-dragging on the whole thing; the people involved not only wanted to be able to query the system for its metadada, but wanted it written exactly right the first time.

Meanwhile, my pet corner of the universe had a similar, but much more contained, problem: the version number in the corner of one of the apps had to be adjusted by hand every build, and, as tedious and pointless tasks tend to do, it had been ignored for a while. So I wrote about 100 lines of code to pull out the Maven metadata and splat the version number from that into the corner of the screen, and called it a day.

That code is still the only actual code that anyone’s written to work with Maven metadata. Other projects are using it now, because it works, and it exists right now.

The Extreme Programming propellerheads in the audience are probably nodding along right about now and muttering something about “spike solutions”.

Recent experiences with Bamboo, which is for the most part an excellent tool, have inspired me to play around with various CI systems. I tried out Continuum, which I found to be both too clever and too stupid about Maven builds (which we use), and the old standard, CruiseControl, which I found frankly annoying to configure and manage. Annoying tools get ignored, not used to their fullest. Someone pointed me at Hudson, which is even newer than Bamboo, so I tried it out.

I think I’m in love.

The Hudson guys have taken a lot of hints from other CI tools. The builder configuration is Maven-aware, but not clever about it, which makes it very comfortable to use for Maven projects. The configuration is very straightforward, well documented, and handled entirely inside the webapp, which is in turn capable of self-hosting as a standalone program if you don’t have or don’t want to run a java web container like Tomcat or JBoss. The Hudson site does something exceptionally smart with this: they package the self-hosting version as a Java Web Start application, which means you can click a button in your browser and have a working build server in a matter of seconds.

However, a shiny demo system and simple configuration are not the only key features. I’m more interested in how Hudson behaves under a variety of “evil” build situations: does it fail sanely if the disk is full, and can it recover from non-responsive builds? In order to do this I put together a set of Maven build plugins that do the following things:

• One writes an infinite number of zeros to a file, filling the disk.
• Another spawns an unlimited number of threads, all of which live forever and none of which ever sleep.
• A third consumes the entire Java heap by creating and retaining objects in a loop.

Each of these “problem” builds can be run independantly of any others. I’ve intentionally skipped over idcmp’s fourth case: none of these explicitly ignores signals, since that’s hard to do from Java and unlikely. However, the thread-spawning build tends to ignore signals as a side-effect of swamping the CPU with context switches.

I set up the latest Hudson stable build (at the time of this writing, 1.156) as a self-hosting application in an Ubuntu 7.04 VMWare instance along with Maven 2.0.7 and ran each build individually as a Maven build.

Heaphell

The Heaphell build ran mvn hellbuild:heaphell, which causes Maven to exhaust its stack. If Maven runs in its own process, this is fine: Maven fails the build with an OutOfMemoryError and the build server itself is unaffected. On the other hand, if the build server is “clever” and tries to run Maven directly, the OutOfMemoryError may take out part of the build server as well.

Hudson handled this well. The build failed, but Hudson remained rock-solid, even after several heap exhaustion failures. On a lark I set MAVEN_OPTS to ‘-Xmx2G’ for the build and rebuilt it; it took a little longer to fail, but Hudson didn’t seem to mind.

Threadhell

The Threadhell build ran mvn hellbuild:threadhell, which causes maven to spawn threads calculating x = sin (x) indefinitely. This build gave me trouble on my Mac: the java process Maven ran in started ignoring signals and had to be killed the hard way (kill -9 or the Force Quit button in Activity Monitor).

Since this build will never end on its own, I forcibly killed it after a minute of activity. Or rather, I tried to: even the instance’s console was having trouble coping with the CPU load. Ten minutes after I stopped the build, Hudson’s Maven subprocess was still chewing up 99.7% of the CPU. Hudson did survive the anvil I dropped on the agent subprocess without complaint, but it wasn’t able to rein in its builder on its own.

Diskhell

Potentially the most destructive of the hellbuild plugins, and the reason I ran this in VMWare in the first place, Diskhell ran mvn hellbuild:diskhell, which fills an arbitrary file with zeros until it can’t fill it any more. The Hudson installation I tested this on intentionally had the work directory on the same partition as, well, the entire rest of the system.

Since Hudson and its build agents ran as a normal user, the resulting mess left the system operable. Hudson was even able to perform further builds using the few pages the Diskhell build left available. Once the disk really filled up, and df reported zero pages available, Hudson still remained operable (and even attempted builds), which impresses me a bit. It wasn’t able to do much useful work; eventually even its ability to remember that it was planning to build something failed as the build queue log filled up, but it never crashed and it recovered gracefully after I deleted the gigantic file the Diskhell build left behind. Unlike Bamboo, I wasn’t able to convince Hudson to mangle its own config files: attempting to change the configuration caused an ugly stack trace instead.

Science: It Works.

Overall, Hudson can be recovered from a few weird states without restarting the build server. However, it doesn’t seem to be able to recover itself: it won’t delete the build results on its own, for example, and it can’t actually kill off an unresponsive build agent. Since I consider Bamboo’s configuration-eating behaviour a bug, that puts Bamboo and Hudson about on par with each other for features; the choice between them comes down to price and preference.

The sources for the ‘hellbuild’ maven plugin and the example project are available under the MIT license from our public Mercurial server, or you can use it in your own builds by adding our public Maven repository as a plugin repository in your own build. If you cook up something useful, please submit a patch or a Mercurial bundle back to us at patches@unreasonent.com.

It never rains, does it? The world has to find some way to add that extra little something to any mishap to turn it into a minor disaster.

The machine hosting this site was offline due to lightning-induced damage to precisely one component: the hard drive holding the /var partition. The rest of the machine was completely unaffected, which is at best highly suspicious and at worst a sign that we’re cursed.

My post about backups seems like rather heavy-handed foreshadowing, now.

It’s merrily chugging through an install of Ubuntu Server 7.04 (the beta of 7.10 is dying over some libncurses issue), at the moment. Hopefully, it’ll detect the NIC I added after it didn’t detect the built-in PCI-E one.

Meet the machine:

• 1.8 gHz Pentium Dual Core - a good price-performance tradeoff for what we’re doing with it, and it has more cache than the other budget contenders.
• Asus P5KPL-VM - comes with everything up to and including the kitchen sink. Too bad the gig LAN isn’t supported. It also has only one IDE channel—which is one more than some of the new Dell business machines I’ve been working with lately.
• 2GB DDR2-800 - I got the Kingston ValueRAM because the fellow at my friendly cheapo computer shop said that there was a timing issue with the cheaper (but shinier) OCZ stuff. I took him on his word.
• 320gb of Seagate SATA. Jiggabytes per dollar: Excessive. Cache: 16 mb.
• Case/PSU are an unused Antec Sonata (mmmm), courtesy of Boyfriend.
• Everything else is inconsequential.

So, for well under $400 CAD we have a totally sweet dev machine with a public IP (TekSavvy rocks) and seriously excessive storage. Life is good.

If only the install would finish so I can get along with granting Owen ssh access :)

-Leigh

On the advice of the inestimable jwz, I spent today automating backups to an external drive from the source control system. I figure that that machine will eat its own drives sooner or later, and losing several months’ work would be extremely unpleasant.

The prototype registration frontend is coming along well. Certain people are trying (fairly successfully) to convince me that outsourcing registration and billing is a wise idea; having looked at the fundamental complexity of billing, I’m inclined to agree. Nevertheless we still need a way to create accounts during development and testing that’s more advanced than “INSERT INTO users …”.

Development’s slowed down a bit while I muse over the pieces I need for one of the harder problems. It turns out that allowing something that seemed harmless has compounded the difficulty elsewhere. Dastardly things, these computers, always full of traps for the unwary.

The lovely people on the internet solved my cafe-finding problem for me, so today was the first day spent coding at The End. Not entirely coincidentally, today was spent getting some of the more useful client-facing services talking to the login system. I learned two new and important things:

1. Lattes make great breakfast.
2. A certain middleware vendor’s documentation tells vicious lies.

I’m fairly accustomed to third-party software being erratically documented and somewhat creative in its interpretation of specs, but this is a whole new level of incompetence. Not all is lost, thankfully, since I already had another approach running, but the feature I wanted to use would really have improved testability. As it is, in order to get the same level of test coverage and confidence I’ll end up writing about three times as much code.

Black cat not happy.

With that wonder out of the way, though, things went reasonably well. The prototype server I’ve been hacking at is now at least vaguely aware of whether or not the client has logged in and willing to shout mean things at clients that haven’t. All in all, a fairly productive day.

Making games, like any other worthwhile task, is a phenomenal amount of work. Fortunately, there’s more than one of us working here. We are, in no particular order:

• Owen Jacobson, lead shouting vagrant
• Leigh Honeywell, spooky oracle
• Jeremy Wilson, cackling madman