Asynchronous Games in Firebase; pt I

March 11th, 2010 by larsan

Since the social web started to expand, asynchronous multiplayer games seems to really have taken off. So my immediate question is obviously, can I do that on Firebase? And the short answer is: of course. For the long answer, please read on.

Let’s recap first. In a traditional synchronous game you play against other people in real-time, everyone you play against will have to be online at the same time. But given the nature of our everyday Internet use, a different pattern has emerged: in asynchronous games not all players have to be online at the same time. For example, turn-based games can easily be made asynchronous, you play your turn and then sometime in the future you opponents will have been online and performed their moves. As such, a classic game such as chess in play-be-email mode is a prime example of an asynchronous game. But you’re not limited to turn-based games, you can define the order to act on time or any metric you’d like, the only mandatory feature is that all players do not have to be online at the same time.

Firebase is fundamentally a synchronous multiplayer server. The technical difficulties Firebase set out to solve are all issues arising from the kind of distributed computing you get when a couple of thousand players have to interact in real-time. So normally you’d build an asynchronous game on a traditional web platform, as the demand on timely updates just got a lot less urgent standard web techniques can be easily applied. But having said that, is there anything you’d gain by using Firebase?

I believe there is: A synchronous option. Easy to overlook, this is actually a big one. If you build your asynchronous game on Firebase from the start, you have an immediate option of including synchronous play. Or better yet, the difference between synchronous and asynchronous could be a simple configuration issue. And by the way, who says it’s either or? Why not both?

Let me explain the above in an example: At the moment I’m hacking on a Kalaha game for Facebook on my past-time. And obviously it’ll be asynchronous, players will take their turn while logged in, and the game mechanics will inform you of your move via the Facebook streams. But if two players are online at the same time and want to play against each other, why shouldn’t they? And would quick matches be fun, say max 5 seconds per move? And how about tournaments?

Here’s some other, perhaps minor, points to consider:

  • Duplex communication: implicit in the entire discussion is that Firebase can push events both ways from and to clients and server. In HTTP you’re forced to use Comet patterns, whereas Firebase uses a persistent connection and any actions are delivered to the clients immediately.
  • Latency: Just because you’re building an asynchronous game doesn’t automatically mean latency is not important. If your game is based on actions against a common notion of time, you might be interested in latency anyway. Firebase has an very low latency compared to most application/web servers.
  • Bandwidth: If it turns out you have a hugely successful game, and a large portion of the traffic is made up of game data (as opposed to actual web pages or other contextual data) you’ll be happy just to avoid the HTTP headers and the overhead they automatically bring to the table. Firebase uses TCP only at the moment, thus dramatically reducing the bandwidth requirements.

There are other less apparent options as well, but lets be honest, anything but a huge advantage would probably be negated by the additional platform with the integration and administration it’d mean. That is to say, if you can built it using standard web tools only, why add another platform with which to communicate, integrate and administrate?

So that’s my declaration: Using Firebase gives you the entire range from highly synchronous to entirely asynchronous on one platform! Next post I’ll sketch a proposal on how to actually do it as well. Stay tuned!

Script Support in Firebase

March 10th, 2010 by larsan

Now we have released a candidate for script support in Firebase! This is something we’re very excited about as it means no more Java (unless you want to of course, old hands like me aren’t likely to change in a hurry).

This is a first release so there’s no support for tournaments or services yet. But is not far off.

We’re using Java’s built in scripting support under the hood. It turned out to be not to trivial, but not very hard either. The interesting issues are likely to arrive when we start optimizing and bug hunting. And speaking of optimizing, I ran a few bots, say 50, against a very small script (basically the equivalent of a hello world) and on avarage the bots returned on 10 ms. That’s 10 ms for network latency, Firebase internals, and script evaluation for each event. Pretty damn good! Next step there will be to start optimizing depending on the script implementation, cashing compiled scripts, mutli-threading etc.

One major up-shot of writing on a script language and re-evaluating the script for each event is the velocity: you don’t have to restart Firebase when you change code, the script is re-evaluated automatically. The round-trip time is cut dramatically!

And… You want to see code? Here’ you are, this is the server part of the Hello World tutorial, written in…

JavaScript:

function handleDataAction(action, table) {
    _log.debug('Entering handleDataAction');
    var data = _support.getActionDataAsUTF8(action);
    var playerId = action.getPlayerId();
    var outAction = _support.newGameDataAction(playerId, table);
    _support.setActionDataAsUTF8(outAction, data);
    table.getNotifier().notifyAllPlayers(outAction);
    _log.debug('Exiting handleDataAction');
}

Ruby…

def handleDataAction(action, table)
    $_log.debug("Entering handleDataAction")
    data = $_support.getActionDataAsUTF8(action)
    playerId = action.getPlayerId()
    outAction = $_support.newGameDataAction(playerId, table)
    $_support.setActionDataAsUTF8(outAction, data)
    table.getNotifier().notifyAllPlayers(outAction)
    $_log.debug('Exiting handleDataAction')
end

Python…

def handleDataAction(action, table):
    _log.debug("Entering handleDataAction")
    data = _support.getActionDataAsUTF8(action)
    playerId = action.getPlayerId()
    outAction = _support.newGameDataAction(playerId, table)
    _support.setActionDataAsUTF8(outAction, data)
    table.getNotifier().notifyAllPlayers(outAction)
    _log.debug('Exiting handleDataAction')

Groovy…

def handleDataAction(action, table) {
    _log.debug('Entering handleDataAction')
    data = _support.getActionDataAsUTF8(action)
    playerId = action.getPlayerId()
    outAction = _support.newGameDataAction(playerId, table)
    _support.setActionDataAsUTF8(outAction, data)
    table.getNotifier().notifyAllPlayers(outAction)
    _log.debug('Exiting handleDataAction')
}

Cool, eh?

You’ll notice some strange objects above. We bound some helper objects in the evaluation context, “_log” a Firebase Log4j logger, “_support” a tool for string to byte conversion etc, and some other helpful stuff.

The JavaScript Hello World can be found here.And tentative documentation here. Have fun!

Guice Support in Firebase

March 4th, 2010 by larsan

I’ve always wanted to add dependency injection support to Firebase, and today we released a candidate for Guice! And if you ask me, it’s very cool indeed.

The documentation is a bit sparse at the moment, but can be found on our wiki. The rest of the post I’ll just show how a small fictional game would look using Guice.

To start with, the Guice support comes in a set of abstract base classes, one for each Firebase artefact. And to use those you’d have to add a dependency to you Maven build (I’ll assume Maven here, you can of course use whatever you’d like):

<dependency>
  <groupId>com.cubeia.firebase</groupId>
  <artifactId>guice-support</artifactId>
  <version>1.0-RC.1</version>
</dependency>

And if you haven’t already got it, you’d need our repository as well:

<repository>
  <id>cubeia-nexus</id>
  <url>http://m2.cubeia.com/nexus/content/groups/public/</url>
  <releases>
    <enabled>true</enabled>
  </releases>
  <snapshots>
    <enabled>true</enabled>
  </snapshots>
</repository>

Now your all set to go, just extend GuiceGame and return the class of you game processor within the configuration, like so:

public class MyGame extends GuiceGame {
    public Configuration getConfigurationHelp() {
        return new ConfigurationAdapter() {
            public Class getGameProcessorClass() {
                return MyProcessor.class;
            }
        }
    }
}

So what’s the magic then? It is this: The class MyProcessor will be instantiated by Guice and can therefore contain injections. And further, it will be done in a custom scope, per event, thus isolating instances nicely.

You can also add your own modules to the injection context, again by overriding a method in GuiceGame:

protected void preInjectorCreation(List list) {
    list.add(new MyGameModule());
}

Which means, you can inject not only stuff from the current table but also, your own classes. So if we continue:

public class MyProcessor implements MyProcessor {

    /*
     * This is probably configured in the "MyGameModule" configured
     * in the guice game extension.
     */
    @Inject
    private MyHandler handler;

    /*
     * This is a speciality, you can inject Firebase services
     * right into your classes.
     */
    @Service
    private ScriptSupport support;

    /*
     * And another shortcut, if you use Log4j, we have a
     * a helper annotation for you...
     */
    @Log4j
    private Logger log;

    public void handle(GameDataAction action, Table table) {
        // do something here eh?
    }

    [...]

}

That should give you the idea. You can inject Firebase services as well as a logger (and remember, if you don’t use Log4j, Guice support the Java utility logging package from scratch). There’s a couple of things not shown here, for example, you can inject table members directly into the classes and the state object, so you don’t have to pass those around.

Any catch? Well, when you create your own modules you’ll need to keep in mind that the processor will only work in a custom scope, called EventScope. So if you have something which needs to be bound not as a singleton or in the default scope, you’ll probably need to do something like this:

bind(MyHandler.class).to(MyHandlerImpl.class).in(EventScoped.class);

And that’s it! In a few days we’ll release our script support, which is as you might imagine built on top of the Guice support. And so far? I love it!

A Google Analytics Plugin for Nexus

February 11th, 2010 by larsan

Since Firebase Community Edition uses Maven heavily, I realized I’d like to track our Nexus repository in Google Analytics. The Nexus Book says that there exists such a plugin already, but apparently now one knows where it is. So here’s my attempt to write a simple one.

If you’re an impatient kind, here’s the source code.

I created the project  in Eclipse, instructions here.

Now, let’s start off with a “repository customizer” which is Nexus extension point we’ll use to insert a custom request processor into every repository…

public class GaTrackerRepositoryCustomizer implements RepositoryCustomizer {

    private final Logger log = LoggerFactory.getLogger(getClass());

    @Inject
    @Named("gaTracker")
    private RequestProcessor gaTracker;

    @Override
    public void configureRepository(Repository rep) throws ConfigurationException {
        log.debug("Attaching tracker to: " + rep.getName());
        rep.getRequestProcessors().put("gaTracker", gaTracker);
    }

    @Override
    public boolean isHandledRepository(Repository rep) {
        boolean b = true;
        log.info("Handles repository '" + rep.getName() + "': " + b);
        return b;
    }
}

Not too complicated. We’re using injection to get hold of the actual tracker components and we’re inserting it into all repostories.

Wait, all repositories? Yes, and it’s problem I haven’t really figured out yet. Ideally I’d like to track “hosted” repositories only. However, we’ve configured all our builds to use a few common development “groups” for convenience. However, Nexus seems to treat groups as first class members, so even though an artifact may be deployed in a hosted repository while accessed through a group, the request processor will not get notified for the hosted repository, only the group. I tend to see “groups” as equivalent to “views” in which case I’d expect the hosted repository to be called as well, but, no…

Now, let’s create a request processor which will react when someone “reads” a path in a repository.  We’ll take it in pieces…

@Named("gaTracker")
public class GaTrackerRequestProcessor implements RequestProcessor {

    public static final String GA_TRACKER_ID =
        System.getProperty("cubeia.nexus.gaTrackerId");
    public static final String GA_REFERER_URL =
        System.getProperty("cubeia.nexus.gaRefererUrl");

    private final Logger log = LoggerFactory.getLogger(getClass());
    private final JGoogleAnalyticsTracker tracker;

    public GaTrackerRequestProcessor() {
        if(GA_TRACKER_ID == null) {
            String msg = "Missing system property 'cubeia.nexus.gaTrackerId'";
            throw new IllegalStateException(msg);
        }
         log.info("Creating new tracker, with id: " + GA_TRACKER_ID);
        tracker = new JGoogleAnalyticsTracker("nexus", "read", GA_TRACKER_ID);
        checkRefererUrl();
        adaptLogging();
    }

    [...]

We configure the plugin via system properties (not too beautiful, I know), the “tracker id” is the google tracking code id and is mandatory, and the “refer url” will be set on the call to GA if available.  We’re using the JGoogleAnalytics library to call GA for us. Also, I’m being naughty and throwing an illegal state exception if the tracker id is missing, since GA updates every 24 hours we’d like to be notified on errors early.

There’s  two methods above, one sets the logging in the tracker code to use to slf4j logger instead and the other checks for and sets the referer URL:

private void adaptLogging() {
    /*
     * Adapt the logging to use slf4j instead.
     */
    tracker.setLoggingAdapter(new LoggingAdapter() {

        public void logMessage(String msg) {
            log.debug(msg);
        }

        public void logError(String msg) {
            log.error(msg);
        }
    });
}

private void checkRefererUrl() {
    if(GA_REFERER_URL != null) {
        /*
         * If we have a referer URL we need to set this. However, the
         * tracker does not have a getter for the URL binding strategy, so
         * we'll simply create a new one, ugly, but works.
         */
        log.info("Modifying GA tracking to use referer URL: " + GA_REFERER_URL);
        GoogleAnalytics_v1_URLBuildingStrategy urlb;
        urlb = new GoogleAnalytics_v1_URLBuildingStrategy("nexus", "read", GA_TRACKER_ID);
        urlb.setRefererURL("http://m2.cubeia.com");
        // set new referer
        tracker.setUrlBuildingStrategy(urlb);
    }
}

Not too complicated eh? The only thing to note is that the only way to set the refer URL isby creating a new URL building strategy. Well, I can live with that.

Before we go on we’ll create a small subclass on FocusPoint which we’ll use for tracking. Since JGoogleAnalitycs is made primarily for applications the focus point will URL encode itself, however, that won’t work for us, so we need to override it’s getContentURI method:

/**
 * Simple inner class that adapts the content URI to
 * not be URL-escaped.
 */
private static class Point extends FocusPoint {

    public Point(String name) {
        super(name);
    }

    @Override
    public String getContentURI() {
        return getName();
    }
}

And finally we’ll tie it all toghether. We’ll react on “read” actions, create a URI (of form ‘//path’) and track asynchronously (which will spawn a new thread for calling GA:

 public boolean process(Repository rep, ResourceStoreRequest req, Action action) {
    if(action == Action.read) {
        /*
         * 1) create path by appending repo path to repo id
         * 2) create a subclass of focus point that handles proper URI's
         * 3) track asynchronously, this will perform the tracking on a new thread
         */
        String path = rep.getId() + req.getRequestPath();
        log.debug("Tracking path: " + path);
        FocusPoint p = new Point(path);
        tracker.trackAsynchronously(p);
    } else {
        log.debug("Ingoring action '" + action + "' for: " + req.getRequestPath());
    }
    return true;
}

And that’s it. It’s not perfect though ideally I’d like to track hosted repositories only, I’d like to avoid tracking crawlers and I would prefer not to configure via system properties (hint for you Nexus bundle users out there, you can set system properties in the “conf/wrapper.conf” files), but I’ll leave those for a rainy day.

Here’s the source code as a Maven project.

Enjoy!

Update: If you download and try to compile, you will probably be missing the JGoogleAnalytics lib. You’re welcome to use our, GA-tracked, repository if you like :-)

<repository>
  <id>cubeia-nexus</id>
  <url>http://m2.cubeia.com/nexus/content/groups/public/</url>
  <releases>
    <enabled>true</enabled>
  </releases>
  <snapshots>
    <enabled>true</enabled>
  </snapshots>
</repository>

Problems with Comments + Flash

February 10th, 2010 by larsan

For some reason I can’t seem to comment at the moment. So if you’re experiencing problems, please bear with us.

In the last post, netsql asked about action script support (as upposed to Flex) for Firebase, and I’ll post my answer here for the time being: Good point, it should be documented. As far as I know, but I’m no Flash coder, the API is the same. So you should be able to download it and then use the Flex examples to find your way.

Drop me a line (lars.j.nilsson at cubeia dot com) and I’l let you know when we have something up on the wiki. Otherwise just hang around, we’ll get it updated in the next few days.

Update: Oh, apparently commenting is on again.

Write a multiplayer game in 10 minutes or less!

February 10th, 2010 by larsan

As I was surfing along the other day it struck me that one of the coolest things about Firebase Community Edition is how incredibly fast you can get going. Do you think the title is a boast? Well, in a manner of speaking it is,  you see: we’re using Maven to build, and if you haven’t used Maven to build a Flex/Flash client before, Maven is going to start with downloading half of the Internet for you, and that will inevitably slow you down and may take a few minutes. But hear me! If you have used Maven before, and if you allow for the first time Maven will download the artifacts needed to compile the Firebase game and the Flex client, then I stand firm: you will have a Flex client and a Java server going in less than 10 minutes!

Do you want to get started on a multiplayer game really, really fast? Here’s two different ways:

  • The Extreme Quick Start – This hard-core, and Maven only, quick start will have you up in less than 5 minutes (excluding Maven download times). It does not however, send actual game actions between the client and the server, all you can do is join/leave tables and chat with other players… Excuse me? All you can do?! It’s completely awesome if you ask me.
  • The Beloved Hello World – This tutorial can be done with Maven and optionally Flexbuilder. It will explain along the way what happens, and it will also replace the Firebase standard chat with game actions (also chat) showing you how to communicate properly between game server and client. 10 minutes? Well, if you’re impatient and a fast reader, or if you do it twice you will most certainly beat the 10 minute mark.

If you ask me, and I’m obviously biased, this is extremely cool. Of course, this isn’t actually a game yet and there’s a lot more to learn before launching your own international success on Firebase, but hell, you want to write a game? Hop right to it!

Update: The commenting system seems to be behaving badly.  Even I can’t seem to comment at the moment. Please check the main blog for updates. I’ll be looking at switching blog system now…

Pomodairo v1.7 Released

February 3rd, 2010 by fredrik

We have released a new version of Pomodairo, a tool that is designed to help you work with the Pomodoro technique.

The new version is 1.7 and contains a cleaner graphical interface and some experimental Twitter integration. Check it out here: http://code.google.com/p/pomodairo/

pomodoro client

Happy Pomodoroing!

Fredrik

Firebase CE – What’s next?

January 26th, 2010 by larsan

So, we’ve released Firebase Community Edition. Now there’s a Java game server which is open source and available to be used. Now what?

Well, here’s what! There’s a couple of things we’d like to get out of the door immediately and some of them will start appearing soon indeed:

  • Script support: Java 6 has build in support for JavaScript and can be extended to support a rather large list of script engines. We’d like to add support for writing your game using any of these script languages. We’d love to see the first Ruby+Flash game out there!
  • IDE support: We already have proto code for some Eclipse plugins lying about, and these should be polished up and released. Of course, you can always use our Maven archetypes and plugins, and then import into Eclipse, but direct support would be nice too. Of course, if anyone want to use NetBeans weed have to figure out something for you to eh?
  • Documentation: It’s sparse at the moment and needs to be fleshed out. We hope you’d like to help us here by simply telling us what is missing and asking us about all those things weäve forgotten to write down. Or indeed wasn’t explained properly.
  • IoC support: This is a biggy. Again we have proto code for for Guice lying about which needs to be fixed and published, but obviously we need to add direct support for Spring as well. Actually, you can write your components in Guice now but you’d have to wire it together yourself. Spring needs to be tested, so let us get back to you on that, ok?

And that’s what we want to do the immediate future. Watch this space, this is going to be fun!

Performance Matters

January 21st, 2010 by fredrik

Scale up or scale out?

I often advocate scale-out architectures. When it comes to choice of technology I often side with the guys at YouTube (http://highscalability.com/youtube-architecture) and state that you should choose the technology that allows you to be as productive and creative as possible. There is a reason we do not build web based systems in assembler or pure C.

Granted, using a low level languages may be faster but there is usually a high cost to pay in the development phase. Besides, most time are spent on remote calls anyway, right?

Even though I consider all of the above to be true, I want to balance the discussion a bit by talking a bit about the impact of performance, or rather the lack of performance, in medium to large systems. By medium to large I mean about 10 to 100’s of servers.

For the sake of argument and simplicity I will define performance as the possibility to execute the same amount of meaningful work to the end user with a lesser number of physical servers. This may not be the most stringent or correct definition, but it will do for this post.

Complexity

Consider hosting, monitoring and maintaining a system that consists of 4 servers (e.g. 2 frontend/business servers and 2 database servers). Now consider the same system scaled out to 100 servers (e.g. 80 frontend/business servers and 20 database servers). What are the difference in running the smale scale system compared to the large one? More specifically;

What does this mean to:

  • Deployment? 2 servers are easy to do manually, but 80?
  • Rollbacks of releases?
  • Monitoring? 2 processes may fit nice in a screen, but 80?
  • Hardware failure?
  • Redundancy?
  • Network routing?

How does it compare to monitor and check the pulse on 4 servers and 100 servers?

Obviously it will be more complex to care for a larger system, but my argument is that the complexity grows quickly and it grows in more than one aspect. Increased complexity will also affect many aspects of the daily routines and project cycles in the company. Costs for maintenance will certainly go up, but most likely, project throughput will also decline. Releases and infrastructural changes must suddenly be coordinated and carefully planned for. New functionality and added services to the system must consider a more intricate integration. More constraints, such as network bandwidth and increased RPC’s will start to play a part. What is the total cost for the company?

Hardware

What about machine failure? According to this blog, http://www.linesave.co.uk/google_search_engine.html, Google has about 60 000 servers and predict that 60 machines will fail everyday. This means that a server has a predicted failure chance of 0.001 every day. Below is a chart for the chance of machine failure within a month.

Server failure prediction

As you add servers to the system, the chances for a single server downtime increases. This will put additional load on the operational personel.

Real life example

Our primary product at Cubeia is Firebase, a gameserver tailored for casual games. If we look at one of our competitors (whose name I will not mention here), we can compare our deployment requirements for a poker network setup targeted for 25 000 concurrent users. Running on Firebase we could almost host this on a single server (v1.7, octocore, 4G RAM, cost approx €2000), but lets scale out to four servers for redundancy (i.e. a server failure will not bring down the system).

Our competitor states a need for:

  • 13 Lobby servers
  • 50 Poker game servers

All in all, 63 servers for running the same functionality (assumingly since we cannot compare every detailed aspect).

What are the costs of running a system with 4 servers versus a system with 63 servers?

Predictions for monthly machine failure:

  • 4 servers: 11.31%
  • 63 servers: 84.91%

As a sidenote, according to this Gartner press release, http://www.gartner.com/it/page.jsp?id=1015715, a single x86 server costs about $400 per year in power only. Just the power saved with a 2 server system would be about $23 600 per year.

Some Last Words

I am not advocating that you should spend an insanely amount of man years to polish every function call and algorithm to achieve performance in it’s most glorious perfection. If you are a startup or a small scale company then agility and release speed is probably the most important thing to you right now. But as with everything in life, there is another side to consider as well and if your system is growing this side will become increasingly important.

So, my point is; buy that extra core-server, go for the SSD-disks in your database, do remove unnecessary CPU intensive algorithms, work out contentions and bottlenecks in your implementation! And be proud of it!

Keeping complexity and deployment sizes down will be important as you grow.

Fredrik Johansson is a founder and CEO of Cubeia Ltd.

You can contact him at: fredrik.johansson(at)cubeia.com

FCE: The Upgrade Path

December 7th, 2009 by larsan

So if you write your game with Firebase Community Edition (FCE), what happends then? After all, writing a nice game and getting players is only the first step on the journey. Well, basically this is what we’re thinking (and the information here should be taken with a pinch of salt, we’re still thinking).

If  you have a sucessfull game on FCE but don’t wan’t to share your source code? There’ll be a Firebase Standard Edition (FSE), identical to the FCE with the exception of a proprietary license.

How about scalability? Well, if you want to, you can sticky-balance sessions across multiple single instances, so in theory you can scale both FCE and FSE, but if you want scalability and high availability included, you can upgrade to the Firebase Enterprise Edition which gives you the entire package.

  • FCE -> (success, a couple of thousand players, limited budget) -> FSE
  • FCE -> (success, a lot of players, in need of scalability) -> FEE
  • FSE -> (success, needs HA and scalability) -> FEE

In terms of support, there’ll be an entire range of options, starting from the Basic Support which will offer a ticketing system for bugs, API and maintenance questions, all the way up to Enterprise Support with full telephone support, dedicated personnel, dedicated development branch and customizes hot-patches etc. But I’ll go through those in a separate post. Stay tuned!