Monthly Archives: September 2011

How to Handle Client Reconnects

Cubeia Firebase comes with a very good High Availability support right out from the box: a Firebase cluster aren’t supposed to drop any events at all within reasonable, pragmatic, boundaries. For reference, our release testing is loading a cluster with 1000 events per second when testing fail-over. This corresponds to at least 20 000 poker players (depending on player speed), so it should be pretty safe even for you.

The Problem
Internally Firebase replicates events and state to minimize the risk of data loss, however, there is one instance where this is particular hard, and that is between the clients and the server. And here’s why:

Because Firebase is not one-to-one when mapping incoming events to players, but rather one-to-many we have chosen to not extend the transaction boundaries to the clients themselves as this would be extremely costly.

This means that if a client looses its TCP connection unexpectedly, there may be messages lost. This might come about from several scenarios, for example:

  • A client role node in the Firebase cluster crashes. This will normally disconnect all players from the node, and upon reconnection they will be load balanced to another server.
  • The client application crashes. When opened again, the client will reconnect to the Firebase cluster.
  • A glitch in the routing between the client and the server disconnects the client.

In any of the cases we’ll look at today, the client will be able to realize it has lost the connection and reconnect.

Preconditions
The only thing that is really necessary to start handling reconnects gracefully is for the client to recognise it is reconnecting. If it is handled in-process this should be trivial, and when a client starts it may look to a cookie or a saved state to see if it was closed correctly, ie. a “dirty state” flag.

Also, you need to mark crucial events with ID:s and optionally a flag which can tell you if an event is “resent” (more on this below), and also make sure such events have corresponding answers from the server. Remember that Firebase is an asynchronous system, so you’ll send a critical event, then some time later, you will get an answer, hence the importance of an event ID.

And what events are “critical”? It’s up to you really, but probably those that can significantly alter the outcome of the game. The events that you really don’t want to get lost. But the key word is “significantly”, you probably don’t want to enforce a call/response-pattern on all events in all games. In gambling most events such as bet, call, fold etc are significant, whereas in a social shooter perhaps very few are.

Idempotency to the Rescue
In event processing idempotency is usually meant as the ability to resend events without changing the initial outcome more than once. In an idempotent system, if event X is sent ten times, only the first received will be executed upon, the rest will be ignored. For Firebase this means that the client should be able to send a critical event several times and the server will simply respond as if the event is executed if it is already acted upon.

Let’s break it down:

  • The client must keep a back log of sent “critical” events which it has yet to receive answers to. So for each critical event it will save an ID in a map together with the entire event, and for each response to a critical event the ID will be removed from the map. If the client is not awaiting any response the map will be empty.
  • When a reconnection occur, the client should check if the above map is empty, if not any event therein may theoretically have been dropped. Each event in the map should be resent, optionally with a flag detailing that this is indeed a resent event.
  • The server should keep a map of ID to response objects for a reasonable amount of time, or be able to calculate a response without changing the internal state. So if it receives an event which it has already executed upon, it should recalculate the response, or fetch the response from a map.

Example: Load Test
The load test we’re using internally is very simple: Each client sends a sequence of integers and the server simply echoes the integers right back. So the client may send “5” and will wait for the server to send “5” back. It then waits for X milliseconds before increasing the sequence to “6” and repeat. This simple game allows us not only to determine the exact load at any given time (ie. events per second) but also verify that Firebase honours its FIFO behaviour.

So the client will keep a queue of integers it expects the server to respond to, and it also has an integer sequence to create new events from. The queue of integers acts as the client side map: if it is empty there’s no response outstanding. On reconnect the client checks the queue and resends any integers found in it. The server simply echoes the integers back to the client, but also checks that the sequence is intact (again checking FIFO ordering) except for the case of reconnects when the sequence is allowed to jump back a few steps.

Finally
Hopefully this should have straightened out a few questions. Keeping strict idempotency in any system is tricky, but it will give you a very robust platform, and paired off with Firebase will be damn near bullet-proof!

By |Friday, September 23, 2011|firebase|Comments Off on How to Handle Client Reconnects

Firbease Guice Support Patched

We’ve just patched the Firebase Guice Support to 1.1.1. This patch contains a single bug fix: If you configure your game to use a table listener, this listener needed also to be a tournament table listener previously. But no more!

The new version is available on the Firebase download page.

By |Wednesday, September 14, 2011|firebase|Comments Off on Firbease Guice Support Patched

Pedal to the Metal

We’ve posted a new performance investigation here. In it we’re looking at how “singleton” servers scale up in Firebase 1.7 Enterprise Edition. And the answer: It smokes!

  • We loaded 40 000 poker players on a 3 node cluster of cheap so called “pizza boxes”.
  • Going from 1 to 2 nodes increased throughput with 50%, and from 2 to 3 with an additional 60%.

Read the above one more time: 40 000 players on a cluster that today would cost you less than €1500 is astounding! Granted, there no external integrations involved here, but still I think you’ll agree that it is pretty neat.

By |Tuesday, September 13, 2011|firebase|Comments Off on Pedal to the Metal

Firebase from Scratch, pt II: Diving In

In a series of posts we’ll do “Firebase From Scratch”, an introduction to Firebase and its concepts and ideas. Hopefully, reading this series will give you a firm grasp of what Firebase is and what it can do.

So, you want to write a multi-player game? And perhaps you’re considering Cubeia Firebase as your platform? Then we’ll take a good look at what you actually need to know, and need to do to get going. We’re going to keep it short, and then expand on the different items in later posts.

Client Language
You need a game client, and should decide what language or platform you’ll be using. If you want to deploy it to the web, perhaps flex would be suitable. If you want a heavier client, you should probably go for C++ or Java. For the rest of these posts we’ll assume you went the Flex / Flash track, but Firebase supports Flex and Flash, Java, C++ and C# so it is really your choice!

The Firebase Protocol
The client API of your choice comes with ready made command objects for a number of actions that are taken care of by Firebase, and that you don’t have to do yourself! They are taken care of by the platform, and include:

  • login / logout
  • join / leave game
  • watch / unwatch game
  • query / subscribe lobby
  • send action to game / service
  • etc…

In most cases, the Firebase commands are request / response. You will, for example, send a “login” command to the Firebase servers, and then wait for an answer which will be either a “yes” or a “no” (we’ll discuss Firebase and authentication / login later). The “send action to game” is how you communicate with your game, which is running inside Firebase, and for more on that, see below.

Your Protocol
The Firebase protocol takes care of many “generic” actions for you, but you will need to have a protocol specific to your game.These are the actions your players can make, such as:

  • fire weapon
  • move
  • place bet
  • do the macarena
  • etc…

the actions you actually need will be apparent when you start writing your game, but at this point you should figure out what form the commands will take. Firebase will treat your own commands as binary data, that is to say, you can use any format you want. Many will use XML or JSON which are widely supported in all languages. For larger projects though, we’d recommend you to look at protocol generation schemes such as Google’s Protobuf as it scales better and nicely separates your protocol from your code which is something that becomes important as the game grows in size. You can even use the protocol generation used by Firebase, called Styx; This generator takes a specification in XML and produces serialization factories and command objects in a variety of languages.

Server Language
The server code is the heart of your game. This is where most decisions will take place, where you keep track of the players, send notifications and much more. Normally all Firebase games are written in Java, but you can also write your server code in a number of script languages such as JavaScript or Ruby which is brilliant if you quickly want to prototype a game. Be aware though that the script languages are a bit slower, so if high performance is a must for you, you’ll have to go the Java route.

Building Server Code
How you build your client code is very much up to you and what IDE you’re using. But for the server side code we strongly recommend use Apache Maven for building, simply because there’s some neat plugins and tools already built for you to simplify your life. It is still your choice though and you can use anything from Ant to batch scripts.

Database
You probably want to store results and data somewhere. Firebase will not impose any choice on you and you’re free to use whatever data storage you like best. Firebase contains build in support for most relations databases, such as MySQL or Oracle, via JDBC and JPA, but you can equally well use column databases or simple text files, the choice is entirely up to you!

Ready. Set. Go!
So download what you need. If you went with the Flex/Java route you will need a Java SDK installed together with Apache Maven. Strictly speaking you don’t need an IDE, you could use a simple text editor, but normally you will want to install IDE:s such as Eclipse before going on. And with that done you should be ready to dive in head first in the traditional “hello world” tutorial on our wiki. Or if your feeling brave, the “quickstart extreme – not for the faint hearted“!

Next installment will look closer at the server game itself, its components and functions. Stay tuned!

By |Monday, September 5, 2011|firebase|Comments Off on Firebase from Scratch, pt II: Diving In