I tried some different things over the years, first I toyed with the idea of describing card games as a state machine. I even implemented the basics of Texas Hold’em in UniMod. That quickly got very messy, mostly because of all the special cases but also because it’s actually quite hard to reuse a sub state machine from the outer state machine.

I also considered modelling poker as rules with a rules engine (Drools to be specific). It seems to make sense, poker is a set of rules and a rules engine takes a set of rules and a set of facts and applies these rules. However, these rules engines are meant for “business” rules, meaning they will help you apply the right insurance premium or offer the correct rebate on your order. Having a system where “any” matching rule will fire does not make for very predictable gameplay. In other words, it’s just a bad match.

For a while it seemed like my dream to create “a card rules engine” would remain just that (it was also a case of the goal not being SMART, but I digress). During the last year and a bit, I have found myself writing poker code from scratch again, this time working on Cubeia’s social poker network. As always, there were deadlines and requirements, so I could not sit around and polish on this ivory tower dream of mine. I did however apply a lot of the good design principles I had picked up over the years, mostly from my man crush Miško Hevery.

The key principles in this case were “design for testability” (with the help of TDD) and “composition over inheritance“. And so one evening it hit me that going from where I was to a DSL for poker would not even be a lot of work. So I sat down and spent just over one day before my new and shiny DSL for poker saw the light. Here’s what the definition of Texas Hold’em looks like with this DSL:

    public static GameType createTexasHoldem() {
        return new PokerGameBuilder().withRounds(
                        blinds(),
                        dealFaceDownCards(2),
                        bettingRound(PRE_FLOP, fromBigBlind()),
                        dealCommunityCards(3),
                        bettingRound(FLOP),
                        dealCommunityCards(1),
                        bettingRound(TURN),
                        dealCommunityCards(1),
                        bettingRound(RIVER)).build();
    }

and here’s what Omaha would look like:

    public static GameType createOmaha() {
        return new PokerGameBuilder().withRounds(
                        blinds(),
                        dealFaceDownCards(4),
                        bettingRound(PRE_FLOP, fromBigBlind()),
                        dealCommunityCards(3),
                        bettingRound(FLOP),
                        dealCommunityCards(1),
                        bettingRound(TURN),
                        dealCommunityCards(1),
                        bettingRound(RIVER)).
                        withHandEvaluator(new OmahaHandEvaluator()).build();
    }

You get the idea (the entire code is available from Cubeia’s open source poker repo on BitBucket). Having now achieved this goal of mine I felt very proud and slightly embarrassed that it took me so long to get there.

My takeaway from this and the message I want impress on you is to never give up on your goal and most of all to not get stuck searching for the “one grand unified model” as I did which easily gets you stuck in “analysis paralysis“, but rather break the problem down into pieces and apply adequate design principles to each part.

I hope this gets you motivated to achieve your goal!

Viktor Nordling

Read more here: MySQL Migration Adapter

Note: This will be included in Cubeia Network as of version 1.4.0

Would you want some cloud with your game server?

Cubeia Firebase was initially build for real money gambling and for deployment on dedicated hardware. It makes sense to control your hardware when you’re dealing with monetary systems but as gambling focus has shifted the last couple of years towards social gaming cloud based deployments makes much more sense. In this article we’ll present Cubeia Poker running on Amazon AWS and our initial load tests and evaluation.

Setup
In order to make these tests as real-world as possible on this stage we made the following modifications to our standard suite of software that comprises Cubeia Poker:

• Cubeia Firebase – Normally Cubeia Firebase scales in all directions by using direct-connect inter-cluster communication. But as a cloud deployment usually cannot make hard guarantees on internal bandwidth and latency we settled for a pure scale-out architecture instead. This entailed disabling the internal message bus and state replication mechanisms.
• Cubeia Network – The main challenge here was to scale the wallet functionality. Again, a real money network have demands on consistency that can be relaxed in a social setting. Our modification will be discussed later, but in essence we added sharding of table transactions to scale the load.
• Cubeia Poker – A standard build was used except small changes to provide for the above changes to Cubeia Network. Note that the changes in Cubeia Firebase was completely outside the API and nothing needed to be changes in the Poker code to accommodate them.

Wallet Sharding
The Cubeia Network accounting modules uses dual entry bookkeeping, with static accounts for the main balance of a player and session accounts for all transactions for a player at a particular tables.   When a player signs up and plays at a table, this happens:

1. The player signs up: a static account is created to hold the players balance.
2. The player sits down at a table with X chips: A session account is created for the player and the buy-in sum X is transferred from the players static account to the session account.
3. As player loose and win chips around the table the balances of the session accounts are updated.
4. The player leaves the table: The session account is closed and the resulting balance (ie. win or loss) is transferred to his static account.

The code itself is write optimized and scales well on dedicated hardware. But for this test we opted to run the session accounts, ie table transactions, on separate servers and shard these across the cluster.

Amazon AWS
We opted to run the tests on Amazon AWS in their East Coast data centers, but paid no attention to exact location.

• EC2 was used for all Firebase, Wallet and Bot servers. Pre-setup AMI’s where used with live configuration via Puppet and user data scripts. Only ephemeral storage was used. All servers ran as High-CPU Extra Large Instances (c1.xlarge).
• RDS was used for the databases with MySQL 5 without and replication and read/write setup. Provisioned IO was not used. All instances ran as Extra Large instances with an allocated 5 GB of storage.

It should be noted that apart from the large instance types no effort was made to customize or otherwise tweak the AWS installation, the servers were not even in the same data center.

25 000 Concurrent Players

The Firebase and Bot servers where launched in pairs; one pairing of with one bot server with the bot server running 25 000 players. One Wallet server where handling the static account and one the session accounts. The CPU load on the Wallet servers where 25 and 20 percent respectively, and 12 and 18 percent for the matched databases (eyeball figures).

50 000 Concurrent Players

As the load on the Wallet server was perfectly acceptable the only addition for this step was one more pair of Firebase and Bot servers.   The CPU load on the Wallet servers rose to 50 percent for both server, and 25 and 30 percent for the matched databases (eyeball figures).

75 000 Concurrent Players

One more Firebase and Bot pair was added. Another database shard and session account Wallet server where added together with an additional static account Wallet server to share the rising CPU load. The load on the Wallet servers matched the expected loads from 50′ + 25′ above.

100 000 Concurrent Players

For the final load a last Firebase and Bot pair of server where added.  Again the load on the Wallet servers matched the expected loads from above, ie. 50′ + 50′.

Test Results
Below follows some  evaluation of the actual numbers pulled from the servers during this test. Please note that the game play frequency oscillates naturally and therefore small variations on load and relations are expected.

The above chart shows the executed game events in a Firebase server correlated with Poker hands per minute.  At full load we were executing 4264 incoming events per second and generating a whooping 13 204 hands per minute (220 hands per second).

As expected from the previous graph the relations between executed events and hands per player stays flat. Each player generates approximately 2.4 events per minute and you can expect approximately 0.14 hands per minute and player.

Poker is a naturally “talkative” game. In Cubeia Poker each executed event generates approximately 10.5 outgoing events. At full load the system was executing 4264 events per second and broadcasting 46 264 events per second (approximately 1050 and 11 500 per Firebase and Bot server pair).

Using this we can calculate the cost per player to run this cluster. Two things should be taken into account here: 1) bandwidth is not included; and 2) no attempt was made to minimize the server cost by running the most effective instance type for each server. For example, each Firebase server had a CPU load of approximately 50-65% during the test and could possibly have been run on cheaper instance types. But apart from this the monthly server cost of running 100 000 concurrent poker players is $5378 which corresponds to $0.054 per player. The slight increase in cost at 75′ concurrent players is because the scaling model is not perfectly linear across this test, between 25′ and 50′ only one server was added, but between 50′ and 75′  four servers entered the cluster.

Conclusion
The total cost of running a Cubeia Poker cluster on Amazon is very low and decreases with more players despite the fact that Firebase is a heavy server (due to it’s real money origin) and that Poker is a very event heavy game. The scalability model works very well provided the limitations.

Ma’, can I go out and play with my one million friends?

In order to take this further the static account database needs to be sharded. Whereas session accounts deals with player to player interactions, static accounts do not, and the sharding is therefore relatively simple and can be done using user ID or any other available metric. One million? Sure, why not?

About the author: Lars J. Nilsson is a founder and Executive Vice President of Cubeia Ltd. He’s an autodidact programmer, former opera singer, and lunch time poet. Any questions are welcome at lars.j.nilsson in Cubeia’s domain.

First a little bit of background: Cubeia is a technology company. We work as expert consultants with some of the larger companies in the iGaming business. We also are amendable to revenue share deals which makes us interesting for entrepreneurs. So on the one hand we have established relationships with existing companies and on the other hand start-ups.

Many NDA’s are simply badly written. Others contain possible death traps for a company like ours. But mostly, these are our problems:

• Too early: We find ourselves in a position much like an investor or a company with an established RD department. If you want to explore an idea with us that’s fine, but don’t expect us to sign an NDA at an exploratory stage, if we did we’d soon have drawers full of them.
• Non-mutual: If you tell me your secret I’ll tell you mine. There’s usually no good reason for an NDA not to be mutual unless it is explicitly meant as a possible weapon. And if that’s your kind of game, don’t expect us to play along.
• No expiration: If the NDA doesn’t expire we’d soon be drowning in NDA’s. Each new agreement would just pile on the stack of existing ones, and the accumulated legal risk would increase each time we sign a new one. Also it often makes little sense: do you really have an idea so powerful that it needs to be kept under wraps forever?

Here’s an interesting detail: Cubeia often insists on Swedish jurisdiction for our agreements, and if I’ve understood it correctly, for any piece of information to be confidential according to Swedish law it not only needs to identified as such, it must also be business critical and the holder of the information must be able to show that they’ve taken steps to actually keep it secret. So not just any piece of data will do, and if you want to press claims based on the disclosure of information you also must make sure that you yourself has kept it secret. Which makes sense to me, and also renders many of the NDA’s I’ve seen fairly useless.

So where does this leave us? Obviously Cubeia will sign an NDA if needed. But you need to convince us that it is worth the time and the risk. By making it mutual, keeping it simple and supplying an expiration date the chances we’ll find it acceptable increases significantly. Also understand that we will not sign any NDA based on ideas only, or while we’re still only exploring possibilities of a deal.

In software development it is recognized that new code, however well written, will always increase complexity and risk. The same goes for legal documents, no matter how well written, the simple fact that there is  one more of them increases complexity, which equals increased cost of maintenance and increased risk.

Class loading in Java is often misunderstood and derided. I’ll not spend my time here defending it, but rather try to find a way to gain more information from multiple class loaders.

You’ll find complete source code for this article in the resources at the end.

The Problem

In any non-trivial class loader hierarchy, there’s always a danger of mixing up the class files on deployment. Code deployed on an application server or similar systems will usually share classes. Using the diagram to the right we can easily envision “artifact 1” and “artifact 2” sharing some classes, but if the packaging is not exactly right, or if we have static dependencies, we may be hit with class loading difficulties

We should also remember that the called class loader will usually ask its parent class loader first, and only when the parent class loader returns null, look to its own classes. Using the diagram above as an example, if the class loader for “artifact 1” is asked for a class it will first ask the class loader “deploy” witch in its turn will ask “system”, so the actual lookup chain will be in this order: system > deploy > artifact 1.

ClassNotFoundException

This exception means that a class file could not be found when one of the following three methods were called:

ClassLoader.loadClass(String)
ClassLoader.findSystemClass(String)
Class.forName(String)

Given the string name the class loader will usually do a normal recursive search through the parents as described above and if the class file is not found, raise the exception.

NoClassDefFoundError

This is a linkage error when a class is implicitly loaded but cannot be found in runtime. In other words, the executing class was compiled with access to the missing class, but when the code is executing the class is missing. In that sense this error is usually caused by a class not found exception.

One common source of these errors is when a parent class loader attempts to load a class which depends implicitly on classes from a child class loader. This is often the case with static factory methods where an API defines a static method like this:

public static Foo loadFoo(String fooClass);

If we assume that this factory is loaded by our “deploy” class loader but we have dependencies in any of the artifacts, there are two plausible scenarios:

• We call the factory and the class we supply as the argument is located in one of the artifact class loaders we will get a class not found because a parent class loader will not search, or even be aware of, its child class loaders.
• We call the factory and the class we supply as the argument is indeed found either in “deploy” or its parent “system”, but that class in its turn depends on a class found only  in one of the artifact class loaders we will get a class def not found error.

Given this little crash course, why is this such a huge problem? The answer is often that we do not get enough information with the errors. It seems to me there are some basic things we would like to know when any of these problems occur:

• If a class is not found by a class loader, what classes does it actually contain? Or rather, from where does it read its classes?
• If a class is not found by a class loader, is it perhaps loaded by another class loader but one which it does not have access to?

This information can only be supplied by the class loaders themselves. Which is fine if you write your own server and can add debugging code as you please, but not so interesting if you use an application server of any sort.

Enter AspectJ

Now then, could we use AspectJ to help us out? After all, it is perfectly reasonable to define “points” in the executing code we can use to gather the information we need.

Let’s see then, in order to answer our two main questions above we need to:

• Gather data when a class is loaded anywhere in the system, to
  • Maintain a map between class names and their loaders.
  • Keep a set of known resources for each class loader.
• Intercept our two exceptions and print some debugging.

Pointcut: Loading a Class

Looking at the class loaders, it turns out that there’s a very handy pointcut we can make: the call to load a class in the class loader.

pointcut source(String s, ClassLoader l) : 
        target(ClassLoader+) && 
        target(l) &&
        args(s) &&
        call(Class loadClass(String));

This pointcut has two arguments, the name of the class being asked for, and the class loader which is called. It intercepts the call to “load class” in ClassLoader or any of its descendants (that’s what the ‘+’ means).

Gather Data

Using the pointcut above we can now solve our first problem, maintain our maps of debugging data. Well start with defining the advice

after(String s, ClassLoader l) returning (Class cl) : source(s, l) {
    // do something here...
}

Some things to note here, this advice will only be called upon normal execution, because it is defined as “returning”, so we don’t have to worry about exceptions. When the advice executes we will also have access to three context items, the called class loader, the name of the class and the loaded class.

I’ll not add all code here, you find the source code of this article in the resources at the end, but what we need to do in the above advice is to map the class name to the class loader, and to create or update a set of resources associated with the class loader to include the code source of the loaded class.

All classes loaded by a secure class loader should have a “code source”. This represents the location the class was loaded from, in other words a file path, a JAR archive etc. It is available in the protection domain given the class, so you can extract the exact code source URL like this:

CodeSource src = .getProtectionDomain().getCodeSource();
URL url = (src == null ? null : src.getLocation());

So in effect the advice will look like this:

after(String s, ClassLoader l) returning (Class cl) : source(s, l) {
    /*
     * map class name to class loader
     */
    registerClassLoader(cl, l);
    /*
     * find the class code source and add it to a 
     * set mapped to the class loader
     */
    registerSources(cl, l); 
}

Debugging: Catch the ClassNotFoundException

Now, let’s catch the class not found errors. We’ll do this using the same pointcut as above but with another advice:

after(String s, ClassLoader l) throwing (ClassNotFoundException e) : source(s, l) {
    // do something here...
}

This advice will only run when a class not found is raised by the call identified by the pointcut. We could catch the exception before it is handled (using the “handler” directive) but since we want to know the class loader anyway, this method is a bit more effective.

Printing the debug information here will be rather straightforward (again, the complete sources can be found at the end of this article):

after(String s, ClassLoader l) throwing (ClassNotFoundException e) : source(s, l) {
    /*
     * get the code sources for the given class loader
     */
    LoaderSources fail = getSourcesFor(l);
    /*
     * check if can find the class in any other class loader
     */
    ClassLoader real = classLoaders.get(s);
    if(real != null) {
        /*
         * get the resources for the class loader we found
         * and print debug information
         */
        LoaderSources found = getSourcesFor(real);
        DebugPrinter.printClassNotFound(s, fail, found);
    } else {
        DebugPrinter.printClassNotFound(s, fail);
    }
}

Pointcut: Trap the NoClassDefFoundError

The no class def found error is a runtime error making it hard to catch. In this article we’ll simplify things a bit by using the special point “handler” which can be used for advice execution just before an exception is handled by a catch clause.

pointcut noDef(NoClassDefFoundError e) : 
        args(e) && 
        handler(NoClassDefFoundError);

This pointcut simply adds a handler for the error and saves the error as an argument.

Debugging: Handle the NoClassDefFoundError

We’ll use a very simple advice form here:

before(NoClassDefFoundError e) : noDef(e) {
    // do something here...
}

As a no class def found is implicit, and hence can be thrown from method executions as well as calls, the weaving would be rather extensive, so we’ll limit ourselves by using the handler. The handler mechanism has one downside though, it’s hard to get to the class loader from which the problem originated. What we’ll do here is to use the “this” object available from the pointcut which will in effect be the object which contains the catch clause.

before(NoClassDefFoundError e) : noDef(e) {
    /*
     * the class name we're looking for is the 
     * message of the exception with path separators
     * instead of dots, so...
     */
    String name = e.getMessage();
    if(name != null) {
        name = name.replace(File.separatorChar, '.');
        /*
         * use the current object to try to find a relevant class
         * loader, the higher in the stack the exception is caught, the
         * more chance it is this class loader will be of use
         */
        ClassLoader fail = attemptToFindFail(thisJoinPoint.getThis());
        ClassLoader real = classLoaders.get(name);
 
        // print debugging here
    }
}

The Rest is Plumbing

Really, just add you own or download the sources. For your reference this is an example (somewhat abbreviated) output I got when I tested this is in a Tomcat 6 installation:

>>>ClassNotFoundException
Class 'se.xec.commons.path.StringPath' was not found in the called class loader. 
However, it was found in another class loader in the system.
 ---
Failed to find class in ClassLoader: WebappClassLoader
 - with parent class loader: org.apache.catalina.loader.StandardClassLoader@1a85d38
 - with registered code sources:
        file:/test/webapps/tester/WEB-INF/classes/test/Tester.class
 ---
Class found in ClassLoader: WebappClassLoader
 - with parent class loader: org.apache.catalina.loader.StandardClassLoader@1a85d38
 - with registered code sources:
        file:/test/webapps/finder/WEB-INF/classes/test/Tester.class
        file:/test/webapps/finder/WEB-INF/lib/xec.se-commons-1.4.4.jar
<<<

For this test I used two applications. One “finder” application that works as expected and loads the “StringPath” class from a JAR in its library folder, and one “tester” application that cannot find the string path object. As you can see the debug output answers our initial questions, what code source does the class loader know of, and where can the class be found if it exists in the system.

Some Gotcha’s

• Remember that in Java classes are loaded lazily, which means that if a class exists in a code location it does not automatically mean the class is loaded and hence may not be found even with our nice test harness.
• Most application servers have their own variations of class loaders, so you may have to modify my suggestions on a case by case basis. For example, in order for this code to work on Tomcat 6, you need to trap the “find class” method instead of “load class” in the initial pointcut.
• If your no class def found errors contains a stack trace, you may be able to traverse the stack frames within the debugging code – instead of adding an additional catch clause – to make the code more general.
• Some class loader implementations uses class not found exceptions internally when a class is looked up in the parent chain, you may wish to add additional filtering to the debug code to clear up the output.
• The aspect should only be instantiated once, so make sure it is declared with “issingleton”.

The End

This should have given you an idea of what AOP could help you with in regards to debugging class loading. I’ll leave the details of using and configuring the load time weaver for a rainy day.

Resources

The source code for this article
http://www.cubeia.com/docs/articles/cl-debugging/debugging-src.jar

Demystifying class loading problems, Part 2
http://www.ibm.com/developerworks/java/library/j-dclp2.html

AspectJ documentation
http://www.eclipse.org/aspectj/doc/released/devguide/index.html

Setup

The tests were performed on Cubeia Ltd’s load test environment in Stockholm, Sweden. The participating server was configured as follows:

• Dual Core, Intel Xeon 2.4 GHz
• Linux CentOS, Kernel 2.6.18 (EPOLL enabled)
• Oracle Java HotSpot 64-Bit Server VM (16.3-b01, mixed mode)
• 1280M allocated heap memory
• Cubeia Firebase 1.7-EE Release Candidate 3

The Cubeia Firebase load test game was used, and the sample values were:

• CPU load (“cpu idle” sampled from system)
• Throughput (number of game actions executed)
• Response time (average sampled from the bots)
• Bandwidth (sampled from system)

Binary vs Web Socket

The semantics between the two sockets are similar, both are TCP-based using control characters for frame-demarcation. The main difference is that of the frame payload, whereas the standard binary socket uses a native Styx binary packaging, the Firebase implementation of web sockets uses the same packet format encoded in JSON which is a text-based encoding over UTF-8. This will impact the bandwidth reading, and also the CPU performance.

Throughput

We’ll start with throughput as it is an important factor for reading the other results. The load test uses a bot implementation that sends one event per second and bot. Thus the theoretic MAX value for 100 bots are 100 events per second.

The above chart shows the theoretical max, which equals the number of started bots, with the actual throughput in terms of events per second. We can see that the performance is very similar between the two sockets. In fact, it’s not until approx. 1500 events per second the measured difference is bigger than 1%.

For the rest of the performance results below it pays to  remember that the throughput of web sockets trailed off around 1500 events per second as all results will be reported with the theoretical max events per second in order to make them comparable.

CPU Idle

The chart below show the idle percentage on the server as the load increases.

It’s interesting to see that both sockets shows a decline in CPU use around 1500 events per second. This corresponds to the decreasing web socket through put, and one can clearly see on the graph that the diminished throughput makes the CPU utilization grow in parallel between the sockets.

Response Time

Now lets look at response times. These values are sampled from the bot server and averaged over the last 1000 requests. As such it represents a sliding window over the last 1000 events.

Again we see exactly where the throughput of the web socket implementation started diverge. Still, at 2000 events per second the response times are well under 60 milliseconds which is more than enough.

Bandwidth

The bandwidth should be a fixed percentage higher in the web socket implementation than that of the binary socket. The size of the game data does not matter so much as Firebase treats the game data as a byte array, and will Base64-endcode it during transport.

At 1500 the web socket bandwidth tapers off as the throughput slows down. Up to this point we can see that web sockets consume approx 20% more bandwidth than does the corresponding binary socket. However, this test was performedwithout measuring lobby data which would probably slew the result significantly.

Conclusion

After this first glance we can conclude that web sockets performs really well compared to their binary counterparts.  Even the bandwidth difference is not too concerning when the lobby data is not included. For very intensive applications binary sockets still reign supreme but in most cases web socket will prove to be more than enough.

Topology

When a Firebase server is used stand-alone the configuration is called “singleton”, and that server will run all necessary roles complete functionality (master, client, game and tournament). Normally a multi-node Firebase Cluster is configured in a role based topology where different servers handle different roles in the cluster. For example, you may run client sessions on dedicated machines and all game play on others. However, for smaller installations it may be easier to run several singleton servers side by side.

Setup

The tests were performed on Cubeia Ltd’s load test environment in Stockholm, Sweden. The participating servers were configured as follows:

• Dual Core, Intel Xeon 2.4 GHz
• Linux CentOS, Kernel 2.6.18 (EPOLL enabled)
• Oracle Java HotSpot 64-Bit Server VM (16.3-b01, mixed mode)
• 1280M allocated heap memory
• Cubeia Firebase 1.7-EE Release Candidate 3

As load test game Cubeia Poker was used (nightly build 18th Aug 2011), and the sample values were:

• CPU load (“cpu idle” sampled from system)
• Hands per minute (poker hands executed)
• Actions per second (number of game actions executed)

Game Configuration

The game was configured using one speed for all tables, and 10 seats per table. The bots were configured to react within 2.5 and 3.3 seconds on normal actions (see “player speed and hands per minute” below”).

No external integrations such as wallet or databases were used in order to test the Firebase performance isolated. In real world scenarios the performance would be greatly affected by the latency and throughput of such integrations.

All bots were subscribing to the entire lobby. For a large installation (10 000 players or more) this is not recommended as it generates a massive load. Cubeia Firebase has the ability to partition the lobby so that players only needs to subscribe to a part of the whole lobby.

Actions Versus Hands

The actions sampled represents the total number of actions executed by the poker game. This value is directly linked to hands per minute as shown below, using samples from the single server scenario.

As the graph shows, the relationship between hands per minute and actions per second appears to be fixed. Exploring the the full sample set the relationship between these two values seems to fluctuate between 2.3 and 2.6 (events per second / hands per minute).

Player Speed and Hands per Minute

Another relationship which is expected to scale linearly is between the number of players and hands per minute. The bots used for this test reacted to requests in between 3.3 and 2.5 seconds, thus a full table with 10 players is expected to generate approximately one event every 2.8 seconds and, for example, 2 000 players (200 tables) approximately 56 events per second.

Again the graph is sampled from a single server. The relationship between the number of tables and the number of of events seems to by slightly less than assumed above, but this is to be expected when hand start/end and system latency is included in the calculation.

Performance

Limiting Factors

Each test run was ended when the CPU load of the machines reached up towards 80%. In reality the limiting factor was the CPU in relationship with the Java VM Garbage Collector (GC): as the number of player grows, the initial snapshot for lobby subscription gets larger, and when the available overhead gets too small the GC will fail to release enough memory. Large installations (10 000 players or more) should consider allocation significantly more memory to server “client” roles.

This limitation is not unique for singleton-style clusters however: even in mixed topologies large lobbies will put higher demands on the client role heap size.

CPU

The following chart shows the CPU load for 1-3 server together with the corresponding number of players connected.

We can see that the load peaked at 16 000 players for a single server, 24 000 for 2 servers and 40 000 for three servers.

Going from 1 to 2 server increases the maximum load with approximately 50%. This is because a single server does not have to replicate state for fail over, and as such will not scale linearly compared to a single server.

Hands per Minute

The following chart shows the hands per minute for 1-3 servers together with the corresponding number of players connected.

The game performance scales almost perfectly across all servers, and the peak was reached at 2 763 hand per minute when 40 000 players were connected.

The speed of the bots seems to correspond to a factor of 13-14 in relation to the hands per minute, so for example, we would expect approximately 1 400 hands per minute for 20 000 players.

Conclusion

Cubeia Firebase 1.7 Enterprise Edition shows healthy performance and scalability numbers even when configured to use singleton nodes. The hardware for the tests is a few years old and is lacking on RAM, it is recommended that very large installations (20 000 players or more) is set-up on hardware with significantly more memory available.