Month: October 2016

Costs in Multiplayer Games

Want to know how to reduce the operational costs of your multiplayer game or just to start one by planning ahead for ultra low costs? In this article we’re going to talk about the major things that influence the cost of our whole operation in a multiplayer game. Things like: cloud services, regional servers, Content Delivery Networks (CDN), server performance, and a few others. We will approach many possible ways to reduce the costs of these, and hopefully help you come up with a good plan for your network architecture and a great system design.

Cloud Services

This one is a no-brainer when we talk about online games. While it is cheaper to  develop and test your game in a LAN, if you want to release your game to the masses, you will need an infrastructure to put all of the databases, service applications, and game servers.

Using the cloud for hosting our game is without a doubt the way to go.  It will provide a very reliable and redundant service that will guarantee that your game is up all the time.  Moreover, it will also be cheaper than buying your own hardware and hiring redundant power and internet sources to try to keep up with the same options.

Services like Amazon Web Services (AWS) and Google Cloud Platform (GCP) are great for general use cloud operations, including game. We also have some cloud services that specialize in games like AMPD Games services; which focus on delivering a lower latency in cloud services with specialized hardware for gaming operations while keeping the costs even lower than some of these big companies.


Regional Servers

Another big advantage of using cloud services is how easily you can have servers in many locations around the world, a.k.a Regional Servers. And while having more regional servers may help you provide a faster experience that will make your game feel very crisp, each of these servers incurs some costs. The more regional servers you have the pricier it will get.

When you’re first launching your game, it is a good idea to filter down to the fewest amount of Server regions to work with. This is for a few good reasons that are not related to costs.

Depending on how you have planned to update the game servers with the latest builds the time can vary from a few minutes to several hours. The fewer servers to update the easier and faster you’ll be able to fix, patch, and update your servers.

Another reason that is highly overlooked when deciding on how many regions to go for is… The users! Well, having more regions means a better experience for the users right? Not exactly, and here’s why. Yes, many regions = faster response time for the game all around, but what about how many users connect to a single region? This affects the game in a very different way that most people would just not expect.

Having several regions means that users will now divide into all the available regions, which means fewer players per region, which ultimately leads into users playing against the same users all the time or worse… Not having enough users in their region to play with/against.

Think about it, if you put a server in each country of Europe for example, you will most likely have players from that country connecting to their respective server. German players will play against players connected only to the German region server, and the same applies for every other regional server.

It is better to have one server in the center of Europe (i.e Frankfurt) which will host for all of the European players, and another one in the center of North America that will host for North American players, you could also have one in Brazil to serve the Latin American countries, so on. This slightly higher latency will not affect your game as much as not having enough players could.

It is very important to define which countries you think will be playing your game the most and work around these countries. Surveys, or any kind of poll are always a good way to assess your user-base.


ContenT Delivery Network (CDN)

This is what helps you get your game to the masses. When the users buy your game, they have to download it. If you don’t have a CDN, and host the file from a single source, not only  will the download speed will be very slow, but it will also be very expensive for you.

A simple solution and the recommended one is to go with Steam or any of these Game Distribution Services. It is just too much of a hassle to have to deal with distributing your game on top of how your game is doing. Yes, steam takes a cut, but trust me, it is well worth it since they also take care of hosting a page for your game and even some marketing. If you ask me, for all you get, this one is also a no-brainer.


Design for Scalability

This simply means, architect your whole network to be small at first and to be easy to increase in size, in other words to be very elastic.

Never assume the size of your user-base. One mistake I made while I was working on one of my previous projects was to assume that thousands of players will be playing the game from day 1. Big mistake! This did not hurt the game in any sense, but it really hurt our expenses. I planned to have several Databases per region, and separate every service into their own cloud server. Man this was a headache to “Fix”.

I had to migrate tables and data between databases, change some of the code that was relying on having several DBs, join services into a single instance of a server, and restructure how my system was going to dynamically grow. All of this in a separate testing environment to make sure everything was working properly, and then I had to roll out this environment to “Live”. This might not sound so complicated, but it was a handful to work with and very stressful since I was running against the clock, you know… Costs!

In the end, everything worked perfectly with 2 global DBs, 1 regional DB, 1 global instance, and 2 regional instances running the services and the actual game servers. Only 2 of these instances would grow dynamically. The cost reduction was massive, we ended up spending about 28% of what we were spending before.

Don’t make the same mistake.  I had to learn the hard way. Start small, design and architect everything to grow dynamically from a low cost operation. Most importantly Never assume things. This goes hand in hand with our philosophy: Make educated decisions.


Peer2Peer (P2P)

Branching out of the Server oriented path we encounter a system that can get rid of most of our server costs, this is called P2P.

A drawback of doing this is that we will be giving away the entirety of our game. This means that the users will have direct access to the Server and the Client. The reason for this is that instead of hosting game servers we will allow the users to host their own server, either deliberately or unknowingly.

This is a way how P2P gaming works, The client connects to a special server that holds records to all other clients and which ones are running a server, then the server redirects the users to one of these servers being run by a user. In this case the entire game loop will be managed by the users and just the connection will be managed by a server and DB.

We will be playing the game directly user to user, hence the name Peer2Peer. Now there are some major flaws around this method and you can imagine that now our game is completely vulnerable to being hacked and cheated. Another major big concern is the latency and performance of our game, since we cant control what kind of computer the user is using (a potato), the ISP that the users are using, if the user is throttling his connection, or simply using a lot of bandwidth in other applications.

This lack of control will hurt the performance of our game. It will be a more appropriate approach for games that do not require many resources to run. Clearly we need to be OK with the users having full access to our technology.

There might be some cases that this is a good approach for a very specific type of game, but in general it is not recommended to go with a P2P game.


Code Optimizations

Reducing the CPU and Bandwidth load on the servers will allow you to run more games per instance, hence reducing your overall hosting costs.

Some ways that can help you reduce the bandwidth load and the CPU load of your servers are the following:  Separate your server and client code, Do not use State Synchronization, Use a specialized communications library (RakNet), use your own Communication Library (Scokets), Bitwise Optimizations, Object Pooling, and so on.

In a future article I will go in depth into the details of some good optimization techniques for servers and networking in games.

Authoritative vs Non-Authoritative Servers in Games

Now that we have a better idea on how Authoritative and Non-Authoritative systems work, we can make a more educated decision about which one to use for different games. We’ll jump right into the Pros and Cons of each and compare their similarities and differences.

Authoritative Server
  • Pros:
    • Centralized control
    • Harder to get cheats/hacks
    • Easier to maintain
    • Easier to distribute updates
    • Dedicated server
  • Cons:
    • Slower response time
    • Dedicated server


Non-Authoritative Server
  • Pros
    • Faster response time
    • Server has a lower load
    • Dedicated server
  • Cons
    • Easier to cheat/hack
    • Harder to maintain
    • Harder to distribute updates
    • Dedicated server


My personal opinion is that a dedicated server is a very positive, helpful and better way to plan your system, although there may be some people that don’t require or even disagree with having a fully dedicated server for every instance of a game. That’s why its also listed as a con. There might be specific cases in which you want to mix the code of the server and the client and make a single application that holds both. But doing this will add unnecessary extra logic checks that could slow the code and that will definitely make it harder to maintain and distribute.

Distributing a centralized server update to your selected hosts is much easier and cheaper than distributing a client update to every single user. Generally speaking you want to have a dedicated server.


When to use each?

Well its not as simple as saying go for the easier one to maintain, or go for the faster one. A lot of factors will dictate which one you should use.


  • My approach

Based on the experience I have in working with these kind of systems, I have a very good general idea on how to design the system and architect all the services. I already know which type of system is better for what kind of game. While in the design phase of the game I like to start with a Fully-Authoritative Server, and depending on the needs of the game, I start delegating things little by little to the client. The easiest things you can start delegating are User Inputs and the validations of the “own user interaction”; you can go all the way to delegating almost everything to the client’s code. This will transform the server into a simple message router.

Don’t worry.  If you’re planing to make your first multiplayer game I have some general rules to follow on when to use each type of server. Like stated in previous articles, Authoritative Servers are superior for almost every case, except when your game relies on a fast response time like most FPS games.

An analogy would be to compare shooting a target with a bow and arrow versus shooting with a low caliber gun.

  • If the target is somewhat static, you can use an arrow, which allows you to be accurate and do a lot of damage.  However, if you use the gun you will still be accurate but will not do too much damage.
  • If the target is moving, you could still hit the target with the bow and arrow and still do more damage, but you will need to do extra math to compensate for the distance, arrow speed, and target movement speed too. Therefore all of these factors combined will be what we refer to as client prediction.  Now if you shoot the same target with a low caliber gun, you might do less damage but it will definitely be easier since a bullet travels much faster than an arrow.

In these cases Damage would refer to how secure and easy it is to update your code, the Projectile Speed will be the response time, the Target will be your game, and  whether it is moving or not will be the game type.

BTW, this whole analogy refers to how the game feels from a client’s point of view, it does not refer to the actual communication speed. The messages still have to travel back and forth at the same speed and in the same environment in both cases.

This then leaves the question, when should we use Non-Authoritative Servers (i.e. Gun)? First Person Shooters, Third person action games, and fast reaction-time games in general.

On the contrary, when should we use Authoritative Servers (i.e. Bow and Arrow)? MMOs, MOBAs, RPGs, ARPGs, RTS, so on… As long as the lag/latency/slow-response doesn’t affect the gameplay experience directly.

I mentioned before something about delegating most things to the Client’s code. This approach falls out of the scope of either of these 2 types of systems for a few reasons and optimizations that apply to this case. We will call this approach Peer2Peer (P2P) and we’ll talk about this in a later article which will be more related to lowering costs than to technical optimizations to the system.


Please feel free to share comments or questions, I might be missing a few important details. Looking forward to hearing from you.

Non-Authoritative Servers in Games

As we saw on the previous article about Authoritative Servers, there are some games that might be affected by how things are being handled in a game. FPS games for example, would feel unresponsive in a Fully-Authoritative Server, the data from the inputs has to travel all the way to the server, then it has to be validated, the game logic has to be updated accordingly, then the server sends the updated response back to the user, and finally the client has to render these changes to the screen. All this loop can take a while, therefore we have to come up with different ways to handle some things in our games in order to make the most out of them.

Based on this loop, if we break things down a bit, we can delegate some responsibilities to the clients. However, doing this changes many principles that define an Authoritative Server, that’s why we  call these: Non-Authoritative Servers.



  • The Server manages the game logic:

    The server still has to manage anything that is not character related, this includes: Minions, NPCs, world updates, time, physics, etc.

  • THE Client Handles all its logic:

    The client manages all the logic that is directly owned by it. All the character movements (self),  shooting, abilities, and other actions. Manages all the user input and translates it into In-Game character actions.


    The server still has the role of the central communication node. Every message sent over the network between every computer connected to the game goes through the server.  i.e. If a user wants to send an in-game private message (if supported) to another user, this message has to go through the server first and then re-routed to the target user. There is no peer to peer communication.


    Even though the clients manage their own actions, everything still has to be validated by the server in order to be updated into the game. This means that the server has to be able to check the validity of the player’s actions and accept or decline these actions.

Having the client manage some things doesn’t mean the server is now an easier task. We still have to do most of the things on the server side, we’re just handling the user inputs directly on the client and then validating these inputs on the server.

We don’t need to validate all the data, in fact, if we trust our users, we don’t even have to validate any of their actions, which will make everything much faster and feel snappier. But, is it safe to trust our users? While not every person in the world is a hacker, and not every player is a cheater, not validating this info will make our client code very easy to hack and create cheats for our game. Depending on your point of view and philosophy regarding the type of game you’re producing, it might be the worst thing for the user-base or it could not matter that much.

Lets take CS:GO for example, its a highly competitive game that relies on extremely fast reaction times. This game has allowed the clients code to handle many things, and while it works as a great game experience, this has made it easier for many people to cheat “hack” the game. This hacky/cheater culture has created many problems, it is simply not fun to play with people who are cheating. This culture has degraded the whole CS:GO experience, and many users have stopped playing it simply because of this. Lets be honest, you play a game to have fun, and playing with an unfair disadvantage is not fun in any scenario.

We’ve been talking about these things with a dedicated server in mind, but we can also have the client and the server in the same application. Although this is the default way of connecting games with Unity’s Networking, and the way that most tutorials teach you how to use Unity’s Networking, I highly recommend to have the server and client as separate applications. When you launch your game, you  might not want the server to be visible, you can always launch the server in an “invisible” way in the background, and then launch the client which will connect to this server.

This will not only help you keep everything much more organized but it will also allow you to improve and optimize the code for specific tasks. This would also help you when you want to launch your game to the public, and you want to keep the server safe and only share the client.

Many of the optimizations and approaches used in an Authoritative Server can also be used in a Non-Authoritative scenario, as long as they have nothing to do with the actual task that is being delegated to the client. We can include all the core things like optimizing the message, communication protocols, design patterns, object pooling, so on. Keep in mind that some of these could also be used in the client to optimize it’s code.

It takes a lot of trial and error to get to the amount of task delegation that is appropriate for our game. Don’t be afraid to try different things out, there is no “one and only way” to do things, and there can be many right ways to do the same thing. The more things you try, the more you learn, the more you can comprehend and optimize.

In the next article, we’ll be comparing both Authoritative-Server and Non-Authoritative Servers, their pros and cons. We will also talk about when and why to use each.

Authoritative Servers in Games

When we talk about types of servers in  multiplayer games, we usually find two abstract classifications. One allows you to centralize the control of all, or at least most of the core logic in a game, and the other lets you manage things in a more decentralized and independent way. Although each of these classifications can have many types of servers that branch off of them, at the very core, they still follow certain rules and guidelines that classify them under one of these two.

An Authoritative Server allows you to control most aspects and logic of the game from a centralized and dedicated application, a dedicated server. Although you can also have a dedicated server in a Non-Authoritative approach, they manage things in a very different way. They both have some pros and cons and how you choose which one is right for your application is more tied to the type of game rather than how you want to do things, more on this later.



Authoritative Server



    The server manages almost all of the game logic. From the physics and time scale of the game, bullets and changes in the scene,  everything about the characters, and even the invisible states of various things in our game. Basically anything that can have any type of change over the duration of the game.  Leaving all the responsibility of the functionality to the server and the rendering and audiovisuals to the client.

  • The Clients are the puppets:

    Since all the logic is being managed by the server, the clients act like puppets waiting for the next update from the server; this includes the general state of the game and all its moving parts, and the next actions to be performed by the clients, meaning: animations, sounds, etc.  As stated above, the server is the brain, and the clients are the audiovisual representation of the game.

  • Centralized communication:

    The server also has the role of the central communication node. Every message sent over the network between every computer connected to the game, goes through the server.  i.e. If a user wants to send an in-game private message to another user(if supported), this message has to go through the server first and then re-routed to the target user. Therefore, there is no peer to peer communication.

  • Data and message validation:

    Since all the messages go through the server, we have to validate all the messages and data being communicated. This makes it much more difficult to cheat in our game, since every action is being double-checked by the server.

Now onto more technical details of Authoritative Servers. I’m sure you already know by now, that the server code is completely different from the client’s code. Just wanted to make sure that was clear.

I’ll start by listing the things we need to manage on the server side: List of users, and list of movable objects. Simple right? Not exactly, each of these objects can contain a variety of attributes.  It could be beneficial to put the objects into their own separate lists.

Lets take a MOBA for example, there are users that connect to the game to play, but there are also users that spectate the game that have no specific rules on what they’re allowed to see, or they don’t even have a character assigned to them. Now we have 2 lists of users, one for playing users and another for spectating users. On this note, characters are a type of movable object, but this is a very special type of movable object that we want to have easily accessible for faster indexing and easier management. Lets say we also have Bullets, this is a very special type of list that can grow and shrink very quickly and very often. Why is this so special? Well, we might want to put just the bullets in a separate list and have them object-pooled. The same applies for minions. All these things that are “volatile” and recurrent can be put into separate lists just to optimize performance by pooling them.

We may want to have as many specific lists as possible, from which we can check things against. But we don’t want to have way too many lists that it’s actually difficult for us to keep track of which list contains what. A good rule is to have one list for every type of object that we want to be pooled, a list for players and a list for spectators.

In our MOBA example we could have the following lists: players, spectators, minions, jungle monsters, bullets, abilities. It is important to note that each character in a game, whether it’s a player or a minion, can have within itself a list of buffs, debuffs, states, and so on. This is more related to how you inherit your classes and manage your architecture. For instance if you have a Buff-Manager, you might want to have a list just for buffs and have them reference both the source and target players in order to manage these on their own.

When thinking about the Server try to always think about performance, try to optimize as much as you can while trying to keep things organized. When things go wrong, you’ll benefit from having an elegant, readable and organized code. Even though design patterns are considered a good practice, most of the time, please don’t over use them. Design patterns tend to generalize too much on how things are executed and this could incur on some big performance costs.

Even if you use all the design patterns that you could use in your game, it might still work pretty fine. Then, why do I say this? It’s all about costs! Lets say you finish your game and now you are in the process of launching your game to the masses (Yay!). You will most likely , and I recommend that you do, be using a cloud service solution for hosting your servers. You will have to perform a load test to see how many games can run simultaneously on each server. Sadly it’s never as many as you would expect, specially if your code is not optimized for performance! You’ll eventually find yourself trying to optimize your game just to reduce costs instead of investing that time and effort in developing newer and cooler features to expand your game.

Many times I see people overusing design patterns even when they’re not necessary. I personally never understood this, I will write a post about the abuse of design patterns in the future.

There are some other optimizations that can relieve the server from doing all the work. These are the things that are not necessary to keep centralized, that each client could validate without risking the game to be easily hacked, since the server will always check the validity of actions.

Managing of the direct inputs from the users is one. Instead of sending an array of bits representing each of the user’s keys, the client can validate each key-press and decide if it’s allowed to perform the action. A message will then be sent to the server and will then be re-validated. This not only reduces the load on the server, but also reduces a great amount of messages and data sent by the clients. From sending a big message per “Tick” ,containing the state of all the user inputs (per user), to sending a message on specific events containing only the action to be performed.

Another optimization would be to code your own socket communication library, and optimize the messages to send a smaller stream of bytes in comparison to the already developed serialized version used in most game engines. If you’re using Unity you’ll most likely be using RPCs for your message communication, and while it’s a great and stable solution, coding your own library will not only make things a little faster, but it will also grant you a vast amount of knowledge that you cant learn by just making games.

DO NOT – I would highly discourage the use of the state synchronization that comes with the Unity’s network communication solution. Although it has greatly improved a lot over the years, it still has a lot of overhead when sending data, unnecessary data sometimes. Two years ago I ran a Network stress test to compare Unity’s State synchronization against just using RPCs for communications. You’ll be amazed at the difference in performance.

The test was to have 30 people shoot as fast as they could, the bullets would last for 90 seconds, the bullets would move at different speeds and accelerate or decelerate (I was checking the Doppler effect on sound within Unity). Let’s be conservative and say that on average every player was doing 220 clicks per 90 seconds (2.4 per second), these were all gamers btw. So if we do the math, it comes up to almost 6,600 movable objects having their position, speed and acceleration updated to the server, a full Vector3 for each of these. I’m sure it was way more than that, I’m just being conservative. The RPC server had no issue whatsoever keeping up with this, while the state synchronization couldn’t even get 15 people to click over 20 seconds before the lag was noticeable and it started to choke. That should at least give you an idea on the performance differences.(BTW, the Doppler effect in Unity is pretty awesome, it worked surprisingly well)

What about the Client? As you might imagine, taking care of the client is not as confusing as it can be with the sever.  We still have to take care of pretty much the same lists, but with a lot less logic involved. Receiving the messages that update the objects in these lists is pretty much all we need to do to complete the communication process. The rest of the audio-visual stuff is in your hands.

Remember, the server can actually send the client a message when to do an action like cast a spell.  Hence, when you receive that specific message, you can start the character animation and spawn some particles here and there depending on positions and spawn the spell object that is being casted. I’m trusting you on knowing what to do in these cases.  Please ask if you need some help with this, I’ll be happy to help.

Although it’s great to have everything as organized and centralized in an Authoritative Server, this all comes with a cost. The responsiveness on this type of system is slower, and it can get extremely slow, depending on distance, quality of connection, wifi, and many other factors.

In games that depend on fast reaction time like an FPS, this can make them feel unresponsive. Enter Non-Authoritative Server, we will discuss this in the next article.