Category: Development

Optimizing Multiplayer Games

These are some ways that can help us optimize our multiplayer games. Ways in which we can reduce the CPU and Bandwidth load of our hardware and/or cloud instances.

Some reasons you would want to improve the performance of your servers are: to greatly reduce costs, to have faster servers that will allow you to provide a much better experience, or to simply have a much deeper understanding of the technology you’re using, whether you launch your game or not.

A multiplayer game, while similar in some ways to a normal single-player game, is very different in its very core. A well designed multiplayer game will have a totally different system architecture than its single-player counterpart.

We need to be very aware that a single clock-cycle can mean the difference between a good experience and a lag-spike. A little dramatic? Not really, while technology is getting better and better by the second, we still can’t control how our packets travel through the internet. Since we can’t control the packet loss, all the different routing tables, the performance in each node, and many other things, this is why we have to care so much about the performance of our applications. The few things we can actually control, should be optimized for performance as much as possible.


Server and client are different things

If you read my previous article about Costs in Multiplayer Games, you already know this. You need to separate your server’s code from your client’s code.

This comes with many perks, you will have a more organized and dedicated code. Making it easier to maintain, fix, update, and distribute. Think about it, if you only update your client code, you do not have to also update your servers. Same way on the other side, if you update your server’s code, you will not have to update all the users with the new “Compound Code”.

On the same note, you can strip down a lot of unnecessary logic that is only used on “the other side (server or client)”. You can also specialize logic that was too general to be used on both sides.

This will help us by not having that extra logic to take care of, things that do not pertain to our direct application. Validating all this unnecessary logic, just to make sure if certain code should be allowed to run in the server or the client, could leech some precious clock-cycles from our applications.

 


Send Fewer Messages

While you want to keep your game state as updated as possible, it makes a very big difference to send continuous messages compared to sending messages just on events. i.e. Sending the full state of your character (Position, health, mana, the state of very single key, etc..) every frame will have a much higher impact on both your computer and network performance than if you were to send only the required information when it’s needed.

Having smaller specific messages that tell the story of how things are changing is much better than having the whole snapshot of your game for every frame per user.

Having events in your code that run specific messages will help you achieve this. For instance, if you press a key, validate on the client side the action and then send a message specific to this action to the server.  You can also do this on the server side, i.e. When validating if an ability hits someone, you can call an “Ability Hit Event” that will take care of doing damage, user notifications which can include both the aggressor and the victim, and so on. I’m sure you can get creative with these events.

 


Send Smaller Messages

This point goes hand in hand with the previous one. If you already thought about braking down those massive continuous messages into smaller eventual messages, you’re naturally sending less information per message. But there are some very useful ways to make things even smaller.

Why do we want to make our messages so small? Well as stated before, we cant control the internet. We can safely assume that the time it takes a message to be delivered is much bigger than the time it takes our computer to process the same message in many ways. By having fewer and smaller messages, we save ourselves a big percentage of our communications issues.

A great way to make our messages smaller is to see our attributes in terms of bits, yes 1’s and 0’s, I like to call this – Bitwise Optimization. For instance, a Boolean is equivalent to a single bit, we can group up several states in our game that are represented in bools (for example stun, blinded, silenced, …) and pack them into a single Byte. Then send a message with a single byte rather than sending a message with several bools. The reason these two ways are so different is because of serialization. If you’re using a built in communication library like UNET, this serializes the attributes and then sends this serialized byte array. This serialized information of multiple bools ends up being bigger than even a serialized byte.

You can do the same with small numbers that never go above a certain value. Lets take life for example, the life value will always be between [0 – 100]. This value will have a maximum of 7 bits and if you do the same with 3 other values, you now will have four 7bit values that you can easily encapsulate into a UInt32 and still have room for 4 bools. This UInt32 serialized will be smaller than the serialized array of 4 bytes and 4 bools containing exactly the same information.

 


No State Synchronization

State synchronization is getting better and better each day, so much that is getting difficult to distinguish the performance in small applications. Still not the cream of the crop when talking about performance in game communication.

Don’t get me wrong, state sync is great for prototypes, and it is certainly easier to learn how to use than to use a specialized or custom network communications library.

State synchronization has a lot of overhead in many ways, serializing and de-serializing messages, using reflection (that could be Unbuffered). The time it takes to process attributes into and from a message is just a little too big for my taste, not to mention the size in bytes of the serialized messages. Having bigger messages means more/bigger packets, and more/bigger packets lead to… Packet Loss.

Maybe I’m just biased because I like having everything as “Crisp” as possible. My “assembly mindset” gets in the way a little too much. Either a good or a bad thing in your eyes, it has allowed me to see things in different light, and made me want to improve many things  that someone would normally just let go. Yes it takes more effort, time, and the drive to want to improve such “little things”, but trust me, being this meticulous pays off big time.

You will have much more control over the content and frequency/eventuality of your messages. This goes hand in hand also with the two previous points of optimization – Send fewer and smaller messages.

 


Custom Network Library

While many optimizations can be done with a built-in network communications library, having a custom networking library will allow you, by far, the greatest amount of room to make improved optimizations in many ways.

You will be able to control how you encapsulate and deliver your messages. This will allow you to take full advantage of the previously mentioned optimizations and will allow these to have great synergy.  Moreover, it will help you deeply understand how the whole communication process works inside-out, and if done well, it will never make you want to look back.

I will not go into detail on how to code your own sockets communication library, there are several tutorials, videos, and resources out there that explain it very well. I will however encourage you to at least code your own, even if you’re never going to use it in your game. It will open your eyes and allow you to see things in a different light when developing multiplayer games.

 


General Optimizations

I had to include this whole section just to mention some of the more obvious ways to optimize code in general, and things that apply to general coding and not only to Multiplayer Networked Games. i.e. Object Pooling and Design Patterns (as long as you don’t abuse them). All those things that help make your code more efficient will also help make a better networked game.

There are some design patterns like Command Design Pattern that are useful in multiplayer games for specific reasons like recording a whole multiplayer session so you can then save it and replay it in an In-Game-Replay kinda way.

I hope you found this article useful and clear. I know I’m missing some optimization techniques, I was aiming to explain a few of the most useful but lesser known approaches. Please feel free to add questions or comments below.

Costs in Multiplayer Games

Want to know how to reduce the operational costs of your multiplayer game or just to start one by planning ahead for ultra low costs? In this article we’re going to talk about the major things that influence the cost of our whole operation in a multiplayer game. Things like: cloud services, regional servers, Content Delivery Networks (CDN), server performance, and a few others. We will approach many possible ways to reduce the costs of these, and hopefully help you come up with a good plan for your network architecture and a great system design.

Cloud Services

This one is a no-brainer when we talk about online games. While it is cheaper to  develop and test your game in a LAN, if you want to release your game to the masses, you will need an infrastructure to put all of the databases, service applications, and game servers.

Using the cloud for hosting our game is without a doubt the way to go.  It will provide a very reliable and redundant service that will guarantee that your game is up all the time.  Moreover, it will also be cheaper than buying your own hardware and hiring redundant power and internet sources to try to keep up with the same options.

Services like Amazon Web Services (AWS) and Google Cloud Platform (GCP) are great for general use cloud operations, including game. We also have some cloud services that specialize in games like AMPD Games services; which focus on delivering a lower latency in cloud services with specialized hardware for gaming operations while keeping the costs even lower than some of these big companies.

 

Regional Servers

Another big advantage of using cloud services is how easily you can have servers in many locations around the world, a.k.a Regional Servers. And while having more regional servers may help you provide a faster experience that will make your game feel very crisp, each of these servers incurs some costs. The more regional servers you have the pricier it will get.

When you’re first launching your game, it is a good idea to filter down to the fewest amount of Server regions to work with. This is for a few good reasons that are not related to costs.

Depending on how you have planned to update the game servers with the latest builds the time can vary from a few minutes to several hours. The fewer servers to update the easier and faster you’ll be able to fix, patch, and update your servers.

Another reason that is highly overlooked when deciding on how many regions to go for is… The users! Well, having more regions means a better experience for the users right? Not exactly, and here’s why. Yes, many regions = faster response time for the game all around, but what about how many users connect to a single region? This affects the game in a very different way that most people would just not expect.

Having several regions means that users will now divide into all the available regions, which means fewer players per region, which ultimately leads into users playing against the same users all the time or worse… Not having enough users in their region to play with/against.

Think about it, if you put a server in each country of Europe for example, you will most likely have players from that country connecting to their respective server. German players will play against players connected only to the German region server, and the same applies for every other regional server.

It is better to have one server in the center of Europe (i.e Frankfurt) which will host for all of the European players, and another one in the center of North America that will host for North American players, you could also have one in Brazil to serve the Latin American countries, so on. This slightly higher latency will not affect your game as much as not having enough players could.

It is very important to define which countries you think will be playing your game the most and work around these countries. Surveys, or any kind of poll are always a good way to assess your user-base.

 

ContenT Delivery Network (CDN)

This is what helps you get your game to the masses. When the users buy your game, they have to download it. If you don’t have a CDN, and host the file from a single source, not only  will the download speed will be very slow, but it will also be very expensive for you.

A simple solution and the recommended one is to go with Steam or any of these Game Distribution Services. It is just too much of a hassle to have to deal with distributing your game on top of how your game is doing. Yes, steam takes a cut, but trust me, it is well worth it since they also take care of hosting a page for your game and even some marketing. If you ask me, for all you get, this one is also a no-brainer.

 

Design for Scalability

This simply means, architect your whole network to be small at first and to be easy to increase in size, in other words to be very elastic.

Never assume the size of your user-base. One mistake I made while I was working on one of my previous projects was to assume that thousands of players will be playing the game from day 1. Big mistake! This did not hurt the game in any sense, but it really hurt our expenses. I planned to have several Databases per region, and separate every service into their own cloud server. Man this was a headache to “Fix”.

I had to migrate tables and data between databases, change some of the code that was relying on having several DBs, join services into a single instance of a server, and restructure how my system was going to dynamically grow. All of this in a separate testing environment to make sure everything was working properly, and then I had to roll out this environment to “Live”. This might not sound so complicated, but it was a handful to work with and very stressful since I was running against the clock, you know… Costs!

In the end, everything worked perfectly with 2 global DBs, 1 regional DB, 1 global instance, and 2 regional instances running the services and the actual game servers. Only 2 of these instances would grow dynamically. The cost reduction was massive, we ended up spending about 28% of what we were spending before.

Don’t make the same mistake.  I had to learn the hard way. Start small, design and architect everything to grow dynamically from a low cost operation. Most importantly Never assume things. This goes hand in hand with our philosophy: Make educated decisions.

 

Peer2Peer (P2P)

Branching out of the Server oriented path we encounter a system that can get rid of most of our server costs, this is called P2P.

A drawback of doing this is that we will be giving away the entirety of our game. This means that the users will have direct access to the Server and the Client. The reason for this is that instead of hosting game servers we will allow the users to host their own server, either deliberately or unknowingly.

This is a way how P2P gaming works, The client connects to a special server that holds records to all other clients and which ones are running a server, then the server redirects the users to one of these servers being run by a user. In this case the entire game loop will be managed by the users and just the connection will be managed by a server and DB.

We will be playing the game directly user to user, hence the name Peer2Peer. Now there are some major flaws around this method and you can imagine that now our game is completely vulnerable to being hacked and cheated. Another major big concern is the latency and performance of our game, since we cant control what kind of computer the user is using (a potato), the ISP that the users are using, if the user is throttling his connection, or simply using a lot of bandwidth in other applications.

This lack of control will hurt the performance of our game. It will be a more appropriate approach for games that do not require many resources to run. Clearly we need to be OK with the users having full access to our technology.

There might be some cases that this is a good approach for a very specific type of game, but in general it is not recommended to go with a P2P game.

 

Code Optimizations

Reducing the CPU and Bandwidth load on the servers will allow you to run more games per instance, hence reducing your overall hosting costs.

Some ways that can help you reduce the bandwidth load and the CPU load of your servers are the following:  Separate your server and client code, Do not use State Synchronization, Use a specialized communications library (RakNet), use your own Communication Library (Scokets), Bitwise Optimizations, Object Pooling, and so on.

In a future article I will go in depth into the details of some good optimization techniques for servers and networking in games.

Authoritative vs Non-Authoritative Servers in Games

Now that we have a better idea on how Authoritative and Non-Authoritative systems work, we can make a more educated decision about which one to use for different games. We’ll jump right into the Pros and Cons of each and compare their similarities and differences.

Authoritative Server
  • Pros:
    • Centralized control
    • Harder to get cheats/hacks
    • Easier to maintain
    • Easier to distribute updates
    • Dedicated server
  • Cons:
    • Slower response time
    • Dedicated server

 

Non-Authoritative Server
  • Pros
    • Faster response time
    • Server has a lower load
    • Dedicated server
  • Cons
    • Easier to cheat/hack
    • Harder to maintain
    • Harder to distribute updates
    • Dedicated server

 

My personal opinion is that a dedicated server is a very positive, helpful and better way to plan your system, although there may be some people that don’t require or even disagree with having a fully dedicated server for every instance of a game. That’s why its also listed as a con. There might be specific cases in which you want to mix the code of the server and the client and make a single application that holds both. But doing this will add unnecessary extra logic checks that could slow the code and that will definitely make it harder to maintain and distribute.

Distributing a centralized server update to your selected hosts is much easier and cheaper than distributing a client update to every single user. Generally speaking you want to have a dedicated server.

 

When to use each?

Well its not as simple as saying go for the easier one to maintain, or go for the faster one. A lot of factors will dictate which one you should use.

 

  • My approach

Based on the experience I have in working with these kind of systems, I have a very good general idea on how to design the system and architect all the services. I already know which type of system is better for what kind of game. While in the design phase of the game I like to start with a Fully-Authoritative Server, and depending on the needs of the game, I start delegating things little by little to the client. The easiest things you can start delegating are User Inputs and the validations of the “own user interaction”; you can go all the way to delegating almost everything to the client’s code. This will transform the server into a simple message router.

Don’t worry.  If you’re planing to make your first multiplayer game I have some general rules to follow on when to use each type of server. Like stated in previous articles, Authoritative Servers are superior for almost every case, except when your game relies on a fast response time like most FPS games.

An analogy would be to compare shooting a target with a bow and arrow versus shooting with a low caliber gun.

  • If the target is somewhat static, you can use an arrow, which allows you to be accurate and do a lot of damage.  However, if you use the gun you will still be accurate but will not do too much damage.
  • If the target is moving, you could still hit the target with the bow and arrow and still do more damage, but you will need to do extra math to compensate for the distance, arrow speed, and target movement speed too. Therefore all of these factors combined will be what we refer to as client prediction.  Now if you shoot the same target with a low caliber gun, you might do less damage but it will definitely be easier since a bullet travels much faster than an arrow.

In these cases Damage would refer to how secure and easy it is to update your code, the Projectile Speed will be the response time, the Target will be your game, and  whether it is moving or not will be the game type.

BTW, this whole analogy refers to how the game feels from a client’s point of view, it does not refer to the actual communication speed. The messages still have to travel back and forth at the same speed and in the same environment in both cases.

This then leaves the question, when should we use Non-Authoritative Servers (i.e. Gun)? First Person Shooters, Third person action games, and fast reaction-time games in general.

On the contrary, when should we use Authoritative Servers (i.e. Bow and Arrow)? MMOs, MOBAs, RPGs, ARPGs, RTS, so on… As long as the lag/latency/slow-response doesn’t affect the gameplay experience directly.

I mentioned before something about delegating most things to the Client’s code. This approach falls out of the scope of either of these 2 types of systems for a few reasons and optimizations that apply to this case. We will call this approach Peer2Peer (P2P) and we’ll talk about this in a later article which will be more related to lowering costs than to technical optimizations to the system.

 

Please feel free to share comments or questions, I might be missing a few important details. Looking forward to hearing from you.

Non-Authoritative Servers in Games

As we saw on the previous article about Authoritative Servers, there are some games that might be affected by how things are being handled in a game. FPS games for example, would feel unresponsive in a Fully-Authoritative Server, the data from the inputs has to travel all the way to the server, then it has to be validated, the game logic has to be updated accordingly, then the server sends the updated response back to the user, and finally the client has to render these changes to the screen. All this loop can take a while, therefore we have to come up with different ways to handle some things in our games in order to make the most out of them.

Based on this loop, if we break things down a bit, we can delegate some responsibilities to the clients. However, doing this changes many principles that define an Authoritative Server, that’s why we  call these: Non-Authoritative Servers.

 

NON-AUTHORITATIVE SERVER

  • The Server manages the game logic:

    The server still has to manage anything that is not character related, this includes: Minions, NPCs, world updates, time, physics, etc.

  • THE Client Handles all its logic:

    The client manages all the logic that is directly owned by it. All the character movements (self),  shooting, abilities, and other actions. Manages all the user input and translates it into In-Game character actions.

  • CENTRALIZED COMMUNICATION:

    The server still has the role of the central communication node. Every message sent over the network between every computer connected to the game goes through the server.  i.e. If a user wants to send an in-game private message (if supported) to another user, this message has to go through the server first and then re-routed to the target user. There is no peer to peer communication.

  • DATA AND MESSAGE VALIDATION:

    Even though the clients manage their own actions, everything still has to be validated by the server in order to be updated into the game. This means that the server has to be able to check the validity of the player’s actions and accept or decline these actions.

Having the client manage some things doesn’t mean the server is now an easier task. We still have to do most of the things on the server side, we’re just handling the user inputs directly on the client and then validating these inputs on the server.

We don’t need to validate all the data, in fact, if we trust our users, we don’t even have to validate any of their actions, which will make everything much faster and feel snappier. But, is it safe to trust our users? While not every person in the world is a hacker, and not every player is a cheater, not validating this info will make our client code very easy to hack and create cheats for our game. Depending on your point of view and philosophy regarding the type of game you’re producing, it might be the worst thing for the user-base or it could not matter that much.

Lets take CS:GO for example, its a highly competitive game that relies on extremely fast reaction times. This game has allowed the clients code to handle many things, and while it works as a great game experience, this has made it easier for many people to cheat “hack” the game. This hacky/cheater culture has created many problems, it is simply not fun to play with people who are cheating. This culture has degraded the whole CS:GO experience, and many users have stopped playing it simply because of this. Lets be honest, you play a game to have fun, and playing with an unfair disadvantage is not fun in any scenario.

We’ve been talking about these things with a dedicated server in mind, but we can also have the client and the server in the same application. Although this is the default way of connecting games with Unity’s Networking, and the way that most tutorials teach you how to use Unity’s Networking, I highly recommend to have the server and client as separate applications. When you launch your game, you  might not want the server to be visible, you can always launch the server in an “invisible” way in the background, and then launch the client which will connect to this server.

This will not only help you keep everything much more organized but it will also allow you to improve and optimize the code for specific tasks. This would also help you when you want to launch your game to the public, and you want to keep the server safe and only share the client.

Many of the optimizations and approaches used in an Authoritative Server can also be used in a Non-Authoritative scenario, as long as they have nothing to do with the actual task that is being delegated to the client. We can include all the core things like optimizing the message, communication protocols, design patterns, object pooling, so on. Keep in mind that some of these could also be used in the client to optimize it’s code.

It takes a lot of trial and error to get to the amount of task delegation that is appropriate for our game. Don’t be afraid to try different things out, there is no “one and only way” to do things, and there can be many right ways to do the same thing. The more things you try, the more you learn, the more you can comprehend and optimize.

In the next article, we’ll be comparing both Authoritative-Server and Non-Authoritative Servers, their pros and cons. We will also talk about when and why to use each.

Authoritative Servers in Games

When we talk about types of servers in  multiplayer games, we usually find two abstract classifications. One allows you to centralize the control of all, or at least most of the core logic in a game, and the other lets you manage things in a more decentralized and independent way. Although each of these classifications can have many types of servers that branch off of them, at the very core, they still follow certain rules and guidelines that classify them under one of these two.

An Authoritative Server allows you to control most aspects and logic of the game from a centralized and dedicated application, a dedicated server. Although you can also have a dedicated server in a Non-Authoritative approach, they manage things in a very different way. They both have some pros and cons and how you choose which one is right for your application is more tied to the type of game rather than how you want to do things, more on this later.

 

 

Authoritative Server

 

  • THE SERVER IS THE PUPPET MASTER:

    The server manages almost all of the game logic. From the physics and time scale of the game, bullets and changes in the scene,  everything about the characters, and even the invisible states of various things in our game. Basically anything that can have any type of change over the duration of the game.  Leaving all the responsibility of the functionality to the server and the rendering and audiovisuals to the client.

  • The Clients are the puppets:

    Since all the logic is being managed by the server, the clients act like puppets waiting for the next update from the server; this includes the general state of the game and all its moving parts, and the next actions to be performed by the clients, meaning: animations, sounds, etc.  As stated above, the server is the brain, and the clients are the audiovisual representation of the game.

  • Centralized communication:

    The server also has the role of the central communication node. Every message sent over the network between every computer connected to the game, goes through the server.  i.e. If a user wants to send an in-game private message to another user(if supported), this message has to go through the server first and then re-routed to the target user. Therefore, there is no peer to peer communication.

  • Data and message validation:

    Since all the messages go through the server, we have to validate all the messages and data being communicated. This makes it much more difficult to cheat in our game, since every action is being double-checked by the server.

Now onto more technical details of Authoritative Servers. I’m sure you already know by now, that the server code is completely different from the client’s code. Just wanted to make sure that was clear.

I’ll start by listing the things we need to manage on the server side: List of users, and list of movable objects. Simple right? Not exactly, each of these objects can contain a variety of attributes.  It could be beneficial to put the objects into their own separate lists.

Lets take a MOBA for example, there are users that connect to the game to play, but there are also users that spectate the game that have no specific rules on what they’re allowed to see, or they don’t even have a character assigned to them. Now we have 2 lists of users, one for playing users and another for spectating users. On this note, characters are a type of movable object, but this is a very special type of movable object that we want to have easily accessible for faster indexing and easier management. Lets say we also have Bullets, this is a very special type of list that can grow and shrink very quickly and very often. Why is this so special? Well, we might want to put just the bullets in a separate list and have them object-pooled. The same applies for minions. All these things that are “volatile” and recurrent can be put into separate lists just to optimize performance by pooling them.

We may want to have as many specific lists as possible, from which we can check things against. But we don’t want to have way too many lists that it’s actually difficult for us to keep track of which list contains what. A good rule is to have one list for every type of object that we want to be pooled, a list for players and a list for spectators.

In our MOBA example we could have the following lists: players, spectators, minions, jungle monsters, bullets, abilities. It is important to note that each character in a game, whether it’s a player or a minion, can have within itself a list of buffs, debuffs, states, and so on. This is more related to how you inherit your classes and manage your architecture. For instance if you have a Buff-Manager, you might want to have a list just for buffs and have them reference both the source and target players in order to manage these on their own.

When thinking about the Server try to always think about performance, try to optimize as much as you can while trying to keep things organized. When things go wrong, you’ll benefit from having an elegant, readable and organized code. Even though design patterns are considered a good practice, most of the time, please don’t over use them. Design patterns tend to generalize too much on how things are executed and this could incur on some big performance costs.

Even if you use all the design patterns that you could use in your game, it might still work pretty fine. Then, why do I say this? It’s all about costs! Lets say you finish your game and now you are in the process of launching your game to the masses (Yay!). You will most likely , and I recommend that you do, be using a cloud service solution for hosting your servers. You will have to perform a load test to see how many games can run simultaneously on each server. Sadly it’s never as many as you would expect, specially if your code is not optimized for performance! You’ll eventually find yourself trying to optimize your game just to reduce costs instead of investing that time and effort in developing newer and cooler features to expand your game.

Many times I see people overusing design patterns even when they’re not necessary. I personally never understood this, I will write a post about the abuse of design patterns in the future.

There are some other optimizations that can relieve the server from doing all the work. These are the things that are not necessary to keep centralized, that each client could validate without risking the game to be easily hacked, since the server will always check the validity of actions.

Managing of the direct inputs from the users is one. Instead of sending an array of bits representing each of the user’s keys, the client can validate each key-press and decide if it’s allowed to perform the action. A message will then be sent to the server and will then be re-validated. This not only reduces the load on the server, but also reduces a great amount of messages and data sent by the clients. From sending a big message per “Tick” ,containing the state of all the user inputs (per user), to sending a message on specific events containing only the action to be performed.

Another optimization would be to code your own socket communication library, and optimize the messages to send a smaller stream of bytes in comparison to the already developed serialized version used in most game engines. If you’re using Unity you’ll most likely be using RPCs for your message communication, and while it’s a great and stable solution, coding your own library will not only make things a little faster, but it will also grant you a vast amount of knowledge that you cant learn by just making games.

DO NOT – I would highly discourage the use of the state synchronization that comes with the Unity’s network communication solution. Although it has greatly improved a lot over the years, it still has a lot of overhead when sending data, unnecessary data sometimes. Two years ago I ran a Network stress test to compare Unity’s State synchronization against just using RPCs for communications. You’ll be amazed at the difference in performance.

The test was to have 30 people shoot as fast as they could, the bullets would last for 90 seconds, the bullets would move at different speeds and accelerate or decelerate (I was checking the Doppler effect on sound within Unity). Let’s be conservative and say that on average every player was doing 220 clicks per 90 seconds (2.4 per second), these were all gamers btw. So if we do the math, it comes up to almost 6,600 movable objects having their position, speed and acceleration updated to the server, a full Vector3 for each of these. I’m sure it was way more than that, I’m just being conservative. The RPC server had no issue whatsoever keeping up with this, while the state synchronization couldn’t even get 15 people to click over 20 seconds before the lag was noticeable and it started to choke. That should at least give you an idea on the performance differences.(BTW, the Doppler effect in Unity is pretty awesome, it worked surprisingly well)

What about the Client? As you might imagine, taking care of the client is not as confusing as it can be with the sever.  We still have to take care of pretty much the same lists, but with a lot less logic involved. Receiving the messages that update the objects in these lists is pretty much all we need to do to complete the communication process. The rest of the audio-visual stuff is in your hands.

Remember, the server can actually send the client a message when to do an action like cast a spell.  Hence, when you receive that specific message, you can start the character animation and spawn some particles here and there depending on positions and spawn the spell object that is being casted. I’m trusting you on knowing what to do in these cases.  Please ask if you need some help with this, I’ll be happy to help.

Although it’s great to have everything as organized and centralized in an Authoritative Server, this all comes with a cost. The responsiveness on this type of system is slower, and it can get extremely slow, depending on distance, quality of connection, wifi, and many other factors.

In games that depend on fast reaction time like an FPS, this can make them feel unresponsive. Enter Non-Authoritative Server, we will discuss this in the next article.

 

LAG in Games

There are different ways in which Lag can affect a game, from having a constant Lag that will make your game feel a little weird or slippery, to having massive Lag-Spikes that could shuffle everything in your game depending on how you do things.

There could be very different approaches on how to deal with Lag and Lag-Spikes depending on what kind of game you’re doing. Something that has a big impact on your game latency is what protocol you’re using, UDP or TCP.

 

 – UDP vs TCP

Most games use UDP since its generally a faster protocol that does not check the integrity or order of the incoming packets. If you’re using the built in technology on your game engine, it’s probably using a modified version of UDP, that makes it a little more reliable without sacrificing so much on performance and speed.

If you’re programming your own solution for this (Sockets), I would highly recommend to use UDP for most cases, it really makes a huge difference vs using TCP, especially if it’s an online game. When communicating over a network there is always some packet-loss, even more when on the internet vs a LAN. This packet-loss is what slows down TCP so much.

Can packet-loss affect my game when using UDP? Yes, but in reality, not that much. If you think about it, in most games you’re going to be sending several messages a second to keep the server/clients updated, if some of these don’t make it through, it will not affect your game so much since the newer messages that do make it through will update everything anyways.

When would I use TCP? If you’re making a turn-based game that does not require a “real-time” response. Many Mobile games actually use TCP as their protocol. Lets face it, mobile games are not there yet to actually have a fully functional  real time FPS experience compared to computers. There are not many mobile games that actually try to do this.

 

 – LAN Games

Although the technology now is allowing for easier, cheaper, and faster online games, there are still some people that would like to have their game tied strictly to a LAN.

Remember those glorious LAN-Party days? Well these types of games are the ones we’re talking about, from FPS games like in the old days Counter Strike, Quake and Unreal Tournament, to newer games that are leaning more on the indie side like Artemis Spaceship Bridge Simulator.

When making a strictly LAN game, we are tied to making a game that will act or be sold as a product, meaning that there are no monthly fees, no services tied to it, a self-contained product.

There is one big advantage to having a LAN game, and that is, an extremely low Lag. Unless we are connected through a VPN which simulates a LAN and still works, it can definitely introduce some significant Lag into the whole experience.

Why make a LAN game?… Well, if you don’t really have much experience with networked games, this is the place to start. It’s certainly easier and simpler to manage a few IPs inside your LAN than to manage several URLs, especially when you’re not used to it. Having to deal with either your cloud provider or hosting services just adds an extra layer of unnecessary complexity and cost.

If you’re just starting with multiplayer games, I highly suggest you start with some LAN projects, this will get you going and will allow you to learn the most important and useful things on multiplayer games for the least amount of work and virtually no extra cost.

Even though LAN games have very little Lag, there are still some system designs that could make your game feel and play better. These are the differences between a fully authoritative server vs non-authoritative servers (more on these later).

 

 – Online Games

Here is where we really need to worry about Lag and the consequences that it will have in our game. When a game is laggy, it can sometimes feel choppy, slippery, with slow response, or just weird in general. This weird feeling while playing your game could have a great impact on the whole experience for the users playing.

What can we do about Lag? There are several ways we can approach Lag. We will be discussing the most useful and common ways to approach this, including: client predictions, regional servers, network/server optimizations.

 

 – Client Predictions

This simply means that our Client (whatever the user is running on his computer)  application will try to predict all the movable object’s positions over time in case there is a Lag-Spike or a slight disconnect.

Lets review a simple example. Assume you’re running your online game and the server updates the position of another character to your Client. Based on the previous position and in relation to the new position, the client will know the direction in which the updated character is moving. This is usually known as interpolation. So if in case of a Lag-Spike the server fails to update the position on the next frame, the client is able to “predict” the position of the character for the next frame. This is assuming that the character will keep doing the same action.

So if we didn’t have any type of client prediction and had a Lag-Spike, we will have to wait for the server to reconnect. And whenever the server updates the position of everything in the game, things will just snap to their new position. Even when we don’t have considerable Lag-Spikes, just having a rather high Lag will make our game look and feel very choppy.

A possible negative effect of having a not so good client prediction is that the game will feel more slippery. It is important to know in your code when the client predictions kick in and, if possible, have a variable strength depending on the general latency and/or delay of the messages.

 

 – Regional Servers

One possible solution for handling high latency in online games is to have regional servers.

Lets say the server is physically located in California, and someone is playing your online game in Australia. There would be a massive amount of Lag in this case, considering the distance, many in-between connection nodes, unoptimized routing tables, etc. This is why regional servers are a great way of providing a better service to many regions around the world.

Keep in mind, you need enough users in each region to make this worth it for the users and yourself. Meaning, if there are not enough players connected to a specific region, it is not worth it to have that region functional since it incurs a greater cost to you and affords no player diversity for the users in these low-usage regions.

I would generally recommend to have fewer regions and to have them as centralized to your target markets as possible. For instance, if your target market is English speaking countries, you can have two servers, one centralized in the US that would cover both the US and Canada, and another centered in between the English speaking areas of Europe.

 

 – Network and Server Optimizations

These types of optimizations require a high knowledge of the whole network structure, how your system works, and in most cases they go hand in hand with custom network solutions, meaning that you coded your own socket based communication system.

There are a lot of great improvements to be done if this is the case.  We will go into detail about this in a later article.

Some of these could include:

  • Minimizing message usage – when you code your own communication solution, you can actually control the frequency and/or eventuality of the messages you’re sending. This can improve not only on Server/Client process performance but also better communication. The fewer packages that are sent, the fewer packages that are lost.
  • Minimizing the bytes sent – simply put: the smaller the byte usage in your network messages, the smaller and fewer packages to send and to get lost. There are several ways to do some Bit-wise optimizations that can greatly improve this aspect of your communications.
  • Choosing the specs for your servers once you have an MVP server running, you can stress test and load test your servers, and based on this choose the best configuration for the specs of your servers, whether you’re hosting or using cloud services.

 

 

Please  post some questions, comments, or suggestions. I want to be able to help you guys as much as I can by covering as much as possible. We can go into a very detailed and specific  discussion on whatever you need.

 

 

System Design for Multiplayer Games

Designing a multiplayer game might not seem like something that would require a lot of extra thinking compared to a single player game, and that might be true for some specific scenarios, but in most cases you start designing and suddenly you realize that things are getting a little out of hand.  When you’re finally grasping a general idea of how you want to design certain aspects of the game, you bump into one or two “What Ifs” from which you branch out into a complex spiderweb of design decisions. Before you know it you’re way over your head, confused, lost, and overwhelmed, then you think about all the hours of effort that you might need to put into your game just to get a Minimum Viable Product (MVP) running.

Well that’s what we’re here for, to talk about all these little and sometimes big things that affect our games and how we design them.

This is the first post of a series of articles that talk about the Design of a Multiplayer Game from a technological point of view. This article is aimed for Indie developers who have a desire to create a multiplayer game. I do not intend to tell you how to design your game but to guide you on how to make some design decisions and overcome some bumps in the road you may encounter while developing a multiplayer game. I’ll be highlighting some caveats, listing some common types of multiplayer games, and the work required to get them running. Hopefully this will ease your path into this challenging but exciting experience.

When talking about multiplayer games, we have a few options: Couch-coop, split screen, LAN, and Online games.  This series will regard only LAN and Online games, games that have a network component into them.

In this first article I’ll start by listing limitations, scenarios related to those limitations, and possible solutions for each. Each of the following articles will try to address the former in detail.

Lets get started.

 

 – LAG
I wasn’t sure if I wanted to start with this point or not, due to its complexity and all the things that need to be taken into account while designing a game that might be highly affected by LAG. And yes, some games are more prone to feeling “Weird” or just not working as expected due to LAG. This can be easily overlooked while designing a multiplayer game. Trust me when I tell you, LAG is REAL!

We will look into what are the differences between a Fully Authoritative server vs a more permissive approach. What if your game is Online or you just want to make a LAN game, “a la Counter Strike” when it first started, and possibly sell it as a product. What is Client prediction? What types of solutions are there? What are the Pros n Cons for each of these? What about Lag Spikes? Finally, we will talk about regional servers, and how this affects the game performance and your wallet.

 

 – Price/Cost
Regional servers could really hurt your operational costs. The fact is that all the points we’ve listed so far affect your OPEX somehow. You need to decide between selling your game as a product or as a service.  Those that sell their game as a service include any typical MOBA, Free2Play (F2P) or even an MMO like WoW. We will list and explain the necessary components to develop a game for each of these models. Moreover, we’ll talk about the scale you could start with. It is not only easier but also cheaper to grow something than to shrink down on your Cloud Structure.

What about your Content Distribution Network (CDN)? It is expensive to distribute your game, since there are storage and Bandwidth costs.
One key word is Cloud! Yes, most scenarios should use the cloud, because it’s not too expensive and will give you a lot of flexibility.

 

 – Server Types
What are the differences between having a dedicated socket server vs accessing a REST API? Where should I use each? Things to keep in mind while developing these.

 

 – The Networking Back-End
We’ll be talking about 2 engines in general, Unity and Unreal Engine. Both have a built-in Networking solution. Although these options are great for testing and making a quick draft, you can also create your own networking back-end solution which you’ll have total control over and can allow you to improve a lot on Networking performance, by controlling the size and occurrence of the messages and other cool tricks.

 

 – Game Modes
This is not related to what kind of networking solution you should use, this design decision is more related to how many people play the game and the sad but true possible scenario “What if there are not enough users playing my game?”.  What type of game mode should you start with? A free for all is always a good idea. What about queue times if you have a MOBA type of game? Should you do it match based or have big persistent maps?

These are some considerations to keep in mind while designing your game. We will discuss this in a top-down approach since these considerations come after the type of game you’re developing and not the other way around.

 

 – Multiple Services
There are some development choices that you might be forced to take because of some bottlenecks on some of the services developed. This will allow you to design ahead and have a clear view of what you need to do instead of having to re-factor your whole system in order to accommodate possible expansion scenarios. In short, if you do it right, you’ll do it once and it’ll be clear. We will go a little more into some topics like… What about friends, parties, chat, should I have my own VoIP? How will inter-server communication work?

In any case, we will try to approach most of the possible problems found in designing and developing a multiplayer game. I hope you find these articles useful.

 

After discussing all the parts that I want to point out that are necessary to understand and complete a multiplayer game, I will give some fully detailed examples on some game types such as: MOBA, FPS, RTS, and maybe even an MMO. I could also include some Mobile API based game. This will depend on whatever people requests I get.

 

On a side note: The reason why I say “WE” is because I include you into the discussion and expansion of these topics. It would be great if you can either comment or send me a message about the article regarding its content and  any concerns and specific question you may have. Your feedback will aid in enriching the articles posted here and allow us to further explore solutions for issues that may arise.