Tag: how to

Optimizing Multiplayer Games

These are some ways that can help us optimize our multiplayer games. Ways in which we can reduce the CPU and Bandwidth load of our hardware and/or cloud instances.

Some reasons you would want to improve the performance of your servers are: to greatly reduce costs, to have faster servers that will allow you to provide a much better experience, or to simply have a much deeper understanding of the technology you’re using, whether you launch your game or not.

A multiplayer game, while similar in some ways to a normal single-player game, is very different in its very core. A well designed multiplayer game will have a totally different system architecture than its single-player counterpart.

We need to be very aware that a single clock-cycle can mean the difference between a good experience and a lag-spike. A little dramatic? Not really, while technology is getting better and better by the second, we still can’t control how our packets travel through the internet. Since we can’t control the packet loss, all the different routing tables, the performance in each node, and many other things, this is why we have to care so much about the performance of our applications. The few things we can actually control, should be optimized for performance as much as possible.


Server and client are different things

If you read my previous article about Costs in Multiplayer Games, you already know this. You need to separate your server’s code from your client’s code.

This comes with many perks, you will have a more organized and dedicated code. Making it easier to maintain, fix, update, and distribute. Think about it, if you only update your client code, you do not have to also update your servers. Same way on the other side, if you update your server’s code, you will not have to update all the users with the new “Compound Code”.

On the same note, you can strip down a lot of unnecessary logic that is only used on “the other side (server or client)”. You can also specialize logic that was too general to be used on both sides.

This will help us by not having that extra logic to take care of, things that do not pertain to our direct application. Validating all this unnecessary logic, just to make sure if certain code should be allowed to run in the server or the client, could leech some precious clock-cycles from our applications.

 


Send Fewer Messages

While you want to keep your game state as updated as possible, it makes a very big difference to send continuous messages compared to sending messages just on events. i.e. Sending the full state of your character (Position, health, mana, the state of very single key, etc..) every frame will have a much higher impact on both your computer and network performance than if you were to send only the required information when it’s needed.

Having smaller specific messages that tell the story of how things are changing is much better than having the whole snapshot of your game for every frame per user.

Having events in your code that run specific messages will help you achieve this. For instance, if you press a key, validate on the client side the action and then send a message specific to this action to the server.  You can also do this on the server side, i.e. When validating if an ability hits someone, you can call an “Ability Hit Event” that will take care of doing damage, user notifications which can include both the aggressor and the victim, and so on. I’m sure you can get creative with these events.

 


Send Smaller Messages

This point goes hand in hand with the previous one. If you already thought about braking down those massive continuous messages into smaller eventual messages, you’re naturally sending less information per message. But there are some very useful ways to make things even smaller.

Why do we want to make our messages so small? Well as stated before, we cant control the internet. We can safely assume that the time it takes a message to be delivered is much bigger than the time it takes our computer to process the same message in many ways. By having fewer and smaller messages, we save ourselves a big percentage of our communications issues.

A great way to make our messages smaller is to see our attributes in terms of bits, yes 1’s and 0’s, I like to call this – Bitwise Optimization. For instance, a Boolean is equivalent to a single bit, we can group up several states in our game that are represented in bools (for example stun, blinded, silenced, …) and pack them into a single Byte. Then send a message with a single byte rather than sending a message with several bools. The reason these two ways are so different is because of serialization. If you’re using a built in communication library like UNET, this serializes the attributes and then sends this serialized byte array. This serialized information of multiple bools ends up being bigger than even a serialized byte.

You can do the same with small numbers that never go above a certain value. Lets take life for example, the life value will always be between [0 – 100]. This value will have a maximum of 7 bits and if you do the same with 3 other values, you now will have four 7bit values that you can easily encapsulate into a UInt32 and still have room for 4 bools. This UInt32 serialized will be smaller than the serialized array of 4 bytes and 4 bools containing exactly the same information.

 


No State Synchronization

State synchronization is getting better and better each day, so much that is getting difficult to distinguish the performance in small applications. Still not the cream of the crop when talking about performance in game communication.

Don’t get me wrong, state sync is great for prototypes, and it is certainly easier to learn how to use than to use a specialized or custom network communications library.

State synchronization has a lot of overhead in many ways, serializing and de-serializing messages, using reflection (that could be Unbuffered). The time it takes to process attributes into and from a message is just a little too big for my taste, not to mention the size in bytes of the serialized messages. Having bigger messages means more/bigger packets, and more/bigger packets lead to… Packet Loss.

Maybe I’m just biased because I like having everything as “Crisp” as possible. My “assembly mindset” gets in the way a little too much. Either a good or a bad thing in your eyes, it has allowed me to see things in different light, and made me want to improve many things  that someone would normally just let go. Yes it takes more effort, time, and the drive to want to improve such “little things”, but trust me, being this meticulous pays off big time.

You will have much more control over the content and frequency/eventuality of your messages. This goes hand in hand also with the two previous points of optimization – Send fewer and smaller messages.

 


Custom Network Library

While many optimizations can be done with a built-in network communications library, having a custom networking library will allow you, by far, the greatest amount of room to make improved optimizations in many ways.

You will be able to control how you encapsulate and deliver your messages. This will allow you to take full advantage of the previously mentioned optimizations and will allow these to have great synergy.  Moreover, it will help you deeply understand how the whole communication process works inside-out, and if done well, it will never make you want to look back.

I will not go into detail on how to code your own sockets communication library, there are several tutorials, videos, and resources out there that explain it very well. I will however encourage you to at least code your own, even if you’re never going to use it in your game. It will open your eyes and allow you to see things in a different light when developing multiplayer games.

 


General Optimizations

I had to include this whole section just to mention some of the more obvious ways to optimize code in general, and things that apply to general coding and not only to Multiplayer Networked Games. i.e. Object Pooling and Design Patterns (as long as you don’t abuse them). All those things that help make your code more efficient will also help make a better networked game.

There are some design patterns like Command Design Pattern that are useful in multiplayer games for specific reasons like recording a whole multiplayer session so you can then save it and replay it in an In-Game-Replay kinda way.

I hope you found this article useful and clear. I know I’m missing some optimization techniques, I was aiming to explain a few of the most useful but lesser known approaches. Please feel free to add questions or comments below.

Costs in Multiplayer Games

Want to know how to reduce the operational costs of your multiplayer game or just to start one by planning ahead for ultra low costs? In this article we’re going to talk about the major things that influence the cost of our whole operation in a multiplayer game. Things like: cloud services, regional servers, Content Delivery Networks (CDN), server performance, and a few others. We will approach many possible ways to reduce the costs of these, and hopefully help you come up with a good plan for your network architecture and a great system design.

Cloud Services

This one is a no-brainer when we talk about online games. While it is cheaper to  develop and test your game in a LAN, if you want to release your game to the masses, you will need an infrastructure to put all of the databases, service applications, and game servers.

Using the cloud for hosting our game is without a doubt the way to go.  It will provide a very reliable and redundant service that will guarantee that your game is up all the time.  Moreover, it will also be cheaper than buying your own hardware and hiring redundant power and internet sources to try to keep up with the same options.

Services like Amazon Web Services (AWS) and Google Cloud Platform (GCP) are great for general use cloud operations, including game. We also have some cloud services that specialize in games like AMPD Games services; which focus on delivering a lower latency in cloud services with specialized hardware for gaming operations while keeping the costs even lower than some of these big companies.

 

Regional Servers

Another big advantage of using cloud services is how easily you can have servers in many locations around the world, a.k.a Regional Servers. And while having more regional servers may help you provide a faster experience that will make your game feel very crisp, each of these servers incurs some costs. The more regional servers you have the pricier it will get.

When you’re first launching your game, it is a good idea to filter down to the fewest amount of Server regions to work with. This is for a few good reasons that are not related to costs.

Depending on how you have planned to update the game servers with the latest builds the time can vary from a few minutes to several hours. The fewer servers to update the easier and faster you’ll be able to fix, patch, and update your servers.

Another reason that is highly overlooked when deciding on how many regions to go for is… The users! Well, having more regions means a better experience for the users right? Not exactly, and here’s why. Yes, many regions = faster response time for the game all around, but what about how many users connect to a single region? This affects the game in a very different way that most people would just not expect.

Having several regions means that users will now divide into all the available regions, which means fewer players per region, which ultimately leads into users playing against the same users all the time or worse… Not having enough users in their region to play with/against.

Think about it, if you put a server in each country of Europe for example, you will most likely have players from that country connecting to their respective server. German players will play against players connected only to the German region server, and the same applies for every other regional server.

It is better to have one server in the center of Europe (i.e Frankfurt) which will host for all of the European players, and another one in the center of North America that will host for North American players, you could also have one in Brazil to serve the Latin American countries, so on. This slightly higher latency will not affect your game as much as not having enough players could.

It is very important to define which countries you think will be playing your game the most and work around these countries. Surveys, or any kind of poll are always a good way to assess your user-base.

 

ContenT Delivery Network (CDN)

This is what helps you get your game to the masses. When the users buy your game, they have to download it. If you don’t have a CDN, and host the file from a single source, not only  will the download speed will be very slow, but it will also be very expensive for you.

A simple solution and the recommended one is to go with Steam or any of these Game Distribution Services. It is just too much of a hassle to have to deal with distributing your game on top of how your game is doing. Yes, steam takes a cut, but trust me, it is well worth it since they also take care of hosting a page for your game and even some marketing. If you ask me, for all you get, this one is also a no-brainer.

 

Design for Scalability

This simply means, architect your whole network to be small at first and to be easy to increase in size, in other words to be very elastic.

Never assume the size of your user-base. One mistake I made while I was working on one of my previous projects was to assume that thousands of players will be playing the game from day 1. Big mistake! This did not hurt the game in any sense, but it really hurt our expenses. I planned to have several Databases per region, and separate every service into their own cloud server. Man this was a headache to “Fix”.

I had to migrate tables and data between databases, change some of the code that was relying on having several DBs, join services into a single instance of a server, and restructure how my system was going to dynamically grow. All of this in a separate testing environment to make sure everything was working properly, and then I had to roll out this environment to “Live”. This might not sound so complicated, but it was a handful to work with and very stressful since I was running against the clock, you know… Costs!

In the end, everything worked perfectly with 2 global DBs, 1 regional DB, 1 global instance, and 2 regional instances running the services and the actual game servers. Only 2 of these instances would grow dynamically. The cost reduction was massive, we ended up spending about 28% of what we were spending before.

Don’t make the same mistake.  I had to learn the hard way. Start small, design and architect everything to grow dynamically from a low cost operation. Most importantly Never assume things. This goes hand in hand with our philosophy: Make educated decisions.

 

Peer2Peer (P2P)

Branching out of the Server oriented path we encounter a system that can get rid of most of our server costs, this is called P2P.

A drawback of doing this is that we will be giving away the entirety of our game. This means that the users will have direct access to the Server and the Client. The reason for this is that instead of hosting game servers we will allow the users to host their own server, either deliberately or unknowingly.

This is a way how P2P gaming works, The client connects to a special server that holds records to all other clients and which ones are running a server, then the server redirects the users to one of these servers being run by a user. In this case the entire game loop will be managed by the users and just the connection will be managed by a server and DB.

We will be playing the game directly user to user, hence the name Peer2Peer. Now there are some major flaws around this method and you can imagine that now our game is completely vulnerable to being hacked and cheated. Another major big concern is the latency and performance of our game, since we cant control what kind of computer the user is using (a potato), the ISP that the users are using, if the user is throttling his connection, or simply using a lot of bandwidth in other applications.

This lack of control will hurt the performance of our game. It will be a more appropriate approach for games that do not require many resources to run. Clearly we need to be OK with the users having full access to our technology.

There might be some cases that this is a good approach for a very specific type of game, but in general it is not recommended to go with a P2P game.

 

Code Optimizations

Reducing the CPU and Bandwidth load on the servers will allow you to run more games per instance, hence reducing your overall hosting costs.

Some ways that can help you reduce the bandwidth load and the CPU load of your servers are the following:  Separate your server and client code, Do not use State Synchronization, Use a specialized communications library (RakNet), use your own Communication Library (Scokets), Bitwise Optimizations, Object Pooling, and so on.

In a future article I will go in depth into the details of some good optimization techniques for servers and networking in games.

Authoritative vs Non-Authoritative Servers in Games

Now that we have a better idea on how Authoritative and Non-Authoritative systems work, we can make a more educated decision about which one to use for different games. We’ll jump right into the Pros and Cons of each and compare their similarities and differences.

Authoritative Server
  • Pros:
    • Centralized control
    • Harder to get cheats/hacks
    • Easier to maintain
    • Easier to distribute updates
    • Dedicated server
  • Cons:
    • Slower response time
    • Dedicated server

 

Non-Authoritative Server
  • Pros
    • Faster response time
    • Server has a lower load
    • Dedicated server
  • Cons
    • Easier to cheat/hack
    • Harder to maintain
    • Harder to distribute updates
    • Dedicated server

 

My personal opinion is that a dedicated server is a very positive, helpful and better way to plan your system, although there may be some people that don’t require or even disagree with having a fully dedicated server for every instance of a game. That’s why its also listed as a con. There might be specific cases in which you want to mix the code of the server and the client and make a single application that holds both. But doing this will add unnecessary extra logic checks that could slow the code and that will definitely make it harder to maintain and distribute.

Distributing a centralized server update to your selected hosts is much easier and cheaper than distributing a client update to every single user. Generally speaking you want to have a dedicated server.

 

When to use each?

Well its not as simple as saying go for the easier one to maintain, or go for the faster one. A lot of factors will dictate which one you should use.

 

  • My approach

Based on the experience I have in working with these kind of systems, I have a very good general idea on how to design the system and architect all the services. I already know which type of system is better for what kind of game. While in the design phase of the game I like to start with a Fully-Authoritative Server, and depending on the needs of the game, I start delegating things little by little to the client. The easiest things you can start delegating are User Inputs and the validations of the “own user interaction”; you can go all the way to delegating almost everything to the client’s code. This will transform the server into a simple message router.

Don’t worry.  If you’re planing to make your first multiplayer game I have some general rules to follow on when to use each type of server. Like stated in previous articles, Authoritative Servers are superior for almost every case, except when your game relies on a fast response time like most FPS games.

An analogy would be to compare shooting a target with a bow and arrow versus shooting with a low caliber gun.

  • If the target is somewhat static, you can use an arrow, which allows you to be accurate and do a lot of damage.  However, if you use the gun you will still be accurate but will not do too much damage.
  • If the target is moving, you could still hit the target with the bow and arrow and still do more damage, but you will need to do extra math to compensate for the distance, arrow speed, and target movement speed too. Therefore all of these factors combined will be what we refer to as client prediction.  Now if you shoot the same target with a low caliber gun, you might do less damage but it will definitely be easier since a bullet travels much faster than an arrow.

In these cases Damage would refer to how secure and easy it is to update your code, the Projectile Speed will be the response time, the Target will be your game, and  whether it is moving or not will be the game type.

BTW, this whole analogy refers to how the game feels from a client’s point of view, it does not refer to the actual communication speed. The messages still have to travel back and forth at the same speed and in the same environment in both cases.

This then leaves the question, when should we use Non-Authoritative Servers (i.e. Gun)? First Person Shooters, Third person action games, and fast reaction-time games in general.

On the contrary, when should we use Authoritative Servers (i.e. Bow and Arrow)? MMOs, MOBAs, RPGs, ARPGs, RTS, so on… As long as the lag/latency/slow-response doesn’t affect the gameplay experience directly.

I mentioned before something about delegating most things to the Client’s code. This approach falls out of the scope of either of these 2 types of systems for a few reasons and optimizations that apply to this case. We will call this approach Peer2Peer (P2P) and we’ll talk about this in a later article which will be more related to lowering costs than to technical optimizations to the system.

 

Please feel free to share comments or questions, I might be missing a few important details. Looking forward to hearing from you.

System Design for Multiplayer Games

Designing a multiplayer game might not seem like something that would require a lot of extra thinking compared to a single player game, and that might be true for some specific scenarios, but in most cases you start designing and suddenly you realize that things are getting a little out of hand.  When you’re finally grasping a general idea of how you want to design certain aspects of the game, you bump into one or two “What Ifs” from which you branch out into a complex spiderweb of design decisions. Before you know it you’re way over your head, confused, lost, and overwhelmed, then you think about all the hours of effort that you might need to put into your game just to get a Minimum Viable Product (MVP) running.

Well that’s what we’re here for, to talk about all these little and sometimes big things that affect our games and how we design them.

This is the first post of a series of articles that talk about the Design of a Multiplayer Game from a technological point of view. This article is aimed for Indie developers who have a desire to create a multiplayer game. I do not intend to tell you how to design your game but to guide you on how to make some design decisions and overcome some bumps in the road you may encounter while developing a multiplayer game. I’ll be highlighting some caveats, listing some common types of multiplayer games, and the work required to get them running. Hopefully this will ease your path into this challenging but exciting experience.

When talking about multiplayer games, we have a few options: Couch-coop, split screen, LAN, and Online games.  This series will regard only LAN and Online games, games that have a network component into them.

In this first article I’ll start by listing limitations, scenarios related to those limitations, and possible solutions for each. Each of the following articles will try to address the former in detail.

Lets get started.

 

 – LAG
I wasn’t sure if I wanted to start with this point or not, due to its complexity and all the things that need to be taken into account while designing a game that might be highly affected by LAG. And yes, some games are more prone to feeling “Weird” or just not working as expected due to LAG. This can be easily overlooked while designing a multiplayer game. Trust me when I tell you, LAG is REAL!

We will look into what are the differences between a Fully Authoritative server vs a more permissive approach. What if your game is Online or you just want to make a LAN game, “a la Counter Strike” when it first started, and possibly sell it as a product. What is Client prediction? What types of solutions are there? What are the Pros n Cons for each of these? What about Lag Spikes? Finally, we will talk about regional servers, and how this affects the game performance and your wallet.

 

 – Price/Cost
Regional servers could really hurt your operational costs. The fact is that all the points we’ve listed so far affect your OPEX somehow. You need to decide between selling your game as a product or as a service.  Those that sell their game as a service include any typical MOBA, Free2Play (F2P) or even an MMO like WoW. We will list and explain the necessary components to develop a game for each of these models. Moreover, we’ll talk about the scale you could start with. It is not only easier but also cheaper to grow something than to shrink down on your Cloud Structure.

What about your Content Distribution Network (CDN)? It is expensive to distribute your game, since there are storage and Bandwidth costs.
One key word is Cloud! Yes, most scenarios should use the cloud, because it’s not too expensive and will give you a lot of flexibility.

 

 – Server Types
What are the differences between having a dedicated socket server vs accessing a REST API? Where should I use each? Things to keep in mind while developing these.

 

 – The Networking Back-End
We’ll be talking about 2 engines in general, Unity and Unreal Engine. Both have a built-in Networking solution. Although these options are great for testing and making a quick draft, you can also create your own networking back-end solution which you’ll have total control over and can allow you to improve a lot on Networking performance, by controlling the size and occurrence of the messages and other cool tricks.

 

 – Game Modes
This is not related to what kind of networking solution you should use, this design decision is more related to how many people play the game and the sad but true possible scenario “What if there are not enough users playing my game?”.  What type of game mode should you start with? A free for all is always a good idea. What about queue times if you have a MOBA type of game? Should you do it match based or have big persistent maps?

These are some considerations to keep in mind while designing your game. We will discuss this in a top-down approach since these considerations come after the type of game you’re developing and not the other way around.

 

 – Multiple Services
There are some development choices that you might be forced to take because of some bottlenecks on some of the services developed. This will allow you to design ahead and have a clear view of what you need to do instead of having to re-factor your whole system in order to accommodate possible expansion scenarios. In short, if you do it right, you’ll do it once and it’ll be clear. We will go a little more into some topics like… What about friends, parties, chat, should I have my own VoIP? How will inter-server communication work?

In any case, we will try to approach most of the possible problems found in designing and developing a multiplayer game. I hope you find these articles useful.

 

After discussing all the parts that I want to point out that are necessary to understand and complete a multiplayer game, I will give some fully detailed examples on some game types such as: MOBA, FPS, RTS, and maybe even an MMO. I could also include some Mobile API based game. This will depend on whatever people requests I get.

 

On a side note: The reason why I say “WE” is because I include you into the discussion and expansion of these topics. It would be great if you can either comment or send me a message about the article regarding its content and  any concerns and specific question you may have. Your feedback will aid in enriching the articles posted here and allow us to further explore solutions for issues that may arise.

Starting a blog

Ever wanted to start writing a blog, or even wondered how to start one? I’ll share my experience and resources I used to get this one going.

If you read my Hello world post, then you already know why I started with this blog, basically to be able to help anyone interested in creating something a little more complicated than the entry level demo kind of game that most people create when starting to use a game engine like Unity or UE4.

Since I don’t really have any experience in writing blogs or anything similar to it, my whole life I’ve only written code, I needed some guidance on how to properly start a blog, what were the best or at least better options for someone that is just starting out. While browsing and searching for a tutorial or some type of guide I bumped into the Youtube channel of Simple Programmer by John Sonmez. I watched a few very interesting videos related to development, how to improve productivity, great books for developers, and several other topics. He mentions his free course on how to start your own blog.

In this course, he breaks it down step by step and sends e-mails twice a week, including instructions and some good options for starting a blog, i.e. ready made blogs like wordpress.com vs hosting your own blog with blue host using wordpress. Just so you know, I ended up choosing the latter. He also sends a lot of useful tips on how to increase traffic to your blog, and things I never even thought about like why is it such a great idea to start writing your own blog to promote yourself.

If you want to check his free course, this is the link Create-Your-Blog, highly recommended if you want to start a blog of your own.

One thing I would always recommend to anyone, is to do your research before starting anything, whether it’s a blog, a game or anything in life. You might encounter something that sounds amazing from the perspective of whoever is explaining it, but their needs may differ a lot from your own needs. Try to always make an educated decision on whatever you’re doing, learn from your own mistakes, and even better from other people’s mistakes.

Please let me know what you think about starting a blog, if this was helpful to you and/or if I could use any other resources to improve things in this blog, from my writing to themes and how to manage a website, etc.