Wargaming speeds up game build releases with Resilio Platform

Introduction

Have you played World of Tanks? Over 180 million players have played the team-based massive multiplayer online action game, or one of the other titles in the series, World of Warplanes or World of Warships. This places Wargaming as one of the leaders in the free-to-play MMO market. Maintaining a leadership position in the hyper-competitive gaming market isn’t easy, and Wargaming has to keep players engaged through fast-paced game update cycles.

Challenge

It takes a large army to create such complex games, and Wargaming has over 4500 employees, located in 20+ offices and development centers across the world, working on titles for mobile, PC, MAC and various gaming consoles. Each title has its own producers, designers, developers and publishers, some of whom are third party studios who need to plug in - securely - to the Wargaming infrastructure.
‍
A key step before publishing a game update to fans, is the game build, a combination of compiled code and media assets, like textures, audio and video content. Multiple teams from many offices contribute to each game build, and the build itself needs to be transferred between the different teams as fast as possible, but also in a reliable and manageable manner.  
‍
It may not sound like a big challenge until you dive into all of Wargaming’s requirements. Each build is large, with 50GB and up of code, audio and video. Those big files have to be transferred over the available network channels, which can be great, or a not so great 10Mbps with high latency and packet loss. Regardless of the network channel limitations, the build has to go through fast.
‍
Wargaming were certain they needed to solve this with a peer-to-peer solution that could overcome network channel limitations by using multiple nodes simultaneously. At the same time, they required a very robust set of centralized management features from this business-critical network component. They must be able to track progress and manually stop and restart transfers from a web interface, and also control bandwidth consumption. They work with 3rd parties and must be able to secure access with granular controls. Cross platform support and simple integration with the rest of their stack and into their workflow were also important.

Solution

The engineering team evaluated 14 different synchronization solutions, taking a deep dive into 4 solutions. When they took Resilio for a spin they were pleasantly surprised. “Our engineers deployed a successful test version in a hour - without reading a ton of manuals - that delivered the best result of all the solutions we tried,” says Roman Sakno, Enterprise Architect at Wargaming.
‍
All in all, Resilio was the clear winner because of its combination of very high performance and across the board simplicity. The pure peer-to-peer architecture yielded significant speed improvements. “Resilio met all our technical requirements, but was also very easy to deploy and integrate into our workflow. We also liked the fact that the simplicity carried over to transparent pricing and clear documentation” explained Sanko.
‍
Wargaming has successfully implemented Resilio Platform in 10 geographical locations. On the one hand it is used to synchronize BitBucket builds using the REST API. It is also tightly integrated with TeamCity and Jenkins, and is used to synchronize builds to DevOps teams located on different continents.

Benefits

• Robust performance over limited bandwidth network channels with latency and packet loss by incorporating WAN optimization
  

• Speed up transfers of very large files with multiple types of content - binary, audio, video
  

• Fine-grained user access control, including management of 3rd parties
  

• Web-based tracking of synchronization progress with manual interrupt & resume and exception handling (e.g. lack of disk space)
  

• Cross-platform support (Linux, Windows, Ubuntu servers and CentOS), as well as support for all package formats
  

• Simple integration with CI/CD tools Jenkins and TeamCity
  

• Flexible API for integration with existing workflow
  

Technical Description

The main collaborative development tool is BitBucket, and it’s integrated with Resilio using commit hook and its REST API. When someone changes the file in Git LFS hosted on BitBucket, it notifies Resilio Platform to synchronize file from Git LFS in the current location with another location.

Primarily, the integration is used to transfer audio/video/textures from our designers. On other side, Resilio is tightly integrated with TeamCity and Jenkins. When a build is completed then CI just calls Resilio via API to start transfer and send file to the DevOps team, which is located on the different continent.