Cloud Bursting: What It Is, How It Works, Benefits, and Solution

The widespread popularity of cloud service providers — like Amazon Web Services (AWS), Google Cloud (GCP), Microsoft Azure, and a plethora of smaller ones — has given companies the ability to utilize compute resources in flexible and cost-efficient ways. 

Instead of relying solely on on-premises data centers, teams can use reliable and secure public cloud infrastructure to meet workflow demands. One of the most popular ways to do that is cloud bursting — a method of leveraging cloud resources whenever on-premises infrastructure is at peak capacity.

In this guide, you’ll learn everything you need to know about cloud bursting, including:

• What it is.
• How it works.
• Its 3 biggest benefits.

In the second part, we’ll also explore how the Resilio Active Everywhere Platform — our ultra-low latency, high-performance synchronization system — can let you burst to any location in the cloud or on-premises across any network quickly, reliably, and efficiently. 

Resilio overcomes latency across sites to burst to anywhere at any time — on a schedule, in real-time, or on-demand. It also delivers 2–10x the performance of NFS-based caching solutions, as well as transfer and sync at speeds that are 3–10x faster than traditional hub and spoke solutions.

In a nutshell, Resilio provides:

• Caching flexibility — You can use Resilio to cache files across any location or storage system, whether in the cloud or on-premises. You also have granular control over caching behavior, including selective caching, pre-caching, cache pinning, hydration, and more.

• Cloud and storage flexibility — You can use Resilio with any cloud provider (GCP, AWS, Azure, Backblaze, Wasabi, Backblaze, etc.), device (desktops, laptops, servers, IoT devices, etc), storage type (file, block, object, NAS, DAS, SAN, etc.), and operating system (Linux, Windows, macOS, Unix, etc).

• Centralized management and automation — Resilio’s Central Management Console makes it easy to set up and manage jobs across your entire environment, even in multi or hybrid cloud environments. Plus, any UI action — including caching, hydration, and synchronization — can be automated using our API. 

• Unmatched speed and reliability — Resilio uses a unique peer-to-peer (P2P) architecture that removes single points of failure (SPOFs) and guarantees maximum transfer and sync speeds. Our engineers have tested and achieved speeds of 100+ Gbps per cluster.

• UDP-based WAN Optimization — Resilio’s proprietary WAN optimization protocol overcomes latency and packet loss to ensure predictable transfer and sync times, irrespective of network quality. It also lets you fully utilize any network, including VSATs, cell, Wi-Fi, and any IP connection.

• Full-mesh synchronization — Resilio enables bidirectional and N-way sync within and across sites, whether in the cloud or on-premises. You can also use it to sync fully or partially, as well as make updates in real-time or on demand.

• Organic scalability — Resilio lets you scale your environment by adding nodes (i.e., agents), which scale out your storage performance without disruption. Thanks to the P2P architecture, the more endpoints you add, the better Resilio’s performance gets. 

Organizations use Resilio Connect to provide fast, reliable file sync and access for many use cases, such as cloud bursting, VDI profile sync, disaster recovery, multi-site collaboration and remote work, edge synchronization, server synchronization, and more. To see how Resilio can benefit your business, schedule a demo with our team.

What is Cloud Bursting?

Cloud bursting is a method of using public cloud resources whenever your on-premises data center or private cloud infrastructure is at peak capacity. By bursting to the cloud, organizations can ensure business continuity during peak demand periods, harness excess capacity at the right times, and manage their costs.

For example, cloud bursting is often used in rendering, where the demand for computing power can fluctuate by days or hours. Teams can burst to the cloud during the process in order to utilize all their compute resources, regardless of where they’re located. They can also achieve cost savings using cheaper ephemeral cloud computing resources and scaling their usage down when those additional resources are no longer required.

Cloud bursting is also employed in other scenarios, including high-performance computing (HPC), big data analytics, marketing campaign launches, and more.

How Cloud Bursting Works (Implementation Methods)

Cloud bursting can be implemented in a variety of ways, with each cloud provider offering a different mix of services for the job, including gateways, caching solutions, observability tools, automation platforms, and so on. 

We can broadly organize these implementation methods into the following categories:

• Manual, where teams provision and deprovision cloud services by hand. This can be difficult to scale as you can’t always count on a team member provisioning a service at exactly the right time. That’s why manual bursting is usually employed in very specific scenarios, such as testing and temporary deployments.

• Automated, which relies on tools (like Resilio) to automatically provision cloud resources on your behalf. You set the parameters beforehand — by creating policies and setting capacity limits — and the tools burst to the cloud when those parameters are met. This method is much easier to scale and maintain, as it drastically reduces the probability of human error.

• Distributed load balancing, where workloads are operated simultaneously between the public cloud and your on-prem infrastructure. To implement it, you need a standby deployment to run in the public cloud, so that it can scale up as needed. You also need to set load thresholds in your on-prem infrastructure and distribute them as necessary.

The 3 Biggest Benefits of Cloud Bursting

As we said, cloud bursting can be used in many different ways, scenarios, and industries. In most cases, companies that implement cloud bursting do so to:

• Get flexible access to computing resources. Cloud bursting enables you to use the necessary computing resources exactly at the time you need them. This means you can harness today’s massive cloud computing capabilities in a strategic, cost-effective way that guarantees business continuity.

• Reduce costs and work more efficiently. When done right, cloud bursting utilizes low-cost resources only when they’re necessary. As a result, you can control costs and ensure that no resources are provisioned and unused. In some cases, cloud bursting can remove the need to invest in additional on-premises infrastructure that’s expensive and difficult to set up. 

• Avoid service disruptions. Whenever there’s high demand for your services, cloud bursting directs the overflow traffic that can’t be handled by your on-prem infrastructure toward a public cloud. This ensures business-critical applications continue to work without interruption during crucial periods.

How Resilio Can Help You Burst to the Cloud Quickly, Efficiently, and Reliably

Resilio is a high-performance, scalable, and efficient file caching, replication, and synchronization system. It’s ideal for many burst-to-cloud use cases — including speeding up rendering and HPC — as it lets you overcome latency across sites to burst to locations in the cloud or on-prem.

You can use Resilio to harness excess compute capacity when needed and control costs. Additionally, our solution lets you cache, transfer, and sync data across geographically distributed locations and gives you centralized control over the process. 

This is possible thanks to Resilio’s:

• Management and automation features, which let you burst to the cloud in real-time, on a schedule, or on demand, depending on your workflow needs.

• Policy-driven cache management, which gives you granular control over pre-caching, pinning, download priorities, and dehydration.

• Versatility, which enables you to use Resilio with any on-prem infrastructure (servers, laptops, mobile devices, NAS, DAS, SAN, etc.) or cloud provider (AWS, GCP, Azure, Backblaze, Wasabi, etc.).

• Efficiency capabilities, including selective caching, sync, and downloads. Plus, our solution only transfers the changed portion of files, so no unnecessary data gets moved across the wire.

• UDP-based WAN optimization protocol, which ensures you can utilize any connection — including VSAT, Wi-Fi, cell, and any IP connection — while overcoming the impact of latency and packet loss.

• P2P architecture, which eliminates SPOFs, guarantees maximum transfer and sync speeds, and makes your environment organically scalable.

• Native security features, which protect your data and rest and in transit.

Efficient Burst-to-Cloud, File Synchronization, and File Access Across Any Location

With Resilio, you can burst to the cloud (or other on-prem locations) whenever you need to — on a schedule, using file caching policies, in real-time, or on-demand. 

You can do this manually via the Central Management Console — a user-friendly GUI accessible from any browser — through an API, or through a command-line interface and scripting. Regardless of which option you choose, you can use Resilio to:

• Create, manage, and monitor replication jobs.

• Schedule or prefetch files to completely automate the burst-to-cloud process. 

• Configure automated policies that govern how data is cached, synced, and accessed across your entire on-premises and cloud environment from one place. For example, you can set a policy to purge files from a cache after a certain number of days, as well as control pre-caching, pinning, and dehydration.

• Adjust and manage bandwidth at each endpoint and create profiles that govern how much bandwidth is allotted to each endpoint at certain times of the day and on certain days of the week.

Using Resilio’s policy-based automation, you also have access to plenty of features that improve efficiency and productivity, including:

• Selective caching: You can choose which files and folders are cached locally, and create policies that automate cache management and data movement. 

• Full or partial downloads: You can download full files and folders or just the portions of files/folders that you need. This is a great way to get instant access to the necessary files, while also minimizing data egress charges.

• Selective synchronization: You can choose which files are synced to which specific endpoints. This capability is ideal for remote work scenarios that require granular control over how and where files are synced to ensure end-users have the data they need, with minimal egress costs.

Finally, Resilio provides a single, unified file interface that operates much like popular file management tools, such as MacFinder, Microsoft OneDrive, and Windows File Explorer. Thanks to this storage gateway, end-users can always get low-latency access to files, regardless of where the data is stored and where they’re located.

Flexibility to Use Any System, Network, and Storage Type

Unlike many proprietary solutions, Resilio is an agent-based, vendor-agnostic system that’s built on open standards. This means you can quickly set it up on your existing IT infrastructure (in the cloud or on-premises), without needing to buy or manage new hardware.

You can use Resilio with just about any:

• Device, such as desktops, laptops, file servers, NAS/DAS/SAN devices, mobile devices (Resilio offers iOS and Android apps), and IoT devices. 

• Public cloud provider, such as AWS, Azure, GCP, Backblaze, Wasabi, MinIO, and more. 

• Operating system, such as Linux, Microsoft Windows, macOS, Unix, FreeBSD, OpenBSD, Ubuntu, and more.

• Virtual machine, such as VMware, Citrix, and Microsoft Hyper-V.

Thanks to this versatility, you’re never locked into a vendor or system. You can freely cache, move, and sync data across cloud providers, data centers, and storage types, as well as burst to any location in the cloud or on-prem.

Reliable, Low Latency Data Transfer and Sync Across Any Network

P2P Architecture for Speed, Reliability, and Scalability

Resilio Connect is one of the very few file synchronization solutions that can sync in real time and in any direction. 

This is possible thanks to its P2P replication architecture that lets every endpoint in your environment share files directly with every other endpoint. Plus, Resilio also uses file chunking to break down files into small pieces that can be transferred independently of each other.

For example, say you wanted to sync a file across five endpoints. Resilio could split that file into five chunks and each endpoint could work together to share them across your environment. Endpoint 1 can share the first chunk with Endpoint 2. Endpoint 2 could immediately share that chunk with any other endpoint, even before it receives the remaining four file chunks. 

Put simply, Resilio turns your infrastructure into a distributed mesh network, effectively giving every endpoint the power of a data center. 

It leverages every endpoint in your environment in order to distribute network and CPU load across your system, reduce the load on your servers and internet channels, and improve data availability.

This unique combination of capabilities delivers some crucial benefits:

• Incredibly fast transfer and sync speeds. In a P2P environment, all endpoints can share data between each other. Combined with file chunking, this results in transfer and sync speeds that are 3–10x faster than traditional hub and spoke solutions.

• N-way synchronization. Resilio can sync data in any direction — one-way, two-way, one-to-many, many-to-one, or N-way. N-way sync enables you to keep hundreds or even thousands of endpoints synchronized simultaneously. This makes Resilio invaluable for use cases where data needs to be synced as fast as possible, like server synchronization, remote work collaboration, hot-site disaster recovery, and more.

• Organic scalability. The P2P architecture makes Resilio an organically scalable solution. This means that it performs better as you add more endpoints — it can sync 200 endpoints in roughly the same time it takes most hub-and-spoke solutions to sync just two. You can also sync objects of any size, type, and number.

• High reliability. There’s no SPOF in a P2P environment. If any server or network goes down, Resilio can dynamically route around the outage and find the optimal path to deliver files to their destination. If a transfer is interrupted, Resilio can perform a checksum restart to resume the transfer where it left off and will retry all transfers until they’re complete.

Proprietary WAN Optimization Protocol for Maximum Utilization of Any Network

In some burst-to-cloud scenarios, your data may have to travel over large distances and across low-quality, unreliable networks. These networks often suffer from high latency and varying degrees of packet loss.

Resilio overcomes these challenges with its proprietary WAN acceleration protocol known as Zero Gravity Transport™ (ZGT). ZGT eliminates latency and allows you to fully utilize any network by optimizing traffic over any network, including VSAT, Wi-Fi, cell (3G, 4G, 5G), and any IP connection.

It accomplishes this using:

• A congestion control algorithm: ZGT uses a congestion control algorithm that constantly probes the RTT (Round Trip Time) to calculate and maintain the ideal data packet send rate. This enables it to maintain a uniform packet distribution over time.

• Interval acknowledgments: Rather than acknowledging each packet receipt, ZGT sends acknowledgments for groups of packets.

• Delayed retransmission: Rather than immediately retransmitting lost packets, ZGT retransmits lost packets in groups once per RTT to reduce unnecessary retransmissions.

The combination of a P2P architecture, file chunking, and WAN acceleration makes Resilio a high-performance, ultra-low latency solution. It can transfer data in predictable time frames even across low-quality networks at the far edge.

Built-in End-to-End Security

Resilio comes with native, enterprise-grade security capabilities that keep your data protected at all times, so you don’t need to invest in separate security software. These features are reviewed by 3rd-party security experts (TPN Blu certified and SOC2 certified) and include:

• End-to-end encryption: Data is protected at rest and in transit via AES 256-bit encryption.

• Cryptographic data integrity validation: Resilio uses data integrity validation to ensure data arrives at its destination intact and uncorrupted. Damaged files are immediately discarded and scheduled for retransmission.

• Immutable copies: Resilio stores immutable copies of data in the cloud to protect against ransomware.

• Mutual authentication: Every endpoint must provide an authentication key before it can receive files, ensuring your data is only delivered to approved endpoints.

• Granular permission controls: You can limit access to certain data to privileged employees by controlling who is allowed to access specific files and folders.

• Forward secrecy: Resilio protects each session with one-time session encryption keys.

• Proxy server: Resilio’s Proxy Server capabilities support advanced firewall configurations and enable you to control user and script access on specific machines.

Use Resilio Fast, Efficient, and Reliable Burst-to-Cloud

Resilio is the perfect solution for teams looking for a high-performance, ultra-low-latency solution that can help them burst to any location for use cases like rendering, HPC, and more.

It gives you the freedom to cache, access, and sync data across all locations while overcoming latency and working with your choice of devices, storage, and cloud providers.

In short, Resilio offers:

• Flexibility: You can deploy Resilio agents on any on-premises storage device, cloud storage platform, and operating system. This means you’re free to cache, sync, and move data across data centers, cloud providers, hybrid cloud setups, and much more.

• Centralized management and automation: You can easily bust to the cloud or other locations across your infrastructure in real-time, on a schedule, or on demand. You can also use Resilio’s Central Management Console and REST API to script any type of functionality your workflows require, regardless of whether it’s related to data caching, synchronization, or transfer.

• Speed and reliability across any network: Our solution uses a P2P replication architecture that eliminates single points of failure and provides fast synchronization. It also leverages a proprietary WAN acceleration protocol to optimize transfers over any network regardless of quality, latency, or packet loss.

• Efficiency: You can install Resilio on your existing infrastructure, so there’s no need to invest in new hardware or security solutions. You can also increase productivity and minimize cloud costs with features like selective sync, local caching, and partial downloads.

• Built-in security: Resilio includes native security features that protect your data at rest and in transit. You can be sure that your data is always secure, without having to invest in 3rd-party security tools and VPNs.

Organizations use Resilio Connect to cache, sync, and access data for media workflows (Turner Sports, Innovative), gaming (Wargaming, Larian Studios), remote operations (Mercedes-Benz, Buckeye Power Sales), and more. If you want to learn how Resilio Connect can also help your business, schedule a demo with our team.