Whether you use real-time replication software for disaster recovery, server synchronization, remote work, VDI profile sync, or to distribute software updates, your organization needs a solution that instantaneously and efficiently syncs file changes across your entire replication environment.
The problem with the majority of real-time replication software is that, while they can begin to sync immediately, they can’t actually update every system in your environment simultaneously. Most replication solutions work on a point-to-point transfer architecture where files are transferred between two systems, at most, in one of two ways:
- One-way transfer: Files are transferred one-way between system A and system B.
- Hub-and-spoke transfer: When one system (A) needs to sync to other systems (C, D, E, and/or F), the file transfer must first go through a hub server (system B), which then transfers to all the other systems point-to-point.
The point-to-point model significantly reduces replication speed, as file changes must first be replicated to the hub server before they can be transferred to any other server (a process known as cloud-hopping). Then, when replicating file changes to other parts of your environment, replication must complete between two systems (e.g., from system B to system C) before it can begin on another system (from system B to system D).
For some organizations, this replication model may suit their needs. But for organizations that work with large files, large numbers of files (i.e., millions of files that need to be replicated), or need to synchronize, distribute, or ingest files across multiple endpoints simultaneously, a faster real-time replication solution will be required.
In this article, we’ll share 6 of the best real-time replication solutions you can use. We’ll also discuss our replication software, Resilio Connect, and how it uses a P2P (peer-to-peer) transfer architecture, WAN optimization, and more to provide ultra-reliable true real-time replication across your entire environment.
To learn more about how Resilio Connect can help you optimize file synchronization, schedule a demo.
Resilio Connect
Resilio Connect is a real-time replication software that uses a P2P sync architecture, proprietary WAN acceleration technology, and enterprise-grade security to:
- Achieve the highest file replication speeds in the industry (10+ Gbps).
- Easily scale to support big dataworkloads (i.e., transferring large files, many files, and replicating to many endpoints).
- Reliably transfer over any LAN or WAN.
Resilio is used by leading companies such as Turner Sports, Skywalker Sound, Mercedes Benz, Linblad Expeditions, and more.
Faster Replication with P2P File Transfer
Most other replication solutions transfer files point to point using a dedicated server model, which includes:
- A hub server: Hub servers can receive files from and transfer files to any device in the replication environment.
- Client servers: Client servers can send files to or receive files from the hub server, but they can’t communicate with each other.
Since client servers can’t send files to each other, any file transfer between two client servers must first go through the hub server. For example, if Client A wants to replicate file changes to other clients in the network, it must first send those changes to the hub server, which will then sync with other devices.
This is a process known as “cloud-hopping”, and it inherently slows down replication. Files can only be replicated one at a time. And hub servers can only replicate files to one client at a time.
But with Resilio’s P2P transfer architecture, every device in your replication network can share files with every other device. P2P transfer delivers:
(1) Parallel Transfer & N-way Synchronization
Resilio splits files into multiple file blocks, then transfers each block independently to multiple destinations. Every device in your network can transfer file blocks to any other device. And transfers can occur concurrently, resulting in 3 – 10x faster transfer than any client-server solution.
For example, imagine Device A wants to replicate files to Devices B, C, D, and E. Device A can transfer the first file block to Device B. Once it receives the file block, Device B can share it with any other device in the network while Device A sends the remaining file blocks.
Because of this, Resilio is one of the only real-time replication solutions that can perform true bidirectional sync, including:
- One-to-one transfer
- One-to-many transfer
- Many-to-one transfer
- N-way transfer
(2) Active-Active High Availability
With Resilio’s P2P architecture, every device (that’s running a Resilio agent) in your network can act as a server. If one device fails, Resilio agents can always access data and services from other devices. Resilio has no single points of failure, and each device can request data and services from the device closest to it.
Larian Studios, an award winning game developer, uses Resilio to sync nightly builds and software assets. According to Producer Octaaf Fieremans: “The nightly builds that we pump out involve a large amount of data. Getting that synced to every QA team in each studio proved to be quite difficult with the software that we had. With Resilio, as soon as our build servers have a build ready, it’s using the entire throughput of our gigabit lines to distribute it to the other studios. So QA can start in the morning on a fresh build.” Learn more about how Resilio helped Larian Studios achieve 5x faster transfer speed.
Optimized WAN Transfer
Resilio Connect uses a proprietary transport protocol — Zero Gravity Transport ™ (ZGT) — to optimize the speed, security, and reliability of data transfer over high-latency WANs.
When transferring over a WAN, traditional transfer protocols (such as TCP/IP) suffer from high packet transmission time and packet loss. But ZGT minimizes these issues and accelerates transfer across any network using:
(1) Bulk Transfer & Interval Acknowledgement
The sending device sends file packets periodically with a fixed packet delay in order to create a uniform transfer over time. And rather than sending back acknowledgements for every packet, it sends acknowledgements for groups of packets and includes additional information about lost packets.
(2) Congestion Control & Delayed Retransmission
ZGT ™ uses a congestion control algorithm to calculate the ideal send rate. It periodically probes the RTT (Round Trip Time) in order to monitor and maintain an ideal send rate. And it decreases unnecessary retransmissions by only retransmitting lost packets once per RTT.
Further reading: To learn more about how Resilio optimizes WAN transfer, read our WAN optimization whitepaper.
Resilio Connect’s WAN optimization also enables it to ingest from edge to core — i.e., acquiring, ingesting, and syncing files over unreliable networks to a centralized location. For example, someone filming a nature documentary in a remote location (where network quality is unpredictable and varies in quality) can quickly upload and sync their footage across your entire environment.
Northern Marine Group (a subsidiary of Stena AB) uses Resilio to synchronize data with their geographically distributed vessels (oil tankers, gas carriers, and a small cruise vessel)over spotty satellite connections and bandwidth constraints. According to Stena’s head of IT: “In the future, Resilio Connect and its API will vastly improve our database replication mechanism. More importantly, it will allow us to realistically increase the frequency of replication, so that the information passing back and forth comes far more in real-time and we’ll be able to automatically remediate the discrepancies.” Learn more about how Resilio helped them potentially save millions by reducing time to compliance.
Organic Scalability
Replication solutions that rely on a dedicated server model scale poorly. Adding more endpoints reduces replication speed because:
- Every change to a file must be transferred to the hub server before it can reach the other clients.
- The hub server must complete replication on one client before it can begin on another. The more clients in your network, the longer replication will take to complete.
In short, more servers put more strain on the hub server, and performance degrades across every device trying to sync.
To avoid this problem, many organizations invest in additional hub servers and design their replication environment to balance the load between the two — an endeavor that requires significant capital expenditure and man hours to build and maintain.
But Resilio’s P2P transfer architecture is not only organically scalable, but performance improves as more servers are added. Since every device can share data with each other, adding more devices inherently enhances Resilio’s performance, accelerating sync jobs across your environment.
For example, Resilio Connect will transfer 50% faster than any dedicated server solution in a 1:2 transfer scenario. And it will be 500% faster in a 1:10 transfer scenario.
Because supply increases with demand, Resilio is a perfect solution for applications that involve massive files, high quantities of files, and/or many users.
Optimized Checksum Calculations
A checksum is an identification marker that changes whenever a file gets changed. Resilio is designed to use optimized checksum calculations and real-time notification events from the host OS to perform incremental replication — i.e., detecting and replicating only changed portions of a file.
Many other replication solutions (like DFSR) replicate entire files. By replicating file changes only, Resilio increases transfer speed and reduces the load on your network.
Reliability & Fault Tolerance
Unlike dedicated server replication solutions, Resilio has no single point-of-failure. In a P2P network, if one device or network goes down, any device can still receive data or service from any other available device. And Resilio will retry transfers until they’re complete, and can dynamically route around outages.
File synchronization with Resilio Connect is also reliable and resilient. In the event of a network failure in the middle of a transfer, Resilio can perform a “checksum restart” and resume the transfer where it left off. And it uses cryptographic file validation to protect data integrity and ensure replicated files remain intact and uncorrupted.
Resilio is an ideal solution for fast disaster recovery. Its omnidirectional transfer architecture and ability to fully utilize any network connectivity in your environment enable it to meet sub-five-second RPOs (Recovery Point Objectives) and RTOs (Recovery Time Objectives) within minutes of an outage.
Resilio can be deployed on any infrastructure (edge, on-premise, cloud, or hybrid cloud sync). Its BYO storage model means you can install it on your existing infrastructure and be replicating in as little as 2 hours.
The IT manager at an employee benefit administration company said: “Our allocated time shifted from troubleshooting DFS replication to simply monitoring automated jobs with Resilio Connect. Replication is fully automated and highly available and DR capable. In a true disaster, if one site does go offline and is truly down, we are still up and running because the sites are in sync and data on the live site is still available to users.” Learn more about how Resilio helped them maintain continuous data availability.
High Visibility & Centralized Management
Resilio Connect’s dashboard, real-time notifications, and detailed logs provides insight into and control over how replication occurs in your environment. Adjust key parameters like:
- Managing bandwidth: Increase file sharing efficiency by setting bandwidth usage limits based on the time of day or the day of the week. And you can assign different schedules to different replication jobs and agent groups.
- Network & storage stack: Optimize performance by adjusting buffer size, packet size, disk io threads, data hashing, and more.
Using Resilio’s REST API, you can script any functionality, report on data transfers in real-time, manage agents, and more.
To learn more about how Resilio Connect can help you optimize file synchronization, schedule a demo.
PeerGFS
PeerGFS is a real-time data replication software that, like Resilio, supports hybrid cloud deployments. But unlike Resilio, it’s based on a point-to-point architecture. Enterprise organizations can use PeerGFS to replicate data and minimize downtime with Active-Active High Availability (though, as a dedicated server solution, PeerGFS is slower and less reliable than Resilio).
PeerGFS provides a management center to monitor the state of replication in your environment and configure sync parameters. It also provides secure transport within transit encryption and Malicious Event Detection.
Suresync
SureSync is a reliable and secure database replication provider that can transfer on any LAN or across any WAN. It uses remote differential compression to only copy the changed portions of files and optimize the replication process.
Using Suresync’s Enterprise Status panel, you can monitor all of your replication jobs and control which users have access to certain SureSync jobs.
Goodsync
GoodSync is a data replication and backup solution for Windows and Linux servers, NAS, and mobile devices. It increases transfer speed by running syncs in parallel threads and uses bit-by-bit synchronization for data protection.
Goodsync uses AES-256 to encrypt enterprise data at rest and in transit, and provides detailed logs of all file changes for analysis. Because they offer both a free and paid version of their software, Goodsync is a cost-effective tool that can compete with any other on pricing.
IBM Aspera Sync
Aspera Sync is a high performance data replication tool that is part of IBM’s business intelligence suite for data management. It easily integrates into media workflow ecosystems, making it a great solution for most media use cases.
Aspera optimizes WAN transfer by using UDP to fully tap into network bandwidth. It also uses log-based change data capture (cdc) to provide faster replication of data sources to destination data warehouses. And it’s unique in that it uses blockchain technology to transfer data securely and prevent data loss.
Syncthing
Syncthing is a free continuous file synchronization program used to store and sync data between multiple computers. It works on Mac OS X, Microsoft Windows, Linux, FreeBSD, Solaris, OpenBSD, and more.
Syncthing uses TLS encryption and cryptographic certification to protect your data pipeline. And its open protocol and source code give you complete control over where and how your data is transmitted.
True Real-Time Replication with Resilio
While all of the software on this list can be used to sync data across your replication environment, only Resilio provides true real-time, bidirectional synchronization. And Resilio can be deployed on your existing storage infrastructure, so you can have it up and running with no operational disruption in as little as 2 hours.
To learn more about how Resilio Connect can help you optimize file synchronization, schedule a demo.
