While many organizations use Rsync to sync files between Linux servers, Rsync (and other synchronization solutions) suffer from limitations, particularly for large synchronization deployments, such as:
- Most solutions can’t provide true real-time file synchronization, instead they rely on interval syncs.
- Most solutions experience replication errors when syncing large file directories, large files, or syncing to many endpoints.
- Most solutions aren’t optimized for transfer over WANs.
If your organization needs to sync large files, large numbers of files, or sync files across many Linux servers, then Resilio Connect is the best replication solution.
In this article, we’ll discuss:
Resilio Connect is an enterprise-ready solution that can be used for Linux distribution, synchronization, and consolidation. It utilizes P2P file transfer and WAN acceleration technology to deliver the fastest replication speeds in the industry (10+ Gbps per server) over any network. Leading organizations in tech (Match.com, Kaspersky), gaming (Larian Studios, Blizzard), media (Turner Sports, Skywalker Sound) and more use Resilio Connect to synchronize data.
To learn more about using Resilio Connect to sync files between your Linux servers, schedule a demo.
4 Reasons to Use Resilio Connect for Real-Time Linux Sync
Resilio Connect is a superior Linux file synchronization solution because it:
- Uses P2P (peer-to-peer) file transfer to provide fast, reliable replication.
- Can synchronize files in real-time in any direction.
- Uses WAN acceleration technology to optimize transfer over high-latency WANs.
- Is designed to be highly secure and easy to use.
1. Faster, More Reliable Sync with P2P Architecture
Other Linux file sync solutions, like Rsync or Unison, use a point-to-point file transfer architecture. This means that these solutions can only copy files from one server to another sequentially — i.e., only two servers can share files at a time.
This creates a host of problems that make these solutions problematic for organizations that need to sync large files, sync large numbers of files, or sync files across multiple endpoints.
When you can only sync point to point, large sync jobs and synchronization to many endpoints can take a long time. And if synchronization between any points (i.e., between any two servers) is interrupted or delayed (due to slow network connections), then every other server must wait to receive their files or file updates.
But Resilio Connect uses a P2P transfer architecture and file chunking to provide faster, more reliable file synchronization than any other solution.
With P2P transfer, every device in your environment can communicate with every other device. With file chunking, a file is split into several blocks that can transfer independently of each other. Using these two transfer tactics, Resilio can:
- Simultaneously transfer files across all devices.
- Eliminate any bottlenecks and single points of failure.
- Organically scale to support your synchronization needs no matter how big they become.
Simultaneously Transfer Files
Imagine you have 10 servers in your sync environment, and Server 1 needs to sync a file with the other nine servers. Server 1 can split that file into six chunks and begin sharing them with Server 2.
Once Server 2 has received the first chunk, it can immediately begin sharing it with Server 3 (or any other server), even before it gets the rest of the file chunks. And Server 3 can share any file chunks it receives as soon as it receives them.
Using file chunking and P2P transfer, Resilio can sync files across your Linux servers 3-10x faster than any other point to point sync solutions.
Eliminate Single Points of Failure
There are two types of point to point replication topologies:
- Client-server: One server is designated as a hub-server, and the other servers are client servers. The hub-server can share files with any client server. But the client servers can only share files with the hub-server, not each other.
- “Follow-the-sun”: Replication occurs from one server to another sequentially — i.e., Server 1 syncs with Server 2; then Server 2 syncs with Server 3; and so forth.
In a client-server topology, your entire replication environment is dependent on the hub-server. The hub-server must always be working and online, or replication can’t occur. This creates a huge point-of-failure, as servers can go down due to hardware failures, network failures, maintenance issues, etc.
And in both replication topologies, synchronization can only occur between two devices at a time. As previously stated, any sync delay between two servers can prevent every other server from receiving the files they need.
But with P2P replication, there are no points of failure. If one device goes down, the necessary files or services can be retrieved from any other device in your environment. Resilio can utilize the full bandwidth of your entire environment to balance the network load and maximize resource utilization. And it can dynamically route around outages (network failures, slow networks, etc.) to synchronize files as quickly as possible.
Organic Scalability
In a point to point environment, the larger your replication needs grow (i.e., syncing more files, syncing larger files, or syncing to many endpoints) the longer it will take to synchronize across your entire environment.
But Resilio Connect’s P2P replication topology makes it ideal for large sync jobs and enterprise deployments. Since every server is equally privileged and can sync simultaneously, adding more endpoints increases performance and resources. Each device can contribute resources (i.e., bandwidth, network, CPU, etc.) to sync jobs, and Resilio can balance the load across your environment.
Resilio has been tested and proven to replicate 250+ million files in a single job. And, in a 1:2 scenario, it can sync files 50% faster than point to point solutions. In a 1:10 scenario, Resilio can sync 500% faster.
2. Sync in Any Direction in Real-Time
Point-to-point transfer and sync solutions can only sync files between at most two computer systems at once. Most tools, like Rsync, can only do this “one-way”, i.e., they’re uni-directional and do not support two-way, bidirectional sync. Some tools like Unison can do this two-way but are also limited to syncing between 2 servers or in a hub-and-spoke.
Moreover, tools like Rsync do not provide true real-time synchronization. Instead, they sync in programmed intervals — i.e., every 1 minute, 5 minutes, 10 minutes, etc.
And to perform synchronization, these solutions must scan your entire file system. Performing system scans can take longer and longer as your file directory grows (i.e., more files, larger files).
But Resilio Connect uses optimized checksum calculations — i.e., identification markers for each file that change whenever a change is made to the file — and notification events from the host OS to detect file changes and perform true real-time replication.
Resilio can also sync files in any direction to support any replication scenario. It can sync one-to-one, two-way, one-to-many, many-to-one, and N-way. If you need to sync large file directories across many endpoints — such as when distributing software updates, syncing web and app servers, or syncing files with remote work teams — Resilio Connect is the perfect solution.
3. Optimized Sync over WANs
If you need to sync files over long-distance, high-latency WANs, then you need a sync solution that’s optimized for WAN transfer.
Most sync solutions use transfer protocols that aren’t designed for WAN transfer. But Resilio Connect uses a proprietary WAN transfer protocol known as Zero Gravity Transport™ (ZGT).
ZGT provides the fastest transfer speeds over any WAN connection by:
- Using a congestion control algorithm to calculate the ideal packet send rate and create a uniform packet distribution over time.
- Using interval acknowledgements for groups of packets, rather than sending acknowledgements for every packet receipt.
- Restransmitting lost packets once per RTT (Round Trip Time).
4. Secure & Easy to Use
While Rsync and other linux sync solutions use command-line interfaces, Resilio Connect was designed to be user-friendly.
Resilio can be set up and begin replicating in as little as 2 hours. As an agent-based software, Resilio Connect can be deployed:
- On your existing infrastructure.
- On-prem, in the cloud, or in hybrid-cloud environments.
- On popular operating systems — such as Windows, Linux, Android, Mac, Ubuntu, Unix — and can operate cross-platform.
- On industry-standard servers, desktops, mobile devices, and laptops,
- On VMware, Citrix, and other virtualization platforms.
And with Resilio’s Management Console and REST API, you can control every aspect of replication in your environment:
- Receive real-time notifications and detailed replication logs.
- Control user permissions.
- Adjust buffer size, bandwidth, disk I/O threads, and other replication parameters.
- Script and automate replication jobs.
Resilio Connect also employs state-of-the-art security features to protect your data at rest and in transit, such as:
- AES 256 end-to-end encryption.
- Mutual authentication to ensure your data only gets delivered to designated endpoints.
- Cryptographic data validation integrity to ensure your files remain uncorrupted.
How to Sync Linux Servers with Resilio Connect
Creating sync jobs with Resilio Connect is much simpler than other solutions and requires no coding knowledge.
Use the following tutorial to set up Resilio Connect and begin syncing on your Linux servers:
1. Install Resilio Agents on Your Linux Devices
First, you must install Resilio software agents on all of your Linux servers or devices. You can install Resilio Connect agents using:
- Apt-get managers.
- Yum package managers.
- Deb packages.
- Rpm packages.
For more information on how to install Resilio agents on Linux, use this step-by-step guide.
2. Install the Resilio Connect Management Console
You’ll need the Resilio Connect Management Console to configure and manage synchronization jobs.
To install the Management Console, run the “resilio-connect-console-windows.msi” installation package and follow the directions on the installation wizard.
The Management Console registers itself as a service and should run automatically after it is installed.
For more information on installing the management console, click here.
3. Enable Incoming Connections to the Resilio Connect Console on Your Firewall
If you’re using a 3rd-party firewall, enable incoming connections to ports used by Resilio Connect’s Management Console.
For more information on Resilio Connect port protocols, click here.
4. Apply Your License and Prepare Config File for Your Agents
a. Add Self-Signed Certificate
Navigate to your Resilio server Web UI at https://address:8443. The address might be localhost and the IP or DNS name of the server that you installed the Console on.
You will see a warning. Ignore the warning and add the self-signed certificate to the exception list:
b. Set Up Your Login Information
The Management Console will request that you specify the admin login and password, as well as agree to the end-user license agreement.
More on this step: Password policy
c. Apply the License File
The Management Console will then ask for a license file. Pick the license file that you were provided by Resilio Support (or your sales representative) or that was attached to the trail registration email.
More on this step: Applying license
Once the license is applied, Console lets you through to its interface. There you will see tabs with Jobs, Agents, Groups and other configuration parameters.
5. Organize Your Agents and Set Up Sync Jobs
- Click “Configure jobs”.
- Click “Create new job”.
- For Job Type, pick Synchronization.
- Give it a name and description.
- Select the agents that will be able to sync file changes with other agents, and which will have Read-Only privileges..
- On step “Path”, specify the paths to the share on agents.
To learn more about synchronization jobs, click here.
Use Resilio to Sync Linux Servers
Rather than struggling with a host of difficult Rsync commands or Linux commands, use Resilio Connect to synchronize files across your Linux servers.
Resilio provides:
- Fast (10+ Gbps per server), reliable synchronization
- WAN acceleration for optimal sync across high-latency networks
- Real-time synchronization in any direction
- Ease-of-use and end-to-end security encryption
To learn more about using Resilio Connect to sync files between your Linux servers, schedule a demo.
