Most traditional file replication solutions aren’t designed to handle the workloads required of modern app and web content replication deployments. Not only do these solutions experience frequent breakdowns and replication delays that can harm the user experience, they aren’t designed to optimally handle large deployment schemes — e.g., synchronizing content (web content, software updates for apps, etc.) across many web servers as fast as possible.
In this article, we’ll discuss what to look for in web content replication software as well as five excellent solutions for large replication deployments — starting with our own solution for web and app server synchronization, Resilio Connect.
Resilio Connect is an organically scalable web content replication solution that can quickly detect and replicate files and file changes in real-time. It uses a P2P transfer architecture and WAN optimization technology to deliver some of the fastest replication speeds in the industry (10+Gbps). Resilio Connect has been proven to handle synchronization jobs of 250+ million files, and can update hundreds of servers in about the same time that it takes to update one server.
To learn more about how Resilio Connect can provide fast, reliable web content replication for your app or website, schedule a demo.
What to Look For in Web Content Replication Software
In order to maintain data availability and keep your site users happy, you need a web content replication solution that meets the following criteria:
High replication speed
Your solution must replicate files quickly so users across your entire replication environment have access to the most current version of files.
For example, if a user changes their profile picture on a dating site, all other users accessing the website on any server across the world should immediately have it. The replication speed of your solution will be largely determined by how it replicates files (i.e., replication topology, replication methodology, and more) and other methods and technology that speed replication.
Good scalability
Your solution should easily scale to handle your site replication needs, whether that means replicating large files, a large number of files (i.e., many millions of files), or replicating to a large number of endpoints.
Many organizations also require a solution that can replicate files in different directions — i.e., one-to-one, one-to-many, many-to-one, many-to-many, or N-way synchronization. Regardless of the replication direction, your solution should be able to handle your sync needs in a fixed, suitable time-frame.
Fast replication over WANs
For deployments with web servers spread out across geographic locations and/or using high-latency, unreliable networks, fast replication over WAN networks is a must. Many replication solutions aren’t optimized for WAN transfer, resulting in replication errors (i.e., lost packets) and delays.
Your solution should include WAN optimization technology and be able to quickly replicate content over any network to reliably deliver files to their destination.
Resilience & reliability
Replication environments can be plagued by unreliable networks, hardware failures, software failures, and more. Overcoming these challenges with traditional replication solutions requires spending a lot of time and money developing complex disaster recovery systems (backup systems and servers, failover strategies, etc.). Choosing a reliable web content replication solution can save you time and money and ensure the uptime and availability of your web content.
The 5 Best Web Content Replication Solutions
Resilio Connect
Resilio Connect is an enterprise-proven, real-time file replication and synchronization solution that uses a P2P (peer-to-peer) transfer architecture and proprietary WAN optimization technology to quickly synchronize, distribute, and consolidate files over any network. Because it uses a mesh, P2P network to replicate files, Resilio can organically scale to support the requirements of any large deployment — i.e., replicate files of any size, replicate large numbers of files (tested at 250+ million per job), and replicate to tens of thousands of endpoints.
Resilio Connect can perform file synchronization, consolidation, and distribution. It’s used by leading websites, such as Kaspersky, Match.com, and MixHits Radio.
P2P architecture for fast, reliable synchronization
Resilio’s P2P transfer architecture is one of the most notable features that set it apart from competitors with regard to replication speed and reliability.
Most other replication solutions use one of two replication topologies:
- Client-server replication: In a client-server replication topology, one device is designated as a hub server and the other devices are designated as clients. The hub server can replicate files to any device and receive files from any device. The clients can only replicate files to the hub server, but can’t communicate with other clients. In other words, all replication must first go through the hub server. So if Client 1 wants to replicate file changes across your entire environment, it must first send those file changes to the hub server, which will then replicate the changes to all of the other clients one by one.
- “Follow-the-sun” replication: In a “follow-the-sun” replication topology, replication occurs from one device to the next sequentially. So if Device 1 wants to share file changes with the other devices in your environment, it will first share those changes with Device 2. Device 2 will then share the changes with Device 3, and so forth.
Both of these topologies create bottlenecks in your replication workflow, as replication can only occur sequentially between two devices at a time — either from the hub server to one of the clients devices (in the client-server model), or between two devices (in the “follow-the-sun” model).
With P2P replication, however, every device in your environment is equally privileged. So if Device 1 wants to share files with Device 3, it can do so without first going through a Hub-Server.
Resilio Connect also uses a process known as file chunking. When sharing a file, Resilio will split that file into several chunks that can each transfer independently from the others. This enables all devices to share files simultaneously, resulting in transfer speeds 3-10x faster than traditional replication solutions.
For example, let’s assume you need to replicate a file across seven devices. Resilio will split that file into five chunks. Device 1 can share the first file chunk with Device 2. Once Device 2 has received that first file chunk, it can begin sharing it with Device 3 before it has received the entire file. Every device in your environment can share file chunks with any other device, so all of your devices can work together to speed up file transfer.
P2P file transfer also means that there is no single point of failure in your entire environment. In a client-server topology, replication is entirely dependent upon the hub server, which must be online and functioning at all times in order for replication to work. This requires organizations to invest in complex and costly failsafe setups (i.e., backup servers, failover operations, etc.) to ensure their system never goes down and web content distribution can always occur. Even then, replication is inhibited by:
- The bottleneck created by the need to cloud-hop — i.e., sending a file to the hub server before sending it to the target device.
- The slowest link between the source device and the destination device, as well as any latency that occurs across the end-to-end transmission.
- File size limits.
In a “follow-the-sun” topology, replication can be impeded by any single device in your environment. If, for example, one device is on a slow or unreliable network, all of the other devices must wait for replication to complete on that device before they can receive any files.
These problems are magnified when replicating large amounts of data across many endpoints. With sequential synchronization, large syncs take a long time and data may not be available across all servers when requested (i.e., app users or website visitors will see broken links and missing images on web pages where the server hasn’t yet received the updated files).
But in a P2P architecture, any device can receive the necessary service or files from any other device in your system. If one device is on a slow network (or if the network goes down), the services and/or files can be supplied by a device on a faster network. Your replication environment will never fail because it’s not dependent on any single device.
WAN acceleration for optimized WAN transfer
Most web content replication solutions use transfer protocols that aren’t optimized for transfer over WANs.
WANs are characterized by high-latency and varying degrees of packet loss. Many replication solutions use TCP/IP transfer protocol for transfer over both LANs and WANs. While TCP/IP works well over LANs, it responds poorly to WAN transfer.
For example, when packet loss occurs, TCP/IP treats it as a network congestion issue and will reduce the bandwidth for transfer over the network (i.e., reduce transfer speed). But since packet loss occurs frequently over WANs and is not due to network congestion, this is a poor strategy.
Resilio, on the other hand, uses a proprietary WAN optimization technology known as Zero Gravity Transport™. ZGT optimizes WAN transfer by:
- Creating a uniform packet distribution over time.
- Using interval acknowledgements for a group of packets (that also contains information about lost packets), rather than sending acknowledgements after every packet receipt.
- Using delayed retransmission of lost packets to increase transfer speed.
- Using a congestion control algorithm to calculate the ideal send rate and avoid overloading the network.
Case Study: MixHits Radio
MixHits Radio is a music streaming platform that uses Resilio Connect to update their music metadata across all of their servers. “We have gone from spending 15 hours on average per week troubleshooting conflicts in the prior solution to spending no time at all with Resilio. We configure jobs once in the Resilio Connect Management Console and never have to look at it again.”
Learn more about how Resilio helped MixHits reduce their workload by 60+ hours per month.
Organic scalability and N-way file replication
As mentioned earlier, larger replication workloads (i.e., replicating large files and a bigger number of files) can result in synchronization delays and affect data availability across your environment with traditional replication solutions. These issues can be compounded by complications in your replication system — i.e., hardware failures, network failures, downtime for maintenance, etc.
But Resilio Connect is an organically scalable web content replication solution that actually enhances performance as your environment and replication needs grow.
For example, the more endpoints you add to your system, the faster replication will occur. You’ll have more servers taking part in replication, and Resilio can balance the load across multiple servers to avoid overtaxing any single network or device.
In a 1:2 transfer scenario, Resilio Connect will transfer 50% faster than any dedicated server solution. That increases to 500% faster in a 1:10 transfer scenario.
And Resilio is an enterprise-proven solution that’s designed to handle large deployments. Resilio can replicate files of any size over any network, and has been tested and proven to replicate 250+ million files per job.
With its P2P replication architecture, Resilio can replicate in any direction required, such as one-to-one, two-way, one-to-many, many-to-one, many-to-many (aka N-way sync). This is particularly important when replicating content across many geographically distributed endpoints, as all devices can take part in replication simultaneously and quickly synchronize data so it’s available across your entire environment.
Easy to deploy and manage
Resilio Connect is designed to be easy to deploy and manage. It’s an agent-based software that can be deployed on your existing infrastructure and begin replicating in as little as 2 hours.
You can use Resilio in the cloud, on-premises, or in a hybrid model. It supports a variety of servers, virtual cloud servers, mobile devices, cloud storage services, and desktops. And it can be configured cross-platform on Linux, macOS, iOS, Android, and Microsoft servers.
With Resilio’s centralized Management Console, controlling and managing your replication jobs is easy. You’ll receive real-time notifications and detailed logs that provide insight into your replication environment. You can configure rules and notification parameters, control user permissions, set up Webhooks, adjust replication parameters (such as buffer size, bandwidth usage policies, disk I/O threads, and more).
And you can also use Resilio’s REST API to script any job type, automate replication jobs, and integrate them into your workflow.
Resilio Connect also utilizes state-of-the-art security features that have been reviewed by 3rd-party security experts, such as:
- In-transit encryption using AES 256
- One-time session encryption keys
- Mutual authentication to ensure data is only delivered to designated endpoints
- Cryptographic data integrity validation that ensures data arrives at its destination intact and uncorrupted
Suresync
SureSync is a real-time replication solution that can reliably replicate files over any network. It uses a remote differential compression algorithm to only replicate changed data. And you can use Suresync’s Enterprise Status Panel to monitor and control your replication jobs and user permissions.
Syncthing
Syncthing is an open-source file sync solution that works on Linux, Microsoft, Mac OS, FreeBSD, OpenBSD, Solaris, and more. It uses cryptographic certification and TLS encryption to protect data. And unlike other tools on this list, Syncthing provides full access to their open protocol and source code to give users more control over how replication occurs in their environment.
GoodSync
GoodSync is a real-time replication solution that can be used for both file synchronization and backup. It speeds up file replication by synchronizing data in parallel threads. It also uses data encryption and bit-by-bit synchronization to protect file data.
PeerGFS
PeerGFS provides real-time file replication that can be easily deployed in multi-site, multi-platform and hybrid environments. It uses a remote differential compression algorithm to only replicate changed file data and increase replication speed. And it uses Malicious Event Detection to monitor file activity for suspicious patterns and protect your data.
Use Resilio for Web Content Replication
Resilio Connect is the best web content replication solution on the market, particularly for large web and app deployments that require fast synchronization to many endpoints.
Resilio Connect is able to outperform competing solutions because it:
- Uses P2P file transfer: Every device in your environment can take part in replication jobs, enabling your system to easily overcome any network/bandwidth limitations and sync 3-10x faster than traditional solutions.
- Utilizes proprietary WAN optimization technology: Resilio’s proprietary WAN transfer protocol, Zero Gravity Transport™, enables it to quickly and efficiently replicate files over WAN networks.
- Scales organically: Because Resilio replicates using a mesh P2P network, it scales organically — i.e., the more devices you add to your environment, the better Resilio performs. It can also quickly sync files in any direction (one-to-one, two-way, one-to-many, many-to-one, N-way) and across many endpoints in about the same time that it takes to replicate to one endpoint.
- Is easy to deploy and manage: Resilio can be deployed on your existing infrastructure and begin replicating in as little as 2 hours. You can use Resilio’s Management Console to control replication parameters to suit your needs, maximize resource usage, and control costs.
To learn more about how Resilio Connect can provide fast, reliable web content replication for your app or website, schedule a demo.
