While tools like Dropbox, Google Drive, OneDrive, and WeTransfer can be used for file sharing, they’re not as reliable when it comes to:
- Transferring files over WAN networks
- Quickly detecting and transferring file changes in real-time
- Distributing file changes across many endpoints
- Transferring large files and/or large numbers of files (prices increase significantly with file sizes)
Large enterprises and media organizations that need to distribute and synchronize large files among multiple endpoints (e.g., companies with geographically distributed workforces that collaborate on large files, software and app companies that need to distribute large updates to web and app servers, etc.) often require more than just a large file transfer service. They need a solution that can quickly replicate file changes to multiple endpoints and provide:
Fast, Real-Time Replication
If file transfer takes too long, your solution will become a bottleneck in your workflow. Many replication solutions rely on incremental synchronization — i.e., periodically scanning your file directory and syncing all changes at once. But if you’re working with large file directories and/or large file sizes, incremental syncs can take a long time. More importantly, if you’re an organization that relies on timely and reliable access to business-critical files, then you need a solution that detects and replicates file changes in real-time.
Reliability
With legacy file transfer and replication solutions, IT teams are often forced to spend hours each week troubleshooting replication errors. A proper large file transfer solution should reliably deliver files of any size to their destination — intact and uncorrupted — over any network.
Scalability
Remote teams that work with larger files (such as video files or video game files) often have to replicate those files to multiple endpoints and/or replicate in multiple directions. Your solution should easily scale to quickly synchronize files to multiple endpoints, and be able to sync in multiple directions (one-way, two-way, N-way, etc.).
Ease-of-Use
Command-line solutions (like DFSR, Rsync, and Syncthing) can be complex, error-prone, and require a steep learning curve. Your file transfer solution should provide detailed insight into the state of transfer jobs (so you can monitor progress and confirm delivery) and allow you to adjust file transfer parameters so you can optimize resource use and control how replication occurs in your environment. It should also provide you with the flexibility to use your current on-prem infrastructure (servers, storage, networks, etc.) and incorporate the cloud on your own terms.
End-to-End Security
When you transfer and store your data off-prem and/or in the cloud, you expose it to security risks. Which is why you need a file transfer solution that protects your data at rest and in transit, while also ensuring data arrives at its destination uncorrupted.
Most large file transfer and replication solutions fall short on one or more of these criteria. But we built Resilio Connect to excel in all of the above areas, delivering the fastest and most reliable replication of large files of any solution available.
Resilio Connect is a real-time file replication solution that uses a P2P transfer architecture and WAN acceleration technology to deliver the fastest file transfer (10+ Gbps) over any network. It scales easily so it can replicate files to hundreds of endpoints in about the same time it takes to replicate to one endpoint, and can transfer files omnidirectionally.
Connect is used by leading companies in gaming (Blizzard, Larian Studios, Electronic Arts, etc.), media (CBS, Turner Sports, Skywalker Sound, etc.), and other industries.
Resilio can be deployed on your existing infrastructure in as little as 2 hours. To learn more about how Resilio can help you quickly and reliably replicate large files across your environment, schedule a demo.
Fast, Real-Time Replication with P2P File Transfer
One of the primary features that enable Resilio to deliver faster replication than most other solutions is its use of a peer-to-peer transfer architecture.
Most other solutions use either a client-server architecture or “follow-the-sun” architecture. In a client-server architecture, one device is designated as a hub-server and every other device is designated as a client. The hub-server can share files with any client device, but client devices can’t communicate with each other. In order to transfer a file from one client to another, the file must first be sent to the hub-server, which then transfers it to each client device individually.
In a “follow-the-sun” topology, file transfer occurs from device to device sequentially. In other words, Device 1 transfers files to Device 2; Device 2 then transfers the files to Device 3; Device 3 then transfers the files to Device 4; and so forth.
Both transfer topologies are impeded by bottlenecks inherent in the way they transfer files. Client-server transfer is impeded by cloud-hopping — i.e., the need to first transfer files to a hub-server before they can be replicated to the clients. And in both models, files can only be transferred to one device at a time. If the file transfer to any device is disrupted (for example, if a device is on a slow network or the network goes down), all the other devices must wait for that file transfer to complete before they can receive files.
But Resilio Connect’s P2P file transfer architecture overcomes all of those limitations. In a P2P topology, every device is equally privileged and can transfer files to any other device. Any changes made to files on Device 1 can be replicated directly to Device 2, 3, 4, 5 etc. — so there are no bottlenecks in your replication workflow.
Resilio also uses two other replication tactics that enable it to transfer files faster than other solutions on the market:
1. Simultaneous File Transfer Using File Chunking
Resilio Connect uses a process known as file chunking. When transferring a file, Resilio splits the file up into several blocks. Each block can transfer independently from the other blocks.
This enables all devices in your replication environment to simultaneously take part in file transfer. So, for example, if Device 1 wants to share a file with the five other devices in your environment, it can split that file into six blocks. Device 1 can share the first block with Device 2. Once Device 2 receives that first file block, it can begin sharing it with Device 3 even before it receives the rest of the file.
Every device in your environment can share the file blocks they have with other devices while they await the rest of the file transfer. With every device replicating files simultaneously, file transfer with Resilio occurs 3-10x faster than traditional solutions (delivering 10+ Gbps per endpoint).
2. Real-Time File Transfer with Optimized Checksum Calculations
To detect file changes, many traditional file transfer solutions rely on system scans — i.e., scanning all the files in your file directory to identify any changes. But Resilio Connect detects file changes using:
- Notifications from the host operating system
- Optimized checksum calculations
A checksum is an identification marker assigned to a file. When the file changes, so does the checksum. With OS notifications and checksum calculations, Resilio can identify and replicate file changes in real-time. This is especially helpful if you have large files and/or lots of files in your file directory, as scanning these large systems can be very time consuming.
Case Study: VoiceBase
VoiceBase is a speech-to-text software company that uses Resilio Connect to distribute speech models (50+ GB files) across 400+ endpoints. “Resilio Connect enables us to reliably distribute our code, specifically new language models in a fraction of time. These copy jobs now take an hour, down from eight. Best of all, once Resilio Connect was installed, it just works: We never need to manually intervene in any way.”
Read more about how Resilio enabled Voicebase to reduce replication time by 88%.
WAN Acceleration for Speed and Reliability
- Traditional solutions don’t use WAN optimized protocol
- Resilio uses proprietary WAN protocol
- Uniform packet distribution, congestion control (optimized delivery), interval acknowledgements, delayed retransmission (efficiency)
Most traditional file transfer providers perform poorly over WANs (wide area networks). The file transfer protocols (such as FTP) they use aren’t optimized for WAN transfer.
For example, many solutions use TCP/IP transfer protocol. When confronted with a large amount of packet loss, TCP/IP assumes that the issue is caused by network congestion and slows file transfer to resolve the problem.
But packet loss often occurs over WANs, and slowing file transfer isn’t a great tactic to take in the case of WAN transfer. So if you’re transferring files across high-latency, unreliable WANs, you’ll need a solution that’s optimized for WAN transfer.
Resilio Connect uses a proprietary WAN acceleration protocol known as Zero Gravity Transport™. ZGT optimizes WAN transfer by:
1. Improving packet distribution
ZGT uses a fixed packet delay to maintain a uniform packet distribution rate over time. And it uses a congestion control algorithm that periodically probes the speed of file transfer in order to calculate the ideal send rate. Together, these two processes enable Resilio Connect to provide the fastest WAN transfer speed without overloading the network.
2. Increasing efficiency
With traditional file transfer solutions, devices send acknowledgements after receiving each file packet. To increase efficiency, ZGT only sends acknowledgements for a group of packets. Each acknowledgement also provides information about packet loss, so Resilio can retransmit lost packets periodically and decrease unnecessary retransmissions.
Try Resilio’s transfer speed calculator to see how much time you can save.
Not only does ZGT enable Resilio Connect to perform better over WANs, it also enables Resilio to acquire, ingest, and sync files to a centralized location from the far edge of a network. So, for example, if you’re filming content for a documentary in a remote location with poor network connectivity, you can upload files and sync your footage to the devices across your entire replication environment.
Reliable Transfer with No Single Point-of-Failure
In a client-server topology, your entire replication system is dependent upon the hub-server. The hub-server must be functional and online at all times in order for any replication to occur. But servers can go down for many reasons, including network failure, hardware failures, downtime for maintenance or software updates, and more. This forces organizations to spend large sums of money on backup servers and failover plans in order to prevent their replication environment from going down.
Similarly, the “follow-the-sun” topology can be undone by server failure. If one of the servers or networks fails, replication can’t occur.
But Resilio’s P2P topology means that there is no single point-of-failure. If one device goes down, Resilio can dynamically route around the failure and the required files or services can be obtained from any other device in your environment.
Resilio’s ZGT transfer protocol can also optimize data transfer over any network by dynamically routing around failures, latency (i.e., a device can request data from the device closest to it), and network congestion.
Resilio Connect is also highly resilient. If a file transfer is interrupted mid-transfer, Resilio can perform a checksum restart to resume the file transfer where it left off, rather than starting over from scratch. Resilio will retry any failed file transfer until it is complete.
Because Resilio is able to fully utilize any network connectivity in your replication environment, it can meet sub-five-second RPOs (Recovery Point Objectives) and RTOs (Recovery Time Objectives) within minutes of an outage — making it a perfect disaster recovery solution (or supplement to your disaster recovery system).
Scalability and Omnidirectional Replication
Scaling up your infrastructure to support more data, devices, and/or users in a client-server topology is complex. As you add more replication endpoints (i.e., devices), you’ll also need to add more hub-servers to support them and design a system that balances the load between your servers. This can cost organizations a significant amount of money to create and maintain.
And because replication can only occur from one device to another sequentially, adding more devices considerably increases the amount of time it takes to replicate files across your environment in a “follow-the-sun” topology.
But Resilio’s P2P topology makes it organically scalable. Adding more devices simply means that you have more devices that can take part in replication. In other words, the system scales fluidly on demand and each endpoint provides resources (memory, storage, bandwidth, etc.). And transfer speed increases as you add more devices to your environment.
Resilio Connect can transfer files 50% faster than traditional file transfer solutions in a 1:2 transfer scenario. And transfer will occur 500% faster in a 1:10 transfer scenario.
Resilio can also transfer files omnidirectionally. It can perform a one-to-one sync, two-way sync, one-to-many sync, or N-way sync — making it a perfect replication solution for organizations and applications with many end users who need timely and dependable access to files. With Resilio, you can rest assured that your files are always where they need to be.
Case Study: Larian Studios
Larian Studios is a video game development company that uses Resilio Connect to distribute video game files to their entire development team. “The nightly builds that we pump out involve a large amount of data. Getting that synced to every QA team in each studio proved to be quite difficult with the software that we had. With Resilio, as soon as our build servers have a build ready, it’s using the entire throughput of our gigabit lines to distribute it to the other studios. So QA can start in the morning on a fresh build.”
Read more about how Resilio helped Larian Studios increase transfer speed by 500%.
Easy Management Through Resilio’s Management Console
The value of any solution depends largely on how easy it is to use. And many of our users report that their previous solutions were too complex, difficult to navigate, and provided little-to-no insight into the state of replication in their environment.
But Resilio Connect was designed for ease-of-use, and provides a great degree of insight into and control over your replication process.
Resilio’s Management Console gives you detailed insight by providing event logs and real-time notifications, so you always know the state of replication and whether your files successfully reached their destination.
You can also optimize Resilio’s performance in order to maximize resource usage and minimize costs by adjusting replication parameters, such as:
- Controlling how much bandwidth each device can use, and creating schedules that automate how much bandwidth is allocated to each device at certain times of the day and days of the week.
- Creating rules that control different agent groups and replication jobs, dictate notification events, and govern which files are to be transferred and when.
- Setting up Webhooks, controlling user permissions, and adjusting replication parameters (such as disk io threads, buffer size, and more).
Resilio Connect is flexible and can be easily deployed on your existing infrastructure in as little as 2 hours. It can transfer files of any size (no file size limit) and has been tested to replicate at 250+ million files per job.
It works with popular operating systems and can be configured cross-platform on servers such as Windows, OS X, Android, Linux, and more (Resilio even offers iOS and Android mobile apps, as well as desktop apps for Windows, Mac, Linux, and FreeBSD). And it can be deployed with your preferred cloud storage service, on on-prem hardware, or in a hybrid scenario.
You can use Resilio’s powerful REST API to manage agents, script functionality, generate real-time reports on file transfers, and more.
End-to-End Security
Many off-prem, cloud-based solutions may not meet your organization’s security requirements. Which is why Resilio provides secure file transfer using state-of-the art security features that were reviewed by 3rd party security experts, such as:
- End-to-end encryption of files at rest and in transit with AES 256.
- Cryptographic data integrity validation that ensures data remains uncorrupted.
- One-time session encryption keys that protect your sensitive data.
- Mutual authentication to ensure data is only delivered to designated endpoints.
Use Resilio for Large File Transfer and Replication
While some organizations can get by with traditional file sharing solutions (or sharing files as email attachments on Gmail and other email clients), many enterprise and media organizations require a more robust solution for large file transfer and replication.
Resilio Connect is the ideal solution for organizations that need to transfer and replicate big files. It’s P2P transfer architecture and WAN optimization technology enable it to provide:
- Fast, real-time replication: Resilio provides the fastest transfer speeds in the industry — 10+ Gbps per endpoint, up to 10x faster than traditional solutions.
- Optimized WAN transfer: Resilio can reliably transfer files over high-latency, unreliable internet connections, dynamically route around failures, and sync at the edge of the network.
- Reliable file transfer: Resilio’s P2P architecture means that there is no single point of failure. And it will resume transfers until they’re complete and your files arrive at their destination.
- Organic scalability: Resilio can scale to meet your needs. Performance increases as you add more endpoints. And you can transfer files in any direction.
To learn more about how Resilio can help you quickly and reliably replicate large files across your environment, schedule a demo.
