If you work for a media organization or large enterprise and are responsible for technical solutions for a remote workforce that often shares and collaborates on large files (or large numbers of files), you need a secure large file transfer solution with several key functions and capabilities. For example:
- Speed: No matter how “securely” a tool transfers files, if transfer takes too long, then it becomes a bottleneck in your workflow. The best solutions quickly transfer your files to end users across any network in an acceptable time frame. And if you require instant, “real-time” replication of file changes so everyone on your team always has the most up-to-date version, you need a solution that instantly detects and propagates changes across your network.
- Scalability: Most file transfer solutions can only go one-way: transfer from point A to point B. But in highly-collaborative environments where you must replicate files to many endpoints, the ability to quickly distribute and/or synchronize files omnidirectionally (i.e., from one-to-many, many-to-one, or many-to-many endpoints concurrently) is critical. With Resilio, files can sync one-way, two-way, or even N-way — in about the same time it takes to transfer or sync a file between two endpoints.
- Security: When working with sensitive, private data, your solution needs end-to-end encryption to protect against security threats; nor should it rely on 3rd-party security services.
- Reliability: You shouldn’t have to worry about whether your files will arrive safely (and uncorrupted) at their ultimate destination(s). A proper file transfer tool always delivers complete, updated files to their destination and works around poor network connections and outages.
- Visibility & Management: For peace of mind, your solution should provide detailed insight into the state of all data transfer jobs, and confirm that your files successfully reached their destination. It should also enable you to control file transfer parameters (i.e., bandwidth usage, when files transfer, which files transfer, etc.) to optimize your workflows.
In this article, we’ll discuss why traditional file transfer services and cloud storage solutions (such as Dropbox, Google Drive, Signiant, Aspera, etc.) fall short for serious business users and how our files and server replication software, Resilio Connect, solves common problems for our users.
Resilio Connect is used by leading companies in media (Turner Sports, Skywalker Sound, and more), gaming (Larian Studios, Electronic Arts, etc.), and other industries to provide secure, real-time replication of files across the organization. Unlike other solutions, Resilio Connect uses a P2P (peer-to-peer) transfer architecture, proprietary WAN optimization technology, and more to distribute files concurrently to many endpoints and across any network.
To learn more about how to use Resilio to securely transfer large files in your organization, schedule a demo. Deploy Resilio across your organization in as little as two hours.
Faster File Transfer via P2P Architecture & WAN Optimization
For teams to effectively collaborate on large files (such as video files), everyone must have access to the most up-to-date versions of files as quickly as possible. But traditional file transfer service providers use a client-server transfer architecture, which significantly reduces transfer speed and impedes workflows.
In a typical client-server transfer architecture, one computer is designated as the hub server and all other computers are designated as client devices. The hub server can share files with any of the client devices. But the client devices can only share files with the hub server, and can’t share files directly with any other client device. This creates a lag that results in frustration for any remote team.
Imagine you have 5 teams in different locations collaborating on the same file. If Device 1 wants to share file changes with the other devices in the network, those changes must first be sent to the hub server, which then shares them with Devices 2, 3, 4, and 5 — a process known as cloud-hopping.
Cloud-hopping introduces an unnecessary file transfer step that reduces transfer speed. In addition, the hub server can only share files with each client device one at a time. In other words, the hub server must finish transferring the file to Device 2 before it can begin transferring to Device 3. If one of the devices is on a slow network, experiencing network outages, or impeded in some other way, it can obstruct transfer to every other device in the network.
In contrast, Resilio Connect uses a P2P transfer architecture and WAN optimization to quickly transfer files to every device in your environment and across any network.
P2P Transfer Architecture
In a P2P environment, every computer is equally privileged (based on your settings and rules) and can share files directly with any other computer in your environment. So any changes on Device 1 can be shared directly with Devices 2, 3, 4, and 5 (though you can restrict sync jobs to only certain privileged assets and devices, if you choose) — eliminating the need for cloud-hopping.
Connect also increases peer-to-peer transfer speed using file chunking and optimized checksum calculations.
Optimized checksum calculations
The purpose of checksum calculations is to transfer only the changes made to a file instead of sending the whole file every time a change is made. In this process, every file is assigned an identification marker. When changes are made to the file, the checksum marker also changes.
Resilio uses checksum calculations and real-time notification events from the host operating system (OS) to detect and replicate only the changed portions of a file. This reduces the load on the network and increases transfer speed.
File chunking
With file chunking, files are split into blocks that can independently transfer to multiple destinations. Each device can independently share file blocks with other devices. For example, let’s assume Device 1 wants to share a file with every other device in your environment. It splits the file into six file blocks and begins sharing them with Device 2. Once Device 2 receives the first file block, it can share that block with another device before it receives the complete file.
This dynamic means that adding more endpoints increases file transfer speed. In a 1:2 transfer scenario, Resilio Connect will perform 50% faster (on average) than traditional file transfer solutions. In a 1:10 scenario, transfer will occur 500% faster.
According to Producer Octaaf Fieremans at gaming company Larian Studios:
“The nightly builds that we pump out involve a large amount of data. Getting that synced to every QA team in each studio proved to be quite difficult with the software that we had. With Resilio, as soon as our build servers have a build ready, it’s using the entire throughput of our gigabit lines to distribute it to the other studios. So QA can start in the morning on a fresh build.”
Learn how Resilio helped Larian Studios increase transfer speed by 500%.
WAN Optimization
Traditional file transfer solutions like FTP and other TCP-based solutions aren’t optimized for WANs (wide area networks). So if you need to transfer files across long distances and/or on unreliable networks with high latency or packet loss, file transfer is unreliable.
But Resilio uses a proprietary, UDP-based file transfer protocol known as Zero Gravity Transport (ZGT™). ZGT maximizes transfer speed across any network and minimizes the impact of packet loss and high latency using:
1. Uniform packet distribution & congestion control
In order to provide the fastest file transfer without overloading the network, Resilio uses a fixed packet delay to maintain a uniform rate of packet distribution over time. To calculate this ideal send rate, Resilio uses a congestion control algorithm that periodically probes the RTT (Round Trip Time) — i.e., the time it takes for the destination device to receive a packet — in order to constantly keep the algorithm informed about the current speed of file transfer on the network.
2. Interval acknowledgements and delayed retransmission
With other file transfer solutions, the destination device sends an acknowledgement after receiving each file packet. Resilio sends acknowledgements for a group of packets that provides additional information about packet loss. It then retransmits losts packets once per RTT to decrease unnecessary retransmissions. This process is more efficient.
And because Resilio is optimized for transfer over unreliable networks, it is able to acquire, ingest, and sync files from the far edge of the network to a centralized location. For example, a documentary crew filming content in a remote location can quickly upload and sync their footage with devices across your entire environment.
Try our transfer speed calculator to see how much time we can save for you.
Organic Scalability & Omnidirectional Sync
In a client-server transfer environment, scaling up your infrastructure to support more data, devices, and/or users is costly. A single hub-server can only service a certain number of client devices. As you add more client devices, you’ll also need to add more hub-servers — a significant capital expense. And you’ll need to design a system that balances the network load between the two hub-servers.
But Resilio’s P2P architecture is organically scalable. Every device in your environment acts as a hub-server. So adding more demand (i.e. replication endpoints) inherently creates more supply.
Resilio’s ability to transfer omnidirectionally means that it can rapidly perform a one-to-one sync, one-to-many sync, or N-way sync (concurrently sharing files across your entire environment). This makes it especially valuable for applications that involve many end users that depend on having the right data at the right time. No matter how many endpoints, Resilio reduces overall time to file access and ensures your files are always where they need to be.
Secure File Transfer with End-to-End Encryption
Resilio enables secure file sharing using state-of-the-art security features that were reviewed by 3rd party security experts.
Resilio enables secure file transfer using:
- End-to-end encryption: Your data is encrypted at rest and in transit using AES 256, so it can’t be hacked or intercepted.
- Data validation: Resilio uses a cryptographic data integrity validation process to ensure your data arrives at its destination uncorrupted.
- Forward secrecy: Resilio uses one-time session encryption keys to protect sensitive data.
- Mutual authentication: Your data is only delivered to designated endpoints.
Reliable Transfer & High Availability
With traditional file transfer solutions, your entire transfer infrastructure is dependent upon the hub server. In order for things to work, the hub server must be online and available at all times. But hub server availability can be impacted by any number of problems, including system errors, hardware failure, network disruption, and more.
But Resilio’s P2P architecture and WAN optimization technology make it incredibly resilient and highly available.
No single point of failure
Because Resilio’s file transfer is peer-to-peer and can dynamically route around failures, there is no single point of failure. If one device or network goes down, any device can receive files or service from any other device in the system.
Resilio’s ZGT transfer protocol is sensitive to bandwidth changes and can optimize data transfer over any network. Data is dynamically routed around failures to overcome latency and network congestion.
Highly resilient
In the event that there is a network failure in the middle of a file transfer, Resilio will perform a checksum restart and resume the transfer where it left off. And Resilio will retry transfers until they’re complete.
Resilio is also the ideal solution for disaster recovery because it can fully utilize any network connectivity in your environment in order to meet sub-five-second RPOs (Recovery Point Objectives) and RTOs (Recovery Time Objectives) within minutes of an outage.
Active-Active High Availability
Resilio’s omnidirectional file transfer is perfect for organizations that need an Active-Active availability scenario. Each device can provide data and services to any other device, and all devices can share the load balance between them.
File changes can be replicated across multiple endpoints concurrently, making Resilio perfect for VDI scenarios. One Resilio customer saw a 300% faster time-to-desktop for their VMware Dynamic Environment Manager (DEM) as compared to their previous snapshot-based storage solution. That meant the IT team reduced time-to-desktop by 90 seconds on average for 1,000 remote users.
Flexible deployment
With Resilio’s flexible BYO storage, it can be deployed on your existing infrastructure (on-prem, edge, cloud, or hybrid) and begin syncing in as little as 2 hours. It’s vendor-agnostic and can be configured cross-platform on Windows, Linux, OsX, Android, and iOS.
High Visibility & Control Over File Transfer
Resilio’s dashboard provides peace of mind by giving you detailed insight into your replication environment. It provides detailed event logs and real-time notifications so you’re always clear about the state of replication and ensures your files have reached their destination.
Resilio also enables you to adjust replication parameters in order to optimize performance.
The client includes granular settings to adjust bandwidth usage on each device in a network, and enables you to set usage limits based on the time of the day and day of the week. Easily create profiles that control different agent groups and different replication jobs.
Using Resilio’s REST API, admins can manage agents, control jobs, script functionality, and report on file transfers in real time. Optimize file transfers by adjusting packet size, buffer size, data hashing, disk io threads, and more.
Fast, Secure Large File Transfer with Resilio
Resilio is the ideal solution for organizations that need to securely share and collaborate on large files or large numbers of files. Unlike email clients (such as Google & Outlook), there are no file size limits inhibiting transfer.
Resilio provides the fastest transfer speeds over any internet connection and can support 250+ million files per job, unlimited file sizes, and advanced delivery scenarios like bidirectional and N-way sync.
To learn more about how to use Resilio to securely transfer large files in your organization, schedule a demo.
