TCP (Transmission Control Protocol) has been the dominant protocol used to establish connections and transfer files. However, UDP (User Datagram Protocol) transfer solutions are gaining popularity as a way to quickly move large amounts of data across long-distance, low-quality networks.
For example, Google recently announced that 50% of their Chrome traffic is run by their UDP-based protocol (Google QUIC), and they expect that to increase in the future. That’s because UDP file transfer is faster and more efficient than traditional TCP-based file transfer protocols.
Plus, TCP protocols don’t meet the demands of modern, large-scale data transfer over the WAN (wide area network) because they:
- Require the receiving device to send an acknowledgment after receiving each data packet before the next packet can be sent, which reduces transfer speeds.
- Treat packet loss as a network congestion problem and, in response, slow down transfer speeds. But packet loss is a defining characteristic of WANs, so this response unnecessarily slows down replication.
In contrast, UDP protocols don’t require an ordered delivery of packets or acknowledgments, and don’t respond to packet loss in the same way. This reduces bandwidth consumption and latency, which allows UDP to utilize network connections more efficiently and provide faster file transfers.
That said, UDP and TCP-based approaches also face similar architectural challenges. Most UDP-based file transfer solutions are limited by their point-to-point replication architectures. Point-to-point replication limits transfers and flows to (at most) two servers and creates a single point of failure.
Our UDP-optimized file sync solution, Resilio Connect, overcomes these limitations by using a P2P (peer-to-peer) replication architecture that enables every endpoint to replicate files simultaneously and eliminates single points of failure.
Want to learn how you can use Resilio Connect’s UDP-based transfer for secure, high-speed transfers over any network? Schedule a demo with our team.
In this article, you’ll learn about the 3 top UDP file transfer software solutions. We’ll begin with a deep dive into Resilio Connect, where we discuss how its proprietary UDP-based WAN acceleration protocol and P2P replication architecture allow it to provide faster, more reliable transfers than competing solutions — over any network.
Organizations in media (Turner Sports), gaming (2K Games), engineering (Deutsche Aircraft), retail (McDonalds), and more use Resilio Connect for fast, reliable synchronization of large datasets across many endpoints.
Resilio offers many capabilities that make it the optimal solution for long-distance enterprise transfer and sync workflows, such as:
- Network optimization: Resilio’s proprietary UDP transfer protocol preserves bandwidth and efficiently transfers files over any network (such as VSATs, cell, Wi-Fi, and any IP connection), regardless of distance, latency, or quality.
- Resilient, multidirectional P2P replication: Resilio’s P2P architecture eliminates single points of failure to ensure your data always reaches its destination. It also enables our solution to sync files 3–10x faster than point-to-point solutions, reaching speeds of 100+ Gbps per cluster.
- Centralized management and automation: Resilio’s Management Console allows you to control all endpoints and replication jobs from one centralized interface. You can adjust parameters to optimize resource utilization and costs, and automate jobs to minimize management time. You can also pin traffic to any network using Resilio’s Smart Routing capabilities.
- Flexible deployment: Resilio is an agent-based solution that can be easily deployed in just about any on-premises, hybrid, and multi-cloud environment. You can install Resilio on your existing infrastructure and begin replicating in as little as 2 hours.
- Native security: Resilio includes built-in security features that were reviewed by 3rd party security experts to protect your data end-to-end.
To learn more about how Resilio Connect can help you sync large files over any network and across many endpoints, schedule a demo with our team.
1. Resilio Connect
Resilio Connect uses a proprietary, UDP-based WAN optimization technology and a unique P2P architecture to transfer and synchronize payloads of any size across any network, quickly and efficiently — regardless of distance. It also gets data across networks that have high latency and packet loss in a smart, reliable, and predictable way.
Resilio has several use cases around data transfer, sync, and replication, such as:
- Quickly transferring and syncing large files across many geographically dispersed endpoints.
- Simplifying data replication, sync, ingest, transfer, and access by controlling all aspects from a single Central Management Console.
- Moving large amounts of data from the deep edge into the cloud (or moving data from the cloud to edge locations) over unreliable networks, like Wi-Fi, VSAT, cell, and broadband. Learn more about Resilio’s edge ingest capabilities here.
- And many more.
UDP Transfer for Speed and Reliability Over Any Network
Transfers over long-distance WANs are often plagued by high latency and varying degrees of packet loss. Resilio overcomes these issues with a proprietary UDP-based WAN acceleration transport protocol known as Zero Gravity Transport™ (ZGT).
ZGT is superior to other UDP protocols because it’s:
- Faster: ZGT optimizes bandwidth utilization to provide speeds over 100x greater than existing solutions.
- More efficient: ZGT’s congestion control algorithm optimizes transfer rates to produce a uniform packet distribution over time. It also uses interval acknowledgments and delayed retransmission.
- More reliable: ZGT enables Resilio to work over any type of network connection. And it ensures your files always arrive at their destination uncorrupted.
Speed and Efficiency
ZGT optimizes consumption and bandwidth utilization, allowing it to achieve 100x greater transfer speeds over existing solutions, using:
- A congestion control algorithm: ZGT uses a congestion control algorithm that constantly probes the Round Trip Time (RTT) of a network to identify and maintain the ideal data packet send rate.
- Interval acknowledgments: Unlike TCP protocols that send acknowledgments after each packet receipt, ZGT sends acknowledgments in groups to eliminate replication delays.
- Delayed retransmission: ZGT retransmits lost packets once per RTT to reduce unnecessary retransmissions.
ZGT intelligently and automatically analyzes the underlying network performance by measuring packet loss, latency, and throughput over time. It automatically adjusts to maintain a consistent speed and maximize utilization of the internet connection, learning and adapting to conditions in real-time.
These features reduce traffic over the network to enhance transfer speeds, thus eliminating idle time between teams or across applications. This enhances workflows and speeds up production times in scenarios like:
- Filling data lakes.
- Media production.
- Replicating DFS namespaces.
- Ingesting machine learning data into a GPU public cloud.
- Accelerating software build distribution and CI/CD test cycles.
- Syncing large files with cloud object storage (such as AWS, Azure, Google Cloud Platform, and more).
With ZGT, Resilio can deliver data transfer speeds of 10 Gbps per server, regardless of latency or packet loss. Using horizontal scale-out replication, you can cluster multiple 10 Gbps agents in combination to reach speeds of 100+ Gbps per server.
In fact, our engineers successfully transferred a 1 TB payload across Azure regions in 90 seconds.
Reliability
A common problem with most UDP-based transfer tools is that, unlike TCP, UDP doesn’t have to verify data packets. This can make some UDP transfer solutions less reliable, as files may not be fully delivered to their destination.
However, as we said, Resilio does acknowledge packet receipts. But it does so in intervals for groups of packets to increase efficiency. It also uses cryptographic integrity validation to ensure your files arrive at their destination intact and uncorrupted.
Put simply, ZGT makes Resilio a highly reliable solution that works under the most extreme network conditions. It supports any type of network connection, such as VSATs, cell (3G, 4G, 5G), Wi-Fi, and any IP connection.
You can use Resilio to ingest data from edge devices, as well as at the far edge in areas with little connectivity (e.g., at sea or in countries with underdeveloped network infrastructure). For example, healthcare company Shifo uses Resilio to quickly and reliably sync healthcare data in remote communities, such as Uganda.
Finally, Resilio supports offline work. If you’re working over a shoddy connection that drops in and out, you can continue to work offline and sync files over the LAN. When the connection is reestablished, Resilio will automatically sync your files with all of your other endpoints.
As stated earlier, most UDP-based transfer solutions still suffer from the bottleneck of point-to-point replication architectures. Which is why Resilio’s peer-to-peer replication architecture enables it to really stand out as the best solution for transferring large files over long distances.
P2P Replication That Provides Resilient, Scalable, and Multidirectional Transfer
Another key to Resilio’s superiority is its P2P replication architecture.
Most file transfer tools (including the other two mentioned on this list) use a point-to-point replication architecture deployed in one of two models:
- Hub-and-spoke: In hub-and-spoke replication, there is a hub server and several remote servers. The remote servers can’t send data to each other. All transfers must first go to the hub server, which then replicates the data to each remote server one by one.
- Follow-the-sun: In follow-the-sun replication, data is replicated from one server to another sequentially.
Both of these models are slow and unreliable.
Transfers can only occur between two servers at a time. And transfer speeds are limited by your slowest endpoint.
They also create single points of failure — if any server goes down, it will delay synchronization for your entire environment. And if the hub server goes down in the hub-and-spoke model, replication fails entirely.
But in a P2P environment, every endpoint with a Resilio agent can share files directly with every other endpoint. And all endpoints can work together to share files simultaneously. This enables Resilio to overcome the limitations of point-to-point architectures and deliver file transfer, sync, and replication that’s:
Blazing-Fast
Few file sync solutions offer true real-time replication. But by using optimized checksum calculations (identification markers assigned to each file that change when the file is changed) and real-time notifications from the host OS, Resilio can immediately detect and replicate file changes. Plus, it replicates only the file deltas, making the process extremely efficient.
The real key to Resilio’s blazing-fast speed is due to the combination of file chunking and P2P replication.
File chunking is the process of splitting files into separate chunks that can transfer independently of each other. Every endpoint can then share file chunks concurrently, allowing you to sync your environment 3–10x faster than point-to-point solutions.
For example, if you want to sync a file across five servers, Resilio can split that file into five chunks. Server 1 can share the first chunk with Server 2. Server 2 can immediately share that first chunk with another server, even before it receives the remaining chunks. Soon, every server will be sharing file chunks at the same time.
Multidirectional
P2P sync enables Resilio to sync files in any direction, such as one-way, two-way, one-to-many, many-to-one, and N-way.
N-way sync is another critical component to Resilio’s speed and superiority in use cases such as:
Remote Work
Remote and distributed teams can collaborate on files from anywhere in the world. If any employee makes a change to a file, that file change will immediately sync across every other endpoint in your environment. Everyone on your team will always have the most up-to-date versions of files.
Disaster Recovery
P2P replication allows it to provide Active-Active High Availability for disaster recovery scenarios. It effectively turns every endpoint in your environment into a backup site, and all of your endpoints can work together to bring your system back online in the event of a disaster.
Resilio can achieve sub-five-second RPOs and RTOs within minutes of an outage. And it can dynamically route around outages if any endpoint or network goes down.
Case Study: Skywalker Sound
During the COVID-19 pandemic, Skywalker Sound switched to a remote workforce and used Resilio Connect to sync files between their studios and remote workers.
“I think the concept of people being more distributed and being able to work from home is here to stay. With Resilio in place… we’ve achieved that in such a way that we can have a more diverse workforce and give our staff more flexibility with how they can work in the future.”
Organically Scalable
A P2P environment scales organically. Since every endpoint can share files with each other, adding more endpoints only increases the available resources (CPU, bandwidth, etc.). So adding more endpoints increases transfer speeds, allowing Resilio to sync 200 endpoints in roughly the same time it takes most point-to-point solutions to sync just two.
Resilio can also sync files of any size, type, and number (we’ve successfully synced 450+ million files in a single job).
Resilient
Resilio is also an incredibly resilient and fault-tolerant solution because it:
- Eliminates single points of failure: If any endpoint or network in your environment goes down, files can be retrieved from any other endpoint.
- Transfers reliably: Resilio retries all transfers until they’re complete. If a transfer is interrupted, Resilio can perform a checksum restart to resume the transfer at the point of failure (rather than starting over).
Easy, Centralized Management and Automation
Resilio enables you to control, manage, and monitor your entire replication environment from one centralized location using Resilio’s Management Console which can be accessed via any web browser or through the Resilio portal (you can also use command line prompts).
Through the Management Console, you can:
- Control and monitor all replication jobs.
- Configure notifications to be sent to email or Webhooks.
- Manage all of your endpoints from one centralized location.
- Adjust replication parameters to obtain granular control over how replication occurs, so you can optimize resource utilization and costs.
- Use Resilio’s REST API to script any type of functionality, as well as automate jobs and troubleshooting processes (such as file conflict resolution) in order to minimize management time and human intervention.
- Create bandwidth profiles that govern how much bandwidth is allocated to each endpoint at certain times of the day and on certain days of the week.
Case Study: VoiceBase
VoiceBase provides speech-to-text for audio and video transcription. They use Resilio Connect to distribute large speech model files of 50+ GB across more than 400 servers every 2–4 weeks.
“Resilio Connect enables us to reliably distribute our code, specifically new language models in a fraction of time. These copy jobs now take an hour, down from eight. Best of all, once Resilio Connect was installed, it just works: We never need to manually intervene in any way,”
Read more about how Resilio helped VoiceBase reduce software distribution time by 92%.
Flexible Deployment That Supports Any Scenario
Resilio is built on universal standards and open protocols, allowing you to retain full control over your data and deploy Resilio in almost any environment.
You can install Resilio agents directly onto your devices or virtual machines, and deploy it on your existing on-premises or hybrid/multi-cloud environments IT infrastructure.
Resilio supports just about any:
- Device: Install Resilio on desktops, laptops, file servers, NAS/DAS/SAN devices, IoT devices, mobile devices (Resilio offers Android and iOS apps), and virtual machines.
- Cloud storage: Resilio supports just about any S3-compatible cloud object storage, such as AWS S3, Google Cloud Platform, Azure, Wasabi, MinIO, Backblaze, and more.
- Operating system: Resilio supports just about any operating system, such as Microsoft Windows, MacOS, Linux, Unix, Ubuntu, FreeBSD, OpenBSD, and more.
Resilio also offers Proxy Server capabilities that allow IT teams to:
- Adjust traffic and security firewalls to suit your use case and needs.
- Control who can access your machines in order to protect your data and devices when working with 3rd parties.
Built-In Bulletproof Security to Protect Your Data
Data security is paramount, particularly when transferring files across long distances.
Most file transfer solutions don’t include built-in security features, forcing you to invest in 3rd party security tools and VPNs.
But Resilio protects your data using:
- End-to-end encryption: Resilio uses AES-256-bit encryption to protect data at rest and in transit.
- Cryptographic integrity validation: As stated earlier, Resilio ensures that your files arrive at their destination intact and uncorrupted using cryptographic validation.
- Mutual authentication: Resilio uses bootstrap keys when establishing connections with any endpoint with a Resilio agent on it. Endpoints must provide encryption keys before initiating any transfers. Your files will only be delivered to authenticated endpoints.
- Forward secrecy: Resilio protects sessions using one-time session encryption keys.
- Permission controls: You can control who is allowed to access specific files and folders.
- Access controls: When working with other companies or contractors, you can control who’s allowed to access your machines.
2. IBM Aspera
Aspera Sync is a file transfer solution that’s designed to cater to media companies. It easily integrates with media workflow ecosystems, making it well-suited to media workloads.
Aspera uses a UDP-based transport protocol for accelerated file transfer over long-distance WANs. It also increases transfer speeds by using log-based change data capture to reduce the amount of data being replicated, which is particularly useful for large file transfers.
Aspera offers several features to protect your data, including blockchain technology for extra data security. Still, some Aspera users have experienced security breaches and found Aspera customer support to be unresponsive in these situations.
Check out our article on Aspera Sync alternatives for demanding workflows and remote teams.
3. Signiant
Signiant is a file sync solution that also caters to media companies and easily integrates well with many media workflow solutions. However, Signiant doesn’t support hybrid cloud environments and can’t perform true bidirectional replication — which limits its performance and utility for many use cases requiring large file transfer and sync.
Signiant is known for being easy to deploy and use. It uses TLS encryption to protect data and, like Resilio, can perform checksum restarts to resume interrupted file transfers.
Check out our article on 5 Signiant alternatives for large file transfer and sync.
Quickly Transfer Large Data Across Long Distances with Resilio Connect
Resilio Connect is the best UDP-based file synchronization software solution for transferring large files over WANs (and poor-quality networks) because it provides:
- Optimized transfers over any network: Resilio’s proprietary UDP-based WAN acceleration protocol efficiently and fully utilizes any type of network connection (VSAT, cell, Wi-Fi, IP) to provide transfer speeds 100x faster than other UDP protocols.
- Fast, P2P sync: Resilio’s P2P architecture enables every endpoint to share files concurrently, resulting in transfer speeds 3–10x faster than point-to-point solutions.
- Multidirectional synchronization: Resilio can sync files one-way, two-way, one-to-many, many-to-one, and N-way.
- Organic scalability: Resilio’s P2P architecture scales organically to support environments of any size. Adding more endpoints only increases sync speeds. And it can sync files across 200 endpoints in roughly the same time most point-to-point solutions take to sync just two.
- Resilience: Resilio eliminates single points of failure, dynamically routes around outages, and retries all transfers until they’re complete.
- Flexible deployment: Resilio can be installed on just about any device, cloud storage service, and operating system.
- Centralized management: Resilio gives you granular control over your entire replication environment from a single, centralized location. And you can automate replication jobs and troubleshooting to occur with minimal human intervention.
- Native security: Resilio includes native security features that protect your data at rest and in transit.
To learn more about how Resilio Connect can help you sync large files over any network and across many endpoints, schedule a demo with our team.
