If you need to send large files¹ periodically, Dropbox’s cloud storage service may suffice. But for media companies and other organizations that deal with large project files on a daily basis — and multiple employees contributing to projects — the speed and consistency of large file transfer is critical.
This article discusses the issues you’ll run into when using Dropbox to sync large files, and why Resilio Connect is a much better solution.
Resilio Connect is a fast, easy, and workflow-friendly alternative to Dropbox, Google Drive, OneDrive and other other cloud sync and collaboration tools. The solution works with pretty much any device (Mac, Windows, Android, Linux, etc.) and gives teams a highly reliable and predictable way to sync and share files globally — using any cloud storage or local storage.
P2P-based file replication and sync solution helps organizations that work with big files or a large number of files perform remote work, disaster recovery, software updates, and more. Our software is used by media agencies, post-production professionals, and large companies such as Turner Sports, Skywalker Sound, Wargaming, Mercedes Benz, and more.
To learn more about how we can optimize your synchronization environment, contact our team.
¹With Dropbox Transfer, you can send files up to 100GB (or 250GB with the Creative Tools Add-On) without taking up space in your Dropbox.
Faster File Transfer: P2P vs Cloud-Hop
When real-time file transfer of large files is needed — like 4k or 8k video files, interactive media, architectural diagrams, or even topological maps for mining — Dropbox transfer won’t cut it.
While Dropbox doesn’t have any file sharing or file size limits, it’s a poor tool for transferring large files/large numbers of files because of two issues:
1. Cloud Hopping
Dropbox uses what is known as the client-server, or “hub-and-spoke,” transfer architecture.
In this model, there is one central server — the Dropbox.com cloud — that shares data with other devices in the network. But every other device can only share data with the central server, not directly with each other.
So if Dropbox Folder A wants to share files with Dropbox Folder B, it must first send those files to the central server (Dropbox cloud storage), which will then share the file to Dropbox Folder B — a process known as cloud hopping.
2. Upload Speed and Throttling
When you go to upload files with Dropbox, problems may arise. In order for a device to receive the files or file changes you share, you must upload those file changes and the recipient must download them onto their hard drive on their end.
Upload speed may be slower on your network compared to download speed. Most high-speed internet connections offers slower upload speeds, compared to download speeds.
Another aspect to slow uploads, other than bandwidth (slower bandwidth equates to longer upload times) is network quality. Some networks suffer from high latency and packet loss. These poor conditions can lengthen the time it takes to upload files—or cause uploads to fail completely. Whatever bandwidth you happen to have—you need to know that your files will be uploaded and available to other users in predictable timeframes.
For example, if the files are business critical, you’ll want to know that your files are where you need them—when you need them. That could be the difference in making a deadline or losing business. In some cases, files need to be reliably delivered to as many people as needed; whether it’s a few or a few hundred people.
Alas, tools like Dropbox don’t offer much predictability for uploading very large files, many files, or in situations where updates need to propagate (or synchronize) out to many other people or locations.
Not only that, but Dropbox throttles upload speeds. While it lets you download files as fast as your network can handle, it limits you to 75% of your upload bandwidth — which can be frustrating if you’re in a rush to upload and sync your files quickly.
P2P Architecture Leads to Faster File Transfer
Resilio’s P2P transfer architecture enables it to transfer large files much faster.
In a P2P (peer-to-peer) environment, every device can share files directly with every other device. There’s no middleman, or in this case, middle-server. And when conducting a transfer, Resilio splits files into blocks that independently transfer to multiple destinations.
For example, let’s assume Device A wants to share a large file with Devices B, C, D, and E. Device A can share one file block with Device B. While Device A sends the next block, Device B can share the first block with Device C — and so on.
This P2P environment delivers exponentially faster transfer speed because it:
(a) Eliminates the need for cloud-hopping
Files are sent uninterrupted to their destination. In other words, even if Dropbox and Resilio were transferring at the exact same speed, Resilio files would only need to make half as many stops and would arrive twice as fast.
(b) Enables omni-directional file transfer — that is, you can move files in any direction
With Dropbox, once your data is synced to the cloud, every user can view that data on their device. But to edit the data, they must download it to their device. This download speed can be reduced depending upon:
- How far away the device is from the Dropbox server.
- How many devices are trying to download at once.
But with Resilio’s omni-directional file transfer, each device can receive files from other devices that are closest to it. And the more devices you have in your network, the more bandwidth there is to share files. In other words, speed organically increases with scale.
Better Network Usage: WAN Optimization
Dropbox is dependent upon the network connection over which file transfer is occurring, and it doesn’t include any methods for optimizing network performance.
This can be especially problematic when transferring data over unreliable networks (like mobile and satellite), slow home broadband, and WANs (wide area networks) — i.e., networks that connect computers over a long distance, such as the internet.
When transferring over long-distance WANs:
- Transfer takes longer due to the greater distance.
- There is a greater chance of the data getting lost or corrupted along the way (a.k.a. packet loss).
With Dropbox, upload speed is dependent upon the network connection of the sending device, and download speed is dependent upon the network connection of the receiving device. Any disruptions in these networks — such as congestion due to lots of activity — can reduce your transfer speed. And if many devices are simultaneously requesting data from the server, that can place a heavy burden on the server network and reduce transfer speed.
Resilio overcomes these limitations using WAN optimization and P2P transfer architecture:
WAN Optimization
WAN optimization maximizes the speed, reliability, and security of transfer by controlling how and when file transfer occurs.
A big benefit of Resilio’s WAN optimization technology is giving you predictability: You’ll know exactly how long it will take to move or share your files; to upload, download, or sync files. In that respect, using Resilio is a bit like using UPS or FedEx. Your files get there on time, in a predictable timeframe.
Resilio uses our proprietary transfer protocol, Zero Gravity Transport (ZGT™), to minimize the impact of packet loss and maximize transfer speed across any network using:
(a) Congestion control
ZGT™ uses a congestion control algorithm to calculate the ideal rate at which to transfer file packets. It periodically probes the transfer speed to maintain an optimal send rate.
(b) Packet acknowledgement and retransmission
In most data transfer scenarios, the receiver sends an acknowledgement to the sender everytime it receives a packet of data. That acknowledgement tells the sender whether all the data was received or whether some information was lost (and what info was lost). The sender can then resend the lost information. Frequent acknowledgements can slow transfer speed. So with ZGT™, acknowledgements are sent at intervals for several groups of packets rather than each individual packet. And lost packets are resent in groups too, rather than individually.
Plus, if a connection is interrupted, Dropbox requires the sender to resend the entire file. Resilio intelligently detects where the transfer left off and automatically reinitiates delivery when the connection is restored. When transferring very large files, this can be a big time saver.
P2P Transfer Architecture
In addition to the other benefits we’ve discussed, P2P file transfer enhances network usage and performance.
First, since every device can share files with every other device, the overall load is diffused across your entire network rather than being focused inordinately on one device.
You can also leverage internet channels across all devices to dramatically increase speed. For example, if the network for one device is slow, Resilio can switch to perform transfer from a device with a stronger network connection.
Enhanced Scalability
Both Resilio and Dropbox are able to detect file changes and replicate only the changed portions of files in order to enhance replication speed — known as remote differential compression. (Dropbox also offers selective sync, a feature that allows you to select which Dropbox files get shared and which don’t).
But the more devices that are added to an environment using Dropbox, the worse Dropbox will perform because each device needs to first communicate with Dropbox’s cloud servers and sync with changes on other devices (“cloud hopping”, as we discussed above).
As previously mentioned, larger files and larger numbers of files will take more time to upload and download. Even with remote differential compression, if there are several file changes and the changes are large, sync speed will be impacted.
Because Dropbox is a centralized service, the more demand they experience, the more performance will be impacted. When several people in your network are trying to upload/download files simultaneously, it will take longer than usual to process your files.
By contrast, Resilio increases performance as more devices are added. Each “endpoint” (the device or system running a Resilio agent) participates in sending or synchronizing files. The load is distributed across each of the devices to send your files faster.
On average, Resilio Connect is 10-20x faster than conventional file transfer and sync solutions, depending on the network. Typically, the worse the network conditions, the more improvement you’ll see from Resilio Connect.
More Control & Insight
While Dropbox gives you management control over who on your team can receive, access, and edit files, it doesn’t give you much control over file synchronization.
For example, you can’t adjust how much network bandwidth your Dropbox account uses at a given time. Or the size of packet transfers. And most of how Dropbox syncs your files is controlled behind the scenes by their software.
Resilio’s central management console, by contrast, provides clear visibility and insight into the state of replication — across your entire environment. You can track the progress of individual jobs, get real-time notifications, view detailed logs, and adapt parameters on the fly.
You can allocate bandwidth, view the job status of individual jobs, and automate everything via scripting or through a well-documented API set.
- Bandwidth Control: Set limits on how much network bandwidth Resilio uses at any given time of day or day of the week. You can also create different schedules for different sync jobs. Simply increase bandwidth to increase speed.
- Network Stack: You can optimize Resilio’s performance by adjusting the size of the data packets it transfers, buffer size, and more.
- Storage Stack: Adjust disk io threads, data hashing, and file priorities.
- Functionality: You can use Resilio’s Rest API to script and automate all functionality, manage all agents, control transfer jobs and how they’re performed, or report on data transfers in real-time.
With Resilio’s management visibility and insight, companies can optimize replication performance, control costs and resource usage, and quickly identify and resolve any issues. This frees up lost productivity and improves daily operations for mission-critical data replication.
Sync Large Files with Resilio
While Dropbox is a suitable tool for sharing individual files, docs, and creating shared folders for distributed teams, it’s not built for teams that work with large video files, interactive media, architectural diagrams, and more. And, while it offers unlimited storage space, the reality is that your storage is capped and you’ll need to request more when you reach your limits.
Resilio is a superior sync solution for distributed teams working with large files and large numbers of files. Its P2P transfer architecture and WAN optimization provide faster transfer, optimized network usage, and enhanced scalability.
If you want to learn more about how Resilio can help you optimize your sync environment and workflow, contact our team.
