DFSR Alternative: A Guide to Fast & Resilient P2P File Replication with Resilio

What is DFSR?

DFSR (sometimes written DFS-R), or distributed file system replication, is a feature of Windows Server for replicating files across several servers. Microsoft introduced this feature in Windows NT4.0 as an add-on, and it became a standard feature in Windows 2000 server and on.

Basically, it was an attempt by Microsoft to solve the complex problem of distributing files in real time so teams across several offices could collaborate effectively. Nowadays, the challenge of real time file replication and distribution is much easier to solve, with the advent of the public cloud. Instead of sharing files between offices, people work directly with the cloud. However, the cloud doesn’t solve all the issues in collaboration and you might still need to build a system where several servers need to be synchronized across different offices. Let’s see if DFSR from the early 2000’s is an adequate response to these needs, or if a DFSR replacement should be considered.

When to use a Distributed File Systems Replication Tool vs. When to Find an Alternative

First, let’s see why you might need to synchronize servers directly. One of the reasons might be that you have big files and would require significant time to upload to the cloud. The second case might be that due to a variety of reasons you can’t upload files to the cloud. It might be security concerns, company or government policy etc. Third and most popular reasons that we see is that moving files or synchronizing servers is just a part of your workflow. You are building a business process that is more complex than just uploading and downloading files to the cloud.

DFSR is a standard client-server or 1:1 file synchronization approach that was popular 10-15 years ago. Most other commercial solutions use the same model for synchronization – copy a file from point A to point B. This approach obviously has limitations, such as the availability of A – if server A is not available, then server B and C will not get the data. If there is a resource shortage on server A (networking, memory, CPU) the load cannot be moved to server B, even if it has all the data available.

DFSR Issues & Size Limits

There are a number of other limitations to DFSR. If you need to extend your file synchronization beyond Windows servers, DFSR is not the right tool for the job. You also can’t give different roles to different servers, e.g. if you need some servers to just read information, without being able to change it.

A DFSR-based solution may be effective for some businesses, but let’s take a look into typical DFSR problems that can arise.
‍

DFSR Over Long WAN Connections

When you need to send data to remote locations over mobile or satellite connections that have a very long retransmission time and potential for packet losses, DFSR can run into problems.  It’s based on the TCP/IP protocol, which treats every packet loss as a network congestion and backs off speed in order to reduce the load of the connection. This approach helps applications that are TCP/IP based share networks and collectively agree on the maximum speed they can use for data transfer. In case of wide-area networks (WAN), the packet loss might represent some failure on intermediate device and the channel is often not congested. Therefore, the logic of reducing speed in case of packet loss is not appropriate for WAN connectivity.
‍

TCP/IP Data Delivery Faults

Another problem in the WAN and use of DFSR, relates to the way TCP/IP guarantees data delivery. First, it needs a recipient to acknowledge that packet arrived at the destination. Once a recipient gets a packet, it sends a confirmation packet to the sender acknowledging that it has received the specific packet. The time during which the packet travels from sender to receiver is called retransmission time (RTT). In the local network (LAN) it is below 0.01ms, but in WAN networks it can be as high as 800ms or more. Therefore, a receiver can wait up to a second or even more before it’s able to send another packet. These TCP/IP deficiencies are inherited by DFSR. Overcoming these bottlenecks requires the consideration of a DFSR alternative.
‍

Inability to Deliver Files to Multiple Locations

DFSR is limited in the ability to synchronize or deliver files to more than one destination. It is extremely rare that an organization needs to send or synchronize files just to one location or server. Usually, it is more than one destination server, in more than one location. It such a case, a common approach is to execute jobs one by one. DFSR sends files to one location, then to another.
‍

Poorly Optimized for Real Time File Replication

If you’re working with tens of millions of files and real-time change detection, you’ll find DFSR is not optimized for a large number of files. DFSR speed is usually very slow when you need to synchronize folders with few millions of files since it takes forever to scan through this folder, find changes and transfer them. A better approach would be to use real-time file system monitoring to pick up changes on the fly without the need to browse through the whole directory tree like Resilio Platform.
‍

Lack of Smart Data Routing

DFSR needs static ip:ports to establish a connection to different machines. If a machine has  a new IP:port or is not available, DFSR stops operation and needs a human to re-configure.
‍

Inability To Incorporate Scripting Into DFSR Workflows

Simply put, there is no way to have scripting around DFSR. If you need to build workflows beyond a simple “do something after the file arrives at destination”, there is no way to do so with DFSR.

In contrast to DFSR, alternative replication & file distribution solutions Resilio Platform were developed to be faster under any network conditions and data volumes. This distributed file system replication service is specifically optimized to replicate large volumes of data, and works well with many small files, without any limits.  Besides, with a unique network protocol, it’s sensitive to bandwidth changes and is smart enough to avoid network congestions or use full bandwidth when possible. Resilio Platform lets you take control over the file replication process, see its progress and evaluate the results.

Resilio Platform data replication and synchronization is not limited to the Windows platform only, and can be easily configured as a cross-platform solution (Linux, OS X, iOS, and Android).
‍

File Caching Now Available

Active Everywhere delivers automated cache management controlled by policies in the management dashboard—unique to each caching location.

SCHEDULE A DEMO

Solutions to Common DFSR Problems

How to make DFSR faster

Because DFSR’s basic set of technologies is limited to delta encoding and compression, it means it’s impossible to make DFSR faster. You would need to replace DFSR to get faster performance. If you are looking for a faster alternative to DFSR, consider Resilio Platform, which adds peer-to-peer data transfer, WAN line optimization, smart routing and real-time file system monitoring in order to speed up syncing for today’s enterprise.
‍

DFSR and Transferring Large Files

Because Windows DFSR does not have an optimized way of calculating the checksum of a file, this leads to an extremely long time to calculate file differences, making large file transfers take that much longer. Unlike DFSR speed issues, our DFSR alternative, Resilio Platform, optimizes the checksum calculations quickly, allowing for files of all sizes to sync very fast.

DFSR & End-To-End Encryption

One of the biggest drawbacks of Windows DFSR is that it lacks embedded traffic encryption, which makes it impossible to use DFSR without additional encryption. DFSR currently requires users to install and configure an additional encryption channel – a VPN. Any good DFSR alternative should include encryption within the product so that no additional products or set up is required. Our DFSR replacement, Resilio Platform, uses AES256 in CTR mode to encrypt all traffic that is sent between clients. This also includes both data and all of the control traffic.
‍

DFSR & Static IP Addresses

Another limitation of DFSR is that it requires a static IP and port address for a destination. This makes DFSR impossible to user in a dynamic network environment. With rsync, as soon as an IP address of a server is changed, then rsync will fail to operate. A good DFSR replacement, such as Resilio Platform, will use a dynamic routing approach that specifies when machine A and B need to exchange data. This means there’s no human intervention necessary at all — both machines will use a tracker or multicast to discover the required IP:Port address on the fly.
‍

DFSR & WAN Network Connection

Because DFSR lacks WAN network support, it makes it impossible to use DFSR over long connections for offices located 3,000 miles or more apart. The long distance of the office creates a long travel time for the TCP packet, which increases the chances of packet loss due to equipment failure or congestion. As rsync is built on the TCP protocol, the sync speed will be slow when using these types of networks.

To prevent packet loss over large distances, Resilio Platform uses WAN network support, allowing you to utilize 100% of the available bandwidth in your network totally independent of distance, latency, or loss. To achieve this, Resilio Platform uses UDP based protocol uTP2, which uses bulk packet transfer with selective acknowledgement of lost packets in transferring large files across vast distances.
‍

DFSR & Transferring Files to Several Destinations

Any good alternative to DFSR will use a peer-to-peer approach, which leverages networking between all available offices and significantly speeds up data transfer times. Using this approach splits each file into blocks, then sends these blocks independently. This means that each recipient can send the block to another recipient once it has been received. Using a peer-to-peer approach dramatically speeds up real time syncing operations, since we are not only transferring concurrently to several machines, but also using other network channels to help offload loads from a sender network channel. Resilio Platform uses this approach to deliver large files in real time quickly.
‍

DFSR & NAT

NAT, which is usually a firewall that hides server internal addresses and provides a connection with external IP addresses, makes it impossible for DFSR to connect to a server. In order to do this, you will need to open ports for each incoming connection on the device so that rsync can establish a connection. Resilio Platform, our DFSR replacement, uses NAT traversal techniques that allow a direct connection between computers without the need to open ports.

Summary of Resilio Platform vs. DFSR

Here is a handy summary table of the features needed in a synchronization solution today, and how Resilio Platform stacks up against DFSR.
‍

Features

DFSR

Resilio Platform

Cross platform solution

Max number of servers

256 for each replication group

No limits

Max number of files

70 million (2012R2)
11 million (Server 2008)

No limits

ACL replication

Max file size

250 Gb (2012R2)
64 Gb (Server 2008)

No limits

Data deduplication

WAN acceleration

P2P replication

Time to replicate 3.72 Gb single file

4:00 min

1:39 min

Time to replicate 100k files of 4kb each

46 min

17 min

Delta encoding

Compression

In-transit encryption

WAN optimization

NAT traversal

10M+ files support

Very large folders

Real-time response to changes

About The Author

Konstantin Lissounov  /  Co-Founder & CTO

Konstantin serves as Chief Technology Officer (CTO) of Resilio Inc. and is the co-inventor of BitTorrent Sync, with over twenty years of software development experience. Konstantin is responsible for overseeing the software engineering and development of Resilio core products and sets the technology direction and strategy.