Universal File Delivery: Overcoming Challenges Moving Mountains of File Data

A friend of mine once said:  “There are many paths to the top of the mountain.” 

And so is the case moving and managing massive (and not so massive) unstructured data on a global scale.  Which path will get you and your file data up the metaphorical mountain the fastest, with enough energy and resources to enjoy the view and get your data back down the hill in a coherent state?

In our previous blog we covered some history: consolidating on premises file servers to NAS (network-attached storage) to scale-out and cloud NAS. While storage solutions for unstructured data and cloud file storage have improved over the years, a huge challenge remains:  

Coherently managing large and growing data sets (including large files and many smaller files) across storage silos and other devices–spanning any number of locations and endpoints.  

Should be easy, right? Well, let’s also consider that IT infrastructure teams may also need solutions for:  

• Minimizing downtime and improving high availability and disaster recovery to meet SLAs for recovery times and recovery point objectives and ensuring data protection.
• Having the flexibility and built-in security to support a wide range of enterprise-wide IT initiatives such as remote work, virtual desktop infrastructure (VDI), real-time synchronization, large-scale file distribution, consolidation (ingest), among others. 
• Ensuring that mission critical applications and workflows depending on file data continue operating nonstop, irrespective of location or network quality (no matter how remote or unreliable the network).    
• Minimizing cost and complexity by utilizing in-place IT infrastructure (networks, servers, storage, and other devices) to optimize utilization wherever possible.
• Enabling multi-cloud and hybrid cloud architectures for customer-specific initiatives using cloud services from providers like Amazon AWS, Microsoft Azure, Google GCP, Oracle, Wasabi, among others.
• Moving file-based data rapidly and reliably over any distance and network, at any scale.  
• Avoiding vendor lock-in through the use of open data formats and open, cross-platform, software-only architectures.

These are a short sampling of requirements that enterprise solutions ought to address. This blog explores contemporary strategies for coping with large-scale file data management–including NAS-to-NAS replication, ad hoc file transfer, and cloud file systems–and compares and contrasts these approaches to Resilio’s universal file delivery strategy, powered by Resilio Connect, a centrally managed, software-only system.  

IT Infrastructure 

The following types of infrastructure will be referenced throughout this blog: 

• Core on premises data centers and public cloud providers.  Includes servers, storage, networks, running a variety of OS platforms.  Connectivity varies–but utilizing high performance networks for connectivity at the core will be a goal.  
• Remote and branch offices, home offices, and other remote operations facilities. In some use cases these facilities may be temporary (e.g. construction, live event coverage, on location filming, etc.). In most cases, connectivity is over a wide-area network (WAN) or broadband provider.  
• Mobile and edge devices from desktops to laptops to mobile devices to IoT.  These devices run a variety of operating systems (Windows, Mac, Linux, Android, etc.). Connectivity varies but may include VSAT, LTE/5G, WiFI, or broadband access.

Data Movement Strategies: Summary

Today, there are a variety of popular strategies for replicating and managing data across storage silos, devices, and locations.  These include (but are not limited to): 

• NAS-to-NAS replication:  Use your storage vendor’s replication tools to manage and synchronize data.  If your company has standardized on a single storage vendor across all locations, and your recovery objectives are achieved, these may be “good enough”.  Resilio Platform complements investments in storage and supports most NAS. Resilio Platform adds value supporting remote work, VDI, any-directional data movement, and interoperating with multiple cloud storage providers and operating systems on most any device.
• Ad hoc point-to-point file transfer and replication tools such as rsync (mainly for Linux), Robocopy (Windows) and various WAN-optimized commercial offerings from vendors such as IBM Aspera, FileCatalyst, and Signiant are all point-to-point. These “point solutions” may work great for some use cases but are by no means comprehensive.  Worse, home grown or highly modified solutions may have hidden costs such as failed or unreliable transfers or lost data.  Resilio offers an easy way to eliminate many of these ad hoc approaches–and standardize across the enterprise.
• Global or cloud file systems – Also proprietary storage offerings, but from vendors such as Ctera, Panzura, and Nasuni, among others.  We will go deeper on these solutions below.  These systems include a global namespace, offer file locking, and support common file systems operations (read, write, modify, delete, etc.).  
• Universal file delivery – Resilio’s strategy is built on open, cross-platform, and software-only universal file delivery.  Resilio enables scale-out, “any-directional” real-time file delivery to unify all IT infrastructure and devices (core, cloud, edge) and help IT control and accelerate global data movement, over any location and network.  By contrast to cloud file systems, Resilio Platform complements and seamlessly integrates with Microsoft DFS and NFS 4.0. as well as deployed storage (DAS, SAN, and NAS) infrastructure. 

Platform and Location: Independent Universal File Delivery

In the Resilio model, all (or a subset of) devices and locations (any number of endpoints) can be unified in an open, cross-platform any-directional data control plane.  Some workflows requiring nonstop reliability and predictability depend on highly reliable uni- , bi- , and in some cases multi-directional data delivery (e.g. for remote upload/download or massive file distribution to multiple endpoints).  Resilio’s customers in maritime, oil and gas, transportation, and emergency responders (such as for logistics in fire and rescue, EMTs, marine operators, et al) all require nonstop mission critical reliability. Data needs to be there on time, or within a given transfer window, no matter what. For these use cases, file data is usually transported across unreliable networks such as VSAT, LTE/5G, WiFi, etc. Resilio’s highly reliable transport is WAN optimized, and configurable end-to-end, to ensure data is always delivered in predictable time frames.

In another scenario, some enterprise customers (such as in gaming and tech, media and entertainment, and restaurant services) may require fluid and extreme scaling of the network or data center core for large-scale distribution and synchronization.  Multiple data centers, multiple cloud providers, and any number of server or storage endpoints (running a diversity of operating systems) may need to be managed. In these cases, unifying, controlling and accelerating data movement may require true multi-directional or N-way file distribution or sync, depending on the use case.  Resilio’s gaming customers need massively scalable and reliable distribution to keep meeting their time-sensitive SLAs in support of CI/DI pipelines for game development.  

In any case, Resilio’s universal approach builds on any-directional scalability yet also may leverage deployed technologies such as Microsoft Distributed File System and NFS v4.0 global namespace.  Resilio also offers turnkey multi-cloud support for Amazon S3, Azure Files and Blobs, Azure NetApp Files, Wasabi, Backblaze, Google GCP, and any S3-compatible object storage.

Let’s compare and contrast a few of the different approaches to multi-site and multi-directional data replication. 

NAS-to-NAS Replication

Your traditional storage system vendor likely offers snapshot-based replication between or among its own proprietary, vendor-specific software and/or hardware.   Storage-to-storage replication works great for 1-to-1 (uni- or bi-directional) replication within sites or over low-latency links.  WAN optimization may help depending on the workload–but will add considerable cost, as “in band” appliances or SD-WAN will be required on each site/location.  This will not benefit remote workers or remote endpoints.  

Other points worth considering in your storage vendor replication include:

• Not real-time.  Most replication jobs run on a schedule. Depending on the use case, this is at best infrequently or at most every 5 minutes or so.  By contrast, Resilio synchronizes data in real-time: as soon as a file changes (or in the case of VDI profile sync, when a user logs off) data is immediately replicated.
• Active-passive vs active-active for HA. In most cases, it’s cost prohibitive or impractical to configure active-active high availability in most NAS systems.  The target data is not able to be accessed until there is some failover event. Resilio is active-active. Users can be logged in at any location and have access to their profile and all of their redirected share data.  Resilio enables real-time synchronization.
• Uni- and bi-directional replication–vs “any-directional”. In traditional and scale-out NAS, a destination target requires one source. Resilio supports any-directional: 1:1, 1:Many, Many:1, or Many:Many. Resilio optionally parallelizes data movement, hashing data across multiple endpoints and/or multiple locations in parallel.
• Proprietary vs Open. Resilio is cross-platform and can sync data with any other platforms, including Windows, Apple Mac, Linux, Android, some NAS OSs, Azure Files, Azure Blobs, an Azure VM with block storage attached, any on-prem storage, etc. Resilio’s multi-cloud, cross-platform architecture supports most popular operating systems and cloud storage environments. 
• Storage limited by location or cloud regions. Resilio can sync between any combination of regions without limitation.  Moreover, Resilio offers WAN optimization natively for UDP and TCP to enable full utilization of allocated bandwidth over and LAN or WAN.
  
  In supporting remote work use cases, Resilio for VDI environments enables N-way active-active-active deployments. A customer can have users directed to any number of locations, then a user can log out and log back in again, be directed to some other location, and have local access to their data. Importantly, there is also no failover or failback event or coordination that needs to occur when there is downtime in a location.

Ad Hoc Point-to-Point File Transfer Tools 

Most IT folks are familiar with or using well-known file copy and transfer tools, from SCP to XCOPY to rsync to Robocopy.  If you’re administering DFS, you may be familiar with DFS-R (DFS Replication) and its challenges.  Moving to cloud object storage?  You may be using AWS DataSync, Oracle Sync Files, or Azure File Sync.  Or writing S3-compatible APIs.

These individual tools usually work well for narrow, platform- or service-specific tasks, within their prescribed environments.  Yet, in other cases, for rapid, predictable data movement across platforms–or when supporting large data sets and large files–these tools can be slow, cumbersome, and solely applicable to specific features or cloud storage services.  They do not scale to the needs of many larger or data-intensive enterprises.  Nor are they universally applicable across the enterprise.  

In summary, these tools can be classified as:

• Ad hoc point solutions:  Tools such as DFS-R, rsync, and Robocopy perform some tasks well.  In other cases, there are a number of reasons to deactivate or end-of-life many of these tools, when they be come cost-prohibitive to use, manage, and operate.  Each has trade-offs.  Resilio offers superior replacements for DFS-R, rsync, and Robocopy yet also in universally applicable to other enterprise-wide file transfer and data delivery needs.
• Commercial products from IBM Aspera, File Catalyst, Peer Software, and Signiant, among others, offer solid file transfer with WAN optimization built in.  For some use cases and workflows, these offerings perform well. But their point-to-point architecture and price point may limit their applicability and usefulness across the enterprise.  Resilio is priced aggressively as pay-as-you-grow, accelerates performance from Mbps to 10Gpbs, over any distance, and offers an open, cross platform, universally applicable toolset.
• Point-to-Point (traditional) vs Peer-to-Peer (Resilio).  Point-to-point architectures limit scalability and throughput performance between 2 systems (or endpoints).  There is no way to scale-out or move data beyond the resources contained within an individual endpoint (compute, memory, storage IOPS/throughput, or network bandwidth and throughput).  Multi-threading may help scale up CPU within a server but does not scale-out.  By contrast, Resilio’s scale-out performance is based on a distributed, peer-to-peer architecture.  Resilio’s transport hashes data across (any number of predefined endpoints) to swarm data, finding the best route and ensuring data is delivered reliably and efficiently at prescribed speeds.
• Costly to manage and operate over time.   In IT workflows where an individual or group of individuals have customized, adapted, or otherwise integrated or modified their file transfer products or tools, these integrations can fail or become unsupportable over time.  We’ve all heard stories of the one developer that left the company and forgot to leave instructions for operating or debugging a tool or app. In worst case scenarios, costs are incurred due to the unpredictable and sporadic, error-prone nature of these tools, which may lead to failed transfers, backlogs of errors, or (in the very worst case scenario) data corruption or data loss.

Resilio Connect’s universal file delivery approach always protects data and ensures data integrity. With Resilio, it’s easy to start small and grow and standardize all file delivery tools and operations around a single, open, cross-platform solution–that scales as your data movement needs demand.  

Cloud File Systems (or Global File Systems)

The masters of marketing have been busy.  From the Freedom Filers of Panzura to the UniFS™ of Nasuni, cloud file systems promise to slice, dice, chop, lock, and simplify data access for everyone.  We’ve heard both good stories and bad. In my view, there is no silver, monolithic, file system bullet to solve the challenge of managing all data for everyone everywhere. Alas, proceed with caution when evaluating cloud file systems.  

As of this writing, cloud file systems offer a complete cloud-native or hybrid cloud solution with a variety of caveats depending on the use case.  While compelling, the value proposition is essentially rip-and-replace your existing storage with the cloud file system.  By contrast, Resilio offers a “unify, extend, control, and accelerate” your existing IT infrastructure proposition.

The cloud file system model requires a proprietary file system and all its associated software and hardware components–deployed across each site in your entire IT infrastructure, including your hosted cloud resources on your cloud providers’ infrastructure.  Theses systems are purported to be POSIX compliant.  Thus, consider the performance implications when accessing data that is not cached locally or needs to be read or cached on demand, over a high latency network.  A distributed metadata server architecture supplies all locations–all on premises data centers, remote sites, and multiple cloud providers’ cloud storage.  This strategy relies on caching data at remote sites such as branch offices.  Caching requires proprietary appliances. For some sites, this may work well.  

But what if your enterprise has numerous remote sites? Over 5 or more or imagine 100?  Deploying a caching appliance at each site may be cost prohibitive and impractical.  

Thus, there are some caveats to this cloud file system approach: 

• Cloud native. If you’re a traditional on-premise only or hybrid deployment, or only use the cloud on a project-by-project basis, you will have to go “all in” on the cloud.  Resilio offers compatibility with any IT infrastructure, giving IT the flexibility to use existing IT infrastructure with turnkey multi-cloud support for popular cloud providers.  
• Proprietary data formats (vs open formats). Unlike TAR and ZIP and most native open file system formats, Nasuni, for example, stores data and metadata in a proprietary format.  If you ever have to ditch Nasuni, recovering your data and metadata could be challenging. In the case of Nasuni at least, because the data is stored in a proprietary format, you will need Nasuni to get the data out of the cloud. Resilio, by contrast, simply transfers data in its native format, with options to capture a diversity of metadata associated with your existing file systems.   
• Eggs in one storage basket.  The value proposition from most storage vendors is a single vendor solution: all software and hardware comes from one vendor.  All use cases for file access, data storage, backup, and recovery, etc. rely on one vendor.  Combined with using proprietary formats, some cloud file system vendors offer an “all or nothing” value proposition.  With Resilio, you can leverage your existing storage, servers, and network–and extend those investments to the cloud and deployed devices.  
• Pricing and upfront capital costs.  We’ve heard cloud file systems companies target larger deals with larger customers above 20TBs or $50K depending on who you talk to.  Data intensive SMBs or enterprises with smaller budgets may find global file systems cost prohibitive.  Resilio offers pay-as-you-grow bundles that start small and can be deployed project by project or enterprise-wide.  
• Caveats with global file locking.  We’ve heard numerous issues with global file locking. These depend on the types of applications and use cases.  Resilio advises any customers considering cloud file systems to go through an exhaustive evaluation or POC.  Resilio, by contrast, offloads file locking to the application, file sharing protocol, and/or native operating environment.  
• Caveats with performance (point-to-point vs P2P).  Cloud file systems face the same challenges moving data as any point-to-point architecture.  In cases where scale-out data movement for reliability and performance are required, Resilio offers built-in data resilience for data integrity and predictable transfer performance (from Kbps to 10Gbps and beyond).
• Lack of visibility into monitoring and metrics.  Customers have reported challenges diagnosing problems and troubleshooting events.  Resilio Platform is centrally managed using a scalable management console that supports thousands of clients.  By contrast, one Resilio customer, Kentucky Fried Chicken, manages and monitors 3,500 endpoints using Connect.   

Summary

Resilio does not sell file systems, file storage, file servers, or cloud storage.  Resilio gives enterprise customers an open, software-only, universal file delivery system to rapidly move files of any size and type anywhere, across any location, using your existing IT infrastructure (including storage) and your cloud providers of choice.  

Resilio has no limits on file size or type.  You can move and synchronize the largest data sets–stored on the same NAS your company already owns and manages–on-premise and in the cloud–across any location and network.  Resilio complements your existing IT investments, reducing the burden on IT, while accelerating enterprise file data up the metaphorical mountain.  Cloud computing, cloud data, or on-premises only, the outlook for managing massive data sets across the globe looks pretty good from this viewpoint.

Learn more about Resilio Connect and how it can provide a universal file delivery system to rapidly move files. Or, schedule a demo or start a free trial to see for yourself.