In previous blogs in this series, I talked about the and the fact that in a data pipeline there are into the training cluster. In this blog, I want to go one level deeper and talk about the specific set of choices that you can make to smooth the flow of data into the training cluster. If you think of the GPUs in your training cluster as a high-performance car, a good data pipeline is like the difference between taking that car out on a racetrack or taking it out on the freeway at rush hour. Filesystem Selection for the Data Pipeline As I mentioned in the , object storage is not designed to deliver the level of performance that your data pipeline is going to require—now or in the future. Data Flow into the Training Cluster There are some nuances around the way that data flows into the training cluster that are important to understand. It has the ability to curate the data and lay it out as a set of coalesced file streams that can be nicely aligned with the training cluster, allowing data to stream directly into cluster CPUs to pre-load and feed GPUs. Depending on the types of data sources you have, your data architecture may need to be able to deliver both large sequential reads and small random reads into the training cluster. Single namespace AI datasets have the potential to grow to massive size, leading to tremendous data sprawl. Accommodating this growth requires a scale-out filesystem with a single namespace and the ability to scale performance linearly to a single client node as well as to multiple client nodes accessing the same data in parallel. An architecture that can continue to scale as you add compute and capacity is going to be critical. There can be different types of client access to this single namespace, each with implications for performance. Other training models run synchronously. The training model and its dataset have tight coupling and the dataset is shared across all cluster nodes with simultaneous access. There are other uses cases where a multi-layered neural network trains the layers of the network on different nodes. The nodes serve as a model pipeline where the model progresses from one node to the next. As a relatively new filesystem, HDFS has had limited exposure to diverse data workloads and performance characteristics. Big data vendors have been undertaking significant and proprietary re-writes to deal with the performance needs in the transition from MapReduce to Spark; AI introduces another wrinkle in the HDFS story. Relying on a big-data-specific filesystem like HDFS can mean more data copies and siloes as you find yourself doing yet another data copy from HDFS into a high performance scale-out filesystem for AI. Metadata performance The same access patterns discussed in the preceding section also have implications for metadata performance. Each node in the training cluster may query metadata independently, so metadata access performance must scale linearly with the growth of the filesystem. Metadata access with filesystems such as Lustre and GPFS can become a bottleneck due to reliance on separate metadata servers and storage. Other Factors There are a variety of other factors that you should also take into account when selecting a filesystem for your data pipeline needs. Can the filesystem scale autonomously and automatically without management intervention? How much time and technical expertise does the filesystem take to manage? How easy is it to find people with the necessary expertise? Scale-out filesystems such as Lustre and GPFS can be challenging to configure, maintain, monitor and manage. By comparison, NFS is easy to manage and NFS expertise is widespread. Quality of Service QoS can also be an important element of your data architecture. You may be building multi-tenant training clusters with price tags running into the millions of dollars. QoS plays a key role in your ability to deliver multi-tenancy, enabling multiple activities to share the same resources. Cloning capabilities Part of the multi-tenancy requirement is to satisfy different job functions within your organization. Space-efficient cloning is therefore a must-have for a multi-tenant cluster. Client-side caching Use of a client-side cache helps further accelerate performance by providing a data buffer that enables uninterrupted data flow as the training dataset is accessed from training cluster nodes. A filesystem that supports an ecosystem of client caching products whether open-source or commercial can provide substantial advantages. A variety of open-source and commercial options exist for NFS-based storage. Few if any client-side caching products currently exist for Lustre, GPFS, or HDFS. Almost none are open-source and freely available. For AI that is applied in a post-process fashion—such as for surveillance, fraud detection, etc. However, if real-time performance is a key requirement or a key competitive differentiator, you will likely continue to need a dedicated data copy for the training cluster. Is the filesystem optimized for flash today? Is it seamlessly extensible to support new technologies, and are vendors actively innovating in areas such as NVMe, NVMeOF, NVDIMM and 3D XPoint? Flash today is capable of latencies around 500 microseconds. NVMeoF will take that down to 200 microseconds. NVDIMM, 3D XPoint, and persistent memory are poised to take latencies to sub 100microseconds, sub 10 microseconds, and eventually nanoseconds. Your data pipeline vendor needs to be making sustained investments to keep pace with this evolution, across server-based and shared-storage solutions. Future Proofing your Data Architecture and Filesystem Choice The whole AI field is evolving very quickly, but it can be impractical or impossible to re-build your architecture from scratch every 6 months to a year. As a final consideration, you should try to make technology choices that are as future-proof as possible. The ability to seamlessly and non-disruptively evolve different layers of technology such as filesystem, interconnect, deployment location, media and memory type within a chosen infrastructure provides long-term return on investment and ability to absorb technology evolutions as they occur. You will likely want to factor in past deployment experience, existing deployments, and existing infrastructure. Over time, you may decide the 100GbE or 400GbE roadmap with NFS is better for your needs. A well thought through data architecture is able to accommodate and future proof the solution allowing you to seamlessly switch your filesystem without replacing infrastructure. Similarly, you may choose NFS today but decide you need a SAN, NVMe, or NVMeoF-based filesystem or a persistent-memory-based data layout in the future. A future-proofed architecture allows you to evolve datastore technologies without needing to replace your entire deployed infrastructure. Which Data Architecture Will You Choose? We believe that the combination of NFS running on NetApp AFF storage is a leading contender based on our ability to address these needs and to evolve in place to accommodate the latest technologies. Want to know more about the factors involved in the architecture of a data pipeline for deep learning, specifically data management and the hybrid cloud? Santosh Rao is a Senior Technical Director for the Data ONTAP Engineering Group at NetApp. In this role, he is responsible for Data ONTAP technology innovation agenda for Workloads and Solutions ranging from NoSQL, Big Data, Deep Learning, and other 2nd and 3rd Platform Workloads. He has held a number of roles within NetApp and led the original ground up development of Clustered ONTAP SAN for NetApp as well as a number of follow-on ONTAP SAN products for Data Migration, Mobility, Protection, Virtualization, SLO Management, App Integration and All Flash SAN. Prior to joining NetApp, Santosh was a Master Technologist for HP and led the development of a number of Storage and Operating System Technologies for HP including development of their early generation products for a variety of storage and OS technologies over the years.
