Storage Systems - Almaden - overview
Storage Systems Research group at IBM Research – Almaden has a long history of pioneering work in the data storage field. The group consists of about fifty exceptional engineers and researchers working on the cutting-edge technologies. The diverse mix of skills and expertise allows us to conduct research and advanced development in a wide variety of topics and effectively convert research results into IBM products.
Our research covers all aspects of storage systems and storage solutions, and we explore technologies that can transform storage systems and support new classes of applications. Our goal is to design innovative storage and file systems, simplify and dramatically reduce the cost of storage management, and create new solutions that address the requirements of emerging data-intensive applications. We design and develop novel storage systems that push the limits of today's data storage capabilities. We have projects exploring extreme performance storage systems, systems that can store data reliably for decades, and systems that can store data at a fraction of the cost of today's storage systems.
We collaborate with academic partners and strive for the rapid transfer of our technologies to IBM products by working closely with our product teams. We submit dozens of patents every year and actively publish in major storage and systems conferences.
Below are some of the research directions we actively work on.
• Software for Storage Class Memories
For over half a century, SRAM and DRAM have dominated the cache and main memory tiers in most computer architectures. Unlike the lower tiers of "storage" memory, main memory is byte-addressable, meaning that it can be directly operated on by the microprocessor's load/store instructions. Today, we are on the cusp of a dramatic change in computer architecture. New memory technologies (coined Storage Class Memory – SCM) that have been developed over the last decade are finally reaching product-viability. They promise a new memory tier that provides sub-microsecond access latencies when directly attached to the memory bus, higher throughput than NAND-Flash, and cost/densities that are an order of magnitude improvement over conventional DRAM.
We are exploring the impact of this dramatic shift from a full-stack perspective. Specifically, our work is exploring new software architectures that can manage and unleash the potential of SCM. This includes exploration of new data structures and programming abstractions that can ease "consumption" of SCM technology and the development of high-performance memory management services that both eliminate traditional I/O bottlenecks and provide richer storage-like services (e.g., durable transactions, distributed replication) while retaining a "consumption-as-memory" model. Framing this research is the enablement of cognitive applications and services that consume large, complex data structures and that demand either high-performance multiplexing (such as Cloud microservices) or low-latency, durable data processing (such as financial, critical-systems and IoT).
• Data compression for genomic analysis pipelines
The increased affordability of personal genome sequencing and personalized medicine resulted in a surge of genomic data (about 1/2 TB per person) that needs to be stored and processed. We explore novel approaches to represent genomic data to enable faster execution of the analysis pipelines, and use of smaller storage footprint. Specifically, we work on the seamless integrating genomic data (FASTQ, SAM, and VCF) compression in IBM Spectrum Scale.
• Trillion Operations in a Day (T.O.A.D)
The "Trillion Operations in a Day (T.O.A.D)" project is re-imagining the storage stack for the next-generation of mission-critical, real-time analytics applications that generate ultra-high velocity data. T.O.A.D is a clustered, embedded key-value storage engine that is architected to take advantage of high performance storage and networking technologies.
• Ubiquity container volume service
The Ubiquity container volume service is an open-source unified file and block data management service that provides file persistence within CaaS, PaaS and IaaS cloud deployment runtimes/frameworks such as Docker, OpenStack, Cloud Foundry, and Kubernetes. Data management of persistent volumes is a key challenge of new generation workflows. In many companies, many application workflows require access to persistent storage across numerous runtime frameworks and environments using a variety of storage access protocols. For example, a workflow may begin by having the user upload data using NFS or SMB from a laptop. The uploaded data is then processed and analyzed by dockerized applications using native distributed file system clients. The results are then consumed by the next application in the workflow (which may be running in separate environment such as OpenStack) and/or consumed directly by the user back on their laptop. The Ubiquity container volume service provides seamless access to shared storage across multiple applications running in different cloud runtimes/frameworks.
• MetaOcean
MetaOcean delivers Cognitive Data Management on a petabyte scale by capturing metadata from heterogenous storage and compute environments. Metadata is collected from block, object or file storage and maintained over the data's lifetime. Custom metadata may be added or may be derived from the data itself. The information gathered from various sources is then mined in IBM Watson and Watson Deep Learning Analytics to deliver a cognitive solution to users seeking to derive value, exercise control and safeguard their data. MetaOcean provides a cognitive platform for compute and storage intensive environments, including data management, discovery, governance, workflow, provenance, compliance, audit, intrusion detection and security.
• Mission Critical Analytics for Unstructured Data
Minimizing the time to derive insight from high velocity data is the goal of Mission Critical Analytics. We are building a High Performance Storage Tier for IBM's Spectrum Scale file system utilizing Storage Class Memory in the client's computation nodes. A clustered file system, in the client nodes, allows for a global namespace, efficient sharing of data between nodes and a single point of management across all nodes. We are leveraging Spectrum Scale's performance proven design to architect a decoupled solution that eliminates the classic bottlenecks in a distributed system. We are investigating tradeoffs in POSIX semantics to reduce the file system overhead and make tradeoffs in consistency, availability and durability. The High Performance Storage Tier is intended to serve the needs of analytics and deep learning on high velocity data.
• Software-based, scale-out, high-performance storage systems
The goal of this challenging research project is the design and delivery of a software-based, scale-out, high-performance storage system. As a part of IBM Almaden's Spectrum Scale / General Parallel File System (GPFS) small research team, the research involves new approaches for a system based on networked commodity servers with fast single-port storage devices. Fundamental research challenges include IOPS performance, scalability, resiliency and ease of use.
