Системы хранения данных NetApp FAS, AFF и ASA: Обзор и архитектура

Computer storage product

A NetApp FAS is a computer storage product by NetApp running the ONTAP operating system; the terms ONTAP, AFF, ASA, FAS are often used as synonyms. "Filer" is also used as a synonym although this is not an official name. There are three types of FAS systems: Hybrid, All Flash, and All SAN Array:
NetApp proprietary custom build hardware appliances with HDD or SSD drives called hybrid Fabric Attached Storage (or simply FAS)
NetApp proprietary custom build hardware appliances with only SSD drives and optimized ONTAP for low latency called ALL Flash FAS (or simply AFF)
All SAN Array build on top of AFF platform, and provide only SAN based data protocol connectivity. ONTAP can serve storage over a network using file based protocols such as NFS and SMB, also block based protocols, such as the SCSI over the Fibre Channel Protocol on a Fibre Channel network, Fibre Channel over Ethernet (FCoE), iSCSI, and FC NVMe transport layer. ONTAP based systems that can serve both SAN and NAS protocols called Unified ONTAP, AFF systems with ASA identity called All SAN. NetApp storage systems running ONTAP implement their physical storage in large disk arrays. While most large storage systems are implemented with commodity computers with an operating system such as Microsoft Windows Server, VxWorks or tuned Linux, ONTAP based hardware appliances use highly customized hardware and the proprietary Data ONTAP operating system with WAFL file system, all originally designed by NetApp founders David Hitz and James Lau specifically for storage serving purposes. ONTAP is NetApp's internal operating system, specially optimized for storage functions at high and low levels. It boots from FreeBSD as a stand alone kernel space module and uses some functions of FreeBSD (command interpreter and drivers stack, for example). All NetApp ONTAP based hardware appliances have battery backed non volatile random access memory or NVDIMM, referred to as NVRAM or NVDIMM, which allows them to commit writes to stable storage more quickly than traditional systems with only volatile memory. Early storage systems connected to external disk enclosures via parallel SCSI, while modern models use fibre channel and SAS (Serial Attach SCSI) SCSI transport protocols. The disk enclosures (shelves) use fibre channel hard disk drives, as well as parallel ATA, serial ATA and Serial attached SCSI. Starting with AFF A800 NVRAM PCI card no longer used for NVLOGs, it was replaced with NVDIMM memory directly connected to the memory bus. Implementers often organize two storage systems in a high availability cluster with a private high speed link, either Fibre Channel, InfiniBand, 10 Gigabit Ethernet, 40 Gigabit Ethernet or 100 Gigabit Ethernet. One can additionally group such clusters together under a single namespace when running in the "cluster mode" of the Data ONTAP 8 operating system.

Modern NetApp FAS, AFF or ASA system consist of customized computers with Intel processors using PCI. Each FAS, AFF or ASA system has non volatile random access memory, called NVRAM, in the form of a proprietary PCI NVRAM adapter or NVDIMM based memory, to log all writes for performance and to play the data log forward in the event of an unplanned shutdown. One can link two storage systems together as a cluster, which NetApp (as of 2009) refers to using the less ambiguous term "Active/Active".

Each storage system model comes with a set configuration of processor, RAM, and non volatile memory, which users cannot expand after purchase. With the exception of some of the entry point storage controllers, the NetApp FAS, ASA, and AFF systems usually have at least one PCIe based slot available for additional network, tape and/or disk connections. In June 2008 NetApp announced the Performance Acceleration Module (or PAM) to optimize the performance of workloads that carry out intensive random reads. This optional card goes into a PCIe slot and provides additional memory (or cache) between the disk and the storage system cache and system memory, thus improving performance.

All Flash FAS, also known as AFF A series. Usually, AFF systems based on the same hardware as FAS but first one optimized and works only with SSD drives on the back end while the second one can use HDD and SSD as a cache: for example, AFF A700 & FAS9000, A300 & FAS8200, A200 & FAS2600, A220 & FAS2700 use the same hardware, but AFF systems do not include Flash Cache cards. Also, AFF systems do not support FlexArray with third party storage array virtualization functionality. AFF is a Unified system and can provide SAN & NAS data protocol connectivity, and in addition to traditional SAN & NAS protocols in FAS systems, AFF has block based NVMe/FC protocol for systems with 32 Gbit/s FC ports. AFF & FAS use the same firmware image, and nearly all noticeable functionality for the end user is the same for both storage systems. However, internally data is processed and handled differently in ONTAP. AFF systems, for example, use different Write Allocation algorithms as compared to FAS systems. Because AFF systems have faster underlying SSD drives, Inline data deduplication in ONTAP systems is nearly not noticeable (~2% performance impact on low end systems).

All SAN Array running ONTAP, and based on AFF platform thus inherits its features & functionalities, and data internally processed and handled the same as in AFF systems. All other ONTAP based hardware and software platforms can be referred to as Unified ONTAP meaning they can provide unified access with SAN & NAS data protocols. ONTAP architecture in ASA systems is the same as in FAS & AFF, with no changes. ASA systems using the same firmware image as AFF & FAS systems. ASA is the same as AFF, and the only difference is in the access to the storage over the network with SAN protocols: ASA provides symmetric active/active access to the block devices (LUN or NVMe namespaces), while Unified ONTAP systems continue to use ALUA and ANA for the block protocols. The original ASA platform was first announced in 2019 and re launched in May 2023.

In February 2023, NetApp introduced a new AFF product line code named the C Series. This platform uses quad level cell (QLC) NAND flash and is aimed at competing with the QLC based products already in the market from Pure Storage (specifically FlashArray//C). The C Series has higher latency than typical AFF systems, at around 2 4 milliseconds compared to 500 microseconds on AFF using triple level cell (TLC) media. However, the aim of the platform is to provide a lower price point for customers that might otherwise not choose all flash systems.

NetApp uses either SATA, Fibre Channel, SAS or SSD disk drives, which it groups into RAID (Redundant Array of Inexpensive Disks or Redundant Array of Independent Disks) groups of up to 28 (26 data disks plus 2 parity disks). NetApp FAS storage systems which contain only SSD drives with installed SSD optimized ONTAP OS called All Flash FAS (AFF).

FAS, ASA, and AFF systems are using enterprise level HDD and SSD (i. e. NVMe SSD) drives with two ports, each port connected to each controller in an HA pair. HDD and SSD drives can only be bought from NetApp and installed in NetApp's Disk Shelves for FAS/AFF platform. Some shelves, such as the D4246, can be upgraded from a 6 Gb/s Shelf into a 12 Gb/s with an IOM 12 upgrade. Physical HDD and SSD drives, partitions on them, and LUNs imported from third party arrays with FlexArray functionality considered in ONTAP as a Disk. In SDS systems like ONTAP Select & ONTAP Cloud, logical block storage like virtual disk or RDM inside ONTAP also considered as a Disk. Do not confuse the general term "disk drive" and "disk drive term used in ONTAP system" because, with ONTAP, it could be an entire physical HDD or SSD drive, a LUN, or a partition on a physical HDD or SSD drive. LUNs imported from third party arrays with FlexArray functionality in HA pair configuration must be accessible from both nodes of the HA pair. Each disk has ownership on it to show which controller owns and serves the disk. An aggregate can include only disks owned by a single node, therefore each aggregate owned by a node and any objects on top of it, as FlexVol volumes, LUNs, File Shares are served with a single controller. Each controller can have its own disks and aggregates them where both nodes can be utilized simultaneously even though they not serving the same data.

Advanced Drive Partitioning (ADP) can be used in ONTAP based systems depending on the platform and use case. ADP can be used only with native disk drives from NetApp Disk shelves, FlexArray technology does not support ADP. ADP also supported with third party drives in ONTAP Select. This technique mainly used to overcome some architectural requirements and reduce the number of disk drives in ONTAP based systems. There are three types of ADP: Root Data partitioning; Root Data Data partitioning (RD2 also known as ADPv2); Storage Pool. Root Data partitioning can be used in FAS & AFF systems to create small root partitions on drives to use them to create system root aggregates and, therefore, not to spend entire three disk drives for that purpose. In contrast, the bigger portion of the disk drive will be used for data aggregate. Root Data Data partitioning is used in AFF systems only for the same reason as Root Data partitioning with the only difference that bigger portion of the drive left after root partitioning divided equally by two additional partitions, usually, each partition assigned to one of the two controllers, therefore reducing the minimum number of drives required for an AFF system and reducing waste for expensive SSD space. Storage Pool partitioning technology used in FAS systems to equally divide each SSD drive by four pieces which later can be used for FlashPool cache acceleration, with Storage Pool only a few SSD drives can be divided by up to 4 data aggregates which will benefit from FlashCache caching technology reducing minimally required SSD drives for that technology.

NetApp Flash Pool is a feature on hybrid NetApp FAS systems that allows creating hybrid aggregate with HDD drives and SSD drives in a single data aggregate. Both HDD and SSD drives form separate RAID groups. Since SSD also used to write operations, it requires RAID redundancy contrary to Flash Cache but allows to use of different RAID types for HDD and SSD; for example, it is possible to have 20 HDD 8 TB in RAID TEC while 4 SSD in RAID DP 960 GB in a single aggregate. SSD RAID used as cache and improved performance for read write operations for FlexVol volumes on the aggregate where SSD added as the cache. Flash Pool cache similarly to Flash Cache has policies for reading operations but also includes write operations that could apply separately for each FlexVol volume located on the aggregate; therefore, it could be disabled on some volumes while others could benefit from SSD cache. Both FlashCache & FlashPool can be used simultaneously to cache data from a single FlexVol to enable an aggregate with Flash Pool technology minimum 4 SSD disks required (2 data, 1 parity, and 1 hot spare), it is also possible to use ADP technology to partition SSD into 4 pieces (Storage Pool) and distribute those pieces between two controllers so each controller will benefit from SSD cache when there is a small amount of SSD. Flash Pool is not available with FlexArray and is possible only with NetApp FAS native disk drives in NetApp's disk shelves.

FlexArray is NetApp FAS functionality allows to visualize third party storage systems and other NetApp storage systems over SAN protocols and use them instead of NetApp's disk shelves. With FlexArray functionality RAID protection must be done with third party storage array thus NetApp's RAID 4, RAID DP and RAID TEC not used in such configurations. One or many LUNs from third party arrays could be added to a single aggregate similarly to RAID 0. FlexArray is licensed feature.

NetApp Storage Encryption (NSE) is using specialized purpose build disks with low level Hardware based full disk encryption (FDE/SED) feature and also supports FIPS certified self encrypted drives, compatible nearly with all NetApp ONTAP features and protocols but does not offer MetroCluster. NSE feature does overall nearly zero performance impact on the storage system. NSE feature similarly to NetApp Volume Encryption (NVE) in storage systems running ONTAP can store encryption key locally in Onboard Key Manager or on dedicated key manager systems using KMIP protocol like IBM Security Key Lifecycle Manager and SafeNet KeySecure. NSE is data at rest encryption which means it protects only from physical disks theft and does not give an additional level of data security protection in a normal operational and running system. NetApp has passed NIST Cryptographic Module Validation Program for its NetApp CryptoMod (TPM) with ONTAP 9.2.

MetroCluster (MC) is free functionality for FAS and AFF systems for metro high availability with synchronous replication between two sites, this configuration requires additional equipment. Available in both modes: 7 mode (old OS) and Cluster Mode (or cDOT a newer version of ONTAP OS). MetroCluster in Cluster Mode known as MCC. MetroCluster uses RAID SyncMirror (RSM) and plex technique where on one site number of disks form one or more RAID groups aggregated in a plex, while on the second site have the same number of disks with the same type and RAID configuration along with Configuration Replication Service (CRS) and NVLog replication. One plex synchronously replicates to another in a compound with non volatile memory. Two plexes form an aggregate where data stored and in case of disaster on one site second site provide read write access to data. MetroCluster Support FlexArray technology. MetroCluster configurations are possible only with mid range and high end models which provide the ability to install additional network cards required to MC to function.

With MetroCluster it is possible to have one or more storage node per site to form a cluster or Clustered MetroCluster (MCC). Remote and local HA perter node must be the same model. MCC consists of two clusters each located on one of two sites. There may be only two sites. In MCC configuration each one remote and one local storage node form Metro HA or Disaster Recovery Pare (DR Pare) across two sites while two local nodes (if there is partner) form local HA pare, thus each node synchronously replicates data in non volatile memory two nodes: one remote and one local (if there is one). It is possible to utilize only one storage node on each site (two single node clusters) configured as MCC. 8 node MCC consists of two clusters 4 nodes each (2 HA pair), each storage node have only one remote partner and only one local HA partner, in such a configuration each site clusters can consist out of two different storage node models. For small distances, MetroCluster requires at least one FC VI or newer iWARP card per node. FAS and AFF systems with ONTAP software versions 9.2 and older utilize FC VI cards and for long distances require 4 dedicated Fibre Channel switches (2 on each site) and 2 FC SAS bridges per each disk shelf stack, thus minimum 4 total for 2 sites and minimum 2 dark fiber ISL links with optional DWDMs for long distances. Data volumes, LUNs and LIFs could online migrate across storage nodes in the cluster only within a single site where data originated from: it is not possible to migrate individual volumes, LUNs or LIFs using cluster capabilities across sites unless MetroCluster switchover operation is used which disable entire half of the cluster on a site and transparently to its clients and applications switch access to all of the data to another site.

Starting with ONTAP 9.3 MetroCluster over IP (MCC IP) was introduced with no need for a dedicated back end Fibre Channel switches, FC SAS bridges and dedicated dark fiber ISL which previously were needed for a MetroCluster configuration. Initially, only A700 & FAS9000 systems supported with MCC IP. MCC IP available only in 4 node configurations: 2 node Highly Available system on each site with two sites total. With ONTAP 9.4, MCC IP supports A800 system and Advanced Drive Partitioning in form of Rood Data Data (RD2) partitioning, also known as ADPv2. ADPv2 supported only on all flash systems. MCC IP configurations support single disk shelf where SSD drives partitioned in ADPv2. MetroCluster over IP require Ethernet cluster switches with installed ISL and utilize iWARP cards in each storage controller for synchronous replication. Starting with ONTAP 9.5 MCC IP supports distance up to 700 km and starts to support SVM DR feature, AFF A300, and FAS8200 systems.

NetApp storage systems using proprietary OS called ONTAP (Previously Data ONTAP). The main purpose for an Operating System in a storage system is to serve data to clients in non disruptive manner with the data protocols those clients require, and to provide additional value through features like High Availability, Disaster Recovery and data Backup. ONTAP OS provides enterprise level data management features like FlexClone, SnapMirror, SnapLock, MetroCluster etc., most of them snapshot based WAFL File System capabilities.

WAFL, as a robust versioning filesystem in NetApp's proprietary OS ONTAP, it provides snapshots, which allow end users to see earlier versions of files in the file system. Snapshots appear in a hidden directory: ~snapshot for Windows (SMB) or snapshot for Unix (NFS). Up to 1024 snapshots can be made of any traditional or flexible volume. Snapshots are read only, although ONTAP provides additional ability to make writable "virtual clones", based at "WAFL snapshots" technique, as "FlexClones". ONTAP implements snapshots by tracking changes to disk blocks between snapshot operations. It can set up snapshots in seconds because it only needs to take a copy of the root inode in the filesystem. This differs from the snapshots provided by some other storage vendors in which every block of storage has to be copied, which can take many hours.

Before the release of ONTAP 8, individual aggregate sizes were limited to a maximum of 2 TB for FAS250 models and 16 TB for all other models. The limitation on aggregate size, coupled with increasing density of disk drives, served to limit the performance of the overall system. NetApp, like most storage vendors, increases overall system performance by parallelizing disk writes to many different spindles (disk drives). Large capacity drives, therefore limit the number of spindles that can be added to a single aggregate, and therefore limit the aggregate performance. Each aggregate also incurs a storage capacity overhead of approximately 7 11%, depending on the disk type. On systems with many aggregates, this can result in lost storage capacity. However, the overhead comes about due to additional block checksumming on the disk level as well as usual file system overhead, similar to the overhead in file systems like NTFS or EXT3. Block checksumming helps to ensure that data errors at the disk drive level do not result in data loss. Data ONTAP 8.0 uses a new 64 bit aggregate format, which increases the size limit of FlexVolume to approximately 100 TB (depending on storage platform) and also increases the size limit of aggregates to more than 100 TB on newer models (depending on storage platform) thus restoring the ability to configure large spindle counts to increase performance and storage efficiency.