Мазмұнға сәйкес сақтау механизмі

Type of data storage mechanism

Content addressable storage (CAS), also referred to as content addressed storage or fixed content storage, is a way to store information so it can be retrieved based on its content, not its name or location. It has been used for high speed storage and retrieval of fixed content, such as documents stored for compliance with government regulations. Content addressable storage is similar to content addressable memory. CAS systems work by passing the content of the file through a cryptographic hash function to generate a unique key, the "content address". The file system's directory stores these addresses and a pointer to the physical storage of the content. Because an attempt to store the same file will generate the same key, CAS systems ensure that the files within them are unique, and because changing the file will result in a new key, CAS systems provide assurance that the file is unchanged. CAS became a significant market during the 2000s, especially after the introduction of the 2002 Sarbanes–Oxley Act in the United States which required the storage of enormous numbers of documents for long periods and retrieved only rarely. Ever increasing performance of traditional file systems and new software systems have eroded the value of legacy CAS systems, which have become increasingly rare after roughly 2018. However, the principles of content addressability continue to be of great interest to computer scientists, and form the core of numerous emerging technologies, such as peer to peer file sharing, cryptocurrencies, and distributed computing.

Traditional file systems generally track files based on their filename. On random access media like a floppy disk, this is accomplished using a directory that consists of some sort of list of filenames and pointers to the data. The pointers refer to a physical location on the disk, normally using disk sectors. On more modern systems and larger formats like hard drives, the directory is itself split into many subdirectories, each tracking a subset of the overall collection of files. Subdirectories are themselves represented as files in a parent directory, producing a hierarchy or tree like organization. The series of directories leading to a particular file is known as a "path". In the context of CAS, these traditional approaches are referred to as "location addressed", as each file is represented by a list of one or more locations, the path and filename, on the physical storage. In these systems, the same file with two different names will be stored as two files on disk and thus have two addresses. The same is true if the same file, even with the same name, is stored in more than one location in the directory hierarchy. This makes them less than ideal for a digital archive, where any unique information should only be stored once.

Although location based storage is widely used in many fields, this was not always the case. Previously, the most common way to retrieve data from a large collection was to use some sort of identifier based on the content of the document. For instance, the ISBN system is used to generate a unique number for every book. If one performs a web search for "ISBN 0465048994", one will be provided with a list of locations for the book Why Information Grows on the topic of information storage. Although many locations will be returned, they all refer to the same work, and the user can then pick whichever location is most appropriate. Additionally, if any one of these locations changes or disappears, the content can be found at any of the other locations. Because the keys are not human readable, CAS systems implement a second type of directory that stores metadata that will help users find a document. These almost always include a filename, allowing the classic name based retrieval to be used. But the directory will also include fields for common identification systems like ISBN or ISSN codes, user provided keywords, time and date stamps, and full text search indexes. Users can search these directories and retrieve a key, which can then be used to retrieve the actual document. The timing was perfect; the introduction of the Sarbanes–Oxley Act in 2002 required companies to store huge amounts of documentation for extended periods and required them to do so in a fashion that ensured they were not edited after the fact. A number of similar products soon appeared from other large system vendors. In mid 2004, the industry group SNIA began working with a number of CAS providers to create standard behavior and interoperability guidelines for CAS systems. In addition to CAS, a number of similar products emerged that added CAS like capabilities to existing products; notable among these was IBM Tivoli Storage Manager. The rise of cloud computing and the associated elastic cloud storage systems like Amazon S3 further diluted the value of dedicated CAS systems. Dell purchased EMC in 2016 and stopped sales of the original Centera in 2018 in favor of their elastic storage product. CAS was not associated with peer to peer applications until the 2000s, when rapidly proliferating Internet access in homes and businesses led to a large number of computer users who wanted to swap files, originally doing so on centrally managed services like Napster. However, an injunction against Napster prompted the independent development of file sharing services such as BitTorrent, which could not be centrally shut down. In order to function without a central federating server, these services rely heavily on CAS to enforce the faithful copying and easy querying of unique files. At the same time, the growth of the open source software movement in the 2000s led to the rapid proliferation of CAS based services such as Git, a version control system that uses numerous cryptographic functions such as Merkle trees to enforce data integrity between users and allow for multiple versions of files with minimal disk and network usage. Around this time, individual users of public key cryptography used CAS to store their public keys on systems such as key servers. The rise of mobile computing and high capacity mobile broadband networks in the 2010s, coupled with increasing reliance on web applications for everyday computing tasks, strained the existing location addressed client–server model commonplace among Internet services, leading to an accelerated pace of link rot and an increased reliance on centralized cloud hosting. Furthermore, growing concerns about the centralization of computing power in the hands of large technology companies, potential monopoly power abuses, and privacy concerns led to a number of projects created with the goal of creating more decentralized systems. Bitcoin uses CAS and public/private key pairs to manage wallet addresses, as do most other cryptocurrencies. IPFS uses CAS to identify and address communally hosted files on its network. Numerous other peer to peer systems designed to run on smartphones, which often access the Internet from varying locations, utilize CAS to store and access user data for both convenience and data privacy purposes, such as secure instant messaging.

The Centera CAS system consists of a series of networked nodes (typically large servers running Linux), divided between storage nodes and access nodes. The access nodes maintain a synchronized directory of content addresses, and the corresponding storage node where each address can be found. When a new data element, or blob, is added, the device calculates a hash of the content and returns this hash as the blob's content address. As mentioned above, the hash is searched to verify that identical content is not already present. If the content already exists, the device does not need to perform any additional steps; the content address already points to the proper content. Otherwise, the data is passed off to a storage node and written to the physical media. When a content address is provided to the device, it first queries the directory for the physical location of the specified content address. The information is then retrieved from a storage node, and the actual hash of the data recomputed and verified. Once this is complete, the device can supply the requested data to the client. Within the Centera system, each content address actually represents a number of distinct data blobs, as well as optional metadata. Whenever a client adds an additional blob to an existing content block, the system recomputes the content address. To provide additional data security, the Centera access nodes, when no read or write operation is in progress, constantly communicate with the storage nodes, checking the presence of at least two copies of each blob as well as their integrity. Additionally, they can be configured to exchange data with a different, e. g., off site, Centera system, thereby strengthening the precautions against accidental data loss. IBM has another flavor of CAS which can be software based, Tivoli Storage manager 5.3, or hardware based, the IBM DR550. The architecture is different in that it is based on hierarchical storage management (HSM) design which provides some additional flexibility such as being able to support not only WORM disk but WORM tape and the migration of data from WORM disk to WORM tape and vice versa. This provides for additional flexibility in disaster recovery situations as well as the ability to reduce storage costs by moving data off the disk to tape. Another typical implementation is iCAS from iTernity. The concept of iCAS is based on containers. Each container is addressed by its hash value. A container holds different numbers of fixed content documents. The container is not changeable, and the hash value is fixed after the write process.

Venti: One of the first content addressed storage servers, originally developed for Plan 9 from Bell Labs and is now also available for Unix like systems as part of Plan 9 from User Space. The first step towards an open source CAS+ implementation is Twisted Storage. Tahoe Least Authority File Store: an open source implementation of CAS. Git: a userspace CAS filesystem. Git is primarily used as a source code control system. git annex: a distributed file synchronization system that uses content addressable storage for files it manages. It relies on Git and symbolic links to index their filesystem location. Project Honeycomb: an open source API for CAS systems. XAM: an interface developed under the auspices of the Storage Networking Industry Association. It provides a standard interface for archiving CAS (and CAS like) products and projects. Perkeep: a 2011 project to bring the advantages of content addressable storage "to the masses". It is intended to be used for a wide variety of use cases, including distributed backup, a snapshotted by default, a version controlled filesystem, and decentralized, permission controlled filesharing. Irmin: an OCaml "library for persistent stores with built in snapshot, branching and reverting mechanisms"; the same design principles as Git. Cassette: an open source CAS implementation for C#/. NET. Arvados Keep: an open source content addressable distributed storage system. It is designed for large scale, computationally intensive data science work such as storing and processing genomic data. Infinit: a content addressable and decentralized (peer to peer) storage platform that was acquired by Docker Inc.
InterPlanetary File System (IPFS): a content addressable, peer to peer hypermedia distribution protocol. casync: a Linux software utility by Lennart Poettering to distribute frequently updated file system images over the Internet.