Many internally distributed storage systems – such as Amazon’s S3 service or Riak’s key-value architecture – have similarities in the manner in which data is labelled and subsequently retrieved. This is often because the systems themselves use a distributed hash table or a similar distribution algorithm to disperse and then later find the data they store.
OFS is a python library that seeks to capitalise on their similarities – providing a single, general API to put and get files from one of these services while hiding the specifics of the implementation from the user. This allows for local testing and development before transitioning to using one of the cloud services, services which typically cost real money and slows down testing due to the necessity of communicating with these services over an internet connection.
Characteristics of OFS:
- Uses a ‘bucket/label’ mechanism to identify individual files
- Provides a list of content in a given bucket (as best as that the service can provide)
- Provides per-file metadata in so far as the service can provide (key-value or JSON encode-able data)
- Current backend plugins:
- Local storage – based on the pairtree specification that optimises file-distribution across a native file-system to handle large quantities of files. Uses JSON to encode arbitrary metadata about the files in a given bucket.
- Remote storage (S3 and Archive plugins written by Friedrich Lindenberg (pudo) who has also made large contributions to the codebase):
- Amazon S3
- Riak (in progress)
- Also in progress – a REST Client by Friedrich Lindenberg (pudo)
- One key desire is to provide opaque sharding – breaking up very large files to spread across buckets or even systems to improve performance and the range of services or backend systems OFS can make use of.
It is plain that having the ability to write storage code in a common way, but make use of local as well as remote ‘cloud’ storage is of a great benefit. It encourages file storage to be codified in a distribute-able manner so that scaling later on is easier.
This is a work in progress, but the local implementation is intended to be both a reference implementation as well as useful testing or even production backend for storage. Other backends potentially will have less comprehensive metadata support for individual files, but these ‘limits’ will be included as optional warnings or exceptions once we have a handle on what they are.
Please comment or give feedback on this library. Also, we would welcome any patches for other backend support to the library!