Object storage for Cloud Foundry apps

By Al
One of the key mandates of Cloud Native Apps is to be stateless. This allows them to be ephemeral and to scale out massively. In order to do so they must store their data outside somewhere, but where? The truth is that Cloud Native Apps love “Object storage”.


 Object storage is in way similar to File storage, but unlike File it doesn't have a directory tree. Data is organized in "buckets" which are a single level deep. These "buckets" must have a unique name. "Objects" are files that get stored along with a unique name within the bucket. Apps use this unique identifier retrieve the object. You could say that it is a key-value store where the value portion is a file.

Two factors allow object storage to grow orders of magnitude larger than file storage:
  • The simple (almost absence of) data structure
  • Metadata is stored along with the object
These mean you don't need to preallocate a table of pointers to files and directories as well their ACLs. I bet you have heard the terms FAT32 and NTFS many times. Objects and buckets typically leverage a very simple set of permissions like the one in this picture.


Object storage is very easy to consume for applications. They can use an API like S3 or Swift or in most cases by a simple URL when the app just need to retrieve an object. The image file selected in the previous screenshot could be called back from a HTML page as follows:

<img src="https://a_ppiper_blog.s3.amazonaws.com/3Dprinter-rods.jpg">

AWS S3 is the prime example of object storage. It is potentially the largest storage entity in the world and constitutes the biggest source of revenue for AWS. Its no surprise then that its API (aka S3) is used by other object storage vendors.

During the Pied Piper program we got participants to play also with ECS (Elastic Cloud Storage). This a fast growth storage product for DellEMC. It enables an S3 compatible object storage in the customer's own datacenter, with all the security and governance considerations that come along with that. Availability is another consideration especially after the 5 hour outage AWS S3 experienced in February 2017.

 ECS offers some additional features:
  • Namespaces. With multitenancy in mind, ECS implements "Namespaces" which are a sort of a tenant. This essentially turns it into a 2-level deep object storage. Bucket names must be unique only within a namespace.
  • Strong consistency. Most object stores provide only "eventual consistency", which means that two users accessing the same object in different locations might get different versions of the object if the object has been recently modified. In contrast, ECS implements "strong global consistency" by which all locations know that the object has changed even though they might not have the latest version yet. In consequence they can point the app to get the latest version from a location where it is available.
  • It uses "erasure coding" to protect object data across dispersed geographical locations instead of storing a full copy per location. This translates into $/GB figures that are below most cloud providers
  • Durability (11 x 9’s) and SEC 17-A4 compliant
Of course during the program the main focus was on learning how to use all these technologies. All our coding was done in Python. We leveraged AWS Boto library. The following repo contains the "Boto-S3.py" script which is a concise cookbook with the most common operations on S3 compatible object stores

https://github.com/cermegno/Boto-S3

Once we learned the basics we went on doing a cool exercise of creating a photo-album. If you want to practice you can access the repo here:

 https://github.com/cermegno/Photo-album

We used Cloud foundry in Pivotal Web Services for most of our exercises. This cloud offering has a marketplace where one can add services to apps. However object storage is not a current offering. In order to stay "Cloud Native" we should pass the S3 or ECS credentials using the "user provided environment variable" capability of Cloud Foundry

 There are no differences in the Boto code when dealing with ECS. However there is a small difference in how objects are called back from a URL. The difference stems from the implementation of the "namespace" we mentioned earlier.

 This is the AWS S3 method

http://bucket.s3.amazonaws.com/key

ECS offers two possibilities

http://namespace.public.myecs.com/bucket/key
http://bucket.namespace.public.myecs.com/key

Comments

Post a Comment

Popular posts from this blog

Sending PowerStore alerts via SNMP

By Al
Image
In this previous article we discussed how to leverage PowerStore's powerful API to collect audit logs and send them to a Syslog server . We used Logstash to do that because of its extensive list of plugins Sometimes I come across organizations that still have a requirement to send storage array Alerts to SNMP. As in the Syslog case, ultimately most modern monitoring tools today support collecting information via REST API because that's the way the application world is going, i.e. the cloud native way ... it would be fun to see Prometheus monitoring Kubernetes via SNMP :) However there is still some requirement out there for SNMP traps, so we will explore how to do it. For simplicity we will use the same toolset that we used in the previous article. I encourage you to revisit quickly the article as we did an extensive introduction to Logstash, with practical examples as well as a description of the logs available in PowerStore. I won't cover those aspects in this article I
Read more

Sending PowerStore logs to Syslog

By Al
Image
In this article we will explore how to get the logs from a DellEMC PowerStore array to a Syslog server. For this purpose we will use the PowerStore’s REST API, which is a great piece of engineering and a joy to work with as a developer. If you want to learn more about the PowerStore REST API I strongly recommend you quickly skim through the 2 articles I have written about the REST API. Part 1 - http://anzpiper.blogspot.com/2020/08/intro-to-dellemc-powerstore-rest-api.html Part 2 - http://anzpiper.blogspot.com/2020/08/powerstore-rest-api-best-practices.html In particular the second article demonstrates the capabilities of the REST API query language. This is a great feature I will use heavily in the last section so I strongly recommend you read that one at least As a side note, Syslog was developed in 1980. So this year it has turn 40 years old! That is a long time by any measure ... many of my colleagues were not even born in 1980. But in the technology scale it looks even scarier. 198
Read more

Electronic Nose - eNose

By Al
Image
In the previous article I provided details on VeeFarm which was one of the best projects we saw in ANZ Pied Piper 2019 edition and one of the best projects in the history of the program. The program inspired me and others to work together to showcase a great idea and the Pied Piper program itself during our presales conference. An idea I suggested to enhance VeeFarm was an electronic nose that could be used to detect the optimal ripening of the produce to advise the owner that it is the right time to pick it up the produce. This is an idea I had in the backburner for a while but never found the time to work on. Vee gave me the perfect excuse. The electronic nose, or eNose for short, is a custom PCB that hosts a range of gas sensors. For the proof of concept, I used the cheap MQ sensors. These are designed mostly for hobbyists and makers and they barely cost $1 per sensor. As one could expect, tweaking and calibrating these sensors it is even harder than the most expensive counter
Read more