MongoDB in Cloud Foundry

This is not the first time we discuss NoSQL databases in this blog. In previous posts we have seen how NoSQL databases differ from traditional SQL databases. NoSQL is a growing market forecasted to reach $5 billion this year and within that category MongoDB is the most popular and successful database. According to the latest rankings MongoDB is the 5th database most popular overall just trailing PostgreSQL and 3x more popular than other famous NoSQL databases like Redis, Cassandra and ElasticSearch. The MongoDB company itself had a successful IPO in October 2017

During our team project we decided to use MongoDB because it has a rich query language that allowed us to implement the functionality we were looking for. MongoDB stores BSON (Binary JSON) documents. This is what they look like:

MongoDB is open source and its architecture allows it to scale horizontally in a load balanced fashion by adding more servers to reach very high performance. It also supports indexing for faster queries.

From a physical point of view most MongoDB implementations rely on DAS (Direct Attached Storage) which works well for small deployments but makes it incredibly difficult to manage performance, availability and capacity as the cluster grows. As part of my job I see customers typically starting with DAS and moving to a scale-out SAN like ScaleIO as their needs grow.

For our project we had only 7 weeks and we were going to be in production for less than a week, so it didn't make any sense to build our own infrastructure. Instead we decided to leverage MongoDB service providers. I will discuss in detail our choice of provider and the operational lessons learned in a future article.

If you are interested in learning more about MongoDB from a DBA point of view there are tons of tutorials out there. However if you are interested in talking to MongoDB with Python you could visit the following Github repo we used as part of the Pied Piper program:

https://github.com/cermegno/Mongo-lesson

In the repo you will find a cookbook script called "mongo-test.py". The script will teach you over a very simple example how to Insert, Retrieve, Update and Delete documents. Before you get it to work you will have to install PyMongo:

pip install pymongo

Also before running the cookbook script make sure you adjust the connection string at the top to reflect the details of your database. I personally like to run a virtual machine on VirtualBox with the MongoDB port forwarded to the localhost. This allows me to keep working offline or when I am sitting behind a firewall that blocks MongoDB ports.

However sooner or later I want to run my code in Cloud Foundry. The best practice for Cloud Native apps is to read the details of service connections through environment variables. In particular when you bind a service from the Cloud Foundry MarketPlace to your app the connection string is added to VCAP_SERVICES as follows:

{
  "system_env_json": {
    "VCAP_SERVICES": {
      "mlab": [
        {
          "name": "MyMongo",
          "instance_name": "MyMongo",
          "credentials": {
            "uri": "mongodb://username:password@servername:port/DatabaseName"

Based on the above I like to add these lines to the beginning of my code so that it works seamlessly whether running locally or in the cloud:

if 'VCAP_SERVICES' in os.environ:
    VCAP_SERVICES = json.loads(os.environ['VCAP_SERVICES'])
    CREDENTIALS = VCAP_SERVICES["rediscloud"][0]["credentials"]
    MONCRED = VCAP_SERVICES["mlab"][0]["credentials"]
    client = MongoClient(MONCRED["uri"])
    DB_NAME = str(MONCRED["uri"].split("/")[-1])
    
else:
    client = MongoClient('127.0.0.1:27017')
    DB_NAME = "FoodBlog"

Please note how in the cloud code the database name is given to you by the provider, whereas in the local example you can run multiple databases in your local MongoDB so you get to name them what you like.

Finally I personally find very useful to have graphical tool to do some basic troubleshooting. The best tool I have found is called Robo3T, which is free to download