• 4 min read

Announcing Cage: Develop and deploy complex Docker applications

Announcing Cage: Develop and deploy complex Docker applications

cage

This post is part of our devops series.

At Faraday, we rely heavily on microservices to analyze data, generate maps and make predictions. All our microservices run inside Docker containers, which makes it easy to run our code either locally or in the cloud.

But if you've ever worked on a large project with lots of services, you're aware that local development can be difficult:

  • You need some way to run a complex set of microservices locally on your laptop.
  • You need to be able access the source code for multiple projects easily, edit it locally, and see the changes immediately.
  • You need to remember how to run tests for services written in multiple languages by multiple teams.

Originally, we used docker-compose to work on our services locally. It offered some great features, but it didn't quite do enough:

  • We very quickly wound up needing multiple docker-compose.yml files. For example, you're encouraged to use them to define "task" containers. They're also helpful if you need to selectively restart individual portions of your application or if you use multiple load balancers.
  • docker-compose.yml files often contain a fair bit of duplication. There are ways to reduce this using extends: and env_file:, but it still requires manual maintenance.
  • docker-compose provides limited support for working with multiple source repositories.

But what if there were a tool that made complex microservice projects as simple and easy as a Rails web application? We decided to build some tools and see how simple we could make it.

Introducing Cage

Cage is an open source tool that wraps around docker-compose, and it tries to make local development as easy as possible.

We can get started by using cage to generate a new project:

cage new myproj
cd myproj

Next, we can start up our database server and create a new database. This part should be familiar to docker-compose users:

cage up db
cage run rake db:create
cage run rake db:migrate

Once the database is set up, we can start the rest of the application:

cage up

This should make a web application available at http://localhost:3000/.

Diving deeper

If we open up pods/frontend.yml, we'll see a standard docker-compose.yml file:

version: "2"
services:
  web:
    image: "faraday/rails_hello"
    build: "https://github.com/faradayio/rails_hello.git"
    ports:
    - "3000:3000"
    labels:
      io.fdy.cage.srcdir: "/usr/src/app"
      io.fdy.cage.shell: "bash"
      io.fdy.cage.test: "bundle exec rake"

We see that frontend.yml defines a single web service using the faraday/rails_hello image, with source code available from https://github.com/faradayio/rails_hello.git. (There are also some labels that we'll explain later.)

Let's get this source code and make a change! First, we need to "mount" the source code into our service, and restart the app:

cage source mount rails_hello
cage up

This will clone a copy of the rails_hello source code in src/rails_hello, and mount it into our web service in the directory specified by io.fdy.cage.srcdir above. So we can just go ahead and create an HTML file at src/rails_hello/public/index.html:

<html>
  <head><title>Sample page</title></head>
  <body><h1>Sample page</h1></body>
</html>

If we go back to http://localhost:3000/ and reload, we should see our new page!

Testing and shell access

One challenge on large microservice projects is remembering how to test other people's code! We specified io.fdy.cage.test above, which specifies how to run tests for our web service. We can invoke this as:

cage test web

If we have other services written in other languages, we could also test them using cage test $SERVICE_NAME.

Similarly, if we want to get command-line access to our web service, we can run:

cage shell web

How we built cage

cage is a single binary with no dependencies. It's written in Rust and the Linux version links against musl-libc, so you should be able to install it on any modern Linux distribution using cp.

cage relies heavily on the compose_yml library, which provides a typesafe API for working with the complex data structures in a docker-compose.yml file.

Internally, cage is structured a bit like a multi-pass compiler. In this case, the intermediate language would be docker-compose.yml files, and various transformation plugins each transform the files in some way.

Rust has been a great language for this project:

  • Rust allows us to build fast, standalone binaries.
  • Refactoring Rust code is a joy, because the compiler can catch so much.
  • The cargo build tool and the crates ecosystem is great.
  • Rust's type system allows us keep careful track of exactly what's in a docker-compose.yml file, which fields are optional, and which fields require shell variable interpolation. (It's far more complex than it looks.) Without a strong type system, it would be very easy to overlook an important case when writing a transformation plugin.

Getting started

Cage is still extremely new, and few people outside of Faraday have ever used it! So we encourage you to contact us and to ask us questions.

We're interested in hearing about what works, what doesn't, and what's too confusing. We're also interested in ideas for new features to simplify your development workflow.