We started this 6-part series on how to build web APIs by introducing the variety of architectures that have been proposed or put into use by various languages and frameworks over the years. Among the most commonly discussed architectures online is the Clean architecture, which aspires to produce a separation of concerns by subdividing a project into layers. Each layer abides by the Single Responsibility Principle, ensuring each class is only handling one part of the process, and is more easily and thoroughly unit tested.

Applying Clean to Endpoints

The Clean architecture can be used in many domains. In another blog series we describe how we applied Clean to our mobile applications. Today, we are going to talk about how we apply Clean to API endpoints. We call this the Clean API Architecture:

We have the following layers (and colors) which map to the original Clean architecture:

🔵 FRAMEWORKS & CLOUD
↓
🟢 INTERFACE ADAPTERS
↓
🔴 APPLICATION LOGIC
↓
🟠 ENTITY LOGIC
↓
🟡 DATA

None of the layers have visibility into higher layers. They may have references to their child layer, but definitely not their grandchildren

This diagram also depicts our W-shaped execution flow we described in an earlier blog post, this time represented by arrows that start both at the HTTP layer and again at the Interface Adapter layer from a Queued asynchronous job.

Now it’s time to show you the classes of our architecture.

You had me at Clean

🔵 Frameworks

Any endpoint request must be routed to the appropriate code path through a Load Balancer, Web Server, Application Server, and an API / Web Framework. (We use Sinatra for this last part, but popular frameworks include Rails, Django, Spring Boot, and others). API frameworks offer the most documentation online and is a common (the most common?) architectural structure.

Once a request reaches your API or web framework, however, patterns can diverge widely.

Frameworks like Rails employ an MVC pattern that works well for smaller projects, but have weaknesses when it comes to large, high availability APIs such as ours. Rails models and controllers get fat very quickly when applied to large APIs.

The inevitable course of a Rails model class

Consequently, everything that comes after the Frameworks + cloud layer is (mostly) novel and inspired by the Clean architecture. We will be making the case for our architecture and our decisions in the remainder of this series.

Supporting classes

In any given layer, we will have one or more supporting classes. These are classes that are going to be single-purpose and have no references to other layers above or below them. They help with code re-use and duplication, and enable us to avoid writing complex god classes in a given layer.

We include cloud services as supporting classes of our Framework layer. AWS services like EC2, SQS, RDS and ElastiCache are supporting classes — NOT “inner” or central layer — because, as others have pointed out, the UI and the database depend on the business rules, but the business rules don’t depend on the UI or database.

🟢 Interface adapters

Once a request comes in via our framework, a Controller orchestrates the processing of the endpoint by invoking a Request object to extract each parameter, validate its syntax, and authenticate the user making the request.

Controllers are the first place our application code is introduced. Controllers instantiate our classes and move data between classes in the 🔴 Application Logic layer.

It’s important to note that Controllers are not the only orchestration object in the Interface adapter layer. We also have Jobs, which are used in our asynchronous queue processing layer (more to come).

Supporting classes

Controllers depend on classes including Validators, which check the syntax of incoming data, and Presenters, which format outgoing data, and Response objects, which map objects and/or hashes into JSON, HAML and other formats.

Socket relay classes communicate state changes to the client over a socket communication channel, such as Websockets.

Request classes are typed data structures that bring together the necessary components for the request being made. This is different from standard HTTP requests, which (assuming CGI) are made up of key/value string pairs.

Response classes are like renderers in Rails, and enable you to return HAML, JSON or other types.

Parameter extractors extract data out of the params hash and converts it to properly typed values, such as ints, floats and strings.

🔴 Application Logic

GET request made to read endpoints are next passed to the Application Logic layer where a Service ensures the validity of the inputs, makes sure the user is authorized to access data, and then retrieves data from the Entity Logic Layer through a Repo (for databases) and/or Adapter (for APIs). Service objects return Result objects as defined by dry-monads.

POST, DELETE and PUT requests made to write endpoints do the same thing as read endpoints, but defer processing by enqueuing Service inputs through our queue — Amazon SQS — and write the data to the Entity Logic Layer through a Job or Service.

Jobs are used to orchestrate side effects, such as sending a socket message through a Relay after a data mutation completes

Supporting classes

In our implementation of the Clean API architecture, the Service class itself assembles a distinct collection of Validator classes to provide an additional layer of semantic validation for a request. Thus, we have two layers of validation — syntactic, happening via the Request layer, and semantic, happening via the Service .

🟠 Entity logic

Entity logic refers to components that are common not only to this endpoint, but others as well. Repositoryclasses, which provide us access to persistent stores like Mysql or Postgres databases, and Adapter classes, which provide us access to APIs, including AWS storage apis like S3, ElastiCache and others. We expect classes in this layer to be used over and over again; classes in the layer above are often single-purpose to an endpoint.

🟡 Data

This is ideally a very simple layer that provides an actual interface into our different storage systems. If you use Rails you will likely be receiving ActiveRecord objects; APIs are ideally returning ruby structs.

Testing

In other domains — such as Android or iOS development — we have created interfaces to our data storage layer. We use dependency injection so that, when we run tests, we are doing things like creating mocked, in-memory versions of SQLite data storage, rather than using real filesystem-backed data storage systems.

On our webserver, because of the dynamic nature of ruby, we use stub_const to overwrite singleton cloud services with mocked versions, or we will point singleton cloud services to local docker containers running Redis, Memcached, Mysql, etc.

Storage systems that power the Data layer, such as Mysql and Postgres, that are implicitly at the bottom of the diagram are very likely going to be process-wide singletons that are ideally injected or mocked. ActiveRecord maintains its connection pool; systems Redis and Memcached will also likely need some kind of global pool or singleton managing access.

Stateless HTTP-based APIs, such as S3, DynamoDb, and others, are typically going to be mocked by instance_doubles or overridden connection parameters that point to local mocks.

🙄 Isn’t this overkill?

We promise this won’t lead to another one of these.

We know what you’re thinking. Isn’t this just another example of extra engineering and complexity that I don’t need, and that maybe you don’t need either?

Let’s explore this for a minute or two. Imagine the world’s simplest API — adding a favorite for a profile. This is what it might look like in a simple endpoint:

In fact, this endpoint is implicitly relying on each of the layers we have defined above. Here is how:

The first line, post ‘favorite’ do, is hook into Sinatra 🔵 framework, as you might expect.

The last line is a form of presentation logic that is part of our 🟢 Interface Adapter layer. It is simply a 200 response with no body.

What about the other layers in our Clean Api architecture? Can we see echos of those in this 6-line snippet?

🟢 Request

Params are passed directly without any extraction, and together the two symbols — :target_id and :creator_id — form an implied Request, comprising a target_id and creator_id.

🟢 Controller

Because of the lack of any validation or presentation customizability, there is no need for a controller so there is no equivalent in this code.

🔴 Service

The where clause takes your request — a target_id and creator_id — and finds a domain object — a Favorite.

🟠 Entity Logic

The first_or_create provides the Entity logic — interacting with persistent storage via a special ActiveRecord method

🟡 Data

The Favorite model is equivalent to your Data layer, providing a definition for the object being used.