Rails on Kubernetes with TLS
source link: https://willschenk.com/articles/2021/rails_on_kubernetes_with_tls/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Published July 16, 2021 #rails, #kuberenetes, #terraform, #github
I wanted to see how to really use kubernetes like I'm used to using heroku, so lets recreate everything using terraform, digital ocean, kubernetes and MAGIC!
Sample rails app
Build image
First thing we'll do is to create a docker image that we'll use to build our rails app.
Dockerfile.build
:
FROM ruby:3.0.1
WORKDIR /app
# nodejs and yarn and cloc
RUN curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN curl -sL https://deb.nodesource.com/setup_16.x | bash -
RUN apt-get update && apt-get install -y nodejs yarn
# install bundler
RUN gem install bundler:2.1.4 && gem install rails
CMD bash
Now we can build and run this image to generate our application:
docker build . -f Dockerfile.dev -t railsdev
docker run --rm -it -v $(pwd):/app railsdev
Once you are inside the image, create a new rails app:
rails new favoriteapp -d=postgresql
Then quit out of it.
Developing the app
Now inside of the rails app, we'll create a Dockerfile.dev
that will
let us develop the app:
FROM ruby:3.0.1
WORKDIR /app
# nodejs and yarn and cloc
RUN curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN curl -sL https://deb.nodesource.com/setup_16.x | bash -
RUN apt-get update && apt-get install -y nodejs yarn
# install bundler
RUN gem install bundler:2.1.4
# Bundle gems
COPY Gemfile* /app/
RUN bundle install
# Install node stuff
COPY package.json yarn.lock /app/
RUN yarn install --check-files
COPY . /app/
EXPOSE 3000
CMD rm -f tmp/pids/server.pid;bundle exec rails server -b 0.0.0.0
Now we need to create a docker-compose.yml
to set up the environment.
version: "3.7"
services:
postgres:
image: postgres:13.1
environment:
POSTGRES_PASSWORD: awesome_password
ports:
- "5432:5432"
pgadmin:
image: dpage/pgadmin4:5.4
environment:
PGADMIN_DEFAULT_EMAIL: [email protected]
PGADMIN_DEFAULT_PASSWORD: SuperSecret
GUNICORN_ACCESS_LOGFILE: /dev/null
ports:
- "4000:80"
depends_on:
- postgres
favoriteapp:
build:
context: .
dockerfile: Dockerfile.dev
depends_on:
- postgres
- redis
volumes:
- type: bind
source: ./
target: /app
ports:
- "3000:3000"
environment:
- DATABASE_URL=postgresql://postgres:awesome_password@postgres:5432/favoriteapp?encoding=utf8&pool=5&timeout=5000
- REDIS_URL=redis://redis:6379/0
- RAILS_ENV=development
sidekiq:
build:
context: .
dockerfile: Dockerfile.dev
command: bundle exec sidekiq
depends_on:
- postgres
- redis
environment:
- DATABASE_URL=postgresql://postgres:awesome_password@postgres:5432/favoriteapp?encoding=utf8&pool=5&timeout=5000
- REDIS_URL=redis://redis:6379/0
- RAILS_ENV=development
redis:
image:
redis:
image: redis:6.0.9
ports:
- '6379:6379'
And a nice little .dockerignore
file:
# node_modules
tmp
Now we start it up:
docker-compose up --build
And then we need to create the database:
docker-compose run --rm favoriteapp rails db:migrate
Develop the app
We're going to do some basic stuff here that shows
How to connect to a database
How to connect to redis
How to deploy sidekiq
Scaffold
Then lets create a scaffold for a database object:
docker-compose run --rm favoriteapp rails g scaffold messages body:string processed:boolean
docker-compose run --rm favoriteapp rake db:setup
docker-compose run --rm favoriteapp rake db:migrate
Sidekiq
docker-compose run --rm favoriteapp bundle add sidekiq
Lets turn on the :sidekiq
adapter in config/application.rb
:
class Application < Rails::Application
# ...
config.active_job.queue_adapter = :sidekiq
end
Then lets create a simple job that will process the message.
docker-compose run --rm favoriteapp rails g job process_message
And the job itself app/jobs/process_message_job.rb
:
class ProcessMessageJob < ApplicationJob
queue_as :default
def perform(job)
logger.info "Processing message #{job.id}"
m = Message.find( job.id )
m.processed = true
m.save
end
end
Then we schedule it in app/controllers/messages_controller.rb
, inside
of the create
method:
if @message.save
ProcessMessageJob.perform_later @message
Finally we add the routes in config/routes.rb
:
require 'sidekiq/web'
Rails.application.routes.draw do
mount Sidekiq::Web => "/sidekiq" # mount Sidekiq::Web in your Rails app
resources :messages
root "messages#index"
end
Testing
docker-compose up --build
Now you can visit http://localhost:3000 to see your working rails app.
Add a message, you will see that it's processed = false
, and when you
go back to the index sidekiq should have processed in the message in
the background.
Production Image
Now that we've "developed" our application locally, lets spin it up and deploy it.
Then we need a Dockerfile
to build the thing. Lets create a
Dockerfile.prod
to make it happen.
FROM ruby:3.0.1
WORKDIR /app
# nodejs and yarn and cloc
RUN curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN curl -sL https://deb.nodesource.com/setup_16.x | bash -
RUN apt-get update && apt-get install -y nodejs yarn
# install bundler
RUN gem install bundler:2.1.4
# Set up environment
RUN bundle config set without 'development test'
ENV RAILS_ENV production
ENV RAILS_SERVE_STATIC_FILES true
ENV RAILS_LOG_TO_STDOUT true
# Bundle gems
COPY Gemfile* /app/
RUN bundle install
# Install node stuff
COPY package.json yarn.lock /app/
RUN yarn install --check-files
COPY . /app/
#RUN yarn install --check-files
ARG RAILS_MASTER_KEY
RUN bundle exec rake assets:precompile
EXPOSE 3000
CMD rm -f tmp/pids/server.pid;bundle exec rails server -b 0.0.0.0
Then build the container
docker build . -f Dockerfile.prod -t wschenk/favoriteapp --build-arg RAILS_MASTER_KEY=$(cat config/master.key)
Setting up continious integration
But we don't want to do that all by hand, so lets setup github actions to build and push to dockerhub.
First create a new repository on github. Once you have that add the
remote to the favoriteapp
local git repository.
Now we need to add some secrets and environment variables.
First go to your docker hub security page and create a new access token. Copy this.
Then go to the settings on your github repo, and add the secrets:
DOCKERHUB_TOKEN
the copied tokenDOCKERHUB_USERNAME
Your usernameRAILS_MASTER_KEY
what's in config/master.key
Then we need to create a .github/workflows/build-and-push.yaml
file
that tells GitHub what to do:
name: Build and Push
on: push
jobs:
docker:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
-
name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
-
name: Docker meta
id: meta
uses: docker/metadata-action@v3
with:
images: wschenk/favoriteapp
-
name: Build and push
id: docker_push
uses: docker/build-push-action@v2
with:
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
file: Dockerfile.prod
build-args: RAILS_MASTER_KEY=${{ secrets.RAILS_MASTER_KEY }}
If everything goes well, this will all be pushed to docker hub and we are ready to begin building out the infrastructure.
Note that by default these images are public.
Terraform: Provision the infrastructure
Now that we have a working application that's packaged up in a docker
container, lets define the infrastructure that we will deploy it on.
We are going to use terraform to provision a kubernetes cluster and
postgres cluster on digital ocean, and then inside that cluster we
will setup a deployment
of our application, a job
to run the database
migrations, with a service
and ingress
to present it to the outside
world. We'll use helm
(as part of terraform) to install a redis
instance, cert-manager
to handle certificates, and nginx-ingress
on
the cluster to expose the application.
Finally we will use dnssimple
to make sure that our application has a
name.
The providers
We need tokens from digital ocean and dnsimple (if that's the provider you use, it's easy to swap out for something else.)
The section basically defines the terraform plugins that we will use to provision the platform.
terraform {
required_providers {
digitalocean = {
source = "digitalocean/digitalocean"
version = "~> 2.0"
}
dnsimple = {
source = "dnsimple/dnsimple"
}
}
}
provider "digitalocean" {
token = var.do_token
}
provider "dnsimple" {
token = var.dnsimple_token
account = var.dnsimple_account_id
}
variable "do_token" {
description = "digitalocean access token"
type = string
}
variable "dnsimple_token" {
description = "dnssimple api access token"
}
variable "dnsimple_account_id" {
description = "dnsimple account id"
}
variable "dnsimple_domain" {
description = "dnsimple domain"
}
Cluster
Now we can define the cluster itself.
digitalocean_kuberenetes_cluster
defines the kubernetes cluster
itself, and here we are creating a 3 node cluster.
We also define the kubernetes
and helm
terraform providers here, using
the host
and certificates
that we get from the digitalocean provider.
resource "digitalocean_kubernetes_cluster" "gratitude" {
name = "gratitude"
region = "nyc1"
version = "1.21.2-do.2" # or "latest"
node_pool {
name = "worker-pool"
size = "s-2vcpu-2gb"
node_count = 3
}
}
output "cluster-id" {
value = "${digitalocean_kubernetes_cluster.gratitude.id}"
}
provider "kubernetes" {
host = digitalocean_kubernetes_cluster.gratitude.endpoint
token = digitalocean_kubernetes_cluster.gratitude.kube_config[0].token
cluster_ca_certificate = base64decode(
digitalocean_kubernetes_cluster.gratitude.kube_config[0].cluster_ca_certificate
)
}
provider "helm" {
kubernetes {
host = digitalocean_kubernetes_cluster.gratitude.endpoint
cluster_ca_certificate = base64decode( digitalocean_kubernetes_cluster.gratitude.kube_config[0].cluster_ca_certificate )
token = digitalocean_kubernetes_cluster.gratitude.kube_config[0].token
}
}
Datastores
We are going to setup 2 different datastores, one is a
digitalocean_database_cluster
of postgres with one node, and the other
is redis running on the cluster that we defined (in standalone
). We
are using the bitnami redis helm chart.
I'm also setting a password on the redis instance as an example of how to do this. It's only accessible from within the cluster so I'm not sure it's strictly needed but it can't hurt.
resource "random_password" "redis_password" {
length = 16
special = false
}
resource "helm_release" "redis" {
repository = "https://charts.bitnami.com/bitnami"
chart = "redis"
name = "redis"
set {
name = "auth.password"
value = random_password.redis_password.result
}
set {
name = "architecture"
value = "standalone"
}
}
resource "kubernetes_secret" "redispassword" {
metadata {
name = "redispassword"
}
data = {
password = random_password.redis_password.result
}
}
resource "digitalocean_database_cluster" "favoriteapp-postgres" {
name = "favoriteapp-postgres-cluster"
engine = "pg"
version = "11"
size = "db-s-1vcpu-1gb"
region = "nyc1"
node_count = 1
}
Ingress Controller
We are installing the ingress-nginx
controller here, again using helm.
This will setup the digital ocean load balanacer. The data
terraform
block is there to expose the ip address of the load balancer, which we
will use to setup the DNS name.
resource "helm_release" "ingress-nginx" {
name = "ingress-nginx"
repository = "https://kubernetes.github.io/ingress-nginx"
chart = "ingress-nginx"
}
data "kubernetes_service" "ingress-nginx" {
depends_on = [ helm_release.ingress-nginx ]
metadata {
name = "ingress-nginx-controller"
}
}
output "cluster-ip" {
value = data.kubernetes_service.ingress-nginx.status.0.load_balancer.0.ingress.0.ip
#value = data.kubernetes_service.ingress-nginx.external_ips
}
I use dnsimple for my domain, and I'm calling this site k8
. Why not.
resource "dnsimple_record" "k8" {
domain = var.dnsimple_domain
name = "k8"
value = data.kubernetes_service.ingress-nginx.status.0.load_balancer.0.ingress.0.ip
type = "A"
ttl = 300
}
Cert Manager
cert-manager
keeps track of certificates as a custom resource within
kubernetes. We will use this to get our TLS traffic good to go.
resource "helm_release" "cert-manager" {
repository = "https://charts.jetstack.io"
chart = "cert-manager"
name = "cert-manager"
namespace = "cert-manager"
create_namespace = true
set {
name = "installCRDs"
value = "true"
}
}
Config
Finally, we are going to stick the data that we just got from creating these endpoints into a kubernetes config map that our application will use to wire itself up.
We also create a namespace for all of our app stuff just to keep things organized.
# We use this for the rails master key, adjust to your location
data "local_file" "masterkey" {
filename = "favoriteapp/config/master.key"
}
resource "kubernetes_namespace" "favoriteapp" {
metadata {
name = "favoriteapp"
}
}
resource "kubernetes_config_map" "favoriteapp-config" {
metadata {
name = "favoriteapp-config"
namespace = "favoriteapp"
}
data = {
RAILS_MASTER_KEY = data.local_file.masterkey.content
RAILS_ENV = "production"
DATABASE_URL = digitalocean_database_cluster.favoriteapp-postgres.private_uri
REDIS_URL = "redis://user:${random_password.redis_password.result}@redis-master.default.svc.cluster.local:6379"
}
}
Option 1: ClusterIssuer
custom resource definition
I had some trouble with putting adding this resource before the cluster has started, hopefully they've fixed it in a later release. But in the meantime you may want to only add this file after everything is up.
provider "kubernetes-alpha" {
host = digitalocean_kubernetes_cluster.gratitude.endpoint
token = digitalocean_kubernetes_cluster.gratitude.kube_config[0].token
cluster_ca_certificate = base64decode(
digitalocean_kubernetes_cluster.gratitude.kube_config[0].cluster_ca_certificate
)
}
resource "kubernetes_manifest" "cluster_issuer" {
depends_on = [ digitalocean_kubernetes_cluster.gratitude, helm_release.cert-manager ]
provider = kubernetes-alpha
manifest = {
apiVersion = "cert-manager.io/v1"
kind = "ClusterIssuer"
metadata = {
name = "letsencrypt-prod"
}
spec = {
acme = {
email = "[email protected]"
server = "https://acme-v02.api.letsencrypt.org/directory"
privateKeySecretRef = {
name = "issuer-account-key"
}
solvers = [
{
http01 = {
ingress = {
class = "nginx"
}
}
}
]
}
}
}
}
Option 2: Setup using cluster-issuer.yml
Instead of using the kubernetes-alpha
way of setting up the cluster
issuer, we can do a simple yml
file and do it the kubectl way.
cluster-issuer.yml
:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: [email protected]
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: issuer-account-key
# Add a single challenge solver, HTTP01 using nginx
solvers:
- http01:
ingress:
class: nginx
Then apply it
kubectl apply -f cluster-issuer.yml
And we can look at it like so
kubectl describe clusterissuer letsencrypt-prod
kubectl get cert --namespace favoriteapp
NAME READY SECRET AGE issuer-account-key True issuer-account-key 34m
App deployment
Finally, we define our app itself. It has to moving pieces that can be scaled independantly.
One is called favoriteapp
that is initially set to have 2 replicas.
We define two types of containers here, one is the init_container
that
basically runs on each pod startup to run the migration (command =
["rake", "db:migrate"]
) and the other is the container itself that
serves the rails application on port 3000.
The other is favoriteapp-workers
which runs the sidekiq
command.
resource "kubernetes_deployment" "favoriteapp" {
metadata {
name = "favoriteapp"
labels = {
app = "favoriteapp"
}
namespace = "favoriteapp"
}
spec {
replicas = 2
selector {
match_labels = {
app = "favoriteapp"
}
}
template {
metadata {
name = "favoriteapp"
labels = {
app = "favoriteapp"
}
}
spec {
init_container {
image = "wschenk/favoriteapp:master"
image_pull_policy = "Always"
name = "favoriteapp-init"
command = ["rake", "db:migrate"]
env_from {
config_map_ref {
name = "favoriteapp-config"
}
}
}
container {
image = "wschenk/favoriteapp:master"
image_pull_policy = "Always"
name = "favoriteapp"
port {
container_port = 3000
}
env_from {
config_map_ref {
name = "favoriteapp-config"
}
}
}
}
}
}
}
resource "kubernetes_deployment" "favoriteapp-workers" {
metadata {
name = "favoriteapp-workers"
namespace = "favoriteapp"
}
spec {
replicas = 1
selector {
match_labels = {
app = "favoriteapp-workers"
}
}
template {
metadata {
name = "favoriteapp-workers"
labels = {
app = "favoriteapp-workers"
}
}
spec {
container {
image = "wschenk/favoriteapp:master"
name = "favoriteapp-workers"
command = ["sidekiq"]
env_from {
config_map_ref {
name = "favoriteapp-config"
}
}
}
}
}
}
}
Now that we have the deployments
running, we need to expose them first
to the cluster as a service
(basically this gives them a name and a
port that other kubernetes services can access).
Once that service is defined, we define an ingress
that lets the
outside world connect to the internal service, which in turn connects
to the pods running in the deployment.
resource "kubernetes_service" "favoriteapp-service" {
metadata {
name = "favoriteapp-service"
namespace = "favoriteapp"
}
spec {
port {
port = 80
target_port = 3000
}
selector = {
app = "favoriteapp"
}
}
}
resource "kubernetes_ingress" "favoriteapp-ingress" {
wait_for_load_balancer = true
metadata {
name = "favoriteapp-ingress"
annotations = {
"kubernetes.io/ingress.class" = "nginx"
"cert-manager.io/cluster-issuer" = "letsencrypt-prod"
"cert-manager.io/acme-challenge-type" = "http01"
}
namespace = "favoriteapp"
}
spec {
rule {
host = "k8.willschenk.com"
http {
path {
path = "/"
backend {
service_name = "favoriteapp-service"
service_port = 80
}
}
}
}
tls {
hosts = [ "k8.willschenk.com" ]
secret_name = "issuer-account-key"
}
}
}
terraform
and kubectl
Now we run terraform apply
and, if you've entered in all of your
credentials correctly, the application should start up with all of the
correct datasources, migrations run, and the whole thing.
You can walk through the flow to make sure that the app is working, that things get stored in the database, and that the sidekiq jobs processed what is needed.
You can also configure kubectl
locally so that you can examine the
cluster.
export CLUSTER_ID=$(terraform output -raw cluster-id)
mkdir -p ~/.kube/
curl -X GET \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${TF_VAR_do_token}" \
"https://api.digitalocean.com/v2/kubernetes/clusters/$CLUSTER_ID/kubeconfig" \
> ~/.kube/config
Manually reissuing the certificate
First look to see what the status of your certiticate is:
kubectl get cert --namespace favoriteapp
And you can also look at the certificate request itself to see if everything is good.
kubectl describe certificaterequest issuer-account-key --namespace favoriteapp
Install the cert-manager
plugin locally:
cd /tmp
curl -L -o kubectl-cert-manager.tar.gz https://github.com/jetstack/cert-manager/releases/download/v1.4.0/kubectl-cert_manager-linux-amd64.tar.gz
tar xzf kubectl-cert-manager.tar.gz
sudo mv kubectl-cert_manager /usr/local/bin
Looking at the deployment
Webapp:
kubectl logs --namespace favoriteapp deployment/favoriteapp
Workers:
kubectl logs --namespace favoriteapp deployment/favoriteappworker
Migration:
kubectl logs --namespace favoriteapp jobs/favoriteapp-migration
Deploying a new version
First we make a change to the app, then check it in. Once things are finished building we can manually trigger a restart:
kubectl rollout restart --namespace favoriteapp deployment/favoriteapp
kubectl rollout restart --namespace favoriteapp deployment/favoriteapp-workers
Setting up automatic deployment
We can also extend our github action to use kubectl itself to trigger the deployment. (You'll probably want to add a step in there to run tests also!) This is what that looks like.
First, to you need to add your .kube/config
to the repositories
secrets. First convert to base64, then add a new secret named
KUBE_CONFIG_DATA
:
cat $HOME/.kube/config | base64
Then, add the following steps to build-and-push.yml
:
-
name: Deploy App
id: k8app
uses: steebchen/[email protected]
with: # defaults to latest kubectl binary version
config: ${{ secrets.KUBE_CONFIG_DATA }}
command: rollout restart --namespace favoriteapp deployment/favoriteapp
-
name: Deploy workers
id: k8workers
uses: steebchen/[email protected]
with: # defaults to latest kubectl binary version
config: ${{ secrets.KUBE_CONFIG_DATA }}
command: rollout restart --namespace favoriteapp deployment/favoriteapp-workers
Final thoughts
What a journey this post has been! Stepping back a while bunch it's
not really clear to me that this is an improvement. I've a lot of
applications on Heroku, which has a much simplier workflow. heroku
create
, git push
and there you go. It locks you into a certain way of
doing things and buildpacks, while a bit more constaining compared to
Dockerfiles
are about a zillion times easier to work with.
And on the otherside, you have things like cloud functions, either using something like OpenFaaS or even different deployment models all together. If you are in the Node or Deno ecosystems what's going on with Deno Deploy or even NextJS is a much easier way to actually get something up and running. The level of complexity for kubernetes is truely mind boggling, and a number of times during this write up I was muttering under my breath about a simplier world were we could FTP PHP files around…
Basically, I'm not sure that I often find myself with the problem where kubernetes is the right solution. It's certainly very cool, and the idea of having a bunch of resources that, with a little guidance, and sort of manage and heal themselves is pretty amazing. But I also feel that there's way too much going on than what I properly understand, and it's a lot of ceremony to make stuff happen.
References
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK