alekcz/pcp: PCP: Clojure Processor

README.md

Welcome to PCP

PCP: Clojure Processor -- Like drugs but better

Introduction

Too long have we hustled to deploy Clojure websites. Too long have we spun up one server instance per site. Too long have reminisced about PHP. Today we enjoy the benefits of both. Welcome to PCP.

Status

Active development. Stabilizing.
Latest version: v0.0.1

Goals

Any easy to use, drop-in Clojure replacement for php scripts
Allow multiple website to be hosted on single $5 VPS

Non-goals

Performance. PCP should be sufficient for prototypes and small websites (< 400 req/s)

Demo site

You can view a demo site built in PCP here: https://clojure-pulse.musketeers.io

Overview

Quick start

Install pcp via the installer script:

$ bash -c "$(curl -sSL https://raw.githubusercontent.com/alekcz/pcp/master/install.sh)"

Query the SCGI server status:

$ pcp service status

Create a new project:

$ pcp new project-name

When you create a new project some example code is generated as well which, for convenience, you can view using the local server built in to the pcp utility. This local sever behaves as your pcp site would when deployed with nginx.

$ cd project-name
$ pcp -s public/

Navigate to http://localhost:3000/ and volià.

This is the corresponding code.

(ns index
  (:require [pcp :as pcp] ;this is the core namespace with various useful functions
            [api.info :as a] ;this is another name space in the same project. 
            [garden.core :refer [css]])) ;this is 3rd party namespace from the hosted environment

(def resp 
  (pcp/html
    [:html {:style "font-family: 'Source Sans Pro', Arial, Helvetica, sans-serif;"
            :lang "en"}
      [:head 
        [:title "PCP website"]
        [:style 
          (css [:html   { :text-align "center"}])
          (css [:p.info { :background-color "#EEE" 
                          :font-size "13px"
                          :margin-top "40px"
                          :padding "10px" 
                          :text-align "left" 
                          :width "300px"}])
          (css [:strong { :font-size "13px"}])
          (css [:code   { :font-size "12px" :font-weight "normal"}])
          (css [:.main  { :display "flex"
                          :flex-direction "column"
                          :justify-content "center"
                          :align-items "center"
                          :min-height "90vh" 
                          :font-weight "normal"}])]]
      [:body 
        [:div.main 
          [:img 
            {:src "//raw.githubusercontent.com/alekcz/pcp/master/assets/logo/logo-alt.svg" 
             :width "200px"}]
          [:h1 "Your PCP site is up and running."]
          [:p "Your json endpoint is here " [:a {:href "/api/info.clj"} "here"] 
            [:br]
            "Learn more about pcp at " [:a {:href a/repo :target "_blank"} "here"]
            [:br]
            "Happy scripting!"]
          [:p.info
            [:strong "Request uri "] [:span (str (-> pcp/request :request-uri))]
            [:br] 
            [:strong "Request method "] [:span (str (-> pcp/request :request-method))]
            [:br] 
            [:strong "Request scheme "] [:span (str (-> pcp/request :request-scheme))]
            [:br] 
            [:strong "User agent "]
            [:br] 
            [:code (str (-> pcp/request :http-user-agent))]
            [:br]]]]]))

(pcp/response 200 resp "text/html")

When navigating to your site if you see Connection refused (Connection refused) it means the SCGI server not running. It could still be booting or be down.

You can find instructions on replacing php and deploying to production here

How PCP works

PCP has two parts the utility is simple binary, built with GraalVM, that allows you work effectively with pcp.

PCP: Clojure Processor -- Like drugs but better

Usage: pcp [option] [value]

Options:
  new [project]           Create a new pcp project in the [project] directory
  service [stop/start]    Stop/start the PCP SCGI server daemon
  passphrase [project]    Set passphrase for [project]
  secret [path]           Add and encrypt secrets at . or [path]
  secret [path]           Add and encrypt secrets at . or [path]
  -e, --evaluate [path]   Evaluate a clojure file using PCP
  -s, --serve [root]      Start a local server at . or [root]
  -v, --version           Print the version string and exit
  -h, --help              Print the command line help

The heavy lifting is done by an SCGI server. This server runs on port 9000 and receives requests from the local pcp server or from nginx. The SCGI server is an uber jar that runs as a daemon.

System requirements

PCP is designed to work on Linux and OSX. It requires the following to work:

java 8+
systemd / launchd
persitent disk space
nginx (for production deployment)

Project structure and requiring files

Requiring files in PCP mostly works as it does in Clojure, the primary exception being that a project cannot require a file outside of it's root. Given a project like this:

.                           # project root    
├── .pcp      
│   └── a24d...cfdcb8.npy  # encrypted secret, touch this and bad things will happen    
├── public                 # server root    
│   ├── index.clj          
│   └── login.clj    
│   └── dashboard.clj    
│   └── logic    
│       └── orders.clj    
│       └── payments.clj    
│       └── stock.clj    
│       └── users.clj    
├── pcp.edn                # pcp configuration file    
└── README.md              # surely you don't need a comment to tell you what this is.

Your dashboard could require business logic as follows

(ns dashboard
  (:require [logic.users :as lu]
            [logic.orders :as lo]
            [logic.stock :as ls]
            [logic.payments :as lp]))

Environment variables and secrets

To allow API keys, tokens and the like to be stored and retrieved securely, PCP encrypts them and stores them in the project. Secrets are created using the PCP CLI. They are encrypted using the passphrase selected. Here is an example of the process.

$ pcp secret
--------------------------------------------------
Set an encrypted secret variable for project: pcp-demo
Please ensure you use the same passphrase for all your secrets in this project
and that you add your passphrase to your production server using:
  pcp passphrase pcp-demo
--------------------------------------------------
Secret name: GITHUB_PERSONAL_ACCESS_TOKEN
Secret value: ghp_eR17e5vHq0Sdmj22oracJd0Y7je1IM3g7oPV7yT7sq31
Passphrase: my-super-secure-passphrase-that-i-will-also-store-on-the-server    
encrypting...
done.

Symmetric encryption is used, so the passphrase needs to be added to the server too.

$ pcp passphrase pcp-demo
--------------------------------------------------
Set passphrase for project: pcp-demo
This passphrase will be used for decrypting secrets at runtime.
--------------------------------------------------
Passphrase: my-super-secure-passphrase-that-i-will-also-store-on-the-server

PCP namespace

The following functions are part of the core PCP namespace and are made available for convenience.

pcp/persist

(pcp/persist :cache-key f-on-miss)
This macro allows expensive operations to only be recomputed if they are not in the cache.
On a cache miss the f-on-miss is computed, its value stored in the cache and returned. Caches are isolated across projects. Pages in the same project share a cache keys. The cache uses TTL (30 min) as its invalidation strategy.

pcp/clear

(pcp/clear :cache-key)
Removes key from the cache

pcp/request

pcp/request
The request map. The request map conforms to the ring spec.

pcp/response

(pcp/response [status body mime-type])
A convenience function for generating a response map. Responses are simply Clojure maps that confirm to the ring spec and can be written by hand too.

pcp/render-html

(pcp/render-html options & content)
Renders html from Clojure data strucutures using hiccup

pcp/render-html-unescaped

(pcp/render-html & args)
Renders html from Clojure data strucutures using hiccup. Does not escape html tags strings. Use with care.

pcp/secret

(pcp/secret "SECRET_NAME")
Retrieves secret from project. The secret is read from disk. It may be worthwhile using pcp/persist to improve performance. See Environment variables and secrets for more info.

pcp/now

(pcp/now)
Returns the current time in milliseconds (according to your server).

pcp/slurp

(pcp/slurp "../file-name")
Opens a reader on f and reads all its contents, returning a string. Cannot access files higher up that project root (i.e. server root -1). Does not currently accept any optional arguments.

pcp/spit

(pcp/spit "../file-name" "like drugs but better")
Opposite of slurp. Opens f with writer, writes content, then closes f. Cannot access files higher up that project root (i.e. server root -1). Does not currently accept any optional arguments.

Other built-in namespaces

In addition to the core clojure namespaces available in sci, the following namespaces are also available.

clojure.string
clojure.core.async
clojure.edn
cheshire.core
selmer.parser
selmer.filters
org.httpkit.client
org.httpkit.sni-client
next.jdbc
honeysql.core
honeysql.helpers
postal.core
tick.alpha.api
buddy.sign.jwt
buddy.sign.jwe
buddy.core.hash
buddy.core.codecs
buddy.core.keys
buddy.auth.backends
buddy.auth.middleware
buddy.hashers
garden.core
garden.stylesheet
garden.units
konserve.core
konserve.filestore
konserve-jdbc.core

Performance

A modest PCP website can handle it's own pretty well.
This test was run for 80 minutes with 400 VUs and 400 req/s. Of the 1.9M requests generated, only 26 failed.

You can see the spike in CPU and bandwidth on the droplet, but the CPU never saturates.

These are by no means comprehensive benchmarks but give you a sense of what a PCP server can withstand for simple use case. Going above 400 req/s generally results in bad things happening. You can test your own site using k6 with the instructions in loadtest.js

Roadmap & releases

Release 0.0.1

Run arbitrary scripts
Generate HTML with hiccup
Invoke scripts from Nginx
Emulate of Nginx environment locally
Store secrets in projects
Restrict slurp and spit to project root
Cache expensive operations
Expose request data to scripts
Require external namespaces as in Clojure
Generate new project from CLI
Document PCP
Load test a production deployment

Release 0.0.2

Perfomance improvements

Release 0.1.0

Store passphrases with konserve
Add sponsorship button
Deploy without logging in to VPS
Switch to malli-cli
Automate deployment from repo

Release 0.2.0

Add PEDL (PCP, Nginx, Datahike Server, and Linux) stack template on Digital Ocean.
Other things that even the wisest cannont tell; things that were, things that are, and some things... that have not yet come to pass.

Special thanks

For the guidance and examples, special thanks to

License

PCP is licensed under the MIT License.

GitHub - alekcz/pcp: PCP: Clojure Processor -- A Clojure replacement for PHP