TensorFlow as a Service

TensorFlow as a Service (TFaaS)

A general purpose framework (written in Go) to serve TensorFlow models. It provides reach and flexible set of APIs to efficiently access your favorite TF models via HTTP interface. The TFaaS supports JSON and ProtoBuffer data-formats.

The following set of APIs is provided:

• /upload to push your favorite TF model to TFaaS server
• /delete to delete your TF model from TFaaS server
• /models to view existing TF models on TFaaS server
• /json to serve TF model predictions in JSON data-format
• /proto to serve TF model predictions in ProtoBuffer data-format

From deployment to production

➀ install docker image (TFaaS default port is 8083)

docker run --rm -h `hostname -f` -p 8083:8083 -i -t veknet/tfaas

➁ upload your TF model to TFaaS server

curl -X POST http://localhost:8083/upload
-F 'name=ImageModel' -F 'params=@/path/params.json'
-F 'model=@/path/tf_model.pb' -F 'labels=@/path/labels.txt'

➂ get your predictions

curl https://localhost:8083/image -F 'image=@/path/file.png' -F 'model=ImageModel'

Fore more information please visit curl client page.

TFaaS interface

Clients communicate with TFaaS via HTTP protocol. See examples for Curl , Python and C++ clients.

TFaaS benchmarks

Benchmark results on CentOS, 24 cores, 32GB of RAM serving DL NN with 42x128x128x128x64x64x1x1 architecture (JSON and ProtoBuffer formats show similar performance):

• 400 req/sec for 100 concurrent clients, 1000 requests in total
• 480 req/sec for 200 concurrent clients, 5000 requests in total

For more information please visit bencmarks page.

More information

• Install instructions to build TFaaS from source code
• End-to-end example of serving TF model in Go-server
• Demo
• CMS experiment use-case