Embed Data in Front-end Go Code

April 08, 2017

Embed data/assets/resources/binary in Go code, and compiled by GopherJS to a single JavaScript file which runs on the front-end browser.

I read interesting discussions on Reddit [2] and found that there are several packages which let developers embed data/assets/resources/binary in the Go code. Pakcages such as fileb0x, go-bindata, and go.rice are mentioned in the Reddit post, and I made some survey and decided to give fileb0x a try.

go-bindata has more stars and popular on GitHub. The reason why I do not choose go-bindata is that the way that go-bindata embed data in the code is very similar to the way I do in my current project. I use the template package in Go standard library to generate and embed data in Go code. fileb0x, however, embed the data in a separate package that we can import and use in our code, which makes the code more clean, readable, and easy to understand. This is the main reason I choose fileb0x instead of go-bindata.

Now I will show how I embed data in code that runs on front-end browser. I have the following data of succinct trie to be embedded in the code:

108K  website/rd.txt
4.0K  website/strie_node_count.txt
1.5M  website/strie.txt

Install fileb0x first:

$ go get -u github.com/UnnoTed/fileb0x

Next, create a config file to tell fileb0x where to find the data and build a Go package which has the data embedded in the code. The name of output Go package is paliDataVFS. For more detail of config file, see yaml config example.

pkg: paliDataVFS
dest: "github/repo/paliDataVFS/"

fmt: false

compression:
  compress: false
  method: ""
  keep: false

clean: false
output: "ab0x.go"
unexporTed: false
spread: false
debug: false

custom:
  - files:
    - "./website/rd.txt"
    - "./website/strie_node_count.txt"
    - "./website/strie.txt"

    base: "./website/"
    prefix: "public/"

Save the config as vfsb0x.yaml, and run fileb0x to create Go package:

$ fileb0x vfsb0x.yaml

The output will be github/repo/paliDataVFS/ab0x.go, which contains the data in it. Then I create a repo in GitHub, which is github.com/siongui/paliDataVFS, and commit the ab0x.go to the repo.

Install the package created by fileb0x:

$ go get -u github.com/siongui/paliDataVFS

Now we can access the data in our front-end Go code as follows:

import (
      "github.com/siongui/paliDataVFS"
      "strconv"
)

func GetTrieDataFromVFS() (succinctTrieDataBlob string, rankDirectoryDataBlob string, succinctTrieNodeCount uint) {
      tmp, _ := paliDataVFS.ReadFile("public/strie.txt")
      succinctTrieDataBlob = string(tmp)
      tmp, _ = paliDataVFS.ReadFile("public/rd.txt")
      rankDirectoryDataBlob = string(tmp)
      tmp, _ = paliDataVFS.ReadFile("public/strie_node_count.txt")
      tmp2, _ := strconv.ParseUint(string(tmp), 10, 64)
      succinctTrieNodeCount = uint(tmp2)
      return
}

After coding finished, use GopherJS to compile the code to JavaScript! It works!!!

Tested on: Ubuntu Linux 16.10, Go 1.8.1, Chromium Version 57.0.2987.98 Built on Ubuntu , running on Ubuntu 16.10 (64-bit).

References:

[3]GitHub - UnnoTed/fileb0x: simple customizable tool to embed files in go

[4]GitHub - jteeuwen/go-bindata: A small utility which generates Go code from any file. Useful for embedding binary data in a Go program.

[5]GitHub - GeertJohan/go.rice: go.rice is a Go package that makes working with resources such as html,js,css,images,templates, etc very easy.

[6]GitHub - inconshreveable/go-update: Build self-updating Golang programs

[7]GitHub - shurcooL/vfsgen: Takes an input http.FileSystem (likely at go generate time) and generates Go code that statically implements it.