Connecting the opam bench to Gitter

September 3, 2019

The Coq opam bench system detects bugs in opam packages from the Coq repository. However these bugs may be hard to follow. Indeed, they are in a table one needs to consult regularly. We present a system to post results in a Gitter channel coq/opam-bench-reports. Thanks to this channel, we can more easily track down the latest build failures.

The idea of publishing results in Gitter was given by Emilio Jesús Gallego Arias in coq/issues/10418. Improvements were added from suggestions from Emilio Jesús Gallego Arias and Karl Palmskog.

Implementation

The code to publish results in Gitter is at github.com/coq-bench/make-html. The implementation is in Ruby and shares code with the bench website generator. We extract the results from a CSV database generated by the project github.com/coq-bench/run which tests the packages.

To connect to Gitter, we use the ruby-gitter package. The code goes as follows:

# Use the developer token from the Gitter documentation
# (what is possible as very few messages are sent)
client = Gitter::Client.new(token)

# Find the room id
room = client.rooms.find {|room| room.name == "coq/opam-bench-reports"}
room_id = room.id

# Construct the message with the results
message = "..."

# Actually send the message
client.send_message(message, room_id)

We use a dedicated coq-opam-bench-gitter-bot user to publish on Gitter and sign the messages. We retrieve the results of the past 72 hours, and run the code every 72 hours with a shell command:

while true; do sleep 72h; date; git pull; ruby push_to_gitter.rb ../database gitter-token coq/opam-bench-reports 72; done

Note the git pull of hot code swapping! This is useful to integrate small bench updates without breaking the 72 hours rythm.

We either display the list of packages which failed, or a success message.

Results

We went from dozens of errors every 72 hours (when setting up the system) to just a few these days. Probably, this was mainly due to the bug corrections themselves but the Gitter reports helped. The Gitter reports also created a channel to talk about opam bugs.