Using Raspberry Pi to Migrate GitHub Runners to Self-Hosted Ones
source link: https://hackernoon.com/using-raspberry-pi-to-migrate-github-runners-to-self-hosted-ones
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Using Raspberry Pi to Migrate GitHub Runners to Self-Hosted Ones
5min
by @nfrankel
Nicolas Fränkel
@nfrankel
Developer Advocate for Apache APISIX *** Learner *** Author of...
Too Long; Didn't Read
audio
element.@nfrankel
Nicolas FränkelDeveloper Advocate for Apache APISIX *** Learner *** Author of http://leanpub.com/integrationtest
Receive Stories from @nfrankel
Credibility
Recently, I mentioned how I refactored the script to keep my GitHub profile up-to-date. Since Geecon Prague, I'm also a happy owner of a Raspberry Pi:
https://twitter.com/nicolas_frankel/status/1715313240568197233
Though the current setup works flawlessly - and is free, I wanted to experiment with self-hosted runners. Here are my findings.
Context
GitHub offers a large free usage of GitHub Actions:
GitHub Actions usage is free for standard GitHub-hosted runners in public repositories, and for self-hosted runners. For private repositories, each GitHub account receives a certain amount of free minutes and storage for use with GitHub-hosted runners, depending on the account's plan. Any usage beyond the included amounts is controlled by spending limits.
Yet, the policy can easily change tomorrow. Free tier policies show a regular trend of shrinking down when:
- A large enough share of users use the product, lock-in
- Shareholders want more revenue
- A new finance manager decides to cut costs
- The global economy shrinks down
- A combination of the above
Forewarned is forearmed. I like to try options before I need to choose one. Case in point: what if I need to migrate?
The theory
GitHub Actions comprise two components:
- The GitHub Actions infrastructure itself. It hosts the scheduler of jobs.
- Runners, who run the jobs
By default, jobs run on GitHub's runners. However, it's possible to configure one's job to run on other runners, whether on-premise or in the Cloud; these are called self-hosted runners.
The documentation regarding how to create self-hosted runners gives all the necessary information to build one, so I won't paraphrase it.
I noticed two non-trivial issues, though. First, if you have jobs in different repositories, you need to set up a job for each repository. Runner groups are only available for organization repositories. Since most of my repos depend on my regular account, I can't use groups. Hence, you must duplicate each repository's package on the runner's Pi.
In addition, there's no dedicated package: you must untar an archive. This means there's no way to upgrade the runner version easily.
That being said, I expected the migration to be one line long:
jobs:
update:
#runs-on: ubuntu-latest
runs-on: self-hosted
It's a bit more involved, though. Let's detail what steps I had to undertake in my repo to make the job work.
The practice
GitHub Actions depend on Docker being installed on the runner. Because of this, I thought jobs ran in a dedicated image: it's plain wrong. Whatever you script in your job happens on the running system. Case in point, the initial script installed Python and Poetry.
jobs:
update:
runs-on: ubuntu-latest
steps:
- name: Set up Python 3.x
uses: actions/setup-python@v5
with:
python-version: 3.12
- name: Set up Poetry
uses: abatilo/actions-poetry@v2
with:
poetry-version: 1.7.1
In the context of a temporary container created during each run, it makes sense; in the context of a stable, long-running system, it doesn't.
Raspbian, the Raspberry default operating system, already has Python 3.11 installed. Hence, I had to downgrade the version configured in Poetry. It's no big deal because I don't use any specific Python 3.12 feature.
[tool.poetry.dependencies]
python = "^3.11"
Raspbian forbids the installation of any Python dependency in the primary environment, which is a very sane default. To install Poetry, I used the regular APT package manager:
sudo apt-get install python-poetry
The next was to handle secrets. On GitHub, you set the secrets on the GUI and reference them in your scripts via environment variables:
jobs:
update:
runs-on: ubuntu-latest
steps:
- name: Update README
run: poetry run python src/main.py --live
env:
BLOG_REPO_TOKEN: ${{ secrets.BLOG_REPO_TOKEN }}
YOUTUBE_API_KEY: ${{ secrets.YOUTUBE_API_KEY }}
It allows segregating individual steps so that a step has access to only the environmental variables it needs. For self-hosted runners, you set environment variables in an existing .env
file inside the folder.
jobs:
update:
runs-on: ubuntu-latest
steps:
- name: Update README
run: poetry run python src/main.py --live
If you want more secure setups, you're on your own.
Finally, the architecture is a pull-based model. The runner constantly checks if a job is scheduled. To make the runner a service, we need to use out-of-the-box scripts inside the runner folder:
sudo ./svc.sh install
sudo ./svc.sh start
The script uses systemd
underneath.
Conclusion
Migrating from a GitHub runner to a self-hosted runner is not a big deal but requires changing some bits and pieces. Most importantly, you need to understand the script runs on the machine. This means you need to automate the provisioning of a new machine in the case of crashes. I'm considering the benefits of running the runner inside a container on the Pi to roll back to my previous steps. I'd be happy to hear if you found and used such a solution. In any case, I'm not migrating any more jobs to self-hosted for now.
To go further:
Originally published at A Java Geek on March 10th 2024
Recommend
-
16
Self hosted Github Actions runners in Kubernetes has_many :codes Tips and walkthroughs on web technologies and digital life
-
14
Deploy self-hosted GitHub Actions runners for Red Hat OpenShift Skip to main content Note
-
8
Connecting to a private network from GitHub-hosted Actions runnersSeveral ways for GitHub-hosted Actions runners to connect to resources on your private network. ...
-
6
Managing self hosted CI runners at scale with EC2 spot instancesA cost effective solution for running self hosted runners at scale using Github actions and AWS EC2 Spot instances
-
2
Today, GitHub is releasing a public beta for all new, more powerful hosted macOS runners for GitHub Actions. Teams who are looking to speed up their macOS jobs now have new options to meet their needs.
-
6
How to Use Gith...
-
6
Migrate Data From Atlas to Self-Hosted MongoDB October 11, 2023 Sudhir Gautam In this blog post, we will discuss how we can migr...
-
7
This post will cover some general and high level troubleshooting when trying to use Container App Jobs as a self-hosted runner or agent Overview Since you can run self-hosted runners for GitHub Actions and Azure Pipe...
-
8
GitHub-hosted runners: Double the power for open sourceGitHub Actions continues its industry-leading support for the OSS community by doubling the Windows/Linux machine size to 4-vCPU runners at no cost...
-
3
Dependabot on GitHub Actions and self-hosted runners is now generally availableA quick guide on the advantages of Dependabot as a GitHub Actions workflow and the benefits this unlocks, including self-hos...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK