100 Days of HomeLab - 1 Year Later

A year ago I started a challenge that encouraged everyone to join the #100DaysOfHomeLab challenge, a challenge designed to help improve your skills in IT. This is similar to any of the “100 Days” challenges - pick a topic, stick with it for 100 days, and form a habit. Some of you might be asking what a “HomeLab” is, and I think in it’s simplest terms it’s a “lab environment” mostly at home. Think of this as a test environment to learn about technology without the fear of breaking anything. If you’d like to learn more about HomeLabs and how to get started, I summarized it in a video.

After creating the challenge, I had lots of folks join in on Twitter, Mastodon, YouTube, Instagram and many other social networks using the hashtag #100DaysOfHomeLab, and it’s still going today!

365 Days of HomeLab

I ended up sticking with it, posting on socials (when I remembered), and took it all the way to 1 year! While some posts seemed redundant and repetitive, I kept on building, breaking, learning, and posting. Once I hit 100 Days, I decided to see how long I could go. 100 days turned into 200 days, and 200 days turned into 300… and today I hit 365 days. Looking back at my very first tweet, it seems I missed a few days of sharing, or I am really bad at math. If you can spot where I missed or messed up, let me me know in the comments below! 😀

OK, my turn 😀

Day 1 #100DaysOfHomeLab

Since Day 0 was spent launching the video, planning, connecting with people, and celebrating 100k - today will be spent:

Operationalizing the self-hosted website & Bot pic.twitter.com/5EObleZiAV

— Techno Tim (@TechnoTimLive) June 12, 2022

So what did you learn?

Over the last year I learned so much about HomeLabbing, but specifically Docker, Kubernetes, networking, ZFS, GitOps, and many other related technologies. You can see all of my 100 Days of HomeLab tweets here however I will summarize some of the topics.

Bots & Website

I started out by creating a Twitter bot that would retweet everyone who was joining the challenge. I felt like this was important to build and grow a community around HomeLab and a simple way to bringing people together. This is a self-hosted bot that I wrote myself, and even open sourced the code!

I also decided to create a 100 Days of HomeLab website so people could learn more about the challenge and even showcase some of the creators I worked with to make this possible. Huge thanks to all creators, featured on this page or not, who joined in on the fun!

What not to do

Another bucket of learnings were what not to do. This can be seen as mistakes but I looked at them as opportunities. These were things like:

• Backup everything you can’t recreate (even if you think you don’t need it)
• Test your backups
• Don’t run beta firmware in production
• Don’t run early access firmware in production (I had to list this twice in case I think about it again)
• Don’t deploy and walk away (for hours)
• Try not to test in production (although sometimes there isn’t any way around it)
• Don’t over buy or over build hardware, unless you think you can repurpose or sell it easily in the future
• You may not always need the fastest connection like 10G (things will just take a little longer)
• Don’t delete CRDs in k8s unless you are certain you don’t need any of the resources

What to do

It wasn’t all bad, I also picked up some good habits and learned what I should continue doing in the future:

• Build a small test environment, even if it’s hard
• Create good documentation and write it as if someone other than yourself will have to follow the steps
• Write tests, when you can, even even when you think you can’t
• Have just enough hardware, repurpose what you can
• A little battery backup goes a long way
• Use DNS when possible (even if it’s painful)
• Use certificates, always
• Use strong randomized passwords
• Set up monitoring & alerting, even if it’s just to yourself!
• Keep it simple. Seems like an oxymoron for HomeLab, but the simpler you keep it, the easier it will be to support

Hardware

I made lots of changes to my HomeLab over the past year, from a pile of machines on a shelf, to an open post rack, to a fully enclosed server rack in a room I converted to a server room. With this came new challenges like networking, power, and even RGB. I was sent a Storinator from 45 Drives and really expanded my storage while deprecating my old Disk Shelf. I also picked up a handful of low power devices and built a small low power cluster of Intel NUCs and rack mounted them in my server rack!

Automation

I also got to dive into Ansible deeper than ever before! Ansible is a powerful tool for automating things, especially infrastructure. I automated things like updates, configuration of my machines, password changes, and even building a fully HA Kubernetes cluster with k3s. The time spent learning this tool has already paid back in dividends compared to the time I would put into doing these task manually or even worse, pile up tech debt because I would skip them.

I also picked up Terraform too! Terraform is one of those things you may not ever learn until you need to. It’s definitely been eye opening building up new infrastructure with Terraform. Every time I see a form or a UI to create some sort of Infra, I automatically think about how I can automate this with Terraform… but thinking and doing are two different things and I need to start doing this more often. I’ve already figured out how to apply Terraform to Cloudflare DNS and will be applying to more systems in the future.

Networking

Once of the biggest changes to networking wasn’t new hardware or network speed, but VLANs. I implemented VLANs here to keep all of my network traffic segmented according to the roles these devices fill. For instance I created an IoT vlan for all of my IoT devices, a Camera VLAN just for secure video devices, and Server VLAN for my servers that are used for public facing services. This helps ensure that not only am I not mixing traffic, but also minimizing the blast radius if one of my devices were to become compromised. I talked about this and more security recommendations in a video here. Highly recommended if you are going to self-host anything.

Kubernetes

The next big theme is Kubernetes and has been a theme on my channel almost since the beginning. I now run 3 HA Kubernetes clusters at home. That might sound crazy, but it’s true. It’s taught me so much about how to build, support, and maintain one of the most popular technologies in the world. It’s been challenging but rewarding at the same time. I ended up going all in and migrating all of my Docker only hosts to Kubernetes. I no longer have single Docker hosts (pets) and now have more Kubernetes nodes (cattle). Once I moved everything to Kubernetes, I quickly learned that I needed a better way to manage it than just a UI or applying manifests from the CLI.

(GitOps has entered the chat)

GitOps

Git Ops, such a such a huge term and people have varying opinions on where it starts and where it ends - but it’s the idea that Git is the source of truth to deliver infrastructure as code. What does that mean for me and in my HomeLab? For me it means that my Kubernetes cluster (and custom code) is source controlled in Git and the only way to get those changes applied is through CI. This was one of the most rewarding things I have learned about during my 100 Days of HomeLab. All 3 of my Kubernetes clusters are defined in code (YAML) in a Git repository and when I need to make changes I just commit them to my repo and push them up and FLUX takes care of the rest. It has not only taught me how to deliver infrastructure as code but also taught me about secret management with SOPS which is such a valuable lesson, Kubernetes or not. I will be looking to expand into more IaC this year and beyond because this is truly the future of infrastructure.

Community

Last but definitely not least is community. Doing this challenge has taught me that there are so many other people out there just like me, trying to build/break/fix/learn with a lab environment at home. There are countless times where I have been inspired from others or even found better, more efficient way to accomplish things by interacting with the HomeLab community. I have even picked up new tech all thanks to you. I have met lots of people on socials and will continue to follow your journey!

So, what are you waiting for? Want to join the 100 Days of HomeLab Challenge? You’re just one click away!

Join the conversation

Day 365 #100daysofhomelab

One year. It's been a year since I have started the 100 days of #homelab and I've learned a lot. Here are my learnings in a thread 🧵

— Techno Tim (@TechnoTimLive) June 20, 2023

Links

⚙️ See all the hardware I recommend at https://l.technotim.live/gear

🚀 Don’t forget to check out the 🚀Launchpad repo with all of the quick start source files