Simplification and consolidation

December 20, 2023

Where I started

6 years ago I was in a different place, I was starting my journey as a corporate DevOps engineer. I wanted to tinker and self-host everything that I could, being able to do it myself and have resiliency if there was an internet outage really appealed to me. I ran my own email server, HomeAssistant, small kubernetes clusters with helm charts and all. It was fun and I learned a lot, but as the years went by I realized that as my life got busier I would have less time to troubleshoot. I wouldn’t have a couple hours to troubleshoot why my emails were being flagged as spam, I wouldn’t have time to figure out why my ceph cluster is reporting as corrupted and crashing my services. The more things like that happened, the more overwhelmed I became and the less I wanted to deal with it and the more it made sense to simplify and consolidate. So thats what I did. This post is dedicated to displaying the transition I’ve made to a much more stable, consistent, and simple setup that causes me very little issues or time cost.

Before: Mail in a Box, After: Custom domain hosted on Fastmail

Setting up and running MIAB was super simple thanks to their installer, and initially it was great. However the more I started relying on it, the more the flaws started showing:

Email was only available to send/receive if I was on the same network as MIAB (this could have been resolved by using a public VPS or VPN, but the other issues made it non-viable)
Email sent via MIAB was either flagged as spam, rejected, or just lost due to using my residential IP or one with zero “reputation” at all (this could have been resolved by using a mail forwarder service like SES, but there was no support for it in MIAB)
Downtime meant mail was missed, so updates and stability was critical. Unfortunately in any selfhosted homelab, stability is never a common resource. As a result, I purchased a plan on Fastmail and setup my domain’s MX records to point to Fastmail. Absolutely zero complaints so far, their mobile app is great and I no longer have reputation issues.

Before: Tinc VPN, After: Tailscale

Before wireguard and tailscale became popular, there was openvpn/IPsec and tinc. All the articles I read made it seem setting up openvpn and IPSec tunnels look daunting and confusing so I searched for alternatives and found Tinc. It seemed pretty simple to configure, open up some ports on a public IP, pass around some PSKs and boom you have your own VPN! Throughput and speeds weren’t the best, but it worked. I used Tinc for a long time, but it started showing it’s age when wireguard and tailscale first came out. Tailscale ended up being my VPN of choice because it came with all the benefits of wireguard but with shiny features built on top like:

Being able to use Mullvad VPN endpoints as exit nodes on your Tailscale network
Magic DNS, SSL certificates per node, tailscale funnel and tailscale serve
tailscale drop for sharing files on your Tailscale network
Exposing subnets and routes to machines not on the Tailscale network via Tailscale nodes Tailscale became a no brainer for me and has extremely simplified my VPN network to the point where I don’t really think about it anymore.

Before: HomeAssitant, After: Nothing

HomeAssistant was great for me initially. I was enthralled by it to the point where most tech purchases I made were based around if they had integrations with HomeAssistant or not (cameras, lights, thermostats, etc). Once I got it all setup I realized how pointless it was for me. Presence detection was janky at best, usually requiring multiple trackers to average out and guess if a person was home or not. Smart lights usually required some sort of separate hub in order to talk to them, and turning off light switches like most people do caused connectivity issues (can’t turn a light on if it doesn’t have power). Thermostat control was nice, but the integrations are created and supported by people in their free time so it comes with a lot of end-user troubleshooting in order to get things working. At the end of the day it just wasn’t worth it for me, and I didn’t get value out of it so I removed it.

Before: searX, After: Whoogle

One of the next major steps I wanted to take was to get away from google search. Not only is it filled with trackers and ads, but it also automatically ties whatever I search to my work google account. I do not want my employer seeing everything I search google for, and I don’t want targeted ads based on what I search. So naturally my search lead me to searX, its a self-hosted meta-search engine with a long list of providers. I was able to install it and get it up and running fairly easily, and even configured the optional addons for adding image support and rate-limiting searches so it doesn’t get abused when I expose it to the public internet. I probably used it for almost a year before I realized that the search results I was getting just weren’t good enough, and I ended up going back to google search for more accurate results. Then I discovered Whoogle, which is a semi-anonymous proxied version of google search. It has a similar UI with none of the tracking or ads, and proxies images from google images as well. A lot of the features you’re used to through google search are built into Whoogle as well, so it was a smooth transition for me.

Before: caddy, After: HAProxy on Pfsense

Let me start off by saying caddy is a fantastic piece of software that I’ve used for years. A static binary that handles LetsEncrypt certificates and a fantastic plugin selection made it a no-brainer for me. Nginx seemed a bit too much for what I was looking for, HAProxy felt like I was speaking a different language, so caddy was my choice. Setting up the configuration was simple enough, it handled the certificate management for me and I even set up a custom built docker image that included the binary and the plugins I used. In the end the decision to abandon caddy was more for consolidation and to reduce the number of hops to reach the services I hosted. HAProxy is extremely efficient and I was able to configure and run it via PFSense’s UI configuration after reading the documentation a few times.

Before: Proxmox/Kubernetes, After: Docker on unRAID

When k8s first started becoming popular I thought it was the best thing ever. Automatic application monitoring and scaling? Rolling upgrades built in? IaC by default? It was great. I attended the Kubecon convention multiple years in a row and definitely drank the kool-aid. I bought 3 SFF PCs and installed Proxmox on all 3 of them, set up a HA Cluster and then deployed k8s on the newly created VMs. Everything was great, I was using helm charts, cert-manager, the works. Then a week goes by and my cluster is offline. I try to dig into the issue but I’m not sure where to start, after googling various log errors I finally resolve the issue and get everything back up and running. Then a week later my ceph cluster used for persisten storage gets corrupted because of networking issues and my cluster is “stuck” again. By the time I resolved that issue (I had to wipe and redeploy from scratch, which isn’t hard when you have everything stored as manifest files) there was a new k8s version update that had major breaking changes and broke everything all over again. I finally realized that k8s can be a great asset, but running it alone makes it a nightmare. I looked into paying for a managed k8s service, but the pricing quickly made it a non-viable option for me. So I ended up converting all my manifests and helm charts to plain docker images that I run on my unRAID server and things have been smooth ever since.

Before: Gitlab/OneDev, After: Github and Github Actions

Back when Microsoft first aquired Github, I was among the many that initially jumped ship. The major competition at the time was Gitlab so I went off to figure out how to get my own instance up and running. Unfortunately they didn’t have any sort of docker-based deployment and the only supported way was through a VM. So naturally I spun up a VM on my unRAID server and ran their installer scripts, next thing I knew I had my own instance up and running. For the most part it worked flawlessly, and it came with a wide array of features, including a docker registry and CI/CD pipelines built into the service. Unfortunately I didn’t take advantage of most of the features Gitlab offered and would frequently run into issues where docker images would fill up the hard drive and I would have to SSH in and manually garbage collect. After getting tired of doing that, I bounced between some other alternatives like OneDev or Gitea. Unfortunately selfhosting a tool that holds the code and images for your other selfhosted services means that any issues brought almost everything down around it. So I decided to migrate back to Github and use a handful of private repositories in hand with Github Actions and I’ve been using that ever since.

Before: Daily backups via scripts for VMs across network, After: Docker on unRAID

Initially I had multiple VMs and docker hosts spread across multiple physical hosts for high availability. This quickly became a pain trying to keep everything up to date and consistent backups. I started trying to do simple rsync jobs across the network, or NFS mounts for remote storage/backup, but networking hiccups and cronjob nuances kept causing issues for me. I decided to focus more on having stateless containers/workloads so backups weren’t an issue for me since there was nothing to back up. Due to other architectural decisions mentioned earlier, I ended up running all my services on my unRAID server which has docker and VM backups built in. The backups are then synced offsite to my backup NAS via SyncThing.

Before: Custom twitch/youtube downloader, After: ytdl-sub

I had a list of Youtube channels I subscribed to as well as some twich streamers I liked. Unfortunately I didn’t have time to watch 4+ hour streams and preferred watching VODs on my free time, but I didn’t like the ads. Youtube was the same, I wanted to watch the videos, but without the ads and preferably have it all integrated directly into my Plex server since thats where I watch most of my TV and Movies already. Initially there was no tool that I could find that would do this automatically, so I decided to write one myself. To keep this brief, it was a python script that took a list of youtube and twitch channels and searched for new videos. If it found any it would download them (keeping a rudimentary history via text log) and put them in my Plex folder. It worked alright but integrating it into Plex was a challenge. I tried a few more “alternative” youtube frontends/self-hosted services and eventually settled on ytdl-sub which did exactly what I wanted and did a fantastic job of integrating it into Plex for me. I couldn’t be happier with how it works.

Before: Bazarr, After: Subgen

Subtitles have always been a need for me, between poor sound balancing in movies or watching foreign shows its something I can’t seem to live without. Bazaar seemed to fit the bill for this quite well initially, at least until I started using Tdarr to convert/clean up my movies and shows (stripping unecessary subtitles and audio tracks, putting them in easily streamable containers and formats). The problem I ran into is changing the files meant Bazaar would have a hard time finding the matching subtitle files. Then I found Subgen, which users a very optimized LLM to process and transcribe the audio in movies and shows into subtitle files in realtime (a normal movie is processed in about ~20 minutes). It also uses a different subtitle extension so it doesn’t overwrite/replace subtitles that may already exist. Running this service is very resource intensive so it isn’t something I run all the time, but it does a pretty good job and completely eliminates the need for a serivce like Bazaar or having to search for subtitles myself.