A URL shortener with an interesting storage backend

May 17th, 2021 · Tech · golang programming dns

If you follow my blog, or happen to keep up to date with what I create, then you will know that I have been using YAML as a storage backend for my applications for a while now. It’s relatively low maintenance but it is limited in terms of distributed systems. Sharing data across distributed systems is an extremely tricky feat. You have to be sure that data is propagating appropriately and caching is sensible so that you aren’t overloading your data source, but likewise aren’t serving stale data.

What if I could tell you that there was a data store you could be using that was redundant and distributed by design, allowed granular caching per record in your dataset, and is incredibly simple to set up and maintain. Snake oil this is not, this is DNS.

Now, storing data in DNS is not a new fad, in fact I spoke about this back in 2018. More recently there have been countless memes on social media about using Amazon’s Route53 DNS service as a data store, so I felt it was time to upgrade my URL shortener from using YAML to DNS as its storage medium.

What’s your favorite database?
Mine is Route 53.
It has a 100% SLA, you can query it, and DNS is fundamentally a large key-value store.
Listen to @AWSMorningBrief here: https://t.co/6OlOVFakko

— Corey Quinn (@QuinnyPig) November 13, 2019

The resulting Go code is 59 LOC, including comments and double checking the resulting URL to forward to isn’t relative, seeing as we can’t even trust url.ParseRequestURI to only give us absolute paths any more (not bitter).

So how does this URL shortener work? Well like most it has a base URL and then a shortcode at the end which correspods to a URL to redirect to. We also have a base domain which may or may not be the same as the base URL, and the flow happens thusly:

1. HTTP request to https://[base URL]/[shortcode] (e.g. https://ols.wtf/_orgy)
2. DNS TXT record lookup against [shortcode].[base domain] (e.g. orgy.short.ols.wtf)
3. Contents of the TXT record are used to perform a 302 redirect (e.g. https://ols.wtf/2021/05/10/orgy-personal-tech-stack.html)

You want to create a new redirect? Create a TXT record with the URL to redirect to on your dedicated URL shortening subdomain. With these being individual DNS records, you can chop and change the TTL on each of them as you wish, and now you can run as many instances of this as you want with their distributed yet authoritative record of your data. Fantastic!

Praise for dns-url-shortener

• “significantly less ridiculous than using YAML as a datastore” - hoistbypetard on irc

Do you have a comment to make on this post? Start a discussion in my public inbox by emailing ~ols/[email protected]. You can see the inbox here.