I have made mistakes I hope to never make again.

I have:

• Deleted production data
• Pushed some code which actually caused a system-wide infinite loop in the cloud environment. My bug showed up in their status page 😆
• Made a change which took down DEV for 10+ hours

Harder to sum up, but I've definitely made software design choices which caused problems later, and are still causing problems today. The notorious technical debt.

All of these experiences give me caution which I hope to pass on to others, but it's also important to know that some mistakes are not overly avoiding. As they say in CSS, if everything is !important, nothing is !important. I think mistakes related to data and privacy are the top mistakes to avoid. Any mistake with a known remedy or short-lived impact are not dramatically worth avoiding.

So back in the day, I was working for a tech startup in Boston's metro-west area. I was one of the front-end devs, and we were building some great tech that was being fed a transcript via an API. But the boss wanted to get experimental.

"Wouldn't it be great if, as we progress through the transcript, we could pop up related material at a particular time? Say, the transcript mentions Steve Jobs, we pull up his Wikipedia profile and the top five news stories, or it mentions Microsoft, we display their stock ticker, business news, and related content on the fly?"

And yeah, that sounded neat. But let's be real - it was 2007.

So I tried, I found a great early BaaS from Yahoo! called Pipes. Way ahead of its time, but it allowed us to scrape, munge and combine feeds, based on what we passed to a call to a particular pipe. It was a meta-API, of sorts.

So yeah. The first few days of playing with the parts, things were looking very promising. The Pipe was working as I wanted, it could handle different types, it could return different data, and I could consume it however I like. Sweet!

And then at scale. From the live transcript. That's where I broke things.

See, in one way you can take a particular key term at a particular point in time, pass the request, get the data. With a very small change, you can take all the key terms, from all the points in time, and request them all. For each key term. Thousands per minute.

I inadvertently used Yahoo! for an ugly sort of DDoS on multiple search engines and sites. Boss and I had to go to a meeting with numerous tech heads, explain what our code did and why - they were more curious about the use case and the "how and why", and less about blame.

Honestly, I think we learn more from those moments someone breaks stuff.

UPDATE users SET password='...';

Comment button Reply

Thread

Thread

New branch for debugging, pulled from prod and not test. The picture speaks for itself….

Comment button Reply

Lol not nearly my biggest mistake ever but a particularly frustrating (and now funny one) back when I was building my photography section for my personal site:

I don't remember the exact filename but I had a file named something like "3DCssRenderer.js". I was developing locally on Windows and deploying to an Ubuntu server (obv not best idea as we'll learn in a moment but I was being lazy)

The code was working 100% fine on my machine but when deployed to the remote server the 3DCssRenderer.js couldn't be found which wasn't an error I was anticipating

After like 6 hours of tedious debugging I eventually realized the file was named 3DCssRenderer.js (uppercase C) but somewhere I made a typo and something was looking for a "3DcssRenderer.js" (lowercase c).

Windows isn't case sensitive but Linux is hence why it worked locally and not remotely (and of course it was "C" vs. "c" which isn't obvious at quick glance). Sure enough fixed the typo and it worked perfectly after that 😅 This experience has actually helped me a few times now helping colleagues solve similar bizarre bugs

Comment button Reply

When I read the question, I immediately thought of starting CSS but on a shit*y foundation. It can't be the worst thing, but it sucks to start all over again when you've made good progress. 😶

Comment button Reply

Directly coding wise, I was adding to an enum list and thought it would be helpful to add it alphabetically so you'd see it when scanning the list. This gave the enums different values as they were assigned automatically as incrementing values. Very strange program behaviour followed.

Related to coding but not exactly, I almost lost two weeks worth of work by not know what I was doing in Git

I am not doing this for long, however my biggest mistake so far was to set up a cronjob wrong. Instead of sending an email to all inactive accounts on the 15th of a month it send an email every minute to all inactive accounts on the 15th of the month. Over 300 people received 600+ email from us.

Not communicating when things started to go off rails from the estimate on a big development.

One of my biggest mistakes was when I was the Lead Backend Dev at Eden Life Care(this startup never launched btw). If I remember correctly, I kept chaning dependencies for authentication/authorization for the project. It came to a point where I actually had to restart the project when I was still the only dev working on it(this was a few days after I started the project - which was a good thing).

Another mistake was not focusing on coding for music. I started playing the guitar when I was 13 and been in a band since I was 15 years old and have just started coding with music.

It was long time ago, when we were using FTP clients. I accidentally moved everything from root to one of the folders (without realising). This was on the website that counts its visitiors in millions. After the initial shock of everything not working, it took me almost 30 minutes to figure out what was going on. 😅

I had some dodgy "retry on network fail" logic in our client android app that accidentally retried twice for each failure (which of course escalated quickly). Tested fine, got pushed to production and then we started getting messages from the server team.

So I basically created a bot army out of our android users and got them to DDOS our own servers 😂

That was a decade ago, but to this day I try and avoid any automatic retries. If it's an option, I much prefer to let the user chose to retry manually

Blowing a capacitor on my monitor while testing a VGA display driver I was writing. Loud bang and a puff of smoke. Oddly, the monitor still worked! 😛

Delete from tablename ...was not my dev environnement 😥

Back in the day when drag n drop SFTP was a real thing… overwriting a folder that crashed a live multi template site (of multiple vendors).. basically crashing 5 insurance sites in a flash 😂