Ask HN: Teaching Greybeard IT

I’ve had similar thoughts. I wrote a lot of docs on an internal wiki at a prior company, and also wrote some public docs on my site. In the end nobody really read it and it was too much effort to keep it up to date. That said, I would love to learn from you even though I’m a PM now. I have an insatiable curiosity about how things /really/ work that served me well for more than a decade as an SRE/DevOps/SysAdmin.

Ask the new hires if they care. They probably realize they need to learn AWS, Terraform, Kubernetes and Leetcode, anything else is secondary.

The guys 5-10 years of experience are likely more interested.

But when the black box stops working, how do you find out where to look to fix it? I get that the start of the art is advancing ever onward...but there's still a MAC address involved...waaaaay down the stack.

I was just thinking the other day that a non-profit wiki for sysadmins would be invaluable.

It could cover all the basics but also feature case-studies of real-world examples of solving problems, so there could be a sort of knowledge base too that would attract seekers and lead them into the foundational content.

10 or 15 years ago, I remember being able to find detailed problem solving information on Google. Today the search results are a morass of affiliate and SEO content multiplied by plagiarism served from a mountainous dung heap of user-generated blather.

I'd love to share security experience, suitably redacted, for experience learned from past engagements...it's easy to do at the water cooler, but that audience is limited.

Worked in info sec for a while now. Allow me to explain how every one of those stories go.

"We told them that you can't be running a production service on a Windows 7 desktop underneath the developers desk. They responded there was no budget for a server this year and that they couldn't take the time to port it anyway since they were too busy. We went to management and they told us to stop bothering the developers and they didn't give a damn about security that was our job so fix it. [3 minutes later] so anyway after they exfilled with the passwords and the social security info of all the employees we managed to get things restored from our 3 month old backups, and the CEO fired everyone in security. So that's why I'm currently in the job market."

"Ya so we got this meeting invite from some PM and get to his meeting with like 3 execs two dozen devs and enough middle management to make a B2B salesman weep for joy. They start the meeting by thanking everyone for the past two years of diligent work and the nights and weekends and promised to reimburse everyone for the legal costs associated with the ongoing divorces. Finally they ask us as security if we can give it the final approval so it can go into production at midnight. We explain we've never even heard of this project till now and what the hell is going on. Then one of the anonymous herd of grey suited PHB explains that they didn't invite us to meetings or ask for our help because they needed to "move fast and break things" and that security would've just slowed down their rockstar ninja wizard developers. Meanwhile my coworker has been poking at this for the past 10 minutes of the meeting and says there is no way in hell this thing is ready to ship. When asked why he pointed out that passwords were being sent in plaintext via a GET parameter in every request, every field was a SQL injection vulnerability and for some reason he was able to randomly kill processes by PID running on the server if he created a username that had a non ASCII character. The PM who called the meeting said we couldn't let good be the enemy of perfect and they were shipping anyway, which was met with thunderous applause. Well then you know what happened after, the lawsuits have mostly died down and the good news is my defense proved I didn't have any personal liability."

I think a LOT of this can be distilled down to a pessimistic anecdote...that doesn't mean they're not relevant.

Like how the 3rd part vendor likes to make passwords like %companyName%%YearOfEngagement%%symbol%

and how that might be bad, especially if your NAS admin console is discovered to be internet facing.

100x this.

If you’re not in an org that takes this stuff seriously, don’t walk, run.

This; something open ended, between a wiki and stackoverflow, would be a wonderful resource to offer curious minds.

For anyone wishing collaborate on this, I want to help or at least be in the loop so I can contribute an article or three! Let's band together.

Email in profile.

Write your own blog. Start with just plain text files served via lightweight web server of choise just to get over the starting hurdle.

I think you probably have a lot of valuable knowledge that should be written down.

What knowledge about networking do new hires lack? How does one get comfortable with networking? I'm an undergrad, and my plan for that topic was to learn how to read pcap files and also learn to use wireshark to monitor my network.

it's all so ad-hoc and random. Firewalls have expensive and cheap operations...passing a packet? Cheap. Opening the packet and rewriting the header? Expensive.

That can be leveraged for evil.

How do you even organize that kind of random trivia?

> How do you even organize that kind of random trivia?

Having been slowed down in the past by this kind of thinking: My advice is don't.

You throw the information together and forget about organization (as in a final presentable organization), you just write. Use systems like tags (metadata) to attach concepts and use queries on the tags to find related things. As you build up the knowledge base you can start organizing it more properly, into real essays or chapters and sections. Not all knowledge is amenable to pre-organization before you begin working, and even if you try to have some initial organization as you start writing you will find missing details and then have to fit them in.

I don't know much about firewalls. I wasn't aware they could alter traffic, for example. What do you mean it can be leveraged for evil?

As for organization, I just dump everything I learn in a markdown file on that topic, then look for structure later. So I would just open a file, call it `firewalls.md` and write that down in there. You can accrete a lot of written knowledge over time this way.

They can be used for all types of stuff.

Imagine two companies merging. They both used 10.100.0.0/16. If you were in one of the companies wouldn't it be nice if the other re-ip'd?

... Guess what.. they won't and you don't want to. So you make a box, that turns your ips into 10.101.0.0/16 and theirs into 10.102.0.0/16... and you leave your IPs alone :).

Yeah, some things won't work across that bridge. But it'll get you started.

Firewalls can do really, really evil stuff. DNS? Yeah, we'll answer ALL dns queries.. Even ones you send to 8.8.8.8, or 1.1.1.1...

The BOFH playbook is large, and varied. :)

As a retired sysadmin with 30 years experience, I can tell you with a good amount of certainty that nobody will care if they aren't already sysadmins. It is viewed as entirely unimportant to understand these things, and the only thing you need to get a job is good whiteboard/leet coding skills.

The world changed.

It's unimportant to a certain kind of IT person that's always been out there. But it's useful to people that are curious on how things work...I really don't think those people went away.

I’m not a sysadmin, but want to at least read the contents page - after 20 years full-stack dev I’m painfully aware there’s a lot of things I don’t understand lower down the stack.

What is a "full stack" developer? This is something that always bothered me...

A developer that writes Javascript and sometimes runs it in Node as a server process or as a desktop app in Electron, but actually doesn’t know the /full/ stack at all.