Fresh pcaps, free for the asking

Fresh pcaps, free for the asking

One thing network people hate is having to get access to a machine (or groups of them) in order to run tools like tcpdump. They want a packet trace from somewhere, and having to obtain not just ssh access but sufficient powers to open a raw socket and start sniffing (typically root-level) is a pain. It should be unsurprising that someone eventually wrote a tool which solved the problem for them.

How did it work? Well, the machines in question (all of them) were already running some "agent" software which logged interesting things about the network activity of each host. It had root powers on the machines, and took requests from the network. Adding the packet trace capability was just a matter of extending what was already there.

The code was written, it was attached to the service, and it was shipped with little fanfare. It just kind of hung around after that point, doing its job while presumably reducing some of the more annoying angles for the humans.

Then, one Friday afternoon, rather late, someone decided it was time to post one of those feel-good announcements about what they had accomplished. This was quite a while after it had shipped, but maybe performance reviews were coming up? Nobody's really sure why it was announced at that point, or why it was late on a Friday, but that's what happened.

A few minutes later, one of those people who looks at everything and thinks "is that safe, or could someone hurt themselves or others with it" saw the post and the wheels started turning. Instead of going and digging in to the code (which would require pulling out the laptop and getting back online), they stayed on their little mobile app and replied to the post, instead. They asked some questions: who has access to this, and how is that enforced, exactly?

The hope was to get an answer back like "the network team, specifically anyone in LDAP group X" for the first part, and "we use a (tech) (tech) in the (tech) (tech) part of (tech) to verify their identity". However, unlike what happens with Star Trek scripts, the "tech" items here would actually be appropriate and solid for the task at hand. They'd look even smarter and people would feel good about the whole thing: a tool that saves time and doesn't open more holes!

The answer which came back, to put it charitably, fell short.

"We went and looked... turns out... everyone."

They had added the method to the service but hadn't added any authentication to determine who was making the request, or authorization to make sure they had the right to make it. As a result, anyone could switch it on and get tasty, tasty packet traces from any box in the fleet.

All you needed to do was send a properly-formatted network packet, and it would do the work for you. Want to see what's happening on a sensitive system that handles money, or HR data, or really, anything else run on these machines? Turn on a packet trace and hope they "forgot" to encrypt their traffic in flight.

As you probably guessed, this was unacceptable, and so, that Friday evening, the team which produced the service, the security responders, and that engineer who tends to look for the holes in things didn't get to knock off work for the weekend. Instead they had to figure out how to turn it off (even if that meant breaking things which relied on it), get that pushed out everywhere ASAP, and then see if they can leave it like that until Monday, or if they have to actually solve for the identity and permissions stuff right then and there.

This also meant people had to go digging to (try to) see if it had already been exploited. Not everyone waits for the announcement. Some people just go around thinking "hmm, what's already listening on the network and has interesting powers and curious-sounding RPC methods?", and those people would be way ahead of you.

Finally, there's the human angle to this. Obviously, a SEV was opened for this. The "user impact" field was written to be something like this by the engineer who discovered it:

Any host running (tech) since (commit hash) which was released (month) (year) would have provided packet dumps to anyone who asked politely via (tech).

Someone (who had worked on the service) couldn't leave it alone. They had to edit it. They added this:

It is perceived to have security risks associated with such data export.

In other words, "someone annoying thought this is a big deal and that's why we're working late tonight and into the weekend".

Side note: the code change which introduced this had a comment from one person saying "you know this might be a bad idea, right?"... but it was committed and shipped anyway. Oh, random commenter, if only they had listened to you.