7

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contribution...

 2 years ago
source link: https://lwn.net/Articles/900671/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

[Posted July 11, 2022 by jake]

On his blog, Armin Ronacher comments about a recent security key giveaway by the Python Package Index (PyPI) to provide two-factor authentication (2FA) tokens to the maintainers of the "critical" projects on the index. While (eventually) requiring maintainers to use 2FA before being able to update PyPI packages is reasonable, Ronacher worries about where the idea might lead:

Instead of putting the burden to the user of packages, we're now piling stuff onto the developer who already puts their own labor and time into it. From the index' point of view there is a benefit to not enforce rules on everybody as some of these rules might make the use of the index burdensome, but putting the burden only on critical packages does not hurt the adoption just as much. As mentioned earlier I would not make the case that 2FA is not burdensome, it's a sensible thing. But clearly the index considers it burdensome enough to not enforce it for everybody. More importantly though is what could come next.

There is a hypothetical future where the rules tighten. One could imagine that an index would like to enforce cryptographic signing of newly released packages. Or the index wants to enable reclaiming of critical packages if the author does not respond or do bad things with the package. For instance a critical package being unpublished is a problem for the ecosystem. One could imagine a situation where in that case the Index maintainers take over the record of that package on the index to undo the damage. Likewise it's more than imaginable that an index of the future will require packages to enforce a minimum standard for critical packages such as a certain SLO for responding to critical incoming requests (security, trademark laws etc.).


(Log in to post comments)

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 11, 2022 13:45 UTC (Mon) by bluca (subscriber, #118303) [Link]

One more article to file under the "but why are distributions perceived to be safer to use than language-specific forges?" directory - all of these things are taken for granted and entirely missing on PyPI (and other language forges):

> One could imagine that an index would like to enforce cryptographic signing of newly released packages. Or the index wants to enable reclaiming of critical packages if the author does not respond or do bad things with the package. For instance a critical package being unpublished is a problem for the ecosystem. One could imagine a situation where in that case the Index maintainers take over the record of that package on the index to undo the damage. Likewise it's more than imaginable that an index of the future will require packages to enforce a minimum standard for critical packages such as a certain SLO for responding to critical incoming requests (security, trademark laws etc.).

And the fact that they are missing means the burden is on each and every user. Which means that the same work is repeated many times, if it is done at all. A large corporation can afford to sink bajillion dollars in making the supply chain ending at these language forges somewhat sane, but can the same be said for small companies and projects?

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 11, 2022 14:01 UTC (Mon) by atnot (subscriber, #124910) [Link]

> Likewise it's more than imaginable that an index of the future will require packages to enforce a minimum standard for critical packages a certain SLO for responding to critical incoming requests

If one looks at recent happenings in the open-source world, that doesn't seem very far off at all. See for example this recent news item[1] in response to a call by people at Google for extreme measures including complete legal registration(!!!) of everyone working on software that *they* have happened to *decide* to use in a critical context[2].

[1] https://lwn.net/Articles/866909/
[2] https://security.googleblog.com/2021/02/know-prevent-fix-...

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 11, 2022 14:38 UTC (Mon) by nim-nim (subscriber, #34454) [Link]

Direct distribution via “dev-friendly” ecosystem-specific packaging systems just means distribution-related burdens are pushed on devs, since there are no middlemen left.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 11, 2022 15:09 UTC (Mon) by pizza (subscriber, #46) [Link]

Yeah, I never understood this "waah, getting our stuff into distrubutions is too haaaaard" attitude; take away the work that distributions do, and the developer's workload _increases_ in all but the most trivial cases.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 12, 2022 2:20 UTC (Tue) by notriddle (subscriber, #130608) [Link]

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 11, 2022 14:49 UTC (Mon) by tialaramex (subscriber, #21167) [Link]

Mmm. I think the reason they picked 1% "critical" is that nobody wanted to give them money for hundreds of thousands of FIDO Security Keys.

The thing about the Security Keys, which PyPI doesn't make a big deal of in their announcement - but will be apparent to people who have them - is that they're _much_ less burdensome than approaches like TOTP as well as obviously delivering better security properties. You just tap the Security Key to have it signify that yup, you are still you, done. There are no codes to manually transcribe or copy-paste, there's no step where you, the poor human, are given responsibility for inhuman tasks like correctly matching text strings (is this lwn.net or lvvn.net? My typeface could make those look similar).

So this is actually much less friction (at least if you aren't Russian or otherwise unable to take advantage of their offer) than just switching on TOTP for these "critical" packages.

Suppose PyPI had decided to instead spend this money on having humans audit top-1% critical package changes. Briefly that's better, perhaps, than 2FA but I don't think the money would last long, and of course once the money runs out you go back to the same state as before.

It's actually disappointing to me that most organisations don't give these things (FIDO Security Keys) away, at my own employer I raised this, but it got shot down in a sort of "I don't understand why this is worth having" way that did not give an improved impression of senior management's security know-how. Actually all I wanted was the option to use the Security Keys I already own instead of TOTP, but they were resistant to that idea too.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 11, 2022 16:25 UTC (Mon) by NYKevin (subscriber, #129325) [Link]

At Google, we actually go further than merely giving away keys. If you want to log into your corp account, you must use a security key. No key, no login. IT will issue you a key on day 1, and you can easily get extras if you need them. To my understanding, this has dramatically reduced if not eliminated phishing of corp credentials. I really wish other companies would do that.

If you have a personal Google account, you can apply similar restrictions to that account, but you unfortunately have to come up with the hardware yourself: https://landing.google.com/advancedprotection/

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 11, 2022 17:55 UTC (Mon) by iabervon (subscriber, #722) [Link]

I think it makes sense to have additional effort required to be a package that people are going to rely on and urgently need help with if they find problems, as compared to just being able to develop and share code. Put another way, there should be some work that developers can refuse to do if they don't want to take on ongoing responsibility, so that their projects don't become critical projects if they want to be done with working on the project when the code meets their own needs.

Really, the question is: "Do you want to get panicked emails on weekends about this code, or do you want to let someone else sign up to get panicked emails about this code, or do you want to tell people not to use this code if they might write panicked emails?" These are all possibilities an original author might reasonably choose, and the choice should influence how the code gets used by other people and what policies get built around the project.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 11, 2022 18:16 UTC (Mon) by farnz (subscriber, #17727) [Link]

The problem right now is that various open source packages have evolved organically into packages that people rely on, and thus some developers are being told "hey, this thing you published? It's critical infrastructure now, and we need you to stop treating it like a spare time project and start treating it like something that we all depend on". And this is a problem, because some of those developers don't want to act like it's critical infrastructure - and definitely don't want to do that if they're not being paid for it - but if we simply rip their packages out of the ecosystem, we destroy the critical infrastructure.

I don't, by the way, have a good answer to this - there's lots of bad answers, but the fundamental is that we've taken dependencies in critical infrastructure on code that no-one is willing to maintain as critical infrastructure, and now we're paying the price of not thinking about maintenance of those dependencies up-front.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 11, 2022 18:49 UTC (Mon) by bluca (subscriber, #118303) [Link]

If only we had some projects that volunteer (or get payed to) curate upstream software and distribute it to users, and take all these burdens away from the developers. Imagine how nice that would be. Unfortunately I can't think of anything like that!

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 11, 2022 22:19 UTC (Mon) by iabervon (subscriber, #722) [Link]

Really, I think the question is: why can't the Red Hat or Debian maintainers of packages of Python code maintain the package on PyPI, ideally together with each other and people who review it for distribution to macOS-based computers? The process of making an RPM or .deb could be completely automated, based on a PyPI version having a signature from the Red Hat or Debian maintainer, and the Red Hat or Debian version of pip could care about whether there's a signature available from that distro's maintainer for the PyPI packages.

It makes sense that Red Hat and Debian would have differences between their packages for util-linux or gstreamer, reflecting different kernel features and decisions about backends, but Python library packages would essentially come down to whether version 6.0.1 or 5.8.20 is what's gone through their QA (that is, is the distro seeing compatibility problems with 6.x leading to using a 5.x version with backports, or is the distro happy with 6.x for all dependent packages, such that they've stopped bothering to vet 5.x versions).

Furthermore, there are other groups that would have good guidance, such as the IT support staff for the biology department at a university, who can test whether a new version of a library works as a dependency for a program that processes brain images, or whether it'll corrupt files if you try to do a demo on an Apple laptop at a conference.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 11, 2022 22:51 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

My experience is that PyPI is fine for shipping packages which only have Python interfaces and falls flat for anything else. C APIs tend to get stuffed into some private directory in a way that is no better than any other "bundle my dependencies" way.

For example, there is `h5py` which ships an embedded HDF5 with no headers or anything. How is a project that wants to collaborate with it over the HDF5 C API supposed to work in this situation? I don't know how much it improves things since distros are basically going to have some subset provided by their build infrastructure and any pure Python from PyPI. I suspect such a hybrid setup would be more difficult.

Instead, it might be better to have the distro provide some index for "we provide these packages" that `pip` can use as "just another PyPI" repository instead of trying to wholesale replace PyPI itself. It'd be nicer if distro packages didn't require root to work, but that's a separate problem.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 11, 2022 23:01 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link]

Really, I think the question is: why can't the Red Hat or Debian maintainers of packages of Python code maintain the package on PyPI, ideally together with each other and people who review it for distribution to macOS-based computers?

My impression from what others have said on this topic is that a lot of the "maintainers of packages" are packagers first and maintainers of the code a distant second. They tend to see their function as taking upstream code, fixing any way that it doesn't meet the project packaging guidelines, and converting it into a package for their distribution. They can also serve as a conduit for upstreaming bug reports. They don't really review the code except to make sure it follows packaging guidelines, like not including vendored libraries and putting files in the right locations. Note that this isn't restricted to the people packaging programming libraries like PyPI; there's a ton of other software where the packager really just converts upstream code into the right packaging format.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 12, 2022 3:39 UTC (Tue) by pabs (subscriber, #43278) [Link]

As a maintainer of Python packages in Debian, I want to base Debian packages on upstream git, *not* on PyPI, because of things like missing files, pre-generated files and so on that are present in PyPI source tarballs but not the upstream VCS. We already have tools for automated packaging of Python and other ecosystems, but not everything can be easily automated and some stuff like reviewing code changes is not possible to automate.

https://hamburg-2022.mini.debconf.org/talks/5-automating-debian-packaging/
https://wiki.debian.org/AutomaticPackagingTools
https://wiki.debian.org/CopyrightReviewTools

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 12, 2022 3:39 UTC (Tue) by timrichardson (subscriber, #72836) [Link]

This ship has sailed, but anyway: traditional linux package distribution was not designed to solve supply chain security risks. It solves dependency management and binary distribution, and in doing so it introduces barriers to distribution which limit availability of projects and which limit updates of projects, both of which cause their own serious problems. Your sarcasm is I suppose frustration at losing the argument.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 11, 2022 22:42 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link]

One possible solution to this problem is to be upfront with users about the status of packages they might consider turning into critical infrastructure. At the very beginning, developers could express their level of interest in long-term maintenance of the project, and this would be given a prominent place on the project page. Developers who are considering using the package would then be able to base their decisions on the level of support promised. Some possible support levels:

  1. This is a toy I am sharing because I thought other people might find it amusing
  2. I don't intend to support this project in the future, but I'd be happy if someone else took it over
  3. I don't intend to support this project in the future, but I will help get someone else up to speed if they want to take it over
  4. I intend to support this project as a hobby
  5. I intend to support this project if someone is willing to pay me to do so
  6. My employer has promised to pay for my time supporting this project
  7. This project is supported by a team, so it is not vulnerable to a single programmer leaving
  8. This project is considered critical infrastructure by a major supporter who has promised to maintain support
  9. This project is part of the language's core infrastructure and will not be dropped from the standard distribution without deprecation notices and a proposed replacement

With that kind of notice, developers would be able to come up with their own standard for deciding what level of support they want from anything they're going to use in their project. If you want to develop critical infrastructure, you either need to build on projects that have a reasonable level of promised support or be willing to take over maintenance if/when the existing developer decides to give them up.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 12, 2022 1:04 UTC (Tue) by pizza (subscriber, #46) [Link]

The problem with your "support progression" is that having to declare it in advance means nearly nothing will ever escape (1) or (5). Because who in their right mind would ever commit to anything more than that, in advance?

After all, the difference between "toy" and "mission critical" is determined by the user, which can be many, many steps removed from the original author. Who could have been hit by a truck since they promised to fully support everything, for free, forever.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 12, 2022 7:55 UTC (Tue) by NYKevin (subscriber, #129325) [Link]

I do believe that was the point of rgmoore's proposal: Lots of stuff is *already* stuck at (1) or (5) under the status quo, but we don't have that information recorded anywhere. If we told people "this item is a (1)" upfront, then they'd be able to use that information to make a rational decision about whether it's really a good idea to take this particular library as a dependency. When, subsequently, bugfixes or other improvements are desired, the downstream project knows perfectly well that it has to fork the library and make the fix itself, and need not waste time agonizing over whether to rip the dependency out or try to convince the old maintainer to come back and write more code for free.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 12, 2022 3:23 UTC (Tue) by dkg (subscriber, #55359) [Link]

I like the idea of this, even though i'm not sure that people's *intent* is a sufficient piece of information to base such a dependency decision on.

Even more daunting, though, is what "support" actually means. There are many developers whose idea of "support" is "if you run the latest version and you find a bug i'll fix it in the next version."

That's not bad -- and it's certainly better than "here's a fun toy, knock yourself out." But it doesn't come close to anything like "we promise to provide API (or ABI) stability for at least three years from each major release, while fixing all reported security vulnerabilities for each supported major release."

When a project is one piece of a larger toolchain, particularly a system-critical toolchain, this kind of stability is what's really needed. Sometimes I feel lucky if an upstream developer even understands when they're breaking the API (or ABI) to introduce $COOL_NEW_FEATURE, let alone committing to backporting security fixes to older variants, and not introducing bugs alongside the new features when they don't break the API. ☹

And i should say: i get it! API (or ABI) stability is a subtle concept, implementation is actually even more subtle, and stable maintenance across multiple branches of development feels more like drudgery and bookkeeping than the fun and sexy hacking you get to do when building out a new feature.

How many software projects have this kind of explicit commitment to sober, stable, boring support?

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 12, 2022 9:38 UTC (Tue) by farnz (subscriber, #17727) [Link]

We don't even have the tooling to make that sort of sober boring support easy; the only two projects I can think of off the top of my head that make that level of commitment to ABI stability are the Linux kernel (but only the parts accessible via syscall ABI, not internal APIs), and glibc, which has complex build scripts (mix of Makefiles and awk) to detect obvious ABI changes and apply symbol versioning to make it possible to retain stability.

Given the lack of easy-to-use tooling to detect a build-breaking API change, it's no surprise that projects don't do a good job of it - I can't in any mainstream OSS language just run a tool and have it detect all cases where exported APIs have changed such that a project depending on me will no longer build, and that's table stakes for stable APIs.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 12, 2022 3:42 UTC (Tue) by pabs (subscriber, #43278) [Link]

You can kind of do that on GitHub by not enabling issues, not enabling discussions, not enabling the security tab etc or just archiving the project. The only thing you can't easily do is have an actively developed project that doesn't accept external contributions.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 12, 2022 3:46 UTC (Tue) by timrichardson (subscriber, #72836) [Link]

The fundamental question is should the package provider better information to allow users to make informed decisions, or should pypi impose solutions. Maybe both. To your point, there is already this: https://libraries.io/pypi/requests which attempts on objective measurement.

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 12, 2022 3:53 UTC (Tue) by timrichardson (subscriber, #72836) [Link]

James Bennett (django) posted this in response, where he observes that most significant publishers to PyPi are also consumers of other packages, and that there are social obligations on publishers which other publishers implicitly take advantage of; he rates his opinion as at the other end of the spectrum to Armin Ronacher. It's a good read, and we wouldn't have it without Ronacher's post.

https://www.b-list.org/weblog/2022/jul/11/pypi/

Ronacher: Congratulations: We Now Have Opinions on Your Open Source Contributions

Posted Jul 11, 2022 21:00 UTC (Mon) by martin.langhoff (subscriber, #61417) [Link]

This is where "contribute to a large community-maintained library or framework" is _so_ much better than "create your own super clever small library - that if it gets popular, you're on your own to maintain, grow community around, etc".

The cultural norms of "lone individual library marker" had their moment, but as the OSS community and stack matures, it's all about responsibility. Burden, if you will. A community of maintainers, with code review, etc is _so_ much better.

Redemption by Tool

Posted Jul 12, 2022 5:03 UTC (Tue) by oldtomas (guest, #72579) [Link]

There's an antipattern in our industry I call "Redemption by Tool".

The current run towards 2FA fits that antipattern perfectly. Perhaps Google, Facebook et al are promoting it somewhat (or surfing on it, I don't know). After all, their business model relies on regarding people as cattle. Taking the individual person out of the security equation should be a significant cost-saver there.

Me? I think security *involves* people. First would be that each person understands their security situation as well as possible. Second, they decide whether they go with a 2FA dongle, an encrypted key file on their device or whatever else.

But then you can't regard them as cattle.


About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK