I think the specification should use some improvement here.

In particular (and as you know), the algorithm for outsetting a corner with a radius and the algorithm for insetting a corner with a radius are not symmetric. This is by design. Because of this, we should be careful to apply the relevant algorithm (outset or inset) exactly once, at least most of the time. However, it's possible that for an outline or a border, we should compute one of the edges (inner and outer) of the outline or border from the other. (For borders, this is always the case, because the starting point of these computations is the curvature of the outer edge of the border.)

I don't see anything in the current specification that defines this rules for outsetting and insetting curves for outlines.

I think we should probably fix the specification to more clearly define these rules for outset/inset of curves as an algorithm that can be referenced by elsewhere in the relevant specifications, and then reference this algorithm more clearly to do what is needed.

The parts of the outline are not required to be rectangular. To the extent that the outline follows the border edge, it should follow the border-radius curve.

I take this as implying that outlines should use the typical algorithms for outsetting or insetting rounded corners, but yeah it should be clearer.

I guess the problem you were referring to is that, especially with thick outlines, it may happen that the outer edge of the outline is outside the border, while the inner edge is inside due to a negative outline-offset. In that case, should the outer edge use the offset algorithm, and the inner edge use the inset algorithm? Should we only use one algorithm for one of the edges and derive the other from the former?

This illustrates what would happen when using the currently specced algorithm for outset for the outer outline edge, and the inset algorithm for the inner outline edge. So the cyan areas would be the outline, and the thin black line shows the border area (in reality it would be covered by the outline).

I don't think we want this to happen (deriving the inner outline edge from the outer one):

The most extreme cases of the problem are certainly ones where the outer edge of the outline is outside the border edge and the inner edge of the outline is inside the border edge -- but the question of what is derived from what still matters for other cases as well.

As depicted in my last comment, deriving the inner outline edge from the outer can look quite wrong.

Deriving the outer outline edge from the inner one doesn't seem great either: if the inner edge is inside the element, its radius may have been shrunk to 0, and then the outer edge wouldn't be rounded either.

So I think it will look best if both the inner and the outer edges are derived directly from the border edge.

There's still the question of how to ensure that the outline remains visible when outline-offset is very negative. I guess we should floor by the infimum of the offsets that would result in a positive area within the inner outline edge, but computing this value doesn't seem trivial in general.

I think the specification should use some improvement here.

It sure could be more specific, but to some degree, keeping it very open ended was deliberate. I seem to remember that @tantek was a fairly strong advocate for that. The reasoning being that outlines being used for UI purposes warranted giving the UA room to innovate as to what was the best UI for such things.

I would tend to think that giving the UA a lot of leeway for outline-style: auto, which tends to be used for UI purposes, is a good idea and leaves the rendering largely undefined anyway, but that for other values, given that the appearance is author controlled anyway, we should rather aim for a precise definition and and interoperability. In which case, this issue is just one part of a bigger chunk of work, because outline rendering is all very vague at the moment.