[OpenGL and Vulkan Interoperability on Linux] Part 10: Reusing a Vulkan stencil buffer from OpenGL

This is 10th post on OpenGL and Vulkan interoperability with EXT_external_objects and EXT_external_objects_fd. We’ll see the last use case I’ve written for Piglit to test the extensions implementation on various mesa drivers as part of my work for Igalia. In this test a stencil buffer is allocated and filled with a pattern by Vulkan and then it is used in OpenGL to render something else. We validate that the pattern has been imported correctly and we repeat the process for other depth-stencil formats.

vk-stencil-display test is implemented in tests/specs/ext_external_objects/vk_stencil_display.c inside Piglit.

The code is not upstream at the time of writing this blog post but it can be found in piglit merge request MR!330.

In vk-stencil-display test:

• An external Vulkan stencil buffer is used in a Vulkan renderpass and is filled with a recognizable pattern.
• Then, it gets imported into OpenGL and used to render a fullscreen quad with the stencil test enabled.
• We validate that import was successful: if so, the OpenGL quad must be masked by the stencil pattern.
• The process is repeated for different depth/stencil formats.

Step 1: An external stencil buffer is allocated and used in a Vulkan renderpass

We’ve seen how we allocate external images from Vulkan in Parts 2 and 3.

As in this test we check different depth|stencil formats, we have a subtest per format where we allocate external Vulkan images and use them in OpenGL rendering. Everything is implemented in run_subtest, which is very similar to what we’ve seen in Part 9 of these blog posts.

The images that are used in each subtests are allocated in vk_subtest_init (see code snippet below) that is called per test after we initialize some common Vulkan and OpenGL structs in piglit_init. The external allocation has already been explained in Parts 2 and 3. This step is also quite similar to the first step of Part 9 (where we had enabled the depth test in each renderer).

To use the stencil buffer in the Vulkan renderpass we need to enable the stencil test in each renderer so that it gets enabled in the pipeline and renderpass. For that we set the enable_stencil value to true in vk_create_renderer.

Setting enable_stencil to true when creating the renderer enables some extra settings of the Vulkan renderpass/pipeline (see tests/spec/ext_external_objects/vk.[hc]):

What we want to achieve at the end is to fill the stencil buffer where we have rendered geometry.

And so, we need to:

1. enable the stencil test to be able to write stencil values to the depth/stencil buffer: ds_info.stencilTestEnable = VK_TRUE;.
2. disable the depth test: ds_info.depthTestEnable = VK_FALSE; (we can’t write depth and stencil values at the same time in the buffer).
3. enable depth writes: ds_info.depthWriteEnable = VK_TRUE;. In other words allow writing to the “depth” buffer which happens to be the depth/stencil buffer. The name is a bit confusing but we need to enable this setting to write stencil values.

With the rest of the settings we make sure that the stencil test runs for all pixels (VK_COMPARE_OP_ALWAYS), and when we render a pixel the stencil value in the stencil buffer gets replaced (VK_STENCIL_OP_REPLACE) with a reference value (1). Compare and write masks (0xffffffff) select the bits of the unsigned integer stencil values participating in the stencil test and the values updated by the stencil test in the framebuffer attachment. In this test we want all bits to be updated before the buffer gets imported.

We initialize the stencil buffer to 0 (check vk_draw in vk.[hc]). Let’s see how the stencil pattern is created by calling vk_draw:

The Vulkan renderpass uses the same pair of vertex and pixel shaders we’ve seen in Part 9 when we exchanged the depth buffer:

and FS:

We render a quad in depth 0.33 and we fill it with some (not important) color. The quad in depth 0.33 doesn’t cover the whole framebuffer. If we could display it it would appear like a small quad in the middle. As the stencil test runs for all pixels we draw and replaces the equivalent stencil buffer values (that are 0 because of the clear value) with one, the stencil pattern at the end of the renderpass will be a framebuffer filled with 0 everywhere and 1 in the middle. Something like this:

After it is filled we can try to import the buffer in OpenGL. Note that we used the depth attachment as there’s no separate “stencil attachment”. Depth and stencil share the same buffer and the attachment is named after the depth.

Step 2: The external stencil buffer gets imported and used by OpenGL to render something new

Importing the external stencil buffer is similar to the imports we’ve seen so far in Parts 2 and 8: we first create a memory object from Vulkan memory and then a texture of that memory object:

Then, we need to render something that can help us test if the import was successful. And so,

• We render a blue fullscreen quad with the stencil test enabled and using the stencil buffer as stencil attachment to the target framebuffer.
• We display the result with the depth and stencil tests disabled. We expect to see the blue quad being masked by the stencil pattern (blue where stencil has ones and green where it has zeros).

To render with OpenGL using the Vulkan stencil buffer we need to attach it to an OpenGL FBO that will serve as the OpenGL render target. We’ve created such an FBO in gl_init (called from piglit_init), and so we can call gl_subtest_init per subtest to attach each subtest’s depth/stencil image and then render a quad to the color attachment with the stencil test enabled and a stencil function that runs for all pixels and only writes them when the stencil value is one: (glStencilFunc(GL_EQUAL, 1, 0xFFFFFFFF);). The goal is to visualize the stencil pattern we’ve seen above using blue (the color we paint geometry) where we have rendered pixels (which is where Vulkan filled the external stencil buffer with ones) and green (framebuffer color) where we haven’t rendered (which is where Vulkan let the pixels have the value they took at initialization which is the clear value and is zero):

After that, we must display the result of rendering with the imported stencil buffer, to validate the result and tell if the stencil import was successful.

We map the framebuffer and we disable the depth and stencil tests. Then we map the color attachment of the OpenGL FBO on a piglit quad and we check if it contains a small quad inside a blue quad by probing the color in a few pixel positions:

And that’s it. If the drivers had correctly exported and imported the stencil attachment, piglit_present_results will display the green/blue pattern from the beginning of this post.

Closing, in this test we’ve seen how to reuse an external Vulkan stencil buffer from OpenGL. I’ve used a Piglit test I’ve written to check our EXT_external_objects and EXT_external_objects_fd extensions implementations in various Mesa3D drivers as an example.

That was the final EXT_external_objects(_fd) use case I’ve written for Piglit to test different mesa drivers implementations of the feature and there won’t be a follow-up example unless we see that we need to write more tests. But I might blog post about the updated status of the Mesa3d drivers implementations at some point in this category, as most of our WIP and under review branches I’ve mentioned in my XDC presentation are now upstream.