CameraX is a Jetpack support library that is built to help make camera app development easier. It supports different use cases such as ImageCapture, Preview and ImageAnalysis, which could seamlessly combine with ML Kit or TensorFlow Lite. This opens up a lot of possibilities to create applications like text recognition, image labeling, and even detecting and recognizing objects using your own trained TensorFlow Lite model. However, image format conversion between CameraX and these libraries is non-trivial and sometimes time-consuming. In this blog, we will talk about a new feature we recently built to support YUV to RGB conversion for CameraX ImageAnalysis, including why we built it and how to use it with just a small amount of code.

Background

CameraX produces images with YUV420_888 with Luma (Y), Chroma (U, V) and Paddings (P) with 8 bits for each channel. YUV is a generic and flexible format, it allows for OEM variations on different devices, which covers a lot of ImageAnalysis use cases. However, many apps still rely on the RGB format. In our developer community, YUV to RGB conversion is one of the most highly-requested features, because RGB format is popular, easy to work with, and is sometimes required, such as with TensorFlow Lite models). Let’s take a look at the YUV and RGB formats first.

YUV_420_888 format

YUV format can also be referred to as “YCbCr”. It includes planar (e.g. I420), semi-planar (e.g. NV21/NV12) and packed formats (e.g. UYVY). YUV_420_888 is a generic YCbCr format, capable of describing any 4:2:0 chroma-subsampled planar or semi-planar buffer (but not fully interleaved), with 8 bits per color sample. The Y-plane is guaranteed not to be interleaved with the U/V planes (in particular, pixel stride is always 1). The U/V planes are guaranteed to have the same row stride and pixel stride.

RGBA_8888 format

RGBA_8888 is a standard RGB format with red, green, blue, and alpha channels, and each channel has 8 bits. The major goal of the conversion is RGB color space. RGB is simpler with fewer variations.

API Implementation

We have evaluated 3 approaches for YUV to RGB conversion

1. Java/Kotlin
2. Renderscript
3. Native (C/C++ and NDK)

A Java/Kotlin implementation of image processing comes with long computation time and garbage collection pressure. Renderscript is a candidate for computationally-intensive tasks such as YUV to RGB. However, it has been deprecated starting from Android 12.

Considering future extensibility and compatibility, we decide to use a native approach (libyuv + NDK). Libyuv is an open source project that includes YUV scaling, conversion, and rotation functionality. With all things considered, CameraX color conversion pipeline can be viewed at a high level as follows:

Our pipeline still produces ImageProxy as the output for backward compatibility. ImageProxy is a wrapper class for media.Image, which is an image buffer produced by the Android framework. To produce a converted Image, the Java/Kotlin layer can get an input Image from a Surface via dequeueInputImage() and then write Image data into it using ImageReader and ImageWriter. Since ImageWriter was added in API 23, we use ANativeWindow and its buffer to produce output images in RGBA format to support more API levels.

For input, we support different variations of YUV_420_888 formats (I420, NV12, NV21, etc.) internally in CameraX. For output, we support RGBA format now but could extend support to other RGB formats in the future.

Since we use libyuv as a new dependency, the size of our library increases by about 50 KB.

API Usage

Starting with CameraX 1.1.0-alpha08, apps can choose the output image in YUV_420_888 or RGBA_8888 format by using setOutputImageFormat in an ImageAnalysis configuration.

Once RGBA_8888 is selected, the output image format will be PixelFormat.RGBA_8888, which has only one image plane (R, G, B, A pixel by pixel) with paddings. Traditionally, the Android Framework supports image buffer formats with a subset of both PixelFormat and ImageFormat.

By comparison, if YUV_420_888 is selected, the output image will be ImageFormat.YUV_420_888, which has 3 separate image planes (Y, U, V).

Performance

We measured the speed and compared the results with those of Renderscript on different Android versions and devices. Overall, our pipeline with libyuv outperforms our reference Renderscript implementation on different devices for different resolutions and Android OS versions.