The dynamic range of a scene in the real world exceeds camera ability to perceive due to photon response limitation of CMOS sensor. The most common solution is to use multi-exposures to capture different scene dynamic ranges and stitch them into a wider dynamic range.

In camera solutions, 4-exposure 24-bit High Dynamic Range (HDR) theoretically is more expensive and better than 3-exposure 20-bit HDR. However, the processed result of 4-exposure HDR seems worse than 3-exposure HDR on tone reproduction if sensor configuration and ISP tuning are not managed correctly.

This article describes:

• The difference between 3-exposure HDR content and 4-exposure HDR content
• How the Mali Image Signal Processor (ISP), especially Iridix, the local tone mapping block, processes them differently

Analysis

Figure 1 shows how 3-exposure HDR and 4-exposure HDR content are combined in the sensor side:

• Long exposure captures the darkest intensities.
• The medium exposure captures the dark-medium intensities.
• The short exposure captures the medium-high intensities.
• The very short exposures (for 4-exposure only) capture the brightest intensities.

These exposures are normalized to 20-bit or 24-bit, depending on the supported bit depth of 3-exposure or 4-exposure.

Figure 1:  4-exposure HDR data intensity response and linearization

Note: The most advanced sensors use different architectures to control the sensitivity of the 4 exposures: dual conversion gain and/or split pixel. In this generalized example, we use names like Long and Short to explain the relative sensitivity (exposure ratio), but the integration time is not the only factor.

• Ratio_lm stands for the ratio between long and medium exposures
• Ratio_ms stands for the ratio between medium and short exposures
• Ratio_svs stands for the ratio between short and very short exposures

Mali ISPs, such as Mali-C71 and Mali-C78, have 24-bit data width to process the imaging data with MSB alignment:

• For a 3-exposure 20-bit HDR data input, the ISP left shifts 4bit to align the input to 24-bit
• For 4-exposure 24-bit HDR data input, the ISP matches the input depth well and performs no shift operation.

As a result, when you have same long exposures, 3-exposure 20-bit HDR mode looks 16x brighter than 4-exposure 24-bit HDR mode. Figure 2 and Figure 3 show such brightness difference between a 4-exposure 24-bit image and a 3 exposure 20-bit image.

Figure 2: 4-exposure 24-bit image (no gamma encoded)

 

 Figure 3: 3-exposure 20-bit image (no gamma encoded)

However, the Signal Noise Ratios (SNR) for the dark pixels are same for both 3-exposures 20-bit and 4-exposures 24-bit because of the same long exposures. Applying the digital gains to the dark pixels for 4-exposures does not change the SNR. 

In the ISP, there are two kinds of digital gains:

• Digital gains

Digital gains are applied globally to the whole image that auto-exposure strategy should control these gains carefully to avoid damaging the bright pixels by clipping. In LDR sceneries, applying a lot of digital gains would not be a problem, while in HDR sceneries, applying digital gains would probably break the highlights.

• Iridix gains

Iridix gains are applied locally and adaptively to the image, based on the image content. Generally, more gains for dark pixels and less gains for bright pixels are desired for reproducing an HDR image.

Solution

This section describes how to tune 24-bit HDR data to align the brightness with 20-bit HDR data without breaking the highlights.

Figure 4 shows typical main blocks to process HDR data in the Mali ISPs:

• Gamma_FE is the module to decompand, for example, the 12-bit PWL compressed data to 24-bit linear data for most blocks processing.
• Iridix is the local tone mapping engine to compress the dynamic range from 24-bit to 14bit for the precision management, RGB gamma is the module to mapping from linear to standard gamma to display.

 

Figure 4: typical ISP main blocks to process HDR data

To tune iridix by compressing the dynamic range from 24-bit to 14bit, you can apply a much steeper Asymmetry Curve to amplify the dark regions while we may use moderate asymmetry curve for 20-bit HDR data and increase the dark_enh setting to amplify the dark region in further.

The following table shows the different iridix effect under different combination of asymmetry curve, dark_enh and strength_inroi with 20-bit and 24-bit image data.

 

Figure 5: 3-exposure 20-bit iridix processed result with default asymmetry curve

 

Figure 6: 4-exposure 24-bit iridix processed result with default asymmetry curve

 

Figure 7: 4-exposure 24-bit iridix processed result with steeper asymmetry curve

Conclusion

The Mali ISP can maximize the 24-bit HDR content reproduction with the use of iridix local tone mapping. After the tunning, 24-bit solution has better highlights preservation and the same level dark intensities perception as 20-bit solution.