OpenHarmony相机用户态驱动框架

OpenHarmony相机用户态驱动框架-51CTO.COM

作者：郭新星 2022-04-21 11:26:31

相机作为智能手机上少有的成长空间不错的，能够做出差异化的功能，每年都能成为各大Android手机厂商争相宣传的亮点。

https://ost.51cto.com

相机作为智能手机上少有的成长空间不错的，能够做出差异化的功能，每年都能成为各大Android手机厂商争相宣传的亮点。众所周知Android采用Linux 作为其内核，而Linux采用的开源协议具有传染性[1]，导致Android HAL[2]成为了手机厂商们竞争的重要战场。随着OpenHarmony 3.1[3]的发布，相机模块也逐渐完善起来，目前提供了基础预览和拍照的能力。OpenHarmony中，相机用户态驱动框架承担了和Android Camera HAL一样的角色，这部分位于OpenHarmony的HDF[4]中，对上实现相机HDI[5]接口，对下实现相机Pipeline模型，管理相机各个硬件设备。

相机用户态驱动框架(下图的CameraHost 部分)总体可以分为三层，HDI实现层，实现相机标准南向接口;框架层，对接HDI实现层的控制、流的转发，实现数据通路的搭建、管理相机各个硬件设备等功能;适配层，屏蔽底层芯片和OS差异，支持多平台适配。

HDI Implementation：对上实现HDI接口，向下调用框架层的接口，完成HDI接口任务的转发。

Buffer Manager ：屏蔽不同内存管理的差异，为子系统提供统一的操作接口，同时提供buffer轮转的功能。

Pipeline Core ：解析HCS配置完成pipeline的搭建，调度pipeline中的各个node完成流的处理。

Device Manager：通过调用底层硬件适配层接口，实现查询控制底层设备、枚举监听底层设备的功能。

Platform Adaption ：屏蔽硬件差异，为Device Manager提供统一的操作底层硬件的能力。

Shell
drivers/peripheral/camera
|-- README_zh.md
|-- bundle.json
|-- figures
|   `-- logic-view-of-modules-related-to-this-repository_zh.png
|-- hal
|   |-- BUILD.gn
|   |-- adapter
|   |-- buffer_manager
|   |-- camera.gni
|   |-- device_manager
|   |-- hdi_impl
|   |-- include
|   |-- init
|   |-- pipeline_core
|   |-- test
|   `-- utils
|-- hal_c
|   |-- BUILD.gn
|   |-- camera.gni
|   |-- hdi_cif
|   `-- include
`-- interfaces
    |-- hdi_ipc
    |-- hdi_passthrough
    `-- include

HDI Implementation中的预览流程

接下来我们通过已经发布的OpenHarmony 3.1开源代码，来看看预览是怎么完成的吧。

drivers/peripheral/camera/hal/test/v4l2/src /preview_test.cpp存放了针对v4l2的预览测试代码，入口如下：

C++
TEST_F(UtestPreviewTest, camera_preview_0001)

{
    std::cout << "==========[test log] Preview stream, expected success." << std::endl;
    // Get the stream manager
    display_->AchieveStreamOperator(); // 获取stream operator
    // start stream
    display_->intents = {Camera::PREVIEW}; // 预览流
    display_->StartStream(display_->intents); // 起流
    // Get preview
    display_->StartCapture(display_->streamId_preview, display_->captureId_preview, false, true);
    // release stream
    display_->captureIds = {display_->captureId_preview};
    display_->streamIds = {display_->streamId_preview};
    display_->StopStream(display_->captureIds, display_->streamIds);
}

先获取stream operator实例

C++
void testdisplay::achievestreamoperator()
{
    // create and get streamoperator information
    std::shared_ptr<ohos::camera::istreamoperatorcallback> streamoperatorcallback =
        std::make_shared<ohos::camera::istreamoperatorcallback>();
    rc = cameradevice->getstreamoperator(streamoperatorcallback, streamoperator);
                    // ........
}

通过前文的streamOperator创建流

C++
void TestDisplay::StartStream(std::vector<OHOS::Camera::StreamIntent> intents)
{
    // ..............................
    for (auto& intent : intents) {
        if (intent == 0) {
            std::shared_ptr<IBufferProducer> producer = IBufferProducer::CreateBufferQueue();
            producer->SetQueueSize(8); // 创建buffer的生产端，并和相应的流进行绑定
            auto callback = [this](std::shared_ptr<SurfaceBuffer> Prebuffer) {
                BufferCallback(Prebuffer, preview_mode);
                return;
            };
            producer->SetCallback(callback);
            streamInfo->streamId_ = streamId_preview;
            streamInfo->width_ = 640; // 640:picture width
            streamInfo->height_ = 480; // 480:picture height
            streamInfo->format_ = CAMERA_FORMAT_YUYV_422_PKG;
            streamInfo->datasapce_ = 8; // 8:picture datasapce
            streamInfo->intent_ = intent;
            streamInfo->tunneledMode_ = 5; // 5:tunnel mode
            streamInfo->bufferQueue_ = producer;
            streamInfos.push_back(streamInfo);
        } else if (intent == 1) {
      // .......................
    }
    rc = streamOperator->CreateStreams(streamInfos); // 创建流
    // ................................
    rc = streamOperator->CommitStreams(Camera::NORMAL, ability); // 提交流
    // .................................
}

下面我们正式进入到hal的源代码中看看是怎么创建流的吧。

C++
CamRetCode StreamOperator::CreateStreams(const std::vector<std::shared_ptr<StreamInfo>>& streamInfos)
{
 // .....
    for (auto it : streamInfos) {
//....
        std::shared_ptr<IStream> stream = StreamFactory::Instance().CreateShared(
            IStream::g_availableStreamType[it->intent_], it->streamId_, it->intent_, pipelineCore_, messenger_); // 创建流实例
// ...
        StreamConfiguration scg;
        scg.id = it->streamId_;
        scg.type = it->intent_;
        scg.width = it->width_;
        scg.height = it->height_;
        PixelFormat pf = static_cast<PixelFormat>(it->format_);
        scg.format = BufferAdapter::PixelFormatToCameraFormat(pf);
        scg.dataspace = it->datasapce_;
        scg.tunnelMode = it->tunneledMode_;
        scg.minFrameDuration = it->minFrameDuration_;
        scg.encodeType = it->encodeType_;
        RetCode rc = stream->ConfigStream(scg); // 依据上文的流信息配置流
// ...
        if (it->bufferQueue_ != nullptr) {  // 绑定前文的生产端
            auto tunnel = std::make_shared<StreamTunnel>();
            CHECK_IF_PTR_NULL_RETURN_VALUE(tunnel, INSUFFICIENT_RESOURCES);
            RetCode rc = tunnel->AttachBufferQueue(it->bufferQueue_);
            CHECK_IF_NOT_EQUAL_RETURN_VALUE(rc, RC_OK, INVALID_ARGUMENT);
            if (stream->AttachStreamTunnel(tunnel) != RC_OK) {
                CAMERA_LOGE("attach buffer queue to stream [id = %{public}d] failed", it->streamId_);
                return INVALID_ARGUMENT;
            }
        }
        {
            std::lock_guard<std::mutex> l(streamLock_);
            streamMap_[stream->GetStreamId()] = stream; // 保存流实例
        }
// ...
}

从上面可以看出，消费端传递到了hal，那必然是由hal从bufferproducer获取buffer，并触发预览的启动流程。那看看AttachStreamTunnel 的实现吧。

C++
RetCode StreamBase::AttachStreamTunnel(std::shared_ptr<StreamTunnel>& tunnel)
{
    if (state_ == STREAM_STATE_BUSY || state_ == STREAM_STATE_OFFLINE) {
        return RC_ERROR;
    }
    tunnel_ = tunnel; // 绑定生产端
    CHECK_IF_PTR_NULL_RETURN_VALUE(tunnel_, RC_ERROR);
    tunnel_->SetBufferCount(GetBufferCount()); // 配置轮转的buffer个数
    TunnelConfig config = {(uint32_t)streamConfig_.width, (uint32_t)streamConfig_.height,
        (uint32_t)streamConfig_.format, streamConfig_.usage};
    tunnel_->Config(config);
    streamConfig_.tunnelMode = true;
    return RC_OK;
}

CreateStream之后便是CommitStream，这里的CommitStream 做了些什么事情呢，我们接着往下看。

C++
CamRetCode StreamOperator::CommitStreams(OperationMode mode,
                                         const std::shared_ptr<CameraStandard::CameraMetadata>& modeSetting)
{
// ......
    std::vector<StreamConfiguration> configs = {};
    {
        std::lock_guard<std::mutex> l(streamLock_);
        for (auto it : streamMap_) { // 获取流的配置，前文CreateStrea时保存的流
            configs.emplace_back(it.second->GetStreamAttribute());
        }
    }
    // 检查流是否被支持
    DynamicStreamSwitchMode method = streamPipeline_->CheckStreamsSupported(mode, modeSetting, configs);
    if (method == DYNAMIC_STREAM_SWITCH_NOT_SUPPORT) {
        return INVALID_ARGUMENT;
    }
    if (method == DYNAMIC_STREAM_SWITCH_NEED_INNER_RESTART) {
        std::lock_guard<std::mutex> l(streamLock_);
        for (auto it : streamMap_) {
            it.second->StopStream();// 如果流被支持，但需要内部重启，这里先停流
        }
    }
    {
        std::lock_guard<std::mutex> l(streamLock_);
        for (auto it : streamMap_) {
            if (it.second->CommitStream() != RC_OK) { // 真正的 CommitStream，下面再细说
                CAMERA_LOGE("commit stream [id = %{public}d] failed.", it.first);
                return DEVICE_ERROR;
            }
        }
    }
    RetCode rc = streamPipeline_->PreConfig(modeSetting); // 把模式传入进行预配置
    if (rc != RC_OK) {
        CAMERA_LOGE("prepare mode settings failed");
        return DEVICE_ERROR;
    }
    rc = streamPipeline_->CreatePipeline(mode);// 创建pipeline
    if (rc != RC_OK) {
        CAMERA_LOGE("create pipeline failed.");
        return INVALID_ARGUMENT;
    }
    DFX_LOCAL_HITRACE_END;
    return NO_ERROR;
}

C++
RetCode StreamBase::CommitStream()
{
// ...
    hostStreamMgr_ = pipelineCore_->GetHostStreamMgr(); //从pipelinecore获取hoststreamanager
    CHECK_IF_PTR_NULL_RETURN_VALUE(hostStreamMgr_, RC_ERROR);
// ...
        info.bufferPoolId_ = poolId_;
        info.bufferCount_ = GetBufferCount();
    //  初始化 bufferpool
        RetCode rc = bufferPool_->Init(streamConfig_.width, streamConfig_.height, streamConfig_.usage,
                                       streamConfig_.format, GetBufferCount(), CAMERA_BUFFER_SOURCE_TYPE_EXTERNAL);
        if (rc != RC_OK) {
            CAMERA_LOGE("stream [id:%{public}d] initialize buffer pool failed.", streamId_);
            return RC_ERROR;
        }
    }
// stream传递到pipelinecore 并进行绑定
    RetCode rc = hostStreamMgr_->CreateHostStream(info, [this](std::shared_ptr<IBuffer> buffer) {
        HandleResult(buffer);
        return;
    });
// ....
    return RC_OK;
}

CreateStream 和CommitStream结束之后便是Capture，这里包含了起流的动作，关键实现如下：

C++
CamRetCode StreamOperator::Capture(int captureId, const std::shared_ptr<CaptureInfo>& captureInfo, bool isStreaming)
{
// ...
//  captureId 捕获请求的id; captureInfo 预览/拍照/录像的参数；isStreaming 连续捕获还是单次捕获（拍照）
    CaptureSetting setting = captureInfo->captureSetting_;
    auto request =
        std::make_shared<CaptureRequest>(captureId, captureInfo->streamIds_.size(), setting,
                                         captureInfo->enableShutterCallback_, isStreaming);
    for (auto id : captureInfo->streamIds_) {
        // 创建捕获请求，并传递给前文创建的流
        RetCode rc = streamMap_[id]->AddRequest(request);
        if (rc != RC_OK) {
            return DEVICE_ERROR;
        }
    }
// ...
}

从上面的代码可知预览、拍照、录像都是通过捕获请求触发，单次拍照则为单次捕获请求，预览和录像则是连续捕获请求。

C++
RetCode StreamBase::AddRequest(std::shared_ptr<CaptureRequest>& request)
{
    CHECK_IF_PTR_NULL_RETURN_VALUE(request, RC_ERROR);
    request->AddOwner(shared_from_this());
    request->SetFirstRequest(false);
    if (isFirstRequest) {
        RetCode rc = StartStream(); // 起流
        if (rc != RC_OK) {
            CAMERA_LOGE("start stream [id:%{public}d] failed", streamId_);
            return RC_ERROR;
        }
        request->SetFirstRequest(true);
        isFirstRequest = false;
    }
    {
        std::unique_lock<std::mutex> l(wtLock_);
        waitingList_.emplace_back(request); // 捕获请求添加到waitingList
        cv_.notify_one();
    }
    return RC_OK;
}

看看StreamStream是怎么实现的吧。

C++
RetCode StreamBase::StartStream()
{
// ...
    RetCode rc = pipeline_->Prepare({streamId_}); //  pipeline先完成一些准备工作
// ...
    state_ = STREAM_STATE_BUSY;
    std::string threadName =
        g_availableStreamType[static_cast<StreamIntent>(streamType_)] + "#" + std::to_string(streamId_);
    handler_ = std::make_unique<std::thread>([this, &threadName] {// 创建轮转线程
        prctl(PR_SET_NAME, threadName.c_str());
        while (state_ == STREAM_STATE_BUSY) {
            HandleRequest(); // 处理捕获请求
        }
    });
// ...
    rc = pipeline_->Start({streamId_}); // 通知pipeline和底层硬件可以开始出帧了
// ...
    return RC_OK;
}

C++
void StreamBase::HandleRequest()
{
    // 如果有 捕获请求下发，则退出等待状态
    if (waitingList_.empty()) {
        std::unique_lock<std::mutex> l(wtLock_);
        if (waitingList_.empty()) {
            cv_.wait(l, [this] { return !(state_ == STREAM_STATE_BUSY && waitingList_.empty()); });
        }
    }
// ...
        request = waitingList_.front();
        CHECK_IF_PTR_NULL_RETURN_VOID(request);
        if (!request->IsContinous()) { // 如果是连续捕获，则保留一份拷贝在waitinglist
            waitingList_.pop_front();
        }
    }
// 处理捕获请求
    request->Process(streamId_);// 最终调用下面的Capture接口
    return;
}

C++
RetCode StreamBase::Capture(const std::shared_ptr<CaptureRequest>& request)
{
    CHECK_IF_PTR_NULL_RETURN_VALUE(request, RC_ERROR);
    CHECK_IF_PTR_NULL_RETURN_VALUE(pipeline_, RC_ERROR);
    RetCode rc = RC_ERROR;
    if (request->IsFirstOne() && !request->IsContinous()) {
        uint32_t n = GetBufferCount();
        for (uint32_t i = 0; i < n; i++) {

            DeliverBuffer();// 单次捕获一次性下发所有的buffer
        }
    } else {
        do {
            rc = DeliverBuffer();// 连续捕获每次下发一个buffer
        } while (rc != RC_OK && state_ == STREAM_STATE_BUSY);
    }
    if (request->NeedCancel()) {// 被取消的捕获则退出
        CAMERA_LOGE("StreamBase::Capture stream [id:%{public}d] request->NeedCancel", streamId_);
        return RC_OK;
    }
    rc = pipeline_->Config({streamId_}, request->GetCaptureSetting());// 通知pipeline配置
    if (rc != RC_OK) {
        CAMERA_LOGE("stream [id:%{public}d] config pipeline failed.", streamId_);
        return RC_ERROR
    }
    rc = pipeline_->Capture({streamId_}, request->GetCaptureId());//  这里的capture指的是pipeline中的source node开始回buffer
    {
        std::unique_lock<std::mutex> l(tsLock_);
        inTransitList_.emplace_back(request);// 处理过的捕获请求存放在inTransitList
    }
    return RC_OK;
}

到这起流的流程就结束了，pipeline回上来的帧通过OnFrame接口处理。

C++
RetCode StreamBase::OnFrame(const std::shared_ptr<CaptureRequest>& request)
{
// ...
    bool isEnded = false;
    if (!request->IsContinous()) {
        isEnded = true;
    } else if (request->NeedCancel()) {
        isEnded = true;
    }
    {
        // inTransitList_ may has multiple copies of continious-capture request, we just need erase one of them.

        std::unique_lock<std::mutex> l(tsLock_);

        for (auto it = inTransitList_.begin(); it != inTransitList_.end(); it++) {
            if ((*it) == request) {
                inTransitList_.erase(it);// 已经回帧的请求，从inTransitList删除
                break;
            }
        }
        if (isEnded) {
            // if this is the last request of capture, send CaptureEndedMessage.
            auto it = std::find(inTransitList_.begin(), inTransitList_.end(), request);
            if (it == inTransitList_.end()) {
                std::shared_ptr<ICaptureMessage> endMessage =
                    std::make_shared<CaptureEndedMessage>(streamId_, request->GetCaptureId(), request->GetEndTime(),
                                                          request->GetOwnerCount(), tunnel_->GetFrameCount());
                CAMERA_LOGV("end of stream [%d], ready to send end message, capture id = %d",
                    streamId_, request->GetCaptureId());
                messenger_->SendMessage(endMessage);
                pipeline_->CancelCapture({streamId_});// 如果此次捕获结束，则取消捕获
            }
        }
    }
    ReceiveBuffer(buffer);// 底层返回的buffer送还到生产端，最终帧数据送到消费端
    return RC_OK;
}

linux和Android的关系 - 知乎 (zhihu.com) ↩︎。
HAL Subsystem | Android Open Source Project (google.cn) ↩︎。
zh-cn/release-notes/OpenHarmony-v3.1-release.md · OpenHarmony/docs - Gitee.com ↩︎。
OpenHarmony HDF 驱动框架介绍和驱动加载过程分析-OpenHarmony技术社区-51CTO.COM ↩︎。
OpenHarmony HDF HDI基础能力分析与使用-51CTO.COM 。

想了解更多内容，请访问：

51CTO和华为官方合作共建的鸿蒙技术社区

https://ost.51cto.com

HDI Implementation中的预览流程

先获取stream operator实例

通过前文的streamOperator创建流

Recommend

Google Finally Announces the Switch to Android App

换电车企，在利好中荆天棘地｜浙氪观察

Telegram Gets Another Major Update on iPhone and Android

Tromey: Faster GDB Startup

인텔 오버드라이브 DX40DP75 SZ956 V1.1 빈티지 CPU-인텔 ' 89' 93

Android Loses Market Share, iOS Grows Significantly

Doing our part: How Google’s network helps internet content reach users

Amazon has a ton of cheap TVs on sale with prices starting at just $149.99

慢雾：揭露浏览器恶意书签如何盗取你的Discord账户

DDoS Protection in the Age of 5G Networks, Edge Computing and Explosive Bandwidt...

About Joyk