ELF2开发板(飞凌嵌入式)搭建深度学习环境部署(RKNN环境部署)
本人主要介绍用于elf2的rk3588开发板的深度学习环境的搭建,和官方的方法不同,对于新手比较友好。零基础即可搭建,本人使用的是WSL2系统,当然使用虚拟机也是可以的,本人主要教学搭建yolov5模型转换为rknn的环境搭建,以及基本步骤,从该文章您可以学习到rk3588等瑞芯微芯片的所有模型环境部署。
安装docker环境(已安装过的可以跳过)
安装docker依赖的基础软件
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
添加官方来源
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
安装 docker
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
RKNN环境搭建
本文主要是使用RKNN-Toolkit2和RKNN Model Zoo将yolov5模型进行量化部署转换为rknn模型,并编译好基本脚本上传至开发板。
模型工具的介绍
RKNN-Toolkit2
RKNN-Toolkit2 是由瑞芯微电子开发的一套深度学习模型优化和推理工具。它支持多种深度学习框
架(如 Caffe、TensorFlow、PyTorch 等)的模型转换为 RKNN 格式,并提供了模型转换、量化、推理等功能。RKNN-Toolkit2 主要面向在瑞芯微 SoC 上进行 AI 应用开发,但也可以用于 PC 平台进行模型的转换、量化、推理等操作。
主要功能
- 模型转换:RKNN-Toolkit2支持将多种主流框架(如 TensorFlow、PyTorch、Caffe 等)训练得到的模型转换为RKNN模型,以便在 Rockchip 的硬件平台上运行。
- 模型优化:工具包提供了模型优化功能,可以对转换后的 RKNN 模型进行进一步的优化,以提高
模型在 Rockchip NPU 上的运行效率。 - 模型部署:通过 RKNN-Toolkit2,用户可以将优化后的 RKNN 模型部署到 Rockchip 的开发板上,进行 实际的推理测试和应用开发。
RKNN-Toolkit2仓库链接:https://github.com/airockchip/rknn-toolkit2
RKNN-Toolkit-Lite2
RKNN-Toolkit-Lite2 是 Rockchip 为其轻量级神经网络处理单元(NPU)提供的一款工具包。它专注于简化和加速深度学习模型在 Rockchip 硬件平台上的部署和推理过程。与 RKNN-Toolkit2 相比,RKNN-Toolkit-Lite2 主要针对轻量级应用和较小的计算需求进行优化。
主要功能
- 模型转换:支持将多种主流深度学习框架(如 TensorFlow Lite、ONNX 等)训练得到的模型转换为 RKNN Lite 模型,便于在 Rockchip 的轻量级 NPU 上进行推理。
- 模型优化:提供了针对轻量级模型的优化功能,以提高模型在 Rockchip 轻量级 NPU 上的运行效率和速度。
- 推理部署:使得优化后的 RKNN Lite 模型可以在 Rockchip 的开发板或嵌入式设备上进行推理,适合实时应用场景。
RKNN-Toolkit-Lite2仓库链接:https://github.com/airockchip/rknn-toolkit2/tree/master/rknn-toolkit-lite2
RKNPU2
RKNPU2 是 Rockchip 推出的一个跨平台的编程接口,主要用于帮助用户部署使用 RKNN Toolkit2导出的 RKNN 模型,从而加速 AI 应用的落地。
RKNPU2 的架构设计目标是优化深度学习模型的执行效率,其核心是一个专门为机器 学习任务定制
的硬件加速器。
主要功能
- 高效性能:RKNPU2 设计的目标是提供高吞吐量和低延迟,使复杂的 AI 算法得以快速执行。
- 低功耗:相比于 CPU 和 GPU,NPU 在执行特定 AI 任务时,能显著降低能耗,延长设备电池寿命。
- 灵活的 API 支持:开发者可以选择多种编程方式,如 C/C++、Python,或者直接使用预先训 练好的模型。
- 跨平台兼容:RKNPU2 可适配多种操作系统,包括 Linux 和 Android,方便在不同设备上部署。
- 开源社区:得益于开源模式,用户可以贡献代码,共同改进和完善项目,推动技术进步。
RKNPU2仓库链接: https://github.com/airockchip/rknn-toolkit2/tree/master/rknpu2
RKNN Model Zoo
RKNN Model Zoo基于 RKNPU SDK 工具链开发, 提供了目前主流算法的部署例程. 例程包含导出RKNN模型, 使用 Python API, CAPI 推理 RKNN 模型的流程,RKNN Model Zoo依赖 RKNN-Toolkit2 进行模型转换.
RKNN Model Zoo仓库链接:https://github.com/airockchip/rknn_model_zoo
环境搭建(干货)
- 下载好瑞芯微的RKNN的docker镜像
RKNN的docker镜像链接:https://meta.zbox.filez.com/v/link/view/ef37a9687973439f94f5b06837f12527,提取码为提取码:rknn
elf2开发板的rknn的版本为2.1,那么我们下载2.1版本的docker镜像,docker后缀名的为docker镜像,另一个为rknn-toolkit2的资源文件,和上文rknn-toolkit2仓库内容基本一致
在rknn-toolkit2仓库的release有不同版本的介绍,例如下图。
- 下载
rknn_model_zoo
和rknn-toolkit2
rknn_model_zoo的2.1版本链接:https://github.com/airockchip/rknn_model_zoo/tree/v2.1.0,直接切换到v2.1.0
分支或者在release
进行下载2.1版本。
rknn-toolkit2的2.1版本链接:https://github.com/airockchip/rknn-toolkit2/releases,同理切换到v2.1.0
分支或者release
中下载v2.1版本
- RKNNdocker镜像的导入以及环境的搭建
加载docker镜像并创建容器
docker load -i rknn-toolkit2-2.1.0-cp38-docker.tar
root@LAPTOP-U638FQQS:/# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
pulsar2 3.2-patch1 e96811ac0921 3 months ago 2.58GB
rknn-toolkit2 2.1.0-cp38 2c0940f24482 6 months ago 6.32GB
sophgo/tpuc_dev latest 191a433358ad 8 months ago 7.75GBdocker run -it -v $HOME:/home/host_home rknn-toolkit2:2.1.0-cp38
apt-get install cmake wget
root@5c542f109d4c:/# ls
bin dev home lib32 libx32 mnt proc root sbin sys usr
boot etc lib lib64 media opt rknn_toolkit2-2.1.0+708089d1-cp38-cp38-linux_x86_64.whl run srv tmp var
root@5c542f109d4c:/#
下载交叉编译工具链并复制到docker容器中(https://releases.linaro.org/components/toolchain/binaries/7.5-2019.12/aarch64-linux-gnu/)
指定交叉编译工具链路径(记得下载x86_64_aarch64,别下载成了i686)
export GCC_COMPILER=~/opt/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu
- yolov5模型转rknn,并编译运行程序
root@5c542f109d4c:/mnt# cd rknn_model_zoo-2.1.0
root@5c542f109d4c:/mnt/rknn_model_zoo-2.1.0/examples/yolov5/model# ./download_model.sh
--2025-02-04 05:44:59-- https://ftrg.zbox.filez.com/v2/delivery/data/95f00b0fc900458ba134f8b180b3f7a1/examples/yolov5/yolov5s_relu.onnx
Resolving ftrg.zbox.filez.com (ftrg.zbox.filez.com)... 180.184.171.46
Connecting to ftrg.zbox.filez.com (ftrg.zbox.filez.com)|180.184.171.46|:443... connected.
HTTP request sent, awaiting response... 200
Length: 28935807 (28M) [application/octet-stream]
Saving to: './yolov5s_relu.onnx'./yolov5s_relu.onnx 100%[========================================================>] 27.59M 10.0MB/s in 2.7s 2025-02-04 05:45:03 (10.0 MB/s) - './yolov5s_relu.onnx' saved [28935807/28935807]root@5c542f109d4c:/mnt/rknn_model_zoo-2.1.0/examples/yolov5/model#
root@5c542f109d4c:/mnt/rknn_model_zoo-2.1.0/examples/yolov5# cd python/
root@5c542f109d4c:/mnt/rknn_model_zoo-2.1.0/examples/yolov5/python# ls
convert.py yolov5.py
root@5c542f109d4c:/mnt/rknn_model_zoo-2.1.0/examples/yolov5/python# python3 ./convert.py ../model/yolov5s_relu.onnx rk3588
python3 ./convert.py ../model/yolov5s_relu.onnx rk3588
转化模型为rk3588平台的rknn模型
输出结果
root@5c542f109d4c:/mnt/rknn_model_zoo-2.1.0/examples/yolov5/python# python3 ./convert.py ../model/yolov5s_relu.onnx rk3588
I rknn-toolkit2 version: 2.1.0+708089d1
--> Config model
done
--> Loading model
I Loading : 100%|██████████████████████████████████████████████| 121/121 [00:00<00:00, 16813.34it/s]
done
--> Building model
I OpFusing 0: 100%|█████████████████████████████████████████████| 100/100 [00:00<00:00, 2365.08it/s]
I OpFusing 1 : 100%|████████████████████████████████████████████| 100/100 [00:00<00:00, 1139.61it/s]
I OpFusing 2 : 100%|████████████████████████████████████████████| 100/100 [00:00<00:00, 1050.44it/s]
I GraphPreparing : 100%|████████████████████████████████████████| 149/149 [00:00<00:00, 6884.39it/s]
I Quantizating : 100%|████████████████████████████████████████████| 149/149 [00:30<00:00, 4.96it/s]
W build: The default input dtype of 'images' is changed from 'float32' to 'int8' in rknn model for performance!Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of 'output0' is changed from 'float32' to 'int8' in rknn model for performance!Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of '286' is changed from 'float32' to 'int8' in rknn model for performance!Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of '288' is changed from 'float32' to 'int8' in rknn model for performance!Please take care of this change when deploy rknn model with Runtime API!
I rknn building ...
I rknn buiding done.
done
--> Export rknn model
done
covert.py代码
转化代码
import sysfrom rknn.api import RKNNDATASET_PATH = '../../../datasets/COCO/coco_subset_20.txt'
DEFAULT_RKNN_PATH = '../model/yolov5.rknn'
DEFAULT_QUANT = Truedef parse_arg():if len(sys.argv) < 3:print("Usage: python3 {} onnx_model_path [platform] [dtype(optional)] [output_rknn_path(optional)]".format(sys.argv[0]))print(" platform choose from [rk3562,rk3566,rk3568,rk3576,rk3588,rv1103,rv1106,rk1808,rv1109,rv1126]")print(" dtype choose from [i8, fp] for [rk3562,rk3566,rk3568,rk3576,rk3588,rv1103,rv1106]")print(" dtype choose from [u8, fp] for [rk1808,rv1109,rv1126]")exit(1)model_path = sys.argv[1]platform = sys.argv[2]do_quant = DEFAULT_QUANTif len(sys.argv) > 3:model_type = sys.argv[3]if model_type not in ['i8', 'u8', 'fp']:print("ERROR: Invalid model type: {}".format(model_type))exit(1)elif model_type in ['i8', 'u8']:do_quant = Trueelse:do_quant = Falseif len(sys.argv) > 4:output_path = sys.argv[4]else:output_path = DEFAULT_RKNN_PATHreturn model_path, platform, do_quant, output_pathif __name__ == '__main__':model_path, platform, do_quant, output_path = parse_arg()# Create RKNN objectrknn = RKNN(verbose=False)# Pre-process configprint('--> Config model')rknn.config(mean_values=[[0, 0, 0]], std_values=[[255, 255, 255]], target_platform=platform)print('done')# Load modelprint('--> Loading model')ret = rknn.load_onnx(model=model_path)if ret != 0:print('Load model failed!')exit(ret)print('done')# Build modelprint('--> Building model')ret = rknn.build(do_quantization=do_quant, dataset=DATASET_PATH)if ret != 0:print('Build model failed!')exit(ret)print('done')# Export rknn modelprint('--> Export rknn model')ret = rknn.export_rknn(output_path)if ret != 0:print('Export rknn model failed!')exit(ret)print('done')# Releaserknn.release()
可以看到rknn模型转化成功
root@5c542f109d4c:/mnt/rknn_model_zoo-2.1.0/examples/yolov5/model# ls
anchors_yolov5.txt bus.jpg coco_80_labels_list.txt download_model.sh yolov5.rknn yolov5s_relu.onnx
root@5c542f109d4c:/mnt/rknn_model_zoo-2.1.0/examples/yolov5/model#
- 编译执行脚本
root@5c542f109d4c:/mnt/rknn_model_zoo-2.1.0# ls
3rdparty FAQ_CN.md README.md asset build-android.sh datasets examples scaling_frequency.sh
FAQ.md LICENSE README_CN.md build build-linux.sh docs py_utils utils
root@5c542f109d4c:/mnt/rknn_model_zoo-2.1.0# ./build-linux.sh -t rk3588 -a aarch64 -d yolov5
输出结果:
root@5c542f109d4c:/mnt/rknn_model_zoo-2.1.0# ./build-linux.sh -t rk3588 -a aarch64 -d yolov5
./build-linux.sh -t rk3588 -a aarch64 -d yolov5
/mnt/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu
===================================
BUILD_DEMO_NAME=yolov5
BUILD_DEMO_PATH=examples/yolov5/cpp
TARGET_SOC=rk3588
TARGET_ARCH=aarch64
BUILD_TYPE=Release
ENABLE_ASAN=OFF
INSTALL_DIR=/mnt/rknn_model_zoo-2.1.0/install/rk3588_linux_aarch64/rknn_yolov5_demo
BUILD_DIR=/mnt/rknn_model_zoo-2.1.0/build/build_rknn_yolov5_demo_rk3588_linux_aarch64_Release
CC=/mnt/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc
CXX=/mnt/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-g++
===================================
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/rknn_model_zoo-2.1.0/build/build_rknn_yolov5_demo_rk3588_linux_aarch64_Release
[ 50%] Built target audioutils
[ 50%] Built target imagedrawing
[ 66%] Built target imageutils
[ 66%] Built target fileutils
[100%] Built target rknn_yolov5_demo
[ 16%] Built target fileutils
[ 33%] Built target imageutils
[ 50%] Built target imagedrawing
[ 83%] Built target rknn_yolov5_demo
[100%] Built target audioutils
Install the project...
-- Install configuration: "Release"
-- Installing: /mnt/rknn_model_zoo-2.1.0/install/rk3588_linux_aarch64/rknn_yolov5_demo/./rknn_yolov5_demo
-- Set runtime path of "/mnt/rknn_model_zoo-2.1.0/install/rk3588_linux_aarch64/rknn_yolov5_demo/./rknn_yolov5_demo" to "$ORIGIN/lib"
-- Installing: /mnt/rknn_model_zoo-2.1.0/install/rk3588_linux_aarch64/rknn_yolov5_demo/./model/bus.jpg
-- Installing: /mnt/rknn_model_zoo-2.1.0/install/rk3588_linux_aarch64/rknn_yolov5_demo/./model/coco_80_labels_list.txt
-- Installing: /mnt/rknn_model_zoo-2.1.0/install/rk3588_linux_aarch64/rknn_yolov5_demo/model/yolov5.rknn
-- Installing: /mnt/rknn_model_zoo-2.1.0/install/rk3588_linux_aarch64/rknn_yolov5_demo/lib/librknnrt.so
-- Installing: /mnt/rknn_model_zoo-2.1.0/install/rk3588_linux_aarch64/rknn_yolov5_demo/lib/librga.so
root@5c542f109d4c:/mnt/rknn_model_zoo-2.1.0#
打包/home/root/rknn_model_zoo-2.1.0/install/rk3588_linux_aarch64/rknn_yolov5_demo
root@5c542f109d4c:/mnt/rknn_model_zoo-2.1.0/install/rk3588_linux_aarch64# tar -zcvf rknn_yolov5_demo.tar.gz rknn_yolov5_demo/
rknn_yolov5_demo/
rknn_yolov5_demo/model/
rknn_yolov5_demo/model/yolov5.rknn
rknn_yolov5_demo/model/bus.jpg
rknn_yolov5_demo/model/coco_80_labels_list.txt
rknn_yolov5_demo/lib/
rknn_yolov5_demo/lib/librknnrt.so
rknn_yolov5_demo/lib/librga.so
rknn_yolov5_demo/rknn_yolov5_demo
root@5c542f109d4c:/mnt/rknn_model_zoo-2.1.0/install/rk3588_linux_aarch64#
上机实验
在上一节已经将开发板联网,直接scp或者拖拽上传到开发板并解压,执行可执行程序输出out.png
照片如下图
tar -zxvf rknn_yolov5_demo.tar.gzroot@elf2-desktop:~# cd rknn_yolov5_demo/
root@elf2-desktop:~/rknn_yolov5_demo# ls
lib model rknn_yolov5_demo
root@elf2-desktop:~/rknn_yolov5_demo# chmod 777 rknn_yolov5_demo
root@elf2-desktop:~/rknn_yolov5_demo# ./rknn_yolov5_demo model/yolov5.rknn model/bus.jpg
load lable ./model/coco_80_labels_list.txt
model input num: 1, output num: 3
input tensors:index=0, name=images, n_dims=4, dims=[1, 640, 640, 3], n_elems=1228800, size=1228800, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
output tensors:index=0, name=output0, n_dims=4, dims=[1, 255, 80, 80], n_elems=1632000, size=1632000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922index=1, name=286, n_dims=4, dims=[1, 255, 40, 40], n_elems=408000, size=408000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922index=2, name=288, n_dims=4, dims=[1, 255, 20, 20], n_elems=102000, size=102000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
model is NHWC input fmt
model input height=640, width=640, channel=3
origin size=640x640 crop size=640x640
input image: 640 x 640, subsampling: 4:2:0, colorspace: YCbCr, orientation: 1
scale=1.000000 dst_box=(0 0 639 639) allow_slight_change=1 _left_offset=0 _top_offset=0 padding_w=0 padding_h=0
src width=640 height=640 fmt=0x1 virAddr=0x0x101eab10 fd=0
dst width=640 height=640 fmt=0x1 virAddr=0x0x10316b20 fd=0
src_box=(0 0 639 639)
dst_box=(0 0 639 639)
color=0x72
rga_api version 1.10.1_[0]
rknn_run
person @ (209 243 286 510) 0.880
person @ (479 238 560 526) 0.871
person @ (109 238 231 534) 0.840
bus @ (91 129 555 464) 0.692
person @ (79 353 121 517) 0.301
write_image path: out.png width=640 height=640 channel=3 data=0x101eab10
root@elf2-desktop:~/rknn_yolov5_demo#