rockchip-linux / rknpu2 Goto Github PK

This copyright looks wrong for a file in a public repository:
/****************************************************************************
*

• Copyright (c) 2017 - 2022 by Rockchip Corp. All rights reserved.
• The material in this file is confidential and contains trade secrets
• of Rockchip Corporation. This is proprietary information owned by
• Rockchip Corporation. No part of this work may be disclosed,
• reproduced, copied, transmitted, or used in any way for any purpose,
• without the express written permission of Rockchip Corporation.

*****************************************************************************/
https://github.com/rockchip-linux/rknpu2/blob/master/runtime/RK3588/Linux/librknn_api/include/rknn_api.h

Please update/fix!

请问在使用了rga进行resize之后就aborted了，是什么原因造成的呢？

post process config: box_conf_threshold = 0.50, nms_threshold = 0.60
Read model/000657.jpg ...
img width = 640, img height = 480
Loading mode...
sdk version: 1.3.0 (c193be371@2022-05-04T20:16:33) driver version: 0.6.4
model input num: 1, output num: 1
index=0, name=image_arrays, n_dims=4, dims=[1, 640, 640, 3], n_elems=1228800, size=1228800, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
index=0, name=outputs, n_dims=4, dims=[1, 8400, 85, 1], n_elems=714000, size=714000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=2.498195
model is NHWC input fmt
model input height=640, width=640, channel=3
resize with RGA!
failed to open RGA:Permission denied.
rga_api version 1.6.0_[4] (d041200 build: 2021-12-01 15:26:14 base: )
rga_im2d: The current RockchipRga singleton is destroyed. Please check if RkRgaInit/RkRgaDeInit are called, if so, please disable them.
rga im2d: rga2 get info failed!

242, check error! Fatal error: The current RockchipRga singleton is destroyed.Please check if RkRgaInit/RkRgaDeInit are called, if so, please disable them.

Hi,

When I tried converting my onnx model, the input model has input of [1,300,300,3]. But the conversion results in a model which thinks it's "NHWC" but is in the format of [1,300,3,300].

我需要在x86 pc上重写数据预处理和后处理代码，但是由于没有对应的x86api,我该怎么调试自己的代码以保证正确性呢？

We tried to inference ssd_mobilenet _v1 model with C++ inference on Rk3566 board. But got this error

Loading model ...
rknn_init ...
I RKNN: set log level to 0
E RKNN: Invalid RKNN format
E RKNN: rknn_query, load_model failed
rknn_init fail! ret=-1

Tried to search the error code it said its Model Execution error, but not able to understand why its not working, because same model worked with python inference on ubuntu 16.04.

手上有个RK3568，做yolov5的算法移植，NPU驱动版本为0.4.2，怎么查询NPU利用率？

3rdparty/opencv/opencv-linux-aarch64/lib/目录下没有提供 libopencv_highgui.a

Is it possible to load more than one model at the same time and run inference on these models in sequence (pipeline) or in parallel (multitasking)?

在rknn_inputs_set处出现错误，ret = rknn_inputs_set(ctx, io_num.n_input, inputs);
128800是640,640,3的tensor，而4915200是12800,12800,3的tensor,我的rknn模型在连板调试中使用Python推理正确，且输入确实是640的，但是这里提醒我模型期望输入却是4915200(1280,1280,3)，不知道为什么，当我讲图片resize成1280后，又出现了新的错误，如下图，我想知道函数rknn_inputs_set中返回的模型期望输入是从哪里计算出来的？因为即使 ret = rknn_query(ctx, RKNN_QUERY_INPUT_ATTR, &(input_attrs[i]), sizeof(rknn_tensor_attr));查询模型输入，返回的也依旧是640,640,3

运行yolov7-tiny的时候出现了floating point exception(core dunp)错误。

该错误信息出现在rknn.build的时候，当rknn.config函数中取消target='rk3588'时没有该错误。

如题，仓库中rknpu2/runtime/RK3588/Linux/里面的东西都是3588的？3588的rknn_server 是升级到了1.40版本，但是sdk怎么版本对不上啊，提交信息中写的是Update librknnrt.so: fix compatibility of RK356X，明明是RK356X的不是3588的吧？找了一圈只找到了armhf的1.40版本的sdk，还是在rknpu2/runtime/RV1106/Linux)/librknn_api/里面看到的，我手上的是firefly itx3588j这个能用吗？我记得应该是aarch64的才对呀

rknn_query从模型获得的信息：

input tensors:
index=0 name=x.1_0 n_dims=4 dims=[1 3 640 640] n_elems=1228800 size=1228800 fmt=0 type=3 qnt_type=2 fl=0 zp=0 scale=1.000000
output tensors:
index=0 name=permute_at_1747_221_out0_223 n_dims=5 dims=[3 80 80 11] n_elems=211200 size=211200 fmt=0 type=3 qnt_type=2 fl=-29 zp=227 scale=0.598226
index=1 name=permute_at_1770_230_out0_224 n_dims=5 dims=[3 40 40 11] n_elems=52800 size=52800 fmt=0 type=3 qnt_type=2 fl=-72 zp=184 scale=0.588694
index=2 name=permute_at_1793_239_out0_225 n_dims=5 dims=[3 20 20 11] n_elems=13200 size=13200 fmt=0 type=3 qnt_type=2 fl=-60 zp=196 scale=1.220445

设置输出自动量化:

    // set output
    outputs = new rknn_output [io_num.n_output];
    memset(outputs, 0, sizeof(outputs));
    for (int i = 0; i < io_num.n_output; i++)
    {
        outputs[i].want_float = 1;
    }

之后

cout << outputs[0].size << endl;
cout << outputs[1].size << endl;
cout << outputs[2].size << endl;

结果输出

844800
1163544915
5394255

如果输出float32的数据，第一个844800是可以理解，但后面两个是怎么回事？
另外，如果outputs[i].want_float = 1;的话就会出现Segmentation fault (core dumped)报错，是因为非量化输出不支持.size参数吗？

谢谢

模型都是用rknntoolkit2-1.3版本转换过来的，python版本的运行正常，C++版本的在初始化时遇到这个问题,与这个问题一样的问题#20

官方有解决的办法吗？不知道是什么原因。但是单独打印rknn_tensor_attr的size却是372。

E RKNN: [14:57:27.201] rknn_query, info_len(372) != sizeof(rknn_tensor_attr)(376)!
rknn_init error ret=-5
index=0, name=, n_dims=0, dims=[0, 0, 0, 0], n_elems=0, size=0, fmt=NCHW, type=FP32, qnt_type=NONE, zp=0, scale=0.000000

sizeof_input_attrs[0]): 372
sizeof(rknn_tensor_attr): 372

报错内容提示：failed to open rknpu module,need to insmod rknpu dirver failed to open rknn device
但是在官网更新了最新的3588J的驱动之后，还是会报这个错误，所使用的demo为这个demo，所使用的模型是这个仓库里面的模型https://github.com/rockchip-linux/rknn-toolkit2/tree/master/examples/onnx/yolov5
所使用的固件是这个网站上的ubuntu固件
https://www.t-firefly.com/doc/download/161.html

发现rk3568 ， 偶发推理不一致 （2g内存rk3568板子）
感觉似乎是在内存比较吃紧的时候， 启动推理， 确定推理的输入是一致的，
但是输出是会偶发出现推理不一致， 而且一旦出现推理不一致的情况， 后面的推理 75%的会出错。
推理不是完全错误的， 错误的推理里面 有一半的结果是正确的， 但是会多出几个错误的检测框

注： 可以多连几个vscode， vscode非常消耗系统内存

Hello,

I'm trying to inference YOLOv4 .rknn converted from .onnx with the help of rknn-toolkit2. It performs well on PC using test.py code from toolkit examples but fails on AIO-3568J using rknpu .cc code. It returns error during inference (rknn.run):

E RKNN: [02:49:39.674] failed to submit!, op id: 227, op name: OutputOperator:139_convolutional, flags: 0x1, task start: 0, task number: 2583, run task counter: 2322, int status: 0

Tried to convert original darknet model to .rknn but received the same mistake. Have anyone tried to inference YOLOv4 rknn model before?

麻烦问下rk3568有没有yolo得demo，可以在此提供一下么?

Hi, this repo supplies runtime. Is there linux driver, whose file extension is .ko modules for rk356x?

文档中https://github.com/rockchip-linux/linux-rga
网址不可用,请及时更新文档

Run the rknn_yolov5_demo on the RK3566, and meeting the error:

I have no idea about this error, please help!!!

./build-android.sh
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ubuntu/Desktop/rknpu2-master/examples/rknn_ssd_demo/build/build_android_v8a
[ 33%] Linking CXX executable rknn_ssd_demo
/opt/tools/rk/android-ndk-r17//toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/bin/ld:/home/ubuntu/Desktop/rknpu2-master/examples/rknn_ssd_demo/../../Android/librknn_api/arm64-v8a/librknn_api.so: file format not recognized; treating as linker script
/opt/tools/rk/android-ndk-r17//toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/bin/ld:/home/ubuntu/Desktop/rknpu2-master/examples/rknn_ssd_demo/../../Android/librknn_api/arm64-v8a/librknn_api.so:1: syntax error
collect2: error: ld returned 1 exit status
CMakeFiles/rknn_ssd_demo.dir/build.make:158: recipe for target 'rknn_ssd_demo' failed
make[2]: *** [rknn_ssd_demo] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/rknn_ssd_demo.dir/all' failed
make[1]: *** [CMakeFiles/rknn_ssd_demo.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

RK3566
Android 11
RAM 2G
运行到rknn_init 时候直接崩溃 运行代码
int HEADDetect::LoadRKNN(const char* cpath)
{
FILE *fp = fopen(cpath, "rb");
if (fp == NULL) {
printf("fopen %s fail!\n", cpath);
return -1;
}
fseek(fp, 0, SEEK_END);
int model_len = ftell(fp);
model = malloc(model_len);
fseek(fp, 0, SEEK_SET);
if (model_len != fread(model, 1, model_len, fp)) {
printf("fread %s fail!\n", cpath);
free(model);
return -1;
}
int ret = 0;
//ret = rknn_init(&rknn_ctx, model, model_len, RKNN_FLAG_PRIOR_MEDIUM | RKNN_FLAG_COLLECT_PERF_MASK);
ret = rknn_init(&rknn_ctx, model, model_len, RKNN_FLAG_PRIOR_HIGH, NULL);
if (ret < 0)
{
printf("rknn_init fail! ret=%d\n", ret);
// goto Error;
}
free(model);
fclose(fp);
printf("rknn_init ret=%d\n", ret);
return ret;
}

Hi do I need to modprobe anything? What driver is needed for this? thanks!

rk3588+linux 想要使用rknntoolkit进行调试 无法用adb连接linux系统的板子

模型转换：rknn-toolkit2 version: 1.3.0-11912b58
最小可复现网络：
class InstanceModel(nn.Module):
def init(self, channels):
super().init()
self.ins = nn.InstanceNorm2d(channels)

def forward(self, inputs):
    out = self.ins(inputs)
    return out

转换后模型：https://drive.google.com/file/d/1MGvJ9SHaY0KDKaMEf6XkhUiROLJk3HTv/view?usp=sharing

linaro@linaro-alip: ~/rknn_yolov5_demo_Linux$ sudo ./rknn_yolov5_demo model/RK356X/yolov5s-640-640.rknn model/bus.jpg
Rga built version:274b345
post process config: box_conf_threshold = 0.50, nms_threshold = 0.60
Read model/bus.jpg ...
img width = 640, img height = 640
Loading mode...

Message from syslogd@linaro-alip at Mar 9 09:14:06 ...
kernel:[ 1150.930469] Internal error: Oops: 96000046 [#3] SMP

Message from syslogd@linaro-alip at Mar 9 09:14:07 ...
kernel:[ 1150.935380] Process rknn_yolov5_dem (pid: 1446, stack limit = 0x00000000aeaa45cd)

Message from syslogd@linaro-alip at Mar 9 09:14:07 ...
kernel:[ 1151.760869] Code: a8c12027 a8c12829 a8c1302b a8c1382d (a88120c7)

想读取视频应该怎么解决

代码：

int main() {
const char *model_path = "/home/white/CLionProjects/yolo5s/model/yolox.rknn";
const char *img_path = "/home/white/Documents/dataset/1716/VOCdevkit/VOC2007/JPEGImages/000002.jpg";
std::cout<<model_path<<std::endl;
rknn_context ctx;
int ret;
int model_len = 0;
unsigned char *model;
model = load_model(model_path, &model_len);
// ret = rknn_init(&ctx,model,model_len,0);
rknn_init(&ctx, nullptr, 0, 0);
std::cout<<ret<<std::endl;
std::cout << "Hello, World!" << std::endl;
return 0;
}
报错信息：
undefined reference to `rknn_init'
collect2: error: ld returned 1 exit status

请问 RK356X 的 rknpu2 的 npu 驱动 galcore.ko 之类 哪里获取。rknpu [https://github.com/rockchip-linux/rknpu/tree/master/drivers/npu_ko] 就提供相对应的驱动。

之前你们用芯原的NPU IP的时候可以用他们提供的 OpenVX 与 TIM-VX 以对接 Apache TVM 等第三方 DNN 部署框架，现在 RK35xx 系列换成瑞芯微自研 NPU 了，就只能用闭源的 RKNN 了，没办法自己扩展。请问你们未来有开发算子库与支持开放标准的计划吗？

model input num: 4, output num: 1
input tensors:
index=0 name=input_imgs n_dims=4 dims=[12 256 128 1] n_elems=393216 size=786432 fmt=0 type=1 qnt_type=2 fl=0 zp=0 scale=1.000000
index=1 name=desire n_dims=2 dims=[0 0 8 1] n_elems=8 size=16 fmt=0 type=1 qnt_type=2 fl=0 zp=0 scale=1.000000
index=2 name=traffic_convention n_dims=2 dims=[0 0 2 1] n_elems=2 size=4 fmt=0 type=1 qnt_type=2 fl=0 zp=0 scale=1.000000
index=3 name=initial_state n_dims=4 dims=[512 1 1 1] n_elems=512 size=1024 fmt=0 type=1 qnt_type=2 fl=0 zp=0 scale=1.000000
output tensors:
index=0 name=outputs n_dims=2 dims=[0 0 6409 1] n_elems=6409 size=12818 fmt=0 type=1 qnt_type=2 fl=0 zp=0 scale=1.000000
E RKNN: [07:24:10.076] Meet unsupported split
Aborted

I downloaded latest rknpu2 and rknn-toolkit2 and run on rk3588 board. And I got the above error. I saw the same issue published but still left open. And there is no fix in official release. Please help!

The RKNN runtime in this repo is still in version 1.0.0. It can't load model files generated by current rknn-toolkit2, which is 1.1.0.

I saw rknpu2 version 1.1.0 is actually available in other places. e.g. https://gitee.com/caesar-wang/rknpu2

Please keep this repo up-to-date. Thank you.

I think RK3566/RK3568 is more than capable of running python. It would be great if this platform can provide the access to all resources likes camera, display, and NPU all in python.
The works to do are pretty much wrapping the C APIs defined below into python.
https://github.com/rockchip-linux/rknpu2/blob/master/Linux/librknn_api/include/rknn_api.h

Hi! I tried use flag RKNN_FLAG_ASYNC_MASK, but it doesn't affec to rknn_run.
Have you any example? Or is it not supported in rknpu2?

模型是picodet 未量化



是这里设置的问题吗？前处理跟python 一致

I use the modified version of the yolov program to train my model and the results are very good, but after converting to rknn, the recognition output will be out of the box. Someone suggested to me because the output dimension of my onnx model is 4 dimensions, and the official one is 5 dimensional, after that I found that when exporting the onnx model without adding the --ignore_output_permute parameter the output is 5 dimensional. But after converting the model, the frame is still messed up in this demo, and there are a lot of coordinates output. Please how can I solve this problem

model input num: 4, output num: 1
input tensors:
index=0 name=input_imgs n_dims=4 dims=[12 256 128 1] n_elems=393216 size=786432 fmt=0 type=1 qnt_type=2 fl=0 zp=0 scale=1.000000
index=1 name=desire n_dims=2 dims=[0 0 8 1] n_elems=8 size=16 fmt=0 type=1 qnt_type=2 fl=0 zp=0 scale=1.000000
index=2 name=traffic_convention n_dims=2 dims=[0 0 2 1] n_elems=2 size=4 fmt=0 type=1 qnt_type=2 fl=0 zp=0 scale=1.000000
index=3 name=initial_state n_dims=4 dims=[512 1 1 1] n_elems=512 size=1024 fmt=0 type=1 qnt_type=2 fl=0 zp=0 scale=1.000000
output tensors:
index=0 name=outputs n_dims=2 dims=[0 0 6409 1] n_elems=6409 size=12818 fmt=0 type=1 qnt_type=2 fl=0 zp=0 scale=1.000000
E RKNN: [07:24:10.076] Meet unsupported split
Aborted

在使用demo中你们自定义imresize对输入图像进行resize后，图片会整体向右移动，应该是imresize函数里的坐标有问题

./rknn_yolov5_demo model/RK356X/yolov5s-640-640.rknn model/bus.jpg
执行代码提示
./rknn_yolov5_demo: error while loading shared libraries: librga.so.2: cannot open shared object file: No such file or directory

文档中有librga.so文件的说明,请问librga.so.2这个文件如何获取?

Now that NPU is under Rockchip IP, it would be helpful to open sorce RKNN and Toolkit for devs to fix bugs and add OPs not coverd by vanilla models like yolo and resnet.

使用 NPU SDK 1.3.0 在 3588S上推理时 rknn_query 竟然报错：
E RKNN: [03:53:15.041] rknn_query, info_len(372) != sizeof(rknn_tensor_attr)(376)!

Loading RKNN Model ...
Initializing RKNN Envs ...
sdk version: 1.3.0 (c193be371@2022-05-04T20:16:33) driver version: 0.6.4
Querying Model Attributes ...
model input num: 1, output num: 1

sizeof(this->input_attrs[0]): 372
sizeof(rknn_tensor_attr): 372

E RKNN: [03:53:15.041] rknn_query, info_len(372) != sizeof(rknn_tensor_attr)(376)!
RKNN_QUERY_INPUT_ATTR error ret=-5

报错部分代码如下：

    // - I/O Attributes
    memset(this->input_attrs, 0, sizeof(rknn_tensor_attr));
    memset(this->output_attrs, 0, sizeof(rknn_tensor_attr));

    cout << "\nsizeof(this->input_attrs[0]): " << sizeof(this->input_attrs[0]) << endl;
    cout << "sizeof(rknn_tensor_attr): " << sizeof(rknn_tensor_attr) << "\n" << endl;
    
    this->input_attrs[0].index = 0;
    ret = rknn_query(this->ctx, RKNN_QUERY_INPUT_ATTR, &(this->input_attrs[0]), sizeof(rknn_tensor_attr));    // Single I/O YOLOX
    
    if (ret < 0)
    {
        printf("RKNN_QUERY_INPUT_ATTR error ret=%d\n", ret);
        return;
    }

    this->output_attrs[0].index = 0;
    ret = rknn_query(this->ctx, RKNN_QUERY_OUTPUT_ATTR, &(this->output_attrs[0]), sizeof(rknn_tensor_attr));    // Single I/O YOLOX
    if (ret < 0)
    {
        printf("RKNN_QUERY_OUTPUT_ATTR error ret=%d\n", ret);
        return;
    }

模型和测试代码在TB-RK3399PRO-D上运行顺利，迁移过来后模型使用RKToolkit2重新转换过，不能理解为什么两个输出的 sizeof(rknn_tensor_attr)不一致，麻烦大神们解答，谢谢！！

Hi, there. I was following the document doc/Rockchip_Quick_Start_RKNN_SDK_V1.3.0_CN.pdf and installed rknn through pip. However, I got the following when running python3 test.py

E build: Catch exception when building RKNN model!
E build: Traceback (most recent call last):
E build:   File "rknn/api/rknn_base.py", line 1680, in rknn.api.rknn_base.RKNNBase.build
E build:   File "rknn/api/rknn_base.py", line 363, in rknn.api.rknn_base.RKNNBase._generate_rknn
E build:   File "rknn/api/rknn_base.py", line 272, in rknn.api.rknn_base.RKNNBase._build_rknn
E build: ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found 

10-25 06:32:13.744 0 0 W : invalid destination resolution act_w = 120, act_h = 0
10-25 06:32:13.744 0 0 E rga2 : req argument is inval
10-25 06:32:13.639 2312 2312 E rkyolo4j: colorConvert failed. Ret: Fatal error: Failed to call RockChipRga interface, query log to find the cause of failure.
10-25 06:32:13.767 0 0 W : invalid destination resolution act_w = 120, act_h = 46080
10-25 06:32:13.767 0 0 E rga2 : req argument is inval
10-25 06:32:13.650 2312 2367 E rkyolo4j: run_yolo: resize image with rga failed: Fatal error: Failed to call RockChipRga interface, query log to find the cause of failure.
10-25 06:32:13.777 0 0 W : invalid destination resolution act_w = 120, act_h = 0
10-25 06:32:13.777 0 0 E rga2 : req argument is inval
10-25 06:32:13.675 2312 2312 E rkyolo4j: colorConvert failed. Ret: Fatal error: Failed to call RockChipRga interface, query log to find the cause of failure.
10-25 06:32:13.802 0 0 W : invalid destination resolution act_w = 120, act_h = 46080
10-25 06:32:13.802 0 0 E rga2 : req argument is inval
10-25 06:32:13.685 2312 2367 E rkyolo4j: run_yolo: resize image with rga failed: Fatal error: Failed to call RockChipRga interface, query log to find the cause of failure.
10-25 06:32:13.812 0 0 W : invalid destination resolution act_w = 120, act_h = 0
10-25 06:32:13.812 0 0 E rga2 : req argument is inval
10-25 06:32:13.711 2312 2312 E rkyolo4j: colorConvert failed. Ret: Fatal error: Failed to call RockChipRga interface, query log to find the cause of failure.
10-25 06:32:13.838 0 0 W : invalid destination resolution act_w = 120, act_h = 46080
10-25 06:32:13.838 0 0 E rga2 : req argument is inval

1.官方yolov7使用rknn-toolkit2导出模型时，QUANTIZE_ON = False。

（QUANTIZE_ON = True时输出的box坐标有值但所有框和类别的置信度都为0）

2.使用未量化rknn模型在板子上进行推理，打印模型的输入输出信息如下：

3.根据官方文档中描述，可得知可以给输入设置uint8 / float16 / float32数据类型

4.进行float32/uint8两种测试，测试输入数据

//输入格式为uint8
cv::Mat orig_img = cv::imread(image_name, 1);
cv::Mat img;
cv::cvtColor(orig_img, img, cv::COLOR_BGR2RGB);

inputs[0].type = RKNN_TENSOR_UINT8;
inputs[0].size = width * height * channel;
inputs[0].buf = (void*)img.data;

//输入格式为float32
cv::Mat orig_img = cv::imread(image_name, 1);
cv::Mat img;
cv::cvtColor(orig_img, img, cv::COLOR_BGR2RGB);
img.convertTo(img, CV_32F, 1.0 / 255, 0);

inputs[0].type = RKNN_TENSOR_FLOAT32;
inputs[0].size = width * height * channel * 4;
inputs[0].buf = (void*)img.data;

5.上述两种不同的输入数据格式都会在ret = rknn_run(ctx, NULL)时报同一个错误

跑的简单图像超分辨率模型，模拟器结果正确，NPU结果差别很大，用的一样的代码和模型，分析得到的余弦距离都是0.99以上