modeltc / united-perception Goto Github PK

Hi, just wonder if benchmark result is trained by EOD, does augmentation strictly aligned with yolov5 original repo?

when i use the script, i says "spring.submit: not found", how to solve it?

Your work is great. I ' m very interested in it.
when i use efl loss replace cross_entropy loss in segmentation semantic, the datasets is cityscapes.
The following problems arose

Looking forward to your reply！

Traceback (most recent call last):
File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/data/juicefs_hz_cv_v3/11105507/AAAI2022/united-perception2/up/main.py", line 31, in
main()
File "/data/juicefs_hz_cv_v3/11105507/AAAI2022/united-perception2/up/main.py", line 25, in main
args.run(args)
File "/data/juicefs_hz_cv_v3/11105507/AAAI2022/united-perception2/up/commands/train.py", line 161, in _main
launch(main, args.num_gpus_per_machine, args.num_machines, args=args, start_method=args.fork_method)
File "/data/juicefs_hz_cv_v3/11105507/AAAI2022/united-perception2/up/utils/env/launch.py", line 68, in launch
main_func(*(args,))
File "/data/juicefs_hz_cv_v3/11105507/AAAI2022/united-perception2/up/commands/train.py", line 140, in main
runner = RUNNER_REGISTRY.get(runner_cfg['type'])(cfg, **runner_cfg['kwargs'])
File "/data/juicefs_hz_cv_v3/11105507/AAAI2022/united-perception2/up/utils/general/registry.py", line 81, in get
assert module_name in self, '{} is not supported, avaiables are:{}'.format(module_name, self)
AssertionError: kd is not supported, avaiables are:{'base': <class 'up.runner.base_runner.BaseRunner'>}

I want to train my custom data with "EFL" or "EQFL"!
Can I directly train yolox model with EFL using coco data format？

do_calib关闭时候，直接进行量化就失败了；

将weight 的perchannel 改成false就可以了； 或者设置self.do_calib = True

我看到您在 issue 11 中回答了在分类任务中如何使用EFL的相关问题，但我对其中的第二点不是十分明白，我应该怎么修改我的梯度指导策略才能更好地应用在分类任务上呢？您能提供一些参考的代码或者给出一些代码上修改的建议吗？谢谢！

单机train的时候当指令ng=1，不论是train还是test都会遇到同样的error；

当 --ng=1的时候出现Error:“`........please make sure to call init_process_group”

hi，你好！

EFL在迁到mmdetection中，私有化数据集，仅将原生的FocalLoss修改为EqualizedFocalLoss，mAP 0.032 比原生 mAP 0.425低很多。
在迁移过程中，有些疑问：

1、模型差异，包括：
UP中的retinanet模型，有RFS、iou_branch_loss、采用hand_craft生成anchor、atss正负样本分配。
mmdet中，bbox回归采用L1 Loss、每个位置生成9个anchor、maxIouAssigner正负样本分配。

2、正负样本梯度比观察
self.pos_neg = torch.clamp(self.pos_grad / (self.neg_grad + 1e-10), min=0, max=1)
print(self.pos_neg)

log.txt.tar.gz
训练过程中，不同类别正负梯度比变化很小，说明EFL并没有起到作用？

EFL的实现、超参等，已经检查多遍。
帮忙分析下，是什么问题？

I found in Inferece.py the prepare are:

def fetch_single(self, filename):
        img = self.image_reader.read(filename)
        data = EasyDict(
            {"filename": filename, "origin_image": img, "image": img, "flipped": False}
        )
        data = self.transformer(data)
        scale_factor = data.get("scale_factor", 1)

        image_h, image_w = get_image_size(img)
        new_image_h, new_image_w = get_image_size(data.image)
        data.image_info = [
            new_image_h,
            new_image_w,
            scale_factor,
            image_h,
            image_w,
            data.flipped,
            filename,
        ]
        data.image = data.image.cuda()
        return data

which image_info max size is 7, so that above index [7] is out of indices. How to resolve?

def permute_preds(self, mlvl_preds):
permute_mlvl_preds = []
for loc_pred, cls_pred, obj_pred in mlvl_preds:
bs, ls, ny, nx = loc_pred.shape
cs = cls_pred.shape[1]
loc_pred = loc_pred.view(bs, self.num_anchors_per_level,
ls // self.num_anchors_per_level, ny, nx).permute(0, 1, 3, 4, 2).contiguous()
cls_pred = cls_pred.view(bs, self.num_anchors_per_level,
cs // self.num_anchors_per_level, ny, nx).permute(0, 1, 3, 4, 2).contiguous()
obj_pred = obj_pred.view(bs, self.num_anchors_per_level,
1, ny, nx).permute(0, 1, 3, 4, 2).contiguous()
permute_mlvl_preds.append((loc_pred, cls_pred, obj_pred))
return permute_mlvl_preds

torch.fx.tracer proxy iterated

When will release new version with segmentation and distillation application?

您好，我自己做的数据集是只有单类别的目标，请问可以用EFL吗？谢谢您。

hi, i'm got the error 'ImportError: cannot import name 'naive_nms' from 'up.extensions.csrc' (unknown location)'

for the naive_nms, it's not in csrc? where it is?

Hi, thanks for your wonderful work. I want to know how to find the code about EFL. I only find the base loss definition.

我想跑bignas用于分类训练，但是bignas的分类配置官方示例的分类配置包含一些目标检测相关的配置 calib_meta_file: /mnt/lustre/share/shenmingzhu/calib_coco/coco_2048_coco_format.json，不知道是干嘛用的。另外能完善下教程吗，直接运行script/dist_train.py会报一堆错。。。

configs中net结构和mmdetection中的anchor，header中的构造参数很难对应，mmdetection pretrain模型不能直接对接TC工具吧

The following error occurs when the environment is configured and the following command is executed (the same error occurs on both Windows and Linux platforms) :

sh scripts/dist_train.sh 2 configs/cls/resnet/resnet18.yaml

No module named 'petrel_client'
init petrel failed
No module named 'spring_aux'

2022-11-28 00:53:13,270-rk0-normalize.py#38:import error No module named 'mqbench'; If you need Mqbench to quantize model,      you should add Mqbench to this project. Or just ignore this error.
2022-11-28 00:53:13,270-rk0-normalize.py#45:import error No module named 'msbench'; If you need Msbench to prune model,     you should add Msbench to this project. Or just ignore this error.

Traceback (most recent call last):
File "D:\anaconda3\envs\python37\lib\runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "D:\anaconda3\envs\python37\lib\runpy.py", line 142, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "D:\anaconda3\envs\python37\lib\runpy.py", line 109, in _get_module_details
__import__(pkg_name)
File "D:\pycharm_work_place\United-Perception\up\__init__.py", line 26, in <module></module>
from .tasks import *
File "D:\pycharm_work_place\United-Perception\up\tasks\__init__.py", line 24, in <module></module>
globals()[fp] = importlib.import_module('.' + fp, __package__)
File "D:\anaconda3\envs\python37\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "D:\pycharm_work_place\United-Perception\up\tasks\det\__init__.py", line 2, in <module></module>
from .models import * # noqa
File "D:\pycharm_work_place\United-Perception\up\tasks\det\models\__init__.py", line 1, in <module></module>
from .heads import * # noqa
File "D:\pycharm_work_place\United-Perception\up\tasks\det\models\heads\__init__.py", line 2, in <module></module>
from .bbox_head import *  # noqa
File "D:\pycharm_work_place\United-Perception\up\tasks\det\models\heads\bbox_head\__init__.py", line 1, in <module></module>
from .bbox_head import * # noqa
File "D:\pycharm_work_place\United-Perception\up\tasks\det\models\heads\bbox_head\bbox_head.py", line 6, in <module></module>
from up.tasks.det.models.utils.assigner import map_rois_to_level
File "D:\pycharm_work_place\United-Perception\up\tasks\det\models\utils\__init__.py", line 3, in <module></module>
from .matcher import * # noqa
File "D:\pycharm_work_place\United-Perception\up\tasks\det\models\utils\matcher.py", line 6, in <module></module>
from up.tasks.det.models.utils.bbox_helper import offset2bbox
File "D:\pycharm_work_place\United-Perception\up\tasks\det\models\utils\bbox_helper.py", line 10, in <module></module>
from up.extensions import gpu_iou_overlap

ImportError:  cannot import name 'gpu_iou_overlap' from 'up.extensions'  (D:\pycharm_work_place\United-Perception\up\extensions\__init__.py)

Why can't the module be imported? What should I do? I hope the kind people can help me solve the problem, I will be grateful!

Hello Author, you're doing a great job. I would like to ask how the Softmax-EQL function proposed in this paper can be used in target detection.

Thank you for the excellent work of MQBench and EOD. I am interested in the work of quantization and I have tried the config of retinanet-r50_1x_quant.yaml. However, there are some errors. Besides, I found that there is no quantitative document in this project. Can you give some suggestions to use the quant_runner.

Here are the errors I encountered when use retinanet-r50_1x_quant.yaml：

error_1

File "/home/user/miniconda3/envs/eod/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
fn(i, *args)
File "/home/user/project/EOD/eod/utils/env/launch.py", line 117, in _distributed_worker
main_func(args)
File "/home/user/project/EOD/eod/commands/train.py", line 121, in main
runner = RUNNER_REGISTRY.get(runner_cfg['type'])(cfg, **runner_cfg['kwargs'])
File "/home/user/project/EOD/eod/runner/quant_runner.py", line 14, in init
super(QuantRunner, self).init(config, work_dir, training)
File "/home/user/project/EOD/eod/runner/base_runner.py", line 52, in init
self.build()
File "/home/user/project/EOD/eod/runner/quant_runner.py", line 32, in build
self.quantize_model()
File "/home/user/project/EOD/eod/runner/quant_runner.py", line 68, in quantize_model
from mqbench.prepare_by_platform import prepare_by_platform
ImportError: cannot import name 'prepare_by_platform' from 'mqbench.prepare_by_platform' (/home/user/project/MQBench/mqbench/prepare_by_platform.py)

solved by modifying the EOD/eod/runner/quant_runner.py 68-72:

from mqbench.prepare_by_platform import prepare_qat_fx_by_platform
logger.info("prepare quantize model")
deploy_backend = self.config['quant']['deploy_backend']
prepare_args = self.config['quant'].get('prepare_args', {})
self.model = prepare_qat_fx_by_platform(self.model, self.backend_type[deploy_backend], prepare_args)

error_2

I can use single gpu train the quant model, but when using multiple gpus I meet the error below, which is still unsolved.

Traceback (most recent call last):
File "/home/user/miniconda3/envs/eod/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
fn(i, *args)
File "/home/user/project/EOD/eod/utils/env/launch.py", line 117, in _distributed_worker
main_func(args)
File "/home/user/project/EOD/eod/commands/train.py", line 121, in main
runner = RUNNER_REGISTRY.get(runner_cfg['type'])(cfg, **runner_cfg['kwargs'])
File "/home/user/project/EOD/eod/runner/quant_runner.py", line 15, in init
super(QuantRunner, self).init(config, work_dir, training)
File "/home/user/project/EOD/eod/runner/base_runner.py", line 52, in init
self.build()
File "/home/user/project/EOD/eod/runner/quant_runner.py", line 34, in build
self.calibrate()
File "/home/user/project/EOD/eod/runner/quant_runner.py", line 84, in calibrate
self.model(batch)
File "/home/user/miniconda3/envs/eod/lib/python3.8/site-packages/torch/fx/graph_module.py", line 513, in wrapped_call
raise e.with_traceback(None)
NameError: name 'dist' is not defined

Where is the code of EFL

Hi, thanks for your wonderful work. I want to know how to fuse the EFL loss with the BCE loss for multi-label classification tasks. Can you give me some suggestions?

Thanks a lot.

Best Regards!
Tan

Could you give me some advice on how to apply EFL to YoloV5

本人使用up框架复现efl遇到如下问题，服务器显卡内存充足，但是一直显示oom，已经检查服务器显卡没有占用显存的僵尸进程，并且设置batch为1仍然显示oom,本人服务器配置如下：
python:3.7
cuda :11.3
torch:1.10.0
gpu:RTX3090
config:configs/det/efl/efl_yolox_medium.yaml
请问可能是什么问题呢

Hello, thank you very much for your work. In EQL, the cross entropy loss is used in the calculation of background categories. In eqlv2, how to deal with the background category.
and
How to try to understand this sentence ：“one task with positive label and C−1 tasks with negative labels are introduced by a single instance.”
look forward to your reply ！

Error log:
dimensions of scale and zero-point are not consistent with input tensor；

运行指令：时候加上一次evaluate，原始pretrain模型无法运行；

在训练分类任务时，如图，我想请教下1处的数据集是哪部分数据，2处的数据集又是哪部分，为什么差别这么大，以哪个为准？

您好，我运行样例的eval或者train都会提示这个错误，这个是和安装时编译有关吗，按说是安装成功了，感谢帮助

Traceback (most recent call last):
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/CondaEnv/UnitedDetection/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/CondaEnv/UnitedDetection/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/United-Perception/up/main.py", line 27, in
main()
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/United-Perception/up/main.py", line 21, in main
args.run(args)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/United-Perception/up/commands/train.py", line 161, in _main
launch(main, args.num_gpus_per_machine, args.num_machines, args=args, start_method=args.fork_method)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/United-Perception/up/utils/env/launch.py", line 68, in launch
main_func(*(args,))
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/United-Perception/up/commands/train.py", line 140, in main
runner = RUNNER_REGISTRY.get(runner_cfg['type'])(cfg, **runner_cfg['kwargs'])
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/United-Perception/up/runner/base_runner.py", line 60, in init
self.build()
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/United-Perception/up/runner/base_runner.py", line 103, in build
self.build_hooks()
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/United-Perception/up/runner/base_runner.py", line 296, in build_hooks
self._hooks = build_hooks(self, cfg_hooks, add_log_if_not_exists=True)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/United-Perception/up/utils/general/hook_helper.py", line 1114, in build_hooks
hooks = [build_single_hook(cfg) for cfg in cfg_list]
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/United-Perception/up/utils/general/hook_helper.py", line 1114, in
hooks = [build_single_hook(cfg) for cfg in cfg_list]
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/United-Perception/up/utils/general/hook_helper.py", line 1109, in build_single_hook
return HOOK_REGISTRY.build(cfg)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/United-Perception/up/utils/general/registry.py", line 111, in build
raise e
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/United-Perception/up/utils/general/registry.py", line 101, in build
return build_fn(**obj_kwargs)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/United-Perception/up/utils/general/hook_helper.py", line 593, in init
m.register_full_backward_hook(_backward_fn_hook)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-vacv/kaichaoliang/CondaEnv/UnitedDetection/lib/python3.7/site-packages/torch/nn/modules/module.py", line 779, in getattr
type(self).name, name))
torch.nn.modules.module.ModuleAttributeError: 'Conv2d' object has no attribute 'register_full_backward_hook'

Hello,
1、no module named 'petrel_client',；
2、no module named 'spring_aux',；
3、import error No module named 'mqbench'；
4、 import error No module named 'msbench.nn',；
5、free(): invalid pointer.
Can you tell me how to tackle those problem?

基于UP量化YOLOX，QAT训练成功，精度无下降，但是在deploy到tengine的时候出现多个问题，
1：miss key, 缺少量化节点

2：fake_quantize_per_tensor_affine() received an invalid combination of arguments
导出onnx时出现参数不对应的情况,以下为报错日志
deploy_tengine_error.txt

环境：
env: Ubuntu 20.04
RTX3060TI
CUDA: 11.4
Name: torch Version: 1.10.0+cu111
Name: MQBench Version: 0.0.6
onnx 1.7.0

Hello, If I train a network with 2 gpus, the real batchsize is batch_sampler? or 2*batch_sampler?
what about the learning rate?

Thank you

Hello, can this loss function be migrated to other algorithms (e.g. s2anet)? If so, can you give some advice?

Since you are based on the sigmoid classifier, I am curious if your detection results on LVIS will have many false positives in the same location but with different categories. The reason why I ask this is that, I used to train one-stage detector on datasets similar with LVIS (which is long tailed logo dataset, with 352 categories), however, I get many FP with different categories at the same location. I'm wondering if you have encountered the same situation. Thanks!

And I think it may be due to the use of sigmoid classifier which consists of multiple independent binary classifiers. It may be not suitable for multi-classification(num of categories > 1000). Of course this is just my conjecture, any advice is welcome...

作者您好，感谢您精彩的工作。我对于下图展示的EFL的公式有一个疑惑希望您能帮我解答一下：
（1）您提及到对于class imbalance严重的类，gamma值应该要很大，但是同样会带来这一类对于最终loss contributions变少，所以您又加上一个weight factor，但是我在复现您代码的时候有个问题，就是您是分别计算每一类的损失然后求和，那您再计算某一类loss的时候，其他类的gamma值是怎么确定的？直接赋值为0吗？因为在您代码中您是直接乘上一个计算得到的gamma(第二个图)，但是这样对于某一类而言，只是class imbalance类的gamma值非常小但是background类的gamma值却有时比较大，您可以再细化一下这个公式的解释吗，谢谢！

请问你有没有在deformable-detr上使用过这个loss。 我现在想在deformable-detr上使用你文章中的loss，但是复现的时候遇到了一些问题想请教你，下面是我的复现代码。
我的问题是：1， 我在代码中标注的变量的维度正确吗？
2， 如果正确的话，我在计算focusing factor时遇到了一个错误，错误我写在了代码里面。
3，我想直接计算梯度，我的代码中计算梯度的方式正确吗？
4. 你的代码中的ignore_index是起到了什么作用呢？产生的mask是想去除掉什么呢？

Hi,

Just wonder if we can load original custom pretrained yolov5(s~x) model using your work?

or any way? if so, I suggest updating an easy example.

Thanks your great work !

configs/quant/det/faster_rcnn/faster_rcnn_r18_FPN_2x_quant_qdrop.yaml 这个例子里面的resume_path这个半截的fp32模型哪里获取？

hi, I'm testing the pretrained models of EFL.

1. Improved Baseline Series
   
   the results is ok.

2. YOLOX* Series
   
   the ap_iou 0.5 is 0.

the attached is the config yaml
test-yaml.zip

In addition, the following modify added to avoid running error.

您好 看您的实验结果有在yolox上应用efl/eqfl的结果？请问有开源的代码么？

When I tried the knowledge distillation and model search parts of the code, I had problems where kd_runner could not be found and bignas_runner could not be found, respectively. Is the warehouse code uploaded incomplete?

看了您这边用了RFS的 APr很高…能有十几，而且和APc和APf差的不多。
我这边用YOLOX直接训LVIS，用LVIS0.5的时候甚至只有百分之1不到的mAP…我不知道是不是自己写错了…

想在分类任务中使用efl loss，不清楚如何使用

Hello, can you answer the mechanism of action of the gradient collection function? Although the gradient gathering function is defined in the forward propagation function, it does not seem to call this function. Even if self.pos_neg.detach() is used, what is the input parameter in the collect_grad() function? Does it really work?

Hi,
no module named 'petrel_client'
ImportError:cannot import name 'DEPLOY_REGISTRY'

Can you tell me how to improve this problem?

请问作者，我运行./easy_setup.sh时一直编译失败，一直报一些环境问题，例如：
FAILED: /media/disk1/NEW/UniHead-main/build/temp.linux-x86_64-3.8/media/disk1/NEW/UniHead-main/up/extensions/csrc/iou3d_nms/iou3d_nms_kernel.o

FAILED: /media/disk1/NEW/UniHead-main/build/temp.linux-x86_64-3.8/media/disk1/NEW/UniHead-main/up/extensions/csrc/cross_focal_loss/cross_focal_loss_sigmoid_kernel.o
等等

请问作者有其他方式创建环境吗？

另外，这三个路径都必须要指定好是吗？请问能否只指定gcc呢？
export PATH=/your/path/to/gcc-5.3.0/bin/:$PATH # gcc path
export LD_LIBRARY_PATH=/your/path/to/gmp-4.3.2/lib/:/your/path/to/mpfr-2.4.2/lib/:/your/path/to/mpc-0.8.1/lib/:$LD_LIBRARY_PATH # lib path
export TORCH_CUDA_ARCH_LIST='3.5;5.0+PTX;6.0;7.0' # cuda list

hi~, Could you release a support demo for yolov6. https://github.com/meituan/YOLOv6