Giter Club home page Giter Club logo

documentation's Introduction

MegEngine

Documentation | 中文文档

MegEngine is a fast, scalable, and user friendly deep learning framework with 3 key features.

  • Unified framework for both training and inference
    • Quantization, dynamic shape/image pre-processing, and even derivation with a single model.
    • After training, put everything into your model to inference on any platform with speed and precision. Check here for a quick guide.
  • The lowest hardware requirements
    • The memory usage of the GPU can be reduced to one-third of the original memory usage when DTR algorithm is enabled.
    • Inference models with the lowest memory usage by leveraging our Pushdown memory planner.
  • Inference efficiently on all platforms
    • Inference with speed and high-precision on x86, Arm, CUDA, and RoCM.
    • Supports Linux, Windows, iOS, Android, TEE, etc.
    • Optimize performance and memory usage by leveraging our advanced features.

Installation

NOTE: MegEngine now supports Python installation on Linux-64bit/Windows-64bit/MacOS(CPU-Only)-10.14+/Android 7+(CPU-Only) platforms with Python from 3.6 to 3.9. On Windows 10 you can either install the Linux distribution through Windows Subsystem for Linux (WSL) or install the Windows distribution directly. Many other platforms are supported for inference.

Binaries

To install the pre-built binaries via pip wheels:

python3 -m pip install --upgrade pip
python3 -m pip install megengine -f https://megengine.org.cn/whl/mge.html

Building from Source

How to Contribute

We strive to build an open and friendly community. We aim to power humanity with AI.

How to Contact Us

Resources

License

MegEngine is licensed under the Apache License, Version 2.0

Citation

If you use MegEngine in your publication,please cite it by using the following BibTeX entry.

@Misc{MegEngine,
  institution = {megvii},
  title =  {MegEngine:A fast, scalable and easy-to-use deep learning framework},
  howpublished = {\url{https://github.com/MegEngine/MegEngine}},
  year = {2020}
}

Copyright (c) 2014-2021 Megvii Inc. All rights reserved.

documentation's People

Contributors

allcontributors[bot] avatar applepi-icpc avatar asakusarinne avatar caowgg avatar chaibyte avatar chenqy4933 avatar cqchu avatar daisycx avatar dc3671 avatar dingshaohua960303 avatar dzy97 avatar getupearlier avatar haolongzhangm avatar heimayouxia avatar kagome1007 avatar liuyibo avatar machinegunlin avatar megengine-bot avatar megvii-mge avatar oblessnoob avatar qsingle avatar randolph87xxx avatar tpoisonooo avatar unhnq avatar wanwan1996 avatar xgbj avatar xxr3376 avatar yeasoon avatar yzchen avatar zhiy-zhang avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar

Forkers

fyh0210 phoenix0526 getupearlier hexieshenghuo kagome112 qsingle jiaochenlu tpoisonooo bigeagle asasooo anjing137 siyuanseever hello-lx chyelang daisycx thuwangzw flyingtan mylaneza asakusarinne xgbj bealwang 15014199818 haolongzhangm unhnq dzy97 dingshaohua960303 applepi-icpc yeasoon liuwei7299 chenqy4933 caowgg ayf-code qianlu5110 howardchina huahua404 cqchu zhaoqyu mrleebin93 kagome10007 c-krsw lhaippp chaibyte machinegunlin kagome1007 shen940415 summer0616 anthonyive chen-2569 liangmuxin la0balang raymondci miaomiaowu229 vaynezed abelgaluowa chenxingqiang dengxingwenbu dingyuqing123 chenchunmei-daphne noticeable

documentation's Issues

增加“点赞”功能

感觉很多地方写的很棒，但是没有地方点赞👍。
众所周知，点赞👍人类的基本需求，所以有没有办法在页面上加个点赞👍功能？

添加 Boostrap 视觉样式

为了增强文档中元素的视觉效果，需要基于 Boostrap 增加一些基础结构，以 raw html 语法提供。

将来也有可能直接作为 sphinx ext 语法，但是目前还没研究里面的机制。

去除 API 中需求文件系统大小写敏感的 tensor.po

由于 megengine.tensor 是 megengine.Tensor 的别名，因此在早起的 API 文档中对这两个 API 都进行了单独的罗列，问题在于 Sphinx 生成 po 文件时这两个文件会在同一路径下，因此就导致了像 Windows 和 MacOS 默认的文件系统在使用 git 的时候会将两个文件看作同一个文件（大小写不敏感）。这样非常不利于其他非 Linux 系统的用户进行协作，该影响可以视作为 Bug.

发现两处相同的文档错误

在“通过向量化加速运算”部分，公式（y1,y2,...,yd）存在问题，最终应当生成一个n个输出值，即（y1,y2,...,yn）

文档顶部导航栏所用 logo 图片更换

目前的 logo 虽然文件体积小，但是非常模糊不清，需要更加合适的前段资源。

最理想的情况是能提供 svg 格式（但由于天元 Logo 里面信息比较多，难以矢量化，强行转的话图片体积反而会变大）

需求的 logo 规格应该在高度 45 px，宽度 100 - 120 px 左右，清晰度要高于现有 logo

旷视x北大《深度学习实践》视频课提供 Notebook 交互式实战材料

计划中... 尚且不确定是否有人力支持做这件事情。

一个好的形式是：

官网有教程，教程顶部是视频，下面是对应的文本和交互式代码（类似于课程笔记）

如果做的话可能要趁这个机会更新一下前沿的一些新内容，配合一些论文的 MegEngine 代码复现。

当前 API 中文翻译存在的 warning 

/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/functional/math.py:docstring of megengine.functional.math.normalize:4: WARNING: inconsistent term references in translated message. original: [':attr:`axis`'], translated: [':attr:`inp`', ':attr:`axis`']
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/functional/tensor.py:docstring of megengine.functional.tensor.scatter:1: WARNING: inconsistent term references in translated message. original: [], translated: [':attr:`source`', ':attr:`index`']
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/functional/tensor.py:docstring of megengine.functional.tensor.scatter:4: WARNING: inconsistent term references in translated message. original: [], translated: [':attr:`source`', ':attr:`source`', ':attr:`index`']
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/functional/tensor.py:docstring of megengine.functional.tensor.scatter:8: WARNING: inconsistent term references in translated message. original: [], translated: [':attr:`inp`']
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/functional/tensor.py:docstring of megengine.functional.tensor.scatter:14: WARNING: inconsistent term references in translated message. original: [], translated: [':attr:`inp`', ':attr:`index`', ':attr:`source`']
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/functional/tensor.py:docstring of megengine.functional.tensor.scatter:19: WARNING: inconsistent term references in translated message. original: [], translated: [':attr:`index`']
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/data/sampler.py:docstring of megengine.data.sampler.StreamSampler:9: WARNING: inconsistent term references in translated message. original: [], translated: [':meth:`〜..StreamDataset .__ iter__`']
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/data/dataset/vision/cifar.py:docstring of megengine.data.dataset.vision.cifar.CIFAR10:1: WARNING: inconsistent term references in translated message. original: [':class:`~.Dataset`'], translated: []
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/data/dataset/vision/cifar.py:docstring of megengine.data.dataset.vision.cifar.CIFAR100:1: WARNING: inconsistent term references in translated message. original: [':class:`~.Dataset`'], translated: []
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/module/adaptive_pooling.py:docstring of megengine.module.adaptive_pooling.AdaptiveAvgPool2d:3: WARNING: inconsistent term references in translated message. original: [], translated: [':attr:`kernel_size`']
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/module/adaptive_pooling.py:docstring of megengine.module.adaptive_pooling.AdaptiveMaxPool2d:3: WARNING: inconsistent term references in translated message. original: [], translated: [':attr:`kernel_size`']
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/core/tensor/megbrain_graph.py:docstring of megengine.core.tensor.megbrain_graph.set_priority_to_id:3: WARNING: Definition list ends without a blank line; unexpected unindent.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.sync_ema_fakequant_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.sync_ema_fakequant_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.tqt_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.tqt_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.calibration_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.calibration_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.easyquant_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.easyquant_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.ema_fakequant_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.ema_fakequant_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.ema_lowbit_fakequant_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.ema_lowbit_fakequant_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
WARNING: Cannot resolve forward reference in type annotations of "megengine.Tensor": name 'scalar' is not defined
WARNING: Cannot resolve forward reference in type annotations of "megengine.Parameter": name 'scalar' is not defined
/data/jupyter/Documentation/source/reference/tensor.rst:1:<translated>:1: WARNING: Inline interpreted text or phrase reference start-string without end-string.
/data/jupyter/Documentation/source/reference/tensor.rst:1:<translated>:1: WARNING: Inline interpreted text or phrase reference start-string without end-string.
/data/jupyter/Documentation/source/reference/tensor.rst:30:<autosummary>:1: WARNING: inconsistent term references in translated message. original: [':class:`Tensor`', ':attr:`~.Tensor.dtype`'], translated: [':class:`Tensor`']
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/module/qat/module.py:docstring of megengine.module.qat.module.QATModule:1: WARNING: inconsistent term references in translated message. original: [':class:`~.Module`'], translated: []
/data/jupyter/Documentation/source/reference/api/megengine.quantization.min_max_fakequant_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.min_max_fakequant_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.observer.PassiveObserver.rst:43:<autosummary>:1: WARNING: inconsistent term references in translated message. original: [':attr:`qparams`', ':func:`~.apply_easy_quant`'], translated: []
/data/jupyter/Documentation/source/reference/api/megengine.quantization.passive_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
/data/jupyter/Documentation/source/reference/api/megengine.quantization.passive_qconfig.rst:1:<translated>:1: WARNING: Inline literal start-string without end-string.
WARNING: Cannot resolve forward reference in type annotations of "megengine.Parameter": name 'scalar' is not defined
WARNING: Cannot resolve forward reference in type annotations of "megengine.Parameter.__init__": name 'scalar' is not defined
WARNING: Cannot resolve forward reference in type annotations of "megengine.Tensor": name 'scalar' is not defined
WARNING: Cannot resolve forward reference in type annotations of "megengine.Tensor.__init__": name 'scalar' is not defined
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/autodiff/grad_manager.py:docstring of megengine.autodiff.grad_manager.GradManager:47: WARNING: inconsistent term references in translated message. original: [], translated: [':meth:`~.record`']
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/autodiff/grad_manager.py:docstring of megengine.autodiff.grad_manager.GradManager:70: WARNING: inconsistent term references in translated message. original: [':meth:`record`', ':meth:`record`', ':meth:`attach`', ':meth:`backward`'], translated: [':meth:`~.record`（或者输入 ‘statement’），若张量是在 :meth:`~.record`', ':meth:`~.attach`', ':meth:`~.attach`']
/data/jupyter/Documentation/source/reference/data.rst:20:<autosummary>:1: WARNING: inconsistent term references in translated message. original: [], translated: [':class:`numpy.ndarray`']
/data/jupyter/Documentation/source/reference/functional.rst:52:<autosummary>:1: WARNING: inconsistent term references in translated message. original: [], translated: [':attr:`index`', ':attr:`axis`', ':attr:`inp`']
/data/jupyter/Documentation/source/reference/functional.rst:52:<autosummary>:1: WARNING: inconsistent term references in translated message. original: [], translated: [':attr:`source`', ':attr:`index`']
/data/jupyter/Documentation/source/reference/module.rst:153:<autosummary>:1: WARNING: inconsistent term references in translated message. original: [':func:`~.matmul`', ':func:`~.relu`'], translated: [':func:`~.relu`']
/data/jupyter/Documentation/source/reference/module.rst:187:<autosummary>:1: WARNING: inconsistent term references in translated message. original: [':class:`~.Module`'], translated: []
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/module/batch_matmul_activation.py:docstring of megengine.module.batch_matmul_activation.BatchMatMulActivation:1: WARNING: inconsistent term references in translated message. original: [':func:`~.matmul`', ':func:`~.relu`'], translated: [':func:`~.relu`']
/home/caoxiaowei/.local/lib/python3.7/site-packages/megengine/module/elemwise.py:docstring of megengine.module.elemwise.Elemwise:1: WARNING: inconsistent term references in translated message. original: [':class:`~.Module`', ':mod:`~.functional.elemwise`', ':class:`~.QATModule`', ':class:`~.qat.Elemwise`', ':func:`~.quantize.quantize_qat`'], translated: [':mod:`~.functional.elemwise`', ':class:`~.Module`', ':func:`~.quantize.quantize_qat`', ':class:`~.QATModule`']

API 文档中的一些类型（包括 typehint）依旧显示完整路径

比如我们暴露给用户这样一个 Class:

megengine.module.Module

它的实际实现在：

megengine.module.module.Module

一些继承 Module 的 Class 如 Sequential 在显示基类时会显示成后者，而不是我们希望暴露给用户的形式。

这个问题目前不打算从下游进行解决，而是交给 Sphinx 的 autodoc 和 autosummary 进行改进。

Refs:

MEPs 计划迁移到 MegEngine 文档

当前问题：

  • 用作 Index 和 Template 的 MEP 需要进行区分，目前没有 MEP Template
  • 一些 MEP 形成过程中的完整讨论和决策建议以 Issue 的形式进行保留，整理后的版本放在对应 MEP 中
  • 目前没有明确的 Process 流程，建议参考 PEP Purpose and Guidelines 重新讨论和拟定
  • 要求所有人开始习惯用 RST 语法，可以在 MEPs 内容中对一些概念进行有效的交叉引用

MegEngine 教程（Tutorials）征集

目前预计结构，还待完善：

  • 天元 MegEngine 快速上手 （快速展示 API）

深度学习零基础 LEARNING-BY-DOING

这一部分面向绝对零基础的 Python 用户，不打算分模块，而是 i + 1 任务式讲解。

  • 天元 MegEngine 基础概念：张量/算子/计算图/求导器/优化器/损失函数
  • 一个稍微复杂些的线性回归模型：数据集基本概念/Dataloader/Sampler/梯度下降
  • 从线性回归到线性分类：不同的机器学习任务类型/理解图像数据
  • 从感知机到深度神经网络：funtional 和 module/模型保存和加载
  • 神经网络训练技巧：...
  • 炼丹并不是深度学习的全部：普及推理部署相关的概念，实际教程放在“用户指南”

这一部分的教程对于有经验的用户来说会显得有些啰里八嗦的，但重点在于培养直觉和思维。

趣味教程

打算在这放一些有意思的 DEMO 案例，带图片，主要是吸引用户兴趣。

预设 QConfig 的 说明不正常

目前是直接来自 QConfig 类的 docstring。

建议修改为以下说明：

name intro
min_max_fakequant_qconfig 一个使用 :class:~.MinMaxObserver 和 :class:~.FakeQuant 的预设 QConfig。
ema_fakequant_qconfig 一个使用 :class:~.ExponentialMovingAverageObserver 和 :class:~.FakeQuant 的预设 QConfig。
sync_ema_fakequant_qconfig 一个使用 :class:~.SyncExponentialMovingAverageObserver 和 :class:~.FakeQuant 的预设 QConfig。
ema_lowbit_fakequant_qconfig 一个使用 :class:~.ExponentialMovingAverageObserver 和 :class:~.FakeQuant 且数值类型为 qint4 的预设 QConfig。
calibration_qconfig 一个对激活值使用 :class:~.HistogramObserver 进行后量化（无 :class:~.FakeQuant ）的预设 QConfig。
tqt_qconfig 一个使用 :class:~.TQT 进行假量化的预设 QConfig。
passive_qconfig 一个使用 :class:~.PassiveObserver 和 :class:~.FakeQuant 的预设 QConfig。
easyquant_qconfig 用于 easyquant 算法的 QConfig，等价于 passive_qconfig 。

CI 逻辑优化

  1. 多语言构建环境 prepare （尤其是 cache 检查）这一块是可以公用的
  2. oss 的部署逻辑现在可以完全由 push main 行为触发

在不同的计算设备之间迁移部分，不是 MegEngine 的设计初衷

MegEngine 的设计是希望用户用 set_default_device 全局指定设备（且我们多卡并行是依靠多进程实现，因此几乎不需要涉及到跨设备），而不是依靠 to 频繁在设备间拷贝（这会使得模型比较低 & 更不容易在推理侧运行）。因此建议在 Tutorial 部分只提 set_default_device 即可，to 的使用属于极特殊情况，建议在高阶部分再提及。

希望添加Hub的文档

加载预训练权重的需求比较常见，希望在文档增加Hub相关的文档方便查找

规范社区 Git 贡献流程

  • 确立分支开发工作流程方案
  • 在文档中对工作流程进行说明
  • 文档本地部署教程
  • 文档本地编辑教程：RST 语法 demo 等

基于 module 的统计参数量的示例code微调

在章节 https://megengine.org.cn/doc/stable/zh/user-guide/tools/module-stats.html 中

from megengine.hub import load
from megengine.utils.module_stats import module_stats

# 构建一个 net module，这里从 model hub 中获取 resnet18 模型
net = load("megengine/models", "resnet18", pretrained=True)

# 指定输入 shape
input_shape = (1, 3, 224, 224)

# Float model.
total_params, total_flops = module_stats(
    net, input_shape, log_params=True, log_flops=True
)
print("params {} flops {}".format(total_params, total_flops))

建议把input_shape改成input_shape_list，一般模型多会吃多个输入，像示例这样的写法容易造成误解，或者增加注释

关于文档的多语言支持

之前调研过 Crowdin 方案，可免费用于开源项目，好处是以第三方的形式接入了一些团队功能且有丰富的平台集成。之前我的个人判断是：由于现有官网 API 没经过组织和整理，一些 API 被遗漏，而一些不该暴露出来的接口也被做了本地化，导致这块实际产出没搞对方向。所以当务之急应该是先把已有的需要被翻译的英文 API 弄完备，再开始将重心转移到 po 文件 msg 的翻译，中途可以做部分内容的翻译流程尝试，看之前已经翻译好的东西能否拿过来用。 @xxr3376 需要找个时间提交开源项目申请表给 Crowdin 团队， NumPy 2020 年底开始尝试用 Crowdin，只做了网站一些 Markdown 的翻译，全靠各国志愿者（贡献者能够更方便地参与到翻译工作中来也是这个平台的优势）。

添加 MegEngine 架构设计

路径：source/development/architecture-design.rst

先在这个 Issue 下面讨论内容结构，预期效果等。

需要补充文档的组织逻辑和 RST 语法

  • 区分新手入门（教程）和用户指南（说明书）的范畴
  • RST 语法参考，比如
    • 一致的多级标题使用规范
    • 图片的插入语法 （figure/image）
    • 交叉引用和超链接

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.