linux-nvme / libnvme Goto Github PK
View Code? Open in Web Editor NEWC Library for NVM Express on Linux
License: GNU Lesser General Public License v2.1
libnvme's People
Contributors
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Stargazers
Watchers
Forkers
johnmeneghini safl clicx taranovk tbzatek hreinecke ravivarma1103 wubo009 martin-belanger shampoo365 birkelund jk-ozlabs codeconstruct surfndez franciscomunoz zmole945 matiasbjorling verschwindibussen metaspace petaio-tw aliakseimakarau igaw mallikarjunasai995 upasana123-tech shaowei sc108-lee minwooim jonathan-teh arunpandian15 wxiong38 maurizio-lombardi ikegami-t pipcet arka1988 tasleson jeffreyalien glimchb nurked samiwaheed hanumanthuh akhilesh-rn-wdc bjpaupor martin-gpy totalcaesar659 eli-schwartz t-8ch anuj7781 ankit-sam sky8273 innogritkenny omersne moshibin mjlee34 ismaell francis-pravin drakedog2008 parkrish lq23546 jinliangw wangmengmeng02 bdrung dankamongmen timberland-sig huns-jeon opensuse sberbz keithbusch vikash-k5 zwxdg jeffreycheung joelgranados wlbe4 jellor usaleem-ix calebsander benetti-engineering-sas brandonmfong vaishali59 haifong cvubrugier steeldomejeff juliensu jamestiotio eddakathle mowangdk oneman solidigm bpaupore-wdc fzcz thesamesam parona-source jgu-pure prashanth-nayak lisating0 budikxyz wchangsong cjp256 natethornton gioragg1 liang3zy22libnvme's Issues
No trsvcid specified for 'rdma'
# nvme connect -t rdma -a 192.1.11.151 -n nqn.XXX -Q 1024 -l 3600
No trsvcid specified for 'rdma'
# nvme connect -t rdma -trsvcid=4420 -a 192.1.11.151 -n nqn.XXX -Q 1024 -l 3600@martin-gpy suggested:
@@ -1014,16 +1014,8 @@ struct nvme_ctrl *nvme_create_ctrl(nvme_
c->cfg.host_iface = strdup(host_iface);
if (trsvcid)
c->trsvcid = strdup(trsvcid);
- else if (discovery)
+ else
discovery_trsvcid(c);
- else if (!strncmp(transport, "rdma", 4) ||
- !strncmp(transport, "tcp", 3)) {
- nvme_msg(r, LOG_ERR, "No trsvcid specified for '%s'\n",
- transport);
- errno = EINVAL;
- __nvme_free_ctrl(c);
- c = NULL;
- }Should kato be set by default when setting persistent?
The function nvme_ctrl_set_persistent() sets the boolean c->persistent as follows:
void nvme_ctrl_set_persistent(nvme_ctrl_t c, bool persistent)
{
c->persistent = persistent;
}
From what I can tell, the boolean c->persistent is not really used for anything (at least from the perspective of configuring the kernel). Should we use c->persistent to set a default KATO value for persistent connections to discovery controllers?
For example:
void nvme_ctrl_set_persistent(nvme_ctrl_t c, bool persistent)
{
c->persistent = persistent;
if (c->persistent && c->discovery_ctrl && c->cfg.keep_alive_tmo == 0) {
c->cfg.keep_alive_tmo = DEFAULT_KATO; /* Default could be 120 sec (as in base specs) or any value we agree on */
}
}
meson test is not generating python module on fedora
this works
docker build https://github.com/linux-nvme/libnvme.git#v1.0-rc5 -f-<<EOF
FROM ubuntu:20.04
RUN apt-get update -y && DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get -y install tzdata
RUN apt-get install -y gcc g++ cmake meson swig libjson-c-dev libssl-dev python3-pip python3-dev pkg-config libuuid1 uuid-dev uuid-runtime
WORKDIR /root
COPY . .
RUN meson setup .build --werror && meson test -C .build --verbose
EOF
this fails
docker build https://github.com/linux-nvme/libnvme.git#v1.0-rc5 -f-<<EOF
FROM fedora:33
RUN dnf install -y gcc g++ cmake meson swig json-c-devel openssl-devel python3-pip python-devel pkg-config libuuid-devel
WORKDIR /root
COPY . .
RUN meson setup .build --werror && meson test -C .build --verbose
EOF
Comparing the output:
Sending build context to Docker daemon 2.592MB
Step 1/6 : FROM ubuntu:20.04
---> 2b4cba85892a
Step 2/6 : RUN apt-get update -y && DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get -y install tzdata
---> Using cache
---> bf1e22384eae
Step 3/6 : RUN apt-get install -y gcc g++ cmake meson swig libjson-c-dev libssl-dev python3-pip python3-dev pkg-config libuuid1 uuid-dev uuid-runtime
---> Using cache
---> 1d36b23ad48d
Step 4/6 : WORKDIR /root
---> Using cache
---> 0ab3d684eeb5
Step 5/6 : COPY . .
---> 3a09d530e53f
Step 6/6 : RUN meson setup .build --werror && meson test -C .build --verbose
---> Running in a916ee9a47a8
The Meson build system
Version: 0.53.2
Source dir: /root
Build dir: /root/.build
Build type: native build
Project name: libnvme
Project version: 1.0
C compiler for the host machine: cc (gcc 9.4.0 "cc (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0")
C linker for the host machine: cc ld.bfd 2.34
C++ compiler for the host machine: c++ (gcc 9.4.0 "c++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0")
C++ linker for the host machine: c++ ld.bfd 2.34
Host machine cpu family: x86_64
Host machine cpu: x86_64
Found pkg-config: /usr/bin/pkg-config (0.29.1)
Run-time dependency uuid found: YES 2.34.0
Run-time dependency json-c found: YES 0.13.1
Run-time dependency openssl found: YES 1.1.1f
Checking if "__builtin_type_compatible_p" compiles: YES
Checking if "typeof" compiles: YES
Checking if "byteswap.h" compiles: YES
Checking if "bswap64" links: YES
Checking if "statement-expr" compiles: YES
Checking if "isblank" links: YES
Configuring libnvme.spec using configuration
Configuring config.h using configuration
Program python3 found: YES (/usr/bin/python3)
Dependency python found: YES (pkgconfig)
Program swig found: YES (/usr/bin/swig)
Has header "Python.h" with dependency python: YES
Build targets in project: 11
Found ninja-1.10.0 at /usr/bin/ninja
ninja: Entering directory `/root/.build'
[1/32] Compiling C object 'ccan/fc15840@@ccan@sta/ccan_str_debug.c.o'.
[2/32] Compiling C object 'ccan/fc15840@@ccan@sta/ccan_str_str.c.o'.
[3/32] Compiling C object 'src/25a6634@@nvme@sha/nvme_cleanup.c.o'.
[4/32] Compiling C object 'ccan/fc15840@@ccan@sta/ccan_list_list.c.o'.
[5/32] Compiling C object 'src/25a6634@@nvme@sha/nvme_filters.c.o'.
[6/32] Linking static target ccan/libccan.a.
[7/32] Compiling C object 'test/9f86d08@@test-zns@exe/zns.c.o'.
[8/32] Compiling C object 'src/25a6634@@nvme@sha/nvme_log.c.o'.
[9/32] Compiling C object 'examples/c590b3c@@discover-loop@exe/discover-loop.c.o'.
[10/32] Compiling C object 'test/9f86d08@@test-register@exe/register.c.o'.
[11/32] Compiling C object 'examples/c590b3c@@display-columnar@exe/display-columnar.c.o'.
[12/32] Compiling C object 'examples/c590b3c@@telemetry-listen@exe/telemetry-listen.c.o'.
[13/32] Compiling C object 'test/9f86d08@@main-test@exe/test.c.o'.
[14/32] Compiling C object 'src/25a6634@@nvme@sha/nvme_json.c.o'.
[15/32] Compiling C object 'src/25a6634@@nvme@sha/nvme_linux.c.o'.
[16/32] Compiling C object 'src/25a6634@@nvme@sha/nvme_util.c.o'.
[17/32] Compiling C object 'src/25a6634@@nvme@sha/nvme_fabrics.c.o'.
[18/32] Compiling C++ object 'test/9f86d08@@test-cpp@exe/cpp.cc.o'.
[19/32] Compiling C object 'src/25a6634@@nvme@sha/nvme_ioctl.c.o'.
[20/32] Generating nvme.py with a custom command.
[21/32] Compiling C object 'src/25a6634@@nvme@sha/nvme_tree.c.o'.
[22/32] Linking target src/libnvme.so.1.0.0.
[23/32] Generating symbol file 'src/25a6634@@nvme@sha/libnvme.so.1.0.0.symbols'.
[24/32] Linking target test/test-register.
[25/32] Linking target test/main-test.
[26/32] Linking target examples/display-columnar.
[27/32] Linking target test/test-zns.
[28/32] Linking target examples/discover-loop.
[29/32] Linking target examples/telemetry-listen.
[30/32] Linking target test/test-cpp.
[31/32] Compiling C object 'libnvme/715d4c0@@_nvme.cpython-38-x86_64-linux-gnu@sha/meson-generated_.._nvme_wrap.c.o'.
[32/32] Linking target libnvme/_nvme.cpython-38-x86_64-linux-gnu.so.
Test filter for common loop back target
Test scan specific controller
Test walking the topology
Test identification, logs, and features
Test filter for common loop back target
Test scan specific controller
Test walking the topology
Test identification, logs, and features
Test filter for common loop back target
Test scan specific controller
Test walking the topology
Test identification, logs, and features
Test filter for common loop back target
Test scan specific controller
Test walking the topology
Test identification, logs, and features
Test filter for common loop back target
Test scan specific controller
Test walking the topology
Test identification, logs, and features
1/7 [Python] import libnvme OK 0.04 s
2/7 [Python] create ctrl object OK 0.07 s
3/7 main OK 0.01 s
4/7 main OK 0.01 s
5/7 cpp OK 0.00 s
6/7 register OK 0.00 s
7/7 zns OK 0.00 s
Ok: 7
Expected Fail: 0
Fail: 0
Unexpected Pass: 0
Skipped: 0
Timeout: 0
Removing intermediate container a916ee9a47a8
---> e61b5baf3950
Successfully built e61b5baf3950
vs
Sending build context to Docker daemon 2.592MB
Step 1/5 : FROM fedora:33
---> c27e4553c753
Step 2/5 : RUN dnf install -y gcc g++ cmake meson swig json-c-devel openssl-devel python3-pip python-devel pkg-config libuuid-devel
---> Using cache
---> bd86792e3b91
Step 3/5 : WORKDIR /root
---> Using cache
---> 749c5fd1693d
Step 4/5 : COPY . .
---> b2218b29bcb8
Step 5/5 : RUN meson setup .build --werror && meson test -C .build --verbose
---> Running in bf672b79708b
The Meson build system
Version: 0.59.4
Source dir: /root
Build dir: /root/.build
Build type: native build
Project name: libnvme
Project version: 1.0
C compiler for the host machine: cc (gcc 10.3.1 "cc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1)")
C linker for the host machine: cc ld.bfd 2.35-18
C++ compiler for the host machine: c++ (gcc 10.3.1 "c++ (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1)")
C++ linker for the host machine: c++ ld.bfd 2.35-18
Host machine cpu family: x86_64
Host machine cpu: x86_64
Found pkg-config: /usr/bin/pkg-config (1.7.3)
Run-time dependency uuid found: YES 2.36.1
Run-time dependency json-c found: YES 0.14
Run-time dependency openssl found: YES 1.1.1l
Checking if "__builtin_type_compatible_p" compiles: YES
Checking if "typeof" compiles: YES
Checking if "byteswap.h" compiles: YES
Checking if "bswap64" links: YES
Checking if "statement-expr" compiles: YES
Checking if "isblank" links: YES
Configuring libnvme.spec using configuration
Configuring config.h using configuration
Program python3 found: YES (/usr/bin/python3)
Dependency python found: YES (pkgconfig)
Program swig found: YES (/usr/bin/swig)
Has header "Python.h" with dependency python: YES
Build targets in project: 11
Found ninja-1.10.2 at /usr/bin/ninja
ninja: Entering directory `/root/.build'
[1/17] Compiling C object ccan/libccan.a.p/ccan_str_debug.c.o
[2/17] Compiling C object src/libnvme.so.1.0.0.p/nvme_cleanup.c.o
[3/17] Compiling C object ccan/libccan.a.p/ccan_str_str.c.o
[4/17] Compiling C object ccan/libccan.a.p/ccan_list_list.c.o
[5/17] Linking static target ccan/libccan.a
[6/17] Compiling C object src/libnvme.so.1.0.0.p/nvme_log.c.o
[7/17] Compiling C object src/libnvme.so.1.0.0.p/nvme_filters.c.o
[8/17] Compiling C object src/libnvme.so.1.0.0.p/nvme_util.c.o
[9/17] Compiling C object src/libnvme.so.1.0.0.p/nvme_linux.c.o
[10/17] Compiling C object test/main-test.p/test.c.o
[11/17] Compiling C object src/libnvme.so.1.0.0.p/nvme_json.c.o
[12/17] Compiling C object src/libnvme.so.1.0.0.p/nvme_fabrics.c.o
[13/17] Compiling C object src/libnvme.so.1.0.0.p/nvme_ioctl.c.o
[14/17] Compiling C object src/libnvme.so.1.0.0.p/nvme_tree.c.o
[15/17] Linking target src/libnvme.so.1.0.0
[16/17] Generating symbol file src/libnvme.so.1.0.0.p/libnvme.so.1.0.0.symbols
[17/17] Linking target test/main-test
1/7 [Python] import libnvme RUNNING
>>> MALLOC_PERTURB_=0 PYTHONMALLOC=malloc PYTHONPATH=/root/.build/libnvme/.. /usr/bin/python3 -c 'from libnvme import nvme'
2/7 [Python] create ctrl object RUNNING
>>> MALLOC_PERTURB_=0 PYTHONMALLOC=malloc PYTHONPATH=/root/.build/libnvme/.. /usr/bin/python3 /root/.build/../libnvme/tests/create-ctrl-obj.py
3/7 main RUNNING
>>> MALLOC_PERTURB_=133 /root/.build/test/main-test
4/7 main RUNNING
>>> MALLOC_PERTURB_=170 /root/.build/test/main-test nvme10
5/7 cpp RUNNING
>>> MALLOC_PERTURB_=181 /root/.build/test/main-test
6/7 register RUNNING
>>> MALLOC_PERTURB_=171 /root/.build/test/main-test nvme10
7/7 zns RUNNING
>>> MALLOC_PERTURB_=120 /root/.build/test/main-test
3/7 main OK 0.02s
――――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――
Test filter for common loop back target
Test scan specific controller
Test walking the topology
Test identification, logs, and features
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
4/7 main OK 0.01s
――――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――
Test filter for common loop back target
Test scan specific controller
Test walking the topology
Test identification, logs, and features
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
5/7 cpp OK 0.01s
――――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――
Test filter for common loop back target
Test scan specific controller
Test walking the topology
Test identification, logs, and features
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
6/7 register OK 0.01s
――――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――
Test filter for common loop back target
Test scan specific controller
Test walking the topology
Test identification, logs, and features
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
7/7 zns OK 0.01s
――――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――
Test filter for common loop back target
Test scan specific controller
Test walking the topology
Test identification, logs, and features
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
2/7 [Python] create ctrl object FAIL 0.02s exit status 1
――――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――
stderr:
Traceback (most recent call last):
File "/root/.build/../libnvme/tests/create-ctrl-obj.py", line 4, in <module>
from libnvme import nvme
ImportError: cannot import name 'nvme' from 'libnvme' (/root/.build/libnvme/__init__.py)
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
1/7 [Python] import libnvme FAIL 0.03s exit status 1
――――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――
stderr:
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: cannot import name 'nvme' from 'libnvme' (/root/.build/libnvme/__init__.py)
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
Summary of Failures:
2/7 [Python] create ctrl object FAIL 0.02s exit status 1
1/7 [Python] import libnvme FAIL 0.03s exit status 1
Ok: 5
Expected Fail: 0
Fail: 2
Unexpected Pass: 0
Skipped: 0
Timeout: 0
Full log written to /root/.build/meson-logs/testlog.txt
The command '/bin/sh -c meson setup .build --werror && meson test -C .build --verbose' returned a non-zero code: 2
Looks like dependency is missing and python extension_module is not generated/called
Missing this line
[20/32] Generating nvme.py with a custom command.
nvmf_add_ctrl always marks ctrl as disovered
Line 562 in 20b6c2d
nvmf_add_ctrl always marks all controllers as discovered. Is this correct? What if the the ctrl passed in was not created via the discovery mechanisme? This looks suspicious.
Missing include <stddef.h>
For the following:
#include <libnvme.h>
int main() {
return 0;
}You get ...
In file included from /home/tasleson/projects/libnvme/src/libnvme.h:18,
from empty.c:1:
/home/tasleson/projects/libnvme/src/nvme/linux.h:110:34: error: unknown type name ‘size_t’
110 | int nvme_get_ana_log_len(int fd, size_t *analen);
| ^~~~~~
/home/tasleson/projects/libnvme/src/nvme/linux.h:13:1: note: ‘size_t’ is defined in header ‘<stddef.h>’; did you forget to ‘#include <stddef.h>’?
12 | #include "types.h"
+++ |+#include <stddef.h>
13 |
wrong hostnqn uuid
The dmi hostnqn uuid code seems broken. I got following downstream bug report:
# Wrong hostnqn UUID:
dhcp-10-231-46-32:~ # nvme gen-hostnqn
nqn.2014-08.org.nvmexpress:uuid:38363739-3539-4d58-5138-33333030585a
# Correct hostnqn UUID:
dhcp-10-231-46-32:~ # dmidecode -s system-uuid
39373638-3935-584d-5138-33333030585a
It looks like nvme-cli is not byteswapping the first 4 bytes.
and I can reproduce it locally.
Python binding test
Add at least a few simple test for the Python binding to the test framework. Currently, we are blind if a change in the shared
library will break the binding.
Is nvme_subsystem_lookup_namespace doing the right thing?
nvme_subsystem_lookup_namespace()
ret = asprintf(&name, "%sn%u", s->name, nsid);
if (ret < 0)
return NULL;(gdb) p name
$1 = 0x715260 "nvme-subsys1n17"
(gdb) p *s
$2 = {entry = {next = 0x70fbe0, prev = 0x70fbe0}, ctrls = {n = {next = 0x7193f0, prev = 0x70ffc0}},
namespaces = {n = {next = 0x715100, prev = 0x715100}}, h = 0x70fbd0, name = 0x70fee0 "nvme-subsys1",
sysfs_dir = 0x70fba0 "/sys/class/nvme-subsystem/nvme-subsys1",
subsysnqn = 0x70fcd0 "nqn.1988-11.com.dell:powerstore:00:f03028e73ef7D032D81E",
model = 0x70fdf0 "dellemc-powerstore", serial = 0x70fe10 "85JZX23", firmware = 0x70fe30 "2.1.0.0",I wanted to lookup nsid=17 but this obviously /dev/nvme-subsys1n17 doesn't exit. Is this function really
doing what's supposed to do?
compare between `mesonpy` and `mesonpep517` python build backends
Issue/reminder for myself @glimchb to learn more and compare:
Versioning : git tags, release, meson project version and library version
what is the right thing to do ?
let's say today we are at https://github.com/linux-nvme/libnvme/releases/tag/v1.0-rc6
today there are too many places to update:
- git tag and github release
- meson project version
- library version (taken from above)
- python package version
when uploading to pypi we can't override the same 1.0 again and again.
See https://test.pypi.org/help/#file-name-reuse
so we want to add rc and treat them as pre-release
see example here https://pypi.org/project/OctoPrint/#history
also see more discussions and systemd and dpdk versioning here #95
[loop] Invalid subsystem records after scanning
This is somewhat similar to #154 except this fails with just a single connection. We've been using the loop transport for simple setups in our CI and found this via segfaulting nvme-cli:
Steps to reproduce:
- Setup a simple target using the loop transport: target.json
# nvme connect --transport=loop --hostnqn=test_hostnqn --nqn=test_subnqn# nvme list --verbose
Subsystem Subsystem-NQN Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys0 test_subnqn
(null) test_subnqn nvme0
Device SN MN FR TxPort Address Subsystem Namespaces
-------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
nvme0 d55ce7cdd937df5c7dea Linux 5.17.0-0 loop (null)
Device Generic NSID Usage Format Controllers
------------ ------------ -------- -------------------------- ---------------- ----------------
nvme0n2 ng0n2 2 1.07 GB / 1.07 GB 512 B + 0 B
nvme0n1 ng0n1 1 1.07 GB / 1.07 GB 512 B + 0 B
# nvme list --output-format=json --verbose
Segmentation fault (core dumped)
(this fails on nvme_subsystem_get_name() returning NULL)
#0 0x00007ffff7ed7dbd in __strlen_avx2 () from /lib64/libc.so.6
#1 0x00007ffff7f78e2d in json_object_new_string (s=0x0) at /usr/src/debug/json-c-0.15-3.fc36.x86_64/json_object.c:1290
#2 0x000000000043e8e5 in json_detail_list (r=0x4d4be0) at ../nvme-print.c:7102
I haven't looked at where this actually comes from, reporting early.
Missing dependencies
Issue:
When building libnvme the build fails due to the missing pyhon3 development headers.
$ meson .build
The Meson build system
Version: 0.54.2
Source dir: /home/ec2-user/libnvme
Build dir: /home/ec2-user/libnvme/.build
Build type: native build
Project name: libnvme
Project version: v0.1
C compiler for the host machine: cc (gcc 7.5.0 "cc (SUSE Linux) 7.5.0")
C linker for the host machine: cc ld.bfd 2.35.1.20201123-7
C++ compiler for the host machine: c++ (gcc 7.5.0 "c++ (SUSE Linux) 7.5.0")
C++ linker for the host machine: c++ ld.bfd 2.35.1.20201123-7
Host machine cpu family: x86_64
Host machine cpu: x86_64
Found pkg-config: /usr/bin/pkg-config (0.29.2)
Run-time dependency uuid found: YES 2.33.1
Run-time dependency json-c found: YES 0.15
Checking if "__builtin_type_compatible_p" compiles: YES
Checking if "typeof" compiles: YES
Checking if "byteswap.h" compiles: YES
Checking if "bswap64" links: YES
Checking if "statement-expr" compiles: YES
Checking if "isblank" links: YES
Configuring config.h using configuration
Configuring libnvme.spec using configuration
Program python3 found: YES (/usr/bin/python3)
Program swig found: YES (/usr/bin/swig)
Dependency python found: YES (sysconfig)
Build targets in project: 10
Found ninja-1.10.0 at /usr/bin/ninja
$ cd .build/
$ ninja
[18/27] Compiling C object 'pynvme/37eb337@@_nvme.cpython-36m-x86_64-linux-gnu@sha/meson-generated_.._nvme_wrap.c.o'
FAILED: pynvme/37eb337@@_nvme.cpython-36m-x86_64-linux-gnu@sha/meson-generated_.._nvme_wrap.c.o
cc -Ipynvme/37eb337@@_nvme.cpython-36m-x86_64-linux-gnu@sha -Ipynvme -I../pynvme -I. -I.. -I../ccan -Isrc -I../src -I/usr/include/python3.6m -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -fomit-frame-pointer -D_GNU_SOURCE -include config.h -fPIC -MD -MQ 'pynvme/37eb337@@_nvme.cpython-36m-x86_64-linux-gnu@sha/meson-generated_.._nvme_wrap.c.o' -MF 'pynvme/37eb337@@_nvme.cpython-36m-x86_64-linux-gnu@sha/meson-generated_.._nvme_wrap.c.o.d' -o 'pynvme/37eb337@@_nvme.cpython-36m-x86_64-linux-gnu@sha/meson-generated_.._nvme_wrap.c.o' -c pynvme/nvme_wrap.c
pynvme/nvme_wrap.c:149:11: fatal error: Python.h: No such file or directory
# include <Python.h>
^~~~~~~~~~
compilation terminated.
[19/27] Generating symbol file 'src/25a6634@@nvme@sha/libnvme.so.0.0.0.symbols'
ninja: build stopped: subcommand failed.
Fix
Add python3-devel to meson dependencies. I would do it myself but I never used meson in dev, so I figured someone else would be faster than me adding this small feature.
long double does not fit with json-c
https://github.com/linux-nvme/nvme-cli/blob/master/nvme-print.c#L188
https://github.com/linux-nvme/nvme-cli/blob/master/nvme.h#L47
128 bits long double is not fit in 64 bits double
https://github.com/json-c/json-c/blob/master/json_object.c#L1066
Add timeout to all commands
@hreinecke pointed out that it doesn't make sense to have only a timeout argument for the ZNS commands. Let's add them to the rest of the commands as well.
See also #94
Update CI workflow for meson build
The CI workflow will need to be updated for the meson build; ideally we should be adding a 'test' step, too, to run sanity tests for each build.
Visibility for library functions __nvme_msg, __nvme_get_log_page
$ nm --defined-only -g ./src/libnvme.so.1.0.1 | grep __
0000000000007d10 T __nvme_get_log_page
0000000000010860 T __nvme_msg
From the naming, it would seem like these shouldn't be exposed?
Should we drop 'attribute packed' in ioctl.h?
We introduced attribute packed to make sure the layout is stable for all compilers (ABI). Though with just the attribute packed the compiler assumes the sizeof(struct foo) is 1, so there are other aligenment necessary. An array of such a type wouldn't have padding after the each object. So we added the attribute align(sizeof(biggest member)). While this fixes the other alignment, it introduces a new problem. The trick to figure out if the struct has changed and hence it is a different ABI doesn't work unless we pad the struct up again.
In my opinion there is no real gain from adding the attribute packed and I think we should just drop it as we reordered all members in the argument structs in ioctl.h so there is no hole.
Any strong opinions for keeping the attributes?
libnvme openssl meson subproject compilation fails
This works fine: ( using pre-installed packages and skips meson subprojects)
docker build https://github.com/linux-nvme/libnvme.git#master -f-<<EOF
FROM fedora:33
RUN dnf install -y gcc g++ cmake libuuid-devel swig python-devel meson
RUN dnf install -y openssl-devel json-c-devel
WORKDIR /root
COPY . .
RUN meson .build && ninja -C .build && meson install -C .build
EOF
This fails: ( uses meson subprojects instead of pre-installed packages)
docker build https://github.com/linux-nvme/libnvme.git#master -f-<<EOF
FROM fedora:33
RUN dnf install -y gcc g++ cmake libuuid-devel swig python-devel meson
WORKDIR /root
COPY . .
RUN meson .build && ninja -C .build && meson install -C .build
EOF
With this error: (looks like installation stage)
[753/754] Compiling C object libnvme/_nvme.cpython-39-x86_64-linux-gnu.so.p/meson-generated_.._nvme_wrap.c.o
[754/754] Linking target libnvme/_nvme.cpython-39-x86_64-linux-gnu.so
ninja: Entering directory `/root/.build'
ninja: no work to do.
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/mesonbuild/mesonmain.py", line 228, in run
return options.run_func(options)
File "/usr/lib/python3.9/site-packages/mesonbuild/minstall.py", line 720, in run
installer.do_install(datafilename)
File "/usr/lib/python3.9/site-packages/mesonbuild/minstall.py", line 512, in do_install
self.install_targets(d, dm, destdir, fullprefix)
File "/usr/lib/python3.9/site-packages/mesonbuild/minstall.py", line 615, in install_targets
raise RuntimeError(f'File {t.fname!r} could not be found')
RuntimeError: File 'subprojects/openssl-1.1.1l/openssl' could not be found
Installing subprojects/json-c-0.15/libjson-c.so to /usr/lib64
Installing subprojects/openssl-1.1.1l/libcrypto.so to /usr/lib64
Installing subprojects/openssl-1.1.1l/libssl.so to /usr/lib64
The command '/bin/sh -c meson .build && ninja -C .build && meson install -C .build' returned a non-zero code: 2
I can't figure out what is missing during compile stage that fails during installation...
Fill holes in ioctl's argument structs
See discussion in #150
doc/meson.build licence
Hi Martin,
the doc/meson.build file contains following SPDX license text"
# SPDX-License-Identifier: BSD-3
Please send a PR which changes this to
# SPDX-License-Identifier: LGPL-2.1-or-later
Also the descriptionis wrong:
# This file is part of STorage Appliance Services (STAS).
Thanks,
Daniel
Python tests fail if library is embedded
When the library is embedded the testsuite fails in the Python tests:
https://github.com/linux-nvme/nvme-cli/runs/4079968220?check_suite_focus=true
I suspect the paths are not correct.
__nvme_msg() is not checking for NULL before dereferencing pointer
Function __nvme_msg() could be called with a NULL r parameter. In fact, we do test for NULL on the very first line of this function. However, we do not check for NULL in other places where we're dereferencing this pointer (see line 58, 67, 72, etc...).
I've seen it happen, which causes the application to coredump.
`SCOPE_DELIMITER` not defined in musl
src/nvme/fabrics.c:328: char *p = strchr(tmp, SCOPE_DELIMITER);
glibc just defines it as so -
#ifdef __USE_GNU
/* Scope delimiter for getaddrinfo(), getnameinfo(). */
# define SCOPE_DELIMITER '%'
#endifBut there is no such definition in musl libc.
Alpine Linux uses this patch -
https://git.alpinelinux.org/aports/tree/community/libnvme/scope-delimiter.patch
--- a/src/nvme/fabrics.c
+++ b/src/nvme/fabrics.c
@@ -325,7 +325,7 @@
nvme_msg(r, LOG_ERR, "cannot copy: %s\n", src);
const char *scope = NULL;
- char *p = strchr(tmp, SCOPE_DELIMITER);
+ char *p = strchr(tmp, '%');
if (p) {
*p = '\0';
scope = src + (p - tmp) + 1;Python tests fail in debug build
When the library is configured as debug build the Python test fail due to missing str_strstr() function from the ccan library.
ZNS user data vs I/O command set specific namespace data structure
The code base has this:
NVME_IDENTIFY_CNS_NS_USER_DATA_FORMAT = 0x09,
NVME_IDENTIFY_CNS_CSI_NS_USER_DATA_FORMAT = 0x0A,
but the ZNS spec says:
4.1.5.3 Identify I/O Command Set specific Namespace data structure (CNS 0Ah, CSI 02h)
So how should we go about this? Avoid using 'userdata' as it's being obsoleted by the spec?
Keep the existing wording for compability? Add a duplicate / cloned value with the new name?
API cleanup: nvme_$NAME_get() vs nvme_get_$NAME()
The current API isn't really consistent in the naming pattern. We should use one an stick to it.
Build documentation error
Hi,
I used "meson .build -Dman=true" then "cd .build, ninja", but got following error messages:
[1/1] Generating man with a custom command.
FAILED: doc/libnvme.1
/usr/bin/sphinx-build -b man /home/zxm/libnvme/doc /home/zxm/libnvme/.build/doc
Running Sphinx v1.8.5
building [mo]: targets for 0 po files that are out of date
building [man]: all manpages
updating environment: 2 added, 0 changed, 0 removed
reading sources... [ 50%] index
reading sources... [100%] libnvme
/home/zxm/libnvme/doc/libnvme.rst:3906: WARNING: Unexpected indentation.
/home/zxm/libnvme/doc/libnvme.rst:6585: WARNING: Unexpected indentation.
/home/zxm/libnvme/doc/libnvme.rst:6583: WARNING: Inline interpreted text or phrase reference start-string without end-string.
/home/zxm/libnvme/doc/libnvme.rst:6586: WARNING: Block quote ends without a blank line; unexpected unindent.
/home/zxm/libnvme/doc/libnvme.rst:10192: WARNING: Inline strong start-string without end-string.
/home/zxm/libnvme/doc/libnvme.rst:10652: WARNING: Unexpected indentation.
/home/zxm/libnvme/doc/libnvme.rst:11132: WARNING: Inline interpreted text or phrase reference start-string without end-string.
/home/zxm/libnvme/doc/libnvme.rst:11153: WARNING: Inline interpreted text or phrase reference start-string without end-string.
/home/zxm/libnvme/doc/libnvme.rst:11153: WARNING: Inline interpreted text or phrase reference start-string without end-string.
/home/zxm/libnvme/doc/libnvme.rst:11180: WARNING: Inline interpreted text or phrase reference start-string without end-string.
Sphinx error:
master file /home/zxm/libnvme/doc/contents.rst not found
ninja: build stopped: subcommand failed.
nvme_read_config should return error code
Parsing a JSON file can fail but the caller has no clue when this happens. Thus this function should return an error code
build warnings on 32 bit architectures
Got some buidl warnings for 32 bit architectures.
https://build.opensuse.org/build/home:wagi:branches:Base:System/openSUSE_Factory/i586/libnvme/_log
[ 45s] In file included from ../src/nvme/fabrics.c:36:
[ 45s] ../src/nvme/fabrics.c: In function ‘nvmf_get_discovery_log’:
[ 45s] ../src/nvme/fabrics.c:781:37: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 5 has type ‘uint64_t’ {aka ‘long long unsigned int’} [-Wformat=]
[ 45s] 781 | nvme_msg(LOG_DEBUG, "%s: discover genctr %lu, retry\n",
[ 45s] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ 45s] 782 | name, genctr);
[ 45s] | ~~~~~~
[ 45s] | |
[ 45s] | uint64_t {aka long long unsigned int}
[ 45s] ../src/nvme/log.h:36:36: note: in definition of macro ‘nvme_msg’
[ 45s] 36 | format, ##__VA_ARGS__); \
[ 45s] | ^~~~~~
[ 45s] ../src/nvme/fabrics.c:781:60: note: format string is defined here
[ 45s] 781 | nvme_msg(LOG_DEBUG, "%s: discover genctr %lu, retry\n",
[ 45s] | ~~^
[ 45s] | |
[ 45s] | long unsigned int
[ 45s] | %llu
[ 45s] In file included from ../src/nvme/fabrics.c:36:
[ 45s] ../src/nvme/fabrics.c:798:36: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 5 has type ‘uint64_t’ {aka ‘long long unsigned int’} [-Wformat=]
[ 45s] 798 | nvme_msg(LOG_INFO, "%s: could only fetch %lu of %lu records\n",
[ 45s] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ 45s] 799 | name, numrec, le64_to_cpu(log->numrec));
[ 45s] | ~~~~~~
[ 45s] | |
[ 45s] | uint64_t {aka long long unsigned int}
[ 45s] ../src/nvme/log.h:36:36: note: in definition of macro ‘nvme_msg’
[ 45s] 36 | format, ##__VA_ARGS__); \
[ 45s] | ^~~~~~
[ 45s] ../src/nvme/fabrics.c:798:60: note: format string is defined here
[ 45s] 798 | nvme_msg(LOG_INFO, "%s: could only fetch %lu of %lu records\n",
[ 45s] | ~~^
[ 45s] | |
[ 45s] | long unsigned int
[ 45s] | %llu
[ 45s] In file included from ../src/nvme/fabrics.c:36:
[ 45s] ../src/nvme/fabrics.c:798:36: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 6 has type ‘uint64_t’ {aka ‘long long unsigned int’} [-Wformat=]
[ 45s] 798 | nvme_msg(LOG_INFO, "%s: could only fetch %lu of %lu records\n",
[ 45s] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ 45s] 799 | name, numrec, le64_to_cpu(log->numrec));
[ 45s] | ~~~~~~~~~~~~~~~~~~~~~~~~
[ 45s] | |
[ 45s] | uint64_t {aka long long unsigned int}
[ 45s] ../src/nvme/log.h:36:36: note: in definition of macro ‘nvme_msg’
[ 45s] 36 | format, ##__VA_ARGS__); \
[ 45s] | ^~~~~~
[ 45s] ../src/nvme/fabrics.c:798:67: note: format string is defined here
[ 45s] 798 | nvme_msg(LOG_INFO, "%s: could only fetch %lu of %lu records\n",
[ 45s] | ~~^
[ 45s] | |
[ 45s] | long unsigned int
[ 45s] | %llu
[ 45s] [10/32] cc -Isrc/libnvme.so.1.0.0.p -Isrc -I../src -I. -I.. -Iccan -I../ccan -Ilibnvme -I../libnvme -I/usr/include/uuid -I/usr/include/json-c -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=gnu99 -fomit-frame-pointer -D_GNU_SOURCE -include libnvme/config.h -fomit-frame-pointer -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -g -fPIC -DOPENSSL_LOAD_CONF -MD -MQ src/libnvme.so.1.0.0.p/nvme_ioctl.c.o -MF src/libnvme.so.1.0.0.p/nvme_ioctl.c.o.d -o src/libnvme.so.1.0.0.p/nvme_ioctl.c.o -c ../src/nvme/ioctl.c
[ 46s] [11/32] cc -Itest/main-test.p -Itest -I../test -I. -I.. -Iccan -I../ccan -Isrc -I../src -Ilibnvme -I../libnvme -I/usr/include/uuid -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=gnu99 -fomit-frame-pointer -D_GNU_SOURCE -include libnvme/config.h -fomit-frame-pointer -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -g -MD -MQ test/main-test.p/test.c.o -MF test/main-test.p/test.c.o.d -o test/main-test.p/test.c.o -c ../test/test.c
[ 46s] ../test/test.c: In function ‘test_namespace’:
[ 46s] ../test/test.c:275:28: warning: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘uint64_t’ {aka ‘long long unsigned int’} [-Wformat=]
[ 46s] 275 | printf("%s: nsze:%lx lba size:%d\n", nvme_ns_get_name(n), le64_to_cpu(ns.nsze),
[ 46s] | ~~^ ~~~~~~~~~~~~~~~~~~~~
[ 46s] | | |
[ 46s] | long unsigned int uint64_t {aka long long unsigned int}
[ 46s] | %llx
[ 46s] ../test/test.c: In function ‘main’:
[ 46s] ../test/test.c:368:80: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 4 has type ‘uint64_t’ {aka ‘long long unsigned int’} [-Wformat=]
[ 46s] 368 | printf(" `- %s lba size:%d lba max:%lu\n",
[ 46s] | ~~^
[ 46s] | |
[ 46s] | long unsigned int
[ 46s] | %llu
[ 46s] ......
[ 46s] 371 | nvme_ns_get_lba_count(n));
[ 46s] | ~~~~~~~~~~~~~~~~~~~~~~~~
[ 46s] | |
[ 46s] | uint64_t {aka long long unsigned int}
[ 46s] ../test/test.c:391:70: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 4 has type ‘uint64_t’ {aka ‘long long unsigned int’} [-Wformat=]
[ 46s] 391 | printf(" `- %s lba size:%d lba max:%lu\n",
[ 46s] | ~~^
[ 46s] | |
[ 46s] | long unsigned int
[ 46s] | %llu
[ 46s] ......
[ 46s] 394 | nvme_ns_get_lba_count(n));
[ 46s] | ~~~~~~~~~~~~~~~~~~~~~~~~
[ 46s] | |
[ 46s] | uint64_t {aka long long unsigned int}
[ 46s] [12/32] cc -Isrc/libnvme.so.1.0.0.p -Isrc -I../src -I. -I.. -Iccan -I../ccan -Ilibnvme -I../libnvme -I/usr/include/uuid -I/usr/include/json-c -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=gnu99 -fomit-frame-pointer -D_GNU_SOURCE -include libnvme/config.h -fomit-frame-pointer -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -g -fPIC -DOPENSSL_LOAD_CONF -MD -MQ src/libnvme.so.1.0.0.p/nvme_linux.c.o -MF src/libnvme.so.1.0.0.p/nvme_linux.c.o.d -o src/libnvme.so.1.0.0.p/nvme_linux.c.o -c ../src/nvme/linux.c
[ 46s] [13/32] cc -Itest/test-register.p -Itest -I../test -I. -I.. -Iccan -I../ccan -Isrc -I../src -Ilibnvme -I../libnvme -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=gnu99 -fomit-frame-pointer -D_GNU_SOURCE -include libnvme/config.h -fomit-frame-pointer -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -g -MD -MQ test/test-register.p/register.c.o -MF test/test-register.p/register.c.o.d -o test/test-register.p/register.c.o -c ../test/register.c
[ 46s] [14/32] cc -Isrc/libnvme.so.1.0.0.p -Isrc -I../src -I. -I.. -Iccan -I../ccan -Ilibnvme -I../libnvme -I/usr/include/uuid -I/usr/include/json-c -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=gnu99 -fomit-frame-pointer -D_GNU_SOURCE -include libnvme/config.h -fomit-frame-pointer -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -g -fPIC -DOPENSSL_LOAD_CONF -MD -MQ src/libnvme.so.1.0.0.p/nvme_tree.c.o -MF src/libnvme.so.1.0.0.p/nvme_tree.c.o.d -o src/libnvme.so.1.0.0.p/nvme_tree.c.o -c ../src/nvme/tree.c
[ 46s] [15/32] cc -Itest/test-zns.p -Itest -I../test -I. -I.. -Iccan -I../ccan -Isrc -I../src -Ilibnvme -I../libnvme -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=gnu99 -fomit-frame-pointer -D_GNU_SOURCE -include libnvme/config.h -fomit-frame-pointer -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -g -MD -MQ test/test-zns.p/zns.c.o -MF test/test-zns.p/zns.c.o.d -o test/test-zns.p/zns.c.o -c ../test/zns.c
[ 46s] [16/32] cc -Iexamples/display-columnar.p -Iexamples -I../examples -I. -I.. -Iccan -I../ccan -Isrc -I../src -Ilibnvme -I../libnvme -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=gnu99 -fomit-frame-pointer -D_GNU_SOURCE -include libnvme/config.h -fomit-frame-pointer -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -g -MD -MQ examples/display-columnar.p/display-columnar.c.o -MF examples/display-columnar.p/display-columnar.c.o.d -o examples/display-columnar.p/display-columnar.c.o -c ../examples/display-columnar.c
[ 46s] ../examples/display-columnar.c: In function ‘main’:
[ 46s] ../examples/display-columnar.c:93:65: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 4 has type ‘uint64_t’ {aka ‘long long unsigned int’} [-Wformat=]
[ 46s] 93 | printf("%-12s %-8d %-16lu %-8d %s\n",
[ 46s] | ~~~~~^
[ 46s] | |
[ 46s] | long unsigned int
[ 46s] | %-16llu
[ 46s] ......
[ 46s] 96 | nvme_ns_get_lba_count(n),
[ 46s] | ~~~~~~~~~~~~~~~~~~~~~~~~
[ 46s] | |
[ 46s] | uint64_t {aka long long unsigned int}
[ 46s] ../examples/display-columnar.c:104:57: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 4 has type ‘uint64_t’ {aka ‘long long unsigned int’} [-Wformat=]
[ 46s] 104 | printf("%-12s %-8d %-16lu %-8d ",
[ 46s] | ~~~~~^
[ 46s] | |
[ 46s] | long unsigned int
[ 46s] | %-16llu
[ 46s] ......
[ 46s] 107 | nvme_ns_get_lba_count(n),
[ 46s] | ~~~~~~~~~~~~~~~~~~~~~~~~
[ 46s] | |
[ 46s] | uint64_t {aka long long unsigned int}
meson -Dman=true doesn't rebuild the man pages
Running 'meson -Dman=true' will not regenerated the man pages; primarily because it's running off a separate subdirectory, and any man pages will be stored there (and not in the master directory 'man/doc'). But more importantly, it will just create single 'libnvme.2' file based on the contents of 'doc/libnvme.rst', and not the individual man pages we have now.
And furthermore, the 'libnvme.rst' file is never regenerated based on the contents of the actual source files.
python.platlibdir and python.install_env are mutually exclusive
nvme_chomp() does not remove trailing spaces
From SUSE bug 1196889:
The SUT was reconfigured with a PURE target. NVMe autoconnect fails with the following error:
"nvme_fc: nvme_fc_parse_traddr: bad traddr string"
I looked at the libnvme-1.0-rc5 source code: src/nvme/fabrics.c. I looked at the arguments for the nvme_create_ctrl() call. I printed the contents of cfg->host_traddr and traddr.
printf("cfg host_traddr = ->%s<-\n", cfg->host_traddr);
printf("traddr = ->%s<-\n", traddr);
Output:
cfg host_traddr = ->nn-0x200008f1eabae2a6:pn-0x100008f1eabae2a6<-
traddr = ->nn-0x524A937186970412:pn-0x524A937186970412
<-
traddr has trailing spaces.
I next examined nvme_chomp(). It's called like this: nvme_chomp(e->traddr, NVMF_TRADDR_SIZE)
**
- nvme_chomp() - Strip trailing white space
- @s: String to strip
- @l: Maximum length of string
*/
static inline void nvme_chomp(char *s, int l)
{
while (l && (s[l] == '\0' || s[l] == ' '))
s[l--] = '\0';
}
When nvme_chomp() is called, s[l] is accessing memory outside of the string buffer. This causes trailing spaces to not be removed. I created a libnvme bootleg by patching nvme_chomp().
static inline void nvme_chomp(char *s, int l)
{
l--;
while (l && (s[l] == '\0' || s[l] == ' '))
s[l--] = '\0';
}
nvme_ctrl_alloc() can't handle duplicate connections
Okay, I admit my use case is a little weird and might not reflect real world scenario: loop transport, two (duplicate) connections, resulting in a single subsystem and two controllers in sysfs. The libnvme scanning logic would however only find one controller.
Lines 1270 to 1271 in ddd725e
Looking at how nvme_lookup_ctrl() is called and how are the keys matched inside I see no way to distinguish seemingly duplicate controller entries, just having different name and sysfs path.
This is a broader issue, posting now in light of ongoing tree scanning logic rewrite in #108 to get some consideration.
# old/nvme list -v
NVM Express Subsystems
Subsystem Subsystem-NQN Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys0 test_subnqn nvme0, nvme1
NVM Express Controllers
Device SN MN FR TxPort Address Subsystem Namespaces
-------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
nvme0 cce1b445357e708d9d62 Linux 5.14.17- loop nvme-subsys0 nvme0n1, nvme0n2, nvme0n3
nvme1 cce1b445357e708d9d62 Linux 5.14.17- loop nvme-subsys0 nvme0n1, nvme0n2, nvme0n3
NVM Express Namespaces
Device NSID Usage Format Controllers
------------ -------- -------------------------- ---------------- ----------------
nvme0n1 1 1.07 GB / 1.07 GB 512 B + 0 B nvme0, nvme1
nvme0n2 2 1.07 GB / 1.07 GB 512 B + 0 B nvme0, nvme1
nvme0n3 3 1.07 GB / 1.07 GB 512 B + 0 B nvme0, nvme1
# git-master/nvme list -v
Subsystem Subsystem-NQN Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys0 test_subnqn nvme1
Device SN MN FR TxPort Address Subsystem Namespaces
-------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
nvme1 cce1b445357e708d9d62 Linux 5.14.17- loop (null) nvme-subsys0 nvme0n3, nvme0n2, nvme0n1, nvme0n3, nvme0n2, nvme0n1
Device Generic NSID Usage Format Controllers
------------ ------------ -------- -------------------------- ---------------- ----------------
nvme0n3 ng0n3 3 1.07 GB / 1.07 GB 512 B + 0 B nvme1
nvme0n2 ng0n2 2 1.07 GB / 1.07 GB 512 B + 0 B nvme1
nvme0n1 ng0n1 1 1.07 GB / 1.07 GB 512 B + 0 B nvme1
meson.build:39:0: ERROR: Dependency "uuid" not found, tried pkgconfig and cmake
The Meson build system
Version: 0.61.1
Source dir: /home/ngd/hossein/libnvme
Build dir: /home/ngd/hossein/libnvme/.build
Build type: native build
Project name: libnvme
Project version: 1.0
C compiler for the host machine: cc (gcc 7.5.0 "cc (Ubuntu 7.5.0-3ubuntu118.04) 7.5.0")18.04) 7.5.0")
C linker for the host machine: cc ld.bfd 2.30
C++ compiler for the host machine: c++ (gcc 7.5.0 "c++ (Ubuntu 7.5.0-3ubuntu1
C++ linker for the host machine: c++ ld.bfd 2.30
Host machine cpu family: x86_64
Host machine cpu: x86_64
Found pkg-config: /usr/local/bin/pkg-config (0.29.2)
Found CMake: /usr/local/bin/cmake (3.23.0)
Run-time dependency uuid found: NO (tried pkgconfig and cmake)
meson.build:39:0: ERROR: Dependency "uuid" not found, tried pkgconfig and cmake
A full log can be found at /home/ngd/hossein/libnvme/.build/meson-logs/meson-log.txt
cat /home/ngd/hossein/libnvme/.build/meson-logs/meson-log.txt
Build started at 2022-02-09T17:43:34.106390
Main binary: /usr/bin/python3
Build Options:
Python system: Linux
The Meson build system
Version: 0.61.1
Source dir: /home/ngd/hossein/libnvme
Build dir: /home/ngd/hossein/libnvme/.build
Build type: native build
Project name: libnvme
Project version: 1.0
Sanity testing C compiler: cc
Is cross compiler: False.
Sanity check compiler command line: cc sanitycheckc.c -o sanitycheckc.exe -D_FILE_OFFSET_BITS=64
Sanity check compile stdout:
Sanity check compile stderr:
Running test binary command: /home/ngd/hossein/libnvme/.build/meson-private/sanitycheckc.exe
C compiler for the host machine: cc (gcc 7.5.0 "cc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0")
C linker for the host machine: cc ld.bfd 2.30
Sanity testing C++ compiler: c++
Is cross compiler: False.
Sanity check compiler command line: c++ sanitycheckcpp.cc -o sanitycheckcpp.exe -D_FILE_OFFSET_BITS=64
Sanity check compile stdout:
Sanity check compile stderr:
Running test binary command: /home/ngd/hossein/libnvme/.build/meson-private/sanitycheckcpp.exe
C++ compiler for the host machine: c++ (gcc 7.5.0 "c++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0")
C++ linker for the host machine: c++ ld.bfd 2.30
Sanity testing C compiler: cc
Is cross compiler: False.
Sanity check compiler command line: cc sanitycheckc.c -o sanitycheckc.exe -D_FILE_OFFSET_BITS=64
Sanity check compile stdout:
Sanity check compile stderr:
Running test binary command: /home/ngd/hossein/libnvme/.build/meson-private/sanitycheckc.exe
C compiler for the build machine: cc (gcc 7.5.0 "cc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0")
C linker for the build machine: cc ld.bfd 2.30
Sanity testing C++ compiler: c++
Is cross compiler: False.
Sanity check compiler command line: c++ sanitycheckcpp.cc -o sanitycheckcpp.exe -D_FILE_OFFSET_BITS=64
Sanity check compile stdout:
Sanity check compile stderr:
Running test binary command: /home/ngd/hossein/libnvme/.build/meson-private/sanitycheckcpp.exe
C++ compiler for the build machine: c++ (gcc 7.5.0 "c++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0")
C++ linker for the build machine: c++ ld.bfd 2.30
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: x86_64
Host machine cpu: x86_64
Target machine cpu family: x86_64
Target machine cpu: x86_64
Pkg-config binary for 1 is not cached.
Pkg-config binary missing from cross or native file, or env var undefined.
Trying a default Pkg-config fallback at pkg-config
Found pkg-config: /usr/local/bin/pkg-config (0.29.2)
Determining dependency 'uuid' with pkg-config executable '/usr/local/bin/pkg-config'
env[PKG_CONFIG_PATH]:
Called /usr/local/bin/pkg-config --modversion uuid -> 1
CMake binary for 1 is not cached
CMake binary missing from cross or native file, or env var undefined.
Trying a default CMake fallback at cmake
Found CMake: /usr/local/bin/cmake (3.23.0)
Extracting basic cmake information
CMake Toolchain: Calling CMake once to generate the compiler state
Calling CMake (['/usr/local/bin/cmake']) in /home/ngd/hossein/libnvme/.build/meson-private/CMake_compiler_info with:
- "--trace-expand"
- "--trace-format=json-v1"
- "--no-warn-unused-cli"
- "--trace-redirect=cmake_trace.txt"
- "-G"
- "Ninja"
- "-DCMAKE_TOOLCHAIN_FILE=/home/ngd/hossein/libnvme/.build/meson-private/CMake_compiler_info/CMakeMesonTempToolchainFile.cmake"
- "."
Try CMake generator: auto
Calling CMake (['/usr/local/bin/cmake']) in /home/ngd/hossein/libnvme/.build/meson-private/cmake_uuid with: - "--trace-expand"
- "--trace-format=json-v1"
- "--no-warn-unused-cli"
- "--trace-redirect=cmake_trace.txt"
- "-DCMAKE_TOOLCHAIN_FILE=/home/ngd/hossein/libnvme/.build/meson-private/cmake_uuid/CMakeMesonToolchainFile.cmake"
- "."
-- Module search paths: ['/', '/home/ngd/cmake-3.23.0-rc1-linux-x86_64', '/opt', '/usr', '/usr/local']
-- CMake root: /home/ngd/cmake-3.23.0-rc1-linux-x86_64/share/cmake-3.23
-- CMake architectures: ['x86_64-linux-gnu']
-- CMake lib search paths: ['lib', 'lib32', 'lib64', 'libx32', 'share', 'lib/x86_64-linux-gnu']
Preliminary CMake check failed. Aborting.
Run-time dependency uuid found: NO (tried pkgconfig and cmake)
meson.build:39:0: ERROR: Dependency "uuid" not found, tried pkgconfig and cmake
Can we expand the scope of nvme_*_args structures?
I'm currently working on expanding the NVMe-MI code to provide more Admin command functionality, and would like to keep the API as consistent as possible with the base admin functionality.
The current code has a few _args structs, like from src/nvme/ioctl.h:
struct nvme_get_features_args {
__u32 *result;
void *data;
int args_size;
int fd;
__u32 timeout;
__u32 nsid;
enum nvme_get_features_sel sel;
__u32 cdw11;
__u32 data_len;
__u8 fid;
__u8 uuidx;
};which is used in:
int nvme_get_features(struct nvme_get_features_args *args);I would like the MI admin functions to re-use this struct, creating something like:
int nvme_mi_admin_get_features(nvme_mi_ctrl_t ctrl, struct nvme_get_features_args *args)However, there are a couple of issues in doing so:
- those
_argsstructs currently exist in theioctl.hheader, which I would consider specific to the ioctl-based interface. MI doesn't use the ioctl interface, therefore I have (so far) avoiding using the ioctl definitions - there are some ioctl-specific members of the
_argsstructs:fdand (to a lesser extent)timeout.
One approach would be to move these out of the ioctl header, and into something more transport-independent, which I could use for the MI interface too. Any preferences on which header would be best? Or should I create a new one?
To address the second item, we could move fd/timeout out of the struct, and into plain arguments of the corresponding functions. However, this would involve a change in the API (is this considered stable now?) - so I'm not sure if that's an issue.
I'm happy to work on this, but just wanted to verify whether this approach is OK before sending patches.
'SPDX-License-Identifier: LGPL-2.1-or-newer' is not a valid tag
The SPDX-License-Identifier LGPL-2.1-or-newer is not a valid tag in
Line 3 in 95df64a
I suspect it should be LGPL-2.1-or-later. Hannes could you please send a PR which fixes this? Thanks
endianity macros: ccan pulled in public nvme header files
Commit fc8073a made use of le16_to_cpu() from <ccan/endian/endian.h>:
Lines 551 to 555 in 16abb44
The ccan headers are obviously not installed as part of libnvme and this is the only use of ccan macros in public headers. The question which way to go is rather philosophical. Do other libc's provide endian.h like glibc?
Define MASK, SHIFT to use NVME_GET
I would like to do some modify based on usage of NVME_GET
I Just wrote only for nvme_id_nsfeat currently, Example below!
Any opinion?
From 519337c64316d8805b3d988f1b475591347f22f2 Mon Sep 17 00:00:00 2001
From: Steven Seungcheol Lee [email protected]
Date: Thu, 16 Dec 2021 17:17:24 +0900
Subject: [PATCH] types: Define enum to use NVME_GET define
Signed-off-by: Steven Seungcheol Lee [email protected]
3 files changed, 32 insertions(+), 27 deletions(-)
--- a/src/nvme/types.h
+++ b/src/nvme/types.h
@@ -1668,32 +1668,37 @@ struct nvme_id_ns {
enum nvme_id_nsfeat {
- NVME_NS_FEAT_THIN = 1 << 0,
- NVME_NS_FEAT_NATOMIC = 1 << 1,
- NVME_NS_FEAT_DULBE = 1 << 2,
- NVME_NS_FEAT_ID_REUSE = 1 << 3,
- NVME_NS_FEAT_IO_OPT = 1 << 4,
+ NVME_NS_FEAT_THINP_SHIFT = 0,
+ NVME_NS_FEAT_THINP_MASK = 0x1,
+ NVME_NS_FEAT_NSABP_SHIFT = 1,
+ NVME_NS_FEAT_NSABP_MASK = 0x1,
+ NVME_NS_FEAT_DAE_SHIFT = 2,
+ NVME_NS_FEAT_DAE_MASK = 0x1,
+ NVME_NS_FEAT_UIDREUSE_SHIFT = 3,
+ NVME_NS_FEAT_UIDREUSE_MASK = 0x1,
+ NVME_NS_FEAT_OPTERF_SHIFT = 4,
+ NVME_NS_FEAT_OPTERF_MASK = 0x1,
};
pre compiled man pages are not installed
The current configuration will only install man pages when they are build from scratch. The pre compiled man pages in the source tree are not considered when installing.
undefined reference to `json_util_get_last_err' on debian buster
On debian 10.5 the latest version of libjson-c3:amd64 0.12.1+ds-2+deb10u1 doesn't have a symbol for function json_util_get_last_err:
ninja:
/usr/bin/ld: src/25a6634@@NVME@sha/nvme_json.c.o: in function json_read_config:
json.c:(.text+0x5f9): undefined reference to json_util_get_last_err
/usr/bin/ld: src/25a6634@@NVME@sha/nvme_json.c.o: in function json_update_config:
json.c:(.text+0xc0c): undefined reference to json_object_to_fd
/usr/bin/ld: json.c:(.text+0xc19): undefined reference to json_util_get_last_err
/usr/bin/ld: json.c:(.text+0xc22): undefined reference to json_util_get_last_err
collect2: error: ld returned 1 exit status
[3/20] Compiling C object src/25a6634@@NVME@sta/nvme_tree.c.o.
ninja: build stopped: subcommand failed.
Could not rebuild
meson .build:
The Meson build system
Version: 0.49.2
Source dir: /home/oschmerse/git/libnvme
Build dir: /home/oschmerse/git/libnvme/.build
Build type: native build
Project name: libnvme
Project version: 1.0.1
Native C compiler: cc (gcc 8.3.0 "cc (Debian 8.3.0-6) 8.3.0")
Native C++ compiler: c++ (gcc 8.3.0 "c++ (Debian 8.3.0-6) 8.3.0")
Build machine cpu family: x86_64
Build machine cpu: x86_64
Found pkg-config: /usr/bin/pkg-config (0.29)
Dependency uuid found: YES 2.33.1
Dependency json-c found: YES 0.12.1
...
Remove systemd dependency
With systemd:
$ ldd /tmp/libnvme/lib64/libnvme.so.1.0.1
linux-vdso.so.1 (0x00007ffcfe8cc000)
libuuid.so.1 => /lib64/libuuid.so.1 (0x00007fccfd0a7000)
libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007fccfcfdf000)
libjson-c.so.5 => /lib64/libjson-c.so.5 (0x00007fccfcfc8000)
libc.so.6 => /lib64/libc.so.6 (0x00007fccfcdbe000)
/lib64/ld-linux-x86-64.so.2 (0x00007fccfd0e8000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fccfcd8b000)
libzstd.so.1 => /lib64/libzstd.so.1 (0x00007fccfcc97000)
liblz4.so.1 => /lib64/liblz4.so.1 (0x00007fccfcc74000)
libcap.so.2 => /lib64/libcap.so.2 (0x00007fccfcc69000)
libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007fccfcb2a000)
libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007fccfcb04000)
without:
$ ldd .build/src/libnvme.so.1.0.1
linux-vdso.so.1 (0x00007fffad9d3000)
libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f163cad7000)
libjson-c.so.5 => /lib64/libjson-c.so.5 (0x00007f163cac0000)
libc.so.6 => /lib64/libc.so.6 (0x00007f163c8b6000)
/lib64/ld-linux-x86-64.so.2 (0x00007f163cb17000)
Missing AENs
Hi @keithbusch
CC: @hreinecke
I use libnvme to connect to a discovery controller. I create a persistent nvme device because I need to monitor AENs. I also register for uevents to know when changes (i.e. AENs) are detected on the persistent device. I do get a uevent the first time I apply changes to the target system. I use this uevent to trigger a call to the nvmf_get_discovery_log() method. But after that, I never get uevents (AENs) for subsequent changes to the target. It's as if libnvme only sends the AER command when the connection is first established, but it does not send another AER after each call to nvmf_get_discovery_log().
My questions are: How can I keep receiving AENs on a persistent device that is managed through libnvme? Is calling nvmf_get_discovery_log() the right way to re-enable AENs or is there another function I should call?
broken scan
hi, i might be missing something with the latest updates in libnvme, but the nvme_scan seems to be broken (from this commit: c1f6a24)
all examples with scans (in the test directory) work before this commit, and then no nvme devices are listed after this commit. See the example below (first with a previous commit that shows 2 NVMe devices in the system, and then with the commit, no updates). May i am missing something obvious...
atr@qemuss20:~/src/nvme-cli/libnvme$ git checkout 10daac024f8d14930f33775eac7765e306923d2e
Previous HEAD position was fb7fdf7 Merge pull request #31 from martin-belanger/fix-compilation-warning
HEAD is now at 10daac0 Merge pull request #29 from martin-belanger/fix-compilation-warning
atr@qemuss20:~/src/nvme-cli/libnvme$ make clean ; make -j
make[1]: Entering directory '/home/atr/src/nvme-cli/libnvme/src'
[...]
make[1]: Leaving directory '/home/atr/src/nvme-cli/libnvme/examples'
atr@qemuss20:~/src/nvme-cli/libnvme$ sudo ./test/cpp
nvme-subsys1 - NQN=nqn.2019-08.org.qemu:nvme-dev
`- nvme1 pcie 0000:00:05.0 live
`- nvme1n1lba size:4096 lba max:262144
nvme-subsys0 - NQN=nqn.2019-08.org.qemu:zns-dev
`- nvme0 pcie 0000:00:04.0 live
`- nvme0n1lba size:4096 lba max:262144
atr@qemuss20:~/src/nvme-cli/libnvme$ git checkout c1f6a24d0e8781ad0fe8e18215957436a504d2de
Previous HEAD position was 10daac0 Merge pull request #29 from martin-belanger/fix-compilation-warning
HEAD is now at c1f6a24 Merge pull request #28 from hreinecke/topology
atr@qemuss20:~/src/nvme-cli/libnvme$ make clean; make -j
make[1]: Entering directory '/home/atr/src/nvme-cli/libnvme/src'
[...]
make[1]: Leaving directory '/home/atr/src/nvme-cli/libnvme/examples'
atr@qemuss20:~/src/nvme-cli/libnvme$ sudo ./test/cpp
atr@qemuss20:~/src/nvme-cli/libnvme$ build warnings for __u64 in printf format string on PPC64
There are few build warnings for test/register.c and examples/discover-loop.c
https://build.opensuse.org/build/Base:System/openSUSE_Factory/ppc64/libnvme/_log
[ 32s] ../test/register.c: In function ‘nvme_print_registers’:
[ 32s] ../test/register.c:71:28: warning: format ‘%llx’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘__u64’ {aka ‘long unsigned int’} [-Wformat=]
[ 32s] 71 | printf("%-10s : %llx\n", "CAP", cap);
[ 32s] | ~~~^ ~~~
[ 32s] | | |
[ 32s] | | __u64 {aka long unsigned int}
[ 32s] | long long unsigned int
[ 32s] | %lx
Command stuck without any return when device not returning command result
Currently I don't have any environment that reproduce this.
When I am able to reproduce this, will do more analysis
- Device stop (no return)
- nvme show-regs ( skip nvme_get_property )
- blocking "nvme_scan" ( it's issuing identify command internally)
it was blocking "nvme_scan" not nvme_get_property.
I just remembered, I blocked nvme_get_property block to not issue command when that problem happens
I tried read with old version of nvme-cli that just read with mapping registers without nvme_scan & nvme_get_property, that works fine.
Originally posted by @sc108-lee in linux-nvme/nvme-cli#1541 (comment)
Ability to have error messages go to different file descriptor or user supplied callback
I think it would be good to have a library user provide a file descriptor for logging or a call back as suggested here: #160 (comment)
Having all these messages to written to stderr isn't ideal for library users.
Installation fails
Hello.
I am trying to install libnvme following the README. I get into this error
$ sudo meson install
ninja: Entering directory `.'
ninja: no work to do.
Installing subprojects/json-c-0.15/libjson-c.so to /usr/lib/x86_64-linux-gnu
Installing subprojects/openssl-1.1.1l/libcrypto.so to /usr/lib/x86_64-linux-gnu
Installing subprojects/openssl-1.1.1l/libssl.so to /usr/lib/x86_64-linux-gnu
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/mesonbuild/mesonmain.py", line 140, in run
return options.run_func(options)
File "/usr/lib/python3/dist-packages/mesonbuild/minstall.py", line 554, in run
installer.do_install(datafilename)
File "/usr/lib/python3/dist-packages/mesonbuild/minstall.py", line 372, in do_install
self.install_targets(d)
File "/usr/lib/python3/dist-packages/mesonbuild/minstall.py", line 468, in install_targets
raise RuntimeError('File {!r} could not be found'.format(t.fname))
RuntimeError: File 'subprojects/openssl-1.1.1l/openssl' could not be found
The contents of subproject/openssl-1.1.1l are as follow :
libnvme/.build/subprojects/openssl-1.1.1l$ ls
generated-config libcrypto.so libcrypto.so.p libssl.so libssl.so.pThe build succeeded and unit tests pass
libnvme/.build$ meson test
ninja: Entering directory `/home/reds/libnvme/.build'
ninja: no work to do.
1/5 main OK 1.32s
2/5 main OK 1.30s
3/5 cpp OK 1.32s
4/5 register OK 1.29s
5/5 zns OK 0.05s
Ok: 5
Expected Fail: 0
Fail: 0
Unexpected Pass: 0
Skipped: 0
Timeout: 0
System information : Running on Ubuntu 21.04 with Linux kernel 5.13.0
Best Regards.
Rick
Add versioning to exported symbols
Add versioning to all exported symbols, see also discussion in #51
Fix spelling in the documentation and extend it
Fix all spelling errors in documentation.
RPM: spec file is missing `BuildRequires:meson`
see https://docs.fedoraproject.org/en-US/packaging-guidelines/Meson/
You MUST add following BuildRequires:
BuildRequires: meson
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.