pybind / pybind11_mkdoc Goto Github PK
View Code? Open in Web Editor NEWPybind11 tool for making docstrings from C++ comments
License: MIT License
Pybind11 tool for making docstrings from C++ comments
License: MIT License
When not providing a LIBCLANG_PATH
environment variable the libclang
shared object library file is search for in
/usr/lib/llvm-*/lib/
directories containing libclang.so.1
. On systems where this directory structure does not match the clang installation the implemented search fails with an IndexError
since it expects to find at least one such path.
Hi,
When trying to use the pybind11_mkdoc tool on the example in the README,it seems to work fine. However, if I add a standard include:
#include <string>
I get the error:
/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/cstdlib:75:15: fatal error: 'stdlib.h' file not found
Command: python -m pybind11_mkdoc example.hpp -o test.hpp
appending -isystem /usr/include
does not help.
I have this file in my /usr/include
directory and the pybind11_mkdoc seems to be adding it to the include directories.
I have no other issues with clang / llvm. Any ideas?
System details:
Ubuntu 18.04.4 LTS
LLVM 9.0.0
clang version 9.0.0-2~ubuntu18.04.2 (tags/RELEASE_900/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
python-clang 11.0
Thanks, Matt.
Hi, I'm trying to run the pybind11_mkdoc
in a single file called score.h
, but I got this error below:
C:\Users\nyck\Desktop\maialib>python -m pybind11_mkdoc maiacore\include\maiacore\score.h
Processing "'maiacore\include\maiacore\score.h'" ..
Waiting for jobs to finish ..
error: error reading ''maiacore\include\maiacore\score.h''
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\threading.py", line 1016, in _bootstrap_inner
self.run()
File "C:\Users\nyck\AppData\Roaming\Python\Python310\site-packages\pybind11_mkdoc\mkdoc_lib.py", line 246, in run
tu = index.parse(self.filename, self.parameters)
File "C:\Users\nyck\AppData\Roaming\Python\Python310\site-packages\clang\cindex.py", line 2722, in parse
return TranslationUnit.from_source(path, args, unsaved_files, options,
File "C:\Users\nyck\AppData\Roaming\Python\Python310\site-packages\clang\cindex.py", line 2837, in from_source
raise TranslationUnitLoadError("Error parsing translation unit.")
clang.cindex.TranslationUnitLoadError: Error parsing translation unit.
The file score.h
is a valid C++ header that contains a valid Doxygen documentation in it.
Please, could you help me?
My system:
If the output file name contains a non-existing path, trying to open it for writing fails, i.e.
$ python -m pybind11_mkdoc -o <non/existing/path/out_file.hpp> input_headers...
results in a FileNotFoundError
.
This can be fixed by adding a call to os.makedirs(os.path.dirname(output), exist_ok=True)
(requires Python 3.2+).
In a new conda environment, I've installed
conda install -c conda-forge python
python -m pip install .
which outputs
Processing /Users/tdegeus/Downloads/pybind11_mkdoc
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting clang (from pybind11_mkdoc==2.6.2.dev1)
Using cached clang-16.0.1.1-py3-none-any.whl (31 kB)
Building wheels for collected packages: pybind11_mkdoc
Building wheel for pybind11_mkdoc (pyproject.toml) ... done
Created wheel for pybind11_mkdoc: filename=pybind11_mkdoc-2.6.2.dev1-py3-none-any.whl size=9378 sha256=b8feb15bae6058b1dfe3085aef3d28bfb96a33aa4df57c21192383ac86085198
Stored in directory: /private/var/folders/4z/74x3z8gs7pq77drzss82p5mc0000gq/T/pip-ephem-wheel-cache-89cdsw8p/wheels/57/8e/75/8ef6e06a545a06d229053aff7b0d90101be8b484c73921802b
Successfully built pybind11_mkdoc
Installing collected packages: clang, pybind11_mkdoc
Successfully installed clang-16.0.1.1 pybind11_mkdoc-2.6.2.dev1
Yet I'm getting
python -m pybind11_mkdoc -o python/docstrings.h include/GooseEYE/GooseEYE.h
Processing "include/GooseEYE/GooseEYE.h" ..
Waiting for jobs to finish ..
Exception in thread Thread-1:
Traceback (most recent call last):
File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/site-packages/clang/cindex.py", line 4136, in register_function
func = getattr(lib, item[0])
^^^^^^^^^^^^^^^^^^^^^
File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/ctypes/__init__.py", line 392, in __getattr__
func = self.__getitem__(name)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/ctypes/__init__.py", line 397, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: dlsym(0x846d0020, clang_CXXMethod_isCopyAssignmentOperator): symbol not found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/threading.py", line 1052, in _bootstrap_inner
self.run()
File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/site-packages/pybind11_mkdoc/mkdoc_lib.py", line 248, in run
cindex.conf.lib.clang_createIndex(False, True))
^^^^^^^^^^^^^^^
File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/site-packages/clang/cindex.py", line 212, in __get__
value = self.wrapped(instance)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/site-packages/clang/cindex.py", line 4217, in lib
register_functions(lib, not Config.compatibility_check)
File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/site-packages/clang/cindex.py", line 4164, in register_functions
register(f)
File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/site-packages/clang/cindex.py", line 4161, in register
return register_function(lib, item, ignore_errors)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tdegeus/miniforge3/envs/doc/lib/python3.12/site-packages/clang/cindex.py", line 4142, in register_function
raise LibclangError(msg)
clang.cindex.LibclangError: dlsym(0x846d0020, clang_CXXMethod_isCopyAssignmentOperator): symbol not found. Please ensure that your python bindings are compatible with your libclang.so version.
Further information:
$ mamba list
# packages in environment at /Users/tdegeus/miniforge3/envs/doc:
#
# Name Version Build Channel
bzip2 1.0.8 h93a5062_5 conda-forge
ca-certificates 2023.11.17 hf0a4a13_0 conda-forge
clang 16.0.1.1 pypi_0 pypi
libexpat 2.5.0 hb7217d7_1 conda-forge
libffi 3.4.2 h3422bc3_5 conda-forge
libsqlite 3.44.2 h091b4b1_0 conda-forge
libzlib 1.2.13 h53f4e23_5 conda-forge
ncurses 6.4 h463b476_2 conda-forge
openssl 3.2.0 h0d3ecfb_1 conda-forge
pip 23.3.1 pyhd8ed1ab_0 conda-forge
pybind11-mkdoc 2.6.2.dev1 pypi_0 pypi
python 3.12.0 h47c9636_0_cpython conda-forge
readline 8.2 h92ec313_1 conda-forge
setuptools 68.2.2 pyhd8ed1ab_0 conda-forge
tk 8.6.13 h5083fa2_1 conda-forge
tzdata 2023c h71feb2d_0 conda-forge
wheel 0.42.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h57fd34a_0 conda-forge
$ mamba info
mamba version : 1.5.3
active environment : doc
active env location : /Users/tdegeus/miniforge3/envs/doc
shell level : 2
user config file : /Users/tdegeus/.condarc
populated config files : /Users/tdegeus/miniforge3/.condarc
/Users/tdegeus/.condarc
conda version : 23.10.0
conda-build version : not installed
python version : 3.11.6.final.0
virtual packages : __archspec=1=m1
__osx=14.0=0
__unix=0=0
base environment : /Users/tdegeus/miniforge3 (writable)
conda av data dir : /Users/tdegeus/miniforge3/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/osx-arm64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/osx-arm64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /Users/tdegeus/miniforge3/pkgs
/Users/tdegeus/.conda/pkgs
envs directories : /Users/tdegeus/miniforge3/envs
/Users/tdegeus/.conda/envs
platform : osx-arm64
user-agent : conda/23.10.0 requests/2.31.0 CPython/3.11.6 Darwin/23.0.0 OSX/14.0 solver/libmamba conda-libmamba-solver/23.11.1 libmambapy/1.5.3
UID:GID : 503:20
netrc file : None
offline mode : False
The problem is persistent when I install clang
from conda-forge
In addition, the problem is persistent in my CI: tdegeus/GooseEYE#105
I guess that that the issue could be that I'm not installing directly using
python -m pip install git+git://github.com/pybind/pybind11_mkdoc.git@master
However, that command times-out (also on the CI)
I have a library that depends on a number of different libraries. I use CMake to find them, and I would like pybind11_mkdoc
to do the same. How do I do that?
Doxygen defines tags like \return
and \param
that have special meanings in documentation. pybind11_mkdoc understands these tags and "translates" them appropriately by reformatting the comment. According to the Doxygen manual,
All commands in the documentation start with a backslash (\) or an at-sign (@). If you prefer you can replace all commands starting with a backslash below by their counterparts that start with an at-sign.
But pybind11_mkdoc does not understand @return
and @param
, and thus does not translate them at all.
When compiling bindings output with g++ --std=c++17 -Wall -Werror -pedantic-errors
, generated header is not conformant:
project/bindings.cpp:168:55: error: ISO C++11 requires at least one argument for the "..." in a variadic macro
168 | py::enum_<PreStatus>(m, "PreStatus", DOC(PreStatus))
Setting the clang include directory as it is implemented right now leads to AttributeError: 'NoneType' object has no attribute 'encode'
erros from within clang/cindex.py
on systems where the expected directories do not exist.
pybind11_mkdoc/pybind11_mkdoc/mkdoc_lib.py
Lines 309 to 313 in 956b3f9
If there are no directories matching the search pattern a default value of None
is used which then leads to passing -isystem None
as an argument to clang that causes the above error.
My C++ code depends on an external library, so I add the flag -I${nlohmann_SOURCE_DIR}/include
to ensure that #include
can find the headers. Now #include
doesn't really need to work for pybind11_mkdoc to do its job, and indeed, if I do not add the flag -I${EXTERNAL_SOURCE_DIR}/include
, pybind11_mkdoc still runs successfully. However, I get the following error message
fatal error: 'nlohmann/json_fwd.hpp' file not found
at the line where this file is #include
d. Can we eliminate this error message and guarantee to the user that pybind11_mkdoc works even if included files can't be found, or should we require that all #include
s succeed?
Currently a function's or method's arguments are output with the following format:
Parameter ``arg1``:
This is the description.
This is a continued line of the description.
Parameter ``arg2``:
This is the description.
Throws:
Exception1 This is when the exception is thrown.
Throws:
Exception2 This is when the exception is thrown.
Sphinx then renders this as (depending on the theme)
Parameter arg1
:
This is the description.
This is a continued line of the description.
Parameter arg2
:
This is the description.
Throws:
Exception1 This is when the exception is thrown.
Throws:
Exception2 This is when the exception is thrown.
The docstring format that Sphinx understands that this closest to is the Google Style docstring (https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html#google-vs-numpy). This would look like the following:
Parameters:
arg1: This is the description.
This is a continued line of the description.
arg2: This is the description.
Raises:
Exception1: This is when the exception is thrown.
Exception2: This is when the exception is thrown.
Which would render as (depending on the theme)
Parameters:
arg1
-- This is the description. This is a continued line of the description.arg2
-- This is the description.Raises:
Exception1
-- This is when the exception is thrown.Exception2
-- This is when the exception is thrown.Please could pybind11_mkdoc output docstrings in a format that Sphinx understands so that they render like any other Python docstring would.
The quickest way to implement this would to aim to have pybind11_mkdoc output Google style docstrings. Given how close the current output format is to the Google docstring style, this boils down to supporting grouped sections (eg Parameters and Raises).
However we could chose to output as rst and still get the desired output with Sphinx. Obviously this would be much more work. For the most flexibility, the user could choose the output format that they desire. Personally I don't need this extra flexibility because I use Google style anyway.
Documentation not explicit on dependency on "libcmake-dev", not having it results in "'NoneType' object has no attribute 'encode'".
Tested on master, on this file: https://github.com/EthicalML/vulkan-kompute/blob/master/single_include/kompute/Kompute.hpp
I also tried on a simpler file (https://github.com/EthicalML/vulkan-kompute/blob/master/src/include/kompute/Manager.hpp) but still get the same error:
Processing "Manager.hpp" ..
Waiting for jobs to finish ..
Exception in thread Thread-1:
Traceback (most recent call last):
File "/home/alejandro/miniconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/home/alejandro/miniconda3/lib/python3.7/site-packages/pybind11_mkdoc/mkdoc_lib.py", line 232, in run
tu = index.parse(self.filename, self.parameters)
File "/home/alejandro/miniconda3/lib/python3.7/site-packages/clang/cindex.py", line 2722, in parse
self)
File "/home/alejandro/miniconda3/lib/python3.7/site-packages/clang/cindex.py", line 2816, in from_source
args_array = (c_char_p * len(args))(*[b(x) for x in args])
File "/home/alejandro/miniconda3/lib/python3.7/site-packages/clang/cindex.py", line 2816, in <listcomp>
args_array = (c_char_p * len(args))(*[b(x) for x in args])
File "/home/alejandro/miniconda3/lib/python3.7/site-packages/clang/cindex.py", line 109, in b
return x.encode('utf8')
AttributeError: 'NoneType' object has no attribute 'encode'
Is this due to my environment? I'm testing in Ubuntu 18, Python3. Happy to provide further details.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.