From #27: it would be cool to be able to have multiple lines in prefaces so that one doesn't have to stack them.

Discussed in #29: let's do it. I somewhat overlooked their utility in a bunch of small code blocks. The indirection is minimal, and any name clashes are resolved anyway. Perhaps something like:

# conf.py
codeautolink_default_imports = ["import lib", "import lib.sub as s"]

As with implicit-imports, we'll not allow newlines in entries.

From #44: recently (in #49) we made it possible to provide parsing for custom code blocks. Apparently some third-party directives have some inconsistencies, so we'll introduce a simple configuration value for extending the list of CSS classed that are used to find code examples for matching.

🥳 Let's also add PyPI classifiers for 3.6-10

Yeah so if the same exact name or attribute is on the same line, our sanity safety checks take hold and links are not inserted. We should try to avoid this, because it's entirely reasonable to use the same definition on the same line multiple times. I like the safety checks though, so maybe we should just improve the logic.

In terms of composite type hints, this is pretty basic. So we could just assume that any optional type hints simply manifest the underlying type. It is the job of the code author to check for None, and any attribute access certainly isn't for the NoneType.

From #44: Currently the extension crashes as we try to listen to a non-existent event if autodoc isn't enabled. The use case is solid though, because intersphinx can be used to make the links. So we could try to hack something together, but that'd most likely be affected by the order of initialising extensions. Probably better to just initialise autodoc in our setup. It shouldn't affect documentation that doesn't use autodoc features. It may hide problems in some configurations that don't include autodoc but intend to use it, but in my opinion it's a worthwhile trade-off.

When partially building documentation, links are only generated to the files that were edited. I'm guessing Sphinx uses clever shortcuts to reduce processing, which leads to the doctrees being read but not analysed, which hurts us. So either we:

• figure out a way to stop Sphinx from taking the shortcuts, which could be annoying for huge builds and overall an unnecessary hurdle
• cache source transformations as well

Looks like on generate module documentation is failing and prints some warnings

[tkloczko@ss-desktop sphinx-codeautolink-0.4.0]$ /usr/bin/python3 setup.py build_sphinx -b man --build-dir build/sphinx
running build_sphinx
Running Sphinx v4.2.0
making output directory... done
loading intersphinx inventory from https://numpy.org/doc/stable/objects.inv...
loading intersphinx inventory from https://matplotlib.org/stable/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [man]: all manpages
updating environment: [new config] 7 added, 0 changed, 0 removed
reading sources... [100%] release_notes
looking for now-outdated files... none found
pickling environment... done
checking consistency... /home/tkloczko/rpmbuild/BUILD/sphinx-codeautolink-0.4.0/docs/src/404.rst: WARNING: document isn't included in any toctree
done
writing... sphinx-codeautolink.1 { release_notes reference about examples example_library } failed

Exception occurred:
  File "/usr/lib/python3.8/site-packages/sphinx/writers/manpage.py", line 467, in unknown_visit
    raise NotImplementedError('Unknown node: ' + node.__class__.__name__)
NotImplementedError: Unknown node: PrefaceMarker
The full traceback has been saved in /tmp/sphinx-err-0zz69rdx.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!

Hello! Thanks for making this extension. It is really cool.

Issue

The following rst snippet

.. code-block:: python

   from dataclasses import dataclass
   
   @dataclass
   class A:
       x: int

Causes the following error when running sphinx:

Extension error (sphinx_codeautolink.extension):
Handler <bound method SphinxCodeAutoLink.parse_blocks of <sphinx_codeautolink.extension.SphinxCodeAutoLink object at 0x000001D6A2FC99A0>> for event 'doctree-read' threw an exception (exception: 'NoneType' object has no attribute '_fields')

Expected behavior

There shouldn't be an error 😄

Steps to reproduce

I believe the rst snippet that I included suffices. Please let me know if I can provide anything else!

Discussed in #29: we currently reject all multiline name chains, and we ought to change that. Let's:

• narrow down the lineno calculations to not start at the very first line of a multiline statement, but instead recognise the previous element that didn't belong to the chain and go on from there
• include whitespace cases to our regex matching

Now it's only consumed in python blocks, which is contradicted by the documentation. Let's fix it.

Issue

This seems to happen on deepcopying a part of the tree when making backref tables. Not always though, but it seems to bug older Python versions consistently.

Expected behavior

No error 😄

I'm not sure if this is a version support issue, or something we can simply fix. We'll have to do some digging.

If a resolved type is a class, currently it is classified as callable and its annotation is fetched. The annotation is only available in classes that have annotations, and more importantly it is a logic error, since the class case is already handled. So let's fix the resolving to exclude classes.

It's a bit awkward to use concat-blocks to concatenate short sections, because it has to be turned off at the end if no other concatenation is wanted during the file. So I see a few options:

• Have concat-blocks follow the section tree to only apply the rule to sections under the particular node
• Introduce a temporary configuration like concat-blocks:: here that only concatenates until the next title. This could be equally frustrating if we don't follow what the global current option to go back to is.

Currently the heading stack for backrefs just gathers all the headings in a file, because depart_title isn't being called. So let's try to fix it or just get the first and last ones like now.

Sometimes the table is not rendered since directives need an empty newline before them to be registered as directives (right?). So let's fix that by adding an extra line before.

Discussed in #29: we should try to support console blocks (i.e. >>> prefixed code).

Parsing tests are easy, but we should also look into testing the extension functionality in some manner other than visual testing with our documentation build.

Hi! Thanks for the library, it looks very promising and I can't wait to try it. I would like to use the extension with ArviZ docs. So far I haven't managed to get it to work so I'll try to go over my experience here so that you can hopefully guide me into opening proper issues for bug reports (if any) and enhancements as well as some PRs to improve documentation of the extension?

The first thing is that it took me a while to see if the extension would work at all in our case, as we don't really have any .. code:: blocks, we use matplotlib's, ipython's or bokeh's directives to get the code to autoexecute and include the output, as well as jupyter notebooks parsed with myst_parser. All three directives seem to also use highlight-python notranslate so it looks like they should be compatible out of the box and I did try using the matplotlib example on your homepage to use the plot directive and worked. But it might be worth it to explain what are the actual requirements for this to work or include some examples in the docs, perhaps with ipython and matplotlib ones that are quite popular to show they work too.

myst seems to use highlight-ipython3 notranslate which I'm not sure how easy it would be to cover too. This page for example is generated from a jupyter notebook: https://arviz-devs.github.io/arviz/getting_started/Introduction.html.

I naively added the highlight-ipython3 notranslate to https://github.com/felix-hilden/sphinx-codeautolink/blob/master/src/sphinx_codeautolink/extension/block.py#L193 to see what happened and added a jupyter notebook file to the example docs in my fork. The code in the notebook was not hyperlinked, but at least it did not break the rendering nor links for the rest of the pages, which for now is good enough.

I eventually moved to try it on in ArviZ docs, installed the extension, added it to conf.py and set codeautolink_autodoc_inject = False as we don't want the auto tables there, they would quickly become quite large and we'd rather users focus on the examples in the docstring for quick reference or on the dedicated plotting, labeling... guides for specific tasks which are also useful even if the exact function is not being used as many arguments are common between most plotting functions for example. But I still haven't managed to get the docs to build, I get the following error:

Extension error (sphinx_codeautolink.extension):
Handler <bound method SphinxCodeAutoLink.generate_backref_tables of <sphinx_codeautolink.extension.SphinxCodeAutoLink object at 0x7f8d6a43b2e0>> for event 'doctree-resolved' threw an exception (exception: 'builtin_function_or_method' object has no attribute '__annotations__')

I am trying to build only part of the docs to see if I can identify what triggers that error, but so far all the subsets I have tried end up just like this. I was also wondering if this step is always necessary or if it's only necessary when planning to generate example tables either automatically or with the directive. In our case, even a hacky workaround that completely disables the feature and only adds links to code would be amazing, not sure if it's possible.

I also removed autodoc in one of the experiments when trying to remove the api section completely from the equation which triggered a whole different error. I didn't completely understood the error nor why was it triggered by autodoc being missing, but I think the extension would be also extremely useful in documentation that doesn't use autodoc at all. I am thinking for example about collections of tutorials and case studies like https://pymc-examples.readthedocs.io/en/latest/ that thanks to the integration with intersphinx could easily link from case studies to the api docs in the main documentation website.

It could be neat to be able to specify exactly the text injected to autodoc entries. So our config value could be either a bool or a custom string.

From #44: let's add the global configuration for concatenation back. Not in its former glory though, let's only have it as a boolean switch.

Since .. code-refs:: is already recommended to be included in reference documentation, we ought to hook into autodoc (if possible) to inject the directive to docstrings. If not possible, we should at least document a way to do it.

Concatenation fails when two blocks contain linkable names. This is because the references are not removed for the previous (concatenated) part, and left to be linked in the original block. So we should filter them out.

Currently the code examples are parsed quite naïvely. We should make the parsing respect scopes on all levels. The current way probably covers the 90 % case, but we ought to get it right.

.. code:: python

    from functools import lru_cache

    @lru_cache
    def f(x): pass

    @lru_cache(maxsize=None)
    def f(x): pass

would ideally have lru_cache autolinked when used as a decorator, but instead we get:

sphinx_codeautolink/extension/block.py:224: 
	RuntimeWarning: Could not match transformation of `lru_cache` 
	on source line 3 <or line 6> in document "example.html", source:

from functools import lru_cache

@lru_cache
def f(x): pass

@lru_cache(maxsize=None)
def f(x): pass

I'm running Python 3.8, sphinx-codeautolink 0.2.1

Not completely sure why, but intersphinx links only seem to work if present in a file a the doc home folder. If in index or examples page they do work, but if in examples/function or api/module/class they point to doc_base_url/correct_intersphinx_link.

One example of this behaviour for example, the numpy module link points tot https://arviz--1923.org.readthedocs.build/en/1923/https://numpy.org/doc/stable/reference/index.html#module-numpy

Currently the object inventory file is assumed to be in objects.inv under the build directory. However, intersphinx documentation says it can be some other file or even format as well. So we should look into the locations and how they are configured.

I didn't give them much thought, but now would be the good time to change them since changes won't disrupt too many users.

• codeautolink_autodoc_inject: fine in my opinion
• code-refs: meh, could be example-links or code-examples
• concat-blocks: fine, but it's a bit ambiguous to a reader that doesn't know it's from this extension. Could be confused to actually concatenating code blocks.
• implicit-import: descriptive once you know it, but again I feel like this could be confusing at first
• autolink-skip: absolutely fine

When parsing, if a node doesn't have a language, the syntax error handling fails. We pass the language as a parameter, so we should use that.

We already use intersphinx internally to get the object inventory file. It shouldn't be too much of a leap to support links to whatever libraries are defined for intersphinx!

Discussed in #29: currently e.g. regex.compile is not being linked (matched) because Pygments assigns a different class to it. Let's take care of that.

I'm no web designer, but I think the default link class looks horrible. Something akin to Sphinx-Gallery's blue on hover would be nice. It could be good to have some separation from the ordinary code even without hover, but we'll have to test it.

• Where should the CSS be put?
• More importantly, what should the CSS be?

Discussed in #29: while discouraged as a style, documentation could really benefit from being able to use star imports, so that the library name doesn't have to be prefixed in every example you write.

We'll assume that the module is importable when parsing, and pull the module dict from there.

I'm all for sane defaults and working out of the box, but I've started to feel like the autodoc integration should be off by default. Projects with lots of examples might not want to list everything all the time.

Currently a bunch of assertions and runtime warnings are scattered across the extension. We should probably:

• Have better error messages that contain useful information for users
• Have an error / warning class of our own

The backreference tables can become quite noisy. It'd be cool to be able to collapse the tables to bring the table into view when required. The autodoc integration arguably could also not be that good to be on by default. But I think that if we make the table to be collapsed by default and continue deleting empty tables, leaving it on is fine.

Classes that return their instance from a method are often annotated with a string of its name. We could recognise the simplest case and resolve the type.

How bout a directive for gathering all possible .. code-refs:: together? Output being something like:

Index of code references:
- sphinx-codeautolink.setup:
  - Examples / Basic use
  - Examples / Implicit imports
- sphinx-codeautolink.parse.Name:
  - Examples / Implicit imports
...

I'm not sure if this would be useful or if we should do it though. But it shouldn't be too difficult.

For anybody that reads this: please thumbs up or comment your thoughts on this being a thing!

Discussed in #29: like pycon, this would be a good addition. We need to take care of the fact that <BLANKLINE> represents a blank line and is replaced by Sphinx in pycon blocks too.

Sometimes the errors masked by the Sphinx build manager are actually user errors. So maybe we should catch them and print the stack trace anyway. I'm not sure when Sphinx masks them though, because sometimes it is printed.

I just discovered this extension from the python-announce mailing list, it looks awesome!

I usually browse https://github.com/topics/sphinx-extension to discover new extensions, would be great to have this extension listed there :)

The effect should be pretty much the same, but there are a few advantages:

• We can implement the extension more tightly in the doctree-read time, meaning that any pending references we insert get resolved. We can control the order of transformers to make sure that code blocks are analysed before building reference tables etc.
• As a result, there's no need for the .. code-refs:: type argument anymore, since we can use the object inventory to determine intersphinx dependencies, and the builtin system uses a class type for all internal refs anyway. I'm guessing they get resolved to the correct object naturally, but intersphinx has a different logic.

Currently we only handle names and attributes that are directly accessed via imports. It would be cool to be able to also link more indirect usage, maybe through inspecting type hints or following assignments around.

For example:

import lib

a = lib.get_thing()  # -> lib.Thing
a.do_stuff()  # linked to lib.Thing.do_stuff

From #44: we should look to extend our parsing. We already magically support a matplotlib directive, because it is parsed to ordinary sphinx nodes, but many others exist as well. So we need to make a call: how do we decide which 3rd party extensions to support?

Python and doctest blocks are supported by Sphinx and the surrounding tooling itself, but I don't really want to go deep into other types, especially ones that have specialised syntax. I wonder if instead of using our own code to sanitise the input, we could use Pygments' lexers to parse instead. I have no idea how they work though.

Here's my absolute minimum: we don't depend on any other libraries in an attempt to support parsing for them. And if parsing special syntax requires more than "remove this prefix, ignore everything starting with this", I'm hesitant to add it, because that also needs tests.

This all being said, I'd like to support at least the most popular options like Jupyter, iPython and MyST notebooks. I'll have a look at the lexers and what kind of output do the various directives produce.

We would benefit from being able to write tests that check that link targets are valid, for example for #56. Sphinx linkcheck exists, but I have no experience with it. Is it meant for this? Something else?

From #5: we have the machinery in place to track subscripts and unpacking assignments by analysing type hints that are nested tuples or lists of simple types:

class A:
    attr: int = 3

def foo() -> List[A]:
    pass

def bar() -> Tuple[A, A]:
    pass

# example code
a = lib.foo()[0]
a.attr  # properly linked
b, c = lib.bar()  # b and c linked to A

I'm not sure what to do with other than constant integer subscripts though.

When creating a link to a file that is in a folder relative to the build dir root, the link references the folder instead of the build directory. Let's fix that.

Discussed in #29: we could also link import statements. It can only make things better. Let's do:

import mod.sub  # mod.sub
import mod.sub as s  # mod.sub, not "as s"
from mod.sub import foo as f  # foo, not "as f"
from mod.sub import *  # nothing linked

Honestly I don't know why I safeguarded against not having a module or qualname attribute by defaulting to an empty string. This is in hindsight obviously not the way to go. In the worst case, which currently happens when the annotation is a string, a bare dot is returned breaking the whole system.