Improvements to and fixes of wikiheaders.pl and sdlwiki SDL API documentation generation. about sdl HOT 10 OPEN

For what it's worth, that thing was going to be daunting in any other language than Perl, too.

That being said, this was a really good example of "I can fix this with regex" that got wildly out of control.

If there's ever time, I would probably rewrite this as such:

• drop MediaWiki support.
• use a proper markdown parser to build an abstract syntax tree
• split the tree by sections for comparison between wiki/headers
• walk that syntax tree to output other formats, like manpages, clearly and cleanly.
• Probably drop Perl, since it scares everyone but me. :)

from sdl.

The documentation is bridged bidirectionally between the wiki and the headers, so Doxygen probably isn't a solution here.

from sdl.

Yes, we've been very happy that people can contribute via GitHub and automatically update the headers for public release.

from sdl.

I know that wikiheaders.pl is daunting, and I would suggest that you create a separate python script that autogenerates the category pages (and other documentation, potentially) and we can bring that into the process as well.

Thanks!

from sdl.

Actually, it looks like category generation is already partially implemented in #7528. @icculus, can you take a look? It would be great to get this in for the 3.0 prerelease.

from sdl.

In theory, an alternative solution might be to switch back to Doxygen or some other general documentation generation tool. Though I would guess that you have good reasons for switching away from Doxygen and not picking a similar tool in the first place.

from sdl.

Bidirectional bridging is an interesting functionality to have. That editing a wiki page automatically somehowupdates the corresponding header in the source code is not a common feature to have in projects, in my experience. I would imagine that it makes it easier to get contributors to update the source code documentation. I do not know what the trade-offs are like in this case. Is bidirectional bridging and editing wiki->updates source code headers something that you have been very happy about, or have been needed?

from sdl.

Yes, we've been very happy that people can contribute via GitHub and automatically update the headers for public release.

Well, people can always contribute via Github and to the headers for public release without a bidirectional bridge, it is just a bit easier with a bidirectional bridge.

Actually, it looks like category generation is already partially implemented in #7528. @icculus, can you take a look? It would be great to get this in for the 3.0 prerelease.

Is that PR for by-name listing, by-category listing, or both? I cannot really tell from the code.

from sdl.

Edit: Fixed wrong file for SDL 3.

I believe that these are the version histories of syncing for the SDL 2 and SDL 3 pages of by-name listing, respectively:

• SDL 2 by-name listing: https://github.com/libsdl-org/sdlwiki/commits/main/SDL2/CategoryAPI.mediawiki
• SDL 3 by-name listing: https://github.com/libsdl-org/sdlwiki/commits/main/SDL3/CategoryAPI.md

Like the live pages, both of those files are missing SDL_Surface even though the wiki page for SDL_Surface exists for both SDL 2 and SDL 3.

The SDL 2 by-name listing was last updated 3 months ago.

The SDL 3 by-name listing was last updated 16 hours ago.

from sdl.

@slouken I think I understand what is happening now (after I learned a minuscule amount of Perl to prod and prick at things), and basically, the reason why SDL_Surface is not included in the by-name category is that the mediawiki/markdown page for SDL_Surface is never generated automatically.

No matter which system for the category pages are used, the category pages appear to rely on there being accurate information in the mediawiki/markdown pages in the footer. When going from the source to the wiki, the process might be as follows: For functions, this footer and other information is parsed from the header files (!!, C++ is not generally easy to parse!), and then categorized somehow. The information is also drawn from existing wiki pages. This is then written to the wiki pages. However, for types that are not functions, nothing is parsed. It was never implemented. So the information for non-function types, like SDL_Surface, nothing is gotten from the header files, and instead everything is just taken from the existing mediawiki file for SDL_Surface, and then written back again. This fits with the mediawiki file for SDL_Surface not having had any changes for 2 years: https://github.com/libsdl-org/sdlwiki/blob/main/SDL2/SDL_Surface.mediawiki . Since the categories in that file does not include CategoryAPI, the SDL_Surface wiki page is never included in the by-name listing.

This also means that, since non-function types are never parsed from the headers, the information in the headers and in the wiki can get arbitrarily out of sync. I have only investigated the direction from header-source to wiki. I do not know what the state of the script is for the other direction.

It would be solved if more non-function types are parsed (and documentation for the other direction is written?) . But the script is already long enough, and C++ is not at all among the easier languages to parse, even if you use proper parser techniques instead of regex (ab)use. I cannot in good conscience suggest anyone to add more parsing of C++ code using Perl regexes, especially to a Perl script that is already 1700+ lines long.

Some of the wiki pages will stay out of sync with the header code unless manually updated, namely the wiki pages for the non-function types, if my understanding of the scripts is correct.

I would personally have off-loaded the burden of parsing C++ for the sake of retrieving documentation, and the other documentation generation tasks, to Doxygen or Sphinx or some other documentation generator. Or maybe at least search for a Perl library or a library in a different language for parsing C++ comments.

I have not investigated the other direction, namely from wiki pages to header documentation.

from sdl.