python / docsbuild-scripts Goto Github PK

Currently we're redirecting from / to /3/ but symlinking from /3/ to /3.7/. I feel like we should redirect instead of symlinking (our builds are not aware of the others, so no canonical links in place).

Which is legit, 3.6 is EOL, but on docs.nyc1.psf.io, to build docs.python.org, we still have a 3.6.9.

ping @ewdurbin.

As the search is done client side, and results will always be better directly, it's not usefull to index it.

I'm not a fan of the actual uses of shell_out like shell_out("chown -R :{} Doc/build/html/".format(group)).

It's a typical "there's a vulnerability here", but in the other hand as all input are from us there is no big security issue.

Yet I'd like to take the time to "fix" it so I'm opening this issue as a reminder. If anyone want to take a look at it, it will be appreciated too: I'm not "reserving" this.

I like "git pull --ff-only" as a human to know when there's a problem. But in scripts like this, it's probably better to just force pull, maybe using:

f"git fetch; git reset --hard origin/{branch}"

Currently, when a release is done, like today with 3.9, we have to maintain /3/ symlinks manually:

# logged as docsbuild, example for /es/ translation:
TRANS=es
cd /src/docs.python.org/$TRANS/
ln -nfs 3.9 3
chown -h docsbuild:docs 3
curl -XPURGE https://docs.python.org/$TRANS/3/
for file in 3/*/* 3/*; do echo "$file $(curl -s -XPURGE https://docs.python.org/pt-br/$file)"; done

It would be nice if the script could do it.

The lines:

for fn ∈ os.listdir(targets_dir):
    if os.stat(os.path.join(targets_dir, fn)).st_ino == target_ino:
        prefixes.append(fn)

from build_docs.py are searching all the http-exposed directories pointing to the currently build directory. It's building a wrong list of directories, not being able to spot he's in a subdirectory:

• target is /srv/docs.python.org/fr/3.5
• so targets_dir is /srv/docs.python.org/fr
• the os.listdir will get the right list of translated versions
• prefixes will only contain the versions like ['3.6'] not ['fr/3.6']
• so to_purge will contain something like ['3.5', '3.5/about.html'] missing the fr/ prefix.

Looks like #10 has happened again, this time for 3.11.

https://github.com/python/cpython/pull/28117/files deleted Doc/library/binhex.rst but it's still online at https://docs.python.org/3.11/library/binhex.html (with an old "Last updated on Sep 02, 2021"). The front page is up to date ("Last updated on Feb 10, 2022")

(Re: https://discuss.python.org/t/pep-594-take-2-removing-dead-batteries-from-the-standard-library/13508/13?u=hugovk)

Like #28, perhaps the dev docs builds should also use rsync instead of cp? It's fairly common for modules to be deleted during the alpha.

As the title say, I forgot:

• To chmod 644
• To set the group to docs
• To purge them when they change

platex is not required if one wants to e.g. only build the english documentation. It would therefore make sense to catch such errors and list the program as not installed.

Traceback (most recent call last):
  File "./build_docs.py", line 802, in <module>
    main()
  File "./build_docs.py", line 741, in main
    version_info()
  File "./build_docs.py", line 626, in version_info
    subprocess.check_output(["platex", "--version"], universal_newlines=True), n=3
  File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'platex'

On the release of 3.7.2rc1, the archives/ were not built automatically (they are built daily).

I bet (didn't checked it yet) that it should fix itself in less than 24h, while the cron passes, but it's too long and it bothers people.

I had to build locally then from the local www/ directory run:

$ rsync -vrhi --include '/*/' --include '/*/*.*/' --include '/*/*.*/archives/' --include '/*/*.*/archives/**' --exclude '*' ./ docs.iad1.psf.io:/srv/docs.python.org/

Then remotely run:

$ sudo chown docsbuild:docs /srv/docs.python.org/*/*/archives/*
$ cd /srv/docs.python.org/; for file in */*/archives/*; do  curl -XPURGE https://docs.python.org/$file; done

to temporary trash-fix this.

Users typically don't want to have /dev/ pages in their search result and indexing /dev/ leads to a messy situation where new pages exists and have a canonical to a non-existing one. We still allow search engines to follow the links from /dev/ in case they found something interesting (outbound links to pypi or whatever).

I ran a quick test on docs.p.o, now that 3.10 doc can be compiled in parallel, for a single build (one language, one version) :

HTML only it takes 2mn in parallel vs 5mn single threaded.

Full (with pdf, epub, texinfo ,,,) it takes 26mn in parllel and 38mn single threaded, it boils down to:

• html 2%
• text 2%
• A4 latex 27%
• letter latex 29%
• epub 13%
• texinfo 25%

I think it would be interesting to dive into this to see if it can be enhanced.

Currently we're using sphinx-build -W to spot errors, but on unmaintained translations like 3.8, as we're using the po files from 3.7, it's normal to have inconsistencies.

Maybe we should detect we're using files from another version and just drop the -W in this case.

Based on discussions in: python/devguide#114

Seems like docsbuild script doesn't remove html for docs that have been deleted.

Thanks @ned-deily and @zware for pointing me this way.

If https://docs.python.org/3/_static/documentation_options.js is nor purged, the page displays the proper version but the JS tells another one, so the version switcher picks the wrong version, leading to some confusion.

The Python 3.8 and 3.9 documentation is now built with Sphinx 2.4.4 to fix warnings:
https://bugs.python.org/issue35293

The Python 3.10 documentation is built with Sphinx 3.2.1.

Can we update Sphinx to 2.4.4, to stay in sync?

According to those log lines:

DEBUG:2018-01-22 02:40:51,100:Running command 'rsync -a  Doc/build/html/ /srv/docs.python.org/fr/2.7'
INFO:2018-01-22 02:40:51,360:513 files changed
DEBUG:2018-01-22 02:40:51,361:Running command 'find -L /srv/docs.python.org/fr -samefile /srv/docs.python.org/fr/2.7'
INFO:2018-01-22 02:40:51,386:Running CDN purge
DEBUG:2018-01-22 02:40:51,386:Running command 'curl -X PURGE "https://docs.python.org/{2.7/,2/,2.7/genindex-Y.html,2[...]

The changed files are listed properly but the curl -XPURGE is done without the language tag in the URL, it misses the /fr/ in this case.

Multiple time in the history of this cron, a sphinx-build is launched while the previous one is not complete.

Also we're only building a few versions, so it does not take too long, but we're adding languages, so it take more and more time.

I propose to convert this script to a daemon, while keeping the possibility to start it one shot as a script manually.

The feature I would like to see in the daemon:

• Availability to build old versions, less often than new versions, so they get the new version links in their menu and deprecation header more easily.
• Listen for notifications from github to start a build right after a push, either on translations or on cpython.
• Avoid building where it's not needed.
• Do not start a build when a build is already running.
• Implement a /health/ endpoint showing the overall health of the build system, it may even be integrated to display a green/red line on status.python.org maybe.

I don't think all those feature are impossible as a cron, typically I could implement "avoid building when it's not needed", just by skipping the call to sphinx-build if the previous git pull has pulled nothing. But overall, I think having it as a daemon could help implementing more things.

I'm opening the issue to gather feedback before starting to work on it.

Looks like /objects.inv (intersphinx thing) has not beed invalidated during the 3.10 release.

So we can do:

/srv/docsbuild/venv/bin/python /srv/docsbuild/scripts/build_docs.py --quick --language en --branch 3.6 3.7 3.8

Instead of:

for branch in 3.6 3.7 3.8; do /srv/docsbuild/venv/bin/python /srv/docsbuild/scripts/build_docs.py --quick --language en 

--branch $branch; done

Actually all CJK{Chinese, Japanese, Korean} PDF are not generated correctly.

The Chinese PDF should be fixed with #70

But I'm not sure what to do with Japanese and Korean. xeCJK is supposed to work for all CJK. But it's best to have some native developer/translator to check it.

Typically this page https://docs.python.org/3/library/functions.html has no tag, it should.

Today, 3.5 is typically never built. But translations are indirectly contributing to it so we may still build it, from time to time.

An idea is to monitor the repositories and build only when necessary.

To try to fix https://bugs.python.org/issue31584 / python/pythondotorg#1207

Ref: python-doc-ja/python-doc-ja#771

When Japanese quotes (e.g. 『, 「) are suitable, we use them in translation.
So we don't want to translate English quotes (", ') to Japanese quotes automatically.

Would you disable smart-quotes option in docutils.conf?

Old documentation pages may dissapear, leading to 404s, like
https://docs.python.org/3/howto/webservers.html.

We should redirect users to the latest version where the page existed, typically for this example, to https://docs.python.org/3.4/howto/webservers.html.

Looks like our logs gets written to nowhere after logrotate passes:

docsbuild@docs:/srv/docsbuild/3.8/locale/ja/LC_MESSAGES$ zgrep 'Build' /var/log/docsbuild/docsbuild.log.1.gz | tail -n 1
INFO:2020-05-23 05:55:46,718:Build start for version: 3.7, language: pt-br

docsbuild@docs:/srv/docsbuild/3.8/locale/ja/LC_MESSAGES$ stat /var/log/docsbuild/docsbuild.log.1.gz
  File: /var/log/docsbuild/docsbuild.log.1.gz
  Size: 31614     	Blocks: 64         IO Block: 4096   regular file
Device: fc01h/64513d	Inode: 2070345     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1011/docsbuild)   Gid: ( 1000/    docs)
Access: 2020-05-23 09:52:46.339287904 +0000
Modify: 2020-05-23 05:55:51.000000000 +0000
Change: 2020-05-23 09:47:27.529719526 +0000
 Birth: -
docsbuild@docs:/srv/docsbuild/3.8/locale/ja/LC_MESSAGES$ stat /var/log/docsbuild/docsbuild.log
  File: /var/log/docsbuild/docsbuild.log
  Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: fc01h/64513d	Inode: 2069477     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1011/docsbuild)   Gid: ( 1000/    docs)
Access: 2020-05-23 09:52:41.815095227 +0000
Modify: 2020-05-23 06:25:19.877979541 +0000
Change: 2020-05-23 09:47:27.529719526 +0000
 Birth: -

@larryhastings has been working hard on solving the long-standing problem of managing updates to the Misc/NEWS file in the cpython repos. That file in each branch is an input to the corresponding documentation build to produce its changelog (e.g. https://docs.python.org/3.6/whatsnew/changelog.html). With the changes that Larry is now rolling out to the various branches (already in 3.5), the Docs build now has a dependency on a new utility called blurb (https://pypi.python.org/pypi/blurb/1.0.4). We need to make sure that blurb is now on the PATH during the Docs build. I'm not quite sure what is the best way to accomplish this on docs.iad1.psf.io. The Makefile currently expects blurb to be installed in the python3 that is available in the Docs build, whether that is a "system" python3 or a venv python3 is a question. So pip installing blurb into the right python3 should solve the problem. See the build recipe in the cpython Doc/Makefile for details (currently 3.5 has been updated but others will follow soon).

The specific pip requirement should be:
blurb>=1.0.4

This is a high priority item because the website docs break as these changes are pushed into the various branches.

One of the tasks of docbuild-scripts/build-docs.py is to rebuild downloadable copies of the documentation suite in various formats (.pdf, .epub, etc) and make them available on the website: for example, https://docs.python.org/3/download.html has links to the various files like https://docs.python.org/3/archives/python-3.6.2rc2-docs-pdf-letter.zip. But since the recent release of 3.6.2rc2, while the online html docs are getting updated - note the rc2 in the links - the downloadable files themselves aren't being generated. Those files are served from the docs server at /srv/docs.python.org/3.6/archives and, at the moment, the most recent update there was on 2017-06-28 when 3.6.2rc1 was still current. For other branches, like 3.5 and 2.7, the archives files seem to be being regularly rebuilt as expected. 2017-06-28 also happens to be the date of the last change to build-docs.py, #11.

And, indeed, looking at the most recent /var/log/docsbuild/python36.log.1, we see:

[...]
build succeeded, 19 warnings.

Build finished; the LaTeX files are in build/latex.
Run 'make all-pdf' or 'make all-ps' in that directory to run these through (pdf)latex.
make[3]: Leaving directory '/srv/docsbuild/python36/Doc'
sed -i 's/makeindex/makeindex -q/' build/latex/Makefile
(cd build/latex; make clean && make all-pdf && make FMT=pdf zip bz2)
make[3]: Entering directory '/srv/docsbuild/python36/Doc/build/latex'
rm -f *.dvi *.log *.ind *.aux *.toc *.syn *.idx *.out *.ilg *.pla
make[3]: Leaving directory '/srv/docsbuild/python36/Doc/build/latex'
make[3]: Entering directory '/srv/docsbuild/python36/Doc/build/latex'
pdflatex  'c-api.tex'
This is pdfTeX, Version 3.1415926-2.5-1.40.14 (TeX Live 2013/Debian)
 restricted \write18 enabled.
entering extended mode
(./c-api.tex
LaTeX2e <2011/06/27>
Babel <3.9h> and hyphenation patterns for 2 languages loaded.
(./sphinxmanual.cls
Document Class: sphinxmanual 2009/06/02 Document class (Sphinx manual)
(/usr/share/texlive/texmf-dist/tex/latex/base/report.cls
Document Class: report 2007/10/19 v1.4h Standard LaTeX document class
(/usr/share/texlive/texmf-dist/tex/latex/base/size10.clo)))
(/usr/share/texlive/texmf-dist/tex/latex/base/inputenc.sty
(/usr/share/texlive/texmf-dist/tex/latex/ucs/utf8x.def))
(/usr/share/texlive/texmf-dist/tex/latex/ucs/ucs.sty
(/usr/share/texlive/texmf-dist/tex/latex/ucs/data/uni-global.def))
(/usr/share/texlive/texmf-dist/tex/latex/cmap/cmap.sty)
(/usr/share/texlive/texmf-dist/tex/latex/base/fontenc.sty
(/usr/share/texlive/texmf-dist/tex/latex/base/t1enc.def)<<t1.cmap>>)
(/usr/share/texlive/texmf-dist/tex/generic/babel/babel.sty

! Package babel Error: Unknown option 'french'. Either you misspelled it
(babel)                or the language definition file french.ldf was not found
.

See the babel package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.296 \ProcessOptions*
                      
? 
! Emergency stop.
 ...                                              
                                                  
l.296 \ProcessOptions*
                      
!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on c-api.log.
make[3]: *** [c-api.pdf] Error 1
make[3]: Leaving directory '/srv/docsbuild/python36/Doc/build/latex'
make[2]: *** [dist] Error 2
make[2]: Leaving directory '/srv/docsbuild/python36/Doc'
make[1]: *** [autobuild-dev] Error 2
make[1]: Leaving directory '/srv/docsbuild/python36/Doc'
make: *** [autobuild-stable] Error 2
make[1]: Entering directory '/srv/docsbuild/python36/Doc'
make html SPHINXOPTS=' -Ea -A daily=1 -A versionswitcher=1'
make[2]: Entering directory '/srv/docsbuild/python36/Doc'
/srv/docsbuild/environment/bin/sphinx-build -b html -d build/doctrees -D latex_elements.papersize= -Ea -A daily=1 -A versionswitcher=1 . build/html 
Running Sphinx v1.3.3
[...]

Currently, when a RM changes a symlink, like ln -nfs 3 3.7.0 (to make /3/ point to 3.7), there's no (documented) way to invalidate the cache for /3/*.

Building the 3.8 doc I got:

[...]
DEBUG:Running command 'cp -a /srv/docsbuild/3.8/cpython-en/Doc/dist/* /srv/docs.python.org/3.8/archives'
INFO:597 files changed
DEBUG:Running command ['find', '-L', '/srv/docs.python.org', '-samefile', '/srv/docs.python.org/3.8']
ERROR:Command failed with output 'find: ‘/srv/docs.python.org/release/3.8.0’: Permission denied\n/srv/docs.python.org/3.8\n'
ERROR:Exception while copying to webroot en version 3.8: 'NoneType' object has no attribute 'replace'

This is because RMs are "mkdir"ing the directory with their account on docs.python.org, like:

drwxrwx---  20 ambv      ambv 4.0K Oct 14 18:05 3.8.0

Initially reported in https://mail.python.org/pipermail/doc-sig/2020-May/004194.html

On docs.nyc1.psf.io:

docsbuild@docs:/srv/docsbuild/3.8/locale/ja/LC_MESSAGES$ git branch
* 3.7
  3.8

From an idea of @Mariatta, when we generate the HTML only for a new branch, we could generate a static HTML page containing a nice message like:

This PDF file is currently being built, please try again in a few moments.

It would allow to split again fast HTML build and delayed PDF builds without having the old flood of "the PDF are broken!" in docs@ mailing list.

see python/pythondotorg#1880

where 3.11/whatsnew/3.11.html have a canonical link to 3/whatsnew/3.11.html which does not exists.

It would avoid having conflicting titles in different versions, even if we do it with a big sed it may help avoiding some SEO "bad points".

docsbuild-scripts could manage the /release/ directory too, this directory is currently manually handled by RMs, see PEP 101.

New build has been created in /srv/docs.python.org/fr/ as:

drwxrwx--- 19 docsbuild docsbuild 4.0K Aug  1 23:19 3.7

expected:

drwxr-xr-x 19 docsbuild docs      4.0K Aug  1 23:19 3.7

The docsbuild script logged:

DEBUG:2017-08-03 08:19:25,143:Running command 'chown -R :docs Doc/build/html/'
DEBUG:2017-08-03 08:19:25,150:Running command 'chmod -R o+r Doc/build/html/'
DEBUG:2017-08-03 08:19:25,156:Running command "find Doc/build/html/ -type d -exec chmod o+x {} ';'"
DEBUG:2017-08-03 08:19:25,180:Running command 'cp -a Doc/build/html/* /srv/docs.python.org/fr/3.7'

I don't expect cp to automatically create the directory, I expect a cp: target ‘/srv/docs.python.org/fr/3.7’ is not a directory.

So I don't know what/who created this directory, but in any cases I think docsbuild-script should probably create them itself explicitly with the right group and mode.

The line shell_out("git -C {} checkout {}".format(directory, branch)) is throwing an exception as the 3.7 branch does not exists in french translation, this is expected and don't crash the script. it just don't compile the unexisting translation.

It should be reported nicely instead of logging a whole stacktrace: misreading it the first time make me though the 3.7 english build was failing.

I'm currently fixing this.

Thoughts: I should probably build an untranslated version using po files from a near version… so if a new version spawn, we don't wait for translators teams to push the branch to build the docs.

The idea is that the cpython Doc/ repository should just care about building the documentation, not linking to other builds and languages from docs.python.org.

Building multiple versions, multiple languages, and linking them together is the role of docsbuild-scripts.

This would also avoid the need to update switchers.js and indexsidebar.html on every cpython branch for every new cpython release and every new language.

What would need to change:

• Drop switchers=1 from cpython/Doc/Makefile
• Drop if switchers block in cpython/tools/templates/layout.html. it already has an alternative. Add an id to the <li> so it can be easily found by the new switchers.js as the placeholders are dropped.
• Move switchers.js to docsbuild-scripts and make it use the <li> new id instead of placeholders if they are not found, and maybe fallback to find the place from other ways so it can work on old branches without commiting to them.
• Write a generic "indexsidebar.html" for local builds, to be commited in cpython, and move the "cross-linked" version of indexsidebar in docsbuild-scripts.
• Move {% trans %} tags in dummy.html so they can still be translated even if versionned in docsbuild-scripts (it's the only remaining part in cpython, hope it won't change too often).

We then need a clean way for docsbuild-scripts to edit the HTML to make it load switchers.js.

Yesterday, I rewrote a large part of the documentation. I merged my PR. One hour later, I expected to see it online to validate the rendered HTML, but it was outdated. The day after? It's still outdated. @JulienPalard told me that it's updated once a day.

Would it be possible to rebuild https://docs.python.org/dev/ more frequently? For example, every hour? Or even at each commit? Every hour would be fine with me.

For curious people, my doc change: python/cpython@4b9aad4

See https://mail.python.org/pipermail/pydotorg-www/2016-November/003921.html for original discussion.

When searching Google for "Python timeit" recently, the first hit was for

https://doc.python.org/2/library/timeit.html

The second hit, unfortunately, was for

https://doc.python.org/3.0/library/timeit.html

The first page of results didn't mention

https://doc.python.org/3/library/timeit.html

at all. It seems that the robots.txt file should be tweaked to strongly discourage search engine crawlers from traversing outdated documentation, at least < 3.2 or < 2.6. It's been a long while since I messed with a robots.txt file (so I won't pretend I could submit a proper PR), but something like

User-agent: *
disallow: /3.0/
disallow: /3.1/
disallow: /2.5/
disallow: /2.4/
disallow: /2.3/
disallow: /2.2/
disallow: /2.1/
disallow: /2.0/

should steer well-behaved crawlers away from obsolete documentation.

Python 3.11a0 was released a day ago. But the switcher still shows 3.10 as latest rather than 3.11 (see https://docs.python.org/3.10/whatsnew/3.10.html).

It would probably be nice if we can generate a sitemap of our documentations.

On cpython main, running:

make -C Doc 'SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc= -j4 -q' SPHINXERRORHANDLING= autobuild-dev

with sphinx 3.2.1 gives:

! LaTeX Error: Something's wrong--perhaps a missing \item.

See the LaTeX manual or LaTeX Companion for explanation.
Type  H <return>  for immediate help.
 ...

l.1768 ...hanged}{Changed in version 3.3: }\item {
                                                  }
?             

It happens in library.tex around those lines:

\begin{itemize}
\DUrole{versionmodified,changed}{Changed in version 3.3: }\item {} 
The \sphinxstyleemphasis{opener} parameter was added.

\item {} 
The \sphinxcode{\sphinxupquote{'x'}} mode was added.

\item {} 
{\hyperref[\detokenize{library/exceptions:IOError}]{\sphinxcrossref{\sphinxcode{\sphinxupquote{IOError}}}}} used to be raised, it is now an alias of {\hyperref[\detokenize{library/exceptions:OSError}]{\sphinxcrossref{\sphinxcode{\sphinxupquote{OSError}}}}}.

\item {} 
{\hyperref[\detokenize{library/exceptions:FileExistsError}]{\sphinxcrossref{\sphinxcode{\sphinxupquote{FileExistsError}}}}} is now raised if the file opened in exclusive
creation mode (\sphinxcode{\sphinxupquote{'x'}}) already exists.

\end{itemize}

Currently, the /dev/ symlink is maintained manually and easy to forgot, and easy to automate, so it should be done automatically.

I mean, when a french browser hits docs.python.org/ it could be redirected to docs.python.org/fr/ in a clean way.

I'm sure some people prefer reading the english version anway, so we need to find a smart way of doing it, a way such that nobody can complain about it.

We may drop the now unused hg compatibility, and make --git the default.