debian / debiman Goto Github PK

View Code? Open in Web Editor NEW

185.0 20.0 46.0 2.35 MB

debiman generates a static manpage HTML repository out of a Debian archive

License: Apache License 2.0

Go 98.75% CSS 1.25%

manpages debian

debiman's Introduction

debiman

Goals

debiman makes (Debian) manpages accessible in a web browser. Its goals are, in order:

completeness: all manpages in Debian should be available.
visually appealing and convenient: reading manpages should be fun, convenience features (e.g. permalinks, URL redirects, easy navigation) should be available
speed: manpages should be quick to load, new manpages should be quickly ingested, the program should run quickly for pleasant development

Currently, there is one known bug with regards to completeness (#12).

With regards to speed, debiman can process all manpages of Debian unstable in less than 10 minutes on a modern machine. Incremental updates complete in less than 15 seconds. For more details, see PERFORMANCE.md.

Prerequisites

mandoc
a local or remote Debian mirror or an apt-cacher-ng running on localhost:3142
a number of Go packages (which go get will automatically get for you, see below)
- pault.ag/go/debian
- pault.ag/go/archive
- github.com/golang/protobuf/proto
- golang.org/x/crypto/openpgp
- golang.org/x/net/html
- golang.org/x/sync/errgroup
- golang.org/x/text

Architecture overview

debiman works in 4 stages:

All Debian packages of all architectures of the specified suites are discovered. The following optimizations are used to reduce the number of packages, and hence the input size/required bandwidth:
1. packages which do not own any files in /usr/share/man (as per the Contents- archive files) are skipped.
2. each package is downloaded only for 1 of its architectures, as manpages are architecture-independent.
Man pages and auxiliary files (e.g. content fragment files which are included by a number of manpages) are extracted from the identified Debian packages.
All man pages are rendered into an HTML representation using mandoc(1).
An index file for debiman-auxserver (which serves redirects) is written.

Each stage runs concurrently (e.g. Contents and Packages files are inspected concurrently), but only one stage runs at a time, e.g. extraction needs to complete before rendering can start.

Development quick start

Set up Go

Install the latest supported version of Go from https://go.dev/dl. If you prefer to install Go from Debian, ensure you get the same version — if you use Debian stable, you likely need to install from backports.

Install debiman

To download, compile and install debiman to ~/go/bin, run:

go install github.com/Debian/debiman/cmd/...@main

Run debiman

To synchronize Debian testing to ~/man and render a handful of packages, run:

~/go/bin/debiman -serving_dir=~/man -only_render_pkgs=qelectrotech,i3-wm,cron

Test the output

To serve manpages from ~/man on localhost:8089, run:

~/go/bin/debiman-minisrv -serving_dir=~/man

Note that for a production setup, you should not use debiman-minisrv. Instead, refer to the web server example configuration files in example/.

Recompile debiman

To update your debiman installation after making changes to the HTML templates or code in your debiman git working directory, run:

go generate github.com/Debian/debiman/...
go install github.com/Debian/debiman/cmd/...

Synchronizing

For https://manpages.debian.org, we run:

flock /srv/manpages.debian.org/debiman/exclusive.lock \
nice -n 5 \
ionice -n 7 \
debiman \
  -sync_codenames=oldstable,oldstable-backports,stable,stable-backports \
  -sync_suites=testing,unstable,experimental \
  -serving_dir=/srv/manpages.debian.org/www \
  -local_mirror=/srv/mirrors/debian

…resulting in the directories wheezy/, wheezy-backports/, jessie/, jessie-backports/, testing/, unstable/ and experimental/ (respectively).

Note that you will NOT need to change this command line when a new version of Debian is released.

When interrupted, you can just run debiman again with the same options. It will resume where it left off.

If for some reason you notice corruption or other mistakes in some manpages, just delete the directory in which they are placed, then re-run debiman to download and re-process these pages from scratch.

It is safe to run debiman while you are serving from -serving_dir. debiman will swap files atomically using rename(2).

Customization

You can copy the assets/ directory, modify its contents and start debiman with -inject_assets pointed to your directory. Any files whose name does not end in .tmpl are treated as static files and will be placed in -serving_dir (compressed and uncompressed).

There are a few requirements for the templates, so that debiman can re-use rendered manpages (for symlinked manpages):

In assets/manpage.tmpl and assets/manpageerror.tmpl, the string <a class="toclink" is used to find table of content links.
</div>\n</div>\n<div id="footer"> is used to delimit the mandoc output from the rest of the page.

interesting test cases

crontab(5) is present in multiple Debian versions, multiple languages, multiple sections and multiple conflicting packages. Hence, it showcases all debiman features.

w3m(1) has a Japanese translation which is only present in UTF-8 starting with Debian jessie. It also has a German translation starting with Debian stretch.

qelectrotech(1) has a French translation in 3 different encodings (none specified, ISO8859-1, UTF-8).

mysqld(8) is present in two conflicting packages: mariadb-server-core-10.0 and mysql-server-core-5.6.

URLs

The URL schema which debiman uses is (<suite>/)(<binarypkg/>)<name>(.<section>(.<lang>)). Any part aside from name can be omitted; here are a few examples:

Without suite and binary package:

With binary package:

With suite:

With suite and binary package:

debiman's People

Contributors

Stargazers

Watchers

debiman's Issues

Add a print stylesheet

The man page should fill the entire page. Whether there should be any borders or headers should be the user’s preference (via the print dialog).

tests: add a blackbox end-to-end test

I.e., add a tiny mirror with one package to testdata/, have the tests run the entire debiman logic on it. This should prevent issues like the one which commit f8ca751 fixed

lower-case section names

e.g. 1ssl (openssl) vs. 1SSL (manpages-fr-extra)

change highlighted panel entry background color

optimization: tokenize HTML or process textually entirely

Tokenizing shaves off about 1 minute on a 6 minute rendering of Debian unstable.

The code is not entirely straight-forward to port due to the HTML-tag-agnostic cross reference detection (e.g. for <i>crontab</i>(5)) which requires us to keep state after all.

If we could improve mandoc’s cross reference detection and id generation, we could probably get away with textually processing the HTML, which has the potential to shave off another 30 seconds.

Provide html anchors for Options

Hello,

Sometimes my usage of manpages is to send someone documentation about a command or specifically an option of a command. It might be nice to have anchors in the options as well to point exactly where somebody wants.

Cheers

table of contents links with spaces don’t work

From the debian-doc mailing list:

One little remark : the links in the table of content are inactives because spaces are not translated to underscores :
<a class="toclink" href="#EXIT%20STATUS" title="EXIT STATUS">EXIT STATUS</a>

And then :
<h1 id="EXIT_STATUS">EXIT STATUS<a class="anchor" href="#EXIT_STATUS">

Un-brand templates

Other people might want to make their debiman repositories publically available (intentionally or unintentionally). The default template should not be Debian-branded so that no confusion is created with regards to whether such repositories are to be considered official or not.

Paragraph (code?) not indented as in man

Looking at crontab(5) in my manpage viewer shows code blocks (is what I think they are) being indented, like so:

will not work as you might expect. And neither will this work

    A=1
    B=2
    C=$A $B

There will not be any subsitution for the defined variables in the last value.

However not on the page, where it is rendered like all other text:

...
will not work as you might expect. And neither will this work

A=1
B=2
C=$A $B

There will not be any subsitution for the defined variables in the last value.

Browsers:

$ firefox -v
Mozilla Firefox 45.6.0
$ chromium --version
Chromium 55.0.2883.75 built on Debian stretch/sid, running on Debian stretch/sid

There should be a diff visualizer for manpages in two or more suites

It would be great/fantastic if we had a manpage diff output facility. Meld is a utility which does similar function except in meld i have to provide the names which need to be parsed. Look forward to having such a service.

Render HTML versions of symlinked manpages instead of symlinking them

…otherwise, the navigation is confusing.

Take for example the crontab(5) manpage, which is present in the cron and bcron-run binary packages. Looking at jessie/cron/crontab.5.en.html, you can see the conflicting packages, but looking at jessie/bcron-run/crontab.5.en.html, you can’t. This is because the latter is a symlink to jessie/bcron/bcrontab.5.en.html, which doesn’t conflict.

We should restructure renderAll so that it first processes regular files, then processes symlinks. That way, we can re-use the content fragment and avoid duplicate rendering cost.

search when viewing a manpage should search within the same distribution as current manpage

Say I am looking at https://manpages.debian.org/unstable/systemd/systemd.directives.7.en.html, that is the unstable version of a manpage. Then I decide to search for systemctl. The result is https://manpages.debian.org/jessie/systemd/systemctl.1.en.html , the stable version. I think the result should come (if possible) from the same distribution.

footer: add mandoc version which was used for conversion

disambiguate languages where necessary

Some (slightly?) different languages have the same word for themselves. Examples:

português (pt, pt_BR)
繁體中文 (zh_HK, zh_TW)
català (ca, ca@valencia)

Example page: https://manpages.debian.org/jessie/deja-dup/deja-dup.1.en.html

add title= argument to opensearch reference for Firefox

Reported via email:

We should replace

<link rel="search" type="application/opensearchdescription+xml" href="https://manpages.debian.org/opensearch.xml">

with

<link rel="search" title="Debian manpages" type="application/opensearchdescription+xml" href="https://manpages.debian.org/opensearch.xml">

Some languages are not displayed

Affected languages (at least):

This is not just a cosmetic issue, the languages are also not clickable.

Example page: https://manpages.debian.org/jessie/deja-dup/deja-dup.1.en.html

Make /<suite> URLs result in contents

package renames break navigation between different Debian versions

Easy to reproduce, either use the search to get to the manpage for systemd-nspawn of Jessie's version and see that you don't find a link to manpages of systemd-nspawn of the Testing and Unstable versions.
Alternatively, browse the manpage repositories of Testing and Unstable to not find the manpages of systemd-nspawn. The same is true for machinectl.
Might this be caused by the move of these tools to a separate package?

Investigate raw manpage deduplication

Manpages can be identical across different Debian versions. Our corpus of extracted manpages weighs about 2.5 GB. We should investigate whether it’d be worthwhile to de-duplicate content (by using hard links, most likely). If the savings are not substantial, it might be more trouble than it’s worth.

launch

This checklist tracks items that need to be fulfilled before we can launch:

After a few days:

de-activate the CGI script

Add (lintian?) tests to enforce assumptions / best practices

verify that manpages with .pl, .py, … suffixes are not accidentally installed into language sub-directories
verify language subdirectories are correctly spelled
verify manpages don’t differ between architectures (FHS violation)
verify correct encoding

webfont lacks latin-ext subset

Polish manpages have Polish national characters substitized from other font.

See at https://manpages.debian.org/jessie/manpages-pl/intro.4.pl.html

add <link rel="alternate" hreflink=""> to pages

…so that search engines can deliver results in the best lanugage for their users. See also https://support.google.com/webmasters/answer/2620865?hl=en

Support man.freebsd.org's syntax

FreeBSD's corresponding service supports passing the section as:

http://man.freebsd.org/printf/3

Would you consider supporting the same URL structure? (If it's unambiguous...) The advantage over http://manpages.debian.org/printf%283%29 is that neither shell-quoting (for the ( )) nor percent-encoding are required, and over http://manpages.debian.org/printf.3 that (a) man pages with dots in their names would work unambiguously (think rsyncd.conf(5)), (b) same URL structure across platforms means fewer things to concern one's brain with.

Thanks for restoring the service!

"table of contents" panel header styling broken on firefox-esr

Currently, e.g. openssl has the red border just around the "table of contents" text, rather than the whole box. This appears to be because that panel uses details and summary tags rather than the div structure of the other panels.

Great upgrade!

Hard links are not extracted from Debian packages

If I type ip into the ‘Directly jump to any manpage:’ box, I’m redirected to https://manpages.debian.org/jessie/freebsd-manpages/ip.4freebsd.en.html and that page says 404.

Have a way to search and show manpages from various suites

The current way to view a manpage from a suite other than stable requires quite a bit of expertise. Having a drop-down list of the various suites should make it easier.

package for Debian

To make deployment and usage easier, we should create a Debian package for debiman.

Also store static assets uncompressed

This makes the gzip-fallback aspect of the webserver config way simpler, and doesn’t waste much disk space.

indentation in code examples is lost

Some manpages have source code examples, like scanf(3).
The generated HTML page loses the indentation, which doesn't look very nice.

text overlaps panels with qutebrowser 0.9.0

reported in https://www.reddit.com/r/linux/comments/5oqrb9/show_rlinux_manpagesdebianorg_has_been_modernized/dclizgd/

Add fallback for the SVG logo

Tor browser in high security mode does not render SVGs.

programmatically query manpage details

Hi,

is there a way to query a raw manpage directly only by its (unambiguous) name? I want to use it from the CLI.

Best,
Andreas

Jump to manpage gives result in simplified Chinese

Steps to reproduce:

visit https://manpages.debian.org/
enter pg_dump in text field
press 'Jump to manpage' (https://manpages.debian.org/jump?q=pg_dump)
get redirected to https://manpages.debian.org/jessie/manpages-zh/pg_dump.1.zh_CN.html

The expected result is to see the man page in English as opposed to the simplified Chinese version.

Always display binary package alongside “other languages” and “other sections”

Currently, binary packages are only displayed when the choices are ambiguous, so as to reduce visual clutter.

Please give this issue a thumbs up/thumbs down reaction depending on your opinion, and we’ll act accordingly.

optimization: reduce auxserver memory usage

Currently, auxserver uses ≈ 200MB of memory, which likely can be significantly reduced.

Reducing the steady-state memory usage leaves us more memory for actual debiman runs and the filesystem cache.

Explore generating apache RewriteMaps

See https://httpd.apache.org/docs/2.4/rewrite/rewritemap.html for details.

This would result in an entirely static page, enabling us to distribute it across different machines more easily.

Optimize webfont loading even more

Currently, we asynchronously load WOFF webfonts unconditionally. Additionally, we should:

skip any loads if the fonts are installed locally
detect whether the browser supports WOFF2 and load WOFF2 instead

Please strip <> from the URL if it in <url> format

Certain man pages put an URL in braces like:

in the source distribution or at
<https://www.openssl.org/source/license.html>.

which is from SSL.7ssl.en.
The rendered version adds the closing > to the URL so you can't click on it.

Investigate also processing Debian experimental

At least one user wanted to access a manpage which was only in experimental. Let’s see how much additional processing time/disk space would be necessary to process experimental as well.

limit maximum width of man content to 80 characters

See https://www.reddit.com/r/linux/comments/5oqrb9/show_rlinux_manpagesdebianorg_has_been_modernized/dcld26d/ for context.

Symlinked manpages with ambiguous target

Symlinks which resolve to a file that is not shipped within the same package might be ambiguous if multiple packages ship the resolved file. We should see if the ambiguity can be resolved, or, if that isn’t possible, if we can at least make a deterministic choice and document this limitation.

The following Debian binary packages are currently affected:

mencoder, mplayer-gui (referencing mplayer.1 from either mplayer or mplayer2)
aterm-ml, aterm (referencing urxvt.1 from either rxvt-unicode, rxvt-unicode-256color or rxvt-unicode-lite)
mingw-w64-tools (referencing pkg-config.1 from either pkg-config or pkgconf)
udhcpc, udhcpd, busybox-syslogd (referencing busybox.1 from either busybox or busybox-static)
mysql-client-5.6, mariadb-client-10.0, mariadb-client-10.1 (referencing either mysql-client-core-5.6, mysql-client-core-5.7, mariadb-client-core-10.0 or mariadb-client-core-10.1)

footer: add snapshot.debian.org link to Debian binary package containing the manpage

Add Debian contrib

As per hmh:

Maybe you could consider adding the manpages from packages in contrib as
well? Unlike non-free, the licenses in contrib are all compatible with
the DFSG, so they must not have any license restrictions that would get
in the way...

Implement slave alternative manpages

A large number of packages makes manpages available via slave alternatives. See https://codesearch.debian.net/search?q=path%3Adebian%2F+update-alternatives+--install for an upper bound.

This is tricky to implement, because slave alternatives are configured in the postinst maintainer script, i.e. via update-alternatives calls in arbitrary shell script. It would be brittle to parse that shell script without running it, not to mention that its behavior might depend on the package actually being installed.

It seems like the only solution is to actually install the package and then introspect the alternatives database, e.g. /var/lib/dpkg/alternatives/vi.

A few test cases are:

vi
pg_dump

Set up continuous integration on travis

To quickly evaluate pull requests and tell others in what state our project is, we should set up travis to run our test suite.

Table columns not aligned

crontab(5) does not align the table in the description.

What I see:

field allowed values
 
----- --------------
minute 0-59
hour 0-23
day of month 1-31
month 1-12 (or names, see below)
day of week 0-7 (0 or 7 is Sun, or use names)

What I expect to see:

field         allowed values
-----         --------------
minute        0-59
hour          0-23
day of month  1-31
month         1-12 (or names, see below)
day of week   0-7 (0 or 7 is Sun, or use names)