theleagueof / fontship Goto Github PK

Since this build system kind of takes over versioning fonts we should document what is happening and how to control it. Designers might be surprised to find the version meta data they set it the font source is actually discarded and replaced with normalized versioning based on Git tags. Furthermore they may not know how to do anything about it.

(Post RIIR #98)

Copy shell completion generation from CaSILE.

See e.g. this comment:

We could make some, but I’m left pondering whether it makes sense for every libre font repo to have it’s own set of idiosyncratic proofing documents rather than, say, a common script repo that generates proofing documents based on character coverage and OTL features?

It should be pretty easy to scan the coverage and features of font sources and generate useful proofs. The trick will be less in generating everything that could be covered and more in filtering the proofs down to just what is most useful/relevant to review.

Possibly leverage fontproof for this.

Right now projects that choose to use this have a roughly 2:30 delay while Github rebuilds the Docker image from scratch on every invocation.

- name: Fontship make
  uses: theleagueof/fontship@master

Gituhb also supports pulling the image from Docker hub, and that only takes about 39 seconds:

- name: Fontship make
  uses: docker://theleagueof/fontship:master

It still doesn't even cache that and downloads the image every time!

I messed around with using fixed commits, tags, etc. and still got rebuilds. There must be a way to stop that nonsense and just use the last built image but I haven't figured out what that is yet.

How are we supposed to pitch a tool to designer types if it doesn't even have a shinny logo?

Unfortunately I'm that long-bearded programmer dude who lives in a terminal (yes, I am in fact composing this very issue in vim!). Somebody else is going to have to take this one on. Does anybody want to give back to @theleagueof for all the free fonts by contributing some cool branding for this project?

Somehow the make dist target thinks it needs to build the CLI binary. This shouldn't be necessary as only the sources are included in the dist, not the actual build. I'm unsure what the proper usage of automake is here so leaving this for later.

Note that CI jobs will probably need separating too.

Also currently make dist fails on the first run for any new commit. I'm guessing this is related and will go away when building the binary doesn't trip up the source builder, but in the event it does not that should subsequently be tracked in a new issue.

VFC/VFJ (latter recommended) should be a standard option as canonical format.

I don't know what open tooling is out there to convert to/from UFO or anything else we can run fontmake on, but it's worth looking into.

Bob and Alice each running fontship to generate fonts from source should both come out with identical products.

To take a current example, if I check out league-spartan and run the ./build.sh tool, I get all the OTF/TTF/VF products regenerated. Git has one look at this and says, "AAAAHHH" and throws up its hands saying a bunch of binary blobs have changed.

Now I realize that we're going to be dropping these binary products from the git repository anyway, but the point holds that it's hard to tell what changes if things are always changing for no reason.

Drilling down a bit more and using fdiff to see what actually changed, it's only a checksum and modification timestamp that changed in each font file.

The conversion process for ALL formats should normalize everything such that there are no values to differ like this unless there are actual differences. Bob should be able to get the same checksum as Alice having each generated the fonts on their own systems. This is a fundamental requirement to know that our thing-a-ma-giggers work.

We are going to need some documentation for this. Usage for the CLI will be available places like fontship --help and man fontship, but some higher level documentation is probably in order — especially given the target audience of non-programmers.

What's the best place to put this?

• Just keep appending to the README?

  High discoverability but in my opinion this is going to get cumbersome. I think the Readme would be more useful if kept to the very basic overview and getting started stuff and deferred to full documentation for more in depth coverage.

• A readthedocs site?

  Since this is pretty much the de facto place for Python documentation and lots of font tooling is in Python, this might be something people are already familiar with. See for example fonttools.

• A Github Wiki?

  It's currently not enabled for this project but Github allows a Wiki per-project, so we could use that.

• A custom static built site using Github Pages?

  Probably the most bang, but also the most buck. Not terribly hard to setup but requires a bit more hands on approach.

We can scrape or guess most of the meta data we need about a project, but not quite all. As time goes on and more font projects use this there will be more things to tweak. This requires some sort of configuration, which needs to be saved somewhere. What format should it be in?

(Note the examples here are contrived, we probably don't need a family-name field at all because we can detect it from sources, but I just made up something to show how different formats might look.)

• JSON

  {
    "family-name": "Foo Bar",
    "hint": {
      "otf": "false",
      "ttf": "true"
    }
  }

  Easily machine readable and interoperable, but not that great for hand editing by non-programmers

• YAML

  family-name: Foo Bar
  hint:
  - otf: false
  - ttf: true

  Much more flexible and much easier to edit by hand especially by non-programmers, but also has the potential to be complex and can be finicky in unexpected ways.

• TOML

  [Fontship]
  family-name = "Foo Bar"
  hint = { otf = false, ttf = true }

  Less finicky, fewer possible syntax rules easy to understand, but not quite as flexible. Possible less widely recognized by tooling.

• Makefile

  FamilyName = "Foo Bar"
  HINT_OTF = false
  HINT_TTF = true

  Great for integrating with our actual build rules because we don't have to parse and output anything, but not so great for font designer types to edit by hand.

• Python

  FamilyName = "Foo Bar"
  Hint = [ otf = false, ttf = true ]

  Also great for integrating with font ecosystem tooling and some font designers will have a smattering of Python already, but discoverability is perhaps much lower with this (and the Makefile option) than with dedicated data formats.

• Something else?

Personally I lean towards YAML, but anything will work really.

When I attempt to build the master branch of https://github.com/alerque/libertinus using v0.7.1 using a local fontship installation, I get an error saying

make: *** No rule to make target '.fontship/LibertinusGitSerif-Regular-subr.otf', needed by 'LibertinusGitSerif-Regular.otf'.  Stop.

My guess is that the project name is inferred to be LibertinusGit instead of Libertinus.
v0.7.0 works without any problems.
(To be honest, I only suspect this to be a fontship issue which is why I’m reporting it here, but I’m not sure.)

Similar to #19, but for sources.

My vote is to keep the font source files front and center in the root of the repository.

The advantages I see in this are:

• Discourages people using the "old normal" combo of sub directories for "sources" and "fonts" used by the vast majority of current font projects I see around Github where "sources" may or may not be up to date, may or may not match "fonts", etc. By putting sources at the root it makes it painfully obvious that they are the main attraction in a repository.

• Less clutter in editors and other UIs that show open file paths. Often long paths are even abbreviated using the trailing path segment for which purpose "sources" is just not useful.

• Reduced navigation and typing, whether in file managers or Github or the CLI the main thing that's changing and being references should have the most straight forward path.

Glyphs introduces several dynamic noisy bits as you use the app that are inevitably merge conflicts in the event of contributions from more than one person or branch. One at least is just a datestamp of the last save. We should be able to write a normalizer that smooths these bits over so that merge conflicts are less common.

Set this up as a smudge/clean filter? Pre-commit hook?

Similar to #36 but for generated artifacts (production font builds, whatever you want to call them) the TTF & OTF files have date stamps in them. This is slightly interesting information but it breaks the concept of reproducible builds. I can't verify that two font builds are exactly the same if they change on every build even when nothing changes.

I propose we set this field on all generated artifacts to the timestamp of the Git commit they were built from.

Hey @micahbrich I think GitHub changed something in their developer agreement and you need to accept the terms on behalf of theleagueof org again:

From this project, Releases tab, click to edit one of the existing releases. The top of the form is "Publish this release" but for me has a warning that an admin needs to accept the agreement first.

This has been outstanding for a couple months now but I just pushed a release through last night no it matters a bit right now that it shows up in the right places.

In the event the project itself doesn't have it's own, we would generate an instruction file (INSTALL.txt or similar) with various platform instructions.

The new color output is pretty in a terminal, but it only outputs it a terminal.

When an interactive environment is not detected it is just normal.

This is an appropriate compromise, but in the case of GitHub Actions runners, some color support does exist. See for example super-linter output:

Figure out how to output even friendlier output in the case of Actions (and possibly other runners).

The project name scraped from the project directory name is obscured in Docker where the volume mount is just "data".

(Post RIIR #98)

Output is either too verbose or too quiet depending on how things worked out.

We should be able wo capture and wrap output on a per-target basis, show what steps are being run, bun hide output except in the event of a failure.

See https://stackoverflow.com/questions/37952098/how-to-run-pre-and-post-recipes-for-every-target-using-gnu-make

There is a GNU Make target called debug that dumps a bunch of variables to aid in debugging the build process. It should be possible to reach just with fontship make debug, but I've known the output filtering has needed the fontship -v make debug verbose flag to work for a while. As I'm working on #120 it seems even that workaround is lost and the output is simply captured all the time. Not so good.

I don't know how to lay this out (external demo repositories? script to generate them internally? tarballs?) but one way or another the test suite for fontship itself must include generating fronts from various different canonical sources. Already I've broken master multiple times with unexpected side effects that don't impact the thing I was testing but do affect other font projects. Testing that fontship itself builds and runs isn't enough, it needs to generate expected results from actual font sources.

Something like FooBar-VF[weight,slant].ttf seems to be the industry practice and that seems useful to me.

Because it will save us from people doing exactly what we don't want: commiting built artifacts to the repo!

I am having some trouble building the Libertinus fonts.
(one thing is that the Libertinus build script only accepts sfdLib 1.0.8, not 1.1.0, because then it has trouble with from sfdLib.utils import GLYPHCLASS_KEY, MATH_KEY.)
There is some trouble with psautohint (see dump below). I tried updating to the git release, updating psautohint to 2.3.0, updating fonttools, but it did not help. I don't know where this error is coming from and it is possible that it isn't a Fontship bug, but I can't track down where it goes wrong. According to the checks, package prerequisities should be satisfied and Fontship compilation goes smoothly, it's only the building that's the problem.
I did not manage to get Docker running so I have just been using the compiled Fontship version. Lately it stopped working, so I decided to upgrade from 0.7.2 to 0.7.6, and that's when I started to have this problem.

┏━ Welcome to Fontship ⁨v0.7.6.r5-gf0302d2⁩
┠─ Everything seems to be ship shape, anchors up!
┣━ Building target(s) using ‘make’
┠┄ Starting make job for target: ⁨.fontship/last-commit⁩
┠┄ Starting make job for target: ⁨.fontship/LibertinusSerif-Regular-instance-hinted.otf⁩
┠┄ Starting make job for target: ⁨.fontship/LibertinusSerif-Semibold-instance-hinted.otf⁩
┠┄ Starting make job for target: ⁨.fontship/LibertinusSerif-Bold-instance-hinted.otf⁩
┠┄ Finished make job for target: ⁨.fontship/last-commit⁩
┠┄ Starting make job for target: ⁨.fontship/LibertinusSerif-Italic-instance-hinted.otf⁩
┠┄ Failed make job for target: ⁨.fontship/LibertinusSerif-Italic-instance-hinted.otf⁩
┠┄ Failed make job for target: ⁨.fontship/LibertinusSerif-Bold-instance-hinted.otf⁩
┠┄ Failed make job for target: ⁨.fontship/LibertinusSerif-Regular-instance-hinted.otf⁩
┠┄ Failed make job for target: ⁨.fontship/LibertinusSerif-Semibold-instance-hinted.otf⁩
┖┄ Dumping captured output of ‘make’
usage: psautohint [-h] [-v] [-o PATH [PATH ...]] [-r PATH] [-a] [-w]
usage: psautohint [-h] [-v] [-o PATH [PATH ...]] [-r PATH] [-a] [-w]
[-g GLYPH_LIST | --glyphs-file PATH | -x GLYPH_LIST | --exclude-glyphs-file PATH]
[-g GLYPH_LIST | --glyphs-file PATH | -x GLYPH_LIST | --exclude-glyphs-file PATH]
[-c | --report-only] [--log PATH] [-d] [--no-flex]
[-c | --report-only] [--log PATH] [-d] [--no-flex]
[--no-hint-sub] [--no-zones-stems] [--fontinfo-file PATH]
[--no-hint-sub] [--no-zones-stems] [--fontinfo-file PATH]
[--print-dflt-fddict] [--print-list-fddict] [--doc-fddict]
[--print-dflt-fddict] [--print-list-fddict] [--doc-fddict]
[--info] [--version] [--traceback]
[--info] [--version] [--traceback]
FONT [FONT ...]
FONT [FONT ...]
psautohint: error: the following arguments are required: FONT
psautohint: error: the following arguments are required: FONT
usage: psautohint [-h] [-v] [-o PATH [PATH ...]] [-r PATH] [-a] [-w]
[-g GLYPH_LIST | --glyphs-file PATH | -x GLYPH_LIST | --exclude-glyphs-file PATH]
[-c | --report-only] [--log PATH] [-d] [--no-flex]
[--no-hint-sub] [--no-zones-stems] [--fontinfo-file PATH]
[--print-dflt-fddict] [--print-list-fddict] [--doc-fddict]
[--info] [--version] [--traceback]
FONT [FONT ...]
psautohint: error: the following arguments are required: FONT
usage: psautohint [-h] [-v] [-o PATH [PATH ...]] [-r PATH] [-a] [-w]
[-g GLYPH_LIST | --glyphs-file PATH | -x GLYPH_LIST | --exclude-glyphs-file PATH]
[-c | --report-only] [--log PATH] [-d] [--no-flex]
[--no-hint-sub] [--no-zones-stems] [--fontinfo-file PATH]
[--print-dflt-fddict] [--print-list-fddict] [--doc-fddict]
[--info] [--version] [--traceback]
FONT [FONT ...]
psautohint: error: the following arguments are required: FONT
make: *** [/usr/local/share/fontship/rules/rules.mk:307: .fontship/LibertinusSerif-Italic-instance-hinted.otf] Error 2
make: *** Waiting for unfinished jobs....
make: *** [/usr/local/share/fontship/rules/rules.mk:307: .fontship/LibertinusSerif-Bold-instance-hinted.otf] Error 2
make: *** [/usr/local/share/fontship/rules/rules.mk:307: .fontship/LibertinusSerif-Regular-instance-hinted.otf] Error 2
make: *** [/usr/local/share/fontship/rules/rules.mk:307: .fontship/LibertinusSerif-Semibold-instance-hinted.otf] Error 2
┎┄ End dump
┗━ Fontship run complete
Error: Custom { kind: InvalidInput, error: "Make returned unknown error." }

Many / most fonts have various features that can be enabled or disabled through OT feature flags. Unfortunately there are still quite a few applications out there that don't support this. Releasing every possible combination of features as separate fonts is silly and unrealistic to do by hand, but if we had a simple way to generate instances automatically or with a simple fontship command people could easily grab the ones they want.

See theleagueof/raleway#5 for an example.

The current strategy of generating a single release bundle with all target formats (TTF, OTF, WOFF, WOFF2 for both static and variable for a potential of 8 formats) is generally useful, but it's not what all projects need.

As a follow up to #88, I suggest we add optional targets that would generate release packaging filtered to specific output format(s).

Nope not a typo for FEA.

As soon as fontFeatures has an official release I can work this into build rules so that FEE files included in project source get built into FEA and into the final fonts automatically.

Anybody with opinions please chime in!

I'm working on the make dist function for fontship and thinking about what folder layout to standardize on. Note this is for what goes in the final ZIP (or otherwise compressed archive) that is intended for font end user distribution — not the Git repository layout.

I maintain hundreds of distro packages for various free fonts and honestly the situation I've observed across a variety of sources is pretty chaotic. Even across fonts from The League of Movable Type there are 3 or 4 basic layouts plus some minor variants! Some ZIPs have loose font files at the top level, some have everything in a project name folder. Some have fonts split by format, some group multiple formats by usage (e.g. EOT and WOFF under 'web'), and so forth.

I think the questions we need to answer are:

1. Do we include a top level folder with the font family name? and a version string?

   My suggestion is emphatically YES on this question. A downloaded ZIP file for release 1.002 of font family Foo Bar should be FooBar-1.002.zip and extract to a top level directory of FooBar-1.002. I'll defend this the merits of this vigorously, but unless challenged I'll just assume this.

2. Should files be grouped, and if so should the main grouping be by format or usage?

   a. If yes, should they be capitalized?

   My vote is for yes, by format, and capitalized (i.e. ditch things like "web/" and just have "WOFF/" etc.)

3. Where should variable fonts fit into this? Should they be stuffed in the per format folder (i.e. static and variable TTFs in TTF/ or separated? If separated should all the formats be lumped together or further split? Should static fonts have a matching structure? Should the static/variable dirs be top level or under formats?

   I'm not so sure on this one. My first idea was to just keep them alongside statics by format:

   TTF/FooBar-Thin.ttf
   TTF/FooBar-Black.ttf
   ...
   TTF/FooBar-VF[weight].ttf
   

   The caveat is that it becomes slightly harder for packagers who package these separately because simple glob expansions like install -Dm644 -t "/usr/share/fonts/TTF/" TTF/*.ttf become more complicated needing to separate static from variable files.

   If we do split them, I vote the split be top-level nod mid-level (i.e. variable/TTF/... not TTF/variable/...) and that static instances be split into a matching folder so everything is at the same level and we emphasize the nature of the files (i.e. static/TTF/FooBar-*.ttf). This should give end users the best shot at both finding what they want out of the package and understanding what is there to pick from. The slight downside is that it would probably be slightly easier on packagers to script installations with the mid-level layout, but since glob expressions could be written either way prioritizing the end user discovery seems to make the most sense. My fear with mid-level layout would be that someone who went straight for the first level OTF folder might be disappointed to find only a static entry and no variable (since it's only in the TTF version).

I don't know what screw got loose for the output of some builds (e.g. this run on Libertinus) to be so verbose, but we shouldn't be defaulting to outputting so much stuff that CI runners truncate the output. Just no.

I think psautohint is the culprit here, but not sure.

Look into setting up tooling such that fontship publish or similar can publish fonts build from release tags directly to Type.World, including from CI.

• GitHub Project
• Developer docs
• API docs

Having ./configure check for run-time dependencies is great for making sure people don't get frustrated with a system that doesn't work and for packagers to make sure they setup dependencies right, but it is somewhat less than idea for two reasons:

1. It means the host system building Docker images using make docker will be expected to have the run-time dependencies, which is unnecessary.

2. It should be possible to build packaging with only build-time dependencies.

A guard and configure flag to optionally skip these checks is probably the way to go.

See googlefonts/fontmake#714

Hasn't hit a release version yet, but when it does we should have a meta data configuration option so project can use it easier than fiddling with custom flag variables.

Add a make target that will generate a package that bundles webfont builds of a font project and any necessary CSS with suitable meta generated from what we already know about a project in a form suitable for submission to an NPM package registry.

Such a package should make it easy to include as a dependency for many web projects that use npm to manage dependencies already to add a font as a dependency with good versioning and easy integration into a site publishing workflow.

I know that we'll need to generate a package.json file, but I don't know what the best file layout should be for such a webfont package. If anybody knows off the top of their head or has examples of specific font projects that have done this well already comments would be much appreciated.

The entire ecosystem of fonts is already way too Latin/English oriented. We don't need to perpetuate that by forcing English-only tooling on font developers. Everything user (developer) facing should be easy to localize. This should be done with with Fluent of course, via python-fluent.

Auto generated variables (mostly instance←→source maps) getting stuffed into a temporary file in the project root (because they are used on first pass of parsing the Makefiles where build dirs might not be known or created yet) are not getting deleted after each make run.

It seems like this is possible, but I haven't found the right combo of Python tools to actually do it yet. If anybody knows or bumps into it being done anywhere...

fontship make -B or fontship make <target> -W <source> aren't passed through, probably caught by Python arg parser. Everything after a make subcommand should get passed straight through.

Github workflow pulls the newest Fontship and that causes an error even with commits where there was none before (I checked). There seems to be a Python module unidecode missing somewhere.
https://github.com/ivo-s/libertinus/runs/1671895283

I am just guessing this is a FontShip bug, because I didn't find any package configuration in the Libertinus workflow file.

Anyone running Google corp hardware can't touch agpl code, so this can't be used in any GF projects until the license is relaxed to GPLv3

Make sure we use --match=... as necessary to only catch and version based on proper font format tags and ignore any other tag names.

Which normalizer to we use? Have this a feature flag to use one or the other?

See for example the difference between a Libertinus release and the CTAN package.

The biggest thing is the included fonts directory should be filtered to just static/OTFS and flattened.

It should be pretty easy to setup runners for most CI systems, somebody just has no need it.

In no particular order besides the likelihood I might personally run across a need I can think of...

• Gitlab CI
• Source Hut
• Cirrus CI
• Buddy
• Circle CI
• Travis
• Drone CI
• Jenkins
• Azure
• AWS Amplify

Feel free to comment on this issue with a request if you have a use for something else. At the very least I can give you some pointers for getting started.

It should be possible to generate fonts with an easy command line flag to toggle OT features on or off by default. This would make it possible for people with old applications that don't support OpenType at all to use their preferred variants and features.

Hey,

apologies if this seems a trivial question. The project looks very useful and the premise makes sense to me, but as for understanding actual usage I am a bit puzzled.

Looking at the Usage section those commands make sense, but I don't understand how the make command is configured, where files need to be, and what exactly those do? Can one add others oneself to run via fontship?

If there any more documentation on that? Or where in the source code can I find those to dig deeper?

Not all fonts will be able to or want to generate all output formats. Some will want both TTF and OTF, others will have special features that only work in one or the other. Some have variable fonts plus static instances etc. For the sake of getting things running I've been assuming all formats generated for League Spartan, but this needs to be a configurable thing.

Given the dominance of macOS in the font design market, easy access to this tooling on macOS will be a deal-breaker feature. Docker is a pretty simple solution, but some people might have more fun closer to the metal. A Homebrew formula would probably get some milage...

The machinery to generate variable fonts from FontForce sources (SFD) isn't here yet. It's possible to do with the addition of a designspace file and on the fly conversion to UFO (which we need anyway for exchange with other contributors), we just need to wire it up to be usable. The end user shouldn't have to worry about more than clean sources with the right names and layout plus a designspace that maps them.

An existing variable font project drawn and built from SFD would be a good place to start. I can mock one up but if there is an open source project out there already we can convert as a proof of concept that would be spiffy.

Because they belong there.