Eventually move to a system where some common glyph images are downloaded as a "base font glyph file", and only include uncommon glyph images in the vector tiles.

@kkaefer curious what you have in mind here. I at one point envisioned a "glyph tile" system where you'd download/cache whole chunks of unicode ranges as you ran into them

http://jrgraphix.net/research/unicode_blocks.php

This would separate the tiles fully from font glyphs, but not sure what other considerations should be involved here.

The Node.js tests should move out with the Node.js bindings. Need to add some unit tests (using Catch, Boost.Test or something else?) to actually test the core C++ library.

Per chat discussion, we'll want to:

fontserver

• Split out a separate .proto file/format (what to call this? glyphrange.proto?)
• Probably nuke any of the vector tile-based augmented handling in fontserver entirely and nuke the tile-based shape call for now
• Add parser/reader to https://github.com/mapbox/fontserver/tree/glyphsproto/test/format

llmr

• Add parser/reader to https://github.com/mapbox/llmr/tree/glyphrange/js/format
• Update integrating code

Because @DennisOSRM said so.
https://github.com/DennisOSRM/Project-OSRM/blob/master/Util/StringUtil.h

From @springmeyer:

It would be able to be installed into /usr/local/, for example, and would have a fontnik-config or a pkg-config entry so apps could find it.

So one somewhat missing piece is how gyp would be used. Currently gyp is largely used when C++ code is vendored into another code base but it should work both ways, you should be able to have a set of gyp files that both allow for:

• you to install a C++ lib into a given --prefix, and
• bundle/vendor it into a larger app and just point to its .gyp file

If you strip node-gyp out you lose things that can be quickly added back. I think the best way to do it is a Python configure/install wrapper. We basically need cpp-gyp, written in Python, that works on Windows/Mac/Linux, and is able to take args like --prefix, --debug, --enable-static, --enable-shared, etc. Python dependency is already there due to gyp being Python.

We discussed in chat that maybe defaulting to Open Sans Regular might not be the most logical — should we change this to Arial Unicode MS Regular?

@mikemorris if this makes sense to you I'm happy to grab

https://github.com/mapbox/node-fontnik/blob/master/vendor/fontnik/src/glyphs.cpp#L151

This could be more memory efficient if we used a set. Something to do for the next release.

@springmeyer would love to get your feedback on this —

node-fontnik will be used in the font uploading API, but @mikemorris and I talked about how we could totally refactor this to no longer use the Mapnik font registry concept and have little name2face maps floating around, but rather pull the streaming logic out so that we could do something like

fontnik.load(fontfile, function(faces) {
  // faces returns an array per face from the file of objects that look like
  // {  face_name: "",
  //    family_name: "",
  //    style_name: "",
  //    codepoints: [],
  //    range: function
  // }
})

(where the FT library/stream is scoped to the callback so that the range function can be repeatedly called inside the callback). And eliminate much of the node-fontnik API.

But, given that the registry actually seems (if I understand correctly) mostly harmless — just a name-to-face map — is it worth doing this? (Or is it harmless?)

The overhaul branch no longer uses C++ wrappers around Freetype objects. @lbud one side effect of this is that we lost the RAII benefit of those classes.

To solve this I propose we use scope guards (http://en.wikibooks.org/wiki/More_C++_Idioms/Scope_Guard). Scope gaurds make sense because in our used of FT objects is now scoped purely to the async methods. Scope guards will help use ensure that the Freetype objects get destructed the same no matter if the function throws an error or returns early.

Here are the current leaks:

• FT_Library is leaked here
• FT_Glyph is leaked here and here and here

For reference I found the memory leaks by:

• booting a linux VM (since OS X does not support valgrind and iprofiler -leaks is not as good as valgrind)
• built overhaul branch of node-fontnik in debug mode
• ran valgrind --leak-check=full node ./node_modules/.bin/_mocha test/
• paid attention to the sections that originated from node-fontnik function calls and said blocks are definitely lost and ignored all others

Valgrind results that showed the memory leaks were:

==5575== 320 bytes in 4 blocks are definitely lost in loss record 230 of 309
==5575==    at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==5575==    by 0x7B5CC6F: ft_mem_qalloc (in /usr/lib/x86_64-linux-gnu/libfreetype.so.6.11.1)
==5575==    by 0x7B5CCB7: ft_mem_alloc (in /usr/lib/x86_64-linux-gnu/libfreetype.so.6.11.1)
==5575==    by 0x7B63177: ??? (in /usr/lib/x86_64-linux-gnu/libfreetype.so.6.11.1)
==5575==    by 0x7B634CE: FT_Get_Glyph (in /usr/lib/x86_64-linux-gnu/libfreetype.so.6.11.1)
==5575==    by 0x78C7FC5: node_fontnik::RenderSDF(node_fontnik::glyph_info&, int, int, float, FT_FaceRec_*) (glyphs.cpp:522)
==5575==    by 0x78C69B7: node_fontnik::RangeAsync(uv_work_s*) (glyphs.cpp:290)
==5575==    by 0x9E0E2F: worker (threadpool.c:74)
==5575==    by 0x9D6640: uv__thread_start (uv-common.c:323)
==5575==    by 0x5A6B181: start_thread (pthread_create.c:312)
==5575==    by 0x5D7BEFC: clone (clone.S:111)

And

==5575== 2,128 (128 direct, 2,000 indirect) bytes in 1 blocks are definitely lost in loss record 273 of 309
==5575==    at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==5575==    by 0x7B5CC6F: ft_mem_qalloc (in /usr/lib/x86_64-linux-gnu/libfreetype.so.6.11.1)
==5575==    by 0x7B5CCB7: ft_mem_alloc (in /usr/lib/x86_64-linux-gnu/libfreetype.so.6.11.1)
==5575==    by 0x7B61431: FT_Add_Module (in /usr/lib/x86_64-linux-gnu/libfreetype.so.6.11.1)
==5575==    by 0x7B5856B: FT_Add_Default_Modules (in /usr/lib/x86_64-linux-gnu/libfreetype.so.6.11.1)
==5575==    by 0x7B585C8: FT_Init_FreeType (in /usr/lib/x86_64-linux-gnu/libfreetype.so.6.11.1)
==5575==    by 0x78C5D40: node_fontnik::LoadAsync(uv_work_s*) (glyphs.cpp:175)
==5575==    by 0x9E0E2F: worker (threadpool.c:74)
==5575==    by 0x9D6640: uv__thread_start (uv-common.c:323)
==5575==    by 0x5A6B181: start_thread (pthread_create.c:312)
==5575==    by 0x5D7BEFC: clone (clone.S:111)

/lib should have get the rm -rf most likely.

We should port the build/packaging system to use Mason for handling deps.

What this will unlock:

• Easily pull in the latest versions of freetype, boost, and protobuf and have them be the same across osx + linux
• No more source compiles needed like are currently done in https://github.com/mapbox/node-fontnik/tree/master/deps
• being able to have the travis build use sudo:false for faster builds

Steps to port things:

• delete deps/build_boost.sh, deps/build_freetype.sh and deps/build_protobuf.sh
• inside deps/travis_build.sh:
  • install mason
  • replace the use of ubuntu-toolchain-r ppa with mason-provided C++11 toolchain: https://github.com/mapbox/mason/blob/master/scripts/setup_cpp11_toolchain.sh (so use clang++ instead of g++)
  • use mason to install and link all deps
  • set environment settings to point to linked deps dir ./mason_packages/.link like export CXXFLAGS="$(pwd)/mason_packages/.link/include" and export LDFLAGS="$(pwd)/mason_packages/.link/lib" and export PKG_CONFIG_PATH="$(pwd)/mason_packages/.link/lib/pkgconfig"
• back in .travis.yml do source ./deps/travis_build.sh to run the script AND inherit the CXXFLAGS and LDFLAGS at https://github.com/mapbox/node-fontnik/blob/master/.travis.yml#L22
• try building the app and if it works confirm with otool -L (for OS X) and ldd (for linux) that lib/fontnik.node is not linking to any system libprotobuf, or libfreetype. If not dynamic linking is present them it means we've successfully statically linked to the mason-provided .a files in ./mason_packages/.link/lib/

Travis built binaries for 0.1.2 so you should be good to publish!

• When does this module go open source?
• Does it get renamed/enter the npm registry?

TL;DR

• good news is that there are definitely commonly used chinese characters that we can bundle together into special glyph PBFs.
• bad news is that this amounts to about ~3000-4000 characters which will amount to a ~2-3MB overhead and it's not likely we can reduce this overhead much.
• other good news is this approach will likely work for hangul (korean) which has similar size/range waste issues though not as bad as cjk.

Notes from weekend analysis (source committed here https://github.com/mapbox/fontserver/tree/char-spec/spec).

Background

• There are a lot of chinese characters -- "Chinese characters are theoretically an open set" -- and a modern dictionary puts the number in the 100,000+.
• The problem we're dealing with is pretty well discussed especially in relation to learning chinese. Most discussions put the "99%" use character count at around 3000-4000 chars (e.g. http://www.commonchinesecharacters.com/Lists/MostCommon2500ChineseCharacters).

Prep

• Sample of text that we need to render. In spec/fixtures/ I've added 12 z14 vector tiles of major chinese cities.
• 4096 most frequent chinese characters, in all name tags from OSM. This is dumped to cjk-osm.json.
• 4096 most frequent chinese characters, from an academic analysis of modern chinese character frequency (http://lingua.mtsu.edu/chinese-computing/statistics/). This is dumped to cjk-modern.json.

Approach

Assuming a character that falls into the 4096 most freq character use can be grabbed from a set of CJK common PBFs or so, we can analyze the range count for our 12 vector tiles:

none (78 ranges)
[ '0-255',
  '19968-20223',
  '20224-20479',
  '20736-20991',
  '20992-21247',
  '21248-21503',
  '21504-21759',
  '21760-22015',
  '22016-22271',
  '22272-22527',
  '22528-22783',
  '22784-23039',
  '23040-23295',
  '23296-23551',
  '23552-23807',
  '23808-24063',
  '24064-24319',
  '24320-24575',
  '24576-24831',
  '24832-25087',
  '25088-25343',
  '25344-25599',
  '256-511',
  '25600-25855',
  '25856-26111',
  '26112-26367',
  '26368-26623',
  '26624-26879',
  '26880-27135',
  '27136-27391',
  '27392-27647',
  '27648-27903',
  '27904-28159',
  '28160-28415',
  '28416-28671',
  '28672-28927',
  '28928-29183',
  '29184-29439',
  '29440-29695',
  '29696-29951',
  '29952-30207',
  '30208-30463',
  '30464-30719',
  '30720-30975',
  '30976-31231',
  '31232-31487',
  '31488-31743',
  '31744-31999',
  '32000-32255',
  '32256-32511',
  '32512-32767',
  '32768-33023',
  '33280-33535',
  '33536-33791',
  '33792-34047',
  '34048-34303',
  '34304-34559',
  '34560-34815',
  '34816-35071',
  '35072-35327',
  '35584-35839',
  '35840-36095',
  '36096-36351',
  '36608-36863',
  '36864-37119',
  '37120-37375',
  '37888-38143',
  '38144-38399',
  '38400-38655',
  '38656-38911',
  '38912-39167',
  '39168-39423',
  '39424-39679',
  '39936-40191',
  '40448-40703',
  '40704-40959',
  '8192-8447',
  '8448-8703' ]

osm (26 ranges)
[ '0-255',
  '20224-20479',
  '20992-21247',
  '256-511',
  '28416-28671',
  '29440-29695',
  '29952-30207',
  '33536-33791',
  '38912-39167',
  '8192-8447',
  '8448-8703',
  'cjk-common-0',
  'cjk-common-1',
  'cjk-common-10',
  'cjk-common-11',
  'cjk-common-12',
  'cjk-common-13',
  'cjk-common-14',
  'cjk-common-2',
  'cjk-common-3',
  'cjk-common-4',
  'cjk-common-5',
  'cjk-common-6',
  'cjk-common-7',
  'cjk-common-8',
  'cjk-common-9' ]

modern (34 ranges)
[ '0-255',
  '22272-22527',
  '256-511',
  '26880-27135',
  '27136-27391',
  '27648-27903',
  '27904-28159',
  '28416-28671',
  '29184-29439',
  '29952-30207',
  '30976-31231',
  '32512-32767',
  '33280-33535',
  '33536-33791',
  '34304-34559',
  '39424-39679',
  '8192-8447',
  '8448-8703',
  'cjk-common-0',
  'cjk-common-1',
  'cjk-common-10',
  'cjk-common-11',
  'cjk-common-12',
  'cjk-common-13',
  'cjk-common-14',
  'cjk-common-15',
  'cjk-common-2',
  'cjk-common-3',
  'cjk-common-4',
  'cjk-common-5',
  'cjk-common-6',
  'cjk-common-7',
  'cjk-common-8',
  'cjk-common-9' ]

The scripts assume we split up the cjk-common PBF into chunks of 256. It looks like you will want most if not all of these 4096 characters as a baseline, always, all the time (if you run the script with 1 or 2 tiles alone you will often end up with 12-16 of the common ranges). You basically end up grabbing the common characters no matter what.

So combining the common glyphs into a single pack and eliminating the non cjk ranges from the list to reduce noise, we're down to:

osm (8 ranges)
[ '20224-20479',
  '20992-21247',
  '28416-28671',
  '29440-29695',
  '29952-30207',
  '33536-33791',
  '38912-39167',
  'cjk-common' ]

modern (15 ranges)
[ '22272-22527',
  '26880-27135',
  '27136-27391',
  '27648-27903',
  '27904-28159',
  '28416-28671',
  '29184-29439',
  '29952-30207',
  '30976-31231',
  '32512-32767',
  '33280-33535',
  '33536-33791',
  '34304-34559',
  '39424-39679',
  'cjk-common' ]

Conclusion + questions

Overall having a cjk-common glyph that takes precedence if a character falls into a fixed list of n (I picked 4096 based on a few runs -- 3000 not enough, 5000 has diminishing returns) looks like a good approach. You'll have most commonly used characters loaded and cached and will fire off requests as normal for other ranges as you hit less common characters.

The discrepancy between OSM's top characters and other analysis' top characters is worth some study though. Note that once we set this list changing it is very painful. It will affect any implementations using these endpoints and likely means we need to spec and version any endpoints around this strictly.

• "Place language" differs from normal usage significantly in english, german, etc. This is likely (?) the case for Chinese as well and could lead to this discrepancy.
• OSM data quality in China is an unknown to me. Setting a list of character frequency list based on the current state of OSM could be a bad idea without some overall confidence in current OSM data in China being representative of Chinese-language maps as a whole and future OSM data/map data.

Next actions

• Similar analysis for Hangul
• Branch of fontserver that can generate a PBF from a charlist rather than a start/end range,
• Branch of llmr that uses a cjk charlist to request the common CJK PBF and fallback to normal range glyphs otherwise,
• Test IRL
• Consult/further research on chinese char freq questions

cc @mikemorris @kkaefer @ansis @nickidlugash

Been passed around several places at this point. Going to stick with it here.

To deploy the font server in a high load environment, we should do these things:

• Cache rendered glyph images of a font in memory. These should be in the low tens of thousands range and should take about a kilobyte each, so we're looking at 20-50MB. The reason for this is that glyph images are expensive to create, but don't change at all once they're rendered. See https://github.com/mapbox/fontserver/blob/master/src/tile.cpp#L633-L640 for some demo code. This also needs support in the renderer, so maybe not an immediate task. (internal memory cache and http-level caching of glyph ranges should be sufficient)
• Don't store/cache the generated tiles, at least not without a strategy of what we're going to do in disk full scenarios.
• Remove the .simplify() calls. They're a hack in the font server to get rid of some of the duplicate geometries, but it's nothing that we should use in production. Rather, we should use them as is. E.g. https://github.com/mapbox/fontserver/blob/master/src/tile.cpp#L251-L276 unions all water polygons.
• Eventually move to a system where some common glyph images are downloaded as a "base font glyph file", and only include uncommon glyph images in the vector tiles. #17 #23
• Try to modify the existing protobuf object rather than copying over everything (vector pbfs are no longer augmented with switch to glyph ranges)
• Remove unused portions of the node module, like the Font object(?)
• Remove Clipper, don't unpack the entire geometry only to repack it unchanged. Again, geometry stuff should be handled upstream.
• Find a way to specify what keys we want to get shaping information for. Right now, we only shape name: https://github.com/mapbox/fontserver/blob/master/src/tile.cpp#L539 (not relevant with switch to glyph ranges and client-side shaping)
• Check for memory leaks of the node extension/library
• Properly pass pango/freetype errors back to node so we can log them for future reference/debugging

Just capturing chat exchange:

I've assumed that some kind of dynamic request-time parameters would be necessary for an editor to work effectively down the road but I can imagine there are reasons this could be a not good idea.

Implementing this atm would be fairly trivial and enable custom fonts for different styles/demos.

Bad idea or worst idea?

Looking forward to your notes.

@kkaefer - would it be possible to add some basic mocha tests?

Port to NAN to develop against both Node v0.10.x and v0.12.x. Some potentially helpful porting details at mapnik/node-mapnik#256.

• 0.8
• 0.10
• 0.11

I've been trying to swap fonts in a map I've created inspired by the mapbox-gl example posted on the Mapbox blog:

https://www.mapbox.com/blog/video-in-mapbox-gl/

I tried swapping out the 0-255.pbf file I generated with node-fontnik (this goes smoothly) and even the range.0.256.pbf file included in the test folder. Although swapping this file work when I use other 0-255.pbf files that Mapbos hosts, I see a Uncaught Error: Unimplemented type: 7 thrown from Protobuf.prototype.skip with either of these?

Should this work? Is there more to the generating a new font for this purpose?

We can get better compression ratios by reducing the granularity of the SDFs -- the following commit rounds each value to the nearest multiple of 8:

c213c12

Visually it's hard to see much difference once rendered and here is the space saving for a 0-255 range:

master 43124 bytes
compress-more 23208 bytes

This is a nice boost we can get now roughly for free since it doesn't affect the format/integrating projects -- later we may try other encodings for the bitmap (PNG, etc.) but those will be format-breaking.

Next actions

• Try this and other settings (4 and 2? 16 is where I started to see artifacts...) with a variety of different fonts
• This could be variable/env based so if you want a very high quality SDF you can set your settings that way

If the server added an indication of where breaking is allowed, and the direction of the text, could we do the rest on the client?

Line breaking implementation in Mapnik: https://github.com/mapnik/mapnik/blob/master/src/text/layout.cpp#L104-201

@kkaefer @ansis

Working on refactoring & deploying this right now: it looks like

• The index.js in this repo actually doesn't get run
• It requires gl checked out into an html directory as a subfolder?
• But when you do that, it redirects to /gl/debug and is blank
• The llmr repository includes this but does not include the server at all and instead only uses the Tile class
• If it only uses the Tile class is that all this should expose?

Would love some background on what's going on here: I will continue to hack around, but some things are pretty confusing in the midst of all of this.

• Do we actually run the fontserver codebase, like run the server in here? Or is it just included by llmr?

How big would the metrics for an entire fonstack be? (everything but the bitmaps)

Text shaping (and then label placement) is blocked by getting the metrics. I'm wondering if it is feasible to ship all the metrics at once, and only download the sdfs in ranges

My current goal:

• Get fontserver working as a library
• Remove fontserver as dep of llmr (have llmr become client-side only)

It would be nice to include metadata about how the sdfs are rendered.

• font size used in the sdf (currently 24)
• distance units per pixel (currently 256 / 8 = 32)
• shift (currently -2 * 32 = -64)
• sdf granularity

Not urgent, but it would be nice to not hardcode these in the renderers.

Explanation of how this metadata is used at mapbox/mapbox-gl-style-spec#66 (comment)

Refs #51 and #38 (comment)

Tracking in configurable branch, currently stuck at https://github.com/mapbox/node-fontnik/blob/ed5cc2727bb4014211c917e7e07385389c834891/src/fontnik/glyphs.cpp#L38-40

../src/fontnik/glyphs.cpp:38:40: error: no matching member function for call
      to 'get'
    const std::string fontstack(params.get(ParamKey::Fontstack));
                                ~~~~~~~^~~
../include/fontnik/params.hpp:55:21: note: candidate template ignored:
      couldn't infer template argument 'T'
    inline const T *get(ParamKey key) const {
                    ^
../src/fontnik/glyphs.cpp:39:36: error: no matching member function for call
      to 'get'
    const std::string range(params.get(ParamKey::Range));
                            ~~~~~~~^~~
../include/fontnik/params.hpp:55:21: note: candidate template ignored:
      couldn't infer template argument 'T'
    inline const T *get(ParamKey key) const {
                    ^
../src/fontnik/glyphs.cpp:40:46: error: no matching member function for call
      to 'get'
    const std::vector<uint32_t> chars(params.get(ParamKey::Chars));
                                      ~~~~~~~^~~
../include/fontnik/params.hpp:55:21: note: candidate template ignored:
      couldn't infer template argument 'T'
    inline const T *get(ParamKey key) const {
                    ^

Just capturing the fact for now. Not sure what triggers atm, will update as I discover more.

Generate patch files to more easily apply changes and update third party libs. Maybe move all third party lib files to a vendor directory as part of this, instead of splitting components into include and src at top level.

https://github.com/mapbox/node-fontnik/blob/master/include/mapnik/README.fontserver
https://github.com/mapbox/node-fontnik/blob/master/include/node_mapnik/README.fontserver

@springmeyer Should mapnik patch work forwards from the whole repo (delete most files in patch) or just rely on copying over a small subset of files and then apply changes?

I will grab this tomorrow, once sourceforge fixes its ssl and my rage subsides.

@mikemorris could you use this library in its existing state to ask a font for which character ranges it covers?

On server the fonts.conf file does not work as is and leads to:

(process:16528): Pango-WARNING **: failed to choose a font, expect ugly output. engine-type='PangoRenderFc', script='latin'
(process:16528): GLib-GObject-CRITICAL **: g_object_ref: assertion `G_IS_OBJECT (object)' failed
(process:16528): Pango-CRITICAL **: pango_fc_font_lock_face: assertion `PANGO_IS_FC_FONT (font)' failed

Using absolute paths works, however.

TODO

• Is there another way to configure these?
• Should node write this file by hand? (gross)

1. start server
2. request a single tile
3. stuck

It seems to be getting stuck at renderer = pango_sdf_get_renderer(); https://github.com/mapbox/fontserver/blob/master/src/globals.cpp#L52

Trying to set this up on Ubuntu 13.10 for llmr. Running into this build error:

> [email protected] install /home/aj/Code/fontserver
> node-gyp rebuild

make: Entering directory `/home/aj/Code/fontserver/build'
  CXX(target) Release/obj.target/fontserver/src/fontserver.o
g++: error: unrecognized command line option ‘-stdlib=libc++’
make: *** [Release/obj.target/fontserver/src/fontserver.o] Error 1
make: Leaving directory `/home/aj/Code/fontserver/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/home/aj/.nvm/v0.10.21/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:267:23)
gyp ERR! stack     at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:789:12)
gyp ERR! System Linux 3.11.0-14-generic
gyp ERR! command "node" "/home/aj/.nvm/v0.10.21/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/aj/Code/fontserver
gyp ERR! node -v v0.10.21
gyp ERR! node-gyp -v v0.10.10
gyp ERR! not ok 
npm ERR! weird error 1
npm ERR! not ok code 0

I've installed libc++ and libc++-dev but it looks like the problem is g++ doesn't support that flag at all. I have clang available - anyone know how I can force the build to use that?

Currently test/format is copied verbatim from llmr for parsing/confirming the output of fontserver. Once there is no circular dependency between these two and/or the format portion of llmr is split out nuke the local version and add as a devDependency.

Get going on this for harfbuzz branch.

@mikemorris

• Be able to package glyph PBFs independently of vector tiles, basically if someone asks for character range <from>-<to>, hand them back the PBF for those characters. Yes, we know this won't handle shaping.
• Hook this up to some HTTP endpoint.
• See how having llmr retrieve glyph PBFs independently of vector tiles behaves/performs.
• This is either a massive fail or promising and we can look into how to approach shaping for character sets that need it.

Background/goal/interest: being able to have a GL spec that is not tied to a running server. e.g. once glyph PBFs are generated for a given fontset they could be distributed/hosted on S3 or any webserver or locally for demos and when explaining how the whole system works to other devs.

@lbud and I just noticed that running:

./node_modules/.bin/node-pre-gyp --debug rebuild --loglevel=verbose

Seemed to set the correct compiler flags for a debug build (-O0 and -DDEBUG) however when debugging the actual backtrace within ~/Library/Logs/DiagnosticReports/ the backtrace did not show much detail. So something is not quite right: Need to pull over the logic from node-mapnik for debug builds to try to get better backtraces working.

All we are seeing right now is:

Thread 7 Crashed:
0   fontnik.node                    0x0000000102a072dd node_fontnik::LoadAsync(uv_work_s*) + 3869
1   node                            0x0000000100447786 worker + 90
2   node                            0x000000010043dd9a uv__thread_start + 25
3   libsystem_pthread.dylib         0x00007fff8f136268 _pthread_body + 131
4   libsystem_pthread.dylib         0x00007fff8f1361e5 _pthread_start + 176
5   libsystem_pthread.dylib         0x00007fff8f13441d thread_start + 13

Upon loading a font, try to find a cached version of the rendered glyph set so that we don't have to do this on every server restart.

Looks cool tho : )

The Specials Unicode range actually ends at 0xFFFF or 65535, NOT 65533. The last gapless block should be from 65280 to 65535, the same size as previous blocks.

• https://github.com/mapbox/node-fontnik/blob/master/API.md#getrangestart-number-end-number-arraynumber
• https://github.com/mapbox/node-fontnik/blob/master/bin/build-glyphs#L27-L33
• https://github.com/mapbox/node-fontnik/blob/master/index.js#L27-L30
• https://github.com/mapbox/node-fontnik/blob/master/test/test.js#L134-L153

Refs mapbox/mapbox-gl-native#931

When starting llmr I'm seeing a crash in fontserver (happens upon first load of the llmr html page). Perhaps a build issue, but curious if you've seen this before @kkaefer ?

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libprotobuf.8.dylib             0x00000001024e6357 google::protobuf::internal::ExtensionSet::Extension::Clear() + 9
1   libprotobuf.8.dylib             0x00000001024e87d5 google::protobuf::internal::ExtensionSet::Clear() + 29
2   fontserver.node                 0x000000010221a302 llmr::vector::tile::Clear() + 18 (repeated_field.h:1085)
3   libprotobuf.8.dylib             0x00000001024ed3cd google::protobuf::MessageLite::ParseFromArray(void const*, int) + 163
4   fontserver.node                 0x0000000102202ab9 Tile::Tile(char const*, unsigned long) + 71 (tile.cpp:57)
5   fontserver.node                 0x000000010220234c Tile::New(v8::Arguments const&) + 190 (v8.h:4297)
6   node                            0x00000001001703f4 v8::internal::Builtin_HandleApiCallConstruct(v8::internal::(anonymous namespace)::BuiltinArguments<(v8::internal::BuiltinExtraArguments)1>, v8::internal::Isolate*) + 548

Hi,
On @yhahn advice in mapbox-gl-js (mapbox/mapbox-gl-js#809) I am trying to extract all the pdfs for the Open Sans font for every charset for every font-face.
His answer is to run the build-glyphs script from node-fontnik.

I installed node-fontnik on a debian and ran the example

build-glyphs "OpenSans-Regular, Arial Unicode MS Regular" glyphs/
I get the following error:
Error: Failed to find face 'OpenSans-Regular' in font set 'Open Sans Regular, Arial Unicode MS Regular'

When I look into index.js, I can see an optionnal "FONTNIK_FONTS" env var that I tried to set to the /fonts dir with no success.

Should this script work out-of-the-box or does it need special configuration?

Thanks a lot!
Fabien

Would be useful to get an API.md written, especially since part of this module is behind the C++ wall

For further review/test/fixing.

× "Komika Hand, Proxima Nova Bold" => Komika Hand Bold
× "Proxima Nova Bold, Arial Unicode MS" => Arial Unicode MS
✔ "Proxima Nova, Arial Unicode MS"
✔ "Proxima Nova Bold"

After a basic brew install boost: https://gist.github.com/tmcw/bf6a2bc9a1097dceeaf8

I'll start a branch off glyphrange that adds tests with error expectations. Here are some things I'm thinking about...

• convert + range -- If the fontstack you specify includes a font that doesn't match any that are registered should it error out? I'm thinking yes.
• range -- test basic input sanity, e.g. start <= end, start is a number >= 0, end is a number >= 0, neither exceed unicode range (????)
• fixed ranges -- do we enforce these at the fontserver level or outside this API https://github.com/mapbox/llmr/blob/glyphrange/js/worker/workertile.js#L145-234