Packaging about afdko HOT 51 CLOSED

IMO PPA/rpm/brew should not be included in upstream projects. That's best left to their respective packagers.

from afdko.

Sorry, I mean, the /setup.py and /debian/rules files and such, not the packages themselves

from afdko.

versionned releases would be a good start: #44

from afdko.

I'm perfectly happy with the AFDKO being installed by other packages. The problem is that AFDKO has not yet been fixed up so that it can be built by a package. The directory tree here is basically just source code, with major third party additions needed to make it work. Building it currently needs developer tools on Windows and Mac OSX, and a rather manual installation process for the necessary Python modules - see "FDK/FDK Build Notes.txt". I will probably eventually get around to simplifying this, but not anytime this year.

from afdko.

If I get someone to prepare a pull request, would that be welcome, and do
you have any guidance for them?

from afdko.

Yes, it would be welcome. The current instructions for building the AFDKO are in "FDK/FDK Build Notes.txt". One of the issues is that some of the third party modules needed by the FDKO have themselves not yet been fixed up to install correctly with setup.py, but I think those developers would also be happy to have someone else do the work.

from afdko.

To help on the Debian/Ubuntu packaging, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762252 and http://anonscm.debian.org/cgit/pkg-fonts/afdko.git

from afdko.

👍 for brew. Looked into doing this awhile ago but was too big an undertaking with the time I had.

from afdko.

I'm curious if anyone else has worked on this recently?

from afdko.

I was writing to some Adobe folks about CFF CharStrings level checking in the Font Validator ( HinTak/Font-Validator#13 ) and thought of scavenging some of Adobe's existing libraries.

It looks like the type 2 cstr parsing code ( https://github.com/adobe-type-tools/afdko/tree/master/FDK/Tools/Programs/public/lib/source/t2cstr) already does some error checking and is exactly what I need, and perhaps also the autohinter module ( https://github.com/adobe-type-tools/afdko/tree/master/FDK/Tools/Programs/autohint ). These are all C-based;
perhaps some of the checkoutline logic too ( afdko/FDK/Tools/SharedData/FDKScripts/CheckOutlinesUFO.py ) but that seems to be wholy python-based.

Anyway, so I search for existing adaptions of the C part of some of those into a shared library / dll / dylib , which brought me to this issue.

FWIW, the freetype binaries bundled with Font Validator ( https://github.com/HinTak/Font-Validator/tree/master/bin/Darwin , https://github.com/HinTak/Font-Validator/tree/master/bin/Win32 , https://github.com/HinTak/Font-Validator/tree/master/bin/Win64 ) were all built via an rpm-driven cross compiler system on fedora. So I am not only looking for rpm packaging, but also cross-compiling support in afdko.

from afdko.

There is also
https://sourceforge.net/adobe/tintin/code/HEAD/tree/tin/src/
and
http://git.savannah.gnu.org/cgit/freetype/freetype2.git/tree/src/cff

from afdko.

from afdko.

No personal involvement. I just like to give people a more complete picture of their available options, so they're making better-informed decisions.

I know that, as you point out, tintin was “dropped” by Adobe in 2010 and it hasn’t been really used for anything in the community AFAIK. I cannot judge on its usefulness (so I trust your judgment here, esp. on the issue of integrating C vs. C++ with C# — I have zero experience with the C# world).

I'm still to some extent baffled by the fact that tintin was virtually ignored by the opensource community. But I know from contacts with various tool developers that especially implementing CFF parsing sensibly can be a bit burdensome — so I just keep pointing out to tintin because it’s already done.

Yet I think you're right that the code in AFDKO is probably much more tested — since the AFDKO tools have been used to generate 99% of OTFs that are on the market.

from afdko.

With regard to tin (I am not sure why it is being referred to as tintin), I spoke with its author right before or right after he left Adobe, and was told that it is incomplete, because some of the components could not be open-sourced, which is why there are no build files. That would explain why it was largely ignored.

from afdko.

Thanks for the info Ken!

BTW, I think the "tintin" name variation comes from the project's URL on Sourceforge:
https://sourceforge.net/adobe/tintin/
though it is called just "tin" everywhere else. :)

from afdko.

@readroberts and others may welcome this news:
http://social.techcrunch.com/2016/03/30/be-very-afraid-hell-has-frozen-over-bash-is-coming-to-windows-10/

I don't think it's just about bash. The fact that a whole Linux subsystem is coming natively to Windows is great news, I think, and will certainly make cross-platform deployment of many opensource tools easier, not just AFDKO. :)

from afdko.

from afdko.

tin is in a sense rather similar to how the Microsoft Font Validator code base was given to me in August last year, 3 months before it was eventually open-sourced - pieces were missing because they cannot be open-sourced, the build system support files (visual studio project files ) were deemed "not useful", or 'may leaking secrets' and removed. Although I convinced them much later that even not-functional build support files are useful, at least for indicating dependencies and mutual relationships of source files - and it is much easier to 'fill gaps', rather than 'start from scratch' - I kept feeling my time are being wasted, when the Microsoft folks kept telling me that 'those are useless and it is cleaner/better to start from scratch': I may still start from scratch, after seeing what was done, but I like the option of not to. So at the beginning I only had the 2003 binary to look at, to try to figure out which source files contributes to which binary dll.

So I'd probably suggest Adobe put a binary which uses tin out, even if it contains parts which cannot be open-sourced, just to convince and show people what could be done with that code.

from afdko.

Somewhat off-topic, to answer my earlier question: the dev info for Adobe Tin is at:
https://sourceforge.net/adobe/tintin/wiki/Developer%20Documentation/ , and a close-source "demo" for its usage is apparently "Adobe Glyphlet Development Kit (GDK) for SING Gaiji Architecture"
at ( http://www.adobe.com/devnet/opentype/gdk/topic.html ), a 21MB download. The former is mentioned in https://sourceforge.net/adobe/tintin/home/Home/ , and should could really be included as a top-level "README-dev.txt" with the source; the latter could be mentioned a bit more prominently everywhere - both in the Home page, a bit earlier in the dev info, and given more important as "featured" as demo, of what Adobe Tin can do.

from afdko.

@HinTak: About five years ago, when someone asked about a Makefile for Tin, the contributing author wrote the following to me in response: It was never the intent to put out the source code for the entire Tin library. There were parts we could not put into open source. Hence there are no plans for a build file. People should think of it as a source code library.

In other words, while the GDK indeed serves as an example implementation, it includes portions that are encumbered by patents.

And, in case you weren't aware, there was a blog article about the Tin open source project that was published six years ago.

from afdko.

@kenlunde : thanks for the notes, and the pointer to the blog article. I think the relationship with GDK deserves more prominence. A working Makefile wold be nice, but even a non-working Makefile (or visual studio project file, or anything else equivalent like eclipse project file), referring to some not-opened component, tells a story about the mutual relationship of the released source files.

I would suggest say, adding a simple harness demo program which, for example, just read each part of a font file and write a new (potentially identical) one, with plenty of comments/status updates on the way. It seems that there is already the beginning of one - I grep'ed for "main" and found "tin/src/main.cpp". It is rather bare at the moment, and needs fleshing out.

Let me share some of my experience with Microsoft's Font Validator. When the source fell into my hands, all the build support files (vs proj files) were removed, as well as the source code corresponding to about 4-5 dll's. Knowing what is missing (comparing with the released binary) helps, and being certain that the code is useful - that something was built from it - helps.

For the specific purpose I had in mind: a CFF parsing library, with error status when something doesn't work (i.e. it should not assume well-formed fonts), tin vaguely fits the criteria. Beyond that, there are the general consideration of availability of documentations, quality (impression) of code, expectation of it will be maintained and bugs found etc would be fixed, etc. Those are somewhat lacking. Also, as a technology preference, integrating a c++ library with c# (or indeed, most scripting or other languages) means writing many "extern C" wrappers around the C++ part; the error status'es are available as C++ exceptions, which is another obstacle - throwing c++ exceptions across language boundary (whether it is C# or even python) is a challenge.

I hope these notes are helpful to somebody else who may be motivated enough to do something useful with that code.

from afdko.

I should mention that the code base of Microsoft's Font Validator contains some code for writing font files. Obviously Font Validator itself does not make use of that; so my guess is that those are likely work-in-progress, and/or there is an internal or proprietary MS sister project which uses that.

I have no idea about the maturity or purpose of that part of FV code. If people are looking for a C#-based kit for manipulating /re-writing font files, such as a digital signature signing tool, there is no need to go elsewhere.

from afdko.

@twardoch , @readroberts : there are already some test reports of the Ubuntu-on-win10 at
http://www.phoronix.com/scan.php?page=article&item=windows-10-lxcore&num=1
I haven't read it carefully yet, but it seems a good deal more impressive than I gave it credit for earlier - apologies :-).

from afdko.

#136 is a blocker for packaging.

from afdko.

@onlyjob : not really. Those "distributed as is" files should be distributed as is (ghostscript/xpdf/popper do that with the Adobe cmap files too; and the ms validator issue file can be regarded as "frozen" - a capture of issues against specific version of ms validator), those build support files are not very portable and probably should be rewritten or removed. They are blockers only for GPL-fanatics who need the free-modification rights. In fact I think none of the files are essential and should be removed from redistribution anyway. I say, if they bother you, just remove them.

from afdko.

The cmap files are "de facto standards", and there is really no incentive to modify them, for interoperability.

from afdko.

Fair enough, @HinTak. Thanks for your comments. Good to know that those non-free files are not essential. It would be great to have them removed.
FYI this have nothing to do with fanaticism. If you read open source definition you'll see that modification rights (i.e. derived works) are essential.
No permission to modify means non-free. Non-free means non distributable in Debian. Plain and simple.
Also free software (i.e. open source) should not depend on non-free stuff.

from afdko.

@onlyjob : I don't think going on and on about philosophical/legal issues (should/should not) is helping the technical issues we are concerned with - how to, and who is to spend time on it - here. Talking about a "theoretical" modification rights when you are not going to modify ever at all, is a distraction.

When somebody have the packaging support files tested and working, we can revisit the "should/should not" question. I suspect the philosophical issue is irrelevant, since the files you talked about just aren't essential and /or will be replaced as a side effect of solving the packing problem. Instead of "blocker", it will simply disappear and becomes irrelevant when the packaging issue is solved.

As I said, if those files bother you, just remove them.

from afdko.

To me this is very practical: if it is non-free (i.e. not open source) then it doesn't worth spending time on it. It is also important to avoid polluting free software with unnecessary stuff.
@HinTak, even if you don't care much about software freedoms there is no need to talk back to those who think it matters...

from afdko.

@onlyjob @HinTak you both have made your points. Please take the open source discussion elsewhere. I've locked posting on this issue for the time being.

from afdko.

Can anyone point me to a setuptools project that installs a C command-line program?

I' m starting to look at repackaging the afdko so that it an installable package from Github. . This is driven by the increased number of third party modules that the afdko depends on, and a high rate of change in those, as well as in increased rate of change and interest in moving the afdko forwards since it has been posted on GitHub. It is clearly no longer reasonable for me to be releasing the Adobe web site builds once or twice a year.

I was thinking of following the example of Khaled's psautohinter and of fontTools, and building an egg package

Building an installer for the command scripts seems simple. However, the FDK includes C command-line programs, and I haven't been able to quickly find documentation for building and installing those with setuptools. Is anyone aware of an example?

I did find the following comment, which describes an approach which clearly would work:
https://stackoverflow.com/questions/33168482/compiling-installing-c-executable-using-pythons-setuptools-setup-py

but which does not support all of the setuptools workflow.

The afdko would undergo several major changes.

• I would abandon the internal Python interpreter. 10 years ago, there was good reason for this, but not anymore.
• the afdko Python programs will get installed as Python command-line scripts in teh user or system - supplied Python interpreter.
• the AFDKO C programs would go to the same location as the command-line Python

from afdko.

That probably will involve two packages, one for a python module, another for the C program, depending on their mutual relationship (or lacking thereof).

from afdko.

Hey Read, it's good to hear you're working on repackaging the afdko.

I would abandon the internal Python interpreter.

Yes, there's no longer need to do this. Also, the efforts should go to porting the code to py2.py3 so that when the python2.7 expiration date comes in a couple of years, the afdko tools will keep working.

the AFDKO C programs would go to the same location as the command-line Python

I'm not sure about this. If these C programs are exclusively meant to be run as subprocess from python modules/scripts, there is no point in having them available on the PATH. You could either embed them as "package_data" inside the afdko's python packages, and use pkg_resources.resource_filename to retrieve their location so they can be run from subprocess. This would be the easiest solution.

Or a better one would be doing what @khaledhosny did for psautohint, i.e. write extension modules that call into the C libraries, and use setuptools to build them in a platform-independent way. Tools like Cython or CFFI make wrapping C/C++ easier (and more portable) than manually writing C extension module with the CPython API. One advantage of this would be that you could use the existing python packaging infrastructure without hacks like in the stackoverflow thread mentioned above.

building an egg package

Note that eggs are no longer the preferred approach for distributing python binary packages. Wheels are the standard way now. The easy_install module in setuptools that produces/installs them (python setup.py install) is de facto deprecated in favour of pip.

Another thing that would be nice to do is split up the many afdko tools into separate packages that can be installed separately but can also require each other. You could use python's namespace packages to make sure that these different distributions end up under the same namespace, e.g. from afdko import makeotf, autohint.

If you like I may be able to help you doing this in my spare time.

from afdko.

The python2 python3 transition is fairly easy to deal with - python itself gives you the 2to3 tool, as well as the from __future module, for migrating python scripts which are meant to run on both python 2 and python 3.

As for the command-line tool part, I assume Read wants it to work on windows too, which would make it a little bit tricky, whether it is as a subprocess or as a C-based dll loaded via CFFI or ctype, if they are tightly integrated; instead of being loosely related as just being generally "it is all from Adobe". I guess you'll know what to do once you start actually doing it.

from afdko.

yes, both the approaches I recommended above (1. embed the executables as package_data and run them as subprocesses, or 2. write fully-fledged extension modules) will work on Windows as well other platforms.

from afdko.

of course, even for the embedded-executable-as-subprocess method, one has to create separate platform-specific binary packages (wheels), e.g. afdko-makeotf-1.0.0-py2.py3-macos_10_6_intel.whl, afdko-makeotf-1.0.0-py2.py3-manylinux1_x64.whl, afdko-makeotf-1.0.0-py2.py3-win32.whl, etc.

from afdko.

@anthrotype Please take a look at the setup.py file on the afdko 'package' branch, and let me know what you think. This builds a Python FDK package with three commands installed in the Python 'bin' dir. The C command-line programs are called by 'bin' scripts that end up calling a single Python function,, which uses the name of the command to call the actual C command line program. Left to myself, I would continue this template for all the other FDK commands for a first pass. I do like the idea of making all the C command-line programs be a separate sub-packages, and it would also be good to clean up the C command program build process to use the configure/make model, but would do this as a later pass.

from afdko.

That does not sound like it needs python at all. Normal shell script knows its own name and can call exec/execvp to morph into another executable. See the man page of your favorite shell (probably GNU bash...) under the section on "exec".

The thing is, python will end up calling the shell anyway, so you might as well cut python out of the way and do it in shell script direct.

from afdko.

For the moment, I am trying to work within the usual constraints of setuptools. This does not offer a way to install non-Python scripts, but it does offer a mechanism to create an executable Python script of user-defined name, which calls a function in my Python package, which is why I did what I describe above. Once I get the first pass done, I will look at how to install the C programs directly in the Python 'bin' directory, which is in the user's $PATH search list.

from afdko.

Hmm, you are talking about situation on windows, right? Talking about Python's bin directory (which is I believe a windows thing - on unices, python is just in /usr/bin) and configure/make is very confusing...

from afdko.

No, I think the directory to which I am referring is always in the Python directory tree: on Windows, it is "C:\Python27\Scripts", on the Mac it is "/Library/Frameworks/Python.framework/Versions/2.7/bin"

from afdko.

from afdko.

I see you are correct. I thought that setuptools.py added Python.framework/Versions/2.7/bin to the users PATH, as I see this in my .profile file. However, I see that with the afdko setup.py , this does not happen, nor is a symbolic link made in /usr/bin. Does anyone know if there is a setup switch to do this, or is custom code needed? At least one of the Python packages I installed in the last two week does do this.

from afdko.

from afdko.

That is. in fact, what I have done, and it is working in the current afdko setup.py, for the three test commands I put in there. I asked in general python help about adding the Python executable directory to PATH command, or sym links in /user/local/bin, and was told that setup.py doesn't do that, so I assume it is custom code.outside of setuptools.

from afdko.

I thought that setuptools.py added Python.framework/Versions/2.7/bin to the users PATH, as I see this in my .profile file

That was placed there by the Python.org installer itself. It's not the responsibility of setuptools, nor of any other specific python package, to configure the system's environment (e.g. extending the $PATH). That should be left to the OS vendor, or to the users when they install a particular python distribution.

Please take a look at the setup.py file on the afdko 'package' branch, and let me know what you think.

Sure, I'll take a look tomorrow, thanks Read.

from afdko.

I agree with @anthrotype that the best way forward is to convert the command line binaries into extensions and call these from the Python code. This simplifies the packaging greatly, and also allows building the C code using Python infrastructure (so no need for separate makefiles, XCode and Visual Studio projects). It can be even faster if the Python code is executing the binary repeatedly (in psauthonint there was a measurable difference when switching from executing the binary to calling the extension).

I prefer writing the extensions in plain C(++) as it is easier for me to resonate with the CPython API and the C libraries I need to call, rather than warping my head around Cython conventions, or having to rewrite the headers for CFFI (and it also requires building shared libraries and dealing with how to install them), but your mileage may vary.

from afdko.

Writing C extension for AFDKO is not that complex actually, one starts with the current C program, renames main() function and replaces its command line handling with PyArg_ParseTuple etc. and wraps it in a Python module.

from afdko.

Fair enough. The AFDKO actually started out with several of the command-line programs wrapped as Python extensions. I stopped doing that because a) the minor Python versions kept changing, and the extensions had to be recompiled with each change, b) FontLab had its own internal Python with different version numbers, and c) I had to do all the compiling and building, as the afdko wasn't opensource. Now that the afdko can be hosted on github and the build process can be automated, I think it makes sense to go this way for the AFDKO. I'd still like to see the build directories for the C command-line programs maintained, even if not used by the setuptools build/install, as it is useful to be able to send copies of 'tx', 'spot', and some of the others to systems where there is either no Python or no AFDKO installed, as well as easier to debug.

from afdko.

This can be closed, no?

from afdko.

Yes

from afdko.