macports / upt-macports Goto Github PK

@Korusuke it would be good to provide some more documentation and/or einformation in the README. I think it's probably enough to add this to the README, but if you feel like an independent documentation file is more appropriate feel free to do so.

So the discussion about this also took place during the proposal period but we were not able to get to a conclusion.

According to the above stats we can see there are inconsistencies in the naming convention of python ports.

Whereas for Perl and Ruby the naming convention is pretty much consistent as the upstream is the same for every port.
This helps in easily automating the packaging process and also provide ease of use.

but in Python, as the upstream are different naming inconsistencies can be seen, it would be nice to follow a naming scheme from now on and it would help in long term maintainability of ports.

Following the pattern of ruby, perl ports we can have the following naming convention:

Upstream Package name: Foo
port name: py-Foo (capitalization should be maintained)
port folder name: py-foo

I would like to get the view of the community on this and if there is any other convention that we should follow instead?

The json file from PyPI contains a requires_python field that upstream can use to specify support Python versions. If this is present (not sure whether upt or upt-pypi currently parses it), we should compare our "default" versions with this specification and only populate python.versions with the ones that are supported.

If the archive is not a .tar.gz file, but uses a different compression algorithm one should set the proper keyword in the Portfile (see here for the options).

It appears that this information is available from upt, probably from upt.upt_pkg.archives. It can be extracted from the _filename attribute or perhaps better archive_type, even though I am not sure that this is set correctly (doesn't appear to be the case for py-algopy). Examples of ports where this is the case are py-algopy for the pypi frontend and p5-term-gnuplot for the cpan frontend.

The py-upt-macports port is currently generating defunct packages, including #11 and printing empty licence name (macports/macports-ports#4489).

Can you please update the Portfile?

@rajdeepbharati

I apologize that I missed this in PR #2 .... but the field name is incorrect licenses is incorrect, it should be license instead.

port lint --nitpick <port> would have warned about this, but that requires to save the generated Portfile in the MacPorts tree and run this command, which I didn't do back then.

It would be nice to use "coverage" in the tests. Upt does it (see https://framagit.org/upt/upt/commit/3dfc1b4c7b3a163212754538b973503263759b6c) and it helps detect which parts of the code are never executed when running the tests.

So after trying a bunch of things like diff-match-patch, normal diff and vaguely trying regex following is the technique that works best at least for situations that I tried.

Let's say we have a Portfile which we need to update and we have decided that we will update version and checksums field.
So first we extract that fields from existing Portfile (yet to be automated).
and make a file which looks like this:

version             0.7

checksums           sha256  7de9a0cc9d55c175b611ed748ee7ea572ec308411934769e2e647d1d26ba2eea \
                    rmd160  6dc1b42ac5760f8225c81b1cd9539332b33b1b4h \
                    size    24223

and then we generate a new file which looks like this:

version             0.9

checksums           sha256  b967cda25e8e5a634a450886e98ebeaf5aa51220e8888a17a34d78d001f96035 \
                    rmd160  aac9b9c2f52ac75eec0700ac996a87f0151923f9 \
                    size    25969

Notice the blank line at end of file, that is apparently important for this to work.

Now we generate a diff for above two files and then apply patch on the original file.
This even updates a file that maybe modified and has different order of fields than what upt usually has.

For example from this:

# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4

PortSystem          1.0
PortGroup           python 1.0

name                py-upt
version             0.7
revision            0

maintainers         {@korusuke somaiya.edu:karan.sheth} openmaintainer
description         Package software from any package manager to any distribution
long_description    ${description}

platforms           darwin
supported_archs     noarch

homepage            https://framagit.org/upt/upt
master_sites        pypi:[string index ${python.rootname} 0]/${python.rootname}
distname            ${python.rootname}-${version}

license             BSD
checksums           sha256  7de9a0cc9d55c175b611ed748ee7ea572ec308411934769e2e647d1d26ba2eea \
                    rmd160  6dc1b42ac5760f8225c81b1cd9539332b33b1b4h \
                    size    24223

python.versions     37

if {${name} ne ${subport}} {
    depends_lib-append \
                    port:py${python.version}-spdx-lookup \
                    port:py${python.version}-setuptools

    depends_run-append \
                    port:py${python.version}-upt-macports \
                    port:py${python.version}-upt-cpan \
                    port:py${python.version}-upt-pypi

    test.run        yes
    test.cmd        ${python.bin} -m unittest
    test.target
    test.env        PYTHONPATH=${worksrcpath}/build/lib

    post-destroot {
        set docdir ${prefix}/share/doc/${subport}
        xinstall -d ${destroot}${docdir}
        xinstall -m 0644 -W ${worksrcpath} README.md LICENSE CHANGELOG \
            ${destroot}${docdir}
    }

    livecheck.type  none
}

to this:

# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4

PortSystem          1.0
PortGroup           python 1.0

name                py-upt
version             0.9
revision            0

maintainers         {@korusuke somaiya.edu:karan.sheth} openmaintainer
description         Package software from any package manager to any distribution
long_description    ${description}

platforms           darwin
supported_archs     noarch

homepage            https://framagit.org/upt/upt
master_sites        pypi:[string index ${python.rootname} 0]/${python.rootname}
distname            ${python.rootname}-${version}

license             BSD
checksums           sha256  b967cda25e8e5a634a450886e98ebeaf5aa51220e8888a17a34d78d001f96035 \
                    rmd160  aac9b9c2f52ac75eec0700ac996a87f0151923f9 \
                    size    25969

python.versions     37

if {${name} ne ${subport}} {
    depends_lib-append \
                    port:py${python.version}-spdx-lookup \
                    port:py${python.version}-setuptools

    depends_run-append \
                    port:py${python.version}-upt-macports \
                    port:py${python.version}-upt-cpan \
                    port:py${python.version}-upt-pypi

    test.run        yes
    test.cmd        ${python.bin} -m unittest
    test.target
    test.env        PYTHONPATH=${worksrcpath}/build/lib

    post-destroot {
        set docdir ${prefix}/share/doc/${subport}
        xinstall -d ${destroot}${docdir}
        xinstall -m 0644 -W ${worksrcpath} README.md LICENSE CHANGELOG \
            ${destroot}${docdir}
    }

    livecheck.type  none
}

Firstly there are a few packages that are not present upstream but even then declared as dependencies so that should not be added by upt-hackage itself. Example: unbuildable, invalid-cabal-flag-settings, etc..
Do we agree on this? ping @Steap

Then there are ports which are present in hackage but we might not want to mention as dependencies ever like bytestring, unix, win-32, base, etc.. so can we comeup with list of such ports and filter them out and I guess all hackage packages would require ghc....not sure about all this.

ping @mojca @essandess

As currently, upt-cpan ignores build dependency they are not available to the backend.

Now my doubt is, do we need "develop" and "configure" dependencies?

As I see in the Port B-utils the build dependency is written in the original portfile is actually from "configure" in the json and the "build" ones from the json are not written.

My view would be to write all three under build dependency?
What are your views on this? @reneeotten @mojca

Also @Steap, should I push a PR for the same to upt-cpan once this is decided as the only change would be to change the mapping of phases_to_upt_phases and change the if statement.

As output option already exists in upt, it was easy to add the feature to upt-macports.
But I am not pushing it right now as there are parts of it that need #19, #20 and #24 to be merged first.

So once that's done I will create a PR for output feature.

As already done in the current PR, now args.packages is a list which upt iterates over and generates all the ports.

So using this list we can simply add a recursive feature like once the frontend returns the list of required packages.
Add them to the list if not already in the list. But in this, there is no check or control by backend which can be added if instead of upt adding to list, the backend does that.

Now the question is in the backend should we "regenerate" ports if already present in MacPorts tree and second, currently if we output to file the program stop on encountering an existing file which would be a problem if we try recursively packaging ports having the same dependency not committed to MacPorts tree.
So should we change this to just logging the error and continue?

In some cases setuptools is returned both in build and lib which should not be there.

Test ports:
py-zope-deprecation (pypi_name : zope-deprecation)
py-buildbot-pkg (pypi_name : buildbot-pkg)
py-fs (pypi_name : fs)

We currently convert the upstream name to lowercase and/or do other manipulations in multiple locations.

For example, in the MacPortsPythonPackage class there is:

    def _pkgname(self):
        macports_name = self._normalized_macports_name(self.upt_pkg.name)
        return f'{macports_name}'

    @staticmethod
    def _normalized_macports_name(name):
        name = name.lower()
        return f'py-{name}'

    @staticmethod
    def _normalized_macports_folder(name):
        name = name.lower()
        return f'py-{name}'

and after the template-clean up there will be another translation of dependency names

def jinja2_reqformat(self, req):
    return f'py${{python.version}}-{req.name.lower()}'

this should really be only done once and re-used in all the other functions.

The functionality to be added is most probably in upt itself and not in upt-macports.
It is inspired by cpan2port, so basically instead of just passing one port at a time we can have below two options:

• upt package -f .. -b .. pkg1,pkg2,pkg3
• upt package -f .. -b .. -i/--input pkgs.list

So here pkgs.list is a file containing package name.

What are your views on this ?

In _create_output_directories, we "scan" the frontend:

    upt2macports = {
        'cpan': 'perl',
        'pypi': 'python',
        'rubygems': 'ruby',
    }

We already do that once in MacportsBackend.create_package(). We should only do it once.

Would it be possible to tag releases of this, and publish this onto PyPI. Thanks.

A few suggestions for the base.Portfile template:

1. adding a comment along the lines of "Portfile generated/maintained using the Universal Packaging Tool (upt)" would be useful to indicate which ports are using upt. (I would add it just before the name field).
2. rephrase the comment about supported_archs for more clarity. How about something like "# uncomment the following line if no architecture-dependent files are installed, otherwise remove"
3. remove lines 16 (print(self._depends)) and 25 (# env.filters['reqformat'] = self.jinja2_reqformat) as they seem unnecessary
4. add ```categories` field (probably commented out since it's difficult to fill in sensible values automatically)
5. add "default" example for "maintainers"; maintainers: {example.org:julesverne @GitHub-handle} openmaintainer
6. follow the order of fields from the guide a bit more closely. Admittedly, some of this is personal preference so it doesn't need to be changed, but please have a look. I would recommend the following order:

# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4

PortSystem          
PortGroup           

name                
version             
revision            

# categories-append
platforms           
# uncomment the following line if no architecture-dependent files are installed, otherwise remove
# supported_archs     noarch
license            
maintainers: {example.org:julesverne @GitHub-handle} openmaintainer        

description         
long_description    

homepage            
master_sites        
distname            
# if needed, add keyword depending on compression type of archive
checksums           

In {ruby,python,perl}.Portfile, the same kind of code can be found:

    {%- if pkg.upt_pkg.requirements.run|length > 0 %}
    depends_lib-append \
                    {% for i in pkg.upt_pkg.requirements.run|sort(attribute='name') %}
                    port:py${python.version}-{{i.name}}{% if not loop.last %} \
                    {% endif %}
                    {% endfor %}
    {% endif %}

The main difference between the templates is the way the dependency is displayed.

1. We could create a Jinja filter (see for instance what is done in upt-openbsd: https://framagit.org/upt/upt-openbsd/blob/master/upt_openbsd/upt_openbsd.py#L79). The filter could easily be tested in the unit tests.

2. Maybe create a macro that prints the dependencies? See https://framagit.org/upt/upt-openbsd/blob/master/upt_openbsd/templates/base.mk#L1

This would reduce the amount of code needed in the templates, and improve readability.

I just added Ruby 2.6 as supported version to the Ruby PortGroup (macports/macports-ports@b4b3225).

After doing so I tried to package a (random) port (rb-treetop) to see how upt performs. It does get all the dependencies and such right, but there are a few issues with the current template. Admittedly, we (or probably I) will need to do some more work on the PortGroup itself to fix some of them.... but that might take a while since I don't have much experience with Ruby nor writing PortGroups ;)

Anyway, the template will eventually need some more changes once the new PortGroup is ready, but for now it should result in a working Portfile with the current PG which isn't the case right now.

Specifically:

1. ruby.branches needs to be before ruby.setup, otherwise there are no subports generated (I would consider this a bug in the PG)
2. for dependencies it shouldn't be port:rb${ruby.version}-<name> but port:rb${ruby.suffix}-<name>

Trying to package its dependency (rb-polyglot) there is an issue with the formatting of the description that makes the Portfile fail the parsing by MacPorts. Fixing that makes both ports (as far as I can tell at least) install correctly.

Finally, it logs the following message (twice):
[ERROR ] [Backend] Could not determine the type of the source archive
That should not happen I think as the source should be always .gem here, but also with the latest release of upt-rubygems I thought is was setting an ArchiveType, correct? You made an exception for rubygems in the base.Portfile template for the archive_type, perhaps the same thing should happen for the logging as well?

Installed:

  py39-upt-cpan @0.6.1_0 (active)
  py39-upt-macports @0.1-20200607_0 (active)
  py39-upt-pypi @0.6_0 (active)
  py39-upt-rubygems @0.4.1_0 (active)

when run upt package -f pypi -b macports -o . -r pyzbar it crashed:

[INFO    ] [Backend] Creating MacPorts package for pyzbar
[INFO    ] [Backend] Found license MIT
[ERROR   ] [Backend] Could not determine the type of the source archive
Traceback (most recent call last):
  File "/opt/local/bin/upt", line 33, in <module>
    sys.exit(load_entry_point('upt==0.12', 'console_scripts', 'upt')())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/upt/upt.py", line 497, in main
    package(name, version, frontend, backend, args.output,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/upt/upt.py", line 382, in package
    backend.create_package(upt_pkg, output=output)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/upt_macports/upt_macports.py", line 258, in create_package
    packager.create_package(upt_pkg, output)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/upt_macports/upt_macports.py", line 20, in create_package
    portfile_content = self._render_makefile_template()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/upt_macports/upt_macports.py", line 57, in _render_makefile_template
    return template.render(pkg=self)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/upt_macports/templates/python.Portfile", line 1, in top-level template code
    {% extends 'base.Portfile' %}
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/upt_macports/templates/base.Portfile", line 41, in top-level template code
    checksums           rmd160  {{ pkg.upt_pkg.archives[0].rmd160 }} \
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/jinja2/environment.py", line 474, in getattr
    return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: list object has no element 0

When testing the templates, I ran into a few situations where while creating the package we would stumble upon an upt.upt.ArchiveUnavailable: No such archive could be found problem.

For example,

py-scitools
[INFO    ] [Backend] Hello, creating the package
[INFO    ] [Backend] Creating the directory structure in /tmp/upt-pypi-packaging
[INFO    ] [Backend] Created /tmp/upt-pypi-packaging/python/py-scitools
[INFO    ] [Backend] Creating the Portfile
Traceback (most recent call last):
  File "/tmp/test-3.7/bin/upt", line 11, in <module>
    load_entry_point('upt==0.9', 'console_scripts', 'upt')()
  File "/tmp/test-3.7/lib/python3.7/site-packages/upt/upt.py", line 302, in main
    backend.create_package(upt_pkg, output=args.output)
  File "/tmp/test-3.7/lib/python3.7/site-packages/upt_macports-0.1-py3.7.egg/upt_macports/upt_macports.py", line 253, in create_package
  File "/tmp/test-3.7/lib/python3.7/site-packages/upt_macports-0.1-py3.7.egg/upt_macports/upt_macports.py", line 22, in create_package
  File "/tmp/test-3.7/lib/python3.7/site-packages/upt_macports-0.1-py3.7.egg/upt_macports/upt_macports.py", line 51, in _create_portfile
  File "/tmp/test-3.7/lib/python3.7/site-packages/upt_macports-0.1-py3.7.egg/upt_macports/upt_macports.py", line 64, in _render_makefile_template
  File "/tmp/test-3.7/lib/python3.7/site-packages/Jinja2-2.10.1-py3.7.egg/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/tmp/test-3.7/lib/python3.7/site-packages/Jinja2-2.10.1-py3.7.egg/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/tmp/test-3.7/lib/python3.7/site-packages/Jinja2-2.10.1-py3.7.egg/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/tmp/test-3.7/lib/python3.7/site-packages/Jinja2-2.10.1-py3.7.egg/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "<template>", line 19, in top-level template code
  File "<template>", line 8, in block "nameversion"
  File "/tmp/test-3.7/lib/python3.7/site-packages/upt_macports-0.1-py3.7.egg/upt_macports/upt_macports.py", line 144, in _python_root_name
  File "/tmp/test-3.7/lib/python3.7/site-packages/upt/upt.py", line 187, in get_archive
    raise ArchiveUnavailable()
upt.upt.ArchiveUnavailable: No such archive could be found

afterwards we do end up with an empty Portfile in the ```/python/py-scitools`` directory. One way around this would perhaps to first attempt to generate a Portfile in a temporary directory and only copy this over once everything finishes correctly. Or, alternatively, once we run into such Exceptions we could do some clean-up.

First of all (as an independent task) it would make sense if at the end we end up with upt as a standalone command, not just upt-3.7, but independent of that ...

I never know how to call upt. So I'm always confused by how little information I get by typing upt -h. This is semi-useless unless you already know what you are supposed to do:

usage: upt [-h] {list-backends,list-frontends,package} ...

I pause a bit and start thinking what to do. Then I figure out I should probably be using package, and then I need a couple of more commands to get the help for package. But then I mix the frontends and the backends, so another two calls just to confirm what is a frontend and what is a backend.

I would like to have either a complete man page, or the above command could at least automatically already list the available frontends and backends and immediately show

usage:
upt package [-h] -f FRONTEND -b BACKEND [--debug] [-o OUTPUT] [-q] package

plus some extra info.

On a somewhat unrelated matter, using cpan2port and pypi2port is somewhat more intuitive. Just a single command with autocompletion that replaces all of the

upt package -f pypi -b macports -o <something>

I wish we had some shortcuts to call when we want to create a package for macports. Something that would already be available as a command. Maybe upt-cpan2port or upt-pypi-macports, or something along the lines. I know this is somewhat incompatible when you need M x N frontends and backends, and then you install or uninstall just one, so handling the symlinks automatically by the upt package itself becomes a bit nontrivial.

opam is the package manager for the OCaml ecosystem. Almost all OCaml libraries and programs have associated opam files.

Generally, it's better for people working in the ocaml ecosystem to use opam for their day to day operations, but if you're building a package that is written in ocaml in macports, you probably don't care and don't want to care about opam and the opam ecosystem. We thus have to maintain a bunch of packages that are also packaged by opam, and it's painful. Automation would help a lot.

quick overview of the opam format

opam file description

repository of opam files used by the opam system

typical opam file for the zarith package

my portfile for zarith, not really made from the opam file

Template for RubyGems frontend is missing and needs to be added.

For few packages in hackage the description looks like:

description         This package provides a pure interface for compressing and \
                    decompressing streams of data represented as lazy \
                    ByteStrings. It uses the \
                    zlib C library \
                    so it has high performance. It supports the "zlib", \
                    "gzip" and "raw" compression formats.

and due to the quotes port lint gives a syntax error:

("extra characters after close-quote").

so should we remove quotes from all description by default for all frontends in upt-macports itself ?

Ideally, the layout/whitespaces within the if {${name} ne ${subport}} { } block should be the same irrespective of which/how many dependency types are present.

Try packaging setuptools, upt and pytest as examples of the different scenarios and make sure the layout is the same as below (i.e., no newline at the beginning of the if-block and one newline between the different dependency types, and the livecheck):

if {${name} ne ${subport}} {
    depends_build-append \
                    port:py${python.version}-setuptools

    depends_lib-append \
                    port:py${python.version}-???

    depends_test-append \
                    port:py${python.version}-???

   livecheck.type  none

If a specific package is not available/not found upstream then recursive packaging stops midway and there's no automagic way to package the remaining packages.

Example: ghci package for hackage, there are many missing packages in hackage.