See the second segment of this comment, quoted here for simplicity:

However, this strategy of using a shell (or Python or Ruby) "launcher", while a popular trick that worked in the past, seems to be breaking down on more recent macOS releases, particularly catalina. See this blog post about how one Emacs mac build's too-clever binary-selector launching script effectively broke the ability to launch the application multiple times. puppy.app seems to have the same problem - launching it from the Downloads folder while it's open doesn't bring it back to the front. By contrast, py2app applications use an executable shim and do not have this problem, since they don't re-exec; testing with my own pygame apps validates this. So you might want to figure out some way to build a C executable that invokes Python as a library.

(extracted from #9)

Motivations:

• Support a --output-format option that triggers different plugins that, themselves, announce the support/require that CLI option.
• Support overriding "default" Python runtime bundles.
• There will probably be more use cases.

FACT:

• As of now, the pup.install-cleanup step compiles the packaged Standard Library (and site-packages) one directory at a time.

THOUGHT:

• Maybe we can compile everything with a single python -m compileall invocation: probably just a matter of figuring out the right CLI flags. :)

BENEFIT:

• Simpler code.
• Marginally faster (?) compilation due to less sub-process spawning overhead.

Nice? :)

WANT NOW:

• Produce user installable MSI artifact.
• Should add entry to the start menu.
• Grab WiX or whatever is needed automatically.

LATER:

• Sign the MSI file.
• Allow for system installable MSI.
• Allow for GUI selectable user/system install?
• Not doing MSI files, but EXE installers, instead.

(creating issue to include cppyy for future evaluation)

TO BE CONSIDERED:

• Found this stackoverflow question about packaging applications that use cppyy.
• Does pup handle packaging such applications?
• If not, do we want it to? Can it be solved with a plugin?

FACTS:

• Packaging proceeds in steps, executed one after the other.
• If a given step fails, it probably makes no sense continuing.
• There's no way to stop other than raising an exception which will propagate to the top of the stack.

IDEA:

• Maybe step execution can raise a specific exception, caught by the step driver, in order to stop.

MOTIVATION:

• The good old UNIX UX of no feedback means "all is good"? :)

See #58 for the reasons why it currently is logging.INFO.

THOUGHTS:

• In the current state, nearly trivial: building ZIP file from the relocatable root should be enough.
• Probably depends on #14.

Objective:

• Identifies and initializes plugins.
• Runs things from the CLI and from the API.
• Supports CLI/API arguments and passes them to the proper "players".

THOUGHTS:

• In particular, the "Development Status" classifier, currently as "1 - Planning".
• I guess we're past that point!

MOTIVATION:

• At least during initial development, "always" logging exception tracebacks is useful.

FACT:

• After having fixed #35, pup is now failing on macOS with Python 3.8.
• Here's a sample output, trying to package mu:

$ PUP_LOG_LEVEL=info pup package --launch-module=mu ../../Python/github.com/mu
20200916 125819 I pup 1.0.0a1 - starting with PID=36760
20200916 125819 I pup.api Package '../../Python/github.com/mu': starting.
20200916 125819 I pup.plugins.metadata Collecting metadata for '../../Python/github.com/mu'.
20200916 125822 I pup.api Step 'mac.app-bundle-template': starting.
20200916 125822 I pup.api Step 'mac.app-bundle-template': completed.
20200916 125822 I pup.api Step 'pup.python-runtime': starting.
20200916 125824 E pup Execution failure: [Errno 2] No such file or directory: '/Users/tiago.montes/Work/Python-Public/x/build/pup/mu-editor.app/Contents/Resources/Python/lib/python3.7/config-3.7m-darwin'
20200916 125824 I pup 1.0.0a1 - done

MOTIVE:

• The quick-and-very-very-dirty "Python Runtime cleanup" is hard-coded for 3.7. :-)

PROPOSED FIX:

• Let's drop the runtime cleanup hacks and create an issue to handle that (kind of) "properly".

Want:

• Should layout a macOS Application Bundle "skeleton" based off a built-in cookiecutter template.

Future:

• Allow template to be overridden.

PLAN:

• Collect and share documentation links about the topic (many of which I've been reading as of recently).
• Get a developer certificate from Apple.
• Explore the process manually / via ad-hoc scripts, confirming that it seems to work.
• Try to validate it in collaboration with someone else, running macOS Catalina or Big Sur.
• Transpose the process into pup code.

NOTE:

• From the investigation so far, it is a very good idea to have a properly layed out, as small as possible macOS Application Bundle to submit for notarization.
• With that in mind, #38 becomes a relevant dependency to this issue's "correct" implementation.

AT LEAST:

• Include notes on new MSI and DMG outputs.
• Review README.

See the first segment of this comment in #43.

From #82:

• Sign the MSI file | Addressed in #97.
• Allow for system installable MSI in #151.
• Allow for GUI selectable user/system install? #151
• Include license agreement | Addressed in #91.
• Add Icon to MSI file -- the file itself can't have one, that's an OS level thing | See #103.
• Address apprently font-related warnings when packaging Mu.

IDEA:

• Signed and notarized macOS application bundles should be packaged into commonly used DMG images.
• DMG filename and volume name should be along the lines of <nice-application-name> <version>.

EXTRA POINTS:

• Should include a nicely layed out link to the /Applications directory and some kind of visual arrow, guiding users to drag (copy) the distributed application bundle there, per platform conventions.

USEFUL REFERENCES:

• https://stackoverflow.com/questions/96882/how-do-i-create-a-nice-looking-dmg-for-mac-os-x-using-command-line-tools
• https://github.com/create-dmg/create-dmg

USE CASE:

• Run pre-flight checks without actually doing the packaging.
• Examples:
  • Network connectivity available, if needed, to download things.
  • Cached downloaded files available (thus downloading not needed).
  • Tools like macOS codesign and others found.
  • More?

POSSIBLE IMPLEMENTATION:

• Each step's __call__ method to include a preview=False argument.

CURRENT macOS FACT:

• Packaged Mu with Python 3.7.
• Checked the value of the bundled Python environment's sys.path with:

$ ./build/pup/mu-editor.app/Contents/Resources/Python/bin/python3 -q
>>> import sys
>>> print(*map(repr, sys.path), sep='\n')
''
'${PWD}/build/pup/mu-editor.app/Contents/Resources/Python/lib/python37.zip'
'${PWD}/build/pup/mu-editor.app/Contents/Resources/Python/lib/python3.7'
'${PWD}/build/pup/mu-editor.app/Contents/Resources/Python/lib/python3.7/lib-dynload'
'${HOME}/.local/lib/python3.7/site-packages'
'${PWD}/build/pup/mu-editor.app/Contents/Resources/Python/lib/python3.7/site-packages'
'${PWD}/build/pup/mu-editor.app/Contents/Resources/Python/lib/python3.7/site-packages/setuptools-49.6.0-py3.7.egg'
'${PWD}/build/pup/mu-editor.app/Contents/Resources/Python/lib/python3.7/site-packages/pip-20.2.2-py3.7.egg'
>>> 

NEEDS CHANGE:

• Two entries should be eliminated from sys.path:
  • The '' empty one, corresponding to the current working directory.
  • The ${HOME}/.local/lib/python3.7/site-packages which, I suppose, holds pip --user installed packages.

WONDERING:

• What kind of sys.path values are we getting on Windows?
• What about Python 3.8 variations?

RELATED:

• This BeeWare briefcase issue, in particular from this comment onwards. :-)

IDEA:

• Drop the date and time.
• Drop the module name.

MOTIVATION:

• Cleaner, hopefuly more readable kind of interactive-ish output.

REPORTED by @tjguk, on Gitter's mu-editor/dev channel:

I gave it a go on Windows. Cloned pup; created a local venv; pip install -e.[dev]; and then ran -- from the pup directory pup package --launch-module=mu c:\work-in-progress\mu-master
It did stuff for a while (albeit with no output) and then I got an error:
20200915 072004 E pup Execution failure: module 'importlib.resources' has no attribute 'files'
This is with Python 3.8.3 on Win10

TO WHICH I RESPONDED:

Thanks for trying it @tjguk! :) Silly me trying to use 3.8 stdlib’s importlib.resources, while using PyPI’s “backported” but more capable on 3.7 - will be an easy fix. FYI, for more “noisy” output, add —log-level=info/debug between pup and package.

FACTS:

• importlib.resources is used to "bundle" cookiecutter templates as packages within pup itself.
• On 3.7 the PyPI backported importlib_resources package is used...
• ...on 3.8, the standard library's importlib.resources is used instead.

CULPRIT:

• The PyPI importlib_resources provides a files function that is not implemeted by 3.8's stdlib.

POSSIBLE FIXES:

• Use PyPI's importlib_resources on both 3.7 and 3.8 (3.9's stdlib seems to include the files function we're using).
• Use alternative implementation, supported by 3.8's stdlib.

FACTS:

• Python packages (and thus, pup packagable applications) tend to have a somewhat technical (-ish) name.
• The case for Mu is mu-editor, which is what people pip-install, for instance.
  (then there's the fact that the pip-installable name is not necessarily the importable one -- see #28).

WANTED:

• It will be nice to have Mu packaged into a distributable file called, say, Mu Editor 1.1.0.zip,
  instead of mu-editor 1.1.0.zip which would be the best pup can do from the the available metadata.

THOUGHTS:

• I've been trying to avoid requiring a "configuration file" for pup to do its thing.
• Given that, AFAICT, there's no way to add custom metadata to a Python distribution (wheel, sdists, etc)...
• ...maybe we'll start off by adding a CLI option / API argument to support that (much like #28, again).

MOTIVATION:

• There's some written words now.

OBJECTIVE:

• Let's have GitHub actions do the Sphinx build...
• ...and figure out a way of having the result published to readthedocs.org.

Used by plugins to know where things go:

• Downloaded file caches?
• Logs?
• Other things?

MOTIVATION:

• Keep redistributable artifact as small as reasonable.
• The Python Runtime includes lots of things that are of no use in most (?) cases: stdlib tests, static libraries, etc.

IDEAS:

• Remove unneeded files for running the application:
  • All the tests in the Standard Library.
  • All (most?) of the scripts/console_scripts (pip? easy_install?), including ones that may be put in place after pip-installing the application and its dependencies.
• (Optionally?) compile the Standard Library.

DETAILS:

• Packaging puppy is failing with the latest PyPI-sourced pup 1.0.0a3 release.

REPRODUCTION:

• Create a Python 3.7 venv and update pip.
• pip install pup -- grabs 1.0.0a3 from PyPI.
• Clone puppy source tree from https://github.com/tmontes/puppy.
• Run pup package . on the source tree root.

RESULT (Windows):

(venv-xirro) C:\Users\test\work\github.com\puppy>pup package .
I pup 1.0.0a3 - starting with PID=3156
I Package '.': starting.
I Collecting metadata for '.'.
I About to run 'c:\\users\\test\\work\\github.com\\puppy\\venv-xirro\\scripts\\python.exe -m pip wheel --no-deps C:\\Users\\test\\work\\github.com\\puppy'.
I pip out: Processing c:\users\test\work\github.com\puppy
I pip out: Building wheels for collected packages: puppy
I pip out:   Building wheel for puppy (setup.py): started
I pip out:   Building wheel for puppy (setup.py): finished with status 'done'
I pip out:   Created wheel for puppy: filename=puppy-1.4.0-py3-none-any.whl size=8648 sha256=ff3f236e235e73f658ce6ae95e4f0786a4065d7e8515647b21a8f910a5c80590
I pip out:   Stored in directory: C:\Users\test\AppData\Local\Temp\pip-ephem-wheel-cache-vyuiro5s\wheels\17\26\43\705bd58c4b717c1af27e3ec8fb612c8da7e59b082813f42eb8
I pip out: Successfully built puppy
I Step 'win.distribution-layout': starting.
C Execution failure:
C Traceback below:
Traceback (most recent call last):
  File "c:\users\test\work\github.com\puppy\venv-xirro\lib\site-packages\pup\__main__.py", line 31, in wrapper
    exit_code = command_function(*args, **kw)
  File "c:\users\test\work\github.com\puppy\venv-xirro\lib\site-packages\pup\__main__.py", line 80, in package
    launch_module=launch_module,
  File "c:\users\test\work\github.com\puppy\venv-xirro\lib\site-packages\pup\api.py", line 41, in package
    dsp.run_pluggable_step(step)
  File "c:\users\test\work\github.com\puppy\venv-xirro\lib\site-packages\pup\dispatcher.py", line 79, in run_pluggable_step
    return self._invoke_plugin(name)
  File "c:\users\test\work\github.com\puppy\venv-xirro\lib\site-packages\pup\dispatcher.py", line 68, in _invoke_plugin
    return plugin(self._ctx, self, **kwargs)
  File "c:\users\test\work\github.com\puppy\venv-xirro\lib\site-packages\pup\plugins\win\dist_layout.py", line 63, in __call__
    result_path = generate.generate_files(tmpl_path, tmpl_data, build_dir, overwrite_if_exists=True)
  File "c:\users\test\work\github.com\puppy\venv-xirro\lib\site-packages\cookiecutter\generate.py", line 264, in generate_files
    template_dir = find_template(repo_dir)
  File "c:\users\test\work\github.com\puppy\venv-xirro\lib\site-packages\cookiecutter\find.py", line 34, in find_template
    raise NonTemplatedInputDirException
cookiecutter.exceptions.NonTemplatedInputDirException
I pup 1.0.0a3 - done

(venv-xirro) C:\Users\test\work\github.com\puppy>

RESULT (macOS):

(venv) tiago.montes (~/Temp/i77/puppy): pup package .
I pup 1.0.0a3 - starting with PID=1032
I Package '.': starting.
I Collecting metadata for '.'.
I About to run '/Users/tiago.montes/Temp/i77/venv/bin/python3.7 -m pip wheel --no-deps /Users/tiago.montes/Temp/i77/puppy'.
I pip out: Processing /Users/tiago.montes/Temp/i77/puppy
I pip out: Building wheels for collected packages: puppy
I pip out:   Building wheel for puppy (setup.py): started
I pip out:   Building wheel for puppy (setup.py): finished with status 'done'
I pip out:   Created wheel for puppy: filename=puppy-1.4.0-py3-none-any.whl size=8621 sha256=cea794985820a128e2dfdbccdb480447e05a0f63cfdb04620aea41962d15af88
I pip out:   Stored in directory: /private/var/folders/0m/xy6669f52zdbb6prv46hrghw0000gp/T/pip-ephem-wheel-cache-as8l2gbp/wheels/fe/32/0e/c92f7a965c6b69e2cc9d2076cc0dd7078458303396513efbcd
I pip out: Successfully built puppy
I Step 'mac.app-bundle-template': starting.
C Execution failure: 
C Traceback below:
Traceback (most recent call last):
  File "/Users/tiago.montes/Temp/i77/venv/lib/python3.7/site-packages/pup/__main__.py", line 31, in wrapper
    exit_code = command_function(*args, **kw)
  File "/Users/tiago.montes/Temp/i77/venv/lib/python3.7/site-packages/pup/__main__.py", line 80, in package
    launch_module=launch_module,
  File "/Users/tiago.montes/Temp/i77/venv/lib/python3.7/site-packages/pup/api.py", line 41, in package
    dsp.run_pluggable_step(step)
  File "/Users/tiago.montes/Temp/i77/venv/lib/python3.7/site-packages/pup/dispatcher.py", line 79, in run_pluggable_step
    return self._invoke_plugin(name)
  File "/Users/tiago.montes/Temp/i77/venv/lib/python3.7/site-packages/pup/dispatcher.py", line 68, in _invoke_plugin
    return plugin(self._ctx, self, **kwargs)
  File "/Users/tiago.montes/Temp/i77/venv/lib/python3.7/site-packages/pup/plugins/mac/app_bundle.py", line 65, in __call__
    result_path = generate.generate_files(tmpl_path, tmpl_data, build_dir, overwrite_if_exists=True)
  File "/Users/tiago.montes/Temp/i77/venv/lib/python3.7/site-packages/cookiecutter/generate.py", line 264, in generate_files
    template_dir = find_template(repo_dir)
  File "/Users/tiago.montes/Temp/i77/venv/lib/python3.7/site-packages/cookiecutter/find.py", line 34, in find_template
    raise NonTemplatedInputDirException
cookiecutter.exceptions.NonTemplatedInputDirException
I pup 1.0.0a3 - done
(venv) tiago.montes (~/Temp/i77/puppy): 

HOWEVER:

• It does work when running pup from its development virtual environment.

THUS:

• We have a meta-packaging issue.
• pup is not properly packaged itself. :-)

REPRODUCING:

• Package puppy on Windows with pup package .../path-to-puppy.
• Don't change working directory and execute build\pup\puppy 1.2.0\puppy.vbs.

RESULT:

• A failure popup window is displayed.

FIX:

• Requires the .vbs script to be CWD-independent.
• This article may be of help: https://www.geekshangout.com/vbs-script-to-get-the-location-of-the-current-script/

MOTIVATION:

• Keep distribution size as small as possible.

MOTIVATION:

• The currently hard-coded approach is a proof-of-concept, not sustainable.

THOUGHTS:

• Be slightly more clever and take platform, architecture, bitness (and more?) into consideration bringing in support for:
  • 64 and 32 bit Windows
  • GLIBC and MUSL Linux
  • ...more?
• Don't grab if there's a compatible but older version in cache?
• Allow overriding default behaviour by specifying a specific relocatable Python Runtime (URL? File?) -- implemented in #125.

FACTS:

• About to release 1.0.0a3 that actually does package, sign, and notarize Mu Editor on macOS.
• Usage is simple, but non-trivial.
• At least document that so as to simplify gathering feedback from early users/tests.

• #16 will provide a built-in template.
• It should be overridable via CLI options / API args (depends on #14).

(it's actually already created, but make it do what it says on the tin!) :)

FACTS:

• Source package metadata is collected with pkginfo from a wheel file produced with pip wheel --no-deps <src>, which is queried and then thrown away (deleted).
• At a later stage, the source package is pip-installed into the relocatable Python environment.

IDEA:

Maybe re-using the wheel could:

• Make things go (ever so slightly) faster?
• Save some compute cycles (and thus, be environmentally friendlier)?
• ...just make sense?

FACTS:

• macOS packaging is minimally working: pup package --launch-module=mu .../mu-repository-root results in an apparently working packaged and redistributable mu-editor.app application bundle.

WANT:

• The same command should result in a directory/zip-file that also apparently works on Windows.

FACTS:

• Current output (STDOUT/STDERR) from sub-processes is logged in an "all-at-once" approach when the sub-process ends.
• Also, STDOUT is logged at DEBUG level.
• Some such runs may be somewhat long-ish (many seconds, few minutes).

IDEA:

• Given that we prefer not to let sub-processes "just" output to STDOUT/STDERR directly...
• ...maybe we could live log such output as it is produced, prefixing it with something like 'pip stdout/stderr'?

THOUGHTS:

• Given that subprocess.run seems to block, not sure how to achieve that.
• Maybe a thread or two are in order? :-)

FACTS:

• pup started off as Python Mu Packager, which makes sense given its original motivation.
• Per the design and initial results I feel very confident that it will indeed support more applications that just the Mu Editor, with no additional effort.

THOUGHTS:

• Having "Python" in the name feels redundant -- it's on PyPI, what should it be related to?!
• Having "Mu" in the name feels like it is scoped too narrow -- it does (and will) package other applications.

THUS:

Let's call it Pluggable Micro Packager instead:

• Still abbreviated as pup (of course!), taking the u as an ASCII simplification of the µ (mu) Greek letter, a common symbol for the word "micro" -- retaining the relation with the original motivation, the Mu Editor.
• Highlights one of its features: pluggability (which will need to be much better designed and implemented, but that's a whole other story).

MOTIVATION:

• Was copied from another project.
• There are some inconsistencies.

Address, at least:

• Context object: what information does it convey? how is it populated? is it mutable?
• Dispatcher: which criteria can/should it use to do its job? does it include a "registry" is will that be a different thing?

Then, maybe:

• Figure out a way that "pluggable behaviours" can get "parameters" from their callers and wire that nicely to both the pup API and the CLI usage.

FACTS:

• As of now, due to the way that the packaged Standard Library is compiled -- in legacy (pre-PEP3147) mode -- the packaging process produces some redundant pyc files living inside __pycache__ directories.
• These probably result from python -m pip execution within the distributable Python environment.

OBJECTIVE:

• Let's remove those __pycache__ directories, shall we?

BENEFIT:

• Smaller distributable artifact.
• ...zero doubts about what is available and imported from where. :)

Motivation:

• Mu's package name is mu-editor...
• ...but it is launched with python3 -m mu.

:-)

Packaging steps will need to know things like:

• Name
• Version
• Copyright
• ...

MOTIVATION:

• For the time being, during development, avoid the need to use the --log-level=info CLI arg (or setting the PUP_LOG_LEVEL environment variable).
• Should simplify initial testing/exploration by others and gathering their feedback.

THOUGHT:

• Maybe later consider changing it back to logging.WARNING.

Ideally, CI should:

• Run tests when src or tests change.
• Build docs when docs change.

• That's the purpose of the pup.python-runtime pluggable step.
• Should put a "Python Runtime" in the directory indicated by the context python_runtime_dir value.

MOTIVATION:

• Share with the Mu development team.
• Test with current master.
• Get early feedback.