nix-community / emacs2nix Goto Github PK

View Code? Open in Web Editor NEW

27.0 10.0 16.0 1.21 MB

Automatically generate Nix expressions for Emacs packages

License: GNU General Public License v3.0

Haskell 86.96% Nix 7.96% Emacs Lisp 4.14% Shell 0.95%

nix emacs-packages melpa nix-expressions

emacs2nix's Introduction

emacs2nix

Automatically generate Nix expressions for Emacs packages.

Be sure to run

git submodule update --init

before building this package!

The scripts elpa-packages.sh, nongnu-packages.sh, nongnu-devel-packages.sh, melpa-packages.sh, and melpa-stable-packages.sh regenerate each package set. They require Nix to build.

To update the ELPA packages, run

# For ELPA packages
./elpa-packages.sh \
  -o $NIXPKGS/pkgs/applications/editors/emacs-modes/elpa-generated.nix \
  --names names.nix

$NIXPKGS should be the path to the Nixpkgs clone which you are updating.

To update the MELPA packages, run

# Use melpa-stable-packages.sh to update melpa-stable-generated.nix instead
./melpa-packages.sh \
  --melpa $MELPA \
  -o $NIXPKGS/pkgs/applications/editors/emacs-modes/melpa-generated.nix \
  --names names.nix

$MELPA should be tho path to a clone of the MELPA repository.

To run one of the elpa2nix or melpa2nix commands outside the shell script wrappers, first enter a Nix shell with the command

nix-shell -A env

which will ensure that all the runtime dependencies are present.

emacs2nix's People

Contributors

Stargazers

Watchers

Forkers

gridaphobe aneeshusa phunehehe bendlas binarin matthewbauer ryantm alyssais collares kennyballou vonfry isabella232 accelbread leungbk

emacs2nix's Issues

Wrong dependencies with MELPA

I'll provide more info later- but something in emacs2nix is messing up my config.

Basically for some packages I have two versions of it. Here's my share/emacs/site-lisp/elpa:

  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 aggressive-indent-20171012.1107
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 anaconda-mode-20171223.1118
  dr-xr-xr-x  33 root wheel 1.1K Dec 31  1969 anything-20170125.1710
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 apiwrap-0.4
  dr-xr-xr-x  11 root wheel  352 Dec 31  1969 apropospriate-theme-20171010.1420
  dr-xr-xr-x  12 root wheel  384 Dec 31  1969 async-1.9.2
  dr-xr-xr-x  93 root wheel 3.0K Dec 31  1969 auctex-12.1.0
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 avy-0.4.0
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 bind-key-2.3
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 bool-flip-20161215.739
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 browse-at-remote-20171115.210
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 buffer-move-20160615.1103
  dr-xr-xr-x   8 root wheel  256 Dec 31  1969 bug-reference-github-20180128.514
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 coffee-mode-20170324.240
  dr-xr-xr-x  50 root wheel 1.6K Dec 31  1969 company-0.9.4
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 copy-as-format-20171215.1616
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 counsel-20180114.1336
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 crux-20180113.251
  dr-xr-xr-x   8 root wheel  256 Dec 31  1969 css-eldoc-20150124.1923
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 csv-mode-1.7
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 c-eldoc-20170917.1502
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 dash-2.13.0
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 dash-functional-2.13.0
  dr-xr-xr-x  14 root wheel  448 Dec 31  1969 diff-hl-20180123.1420
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 dired-hacks-utils-20160527.1436
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 dired-subtree-20170910.1521
  dr-xr-xr-x   8 root wheel  256 Dec 31  1969 dtrt-indent-20171001.1233
  dr-xr-xr-x  24 root wheel  768 Dec 31  1969 elfeed-20180127.1442
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 elf-mode-20161009.48
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 elisp-refs-1.2
  dr-xr-xr-x  16 root wheel  512 Dec 31  1969 elpy-20180119.54
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 epl-0.8
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 esh-help-20170829.2111
  dr-xr-xr-x  27 root wheel  864 Dec 31  1969 ess-20180128.1545
  dr-xr-xr-x   8 root wheel  256 Dec 31  1969 esup-20170923.1328
  dr-xr-xr-x  58 root wheel 1.9K Dec 31  1969 expand-region-20171217.2125
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 find-file-in-project-3.5
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 firestarter-20161219.523
  dr-xr-xr-x   9 root wheel  288 Dec 31  1969 flycheck-31
  dr-xr-xr-x   9 root wheel  288 Dec 31  1969 flycheck-20180123.1419
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 focus-20171203.2103
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 f-0.20.0
  dr-xr-xr-x  26 root wheel  832 Dec 31  1969 ghc-20180121.418
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 ghub+-0.2.1
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 ghub-1.3.0
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 git-commit-2.11.0
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 git-commit-20180126.913
  dr-xr-xr-x   8 root wheel  256 Dec 31  1969 god-mode-20180117.334
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 go-eldoc-20170305.627
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 go-mode-1.5.0
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 go-mode-20170726.555
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 hackernews-20180117.1302
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 haml-mode-3.1.9
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 haml-mode-20170923.2153
  dr-xr-xr-x  90 root wheel 2.9K Dec 31  1969 haskell-mode-16.1
  dr-xr-xr-x  90 root wheel 2.9K Dec 31  1969 haskell-mode-20171022.26
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 helpful-20180120.355
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 highlight-indentation-0.7.0
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 hl-todo-20170621.1257
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 hookify-20141216.1409
  dr-xr-xr-x  13 root wheel  416 Dec 31  1969 hook-helpers-1.1
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 htmlize-20171017.141
  dr-xr-xr-x  11 root wheel  352 Dec 31  1969 hydra-0.14.0
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 imenu-anywhere-20170805.1155
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 imenu-list-20171120.1255
  dr-xr-xr-x  52 root wheel 1.7K Dec 31  1969 indium-20180122.507
  dr-xr-xr-x  24 root wheel  768 Dec 31  1969 irony-1.2.0
  dr-xr-xr-x  24 root wheel  768 Dec 31  1969 irony-20180104.1109
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 irony-eldoc-20170502.1208
  dr-xr-xr-x  13 root wheel  416 Dec 31  1969 ivy-0.10.0
  dr-xr-xr-x  13 root wheel  416 Dec 31  1969 ivy-20180124.1127
  dr-xr-xr-x 136 root wheel 4.3K Dec 31  1969 jdee-20180109.1233
  dr-xr-xr-x  10 root wheel  320 Dec 31  1969 js2-mode-20170721
  dr-xr-xr-x  10 root wheel  320 Dec 31  1969 js2-mode-20171224.1833
  dr-xr-xr-x   8 root wheel  256 Dec 31  1969 js3-mode-20160515.850
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 json-mode-20170719.2205
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 json-reformat-0.0.6
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 json-snatcher-1.0.0
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 julia-mode-20171116.642
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 kill-or-bury-alive-20171231.2218
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 let-alist-1.0.5
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 list-utils-0.4.4
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 loop-1.3
  dr-xr-xr-x  22 root wheel  704 Dec 31  1969 magithub-20180128.1035
  dr-xr-xr-x  60 root wheel 1.9K Dec 31  1969 magit-2.11.0
  dr-xr-xr-x  79 root wheel 2.5K Dec 31  1969 magit-20180129.629
  dr-xr-xr-x   8 root wheel  256 Dec 31  1969 magit-popup-2.12.2
  dr-xr-xr-x   8 root wheel  256 Dec 31  1969 magit-popup-20180119.111
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 markdown-mode-2.3
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 markdown-mode-20180124.138
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 memoize-1.1
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 minimap-1.2
  dr-xr-xr-x  40 root wheel 1.3K Dec 31  1969 mmm-mode-20171212.1428
  dr-xr-xr-x  22 root wheel  704 Dec 31  1969 multiple-cursors-20170908.1452
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 multi-term-20160619.233
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 mwim-20180116.740
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 nix-buffer-20170809.1128
  dr-xr-xr-x  16 root wheel  512 Dec 31  1969 nix-mode-20180126.1935
  dr-xr-xr-x  47 root wheel 1.5K Dec 31  1969 notmuch-20180104.1635
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 org-static-blog-20170706.646
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 page-break-lines-20171210.31
  dr-xr-xr-x  10 root wheel  320 Dec 31  1969 pandoc-mode-20180122.108
  dr-xr-xr-x  44 root wheel 1.4K Dec 31  1969 php-mode-20180128.843
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 pkg-info-0.6
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 popwin-20150315.600
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 projectile-20180128.655
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 pythonic-0.1.0
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 pyvenv-1.11
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 rainbow-delimiters-20170929.432
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 rainbow-mode-0.13
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 restclient-20171203.1248
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 rg-20180122.855
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 rust-mode-20180109.544
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 sass-mode-20161006.2326
  dr-xr-xr-x  23 root wheel  736 Dec 31  1969 scala-mode-20170802.432
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 scss-mode-20180123.908
  dr-xr-xr-x  13 root wheel  416 Dec 31  1969 seq-2.20
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 shrink-whitespace-20150916.1215
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 shut-up-0.3.2
  dr-xr-xr-x  41 root wheel 1.3K Dec 31  1969 smartparens-20180129.939
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 smart-hungry-delete-20170412.643
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 smart-shift-20150202.2325
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 string-inflection-20180102.643
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 sudo-edit-20170803.2052
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 swiper-0.10.0
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 s-1.12.0
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 tern-20170925.1333
  dr-xr-xr-x   9 root wheel  288 Dec 31  1969 tide-20180125.418
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 transpose-frame-20151126.626
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 try-20170226.805
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 twittering-mode-20180107.412
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 typescript-mode-0.2
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 typescript-mode-20180118.2305
  dr-xr-xr-x  22 root wheel  704 Dec 31  1969 use-package-20180127.1413
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 vkill-20091203.1022
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 websocket-1.9
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 web-mode-20180120.1009
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 which-key-20180108.1930
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 whitespace-cleanup-mode-20170505.1923
  dr-xr-xr-x   8 root wheel  256 Dec 31  1969 with-editor-2.7.1
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 xah-math-input-20170905.849
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 xterm-color-20170102.1525
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 yafolding-20170304.1917
  dr-xr-xr-x   6 root wheel  192 Dec 31  1969 yaml-mode-20170727.1531
  dr-xr-xr-x   7 root wheel  224 Dec 31  1969 yasnippet-0.12.2

As you can see I have both the 'stable' and regular version of packages like magit, typescript-mode, etc. Emacs will end up only loading the 'stable' version. Looking into what's going on I think that the dependencies for some package is magit, etc. but it isn't getting that they're the same package.

Produce prettier error messages

Our error messages are detailed, but not very nice looking.

[failure] emacsPackages.ligo-mode: hash mismatch

As reported in NixOS/nixpkgs#217317
Confirmed in https://logs.nix.ci/?key=nixos/nixpkgs.220059&attempt_id=3cd1313f-a442-4976-943d-881a095ff606

error: hash mismatch in fixed-output derivation '/nix/store/gkp5avxisynk9fdiz3cs9qbflzq2kz1j-source.drv':
         specified: sha256-qlLBgKeabDi0UZNOll4PgXquterQN2zHxoY52AQ980Y=
            got:    sha256-wz9DF9mqi8WUt1Ebd+ueUTA314rKkdbjmoWF8cKuS8I=
error: 1 dependencies of derivation '/nix/store/a21rk31ydmrbkp5yxf27dxgx75d5lr00-emacs-ligo-mode-20230302.1616.drv' failed to build

Same issue in nix-community/emacs-overlay#304

Don't update package recipes if the hash is unchanged

Package recipes don't always need to come from the latest version of the MELPA repository. If a recipe's hash hasn't changed, there's no need to update the Git ref. This leads to generating much larger diffs than necessary.

ELPA automation scripts generate a duplicate of shell-command-plus

When update-elpa script is executed inside pkgs/applications/editors/emacs/elisp-packages/ of Nixpkgs repo, it always duplicate the specific recipe shell-command-plus.

Certainly it can be fixed by hand, but it is always a tedious procedure.

Here is the latest time the bug was reported at Nixpkgs:

NixOS/nixpkgs#119586

Some packages have version `nil`

Get package descriptions

Get package licenses

shell-command-plus is added multiple times

If run with an existing elpa-generated.nix file, a new shell-command-plus attribute is added, even if it already exists, resulting in evaluation errors.

error reading recipes: Error in $['perl6-mode']: expected Object, encountered Array

Getting this when trying to generate melpa packages with melpa/melpa@ccdf3598. Worked around for now by just deleting the perl6-mode recipe.

attribute ‘env’ in selection path ‘env’ not found

$ ./melpa-stable-packages.sh --threads 6 --melpa melpa/ -o ~/nixpkgs/pkgs/applications/editors/emacs-modes/melpa-stable-generated.nix
error: attribute ‘env’ in selection path ‘env’ not found

GitLab fetcher uses invalid URL

The GitLab fetcher doesn't seem to be working (anymore?) since the URL scheme it uses is invalid. Issue first reported at nix-community/emacs-overlay#128.

Compare https://github.com/ttuegel/emacs2nix/blob/master/src/Distribution/Nix/Fetch.hs#L228-L230 to how fetchFromGitlab puts together its URL: https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/fetchgitlab/default.nix

Repository configurations

We should support persistent repository configurations allowing the user to do things like:

specify the repository's upstream URL
disable specific packages
pin packages and recipes to certain versions

Make elpa-devel available upstream

I see there is support to grab elpa-devel generated package list. What are the plans to deal with this upstream or in emacs-overlay? (if any).

Summarize changes to each package set

Along the same lines as #8, it would be nice to produce at the end of the run a summary of packages added and removed from the package set.

Don't understand error message

$ ./melpa-stable-packages.sh --threads 4 --melpa ../melpa --output melpa-stable-package.nix shell-switcher
Preprocessing library emacs2nix-0.1.0.0...
In-place registering emacs2nix-0.1.0.0...
Preprocessing executable 'melpa2nix' for emacs2nix-0.1.0.0...
Running melpa2nix...
PackageException "shell-switcher" (ProcessFailed "nix-prefetch-zip" ["--url","https://github.com/DamienCassou/shell-switcher/archive/2c5575ae859a82041a4bacd1793b844bfc24c34f.tar.gz","--name","shell-switcher-2c5575ae859a82041a4bacd1793b844bfc24c34f-src"] BadPrefetchOutput)

what does that mean?

Downloads from EmacsWiki should be versioned

EmacsWiki provides a way to download previous versions of files using urls like
https://www.emacswiki.org/emacs/download/frame-cmds.el?revision=92

So I think that getFetcher at https://github.com/ttuegel/emacs2nix/blob/93b330f51f84e9f6f5ef57936816d6c84d7a05a8/src/Distribution/Melpa.hs#L239 should take some effort to determine proper revision and prevent package breakage after wiki edits.

I'm willing to implement this but I need to know which HTTP library I should add to .cabal and use.

Downloading from the wiki appears to be broken.

I spent some time trying to diagnose what was going on here, but failed. Adding some super-simple debugging output (putStrLn $ T.unpack $ T.unlines $ ["Hashes:"] ++ hashes before hashes is case-analyzed), I would get this:

Hashes:
0n9zagwh3rz7b76irj4ya8wskffns9v1c1pivsdqgpd76spvl7n5
/nix/store/3fp7xny76nx5wfzf4haibwrs0xyd3wni-ac-clang-6b3365063ddfb88d5527618217bb56166349ad4e-src


Hashes:

PackageException "ac-dabbrev" (ProcessFailed "nix-prefetch-url" ["https://www.emacswiki.org/emacs/download/ac-dabbrev.el"] BadPrefetchOutput)

So for a github package (ac-clang), everything works fine. For the wiki package (ac-dabbrev) nix-prefetch-url, there is apparently no output, so nothing to be used. If, of course, I try the same command at the command line, it works fine.

Fetch our own Melpa

Rather than relying on the user to provide a Melpa checkout, we should always fetch the recipes ourselves.

Add nongnu elpa

relative mail on mailing list

The Org package will continue to be available through GNU ELPA (from
https://elpa.gnu.org/packages/) and a new Org-Contrib package will be
available from https://elpa.nongnu.org/nongnu/.

Check that --output specifies a file

I tried running this:

./melpa-stable-packages.sh --melpa ../melpa/ --output /tmp/
[...]

after hours of work, I got this error message:

melpa2nix: /tmp/: openBinaryFile: inappropriate type (Is a directory)

could it be possible to say that right from the beginning? This would have hours of computation :-).

Also, what about accepting a directory as input and writing a file with a standard name in it?

Readme command nix-shell -A env error

Running the command nix-shell -A env causes the error: attribute env in selection path env not found.

Melpa and Melpa stable updating fails with exit code 255

$ ./melpa-stable-packages.sh   --melpa $MELPA   -o $NIXPKGS/pkgs/applications/editors/emacs-modes/melpa-stable-generated.nix   --names names.nix                   
in process emacs -Q --batch -L /home/ryantm/p/melpa/package-build -l /nix/store/bfadm4445a5j7mmj2cmqpazipg4wf21i-emacs2nix-0.1.0.0-data/share/ghc-8.2.2/x86_64-linux-ghc-8.2.2/emacs2nix-0.1.0.0/scripts/dump-recipes.el -f dump-recipes-json /home/ryantm/p/melpa/recipes: died with exit code 255:
Unable to activate package ‘forge’.
Required package ‘graphql-0.1.1’ is unavailable

/nix/var/nix/db is not writable

./melpa-stable-packages.sh --threads 4 --melpa ../melpa --output melpa-stable-package.nix shell-switcher
Preprocessing library emacs2nix-0.1.0.0...
In-place registering emacs2nix-0.1.0.0...
Preprocessing executable 'melpa2nix' for emacs2nix-0.1.0.0...
Running melpa2nix...
(BadPrefetchOutput,"")
PackageException "shell-switcher" (ProcessFailed "nix-prefetch-zip" ["--url","https://github.com/DamienCassou/shell-switcher/archive/2c5575ae859a82041a4bacd1793b844bfc24c34f.tar.gz","--name","shell-switcher-2c5575ae859a82041a4bacd1793b844bfc24c34f-src"] (Died 1 "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100   169    0   169    0     0    459      0 --:--:-- --:--:-- --:--:--   459\n\r100 22707  100 22707    0     0  27327      0 --:--:-- --:--:-- --:--:-- 27327\nhash is 0ia7sdip4hl27avckv3qpqgm3k4ynvp3xxq1cy53bqfzzx0gcria\nerror: Nix database directory \8216/nix/var/nix/db\8217 is not writable: Permission denied\n"))

this is with #13

Set up GitHub Actions for continous integration

This doesn't need to be complicated: we just need to install Nix and build the project. We could also try generating some packages as a quick test, but I don't think it's necessary.

ELPA packages are fetched from unstable url -> not reproducible

Elpa compresses all but the newest versions of a package and keeps compressed archives only of around 20 hand-selected versions. They also keep all packages in their git repo, which is a complete archive.

Details: NixOS/nixpkgs#110796

Unstable version numbers depend on the timezone where the packages are generated

...because they include a timestamp.

Following the README does not work

$ git clone https://DamienCassou:github.com/ttuegel/emacs2nix
$ cd emacs2nix
$ ./melpa-stable-packages.sh ../melpa/
error: opening file ‘/home/cassou/Documents/projects/emacs2nix/hnix/default.nix’: No such file or directory
(use ‘--show-trace’ to show detailed location information)

Parse exception: not enough input

@tobiasBora reports:

Is it for this reason that I can't use emacs2nix to update melpa packages? I get different errors depending on how I run it:

[me@nixos:/tmp/melpa/emacs2nix]$ export MELPA=/tmp/melpa/melpa/
[me@nixos:/tmp/melpa/emacs2nix]$ ./melpa-packages.sh \
>   --melpa $MELPA \
>   -o $NIXPKGS/pkgs/applications/editors/emacs-modes/melpa-generated.nix
melpa2nix: Parse exception: not enough input

Or:

[me@nixos:/tmp/melpa/emacs2nix]$ nix-shell shell.nix 
[nix-shell:/tmp/melpa/emacs2nix]$ export MELPA=/tmp/melpa/melpa/
[nix-shell:/tmp/melpa/emacs2nix]$ sudo ./melpa-packages.sh --melpa $MELPA -o $NIXPKGS/pkgs/applications/editors/emacs-modes/melpa-generated.nix
[sudo] password for me: 
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = "",
        LC_ALL = (unset),
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
error: current Nix store schema is version 10, but I only support 7

README instructions for updating MELPA missing names argument

When I run

./melpa-packages.sh \
  --melpa $MELPA \
  -o $NIXPKGS/pkgs/applications/editors/emacs-modes/melpa-generated.nix

I get

Missing: --names FILE                                                                                                                                                                                           
                                                                                                                                                                                                                    
Usage: melpa2nix [-t|--threads N] --melpa DIR [--stable] --work DIR                                                                                                                                                 
                 (-o|--output FILE) --names FILE [--index-only] [PACKAGE]                                                                                                                                           
  Generate Nix expressions from MELPA recipes

It looks like it wants a names file, but doesn't give an indication of what names file to provide.

Generate Nix expressions directly

Rather than generating a JSON file which gets processed by Nix, we should generate Nix expressions directly. One advantage of this approach is that it bypasses deficiencies in Nix's JSON parser, c.f. #2. Another advantage is that we can put package dependencies in each package's arguments so that overriding dependencies is easier, e.g.

# `foo` requires bar-2.0, but bar-1.0 is the default version
foo = super.foo.override {
  bar = self.bar-2;
}

Hours lost, but I don't know why

After hours of computation

PackageException "zotxt" (ProcessFailed "emacs" ["--batch","-l","../melpa/package-build.el","-l","/home/cassou/Documents/projects/emacs2nix/scripts/checkout.el","-f","checkout-stable","../melpa/recipes/zotxt","zotxt","/tmp/melpa2nix/zotxt"] NoVersion)
PackageException "ztree" (ProcessFailed "emacs" ["--batch","-l","../melpa/package-build.el","-l","/home/cassou/Documents/projects/emacs2nix/scripts/checkout.el","-f","checkout-stable","../melpa/recipes/ztree","ztree","/tmp/melpa2nix/ztree"] NoVersion)
PackageException "zygospore" (ProcessFailed "nix-prefetch-zip" ["--url","https://github.com/LouisKottmann/zygospore.el/archive/1af5ee663f5a7aa08d96a77cacff834dcdf55ea8.tar.gz","--name","zygospore.el-1af5ee663f5a7aa08d96a77cacff834dcdf55ea8-src"] BadPrefetchOutput)
PackageException "zzz-to-char" (ProcessFailed "nix-prefetch-zip" ["--url","https://github.com/mrkkrp/zzz-to-char/archive/efbe99c9163602f23408abaea70ffe292632bf26.tar.gz","--name","zzz-to-char-efbe99c9163602f23408abaea70ffe292632bf26-src"] BadPrefetchOutput)

I got that

$ cat /tmp/melpa-stable.nix 
{ callPackage }: {}

Add emacsmirror repository

Hello,
there are a lot of projects listed on emacsmirror. Here you can find Package (552). Despite asking these Packages are likely to never appear on melpa and in conclusion will not get into nix. Some people are using them via straight.el. Is there a way these scripts can be used to integrate them in nix as a additional repository?

nix-prefetch-zip does not exist

$ cabal run melpa2nix -- --threads 4 --melpa ../melpa --stable --work /tmp/melpa --output /tmp/melpa-stable.nix shell-switcher
Preprocessing library emacs2nix-0.1.0.0...
In-place registering emacs2nix-0.1.0.0...
Preprocessing executable 'melpa2nix' for emacs2nix-0.1.0.0...
Running melpa2nix...
PackageException "shell-switcher" (ProcessFailed "nix-prefetch-zip" ["--url","https://github.com/DamienCassou/shell-switcher/archive/2c5575ae859a82041a4bacd1793b844bfc24c34f.tar.gz","--name","shell-switcher-2c5575ae859a82041a4bacd1793b844bfc24c34f-src"] nix-prefetch-zip: runInteractiveProcess: runInteractiveProcess: exec: does not exist (No such file or directory))

shell.nix does not work anymore

{ nixpkgs ? import ./nixpkgs {}, profiling ? false }:

let
  inherit (pkgs.haskell) lib;   # <-- pkgs doesn't exist
  addCertPath = drv:
    lib.overrideCabal drv
    (args: args // {
      shellHook = ''
        export SSL_CERT_FILE="${cacert}/etc/ssl/certs/ca-bundle.crt"           # <-- cacert doesn't exist
      '';
    });
  addBuildTools = drv:
    lib.addBuildTools drv
    [
      emacs cabal-install               # <-- emacs and the others doesn't exist
      nix nix-prefetch-scripts
      bazaar cvs curl darcs fossil git mercurial subversion
    ];
in
addBuildTools (addCertPath (import ./default.nix { inherit nixpkgs profiling; })).env

SSL_CERT_PATH seems to be ignored

OSX doesn't have /etc/ssl/certs/ca-certificates.crt, so I use the cacert package that nix provides. But emacs2nix doesn't want to use it

$ SSL_CERT_PATH=$HOME/.nix-profile/etc/ssl/certs/ca-bundle.crt ./melpa-packages ...
...
PackageException "abl-mode" (ProcessFailed "emacs" ["--batch","-l","melpa/package-build.el","-l","/Users/gridaphobe/Source/emacs2nix/scripts/checkout.el","-f","checkout","melpa/recipes/abl-mode","abl-mode","/tmp/melpa2nix/abl-mode"] (Died 255 "Command '(env LC_ALL=C /nix/store/n7d0s1wcaffsd4x04apzx4hnllwswh5c-coreutils-8.25/bin/timeout -k 60 600 git clone https://github.com/afroisalreadyinu/abl-mode.git /tmp/melpa2nix/abl-mode/)' exited with non-zero status 128: Cloning into '/tmp/melpa2nix/abl-mode'...\nfatal: unable to access 'https://github.com/afroisalreadyinu/abl-mode.git/': error setting certificate verify locations:\n  CAfile: /etc/ssl/certs/ca-certificates.crt\n  CApath: none\n\n"))
(NoVersion,"")

Note that the error message still lists /etc/ssl/certs/ca-certificates.crt as the cert path.

I think there are two issues here:

The exports in the shell scripts are wrong. If I add a quick echo $SSL_CERT_PATH I see the default /etc/... path. This is probably easy to fix, but I don't know enough bash to help :(
Even if I hard-code the path to the nix-local one, I still get the original error. git grep SSL only shows the shell scripts, so I'm not sure what's going on here, perhaps the subprocesses aren't inheriting the parent environment?

createProcess: runInteractiveProcess: exec: does not exist

$ ./melpa-stable-packages.sh --melpa melpa/ -o ~/nixpkgs/pkgs/applications/editors/emacs-modes/melpa-stable-generated.nix
/nix/store/1rr461bkw19bg4mhlc7hj2vahilaczyx-ghc-7.10.3/bin/ghc-pkg:
createProcess: runInteractiveProcess: exec: does not exist (No such file or
directory)

I'm running latest emacs2nix on top of channels/nixos-16.09.

curl: certificate not found

When trying to update the nongnu repos in https://github.com/nix-community/emacs-overlay, we are getting this curl error:

++ /nix/var/nix/profiles/per-user/root/channels/nixos/pkgs/applications/editors/emacs/elisp-packages/update-nongnu
in process curl 'https://elpa.nongnu.org/nongnu/archive-contents': died with exit code 60:
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This is due to the expiration of the root certificate from let's encrypt: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

Since the nixpkgs is not updated in this repo, the openssl version is too old to be able to handle the transition to the new certificate (from my understanding, I am not an expert in openssl).

Test command

./nongnu-packages.sh --names ./names.nix -o test

Temporary workaround

diff --git a/shell-fetch.nix b/shell-fetch.nix
index d0b36c6..0732317 100644
--- a/shell-fetch.nix
+++ b/shell-fetch.nix
@@ -1,6 +1,7 @@
 let
-  nixpkgs = import ./nixpkgs.nix;
+  nixpkgs = import <nixpkgs> {};
   emacs2nix = import ./default.nix;
+
 in
 
 with nixpkgs;
@@ -9,7 +10,7 @@ stdenv.mkDerivation {
   name = "interactive-${emacs2nix.name}-environment";
   nativeBuildInputs = [
     emacs nix nix-prefetch-scripts
-    bazaar cvs curl fossil git mercurial subversion
+    breezy cvs curl fossil git mercurial subversion
     emacs2nix
   ];
   shellHook = ''

Very slow to start

Each time I want to update the emacs packages in nixpkgs, emacs2nix starts by building hundreds of Haskell packages. This is very annoying.

Package build invocation has changed (again)

I intend to work on this today, so I'm really just leaving this as a breadcrumb should anyone else run into it and intend to work on it.

Migrate repo to nix-community

@ttuegel Would you be open to moving this repository to https://github.com/nix-community?

I think PRs such as #59 would be good if we could merge sooner rather than later, and that's much easier with more maintainers.

I would be very happy to help you out with this migration if you are willing to.

Increase default threads per CPU

One thread per CPU is used by default. Generating packages is heavily IO-bound. My own experiments suggest that four threads per CPU works better. See also #16.

Running melpa-stable-packages.sh starts bash with no other indication

$ ./melpa-stable-packages.sh --threads 4 --melpa ../melpa --output melpa-stable-package.nix shell-switcher
bash-4.3$

what should I do now?

Marmalade support?

I guess I'm really just opening this as a placeholder: I've actually got a branch that includes marmalade support...but it can't work because the https://marmalade-repo.org/ certificate is self-signed, and although I added support for skipping the cert check into elpa2nix, nix-prefetch-url has no way to make such an exception.

emacs2nix --help is unclear

$ ./melpa-stable-packages.sh 
Preprocessing library emacs2nix-0.1.0.0...
In-place registering emacs2nix-0.1.0.0...
Preprocessing executable 'melpa2nix' for emacs2nix-0.1.0.0...
Running melpa2nix...
Missing: --melpa DIR (-o|--output FILE)

Usage: melpa2nix [-t|--threads N] --melpa DIR [--stable] --work DIR
                 (-o|--output FILE) [--index-only] [PACKAGE]
  Generate Nix expressions from MELPA recipes

what do all these options mean?
when I pass --stable, I get Invalid option --stable
same for --work

org-packages.sh fails

Fails with Wrong type argument: listp, <html>

rasen@omicron ~/t/m/emacs2nix> ./org-packages.sh \
                                     -o /home/rasen/prg/nixpkgs/pkgs/applications/editors/emacs-modes/org-generated.nix  \
                                     --names names.nix
in process emacs -Q --batch --load /nix/store/xbxgn357wz474h0qccbgf12f8i89q2rw-emacs2nix-0.1.0.0-data/share/ghc-8.2.2/x86_64-linux-ghc-8.2.2/emacs2nix-0.1.0.0/scripts/elpa2json.el --eval '(print-archive-contents-as-json "/run/user/1000/elpa2nix-archive-contents-24506-0")': died with exit code 255:
Wrong type argument: listp, <html>

master branch at 752fe1b

Drop deprecated packages

Upstream has deprecated CVS and EmacsWiki packages because they cannot be authenticated. They also cannot have stable versions, so we should drop these packages.