Right now, the NotTrusted error displays at most once every 5 seconds, but it's not sensitive to which hook it runs on.

This can happen:

1. User types shadowenv trust and presses enter
2. pre-exec hook runs and reports an error
3. shadowenv trust runs successfully, prints no output
4. the prompt is rendered

The end result is that it looks as if shadowenv trust failed somehow.

We get some information about which hook is running from hookbook. If we would propagate that information into Shadowenv, we could intelligently short-circuit error printing in this case.

It might even make sense to only print errors on prompt generation. (bash-prompt, zsh-precmd, or --on-event fish_prompt)

I can implement the Y-Combinator:

(define y (lambda (f) ((lambda (x) (f x x)) (lambda (x) (f x x)))))

(y (lambda (x y) (x y)))

$ shadowenv exec env
restriction error: max call stack exceeded

Traceback:

  In main, function __shadowenv__foo.lisp
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda
  In main, lambda

error while evaluating shadowlisp

Why does the language need lambda at all? Having a non-recursive let already allows for non-recursive variable declarations, and I don’t think much more is required to describe every sensible configuration one would want.
The error messages are rather non-descriptive, as one can see (no lines).

I’d go even further and argue that everything in “Functions and Macros” besides let should be removed.

Full lambda & application semantics makes implementation of the format non-trivial (goes from a data specification format to a full turing-complete language, which has to be tamed by having an arbitrary maximum stack).

I could not figure out how to concatenate list of strings. For example, I would like to modify the PS1 prompt variable by prepending a string. The function join is supposed to work on lists but in my experience it won't take a list as an input. So this for some reason works (join "," "item1" "item2") but this does not (join "," (list "item1" "item2")). It would also be helpful to have a function that splits a string by white space or by a separator passed as parameter and returns a list of strings. IMHO such and similar string manipulation functions would be quite useful while being just a generalization of what's already available for path manipulations.

env/remove-from-pathlist removes items from the pathlist, but doesn't preserve the index from which they were removed, so when the user leaves the directory in which they were removed, the items are likely not reinserted to the correct location in the pathlist.

We could solve this by keeping richer undo data.

I propose adding -C|--change-dir to Shadowenv, which would change to the indicated directory before executing the command. Behaviour per command:

• shadowenv help: no obvious change
• shadowenv init: no obvious change
• shadowenv hook: evaluate the shadowlisp from the indicated directory rather than the current one.
• shadowenv trust: trust the ./.shadowenv.d from the indicated directory rather than the current one.
• shadowenv exec: Change to the indicated directory before loading the environment and executing the indicated process.
• shadowenv diff: unclear. Probably none? Conceivably we could apply the environment changes indicated by the directory's shadowlisp first.

Example usage:

shadowenv -C /my/project exec irb

I don't actually have any need motivating this, but I think it'd be a nice, obvious, and relatively simple feature.

The conversation on #4 has me thinking a bit about how .shadowenv.d should be structured. I think maybe this makes sense:

.shadowenv.d
├── .error-1-12415
├── .gitignore
├── .trust-ab42714f
└── 500_default.scm

• *.scm: shadowlisp files
• .trust-<fingerprint>: trust files (now with a leading .)
• .error-<n>-<pid>: sentinel file for error n on pid pid, in this project.
• .gitignore: contains:

/.*
!/.gitignore

As a user, it feels more useful to see only the *.scm files.

A user has set REVISION to "$(git ...)", which is kind of useful but:

a) better from a security standpoint to not allow this; and
b) it's shell-dependent.

Let's find a way to pass data to shells as literals.

Shitty proof of concept

$ export VAR=a:b:c:d
$ cat inner/.shadowenv.d/500_custom.lisp
(env/prepend-to-pathlist "VAR" "c")
$ cd inner
activated shadowenv
$ echo $VAR
c:a:b:c:d
$ cd ..
$ echo $VAR
c:a:b:c:d

Undoing this properly would also require similar richer undo data including indexes as described in #16.

shadowenv exec -- nginx -g 'daemon off;'
shadowenv exec -d "${__shadowenv_data}" bash

Mostly useful when the shell shim isn't loaded, shadowenv exec would load the shadowenv from the current directory and exec the indicated command/process.

Here I've made $__shadowenv_data a flag rather than a positional argument, because I anticipate using this primarily in cases where Shadowenv is not loaded as a shell hook.

The first positional argument will be interpreted as the program, and extraneous arguments afterward will be an error.

Passing -- as the first positional argument indicates that all subsequent arguments should be interpreted as the argv of the process to be exec'd.

Example Use-Case

I'd like to be able to write out a LaunchAgent whose ProgramArguments would evaluate a Shadowenv without having to use ['bash', '-c', '...'], e.g.:

    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/bin/shadowenv</string>
        <string>exec</string>
        <string>--</string>
        <string>myprogram</string>
        <string>--with-some-argument</string>
    </array>
    <key>WorkingDirectory</key>
    <string>/path/to/thing/with/env</string>

Interested to hear about how Shopify is integrating Ruby into shadowenv and swapping versions between directories. Right now, most projects come with a .ruby-version file but I can't see a sane way of pulling the value from that file and either setting an environment variable (for tools like rbenv) or shimming in particular versions. I see the documentation moves some values in the environment but it doesn't really keep the source of truth as the .ruby-version file. Would something like (open ".ruby-version") somewhere in the shadowenv configuration cover this?

Tested in zsh.

When shadowenv is not trusted, it prints before and after commands run. This means that we see the message twice. It also means that when we actually run shadowenv trust, we see the message (once). I suggest printing only after the command is run, solving both problems.

So, it would be stupid to think that Shadowenv could allow me to replace my Python and Go virtualenvs? Thanks!

For removing "removing" from pathlist.

error: component 'rustfmt' for target 'x86_64-unknown-linux-gnu' is unavailable for download for channel 'nightly'

rust-lang/rustup#2047

See also:
rust-lang/rustup#1566
rust-lang/rustup#1958

At best-practices.md:38 the original URL https://devproductivity.io/dev-shopifys-all-purpose-development-tool/index.html now redirects to something that is unrelated and does not talk about the dev tool. I found it in the wayback machine at https://web.archive.org/web/20210314190618/https://devproductivity.io/dev-shopifys-all-purpose-development-tool/index.html.

• vim
• emacs
• vscode
• sublime?
• atom?
• RubyMine / jetbrains family? Possibly via EnvFile
• ...

This should, in principle, be pretty easy to build in each case as long as we can:

• Run some code when we "change directory" or open a file;
• Shell out and get the output back; and
• Mutate the process environment.

direnv.vim is probably close to a solution but has no license. Direnv is MIT-licensed so we could just confirm that if it would be useful to start from there.

Some code we wrote to do this in a private repo follows, may be a helpful reference:

# frozen_string_literal: true

require('dev')

module Dev
  class Shadowenv
    module Applicator
      extend Dev::Util::PrivateConstants

      private_constants do
        ASCII_FIELD_SEPARATOR  = "\x1F"
        ASCII_RECORD_SEPARATOR = "\x1E"
        OPCODE_SET_UNEXPORTED  = 1
        OPCODE_SET_EXPORTED    = 2
        OPCODE_UNSET           = 3
      end

      class << self
        # applies a shadowenv found by the normal rules (recursing upward to
        # parents until one contains a `.shadowenv.d`) to the provided
        # `Dev::Context`'s `env`.
        def apply(ctx, dir)
          Dev::Shadowenv::Installer.ensure_installed

          data = ctx.shadowenv_data || ''
          out, err, stat = ctx.capture3('shadowenv', 'hook', '--porcelain', data, chdir: dir)
          raise(Dev::Bug, "shadowenv failed: #{err}") unless stat.success?

          # Fields separated by 0x1F; records separated by 0x1E.
          # example message:
          #   0x02 0x1F F O O 0x1F b a r 0x1E
          #   ^         ^          ^
          #   export    FOO=       bar
          out.split(ASCII_RECORD_SEPARATOR).each do |record|
            opcode, name, value = record.split(ASCII_FIELD_SEPARATOR)
            case opcode.bytes.first
            when OPCODE_SET_EXPORTED
              ctx.env[name] = value
            when OPCODE_UNSET
              ctx.env[name] = nil
            when OPCODE_SET_UNEXPORTED
              # __shadowenv_data specifically is set, but not exported to
              # subprocesses. It lives only in the shell session itself as an
              # unexported variable.
              if name == '__shadowenv_data'
                ctx.shadowenv_data = value
              else
                raise(
                  Dev::Bug,
                  "shadowenv gave us an unexported value that we didn't know what to do with: #{name}",
                )
              end
            end
          end
          nil
        end
      end
    end
  end
end

There are a bunch of built-in functions provided by ketos that we don't document in Shadowlisp in an effort to prevent people from using (and with the intention of removing them someday).

We should actually find a way to mask them and prevent them from being called.

Related: #63

👋 I'm using Shadowenv in a personal project, and I'd like to expose a path that is relative to the directory where the Shadowenv directory lives. Is that currently supported? If not, is this something that aligns with the project, and therefore I can contribute it myself? If so, I'd appreciate any pointers you can give me.

I have a local directory that can be accessed over NFS from other computers. When I set it as trusted locally, it is not trusted when accessed over NFS. When I set it trusted while accessing through the NFS filesystem, it is not trusted as local directory. Is this by design? Can it be fixed? Thank you.

If Shadowenv comes across an invalid dev.yml file, it tries to take the blame. In the output below, you can see it ends with This is a bug in dev or shadowenv., which isn't quite true.

Also, if possible, it would be great if shadownev would report which value was invalid.

┏━━ There was an issue parsing your dev.yml's env section ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
┃ must be a hash with keys and values all Strings
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ (0.0s) ━━
𝒾 generate shadowenv for custom_env
┏━━ There was an issue parsing your dev.yml's env section ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
┃ must be a hash with keys and values all Strings
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ (0.0s) ━━
✗ generate shadowenv for custom_env: failed!
✗ Error Reason:
✗ Dev::Shadowenv::Dep::CannotBuild: dev.yml env was invalid
✗ /opt/dev/lib/dev/shadowenv/deps/custom_env.rb:43:in validate' ✗ /opt/dev/lib/dev/shadowenv/deps/custom_env.rb:21:in compile'
✗ /opt/dev/lib/dev/shadowenv/dep.rb:48:in meet' ✗ /opt/dev/lib/dev/dep.rb:22:in call'
✗ /opt/dev/lib/dev/task.rb:261:in block (2 levels) in run_deps' ✗ /opt/dev/lib/dev/stats/statsd.rb:205:in time'
✗ /opt/dev/lib/dev/task.rb:260:in block in run_deps' ✗ /opt/dev/lib/dev/task.rb:244:in each'
✗ /opt/dev/lib/dev/task.rb:244:in with_index' ✗ /opt/dev/lib/dev/task.rb:244:in run_deps'
✗ /opt/dev/lib/dev/task.rb:234:in block in call_with_dir_set' ✗ /opt/dev/lib/dev/task/formatter.rb:36:in phase'
✗ /opt/dev/lib/dev/task.rb:233:in call_with_dir_set' ✗ /opt/dev/lib/dev/task.rb:187:in block in call'
✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/util.rb:121:in with_dir' ✗ /opt/dev/lib/dev/task.rb:186:in call'
✗ /opt/dev/lib/dev/task_engine/executor.rb:34:in block (2 levels) in do_execute' ✗ /opt/dev/lib/dev/stats/statsd.rb:205:in time'
✗ /opt/dev/lib/dev/task_engine/executor.rb:33:in block in do_execute' ✗ /opt/dev/lib/dev/task_engine/executor.rb:30:in each'
✗ /opt/dev/lib/dev/task_engine/executor.rb:30:in do_execute' ✗ /opt/dev/lib/dev/task_engine/executor.rb:12:in block in execute!'
✗ /opt/dev/lib/dev/util.rb:78:in timed' ✗ /opt/dev/lib/dev/task_engine/executor.rb:11:in execute!'
✗ /opt/dev/lib/dev/commands/up.rb:40:in block in run_locked' ✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/util.rb:121:in with_dir'
✗ /opt/dev/lib/dev/commands/up.rb:39:in run_locked' ✗ /opt/dev/lib/dev/commands/up.rb:33:in block in call'
✗ /opt/dev/lib/dev/commands/up.rb:62:in with_locks' ✗ /opt/dev/lib/dev/commands/up.rb:32:in call'
✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/base_command.rb:24:in block in call' ✗ /opt/dev/lib/dev/stats/statsd.rb:205:in time'
✗ /opt/dev/lib/dev/command.rb:25:in statsd_time' ✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/base_command.rb:23:in call'
✗ /opt/dev/lib/dev/command.rb:13:in call' ✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/executor.rb:17:in block (2 levels) in call'
✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/executor.rb:39:in block (2 levels) in with_logging' ✗ /opt/dev/vendor/deps/cli-ui/lib/cli/ui/stdout_router.rb:164:in with_id'
✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/executor.rb:38:in block in with_logging' ✗ /opt/dev/vendor/deps/cli-ui/lib/cli/ui.rb:143:in log_output_to'
✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/executor.rb:37:in with_logging' ✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/executor.rb:15:in block in call'
✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/executor.rb:47:in block (2 levels) in with_traps' ✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/executor.rb:57:in twrap'
✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/executor.rb:46:in block in with_traps' ✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/executor.rb:57:in twrap'
✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/executor.rb:45:in with_traps' ✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/executor.rb:14:in call'
✗ /opt/dev/lib/dev/entry_point.rb:19:in call' ✗ /opt/dev/bin/dev:41:in block in

'
✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/error_handler.rb:74:in handle_abort' ✗ /opt/dev/vendor/deps/cli-kit/lib/cli/kit/error_handler.rb:20:in call'
✗ /opt/dev/bin/dev:40:in `'
✗ Failed to build shadowenv for custom_env.
✗ This is a bug in dev or shadowenv.

🤷 generate shadowenv for custom_env: failed!

Additional Context

Late this afternoon, Shadowenv started failing for members of the Shopify Fulfillment team. Turns out, this was because we had a port value that was a number not a string. We're unsure why this started failing, as this value has been the same for quite some time, and Shadowenv was enabled yesterday or the Friday before. We were able to fix the root problem here https://github.com/Shopify/fbs/pull/3266, but shadowenv shouldn't try to take the blame if it's a problem with the config file.

Reproduction steps:

mkdir ~/foo
cd ~/foo
rm -r ../foo

Every prompt there-after crashes with:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1188:5
stack backtrace:
   0:        0x1046e3ae5 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h54942195382ebc43
   1:        0x10470bfc1 - core::fmt::write::h03a4fdeef33f04e5
   2:        0x1046e367b - std::io::Write::write_fmt::h2f54eb7e3d98f93c
   3:        0x1046e6353 - std::panicking::default_hook::{{closure}}::h5b49f92bd56be4e7
   4:        0x1046e607f - std::panicking::default_hook::he9bfa0e01e764f9b
   5:        0x1046e6a0b - std::panicking::rust_panic_with_hook::hfdce9374aac935e9
   6:        0x1046e6579 - rust_begin_unwind
   7:        0x10470edac - core::panicking::panic_fmt::h344046753bf14afc
   8:        0x1047160e9 - core::result::unwrap_failed::h5be3faa2da19b654
   9:        0x1044c379b - shadowenv::main::hcf989410cebb90f5
  10:        0x1044cc3a6 - std::rt::lang_start::{{closure}}::hda31d0516cac88b4
  11:        0x1046e6418 - std::panicking::try::do_call::hdad9ba23601b147d
  12:        0x1046f484f - __rust_maybe_catch_panic
  13:        0x1046ed5b5 - std::rt::lang_start_internal::h820f58b1bdc07d9f
  14:        0x1044c3eb9 - main

Shadowenv could have a feature to convert other environment manipulation tools' configs to a format it can understand. For example:

# .env
A=b

$ shadowenv assimilate
assimilated .env as .shadowenv.d/050-dotenv.lisp

; 050-dotenv.lisp
(env/set "A" "b")

This is obviously the most trivial case, but we could do similar things for:

• .envrc
• .ruby-version
• and so on.

Because of Shadowenv's lack of ability to do much of anything dynamic at load-time, this would essentially function as a "compiler" for these environment files.

One trick to try to make these files at least marginally portable would be to replace the user's home directory in any outputs with (concat (env/get "HOME") ...).

If we wanted to keep tabs on a mapping between the original and the assimilated version, we could encode a checksum/hash of the source file in the assimilated file as a comment on the first line. With this, shadowenv or dev could potentially observe these files when entering a directory and inform the user about out-of-date-ness.

Expected: cd into subdir, and we're still trusted
Actual: we're not, and shadowenv trust doesn't even help

Sorry for asking there, but I couldn't find any other discussion channel.

In README there is fragment:

Unlike other tools like direnv, this has the interesting property of allowing us to do things like simulate chruby reset upon entry into a directory without the user having chruby installed (and undo these changes to the environment when cd'ing back out)

But it is still possible in direnv via layout function. So is there anything more except different configuration language (sh vs Ketos)?

Shopify  > cp -a dev-shopify-io up.dev
Shopify  > cd up.dev
shadowenv failure: bad signature

This should be made more actionable.

test 1234

These provide an unnecessary amount of power, as identified in #61.

Related: #62

Shadowenv should behave better when there's a shadowlisp error:

Rather than claiming activation, we should probably suppress the activation message, and maybe even reset the environment to the root/undone state.

Hi there,

When I'm using pipes in the command line such as:

echo "foo" | wc

I get:

-bash: child setpgid (3238 to 3232): Operation not permitted

(but the command actually works)

I tried with an empty .bash_profile and .bashrc no problem.
I also tried with zsh and sh, no problem.

This only occurs when I'm loading dev.sh from my .bashrc in Terminal or ITerm2.
More specifically, if I comment the eval "$(/usr/local/bin/shadowenv init "${__shellname}")", the problem disappears.

Config:

$ system_profiler SPSoftwareDataType
    System Software Overview:

      System Version: macOS 10.14.2 (18C54)
      Kernel Version: Darwin 18.2.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal
      Secure Virtual Memory: Enabled
      System Integrity Protection: Enabled

$ bash --version
GNU bash, version 5.0.0(1)-release (x86_64-apple-darwin17.7.0)

$ sh --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)

$ zsh --version
5.3 (x86_64-apple-darwin18.0)

Here the content of my .bashrc file:

if [[ -f /opt/dev/dev.sh ]] && [[ $- == *i* ]]; then
  source /opt/dev/dev.sh
fi

And the problem occurs with an empty .bash_profile.

Any idea?

It's documented but in actuality it's just a stub currently.

(expand-path "~/foo") should be "/Users/me/foo". I'd expect this to work approximately like File.expand_path(x) in ruby.

The numbered prefix gives the impression that the last would take precedence.

Per documentation: https://doc.rust-lang.org/std/fs/fn.read_dir.html

// The order in which read_dir returns entries is not guaranteed. If reproducible
// ordering is required the entries should be explicitly sorted.

$ tail -n +1 .shadowenv.d/*.lisp
==> .shadowenv.d/000_test.lisp <==
(env/set "FOO" "BAR")

==> .shadowenv.d/100_test.lisp <==
(env/set "FOO" "BAZ")

==> .shadowenv.d/200_test.lisp <==
(env/set "FOO" "QUX")

$ echo $FOO
BAR

Hi there, I am curious why there is no Linux binary on the 2.1.0 release.

Thank you.

I have my atom linter-rubocop package configured to execute /opt/dev/bin/shim bundle exec rubocop, which has worked just fine until recently. Looking into the error log I see an error like

[Linter] Error running RuboCop Error: �[1;34mactivated �[38;5;249ms�[38;5;248mh�[38;5;247ma�[38;5;246md�[38;5;245mo�[38;5;244mw�[38;5;243me�[38;5;242mn�[38;5;241mv�[38;5;240m �[1;34m(node:v8.12.0, ruby:2.5.3)�[0m

Which seems to just say that shadowenv is working fine, but it breaks the rubocop linter. It seems that it is running in a non-interactive shell so it should be easy to not print that status message in this situation.

If a script or app cds into a shadowenv directory (ex: dev's integration feature) that directory might be untrusted and the "directory contains untrusted shadowenv program" error massage would be shown. However if the user is unaware of this change in directory they might be confused in thinking that it's their current directory that's the issue.

This could be avoided by simply including the directory path in the error message.

Could just be a diff, or could color-code additions and deletions in a merged version.

There are some broken links to the local pages on the getting started page.

Just curious if its possible to use it in windows cmd provided that I can successfully compile with windows target.

Is there any special setup I would need to do?

The repo links to shopify.github.io/shadowenv/, but there is no website available at that domain.

Hi,
Is there an example of a Python environment somewhere? Or an example of the command: "source"?
Thanks!