Giter Club home page Giter Club logo

dlx's Introduction

dlx

dlx is a development tool that provisions temporary development environments. It uses lxd and zfs to make efficient, copy-on-write, workspaces from a user-provided template.

Turn a spare computer into a development environment server in just a few minutes.

Why dlx?

I built dlx because I bounce between laptops frequently and I wanted a consistent development environment without having to use something like Syncthing to sync my code. I had an older Intel NUC in the closet doing not very much work, and I wanted to find a way to use it for development.

Components

dlx is composed of the following components:

* [lxd](https://linuxcontainers.org) - the container management system that runs on a remote host, either locally on your network or in the cloud.
* [dlx](https://github.com/bketelsen/dlx) - client program that runs on your local computer.
* distrobuilder - a tool that builds a Linux container image from a template
* Optionally [zfs](https://wiki.ubuntu.com/ZFS)
* Optionally [tailscale](https://www.tailscale.com)

LXD is a containerization technology that is used to create a container that can be used to run a development environment or longer running services. It is a lightweight, easy to use, and highly available containerization technology. LXD has several different options for storage, the best of which for these purposes is zfs. btrfs is also supported. Either will be significantly faster than other LXD storage options, and either supports Copy on Write, which is a feature that is used to make the container's filesystems more efficient.

dlx is a CLI client that uses the LXD API to provision a development environment.

As a convenience, all of the lxc client commands are available as subcommands in dlx. For example, dlx lxc list is equivalent to lxc list. This is possible because the lxc client code is written in Go and also uses the cobra library. Whether this is a good idea or not remains to be seen. For now, it's a nice convenience because it means you're not required to install the lxc client on your local development machine.

How does it work?

dlx uses lxd to provision a new container from a custom Linux image. A typical workflow might look like this:

dlx create projectname

dlx instructs the LXD daemon to create a container called projectname from the image dlxbase. dlxbase is a convention that is used to refer to a base LXD image that exists locally on the LXD server. The documentation includes information on how to edit the template image and create it on the LXD server. This dlxbase image is used to create all containers created by dlx.

dlx also creates a projectname directory on the LXD host filesystem and mounts it into the container in the /home/user directory. This is where your source code should be stored. This allows you to backup your source code from the host using conventional Linux backup tools.

Depending on your needs, you may configure LXD to bridge network traffic between the containers and your local network, giving each container an IP address on your LAN. This is how I use dlx at home. You can also use something like tailscale to provide connectivity to the containers from your VPN network, which would allow you to use dlx to provision a development environment on a remote host that is not on your LAN and can't provide routeable IP addresses.

After the container is provisioned, you can access it over SSH using the IP address of the container. This works perfectly for VS Code's remote SSH feature. If you've configured bridged traffic, the containers appear as peers on your local network, so you can use ssh to connect to them directly. If you're using Tailscale, you can setup Magic DNS to achieve the same effect. Note: Tailscale isn't required for this to work, you can use Wireguard or other VPN solutions, or you can use LXD's built-in port forwarding features to access your containers.

When you're done with the container, you can stop it with dlx stop projectname, and optionally delete it with dlx rm projectname. The host's projectname directory is not deleted, which means that you can later create a new container with the same project name and get the same source code.

Conventions and Assumptions

dlx makes several assumptions about your setup. Not all of these are required, but they are recommended.

* The LXD server is running Ubuntu 18.04 or higher.
* LXD is installed via `snap` on the server.
* Your account on the server has sudo privileges.
* Your account on the server has public and private keys in `~/.ssh`. These are mounted in the containers to allow you to SSH into the containers.
* You have passwordless SSH access to the LXD server.
* Your base LXD image `dlxbase` is configured to create a user with the same username you use on your laptop/desktop.

Using these conventions allows dlx to create a container that requires no extra work to access.

Installation

See the documentation

Usage Scenarios

Locally

Locally I have an LXD server running headless in my closet. I have LXD using a bridge to connect the containers to my local network. When I create a new container, they're immediately available on my local network by using the container name as the hostname.

In the Cloud

I also have an instance of LXD running in the cloud. The LXD server is using the default lxdbr0 bridge to route traffic to and from the containers. The dlxbase image is configured to install Tailscale in each container. After the container is provisioned I use dlx connect to access the container and run tailscale up to authorize the container (this could be automated with a Tailscale Personal Access Token, but I haven't done this yet). Because I have Magic DNS configured on my Tailscale account, the containers are available by hostname in DNS lookups. SSH and VS Code Remote SSH work the same as they do locally.

Security

Contributing

License

Documentation

Documentation (Work in progress!) is available at dlx.rocks.

dlx's People

Contributors

bketelsen avatar christianboehm avatar moonmeister avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dlx's Issues

How to uninstall everything in order to start over

During installation & config of the .05 release I hit at least one failure on Ubuntu 18.04 with SNAP LXD v3.13

$ devlx template create guitemplate --profile gui --provisioners vscode
~ Creating template guitemplate
✗ Unable to create template: waiting for container start: Common start logic: Missing source '/tmp/.X11-unix/X1' for disk 'X1'

But then I realized I didn't see any info in the README files about how to completely uninstall and start over?

thanks

Profiles need to be templates

Profiles are specific to my environment: eg X Display 1, not 0, using bridge br0.
These either need to be specified as params during init or discovered from the environment when running.

release 0.0.3 - fails on - lxdev template create guitemplate --profile gui --provisioners vscode

$ lxdev version
0.0.3

$ lxdev config -c
~ Create configuration
✓ Default configuration file created
✓ Configuration created

bmullan@dadspc:~$ lxdev config -t
~ Create configuration
✓ Templates created
✓ Configuration created

bmullan@dadspc:~$ lxdev profile -w gui
~ Managing profile gui
~ Creating profile gui
✓ Creating profile gui
✓ Managing profile gui

bmullan@dadspc:~$ lxdev profile -w cli
~ Managing profile cli
~ Creating profile cli
✓ Creating profile cli
✓ Managing profile cli

bmullan@dadspc:~$ lxdev template create guitemplate --profile gui --provisioners vscode
~ Creating template guitemplate
✗ Unable to create template: waiting for container start: Common start logic: Missing parent 'enp5s0' for nic 'eth0'

I don't know the interpretation of the above message about - Missing parent 'enp5s0' but my "host" system uses WiFi for internet/network access and not ethernet. But that message may not have anything to do with that as I realize the reference to "enp5s0' could be for an interface "inside" the container.

$ ifconfig
enp10s0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 08:60:6e:87:a8:58 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 728370 bytes 282712698 (282.7 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 728370 bytes 282712698 (282.7 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lxdbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 10.105.172.1 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::d04e:e6ff:fe04:db22 prefixlen 64 scopeid 0x20
ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet)
RX packets 422055 bytes 281775857 (281.7 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 361640 bytes 131269725 (131.2 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:58:bc:68 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

wlxc8d719c17a3d: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.130 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::55f4:b889:ffdc:500b prefixlen 64 scopeid 0x20
ether c8:d7:19:c1:7a:3d txqueuelen 1000 (Ethernet)
RX packets 533271 bytes 640898273 (640.8 MB)
RX errors 0 dropped 38 overruns 0 frame 0
TX packets 185614 bytes 31731616 (31.7 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

make deps fails - go get github.com/gobuffalo/release: open /usr/local/go/bin/release: permission denied

Brian

I finally got around to trying to install lxdev
my system:
ubuntu 18.04
lxd v3.9
go version go1.11.5 linux/amd64

So I followed your github Install steps:

git clone https://github.com/bketelsen/lxdev
cd ~/lxdev
make deps

~/lxdev$ make deps
go get github.com/gobuffalo/release: open /usr/local/go/bin/release: permission denied
Makefile:25: recipe for target 'deps' failed
make: *** [deps] Error 1

Note: I am not a GO programmer at all but I installed GO fine and ran some test builds with it (ie hello) just to check that it compiled okay and "hello" ran ok so I know GO is okay I think.

I did check: github.com/gobuffalo
and there is a "release" folder there?

Any idea what I am doing wrong
thanks
brian

using release 0.0.2 - fails to create new container

finally found out about the 0.0.2 release and downloaded it.

I tried to copy your video and got to creating a new container and it failed.

$ lxdev create newcn
~ Creating container newcn
✗ Unable to create container: creating container: Must specify one of alias, fingerprint or properties for init from image

I am on ubuntu 18.04 and using SNAP LXD

delete infers stop?

A brief look at the code looks like you need to call stop before you call delete, so inferring stop from delete might be good.

permissions problem - lxdev template create clitemplate --profile cli --provisioners go,yadm

When executing:
lxdev template create clitemplate --profile cli --provisioners go,yadm

eventually see the following:

Initialized empty shared Git repository in /home/ubuntu/.yadm/repo.git/
The authenticity of host 'github.com (192.30.253.112)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,192.30.253.112' (RSA) to the list of known hosts.
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
ERROR: Unable to fetch origin [email protected]:bketelsen/dotfiles

Snap package

Hey Brian,

Interested in a snap package for this? We could wire it up to build and push to the edge channel in check in.

I could help you out.

make deps fail

Dear Brian,

make deps fail with error message: cannot load github.com/bketelsen/libgo/events
if I try to curl your url on github (github.com/bketelsen/libgo/events) I got a 404 error

Misc install issues

Hey Brian, Thanks for your work on this, I'm super excited to try this out. I just got it installed but had some issues along the way. Thought I'd share them in case you're able to make the install process more robust.

OS: Ubuntu 18.10

On running make deps it started installing various packages but threw an error

 go: launchpad.net/[email protected]: bzr branch --use-existing-dir https://launchpad.net/~niemeyer/gocheck/trunk . in /home/user/development/go/pkg/mod/cache/vcs/f46ce2ae80d31f9b0a29099baa203e3b6d269dace4e5357a2cf74bd109e13339: exec: "bzr": executable file not found in $PATH

After throwing the error the install appeared to continue. It didn't complete though. Manually installing bzr from apt fixed the issue and the complete install was much longer once that was working.

I continued to have issues with running make test and make install both error with

build github.com/bketelsen/lxdev: cannot find module for path home/user/development/lxdev/packrd
make: *** [Makefile:38: test] Error 1

notice the lack of / in front of home ...here is the cmd-packr.go file itself:

// Code generated by github.com/gobuffalo/packr/v2. DO NOT EDIT.

// You can use the "packr clean" command to clean up this,
// and any other packr generated files.
package cmd

import _ "home/user/development/lxdev/packrd"

I'm quite green with go so I'm not sure what or how packrd is but it seems broken...My fix for this was moving lxdev into my GOPATH and setting GO111MODULES=on. With these two changes test and install worked like a charm.

Let me know if I can provide more info or assist, now off to play...

Feature: Prompt to create templates if there are none

When starting, lxdev should check to see if there are any templates to use for new container creation. If there aren't any, a useful help message should display explaining the template and container creation process. related: #10

profile merging doesn't merge

I was wrong about profiles merging. Last applied key wins, which breaks the whole concept.

Need to:

  • Write a parser for run-cmd and packages
  • Merge the parsed data into a template based on the selected base profile
  • create a new profile with the merged data

ghost template on failed

When creating a template fails the template never get's created on the devlx side but exists in lxc. create needs to cleanup the lxc image on error or correctly create its record of the template as being in a failed state. Currently when listing templates in devlx after this happens it doesn't say no templates...but it doesn't list any either. Doing an rm doesn't help as due to #12 templates aren't actually deleted from lxc.

Investigate Crostini support

Crostini in ChromeOS uses LXC/D under the covers to provide a linux VM. Can lxdev be tweaked (or forked?) to assist with that?

Deleting Templates

Decide what to do about deleting templates. Perhaps first step is to check to see if any containers exist with the template (which is an lxc image), and if none, perform the delete.

If there are some, we can't delete. Display useful error message.

using host SSH-agent

If we mount the unix socket for the ssh-agent on the host into the LXD container then the host can be the source of truth for the container ssh. SSH_AUTH_SOCK enables us to dynamically set this.

This allows not copying .ssh files into new containers and will enable support for yubikeys/gpg keys through gpg-agent.

More info:
https://discuss.linuxcontainers.org/t/how-to-mount-the-ssh-agent-socket-into-lxd-container/2591/3

I intend to work on this at some point but if anyone else wants to please do...though this might be dependent on #24

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.