Giter Club home page Giter Club logo

balena-virt's Introduction

Labs Logo

Balena Virt

Balena Virt is a suite of tools for virtualising Balena OS.

There are a number of options for using Balena Virt:

  1. Running on a Digital Ocean Droplet, providing an easy way to try Balena Cloud without the need for physical hardware, and to provide a powerful development platform.
  2. Running on other VPS services and hardware. See the advanced setup which provides containers that can be used and steps for using Tailscale for multi-device setups.
  3. Running locally on a Linux desktop machine
  4. Turning a single Intel NUC into a small fleet of devices for testing and development
  5. Using the Balena Virt CLI for custom builds

Universal Environment Variables

Joining a fleet automatically

Images can be configured to automatically join the balenaCloud by passing the API_TOKEN environment variable and FLEET environment variable. When passing these variables, the device will default to production mode, unless also passing in the DEV_MODE environment variable.

API_TOKEN="sdfef2fef123"
FLEET="your-fleet"
DEV_MODE=true # Defaults to true when not configured for joining th Cloud, defaults to false when joining the Cloud

Configuring the environment

Default cores, disk size and memory will mirror the system that balenaVirt is running on.

The default OS image is the latest GENERIC X86_64 (GPT) images.

These defaults can be overridden with environment variables. Here are some examples:

IMAGE_VERSION=2.108.28+rev3 # Image type is always GENERIC X86_64 (GPT)
CORES=4
DISK=8G
MEM=512M

Balena Virt on Digital Ocean

Balena Virt can be run on a Digital Ocean droplet with hardware acceleration. This guide provides a link for $200 of free Digital Ocean credit; an installation script; and steps to forward the ports locally. After installation, you will be able to use virtualised OS locally just like any other device, using the Balena tools, e.g:

balena ssh 127.0.0.1
balena push 127.0.0.1

You can scale the Digital Ocean hardware to your development needs. 8 CPUs, 16GB of memory, and SSD drives make development (such as compiling binaries) far more efficient on Digital Ocean than on much of the IoT hardware used in production, while removing any need for manual configuration of development environments on your local system.

You can see some benchmarks where we compared the performance of Balena Virt on a Droplets against some common hardware.

Step 1: Sign up for Digital Ocean and claim the free credit

Sign up for Digital Ocean. Using the button below you will get $200 of free credit for 60 days; plenty to run a powerful virtual device:

DigitalOcean Referral Badge

You will see a banner at the top of the page indicating that the credit is active and you can continue to the sign up page:

Free credit active: Get started on DigitalOcean with a $200, 60-day credit for new users.

Step 2: Create a droplet.

The Digital Ocean documentation provides guidance on how if you need some helps.

We suggest a minimum of 2GB of RAM. Although as your free credit only lasts 60 days, why not fire up a more powerful machine (don't forget to destroy the machine before your 2 months runs out):

8 CPUs
16 GB Memory
320 GB SSD Disk
6 TB transfer
$96 /mo

Step 3: Run the install script

Connect to your Droplet as the root user via the terminal and run the install script:

curl -fsSL https://raw.githubusercontent.com/balena-labs-research/balena-virt/main/vps/install.sh | sudo sh

The default number cores, disk size and memory of the virtualised device will mirror the system that Balena Virt is running on.

Step 4: Forward the Droplet ports to your local system

Mount the running virtualised OS locally with the following, where ip.ip.ip.ip is the IP address of your remote host (for example your DigitalOcean Droplet IP):

ssh -L 22222:10.0.3.10:22222 \
 -L 2375:10.0.3.10:2375 \
 -L 48484:10.0.3.10:48484 \
 [email protected]

You can then use the Balena CLI to interact with the OS by using the local IP address, for example:

balena ssh 127.0.0.1
balena push 127.0.0.1

Other ports can me mapped locally, for example to interact with services on the device:

ssh -L 80:10.0.3.10:80 \
 [email protected]

Advanced Configuration

Advanced documentation is available in the vps folder here. It includes instructions on using other VPS services, and enabling Tailscale for multiple devices on each service.

Balena Virt on Linux Desktop

Are you running Linux on your desktop? You can start a virtual device on your desktop with the following Docker run commands:

Start the networking requirements:

docker run -it \
    --restart always \
    --cap-add=net_admin \
    --network host \
    ghcr.io/balena-labs-research/balena-virt-networking:latest

Start the virtualised OS:

docker run -it \
    --restart always \
    -v bv_pid:/app/pid \
    --device=/dev/kvm \
    --cap-add=net_admin \
    --network host \
    ghcr.io/balena-labs-research/balena-virt:latest

Start additional devices by running the ghcr.io/balena-labs-research/balena-virt:latest command again.

Balena Virt on Intel NUC

Install Balena OS on your NUC using the generic-amd64 image.

Deploy Balena Virt with the one-click install to turn your NUC into 4 Balena OS devices:

deploy button

Change the default number of devices by adding or removing containers from the ./docker-compose.yml file and deploying manually with balena push.

Accessing the devices

Each device will be assigned its own IP and is accessible when you SSH to the Intel NUC.

Alternatively, you can use Tailscale to expose all the running devices to your local system as if they are on your network. On the Intel NUC run the Tailscale container:

balena run -d \
    --name=tailscaled \
    --restart always \
    -e TS_STATE_DIR=/var/lib/tailscale \
    -v tailscale-state:/var/lib/tailscale \
    -v /dev/net/tun:/dev/net/tun \
    --network=host \
    --privileged \
    tailscale/tailscale tailscaled

Then bring up the service with:

balena exec tailscaled tailscale up --advertise-routes=10.0.3.0/24 --accept-routes --reset

The Tailscale container will provide you a URL to access that adds the device to your Tailscale account.

Then enable the subnets from your Tailscale admin panel to be able to use all the devices locally through the IP addresses they are assigned by Balena Virt.

If you would rather not use Tailscale, you can use SSH to forward a virtualised device to a port on your local system:

ssh -L 22222:10.0.3.10:22222 \
 -L 2375:10.0.3.10:2375 \
 -L 48484:10.0.3.10:48484 \
 [email protected]

Balena Virt CLI

Easily run virtual machines on balenaOS with native performance, powered by QEMU and KVM. Guests can even run balenaOS on balenaOS, allowing multiple containerized applications to run concurrently on one physical hardware device.

Additionally, this allows balenaOS guests to take advantage of QEMU features such as shared backing images and disk snapshots via qcow2 images, rolling back to a previous snapshot, temporary snapshots, pause and resume, and live migration to another host.

Deploy with balena push from the ./cli directory.

Grab a balenaOS image for your guest(s) using the dashboard, or CLI:

$ balena os download genericx86-64-ext -o rootfs.img

Images and configs can be copied to the named data volume for the application using SCP, assuming you have SSH access:

$ scp -P 22222 rootfs.img [email protected]:/var/lib/docker/volumes/${appid}_resin-data/_data/

The config can be edited on your device from the dashboard inside the main container, or using SSH:

$ balena ssh mydevice.local main
~ # vi /data/guests.yml

This disk image would then be available inside the container at /data/rootfs.img.

Advanced Configuration

Advanced documentation is available in the cli folder here.

balena-virt's People

Contributors

jakogut avatar klutchell avatar maggie44 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

balena-virt's Issues

Allow specifying an OS version

It is currently not possible to specify which OS version to use. This is because the current OS images are flasher images. Extracting the OS image from inside requires extra code, although there is ongoing discussion on whether the images will remain flasher images, or if they will be divided and available separately. Rather than write the code for extracting the image which is required to allow OS selection, instead we have opted to wait on the images being available separately which will make the implementation far easier and save us the time of adding in the mechanism temporarily.

Starting multiple devices simultaneously results in memory peak

When launching many devices at once, the simultaneous launch creates a memory peak.

'BV maintains two trees of pages, one stable, one unstable. Volatile pages aren't scanned for merging, so during bootup, all of these machines consume approximately N times the allocated memory space. After things cool down, KSM starts working. We could vastly increase the number of machines that can run on a single host by delaying each launch by a reasonable amount of time to allow KSM to merge pages for each new host without becoming overloaded. Maybe a minute or so, depending on boot speed and whatnot'.

Using with Github codespaces

I think this would be awesome to use with Github codespaces, as to have a "ready-to-go" Balena dev environment on the cloud for easy development of a Balena project.

Has anyone tried it out? Would be great to have some sort of template or documentation on setting this up.

unable to get ip in vm

Hello, so i got the qemu0 working, and did the iptables additions inside the host os too, but still no juice.

this is from alpine
5: qemu0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000 link/ether 86:a0:1a:d2:2f:5b brd ff:ff:ff:ff:ff:ff inet 192.168.0.118/24 brd 192.168.0.255 scope global dynamic noprefixroute qemu0 valid_lft 73943sec preferred_lft 73943sec inet6 2a02:8084:9061:2d80:c4a5:444d:4ebd:830e/64 scope global dynamic noprefixroute valid_lft 967085sec preferred_lft 374609sec inet6 fe80::5f09:cb45:42ae:a12/64 scope link noprefixroute valid_lft forever preferred_lft forever0:80/) ...

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master qemu0 state UP qlen 1000 link/ether dc:a6:32:ae:a1:3c brd ff:ff:ff:ff:ff:ff

37: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master qemu0 state UNKNOWN qlen 1000 link/ether fe:61:97:01:bd:f3 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc61:97ff:fe01:bdf3/64 scope link valid_lft forever preferred_lft forever

vm:~$ ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:a5:eb:f6 brd ff:ff:ff:ff:ff:ff inet6 2a02:8084:9061:2d80:5054:ff:fea5:ebf6/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 967084sec preferred_lft 374608sec inet6 fe80::5054:ff:fea5:ebf6/64 scope link valid_lft forever preferred_lft forever

running ubuntu server jammy in the vm,

from the host
~/_data# nmcli c s
NAME UUID TYPE DEVICE
bridge-qemu0 b7d7b2e2-67a9-41ab-92bb-33641d44eb99 bridge qemu0
supervisor0 bd4501c0-8bdc-4627-9f08-cd0b3fe40074 bridge supervisor0
bridge-slave-eth0 20f6005a-ba86-4fd3-99d7-ea8f46f9400e ethernet eth0
tap0 a29322a0-e267-4a97-9e24-97ef90d5cb4b tun tap0

regards,

puc

Document how to find IP addresses of virt devices

Is your feature request related to a problem? Please describe.

When bringing up virt devices the IP addresses are shown in the DHCP logs of the networking container, but there must be a better way to collect these addresses if the logs are not available?

Describe the solution you'd like

A simple command line step to print the addresses of the virt devices in the README.

unable to locate qemu0 - net bridge qemu0: bridge helper failed

Hello,

trying to run this on a raspberry pi 4, the container builds fine, I eventually got the data on the device, but looks like qemu0 can't be found.

i played around a bit hoping that if qemu0 was missing, maybe br0 may be there, but that wasn't there either. virbr0 maybe?
next: testing bridged networking via a docker-compose setting...

any help would be great.

[Logs]    [9/8/2022, 00:29:44] [main] qemu-system-aarch64: -net bridge,br=qemu0: bridge helper failed
[Logs]    [9/8/2022, 00:29:44] [main] QEMU instance with PID 17 exited with code: 1, signal: null
[Logs]    [9/8/2022, 00:29:47] Restarting service 'main sha256:4facc191c9c591c2de12e3927be39b59a060c35a7dc674d82d3c37ec62330e5d'
[Logs]    [9/8/2022, 00:29:47] [main] Spawning 1 aarch64 children using template 'standard'
[Logs]    [9/8/2022, 00:29:47] [main] All instances are running


[Logs]    [9/8/2022, 00:44:21] Service exited 'main sha256:4facc191c9c591c2de12e3927be39b59a060c35a7dc674d82d3c37ec62330e5d'
[Logs]    [9/8/2022, 00:44:21] [main] failed to get mtu of bridge `br0': No such device
[Logs]    [9/8/2022, 00:44:21] [main] qemu-system-aarch64: -net bridge: bridge helper failed
[Logs]    [9/8/2022, 00:44:21] [main] QEMU instance with PID 17 exited with code: 1, signal: null

Provide a method to join new virt devices to an existing fleet

Is your feature request related to a problem? Please describe.

When new virt devices come online it would be convenient to automatically join them to an existing fleet.

Describe the solution you'd like

Maybe an optional env var JOIN_FLEET (and API_TOKEN) could be used to automatically join devices to a balenaCloud fleet? I guess that would require the balenaCLI or at least reproducing the steps taken by the join command over SSH.

Describe alternatives you've considered

I also considered pre-configuring (and maybe preloading) the image, or allowing a custom balenaOS image path, but it seems that a special image release is being used.

"Failed to configure image with the fleet provided." using latest image

Describe the bug
I'm trying to run a Balena virtual device locally and to register in Balena Cloud, however I see a warning message when I run the image and my device is not visible in my fleet in Balena Cloud. This is the message I see when I run the image:

This version of the balena CLI (15.0.3) has expired: please upgrade.
369 days have passed since the release of CLI version 16.0.0.
See deprecation policy at: https://git.io/JRHUW#deprecation-policy

The --unsupported flag may be used to bypass this deprecation check and
continue using this version of the CLI. However, note that the balenaCloud
or openBalena backends may be updated in a way that is no longer compatible
with this CLI version.

Failed to configure image with the fleet provided.

To Reproduce
Steps to reproduce the behavior:

  1. Run
docker run -it \
    --restart always \
    -v bv_pid:/app/pid \
    --device=/dev/kvm \
    --cap-add=net_admin \
    --network host \
	-e API_TOKEN="" \
	-e FLEET="" \
	-e AZURE_DEVOPS_ACCESSTOKEN="" \
	-e BALENARC_BALENA_URL="balena-cloud.com" \
    ghcr.io/balena-labs-research/balena-virt:latest

Expected behavior
Device is visible in my fleet in Balena cloud.

Device Type: Local machine used for development.

qemu processes still present on exit

If using ctrl+c to exit, it reports qemu-system-x86_64: terminating on signal 2 but the process is still running in the background. Each new run creates a new process and they begin to build up. Killing with kill -9 seems to corrupt the .img file (not tested with COW, just raw):

ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.1   4240  1560 pts/0    Ss   13:33   0:00 bash
root         113 71.7 37.3 1300320 373348 pts/0  Tl   13:36   0:51 qemu-system-x86_64 -machine q35,accel=kvm -cpu max -smp cores=4 -m 512M -drive if=pflash,format=raw,u
root         125  0.0  0.2   5892  2776 pts/0    R+   13:37   0:00 ps aux

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.