kevinwortman / tuffix Goto Github PK

I've had a couple of students ask if they can use last year's version of tuffix. I wonder how difficult it would be for us to support a rolling release instead of having to download a new VM image every year.

Would running an updated tuffixize script inside a VM solve this?

hi, I am trying to install Tuffix 2020 for my MacOS Mojave. After import the Tuffix 2020 Edition.ovf, hit Continue button and it got hang while reading applicance 'User/.../Tuffix 2020 Edition.ovf'. I have checked your suggestion and the file ovf is fully download. Please advise. Thanks

We instruct students to ignore messages about upgrading packages or releases. But it would be simpler to just turn off those messages. There's an option to do that in the packages settings dialog.

The newer versions of Ubuntu, or any version running on Windows 11, need a minimum 2CPUs in order to run. With only a single CPU the VM will have a Kernel Panic.

When we switch to per-class packages, include one for CPSC 440 that includes QtSpim and CLI spim.

The ansible syntax for installing multiple packages has changed. Ansible is giving warning messages about the deprecation. We should update the syntax to the suggested alternative.

Hello. I am an incoming freshman in the class of Fall 2020. As the class has its own Linux distro that is Ubuntu-based, I think will help a lot. However, some issues, such as #13, bring up the problems of using a script to bootstrap a distro that is Ubuntu based. This issue will go over some of those disadvantages and how Nix will solve them.

What are the problems?

Reproducibility

This distro seems to be bootstrapped using a shell script. A bootstrap shell script is either stateless or stateful.

A stateless script is not aware of the system state: it installs everything from scratch, and if there are modifications to certain configuration files, the script will apply it anyway and potentially break the system.

A stateful script is slightly better: it is aware of previous modifications, either from its own script or from the state of the system. A stateful script that is aware of enough state will eventually cover all cases. However, it is impractical to cover all cases, which leads us to simplicity.

Simplicitly

Scripts in general have a flaw: they require imperative lines of code executed sequentially that would require the developer to explicitly declare each possible scenario as well as ways to handle them. This is not only time-consuming to develop, it also needlessly bloats up the code.

It could be argued that the 80/20 rule may apply: we could cover 80% of the cases with 20% of the code that we would otherwise have to write/read. However, Nix fixes this in a very elegant way that I will be going into later.

How does Nix solve this?

What is Nix?

Wikipedia states that "NixOS is a Linux distribution built on top of the Nix package manager. It uses declarative configuration and allows reliable system upgrades."

It also states that "Nix is a cross-platform package manager that utilizes a purely functional deployment model where software is installed into unique directories generated through cryptographic hashes."

What does that mean?

First, let's go over declarative configuration. Let's look at the first 7 packages required for the Fall 2020 class:

• a2ps
• autoconf
• automake
• build-essential
• chromium-browser
• cimg-dev
• clang

Here's the same package, except in Nix's declarative configuration format.

{ config, pkgs, ... }: {
    # GNU build tools are covered by Nix in its nix-shell derivative, which I will
    # go over below.
    environment.systemPackages = with pkgs; [ a2ps chromium-browser cimg clang ];
}

There's one difference to the code above: instead of writing a script that executes apt install <pkgs...>, all the packages needed are now written inside a config file that one could so easily install by just importing the file into their system:

{ config, pkgs, ... }: {
    imports = [ ./class-pkgs.nix ];
}

To solve the problems of potentially overriding configuration files, Nix users simply configure their system using the declarative Nix syntax instead of writing a different configuration file for each application/service. This not only simplifies the job by miles, but it would also organize the configuration files in an organized and consistent manner.

What else can we do?

Let's look at how we decided to set the wallpaper. The instructions in the Markdown file writes:

Use Firefox to download a background image, move it to to ~/Pictures, set it as the desktop wallpaper, delete ~/.mozilla

Here's how wallpapers are changed on NixOS GNOME:

{ config, pkgs, ... }:

# Fetch the Tuffnix repository, which would have the wallpaper in its Git
# repository.
let tuffnix = builtins.fetchGit {
    url = "https://github.com/somebody/tuffnix.git";
    rev = "0d05c53204da3b576f810ef2e1312b19bf2420b7";
    sha256 = "1lc26jkjfw1cridymic82lk3zdwhlccs7s5mhkdnz7cbcwllyy54";
};

in {
    # Use the GNOME 3 desktop environment, similar to Ubuntu.
    services.xserver = {
        displayManager.gdm.enable = true;
        desktopManager.gnome3.enable = true;
    };


    # Here we have the user-specific configurations for "student".
    home-manager.users.student = {
        # GNOME wallpapers are stored inside dconf, so we change that.
        dconf.settings = {
            "org/gnome/desktop/background" = {
                # Use the wallpaper in the Git repository.
                picture-uri = "${tuffnix}/background.jpg";
            };
        };
    };
}

As we can see above, Nix not only automates this without any human interactions, it also does this cleanly and declaratively.

Bonus: the Nix shell

Nix provides a nix-shell command which drops the user into a fresh shell instance with additional things installed. This does not globally install anything, which makes it very convenient for development. Here is an example of a nix-shell that brings go (the Golang compiler) into the current shell:

{ pkgs ? import <nixpkgs> {} }:

pkgs.stdenv.mkDerivation rec {
    name = "myproject";
    buildInputs = with pkgs; [ go ];
}

This file would be written into shell.nix. To enter the shell with those packages inside, one only needs to run nix-shell shell.nix.

How does Nix solve the 2 issues above?

The above snippets of code demonstrates a fraction of what Nix could do best: it is a language and package manager that allows expressing system configurations in a simple and declarative manner. What should have been a shell script that would need to run commands and hope that nothing is broken is replaced with a straightforward and declarative configuration file.

A specific example of reproducibility would be the rev Git hash and sha256 output hash provided in builtins.fetchGit. This tells Git that the repository downloaded must be of a certain commit hash and must give an output that matches the given hash. Because of this, all successful outputs of tuffnix will always be the same.

Git repositories aren't the only thing that is hashed. Every single thing managed by Nix will have a reproducible output hash that those things must satisfy. Nix calls this a "derivative". All output derivatives are immutable, meaning they cannot be changed without the hashes being changed as well.

How would NixOS be deployed to students' computers?

That is a great question. This is one of the possible drawbacks in using NixOS over as basic of a distro as Ubuntu: there is no way for graphical installation. NixOS has to be installed in the terminal. It requires manual partitioning before any of the automatic commands could work.

Thankfully, since Nix is reproducible and declarative, it provides tools to both generate a disk image in ISO form as well as virtual manager images with the given Nix configuration files. In fact, one can easily generate a disk image of their own desktop with just a few extra lines AND their system configurations would still be usable.

Nix is a source-based distro with a build cache.

Nix derivatives have to be created from certain pieces of code, so most of what Nix has in its repository are build instructions and configuration declarations. Because of this, Nix is treated as a source-based distro.

However, Nix also has a build cache. Thanks to Nix's reproducible features, all builds always have hashes attached to them, meaning they are immutable for that piece of configuration, making it very easy to distribute Nix builds safely and quickly.

One could also either self-host a binary cache or distribute public build derivatives for others to use. This is extremely useful when it comes to distributing Nix configuration files, as those who install the configurations can expect everything to be downloaded as binaries and nothing to be built.

Conclusion

NixOS can reliably help students as well as others bootstrap their setup. Doing so is a lot better than using a shell script to bootstrap packages and configuration files onto a (hopefully fresh) Ubuntu setup. If done properly, disk images could even be generated that would install everything right during the installation process without any further effort.

Additional tidbits

Tidbits that aren't organized in any specific order of any meaning but is still important enough to be mentioned.

Nix is both rolling release and standard release

The Nix package manager defaults to using a standard release channel (repository) that behaves similarly to Ubuntu's release system. However, Nix users could also opt for the unstable channel, which is rolling. This is extremely useful for those who need the cutting edge packages.

Nix also allows a configuration to use multiple Nix package repositories (nixpkgs) at once. If a use case requires a cutting edge version of clang that isn't available in stable yet, one could always just use the clang of a newer channel while leaving the rest in tact.

Nix's channel versions can be locked

If extreme reproducibility is needed, one could lock their nixpkgs onto a certain commit. This is extremely easy because nixpkgs itself is just a Git repository. As Nix always have build instructions, one could always reliably get the packages as long as the source code is available.

Dependency hell is not a problem on Nix

Nix allows keeping multiple versions of packages on the same system. As each version has a different output hash, Nix doesn't really care about them colliding. Packages that rely on different versions of a dependency will happily coexist.

In the case that a package needs an outdated dependency, as long as the source code is available, Nix can always build that dependency without colliding with anything else.

Nix allows rolling back after each configuration change

Nix creates a "generation" for each configuration rebuild. Although rebuilds will straight up fail if any part of the configuration is erroneous, even if the error somehow slips through, one can always simply reboot to the last generation.

Nix Overlays allow for easy override of packages

While other distros would require rebuilding a package from bare source code if certain changes were to be made (such as patches or cutting edge upgrades), Nix allows overriding existing build instructions. The below code overrides package libhandy's version:

{ config, pkgs, ... }: {
    nixpkgs.overlays = [
        (self: super: {
            libhandy = super.libhandy.overrideAttrs(old: {
                name = "libhandy-0.90.0";
                src  = builtins.fetchGit {
                    url = "https://gitlab.gnome.org/GNOME/libhandy.git";
                    rev = "c7aaf6f4f50b64ee55fcfee84000e9525fc5f93a";
                };
            });
        })
    ];
}

Nix works on MacOS as well

Nix can work natively on MacOS thanks to Nix Darwin. Although certain things may not work, other things like services, packages and user configurations would still work.

Nixpkgs is the second largest package repository ever

According to Repology, nixpkgs contains 51481 packages, which is only behind the Arch User Repository (AUR) at 55557 packages. It is worth mentioning that this number of packages does not make Nix any less stable, unlike the AUR.

Reference

• NixOS manual: https://nixos.org/nixos/manual/
• Nixpkgs manual: https://nixos.org/nixpkgs/manual/
• Uses and applications: https://nixos.wiki/wiki/Applications

We'd like to use OpenGL and GLEW in CPSC 484. Those libraries don't seem to work with the default graphics controller setting, but do work with VBoxSVGA.

The script currently installs libgtest-dev that conflicts with the compiled version of googletest.

I think if we remove the line below, it should resolve the issue.

Hernan Manabat writes

I ran into few instances where users have problem running Tuffix VM because of the CPU Virtualization being disabled by default on some systems. Users need to restart their machine and must go into the BIOS and enable this feature first in order such issue. Perhaps you can mention something like that in your Tuffix VM document's FAQ section.

We should add an item about this to the installation instructions in the VM section.

Add
export EDITOR=atom
to .bash_profile or .bashrc so that git and other tools will open config files in Atom. By default they use nano or vim, neither of which students understand how to use.

We might want to add a checker to see if googletest works as expected. There are cases when the unit test fails due to conflicting googletest versions.

Paul requested that we add packages clang-tidy clang-format for 121.

Hello,
I don't personally need this but thought you'd like to know. It seems the "Google Test Library Build" fails on my installation:

TASK [Google Test Library Build] ***********************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "BUILDDIR=\"/tmp/gtestbuild.$$\"\nDESTROOT=\"/usr/local/\"\nmkdir -p ${BUILDDIR}\ncd ${BUILDDIR}\ncmake -DCMAKE_BUILD_TYPE=RELEASE /usr/src/gtest/\nmake\ninstall -o root -g root -m 644 libgtest.a ${DESTROOT}/lib\ninstall -o root -g root -m 644 libgtest_main.a ${DESTROOT}/lib\n", "delta": "0:00:17.356882", "end": "2020-08-08 05:32:14.465000", "msg": "non-zero return code", "rc": 1, "start": "2020-08-08 05:31:57.108118", "stderr": "CMake Warning at CMakeLists.txt:54 (project):\n  VERSION keyword not followed by a value or was followed by a value that\n  expanded to nothing.\n\n\ninstall: cannot stat 'libgtest.a': No such file or directory\ninstall: cannot stat 'libgtest_main.a': No such file or directory", "stderr_lines": ["CMake Warning at CMakeLists.txt:54 (project):", "  VERSION keyword not followed by a value or was followed by a value that", "  expanded to nothing.", "", "", "install: cannot stat 'libgtest.a': No such file or directory", "install: cannot stat 'libgtest_main.a': No such file or directory"], "stdout": "-- The CXX compiler identification is GNU 9.3.0\n-- The C compiler identification is GNU 9.3.0\n-- Check for working CXX compiler: /usr/bin/c++\n-- Check for working CXX compiler: /usr/bin/c++ -- works\n-- Detecting CXX compiler ABI info\n-- Detecting CXX compiler ABI info - done\n-- Detecting CXX compile features\n-- Detecting CXX compile features - done\n-- Check for working C compiler: /usr/bin/cc\n-- Check for working C compiler: /usr/bin/cc -- works\n-- Detecting C compiler ABI info\n-- Detecting C compiler ABI info - done\n-- Detecting C compile features\n-- Detecting C compile features - done\n-- Found PythonInterp: /usr/bin/python (found version \"2.7.18\") \n-- Looking for pthread.h\n-- Looking for pthread.h - found\n-- Performing Test CMAKE_HAVE_LIBC_PTHREAD\n-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed\n-- Looking for pthread_create in pthreads\n-- Looking for pthread_create in pthreads - not found\n-- Looking for pthread_create in pthread\n-- Looking for pthread_create in pthread - found\n-- Found Threads: TRUE  \n-- Configuring done\n-- Generating done\n-- Build files have been written to: /tmp/gtestbuild.25434\nScanning dependencies of target gtest\n[ 25%] Building CXX object CMakeFiles/gtest.dir/src/gtest-all.cc.o\n[ 50%] Linking CXX static library lib/libgtest.a\n[ 50%] Built target gtest\nScanning dependencies of target gtest_main\n[ 75%] Building CXX object CMakeFiles/gtest_main.dir/src/gtest_main.cc.o\n[100%] Linking CXX static library lib/libgtest_main.a\n[100%] Built target gtest_main", "stdout_lines": ["-- The CXX compiler identification is GNU 9.3.0", "-- The C compiler identification is GNU 9.3.0", "-- Check for working CXX compiler: /usr/bin/c++", "-- Check for working CXX compiler: /usr/bin/c++ -- works", "-- Detecting CXX compiler ABI info", "-- Detecting CXX compiler ABI info - done", "-- Detecting CXX compile features", "-- Detecting CXX compile features - done", "-- Check for working C compiler: /usr/bin/cc", "-- Check for working C compiler: /usr/bin/cc -- works", "-- Detecting C compiler ABI info", "-- Detecting C compiler ABI info - done", "-- Detecting C compile features", "-- Detecting C compile features - done", "-- Found PythonInterp: /usr/bin/python (found version \"2.7.18\") ", "-- Looking for pthread.h", "-- Looking for pthread.h - found", "-- Performing Test CMAKE_HAVE_LIBC_PTHREAD", "-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed", "-- Looking for pthread_create in pthreads", "-- Looking for pthread_create in pthreads - not found", "-- Looking for pthread_create in pthread", "-- Looking for pthread_create in pthread - found", "-- Found Threads: TRUE  ", "-- Configuring done", "-- Generating done", "-- Build files have been written to: /tmp/gtestbuild.25434", "Scanning dependencies of target gtest", "[ 25%] Building CXX object CMakeFiles/gtest.dir/src/gtest-all.cc.o", "[ 50%] Linking CXX static library lib/libgtest.a", "[ 50%] Built target gtest", "Scanning dependencies of target gtest_main", "[ 75%] Building CXX object CMakeFiles/gtest_main.dir/src/gtest_main.cc.o", "[100%] Linking CXX static library lib/libgtest_main.a", "[100%] Built target gtest_main"]}

OS: Ubuntu 20.04.1 (fresh installation)
Uname: Linux mike-KVM 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Virtualization: Qemu/KVM
Used command: wget https://csufcs.com/tuffixize -O - | bash
DE: Gnome

Before installation I updated, installed xfce4 and terminator, and rebooted (did not switch to xfce4 yet).

Thomas Bettens observes

One thing I noticed: while GCC 9 is installed, the default still points to GCC 8.

g++ -v

gcc version 8.3.0 (Ubuntu 8.3.0-6ubuntu1)

/usr/bin/g++ points to /usr/bin/g++-8. Should it point to /usr/bin/g++-9?

g++-9 -v

gcc version 9.1.0 (Ubuntu 9.1.0-2ubuntu2-19.04)

Likely an alternative setting/update somewhere ...

VirtualBox added confirmed support in version 7.0.0.

After following the "Native Installation" instructions to install Tuffix on a clean Linux install, there should be a script that can be run to verify that everything was successfully installed and configured.

Given the fact that the concept behind Tuffix is to ensure that all users are running identical machines, there should be some way to ensure that this is true. Scripts or installs can fail, versions might differ, or other odd issues might occur.

I'd suggest a secondary test script which would go through and run a set of scripts to verify that everything that was supposed to be installed is working correctly. And maybe even throw in a few scripts which should fail given that Tuffix doesn't include certain packages.

Detailed installation instructions are on Google Drive, most probably on someones personal account which risks disappearing overtime.
It can't be modified or improved by external contributors either, as it's read-only.
Please consider enabling or opening access to the project's wiki and moving the document to it under a Free license so others can also contribute.

As of PR 869147c , the tuffixize script installs only the minimal core packages for 120-121-131 C++ programming, and additional libraries for post-intro courses are in playbooks in the sub-tuffix directory. We need instructions on how students should add these additional playbooks.

Since these students will have completed 120-121-131 they will be proficient in git and the terminal, so I think it's OK to tell them to simply clone this tuffix repo and run the ansible-playbook command.