wmutschl / mutschler.eu Goto Github PK

Hello, your guide has helped me a lot from my transition from pop os to Fedora in the past few days.
Could you please add some information in the page about your backup strategy in regards to BTRBK?
I already did setup my system with @ and @home, and have Timeshift working as intended. But I haven't been able to make BTRBK to work. I want to replicate your backup on the internal drive on your linux machine:
Dell Precision 7520 (Fedora Linux)
3. hourly, daily, weekly and monthly BTRFS send/receive snapshots to the internal 512GB SSD drive using BTRBK (automatic)

I also have an internal drive formated to BTRFS, and with a subvolume named fedora. Trying to emulate the author's setup for Example: Backups to USB Disk:

• a backup disk having a btrfs volume mounted as /mnt/btr_backup, containing a subvolume or directory mylaptop for the incremental backups.

Just sharing your btfbk.conf will help me a lot.

sudo apt install gnome-tweak-tool
will not work, but following works
sudo apt install gnome-tweaks
Cheers, great Things to do after install, love them
/henrik

On :: website-academic/content/linux/install-guides/fedora-post-install.md :: section // Pop Gnome Terminal Theme

When attempting to follow the guide I receive a syntax error on lines that include "fi". When changing fi to if I receive "line 69: syntax error: unexpected end of file"

Hi, first off thanks for your awesome guide!

I notice the youtube video is missing on "fedora-btrfs", and the page "fedora-btrfs-raid1" is also missing.

What happened to the Manjaro page (https://mutschler.eu/linux/install-guides/manjaro-btrfs/)?

"This site has moved to https://mutschler.dev" -- must have fallen off the truck...

First of all, thanks so much for your install guide, it's really useful! One question though: In your guide you write

compress=zstd: allows to specify the compression algorithm which we want to use. btrfs provides lzo, zstd and zlib  compression algorithms. Based on some Phoronix test cases, zstd seems to be the better performing candidate. Fedora has started to use v1, I will use v3.

Later then you edit /etc/fstab

sed -i 's/compress=zstd:1/ssd,compress=zstd,discard=async/' /etc/fstab

There, you don't specify a zstd version. Shouldn't it be

sed -i 's/compress=zstd:1/ssd,compress=zstd:3,discard=async/' /etc/fstab

If not, could you please explain why you don't need to specify it but it is still later showing zstd version 3 when you run mount -v | grep /dev/mapper

Hi,

I tried to follow guide to install ubuntu on btrfs with luks, and on optional step 3 there are no /usr/lib/partman/mount.d/70btrfs and /usr/lib/partman/fstab.d/btrfs.
I'm using Ubuntu20.04.1 server.

Maybe guide should be updated?

Hi,

Thank you for amazingly detailed tutorial for Ubuntu-btrfs.

I do however a couple of questions:

1. After following the tutorial can i add and remove additional drives, I am setting up a work station for data processing. My system "Dell Tower" has bays to slot in and out drives. How will this installation guide affect that, in terms of encryption and mounting drives.
2. could I add another normal user "Not super user" to system and allow the user to carry out tasks but limit them to certain things, such as no internet access and just some specific software on the system.

Thank you if you can get back to me as soon as. As i am in the middle of setting up the work station.

Best regards
Monib

Hi,
After doing this tutorial: https://mutschler.eu/linux/install-guides/ubuntu-btrfs-20-04/ , I rebooted and successfully accessed my system.
After upgrading the system, I was asked to reboot to finish the process.
The password open GRUB but when I select my install I have a message: " cryptsetup: Waiting for encrypted source UUID=....."
Then I'm on a Busybox command line.
I tried to remount the partitions in a live session and it's working.

Hello, you have some excellent advise around optimization of FDE + btrfs. I linked here for ubuntu+btrfs-under-luks fde including /boot ...

I noticed many links to your site are broken on the web as you adjust link names; it makes sense what you changed ( https://web.archive.org/web/*/https://mutschler.eu/linux/install-guides/ubuntu-btrfs --> https://mutschler.eu/linux/install-guides/ubuntu-btrfs-20-04/) HOWEVER I the way your site is laid out I had a bit of a time hooking those up, since you have [thank you btw] many different similar pages.

perhaps consider (at a minimum) keeping old links, and an explanation of how you might change them, or use a naming strategy that lets you keep one un-versioned (latest) link, which redirects to the latest versioned page (which can itself hardlink to past versions at the top)

anyway, thanks for your resource & efforts.

Hi!
Thank you so much for these guides. They have been of great help.
On the manjaro-btrfs-luks, there is an issue that has stumped me.

This is on the "Move /var/cache into @cache

The commands

export CRYPTUUID=$(blkid -s UUID -o value /dev/vda2)
echo "UUID=${CRYPTUUID} /var/cache btrfs rw,noatime,compress=zstd:3,ssd,space_cache,commit=120,subvol=@cache 0 0" >> /etc/fstab

specify /dev/vda2 (/sda/2 in my system) which is SWAP in my system (and yours too, following your instructions).

The system won't, therefore, mount the subvol into the SWAP partition.

Could please check and advise if I am missing something?

Thank you,
Brian.

First of all: thank you very much for such a great guide!!

I installed Trisquel 10 and so far it works great. Except for a couple of little changes to adapt it to Trisquel, I entirely followed your very nice guide.

This issue is about a sed command that needs to be checked. I do not open an issue because I do not dare to fix it myself.

I think the problem came right after you mention "Also, let’s use UUIDs instead of /dev/mapper/crypt_vda in the fstab". Then your guide shows the following command:

sed -i "s|/dev/mapper/crypt_vda|$(blkid -s UUID -o value /dev/mapper/crypt_vda)|" /etc/fstab

I think it was there when my fstab file showed the ID numbers but missed to show "UUID=". I had to edit the fstab file to add "UUID=" right before every ID number, and then everything worked ok.

This isn't really an issue as much as it is a request for a guide on how to upgrade the OS. I run this exclusively and use btrfs and am frustrated about the lack of support of the filesystem from System76 in upgrading, installing, etc.

Hello,
I was following the Pop!_OS 20.04 with btrfs-LVM-luks installation guide and now Pop!_OS already has a newer kernel version how do we enable discard?

$ uname -a
Linux pop-os 5.11.0-7612-generic #13~1617215757~20.04~97a8d1a-Ubuntu SMP Thu Apr 1 21:15:20 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

In my case, I saw this guide https://wiki.debian.org/SSDOptimization and added ,discard to /etc/crypttab cryptdata line.

Also following that guide I changed the scheduler to be dead-line:

$ grep . /sys/block/sd?/queue/scheduler
[mq-deadline] none

Were it good changes?

Hello,

I have a question about the "systemd bootloader and kernelstub" part. Do you have to repeat it after updating e.g. release 21.10 or kernel update?

I have a setup of 20.04 according to instructions at https://mutschler.eu/linux/install-guides/pop-os-btrfs-raid1-20-04/ . What is the best practice on upgrading to 22.04?

• Backup all data on separate drive, fresh install of 22.04, and restore the data?
• Live update on top of existing system? Any precautions?

I'm sorry to open this, but I want your advice to recover my system.

Question

• I want to restore my system using a snapshot, but I can't do that since my info.jsons (in my snapshots) are corrupted. Thus, I want to create a valid info.json manually and replace it with the corrupted one. What should I write to info.json?
• Are there any recommended settings to prevent the corruption of info.json?

Situation

I installed Ubuntu Desktop 21.10 on my machine, following your instruction (Ubuntu Desktop 20.04: installation guide with btrfs-luks full disk encryption including /boot and auto-apt snapshots with Timeshift) last week. The system was stable, and I was very comfortable using it.

However, today I accidentally removed a mandatory package, and my system won't boot properly (it shows GNU GRUB version 2.04 instead).

Then I read your article (Timeshift (under construction)), booted my machine from a Live USB stick, and executed

$ sudo cryptsetup luksOpen /dev/nvme0n1p3 cryptdata
$ sudo add-apt-repository ppa:teejee2008/timeshift
$ sudo apt update
$ sudo apt install timeshift

$ sudo timeshift --list --btrfs

/dev/dm-0 is mounted at: /run/timeshift/backup, options: rw,relatime,ssd,space_cache,subvolid=5,subvol=/

E: JSON data must be UTF-8 encoded

(process:8812): Json-CRITICAL **: 02:59:20.107: json_node_get_object: assertion 'JSON_NODE_IS_VALID (node)' failed
E: JSON data must be UTF-8 encoded

(process:8812): Json-CRITICAL **: 02:59:20.108: json_node_get_object: assertion 'JSON_NODE_IS_VALID (node)' failed
E: JSON data must be UTF-8 encoded

[ Omitted since it's massive and repetitive ]

(process:8812): Json-CRITICAL **: 02:59:20.112: json_node_get_object: assertion 'JSON_NODE_IS_VALID (node)' failed
E: JSON data must be UTF-8 encoded

(process:8812): Json-CRITICAL **: 02:59:20.112: json_node_get_object: assertion 'JSON_NODE_IS_VALID (node)' failed
Device : /dev/dm-0 (nvme0n1p3)
UUID   : 17502a7b-cced-4947-9812-ec655bd0c953
Path   : /run/timeshift/backup
Mode   : BTRFS
Status : No snapshots on this device
First snapshot requires: 0 B

No snapshots found

... and saw a bunch of errors.

I opened the GUI version of timeshift and confirmed that my snapshots are saved, but info.jsons are corrupted.

After that, I read this issue that says you can create a valid info.json (by taking new snapshot) and replace it with the corrupted one. That's why I plan to make a new info.json.

The problem is that I'm on a live system (restore only mode) and unable to create new snapshots. I think I should make info.json by hand, but I'm not 100% sure what the content should be.

This is my present situation.

Thank you.

The Ubuntu 20.04 btrfs-luks guide states:

As I have no use for hibernation or suspend-to-disk, I will simply use a random password to decrypt the swap partition using the crypttab

For my use case, I need to have a fully encrypted laptop with suspend-to-disk/hibernate support. How can I do this? The section about it in dm-crypt/Swap encryption is not clear to me.

Hello there!

Your guide was very helpful since I'm also not able to use Timeshift since Fedora 34 (nor even with fresh installation for testing).
It was amazing being able to upgrade to Fedora 36 beta and get back to Fedora 35 after some very, very strange behaviours with a beta system that I was already testing on my external SSD.

So, I used the default Fedora sub volumes and your configuration for BTRBK. It didn't take much time, my system started behaving strangely, after a half hour, some checks on DNF history, Flatpak and Brave browser, out of nowhere (and finally) my Gnome decided to warn me that I was getting out of free space (less than 500MB). I about 100GB free before upgrading to F36.

When I tested TimeShift, I had the same problem, but a bit more because with TimeShift it caused the BTRFS partition to get completely full. As I was just testing, I did a fresh install and restore my backups via Restic.
I setted it on my main drive, 500GB and also to send it to my external USB 1TB drive.
Not sure if there is a way with BTRBK to set and minimum of free space to be alerted, or if I will have to create a script to keep checking it and alert me.

When restoring, I went to the BTRBK git page to follow the instructions. Restoring was really fast and easy, I even double-checked (it's been a while since I used btrfs send and btrfs receive and other commands) if I did things right. Rebooted and everything was there. BUT probably a BTRFS thing, I couldn't delete the Broken sub volume. Kept saying that wasn't empty, even trying with rm and rmdir from another system, and I was just able to do it by removing each folder corresponding to previous snapshots.

All this just to say that is not yet a set-and-forget solution, but not needing to rename sub volumes is already a very super advantage.

Hello,

I have used the Manjaro guide in the past. This is just to inform you that the guide is out of date because manjaro-architect is out of date and its use in newer versions of Manjaro could lead to several issues.

https://forum.manjaro.org/t/maintainer-s-wanted/19502

I think that from now on more than one step will have to be done manually as if it were a pure Arch.

Best

Suggestion: In the tutorials involving grub-btrfs, include a note that for read-only snapshots "/var/log or even /var must be on a separate subvolume", see here.

An unknown number of headaches will not happen. :)

Hi Willi,

I want to encrypt a new installation of Pop!_OS 20.04, but I'm newbie. I follow your tutorial about Pop_OS and btrfs filesystem but I have a classic HDD and I prefer the ext4 filesystem with a traditional partitions (root, swap and home). Following your instructions I can understand something but in the middle I can't continue. Now, I write my process and maybe with your help the initial tutorial can expand for users of ext4, in particular that don't exist tutorials about installation of encrypted full disk with Pop!_OS.

1. Reading about LUKS and LVM one thing is clear: GRUB don't boot with LUKS1, but in your text you wrote that Pop!_OS 20.04 can boot with LUKS2: Pop!_OS uses the systemd bootloader, which can handle luks type 2 encryption just fine at boot time

2. First step is load live USB. Then I use Gparted and created a partition table and layout

• gpt
• a 512 MB EFI (fat32)
• rest of disk without format
• apply changes and tick a efi partition with boot and esp flags.

3. Then create a LUKS partition:

cryptsetup luksFormat /dev/sda2
cryptsetup luksOpen /dev/sda2 cryptdata

pvcreate /dev/mapper/cryptdata
vgcreate data /dev/mapper/cryptdata

lvcreate -n swap -L 8G data
lvcreate -n root -L 30G data
lvcreate -n home -L 100%FREE data

• Note: This scheme is similar to arch wiki example.

4. Next, I will use the Pop!_OS installer to format the partitions and install system, but in this moment I can't continue, because I don't understand your process.

Request:

I read your tutorial about Ubuntu and btrfs-luks, where you don't restart the OS, you return to terminal and write some commands for finish the process. For complete this new version of your tutorial about Pop!_OS we need that you complete the commands for finish the installation process. So we need the post-installation steps for ext4 filesystem.

Additionally, in the Pop!_OS tutorial you don't mention nothing about how to avoid type passphrase twice. Please don't forget that reference.

Finally, another reference important is about swap. In my proposition the swap is part of encrypted partition so I don't know if is necessary create other encrypted partition.

Willi thanks for your time and all your knowledge that you share.

I fixed my error when this occured but showing the user your correct file does not imply strongly enough to add a comma to the line preceding the addition we make in the /etc/kernelstub/configuration file for somebody who doesn't work on this stuff all day long. I only included the stack trace at the bottom so others who see this will understand how important missing punctuation is and how hard it can potentially be to troubleshoot for the non-veteran coder.

So to the meat of my issue:
Can you add a bold note when you show that file's correct contents that they also need to add a comma to the end of the line "splash" like so "splash",?

Other than that I absolutely cannot thank you enough for this document. It scratched an itch I didn't even realize I had.

Below added for explanatory purposes:

Using your Pop!_OS 21.04: installation guide with btrfs-LVM-luks and auto-apt snapshots with Timeshift fails at update-initramfs command with the following stack trace:

root@pop-os:/# update-initramfs -c -k all
update-initramfs: Generating /boot/initrd.img-5.11.0-7620-generic
kernelstub.Config    : INFO     Looking for configuration...
Traceback (most recent call last):
  File "/usr/bin/kernelstub", line 244, in <module>
    main()
  File "/usr/bin/kernelstub", line 241, in main
    kernelstub.main(args)
  File "/usr/lib/python3/dist-packages/kernelstub/application.py", line 142, in main
    config = Config.Config()
  File "/usr/lib/python3/dist-packages/kernelstub/config.py", line 50, in __init__
    self.config = self.load_config()
  File "/usr/lib/python3/dist-packages/kernelstub/config.py", line 60, in load_config
    self.config = json.load(config_file)
  File "/usr/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 20 column 7 (char 363)
run-parts: /etc/initramfs/post-update.d//zz-kernelstub exited with return code 1

This is running exactly as described in your document (with the slight changes for my partition layout, UUIDs, and such.

I've tried this a few times and have always ended up in emergency mode from which I failed to recover. The following things are unclear in the guide:

1. What are Meta+Term+Enter keys supposed to be.. specifically the Term key?
2. The /btrfs_pool and related /etc/fstab operations read like they are being done on a live system. Wouldn't that be leading to data corruption and maybe the emergency mode issue?

Hello! Thanks for making such a comprehensive guide! I ran into an (fixable) issue in the process which may need your attention.

Following the "Pop!_OS 20.04 with btrfs-LVM-luks installation guide and auto-apt snapshots with Timeshift" guide for 21.04 results in the user that is set up by the pop os installer not having a home directory, with the $HOME variable instead being set to the root directory. This causes various issues.

I'm not sure what in the installation process causes this, but the issue can be easily worked around by switching to a tty and making a new user account, giving it sudo privileges, and then removing the first one.

If one were to know this going in, they could make the account in the installer a temporary one that would only be used to create their main account, bypassing the headache of creating a second account to delete the first with the username that you wanted to use and then making a 3rd one with said username.

Using the Pop!_OS 20.04-Raid1 guide to install Pop!_OS 21.04, I found an error with btrfs-progs 5.10.1-1 that is included with the installation media of Pop!_OS 21.04: btrfs-progs throws a "no such directory" error when I get to the line btrfs device add /dev/mapper/data_vdb-root /mnt

I solved this by downloading btrfs-progs 5.13.1-1 and then btrfs device add /dev/mapper/data_vdb-root /mnt worked just like it should in the guide. Version 5.13.1-1 isn't really for ubuntu/pop_os and the error for 5.10.1-1 is acknowledged by a forum post that it was fixed in 5.11, but the ppa's only update to 5.10.1-1 for Pop!_OS 21.04.

Anyway, on a testboot there were some errors like the file manager wouldn't load and the system was generally unresponsive. This was just an initial test on a new media server, to see how the NVME raid would work for the OS disk. I plan to run the guide again and check my setup more carefully in a text editor and see if it solves the weird login behavior.

Hopefully if you make a Raid1 guide for 21.04 this will save you a step of figuring out why btrfs-progs won't cooperate at btrfs device add /dev/mapper/data_vdb-root /mnt

Hey,

I am currently trying to setup Pop OS LTS 22.04 and tried to follow your 21.04 and 21.10 guide and end up with the same issues.
Basically all steps work until the point where the os is installed and I would need to move the files to the sub volumes.
When I just restart the OS the encryption etc. works fine and is indeed btrfs.

First Issue:

Once the intaller is finished installing I am able to open the luks partition but not able to mount it.

$ cryptsetup luksOpen /dev/nvme1n1p6 cryptdata
# Enter passphrase for /dev/nvme1n1p6:
$ ls /dev/mapper
# control cryptdata data-root
$ mount -o subvolid=5,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async /dev/mapper/data-root /mnt
mount: wrong fs type, bad option, bad superblock on /dev/mapper/data-root,
       missing codepage or helper program, or other error

When I just restart the os it boots finde and using df -Th shows that its indeed btrfs

Second Issue:

As I don't really need the snapshots I thought I will just skip the sub volumes then and just apply your fstab optimizations.
So once I am booted inside Pop-OS I execute:

$ sed -i 's/btrfs  defaults/btrfs  defaults,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async/' /etc/fstab
$ sed -i 's/luks/luks,discard/' /etc/crypttab

Once I reboot I will stuck on the Password entry screen (the partition was already unlocked but the screen flashes once and then comes back with the login). When I then switch to terminal mode Ctrl+Shift+F2 I can see the login-promt and login with my user.
When then trying to exit /etc/fstab I will get the error that the file is read-only, as root.

Has something changed with 22.04? It seems like only the Gnome Display Manager fails to start due to some mounting issues?
Did you maybe had a chance to apply your guide to 22.04?

Regards Artur