Comments (16)
Right online shrinking does not work. The need for a different filesystem is indeed a good reason. We had this request at DietPi as well, hence I'm interested to year about cases.
In theory, easy to implement here: https://github.com/RPi-Distro/raspberrypi-sys-mods/blob/0a2e4c3/usr/lib/raspberrypi-sys-mods/firstboot#L45
The question is how to best offer the input:
- Allow to define the end of the partition
- Allow to define the size of the partition (so end depends on boot partition size and location)
- Allow to define the remaining disk space after the partition
Not sure how the test systems at RPi dev side look like, but if there are already tests done with giving rpi-imager/config file/flag first boot inputs, booting a (virtual or real) system and checking back whether the inputs have been successfully applied, checking the partition/filesystem size shouldn't be hard to implement?
OOT: Interesting to see someone using emoncms. We had emonHub as software option in DietPi, but zero reported installs. And since we don't have the hardware to actually test the long untouched implementation, we removed it. Do you use emonHub or other software to read and store the sensor data?
from raspberrypi-sys-mods.
Since this has come up a few times already, it seems like there's some demand for this feature. However, I don't want to change behaviour or add features until we have a test system in place to make sure non-default configurations work. Otherwise, it's easy to have a feature that breaks and we don't find out about it until somebody actually uses it months after an image release.
So yes, but not right now.
from raspberrypi-sys-mods.
Not sure how the test systems at RPi dev side look like, but if there are already tests done with giving rpi-imager/config file/flag first boot inputs, booting a (virtual or real) system and checking back whether the inputs have been successfully applied, checking the partition/filesystem size shouldn't be hard to implement?
There aren't, so that's why I'm keen on getting something like that in place to make sure all configuration options are tested under all the headless setup mechanisms before doing anything else.
from raspberrypi-sys-mods.
There aren't, so that's why I'm keen on getting something like that in place to make sure all configuration options are tested under all the headless setup mechanisms before doing anything else.
Shall those tests run on physical hardware, or would an image "booted" as e.g. simple systemd-nspawn
container via GitHub Actions would be feasible? When its about partition and filesystem resizing within the guest, the only issue I wasn't able to solve yet, is how to inform the host system's kernel about partition table changes, when resizing the partition from within the container, so that resize2fs
can expand the filesystem and does not return "nothing to do".
from raspberrypi-sys-mods.
My suggestion (from our own experiences), is simply the ability to specify a rough size (say 6G) for the rootfs
, and then the system can calculate the best boundary fit for that. A user who knows they want a smaller size root partition, can then sort out any other partition on the remaining space to suit. You can even expand the rootfs
, you just can't shrink it!
I'd suggest a check wrt the minimum allowed.
Personally, something that could be set in a configuration file in the boot
partition. Perhaps added to the Pi Imager as an option as well.
OOT: Interesting to see someone using emoncms. We had emonHub as software option in DietPi, but zero reported installs. And since we don't have the hardware to actually test the long untouched implementation, we removed it. Do you use emonHub or other software to read and store the sensor data?
emonHub is a key element of emoncms in many cases, especially those running on a Pi as it acts as both a software and hardware interface to push data to emoncms (and send on to other systems as well if required). We are in the process of updating the docs.
from raspberrypi-sys-mods.
While this is being discussed, can I float the idea of (optionally?) rounding down common SD card sizes so that images can be easily cloned between cards from different manufacturers of roughly comparable sizes? In other words, define minimum sizes that could be given the labels 16GB, 32GB, 64GB, etc. and round down actual card sizes by a limited amount (up to 1%? 0,5GB?) when expanding the rootfs.
from raspberrypi-sys-mods.
I think for each person happy with the option, another 5 will ask why it doesn't create the extra partition, why the size, format and other options can't be configured, then why can't they add multiple extra partitions and it will keep going from there.
Yes, You can't please all of the people all of the time 😆
However, as a starting point, it would at least provide an option which currently doesn't exist at all.
from raspberrypi-sys-mods.
Shrinking 'rootfs' is difficult
E.g. for a 4 GiB filesystem:
resize2fs /dev/mmcblk0p2 4G
This is difficult?
If you do not mind a question: Why do you need a dedicated data partition on the same drive? What is the advantage compared to a directory on the root filesystem?
from raspberrypi-sys-mods.
resize2fs /dev/mmcblk0p2 4G
Is that resizing a live root partition?
Fresh PiOS 32bit Lite
pi@emonsdtest:~ $ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk0 179:0 0 14.8G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 14.6G 0 part /
pi@emonsdtest:~ $ sudo resize2fs /dev/mmcblk0p2 8G
resize2fs 1.46.2 (28-Feb-2021)
Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing required
resize2fs: On-line shrinking not supported
I think because it is still mounted (as it is boot).
If you do not mind a question: Why do you need a dedicated data partition on the same drive? What is the advantage compared to a directory on the root filesystem?
This is for the OpenEnergymonitor emoncms system. Over the years it suffered with SD Cards failing despite various interventions put in place to buffer data etc. At worst it records multiple sensor data every 5s and writes it to file (and no, we are not going to use a different database engine 😄 ).
Through experimentation, it was determined that using ext2 filesystem provided a more robust target for the data, so we started using that for a data partition. It has been successful and the number of failures is now very low.
We used to use out own init script
(simply replaced the standard one before first boot), but changes to that method have made it more difficult.
# Dev size divisible by 2048
ROOT_DEV_SIZE2048=$((($ROOT_DEV_SIZE / 2048) * 2048 ))
#EmonSD Fix end of Rootfs
if [ $ROOT_DEV_SIZE -gt 14000000 ]; then
#16GB card or above, assign last 10GB for data
TARGET_END=$(($ROOT_DEV_SIZE2048 - 20971520 - 1 ))
elif [ $ROOT_DEV_SIZE -gt 6000000 ]; then
#8GB card, assign last 4GB for data
TARGET_END=$(($ROOT_DEV_SIZE2048 - 7340032 - 1 ))
else
# Assign 2GB to rootfs
TARGET_END=4098047
fi
#EmonSD set start of ext2 partition
EXT2_START=$((TARGET_END + 1))
PARTITION_TABLE=$(parted -m "$ROOT_DEV" unit s print | tr -d 's')
LAST_PART_NUM=$(echo "$PARTITION_TABLE" | tail -n 1 | cut -d ":" -f 1)
ROOT_PART_LINE=$(echo "$PARTITION_TABLE" | grep -e "^${ROOT_PART_NUM}:")
ROOT_PART_START=$(echo "$ROOT_PART_LINE" | cut -d ":" -f 2)
ROOT_PART_END=$(echo "$ROOT_PART_LINE" | cut -d ":" -f 3)
We distribute a pre-prepared SD Card, but also a script system for users to create their own setup.
As far as I can see, it just needs the ability to specify an approximate TARGET_END
in a setting and then calculate the right boundary (I say 'just' 😃 ).
Cheers
from raspberrypi-sys-mods.
Shall those tests run on physical hardware, or would an image "booted" as e.g. simple
systemd-nspawn
container via GitHub Actions would be feasible? When its about partition and filesystem resizing within the guest, the only issue I wasn't able to solve yet, is how to inform the host system's kernel about partition table changes, when resizing the partition from within the container, so thatresize2fs
can expand the filesystem and does not return "nothing to do".
In my experience, containers aren't quite the panacea some people claim they are. I have had plenty of cases where a simple chroot
or a VM just works and systemd-nspawn
just doesn't and I can't go down a rabbit hole each time it happens to try to figure out why.
But, if there's an easy stop-gap solution to the problem, that would be great. Is there a way to simulate different environments? For example, running as an init= script (so there is no systemd in that environment), running a systemd service the way rpi-imager does, through cmdline and "rebooting", or a full fledged environment where you can run an X session and start/stop services freely?
The only way I can think of doing all that would be with qemu-system, which isn't trivial either.
from raspberrypi-sys-mods.
Couple of thoughts
- This should only be available to non NOOBS setups.
- You do already do a similar thing with NOOBS (set different sizes for
rootfs
) it seems. So being able to specify the size used seems to me to be an extension of an existing process. - Do nothing else - just create a smaller root partition.
from raspberrypi-sys-mods.
Interestingly a user on the OEM forum has found a way round this. After flashing, before boot, create a partition on the SD Card which prevents the boot process filling the card with rootfs (not tried it myself).
from raspberrypi-sys-mods.
If anyone comes here looking for a method to do this (prevent firstboot
filling card with rootfs
) - I'll direct you to this post. https://community.openenergymonitor.org/t/creating-an-ext2-partition-and-resizing-rootfs/22786/1
Perhaps simply offering an option to disable the resize operation is sufficient to satisfy those that want to do more with the card.
from raspberrypi-sys-mods.
While this is being discussed, can I float the idea of (optionally?) rounding down common SD card sizes so that images can be easily cloned between cards from different manufacturers of roughly comparable sizes? In other words, define minimum sizes that could be given the labels 16GB, 32GB, 64GB, etc. and round down actual card sizes by a limited amount (up to 1%? 0,5GB?) when expanding the rootfs.
Sounds good, but there's also another plan to deal with this scenario. Probably worth discussing offline.
Perhaps simply offering an option to disable the resize operation is sufficient to satisfy those that want to do more with the card.
I think for each person happy with the option, another 5 will ask why it doesn't create the extra partition, why the size, format and other options can't be configured, then why can't they add multiple extra partitions and it will keep going from there.
from raspberrypi-sys-mods.
The only way I can think of doing all that would be with qemu-system, which isn't trivial either.
A VM of course would be better. I tried to get qemu-system with the included RPi machine up to test Raspberry Pi images once, but mostly failed. It wasn't able to deal with the RPi kernel somehow, but can't remember details. Also you need to provide the kernel and dtb externally since it isn't able to load it from the FAT partition of the image, but would require a special layout for this. So I gave up on this and switched to systemd-nspawn with qemu-user.
Is there a way to simulate different environments? For example, running as an init= script (so there is no systemd in that environment), running a systemd service the way rpi-imager does, through cmdline and "rebooting", or a full fledged environment where you can run an X session and start/stop services freely?
I didn't understand all aspects of the question, but:
systemd-nspawn -bD /path/to/roofs
"boots" an image (e.g. mounted as loop device) in the foreground. It will just execute/sbin/init
by default, but the init command can be configured. So it will be systemd with Raspberry Pi OS images, which should be the only relevant testing case?- It will internally use
/dev/console
and get the login prompt there automatically viaconsole-getty.service
(systemd pulls this in automatically). So autologin would need to be configured with this service, the same way as one would do with[email protected]
. From there you cansystemctl
everything as on VM/physical system. - By default it has no access to drives, but this can be granted with some
--bind=/dev/loop...
arguments, so it can use/etc/fstab
to mount/boot
(if not done from host already), read and control its block device properties. - By default it uses the host network, cannot and does not need to configure network devices, but this can be changed with command options as well to have an isolated network and allow setup via ifupdown or dhcpcd from within the host.
- X or generally GUI is tricky. I tried it once but apart of a bunch of additional
--bind=
mounts to enable access to the graphics interfaces, I failed to get interactive mouse/keyboard input for GUI elements working properly, even with the container started on an actual Raspberry Pi without userspace emulation. - You can start the container via systemd service or other method in background and then use
machinectl
to control/access it from the host.
from raspberrypi-sys-mods.
I currently do this by manually removing the init=
part of the cmdline.txt file after creating the initial (Lite) disk image. You'll then have to boot the Raspberry Pi twice, and on the second boot you can gain access via SSH. I then run an install script that adds a recovery and user partition.
With every new release of Raspberry Pi OS I worry that this hack will no longer be possible. Bookworm added the need to double-boot, for example.
Having an option to set the partition size would be very welcome.
from raspberrypi-sys-mods.
Related Issues (20)
- Dedicated LTS Raspberry Pi OS image for Compute Modules. HOT 3
- Raspberry Pi OS Image is missing Microsoft Repository HOT 1
- [Bullseye] postinst script contains every auto-added block four times HOT 3
- Post-installation script subprocess returned error exit status 126 HOT 5
- Remove 010_pi-nopasswd HOT 7
- New postinst flush code fails if vfat partition not found HOT 3
- [bullseye] FIRSTBOOT exits prematurely on NOOBS/PINN sd card HOT 4
- `init_config` python module named 'toml' is missing in Raspberry Pi OS lite 64bit HOT 2
- missing license HOT 2
- firstboot script hangs if image contains already kernel 6.1
- "imager_custom set_wlan" generates script but does not escape special characters properly HOT 1
- Abstract boot partition path HOT 5
- Exit status of grep HOT 2
- Firstboot fix_partuuid issues
- How to update wifi after first boot? HOT 13
- Serial port symlinks broken in bookworm. HOT 6
- imager_set_custom using uuid which is not always available HOT 1
- Initrd rootfs resize fails on RPi5 HOT 12
- Recent removal of /boot/cmdline.txt link to /boot/firmware/cmdline.txt HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from raspberrypi-sys-mods.