Giter Club home page Giter Club logo

chiagarden's Introduction

ChiaGarden

ChiaGarden is a collection of sysprep tools designed to build, manage, and maintain a PoST (Proof of Space and Time) farm on a linux based system. These tools make it easy to prepare and mount your hard disks, manage and monitor your plots for your farm.

If you agree that typing disk paths is a job you shouldn't be doing, then this is for you. If you are already plotted and have no intention of reformatting anything, do read on: ChiaGarden will let you uniquely label drives with a certain pattern (for example CHIA-[SERIALNR]) which then facilitates subsequent processing.

And if you've already given out creative disk labels (Heinz, Hans and Franz, I guess) or if you think that changing disk labels may be destroying your plots (although that is not the case) here is good news, too: Most of the ChiaGarden tools offer you a choice between two operating modes:

--label search and process plots based on disk labels. --mount-dir search and process plots based on mount points.

As long as your disks are mounted to the same mount point consistently, there's no distinction.

chiainit

chiainit is a bash script for preparing hard drives for PoST farming. It automates the process of formatting and labeling multiple drives at once.

gardenmount

gardenmount is a script for (un)mounting drives based on a specified label prefix. Additionally, this repository provides systemd service files for managing the mounting and unmounting process.

plotting

plotting contains various scripts for managing, monitoring and automating the (re)plotting process.

farming

farming contains various scripts for analyzing farming performance, specifically for chia.

taco_list

taco_list is a simple wrapper script to list your destination disks and execute any command with the given parameters - for example chia_plot_sink

Getting Started

To get started with ChiaGarden, clone the repository, and run the installer script install.sh.

git clone https://github.com/efnats/chiagarden.git
cd chiagarden
sudo ./install.sh

The installer script will guide you through the installation process, copy necessary files, and optionally enable and start the garden-mount.service.

Once you have installed ChiaGarden, use it to ensure all drives containing plot files are labeled with a unique label pattern (default: CHIA-[serialnr]). The chiainit tool can be used to (re)label available drives in your system. Relabeling a drive is a non-destructive action and will not delete any data on the drive.

Refer to the instructions in each directory for further configuration and usage information.

Reaching out

Contributions to ChiaGarden are welcome. Feel free to submit pull requests or open issues to improve the tools and make them more useful for the Chia farming community. Currently the best way to reach out is here via filing a github issue or using the thread I created at chiaforum.com

License

Chia Garden is released under the MIT License. See the LICENSE file for more information.

chiagarden's People

Contributors

efnats avatar mathis436 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

chiagarden's Issues

Question about plot_over

Im trying to configure plot_over. My setup for the mounted drives in /mnt/bay#/serial#/.
when I configure plot_over with

##Chiagarden plot_over config file

#Dry run is enabled by default, to ease your mind. set to false, to actually use plot_over
dry_run=true

minimum free space in GB

min_free_space="200"

desired amount of disks with min_free_space

amount_free_disks="44"

comma-separated list of Plot Compressions levels that you would like to replot. Those will be (gradually) removed.

this example would replot to C18

replot_levels="0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,17,19,20"

Default parameter for your label. Can also be set in commandline.

label="CHIA"

If you didnt label your drives with chiainit, you can set a parent mount dir where your your chia disks are mounted in.

mount_dir and "label" cannot be used at the same time.

mount_dir="/mnt/"

Default parameter for your first subdirectory for plots in Disk. Can also be set in commandline.

subdir=""

maximum search depth for plot files - how many subdir levels should we iterate?

search_depth="10"

rescan every xx seconds. this is AFTER removal of plot files in one iteration is finished. it is okay to set this relatively low (30s)

interval="30"

set to a valid path or leave empty to disable logging. Ignored (set empty) when plot_over is run from systemd

logfile="./plot_over.log"

set to true to display the paths that are searched for plot files

display_search_paths="true"

This is the output
image

If i change the config to use

If you didnt label your drives with chiainit, you can set a parent mount dir where your your chia disks are mounted in.

mount_dir and "label" cannot be used at the same time.

mount_dir="/mnt/bay8/"

The out put is
image

Working as expected.

I have approximately 44 drives that are filled with c8 compressed plots that I want to replot
they are located in /mnt/bay8/ & /mnt/bay9-11/

how can I configure plot-over to read the list of directories past the first set of bay# folders that are under /mnt ?

BUG: chiainit is_drive_mounted() grep pattern matches incorrect targets

Problem

This line in chiainit.py causes is_drive_mounted() to incorrectly report drives as being mounted when they are not.

Proof of concept:

I am trying to initialize /dev/sdb, which is not mounted, but /dev/sdba1 is mounted. The grep expressing matches /dev/sdba1, and thus reports that /dev/sdb is mounted when it is not.

$ drive="sdb"
$ if grep -qs "/dev/${drive}[0-9]*" /proc/mounts ; then echo 'FOUND'; fi
FOUND
$ grep -s "/dev/${drive}[0-9]*" /proc/mounts
/dev/sdba1 /mnt/jbod1/jbod1-00014KYV ext4 rw,noatime 0 0

This was on a Debian Linux system, so YMMV if you are using a different grep variant or have configured bash globbing differently.

Explanation

Typically an asterisk is used in bash with paths for globbing. Ie, if we consider ls /dev/sda[0-9]*, it would only list all of the partitions of /dev/sda since the expansion is occuring via bash globbing.

However, since it's the path is part of a grep PATTERN, instead of globbing, the asterisk is being used as part of the REGEX as a repetition specifier. The default REGEX engine used is BRE, which treats the asterisk as: The preceding item will be matched zero or more times. Thus the [0-9]* pattern makes the digit optional (ie, 'match 0 or more times'). It's effectively just looking for any entry in /proc/mounts that contains /dev/$drive.

Solution

To solve this, simply match one or more instead of zero or more:

  • via BRE: grep -qs "/dev/${drive}[0-9]\+" /proc/mounts
  • via ERE: grep -qsE "/dev/${drive}a[[:digit:]]+"
  • via PCRE: grep -qsP "/dev/${drive}\d+" /proc/mounts

However, it's technically possible in Linux to mount a full-disk filesystem without a partition table (mkfs.ext3 /dev/sdb ; mount /dev/sdb /mnt/something). We probably want to also detect this, just in case! So something like this might be more appropriate:

grep -qs "/dev/${drive}[0-9]* " /proc/mounts

Note the addition of the space at the end of the PATTERN. That would only match the expected $drive OR any of it's partitions, but not any other drive whose name is a superset of our drive. And 1-character bug fixes are cool :)

Thanks!

chia init only fully completes 1 drive when attempting multiple partition and formats

sudo ./chiainit --fstype ext4 --init sdy sdab sdw sdu sds sdq sdz sdx sdaa sdv sdt sdr

Operation Summary
Drives : sdy sdab sdw sdu sds sdq sdz sdx sdaa sdv sdt sdr
Excluded :
Filesystem : ext4
Actions : init

Do you want to continue? (Y/n)
Y
WARNING: You have selected a destructive action (wipe, format, or init).
THIS WILL DESTROY ALL YOUR DATA ON THE SPECIFIED DRIVE(S).
sdy sdab sdw sdu sds sdq sdz sdx sdaa sdv sdt sdr

To proceed, type 'YES I SACRIFICE THIS DATA' and press Enter.
YES I SACRIFICE THIS DATA

Drive /dev/sdy:
Wiping /dev/sdy...
/dev/sdy: 8 bytes were erased at offset 0x00001000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdy: 8 bytes were erased at offset 0x74702555000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdy: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdy: calling ioctl to re-read partition table: Success
Formatting /dev/sdy...
mke2fs 1.47.0 (5-Feb-2023)
/dev/sdy1 contains a zfs_member file system labelled 'mmcuplot2'
Creating filesystem with 1953506304 4k blocks and 1907744 inodes
Filesystem UUID: c226291a-5ae3-4e4c-b4a5-eb71c9dab9be
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544, 1934917632

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

Labeling /dev/sdy1...
Serial Number: R5GG0AUV
UUID: UNKNOWN
New label:

Drive /dev/sdab:
Wiping /dev/sdab...
/dev/sdab: 8 bytes were erased at offset 0x00001000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdab: 8 bytes were erased at offset 0x74702555000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdab: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdab: calling ioctl to re-read partition table: Success
Formatting /dev/sdab...
Labeling /dev/sdab1...
Serial Number: R5G9WN2V
UUID: UNKNOWN
New label:

Drive /dev/sdw:
Wiping /dev/sdw...
/dev/sdw: 8 bytes were erased at offset 0x00001000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdw: 8 bytes were erased at offset 0x74702555000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdw: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdw: calling ioctl to re-read partition table: Success
Formatting /dev/sdw...
Labeling /dev/sdw1...
Serial Number: R5GEN2VV
UUID: UNKNOWN
New label:

Drive /dev/sdu:
Wiping /dev/sdu...
/dev/sdu: 8 bytes were erased at offset 0x00001000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdu: 8 bytes were erased at offset 0x74702555000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdu: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdu: calling ioctl to re-read partition table: Success
Formatting /dev/sdu...
Labeling /dev/sdu1...
Serial Number: R5GEJMBV
UUID: UNKNOWN
New label:

Drive /dev/sds:
Wiping /dev/sds...
/dev/sds: 8 bytes were erased at offset 0x00001000 (gpt): 45 46 49 20 50 41 52 54
/dev/sds: 8 bytes were erased at offset 0x74702555000 (gpt): 45 46 49 20 50 41 52 54
/dev/sds: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sds: calling ioctl to re-read partition table: Success
Formatting /dev/sds...
Labeling /dev/sds1...
Serial Number: R5GBLVZV
UUID: UNKNOWN
New label:

Drive /dev/sdq:
Wiping /dev/sdq...
/dev/sdq: 8 bytes were erased at offset 0x00001000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdq: 8 bytes were erased at offset 0x74702555000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdq: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdq: calling ioctl to re-read partition table: Success
Formatting /dev/sdq...
Labeling /dev/sdq1...
Serial Number: R5G1HRKV
UUID: UNKNOWN
New label:

Drive /dev/sdz:
Wiping /dev/sdz...
/dev/sdz: 8 bytes were erased at offset 0x00001000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdz: 8 bytes were erased at offset 0x74702555000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdz: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdz: calling ioctl to re-read partition table: Success
Formatting /dev/sdz...
Labeling /dev/sdz1...
Serial Number: R5GEU6AV
UUID: UNKNOWN
New label:

Drive /dev/sdx:
Wiping /dev/sdx...
/dev/sdx: 8 bytes were erased at offset 0x00001000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdx: 8 bytes were erased at offset 0x74702555000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdx: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdx: calling ioctl to re-read partition table: Success
Formatting /dev/sdx...
Labeling /dev/sdx1...
Serial Number: R5GBLSLV
UUID: UNKNOWN
New label:

Drive /dev/sdaa:
Wiping /dev/sdaa...
/dev/sdaa: 8 bytes were erased at offset 0x00001000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdaa: 8 bytes were erased at offset 0x74702555000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdaa: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdaa: calling ioctl to re-read partition table: Success
Formatting /dev/sdaa...
Labeling /dev/sdaa1...
Serial Number: 7HK1E73F
UUID: UNKNOWN
New label:

Drive /dev/sdv:
Wiping /dev/sdv...
/dev/sdv: 8 bytes were erased at offset 0x00001000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdv: 8 bytes were erased at offset 0x74702555000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdv: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdv: calling ioctl to re-read partition table: Success
Formatting /dev/sdv...
Labeling /dev/sdv1...
Serial Number: R5GBW83V
UUID: UNKNOWN
New label:

Drive /dev/sdt:
Wiping /dev/sdt...
/dev/sdt: 8 bytes were erased at offset 0x00001000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdt: 8 bytes were erased at offset 0x74702555000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdt: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdt: calling ioctl to re-read partition table: Success
Formatting /dev/sdt...
Labeling /dev/sdt1...
Serial Number: R5GBPL1V
UUID: UNKNOWN
New label:

Drive /dev/sdr:
Wiping /dev/sdr...
/dev/sdr: 8 bytes were erased at offset 0x00001000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdr: 8 bytes were erased at offset 0x74702555000 (gpt): 45 46 49 20 50 41 52 54
/dev/sdr: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdr: calling ioctl to re-read partition table: Success
Formatting /dev/sdr...
Labeling /dev/sdr1...
Serial Number: R5GG7BVV
UUID: UNKNOWN
New label:

Operation complete for all specified drives.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.