openzfsonosx / openzfs Goto Github PK

OpenZFS on Linux and FreeBSD, and, macOS. This is where development for macOS happens.

Home Page: https://github.com/openzfs/zfs/wiki

License: Other

Shell 15.52% Python 1.75% C 76.66% Awk 0.01% C++ 1.27% Assembly 1.86% Perl 0.13% Makefile 0.71% M4 1.81% Roff 0.05% Lua 0.16% sed 0.01% Objective-C++ 0.01% Rich Text Format 0.05% JavaScript 0.02%

openzfs macos osx

openzfs's Introduction

OpenZFS is an advanced file system and volume manager which was originally developed for Solaris and is now maintained by the OpenZFS community. This repository contains the code for running OpenZFS on Linux and FreeBSD.

Official Resources

Documentation - for using and developing this repo
ZoL Site - Linux release info & links
Mailing lists
OpenZFS site - for conference videos and info on other platforms (illumos, OSX, Windows, etc)

Installation

Full documentation for installing OpenZFS on your favorite operating system can be found at the Getting Started Page.

Contribute & Develop

We have a separate document with contribution guidelines.

We have a Code of Conduct.

Release

OpenZFS is released under a CDDL license. For more details see the NOTICE, LICENSE and COPYRIGHT files; UCRL-CODE-235197

Supported Kernels

The META file contains the officially recognized supported Linux kernel versions.
Supported FreeBSD versions are any supported branches and releases starting from 12.2-RELEASE.

openzfs's People

Contributors

Stargazers

Watchers

Forkers

xcodeuuuuu66699 rottegift glessard ilovezfs larsf jmovs somestupidgirl almuslah testlabor k2662

openzfs's Issues

"Disk not ejected properly" when unmounting any devdisk=on dataset

Note that this affects everyone since the pool dataset is devdisk on by default.

joe@joes-Mac ~ % sudo zfs unmount foo/bar 
Volume bar on disk3s2 unmounted

A notification pops up:

DISK NOT EJECTED PROPERLY
Eject "bar" before disconnecting or turning it off

My first guess is that we may need to change the IOkit type of the devdisks to be "removable" drives not "internal" drives.

joe@joes-Mac ~ % zpool version
zfs-2.0.1-0
zfs-kmod-macos-2.0.0-rc6-2-g29afe03e29-dirty
joe@joes-Mac ~ % sw_vers      
ProductName:	macOS
ProductVersion:	11.3
BuildVersion:	20E232
joe@joes-Mac ~ %

kext does not load on 10.16 Big Sur

Appears to be related to the Plugins/KernelDependencies.kext/ that we have to link against things we are not allowed to link against.

A straight load attempt fails with:

Error: Error Domain=KMErrorDomain Code=1 "Unable to resolve dependencies: 'net.lundman.zfs' names a 
dependency on 'net.lundman.kernel.dependencies.0.8.0', which was not found." UserInfo=
 {NSLocalizedDescription=Unable to resolve dependencies: 'net.lundman.zfs' names a dependency on
 'net.lundman.kernel.dependencies.0.8.0', which was not found.}

As if it doesn't look in Plugins anymore. However, when if I attempt to codesign the Plugins kext;

codesign_allocate: can't allocate code signature data for: 
/private/tmp/zfs.kext/Contents/PlugIns/KernelExports.kext/KernelExports (for architecture x86_64) because 
file does not have a __LINKEDIT segment

Which makes me think that they've made a slight change to the requirements of kexts, and kextsymboltool no
longer produces a valid object file.

Similarly, if I try to load the 1.9.4 signed/notarized kexts:

 "Error occurred while building cache:
in '/Library/Extensions/spl.kext/Contents/PlugIns/KernelExports.kext/KernelExports' missing __TEXT segment"

So I think we need to figure out how to make the output of kextsymboltool be accepted under Big Sur.

In addition, the output from kextload/kextutil is not exactly great - the real errors must go somewhere?

panic: reservation_013_pos

panic(cpu 0 caller 0xffffff7fa3e0b3d2): VERIFY3(tx->tx_txg <= spa_final_dirty_txg(os->os_spa)) failed (1274 <= 1271)
Backtrace (CPU 0), Frame : Return Address
0xffffff80833eb390 : 0xffffff8020ffcdcd mach_kernel : _handle_debugger_trap + 0x48d
0xffffff80833eb3e0 : 0xffffff8021181abb mach_kernel : _kdp_i386_trap + 0x19b
0xffffff80833eb420 : 0xffffff8021171ad3 mach_kernel : _kernel_trap + 0x3d3
0xffffff80833eb490 : 0xffffff8020f95a40 mach_kernel : trap_from_kernel + 0x26
0xffffff80833eb4b0 : 0xffffff8020ffd100 mach_kernel : _DebuggerTrapWithState + 0x20
0xffffff80833eb5b0 : 0xffffff8020ffc597 mach_kernel : _panic_trap_to_debugger + 0x247
0xffffff80833eb600 : 0xffffff802186a0cc mach_kernel : _panic + 0x54
0xffffff80833eb670 : 0xffffff7fa3e0b3d2 net.lundman.zfs : _spl_panic + 0xe2
0xffffff80833eb830 : 0xffffff7fa3b3c3dd net.lundman.zfs : _dbuf_dirty + 0x1dd
0xffffff80833eb8d0 : 0xffffff7fa3b3cd84 net.lundman.zfs : _dmu_buf_will_dirty_impl + 0x104
0xffffff80833eb920 : 0xffffff7fa3b3c1b2 net.lundman.zfs : _dmu_buf_will_dirty + 0x22
0xffffff80833eb940 : 0xffffff7fa3c4b8d9 net.lundman.zfs : _zap_lockdir_impl + 0x209
0xffffff80833eba20 : 0xffffff7fa3c4b6a0 net.lundman.zfs : _zap_lockdir + 0x90
0xffffff80833eba90 : 0xffffff7fa3c4d8e7 net.lundman.zfs : _zap_update + 0x67
0xffffff80833ebb30 : 0xffffff7fa3c58830 net.lundman.zfs : _feature_sync + 0xa0
0xffffff80833ebbb0 : 0xffffff7fa3c590cf net.lundman.zfs : _feature_do_action + 0x1df
0xffffff80833ebc40 : 0xffffff7fa3c58da7 net.lundman.zfs : _spa_feature_incr + 0x27
0xffffff80833ebc70 : 0xffffff7fa3c0a779 net.lundman.zfs : _space_map_alloc + 0x49
0xffffff80833ebcc0 : 0xffffff7fa3bffdda net.lundman.zfs : _spa_generate_syncing_log_sm + 0x1da
0xffffff80833ebd90 : 0xffffff7fa3bffa88 net.lundman.zfs : _spa_flush_metaslabs + 0xc8
0xffffff80833ebde0 : 0xffffff7fa3befd46 net.lundman.zfs : _spa_sync_iterate_to_convergence + 0x186
0xffffff80833ebe30 : 0xffffff7fa3bef5ea net.lundman.zfs : _spa_sync + 0x52a
0xffffff80833ebf10 : 0xffffff7fa3c0c458 net.lundman.zfs : _txg_sync_thread + 0x2f8

Reusing the name of a destroyed snapshot can cause unmount trouble

If a snapshot named s0 is

created
mounted
unmounted
destroyed
recreated
mounted

then it cannot be unmounted unless force is used.

Here's a reproducer:

joe@joes-Mac ~ % cat snapshot-reuse-name.sh 
#!/bin/sh -x
zpool create foo disk0
touch /Volumes/foo/hello
sleep 1 && sync && zpool sync
zfs snapshot foo@s0
zfs mount foo@s0
zfs unmount foo@s0
zfs destroy foo@s0
zfs snapshot foo@s0
zfs mount foo@s0
zfs unmount foo@s0
echo "Why did that unmount fail?"
zfs unmount foo@s0
echo "Trying again does not help"
ls -al /Volumes/foo/.zfs/snapshot/s0
mount | grep s0
echo "But it is supposedly mounted"
zfs snapshot foo@s1
zfs mount foo@s1
ls -al /Volumes/foo/.zfs/snapshot/s1
zfs unmount foo@s1
echo "A different snapshot name is fine"
#zfs mount foo@s0
zfs unmount -f foo@s0
zfs mount foo@s0
ls -al /Volumes/foo/.zfs/snapshot/s0
zfs unmount foo@s0
echo "Now everything appears after a force unmount and remount"
zpool destroy foo
joe@joes-Mac ~ %

Demo running it:

joe@joes-Mac ~ % cat snapshot-reuse-name.sh 
#!/bin/sh -x
zpool create foo disk0
touch /Volumes/foo/hello
sleep 1 && sync && zpool sync
zfs snapshot foo@s0
zfs mount foo@s0
zfs unmount foo@s0
zfs destroy foo@s0
zfs snapshot foo@s0
zfs mount foo@s0
zfs unmount foo@s0
echo "Why did that unmount fail?"
zfs unmount foo@s0
echo "Trying again does not help"
ls -al /Volumes/foo/.zfs/snapshot/s0
mount | grep s0
echo "But it is supposedly mounted"
zfs snapshot foo@s1
zfs mount foo@s1
ls -al /Volumes/foo/.zfs/snapshot/s1
zfs unmount foo@s1
echo "A different snapshot name is fine"
#zfs mount foo@s0
zfs unmount -f foo@s0
zfs mount foo@s0
ls -al /Volumes/foo/.zfs/snapshot/s0
zfs unmount foo@s0
echo "Now everything appears after a force unmount and remount"
zpool destroy foo
joe@joes-Mac ~ %

Note: This issue was discovered while investigating why xattr_007_neg fails only if it is run after a successful run of xattr_006_pos. The reason is that they both use the name snap for a snapshot of the $TESTPOOL/$TESTFS dataset (and they mount the snapshot). xattr_006_pos destroys the snapshot during cleanup and a snapshot of the same name is created during xattr_007_neg. Then xattr_007_neg appears to successfully mount snap but it actually hasn't. Attempting to unmount it requires force. If snap is then remounted after the force unmount, the mount is actually successful and the test passes.

Kernel Panic after a couple of minutes while using APFS inside zvol on mirror-pool

System information

Type	Version/Name
Distribution Name	MacOS
Distribution Version	11, Big Sur
Linux Kernel
Architecture	ARM M1
ZFS Version	2.1 rc1
SPL Version

Describe the problem you're observing

Kernel Panic after around 15 minutes of making time machine backup to APFS inside zvol

Describe how to reproduce the problem

Start Time Machine backup to APFS inside zvol, wait

Include any warning/errors/backtraces from the system logs

https://gist.github.com/JMoVS/21e46b2813bf2196fe1876bdccc6f029 (copy pasted version, here the direct file: https://gist.github.com/JMoVS/31ab6dc11516b200799258d5fd6a6905)
https://gist.github.com/JMoVS/1f0231732ea493c816b67ad40a80ad95
https://gist.github.com/JMoVS/33ac87f3c7faa127d6e026d24e96b69d

Big Sur Input/output error when unmounting dataset whose mountpoint is gone

This behavior seems specific to Big Sur as it does not occur on Catalina.

joe@joes-Mac ~ % ./bigsur-unmount-fail.sh 
+ sw_vers
ProductName:	macOS
ProductVersion:	11.3
BuildVersion:	20E232
+ zpool version
zfs-2.0.1-0
zfs-kmod-macos-2.0.0-rc6-2-g29afe03e29-dirty
+ NUMSECTORS=1280000
++ hdiutil attach -nomount ram://1280000
+ mydev='/dev/disk2          	                               	'
+ sudo zpool create foo /dev/disk2
+ sudo zfs snapshot foo@s1
+ sudo zfs create foo/bar
+ sudo zfs rollback foo@s1
+ sudo zfs unmount foo/bar
Unmount failed for /Volumes/foo/bar
Fallback umount called
umount: unmount(/Volumes/foo/bar): Input/output error
cannot unmount '/Volumes/foo/bar': unmount failed
+ sudo zfs unmount -f foo/bar
Unmount failed for /Volumes/foo/bar
Fallback umount called
umount: unmount(/Volumes/foo/bar): Input/output error
cannot unmount '/Volumes/foo/bar': unmount failed
+ sudo zfs unmount foo
Unmount failed for /Volumes/foo/bar
Fallback umount called
umount: unmount(/Volumes/foo/bar): Input/output error
cannot unmount '/Volumes/foo/bar': unmount failed
+ sudo zfs unmount -f foo
Unmount failed for /Volumes/foo/bar
Fallback umount called
umount: unmount(/Volumes/foo/bar): Input/output error
cannot unmount '/Volumes/foo/bar': unmount failed
+ sudo umount -f /Volumes/foo/bar
umount: unmount(/Volumes/foo/bar): Input/output error
+ sudo umount -f /Volumes/foo
+ sudo zfs mount -a
+ sudo zpool export foo
Unmount successful for /Volumes/foo/bar
Volume foo on disk3s1 unmounted
joe@joes-Mac ~ %

The script is here: https://gist.github.com/ilovezfs/92978595c91e28dce0090a5484988cb5

Note that the only way to recover is to umount -f the mountpoint of the parent dataset. zfs unmount -f is ineffective (I suspect the umount fallback is not passing the -f option even zfs unmount -f was called.

On FreeBSD and OmniOS, you can zfs unmount foo/bar && zfs mount foo/bar and do not need to force unmount the parent dataset to recover. On Linux, the operating system automatically removes the mount after a few seconds if its mountpoint has gone missing, and so you just need to run zfs mount foo/bar.

zfs receive from file is broken

The error is "cannot receive new filesystem stream: invalid backup stream"

2.0-rc6:

joes-Mac-2:~ root# zpool create tank disk4
joes-Mac-2:~ root# echo hello > /Volumes/tank/s1.txt
joes-Mac-2:~ root# zfs snapshot tank@s1
joes-Mac-2:~ root# zfs send tank@s1 | zfs recv tank/frompipe
joes-Mac-2:~ root# cat /Volumes/tank/frompipe/s1.txt 
hello
joes-Mac-2:~ root# zfs send tank@s1 > tanks1.bin 
joes-Mac-2:~ root# zfs recv tank/fromfile < tanks1.bin 
cannot receive new filesystem stream: invalid backup stream
joes-Mac-2:~ root# cat tanks1.bin | zfs recv tank/fromfile
joes-Mac-2:~ root# cat /Volumes/tank/fromfile/s1.txt 
hello

Fine in 1.9.4:

joes-Mac-3:~ root# zpool create tank disk0
joes-Mac-3:~ root# echo hello > /Volumes/tank/s1.txt
joes-Mac-3:~ root# zfs snapshot tank@s1
joes-Mac-3:~ root# zfs send tank@s1 | zfs recv tank/frompipe
Mount successful           
joes-Mac-3:~ root# cat /Volumes/tank/frompipe/s1.txt 
hello
joes-Mac-3:~ root# zfs send tank@s1 > tanks1.bin
joes-Mac-3:~ root# zfs recv tank/fromfile < tanks1.bin 
Mount successful
joes-Mac-3:~ root# cat /Volumes/tank/fromfile/s1.txt 
hello
joes-Mac-3:~ root#

zpool status shows "rdisk0s1" instead of "disk0"

Importing a pool -d /dev displays disks incorrectly after a pool has previously been imported with an invariant path.

2.0.0-rc6:

joe@joes-Mac-2 ~ % sudo zpool create tank disk0
joe@joes-Mac-2 ~ % zpool status|grep ONLINE
 state: ONLINE
	tank        ONLINE       0     0     0
	  disk0     ONLINE       0     0     0
joe@joes-Mac-2 ~ % sudo zpool export tank &>/dev/null
joe@joes-Mac-2 ~ % sudo zpool import tank
joe@joes-Mac-2 ~ % zpool status|grep ONLINE
 state: ONLINE
	tank                                          ONLINE       0     0     0
	  media-79CBED37-F792-8C40-A10C-10C5A750F7F9  ONLINE       0     0     0
joe@joes-Mac-2 ~ % sudo zpool export tank &>/dev/null 
joe@joes-Mac-2 ~ % sudo zpool import -d /dev tank     
joe@joes-Mac-2 ~ % zpool status|grep ONLINE 
 state: ONLINE
	tank        ONLINE       0     0     0
	  rdisk0s1  ONLINE       0     0     0
joe@joes-Mac-2 ~ % zdb|grep path   
            path: '/dev/rdisk0s1'

1.9.4:

joe@joes-Mac-3 ~ % sudo zpool create tank disk0
joe@joes-Mac-3 ~ % zpool status|grep ONLINE
 state: ONLINE
	tank        ONLINE       0     0     0
	  disk0     ONLINE       0     0     0
joe@joes-Mac-3 ~ % sudo zpool export tank &>/dev/null
joe@joes-Mac-3 ~ % sudo zpool import tank
joe@joes-Mac-3 ~ % zpool status|grep ONLINE          
 state: ONLINE
	tank                                          ONLINE       0     0     0
	  media-D2C22FEF-550B-A04C-AAFC-FCD70DB47468  ONLINE       0     0     0
joe@joes-Mac-3 ~ % sudo zpool export tank &>/dev/null 
joe@joes-Mac-3 ~ % sudo zpool import -d /dev tank
joe@joes-Mac-3 ~ % zpool status|grep ONLINE          
 state: ONLINE
	tank        ONLINE       0     0     0
	  disk0     ONLINE       0     0     0
joe@joes-Mac-3 ~ % zdb|grep path           
            path: '/dev/disk0s1'

Note that it does not occur until the pool has been imported with an invariant path at least once.

joe@joes-Mac-2 ~ % sudo zpool create tank disk0
joe@joes-Mac-2 ~ % sudo zpool export tank &>/dev/null
joe@joes-Mac-2 ~ % sudo zpool import -d /dev tank
joe@joes-Mac-2 ~ % zpool status|grep ONLINE
 state: ONLINE
	tank        ONLINE       0     0     0
	  disk0     ONLINE       0     0     0
joe@joes-Mac-2 ~ % sudo zpool export tank &>/dev/null
joe@joes-Mac-2 ~ % sudo zpool import -d /dev tank    
joe@joes-Mac-2 ~ % zpool status|grep ONLINE          
 state: ONLINE
	tank        ONLINE       0     0     0
	  disk0     ONLINE       0     0     0
joe@joes-Mac-2 ~ % sudo zpool export tank &>/dev/null
joe@joes-Mac-2 ~ % sudo zpool import -d /var/run/disk/by-serial tank 
joe@joes-Mac-2 ~ % zpool status|grep ONLINE                          
 state: ONLINE
	tank                                                     ONLINE       0     0     0
	  VMware_Virtual_SATA_Hard_Drive-03000000000000000001:1  ONLINE       0     0     0
joe@joes-Mac-2 ~ % sudo zpool export tank &>/dev/null                
joe@joes-Mac-2 ~ % sudo zpool import -d /dev tank                    
joe@joes-Mac-2 ~ % zpool status|grep ONLINE          
 state: ONLINE
	tank        ONLINE       0     0     0
	  rdisk0s1  ONLINE       0     0     0

zfs unmount mountpoint is broken

joe@joes-Mac ~ % zfs mount                     
foo                             /Volumes/foo
joe@joes-Mac ~ % sudo zfs unmount /Volumes/foo
cannot open 'J????????'??*cannot open '? ȫA}#?#???U??7QR?T????	ژ???<r??????|"^?.?
?71?s????+L????z?ٴ?OP/om?Aѡ??8???e,?;Ó~.????и
??[??&?': bookmark delimiter '#' is not expected here
joe@joes-Mac ~ % zpool version
zfs-2.0.1-0
zfs-kmod-macos-2.0.0-rc6-2-g29afe03e29-dirty
joe@joes-Mac ~ % sw_vers
ProductName:	macOS
ProductVersion:	11.3
BuildVersion:	20E232
joe@joes-Mac ~ %

This works fine on Ubuntu and OmniOS and is supported according to the man page.

sa.sa_magic panic in zfs_space_delta_cb()

System information

Type	Version/Name
Distribution Name	macOS
Distribution Version	I've not given it a version yet
Linux Kernel	heh funny
Architecture	x64
ZFS Version	0.8.4
SPL Version

Describe the problem you're observing

panic(cpu 1 caller 0xffffff7fadbeaa0e): VERIFY3(sa.sa_magic == SA_MAGIC) failed (1591864135 == 3100762)

Backtrace (CPU 1), Frame : Return Address
0xffffff80321338c0 : 0xffffff802b29052d mach_kernel : _handle_debugger_trap + 0x46d
0xffffff8032133910 : 0xffffff802b424826 mach_kernel : _kdp_i386_trap + 0x196
0xffffff8032133950 : 0xffffff802b413e42 mach_kernel : _kernel_trap + 0x3c2
0xffffff80321339c0 : 0xffffff802b22bb40 mach_kernel : trap_from_kernel + 0x26
0xffffff80321339e0 : 0xffffff802b28fc25 mach_kernel : _panic_trap_to_debugger + 0x1c5
0xffffff8032133b10 : 0xffffff802b28fa43 mach_kernel : _panic + 0x63
0xffffff8032133b80 : 0xffffff7fadbeaa0e net.lundman.zfs : _spl_panic + 0xce
0xffffff8032133d30 : 0xffffff7fadb00ce5 net.lundman.zfs : _zfs_space_delta_cb + 0x185
0xffffff8032133dc0 : 0xffffff7fad9d0817 net.lundman.zfs : _dmu_objset_userquota_get_ids + 0x367
0xffffff8032133e30 : 0xffffff7fad9f5773 net.lundman.zfs : _dnode_sync + 0x183
0xffffff8032133eb0 : 0xffffff7fad9d24ad net.lundman.zfs : _dmu_objset_sync_dnodes + 0xfd
0xffffff8032133ef0 : 0xffffff7fad9cfcbc net.lundman.zfs : _sync_dnodes_task + 0x3c
0xffffff8032133f20 : 0xffffff7fadc00ac1 net.lundman.zfs : _taskq_thread + 0x3a1
0xffffff8032133fa0 : 0xffffff802b22b0ce mach_kernel : _call_continuation + 0x2e

Describe how to reproduce the problem

zfs_upgrade/zfs_upgrade_004_pos triggers it easily.

Include any warning/errors/backtraces from the system logs

Possible explanation:

openzfs/zfs#2025 (comment)

Since it is relatively easy to trigger on macOS, it may help to debug it, should upstream people be into it.

apfs on zvol is considered as "mounted from a disk image" from the point of macOS

System information

Type	Version/Name
Distribution Name	macOS Big Sur
Distribution Version	11.4
Linux Kernel
Architecture	M1 ARM
ZFS Version	2.1.0 rc1
SPL Version

Describe the problem you're observing

zvol based volumes for time machine aren't showing up as a target as they are, according to log stream --source --predicate 'subsystem == "com.apple.TimeMachine"' --style compact --info --debug`, mounted from a disk image.

Describe how to reproduce the problem

create a zvol, eg
zfs create -s -V 1G pool/foo
use diskutility to format foo as APFS (either with or without partition table)

Include any warning/errors/backtraces from the system logs

How to uninstall OpenZFSonOsX-2.0.0-Big.Sur-11.0.pkg from M1 (installed by mistake)

Hi,
I installed OpenZFSonOsX-2.0.0-Big.Sur-11.0.pkg on MacBookPro M1 (Apple silicon) by mistake.
What is the best way to do a clean uninstall?
Thanks

Snapshots can be unintentionally created

joe@joes-Mac ~ % cat snapshot-palooza.sh
#!/bin/sh -x
zpool create testpool disk0
zfs create testpool/testfs
zfs snapshot testpool/testfs@snap
zfs mount testpool/testfs@snap 
zfs destroy testpool/testfs@snap 
zfs destroy testpool/testfs@snap
zfs snapshot testpool/testfs@snap
zfs mount testpool/testfs@snap  
zfs list -t all                    
umount -f /Volumes/testpool/testfs/.zfs/snapshot/snap
zpool destroy testpool
joe@joes-Mac ~ %

joe@joes-Mac ~ % sudo ./snapshot-palooza.sh
+ zpool create testpool disk0
+ zfs create testpool/testfs
+ zfs snapshot testpool/testfs@snap
+ zfs mount testpool/testfs@snap
ZFS: snapshot mountpoint '/Volumes/testpool/testfs/.zfs/snapshot/snap/'
+ zfs destroy testpool/testfs@snap
cannot destroy snapshot testpool/testfs@snap: dataset is busy
+ zfs destroy testpool/testfs@snap
+ zfs snapshot testpool/testfs@snap
+ zfs mount testpool/testfs@snap
ZFS: snapshot mountpoint '/Volumes/testpool/testfs/.zfs/snapshot/snap/'
+ zfs list -t all
NAME                              USED  AVAIL     REFER  MOUNTPOINT
testpool                         5.14M  18.9G     2.51M  /Volumes/testpool
testpool/testfs                  2.50M  18.9G     2.50M  /Volumes/testpool/testfs
testpool/testfs@snap                0B      -     2.50M  -
testpool/[email protected]     0B      -     2.50M  -
testpool/testfs@Store-V2            0B      -     2.50M  -
+ umount -f /Volumes/testpool/testfs/.zfs/snapshot/snap
+ zpool destroy testpool
Unmount successful for /Volumes/testpool/testfs
Volume testpool on disk8s1 unmounted
joe@joes-Mac ~ %

Where did testpool/[email protected] and testpool/testfs@Store-V2 come from?

Missing cmd/os/macos/zsysctl

OOM error on `zpool import -f` despite successful import

System information

Type	Version/Name
Distribution Name	macOS
Distribution Version	11.4 beta (20F5065a)
Linux Kernel	Darwin 20.5.0 (root:xnu-7195.121.3~4/RELEASE_X86_64)
Architecture	x86
ZFS Version	zfs-2.0.0-rc6 (zfs-kmod-zfs-2.0.0-rc1-452-g964b2a6de7)
SPL Version	?

Describe the problem you're observing

Importing a zpool created on Ubuntu (zfs-0.8.3-1ubuntu12.8) throws an out of memory error, but succeeds in importing.

Describe how to reproduce the problem

Create a zpool on Ubuntu on a removable device
Unmount the device
Unplug the device and plug it into the macOS device
sudo zpool import -d /dev to discover the pool
sudo zpool import <poolname> -f to import

After step 5, an out of memory message will be displayed:

❯ sudo zpool import downloads -f
internal error: out of memory

... but the pool seems to import fine:

❯ sudo zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
downloads   464G   288G   176G        -         -    32%    62%  1.00x    ONLINE  -


❯ sudo zpool status downloads
pool: downloads
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
    The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 00:03:36 with 0 errors on Sun May  9 00:27:46 2021
config:

NAME                                          STATE     READ WRITE CKSUM
downloads                                     ONLINE       0     0     0
  media-CB39844A-1A02-DB44-B898-F37433593365  ONLINE       0     0     0

errors: No known data errors

I'm unsure if the features warning is related to the OOM error, but zpool upgrade also works fine:

❯ sudo zpool status downloads
pool: downloads
state: ONLINE
scan: scrub repaired 0B in 00:03:36 with 0 errors on Sun May  9 00:27:46 2021
config:

NAME                                          STATE     READ WRITE CKSUM
downloads                                     ONLINE       0     0     0
  media-CB39844A-1A02-DB44-B898-F37433593365  ONLINE       0     0     0

errors: No known data errors

Include any warning/errors/backtraces from the system logs

From org.openzfsonosx.zpool-import-all.log:

Mon May 17 16:03:24 EDT 2021
Running zpool import -a
Mon May 17 16:03:36 EDT 2021
Mon May 17 16:03:37 EDT 2021
Launched zpool import -a : 0
Touching the file /var/run/org.openzfsonosx.zpool-import-all.didRun
-zpool-import-all.sh
internal error: out of memory

snapdir ls: /Volumes/mypool/.zfs/snapshot/s1/: Not a directory

Initially things look like they're working:

joe@joes-Mac-2 ~ % sudo zpool create mypool disk2
joe@joes-Mac-2 ~ % sudo chown joe /Volumes/mypool
joe@joes-Mac-2 ~ % touch /Volumes/mypool/s1.txt  
joe@joes-Mac-2 ~ % sudo zfs snap mypool@s1             
joe@joes-Mac-2 ~ % touch /Volumes/mypool/s2.txt
joe@joes-Mac-2 ~ % sudo zfs snap mypool@s2     
joe@joes-Mac-2 ~ % touch /Volumes/mypool/s3.txt
joe@joes-Mac-2 ~ % sudo zfs snap mypool@s3     
joe@joes-Mac-2 ~ % ls /Volumes/mypool/.zfs/snapshot/s1/
s1.txt
joe@joes-Mac-2 ~ % ls /Volumes/mypool/.zfs/snapshot/s2/
s1.txt	s2.txt
joe@joes-Mac-2 ~ % ls /Volumes/mypool/.zfs/snapshot/s3/
s1.txt	s2.txt	s3.txt

But then they look like they're mostly broken:

joe@joes-Mac-2 ~ % sudo zpool export mypool
Unmount successful for /Volumes/mypool/.zfs/snapshot/s1
Unmount successful for /Volumes/mypool/.zfs/snapshot/s2
Unmount successful for /Volumes/mypool/.zfs/snapshot/s3
Volume mypool on disk6s1 unmounted
joe@joes-Mac-2 ~ % sudo zpool import mypool
joe@joes-Mac-2 ~ % ls /Volumes/mypool/.zfs/snapshot/s1/
ls: : Not a directory
joe@joes-Mac-2 ~ % ls /Volumes/mypool/.zfs/snapshot/s2/
s1.txt	s2.txt
joe@joes-Mac-2 ~ % ls /Volumes/mypool/.zfs/snapshot/s3/
ls: : Not a directory
joe@joes-Mac-2 ~ % ls /Volumes/mypool/.zfs/snapshot/   
s1	s2	s3
joe@joes-Mac-2 ~ % sudo file /Volumes/mypool/.zfs/snapshot/s1
/Volumes/mypool/.zfs/snapshot/s1: data
joe@joes-Mac-2 ~ % sudo file /Volumes/mypool/.zfs/snapshot/s2
/Volumes/mypool/.zfs/snapshot/s2: directory
joe@joes-Mac-2 ~ % sudo file /Volumes/mypool/.zfs/snapshot/s3
/Volumes/mypool/.zfs/snapshot/s3: data
joe@joes-Mac-2 ~ % touch /Volumes/mypool/s4.txt              
joe@joes-Mac-2 ~ % sudo zfs snap mypool@s4                   
joe@joes-Mac-2 ~ % ls /Volumes/mypool/.zfs/snapshot/s4   
s1.txt	s2.txt	s3.txt	s4.txt
joe@joes-Mac-2 ~ % touch /Volumes/mypool/s5.txt     
joe@joes-Mac-2 ~ % sudo zfs snap mypool@s5              
joe@joes-Mac-2 ~ % ls -l /Volumes/mypool/.zfs/snapshot/
total 9
-rw-------  1 joe  wheel   3277 May  3 04:14 s1
drwxr-xr-x  6 joe  wheel      6 May  3 04:14 s2
-rw-------  1 joe  wheel  65536 May  3 04:14 s3
drwxr-xr-x  8 joe  wheel      8 May  3 04:17 s4
drwx------  2 joe  wheel      2 May  3 04:14 s5
joe@joes-Mac-2 ~ % mount|grep 's3'
mypool@s3 on /Volumes/mypool/.zfs/snapshot/s3 (zfs, local, read-only, journaled, noatime)
joe@joes-Mac-2 ~ % ls -l /Volumes/mypool/.zfs/snapshot/s3/
ls: /Volumes/mypool/.zfs/snapshot/s3/: Not a directory

zfs send output sometimes incorrect

10:42 PM <ilovezfs_> lundman: it looks like zfs send -p is acting up
10:42 PM <ilovezfs_> joes-Mac:~ root# echo moo > /Volumes/foo/hello
10:42 PM <ilovezfs_> joes-Mac:~ root# zfs snapshot foo@s0
10:42 PM <ilovezfs_> joes-Mac:~ root# zfs send -p foo@s0 > A
10:42 PM <ilovezfs_> joes-Mac:~ root# zfs send -p foo@s0 | tee B > /dev/null
10:42 PM <ilovezfs_> joes-Mac:~ root# ls -l A B
10:42 PM <ilovezfs_> -rw-r--r-- 1 root wheel 1408 May 28 22:40 A
10:42 PM <ilovezfs_> -rw-r--r-- 1 root wheel 2736032 May 28 22:41 B
10:44 PM <ilovezfs_> and tests/functional/cli_root/zfs_send/zfs_send-b.ksh is also struggling
10:48 PM <ilovezfs_> e.g. it can fail with
10:48 PM <ilovezfs_> ERROR: eval zfs send -bcew testpool/backup@s1 | zfs recv testpool/restore exited 1
10:48 PM <ilovezfs_> cannot receive new filesystem stream: incomplete stream
10:50 PM <ilovezfs_> and
10:50 PM <ilovezfs_> ERROR: eval zfs send -p testpool/sendfs@s1 | zfs recv testpool/backup exited 1
10:50 PM <ilovezfs_> cannot receive new filesystem stream: incomplete stream
11:10 PM <ilovezfs_> lundman: also problems with send in the test large_dnode_005_pos
11:10 PM <ilovezfs_> ERROR: eval zfs recv testpool/recv_large_dnode < /private/var/tmp/testdir/ldnsnap exited 1
11:10 PM <ilovezfs_> cannot receive: invalid stream (bad magic number)

"zfs unmount" broken with mimic=hfs for child datasets

The error is cannot unmount 'mypool/fs1': not currently mounted and zfs get mounted also doesn't think the file system is mounted.

2.0 rc6:

joes-Mac-2:~ root# zpool create mypool disk4
joes-Mac-2:~ root# zfs create mypool/fs1
joes-Mac-2:~ root# zfs set com.apple.mimic=hfs mypool/fs1
joes-Mac-2:~ root# zfs unmount mypool/fs1
cannot unmount 'mypool/fs1': not currently mounted
joes-Mac-2:~ root# mount|grep mypool
/dev/disk6s1 on /Volumes/mypool (zfs, local, journaled)
mypool/fs1 on /Volumes/mypool/fs1 (hfs, local, journaled)
joes-Mac-2:~ root# umount /Volumes/mypool/fs1
joes-Mac-2:~ root# mount|grep mypool
/dev/disk6s1 on /Volumes/mypool (zfs, local, journaled)
joes-Mac-2:~ root# zfs mount mypool/fs1
joes-Mac-2:~ root# zpool export mypool
Volume mypool on disk6s1 failed to unmount: dissented by PID 0 (kernel_task)
Fallback umount called
umount(/Volumes/mypool): Resource busy -- try 'diskutil unmount'
cannot unmount '/Volumes/mypool': unmount failed
joes-Mac-2:~ root# umount /Volumes/mypool/fs1
umount(/Volumes/mypool/fs1): Resource busy -- try 'diskutil unmount'
joes-Mac-2:~ root# umount -f /Volumes/mypool/fs1
joes-Mac-2:~ root# zpool export mypool
Volume mypool on disk6s1 unmounted
joes-Mac-2:~ root# zpool import mypool
joes-Mac-2:~ root# zfs unmount mypool/fs1
cannot unmount 'mypool/fs1': not currently mounted
joes-Mac-2:~ root# zfs get mounted
NAME            PROPERTY  VALUE    SOURCE
dank            mounted   yes      -
goodbye         mounted   yes      -
goodbye/myzvol  mounted   -        -
mypool          mounted   yes      -
mypool/fs1      mounted   no       -
joes-Mac-2:~ root# zpool version
zfs-2.0.0-rc6
zfs-kmod-macos-2.0.0-rc6-2-g29afe03e2
joes-Mac-2:~ root# mount|grep mypool
mypool on /Volumes/mypool (zfs, local, journaled)
mypool/fs1 on /Volumes/mypool/fs1 (hfs, local, journaled)
joes-Mac-2:~ root#

In 1.9.4 it's fine:

joes-Mac-3:~ root# zpool create mypool disk1
joes-Mac-3:~ root# zfs create mypool/fs1
joes-Mac-3:~ root# zfs set com.apple.mimic=on mypool/fs1
cannot set property for 'mypool/fs1': invalid property 'com.apple.mimic'
joes-Mac-3:~ root# zfs set com.apple.mimic_hfs=on mypool/fs1
joes-Mac-3:~ root# zfs unmount mypool/fs1
Running process: '/usr/sbin/diskutil' 'unmount' '/Volumes/mypool/fs1' 
Unmount successful for /Volumes/mypool/fs1
joes-Mac-3:~ root# mount|grep mypool
/dev/disk7s1 on /Volumes/mypool (zfs, local, journaled)
joes-Mac-3:~ root# zfs mount mypool/fs1
joes-Mac-3:~ root# zpool export mypool
Running process: '/usr/sbin/diskutil' 'unmount' '/Volumes/mypool/fs1' 
Unmount successful for /Volumes/mypool/fs1
Running process: '/usr/sbin/diskutil' 'unmount' '/Volumes/mypool' 
Volume mypool on disk7s1 unmounted
joes-Mac-3:~ root# zpool import mypool
joes-Mac-3:~ root# zfs unmount mypool/fs1
Running process: '/usr/sbin/diskutil' 'unmount' '/Volumes/mypool/fs1' 
Unmount successful for /Volumes/mypool/fs1
joes-Mac-3:~ root# zfs get mounted
NAME              PROPERTY  VALUE    SOURCE
mypool            mounted   yes      -
mypool/fs1        mounted   no       -
tank              mounted   yes      -
tank@s1           mounted   no       -
tank/foo          mounted   yes      -
tank/foo2         mounted   yes      -
tank/fromfile     mounted   yes      -
tank/fromfile@s1  mounted   no       -
tank/frompipe     mounted   yes      -
tank/frompipe@s1  mounted   no       -
joes-Mac-3:~ root# mount|grep mypool
mypool on /Volumes/mypool (zfs, local, journaled)
joes-Mac-3:~ root# zfs mount mypool/fs1
joes-Mac-3:~ root# mount|grep mypool
mypool on /Volumes/mypool (zfs, local, journaled)
mypool/fs1 on /Volumes/mypool/fs1 (hfs, local, journaled)
joes-Mac-3:~ root# zfs get mounted
NAME              PROPERTY  VALUE    SOURCE
mypool            mounted   yes      -
mypool/fs1        mounted   yes      -
tank              mounted   yes      -
tank@s1           mounted   no       -
tank/foo          mounted   yes      -
tank/foo2         mounted   yes      -
tank/fromfile     mounted   yes      -
tank/fromfile@s1  mounted   no       -
tank/frompipe     mounted   yes      -
tank/frompipe@s1  mounted   no       -
joes-Mac-3:~ root# zpool version
zfs-1.9.4-0
zfs-kmod-1.9.4-0
joes-Mac-3:~ root#

zfs mount with no sudo prints "internal error: out of memory"

In 2.0-rc6 zfs mount tank without sudo prints an out of memory error:

joe@joes-Mac ~ % sudo zfs unmount tank
cannot unmount 'tank': not currently mounted
joe@joes-Mac ~ % mkdir /Users/joe/mymountpoint
joe@joes-Mac ~ % zfs set mountpoint=/Users/joe/mymountpoint tank
cannot set property for 'tank': permission denied
joe@joes-Mac ~ % sudo zfs set mountpoint=/Users/joe/mymountpoint tank
joe@joes-Mac ~ % zfs mount tank 
internal error: out of memory
joe@joes-Mac ~ % mount |grep mymountpoint
joe@joes-Mac ~ % sudo zfs mount tank    
joe@joes-Mac ~ % mount |grep mymountpoint
/dev/disk8s1 on /Users/joe/mymountpoint (zfs, local, journaled)
joe@joes-Mac ~ % zfs unmount tank
Volume tank on disk8s1 unmounted
joe@joes-Mac ~ %

In 1.9.4 zfs mount tank without sudo works fine:

joe@joes-Mac-3 ~ % sudo zfs unmount tank
cannot unmount 'tank': not currently mounted
joe@joes-Mac-3 ~ % mkdir /Users/joe/mymountpoint
joe@joes-Mac-3 ~ % zfs set mountpoint=/Users/joe/mymountpoint tank
cannot set property for 'tank': permission denied
joe@joes-Mac-3 ~ % sudo zfs set mountpoint=/Users/joe/mymountpoint tank
joe@joes-Mac-3 ~ % zfs mount tank        
joe@joes-Mac-3 ~ % mount |grep mymountpoint
/dev/disk6s1 on /Users/joe/mymountpoint (zfs, local, nodev, nosuid, journaled, mounted by joe)
joe@joes-Mac-3 ~ % zfs unmount tank       
Running process: '/usr/sbin/diskutil' 'unmount' '/Users/joe/mymountpoint' 
Volume tank on disk6s2 unmounted
joe@joes-Mac-3 ~ % sudo zfs mount tank
joe@joes-Mac-3 ~ % mount|grep mymountpoint
/dev/disk6s1 on /Users/joe/mymountpoint (zfs, local, journaled)
joe@joes-Mac-3 ~ %

joe@joes-Mac ~ % zpool version
zfs-2.0.0-rc6
zfs-kmod-macos-2.0.0-rc6-13-g818243b98-dirty
joe@joes-Mac ~ %

buffer freed to wrong cache

Interesting to note, it triggers with:

./scripts/cmd-macos.sh zpool create -f -o ashift=12 -O casesensitivity=insensitive -O normalization=formD -O compression=lz4 -O checksum=edonr BOOM raidz2 disk1 disk2 disk3 disk4

but not with just three disks.

Memory allocated from this path:

 <zfs`abd_alloc_struct_impl (abd_os.c:211)> abd_alloc_struct_impl: 0xffffff88928d0000 
  alloc 32864: 0xffffff7f94e8db6b 0xffffff7f94e90209 0xffffff7f94e8e28e 0xffffff7f94f56e3c  
  0xffffff7f94f572a7 0xffffff7f94fc46e6 0xffffff7f94fbfcec 0xffffff7f9513e7d4 0xffffff801219513e

stack:
  zfs`abd_alloc_struct + 11 at abd.c:164:15 
  zfs`abd_get_offset_scatter + 121 at abd_os.c:347:9 
  zfs`abd_get_offset_impl + 62 at abd.c:534:9
  zfs`vdev_queue_io_to_issue + 2796 [inlined] vdev_queue_aggregate + 1057 at vdev_queue.c:896  
  zfs`vdev_queue_io_done + 343 at vdev_queue.c:994:16 
  zfs`zio_vdev_io_done + 198 at zio.c:3854:20

will eventually trigger:

(zfs) <zfs`kmem_error (spl-kmem.c:0)> SPL: kernel memory allocator:
 (zfs) <zfs`kmem_error (spl-kmem.c:925)> buffer freed to wrong cache
 (zfs) <zfs`kmem_error (spl-kmem.c:927)> SPL: buffer was allocated from kmem_alloc_40960,
(zfs) <zfs`kmem_error (spl-kmem.c:0)> SPL: caller attempting free to kmem_alloc_128.
(zfs) <zfs`kmem_error (spl-kmem.c:945)> SPL: buffer=0xffffff88928d0000  bufctl=0xffffff8892966d70  cache: kmem_alloc_128

stack:

    frame #4: 0xffffff8012a6a0cc kernel.development`panic(str=<unavailable>) at debug.c:746:2 [opt]
    frame #5: 0xffffff7f9514858c zfs`kmem_error.cold.1 + 28
    frame #6: 0xffffff887672be30
    frame #7: 0xffffff7f95130e84 zfs`kmem_error(error=<unavailable>, cparg=<unavailable>, bufarg=<unavailable>) at spl-kmem.c:963:3 [opt]
    frame #8: 0xffffff88928d0078
  * frame #9: 0xffffff7f95136f3b zfs`kmem_magazine_destroy(cp=0xffffff88928d0000, mp=0xffffff88821453c0, nrounds=-2147453744) at spl-kmem.c:1666:3 [opt]
    frame #10: 0xffffff88821453c0
    frame #11: 0xffffff7f95132931 zfs`kmem_cache_magazine_purge(cp=0xffffff8880007440) at spl-kmem.c:2875:4 [opt]
    frame #12: 0xffffff887672beb0
    frame #13: 0xffffff7f95137d2e zfs`kmem_cache_magazine_resize(cp=0xffffff8892966d70) at spl-kmem.c:2968:3 [opt]
    frame #14: 0x0000000a7cd9f891
    frame #15: 0xffffff7f9513e7d4 zfs`taskq_thread(arg=0xffffff888000fc40) at spl-taskq.c:1964:3 [opt]
    frame #16: 0xffffff801c1765b0
    frame #17: 0xffffff801219513e kernel.development`call_continuation + 46

stack looks suspect.

realpath returns incorrect results for hardlinked files

It works okay on APFS and HFS, so I guess it's an ZFS specific issue.
Steps to reproduce:

#/usr/bin/env zsh

clang realpath.c -o realpath

mkdir -p broken

touch file
ln file broken/copy

# for some reason without the mv everything works ok
mv broken/copy broken/link

./realpath broken/link
# /Volumes/media/build/broken/link

# might need to adjust this:
sleep 2s

./realpath broken/link
# /Volumes/media/build/broken/file

/usr/bin/stat -f "%N: %HT%SY" broken/link $(./realpath broken/link)
# broken/link: Regular File
# stat: /Volumes/media/build/broken/file: stat: No such file or directory

// realpath.c
#include <stdio.h>
#include <stdlib.h>
#include <sys/attr.h>

int main(int argc, char **argv) {
    char path[PATH_MAX];

    for (int i = 1; i < argc; i++) {
        if (realpath(argv[i], path)) {
            printf("%s\n", path);
        } else {
            printf("bad path: %s\n", argv[i]);
        }
    }

    return 0;
}

Now, I don't care about this realpath.c as I can use the one from coreutils which works correctly, but rust / cargo uses libc with the broken realpath and as the result can't build some crates due to incorrect paths to existing files.

Looking at the implementation of realpath, I see it saves the path to directory of the file and then uses getattrlist to replace filename, except getattrlist returns the filename of the original hardlinked file, i.e. /Volumes/media/build/file:

...
		/*
		 * Save resolved_len, so that we can later null out
		 * the the appended next_token, and replace with the
		 * real name (matters on case-insensitive filesystems).
		 */
		save_resolved_len = resolved_len;
...
		if (getattrlist(resolved, (void *)&_rp_alist, &attrs, sizeof(attrs), FSOPT_NOFOLLOW) == 0) {
...
			/*
			 * attrs already has the real name.
			 */
			// resolved: ./broken/link
			resolved[save_resolved_len] = '\0';
                         // resolved: ./broken/
			resolved_len = strlcat(resolved, (const char *)&attrs.name + attrs.name.attr_dataoffset, PATH_MAX);
                        // resolved: ./broken/file

Other info that might be relevant:

zfs-macOS-2.0.1-1
zfs-kmod-2.0.1-1

macOS 10.15.7 (19H1030)
Darwin mars.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Apr 12 20:57:45 PDT 2021; root:xnu-6153.141.28.1~1/RELEASE_X86_64 x86_64 i386 Macmini6,2 Darwin

Apple clang version 12.0.0 (clang-1200.0.32.29)

Regression: man pages not in folder recognised by terminal

System information

Type	Version/Name
Distribution Name	Big Sur
Distribution Version	11.4
Linux Kernel
Architecture
ZFS Version	2.1 rc1
SPL Version	?

Describe the problem you're observing

For all other commands not from zfs, the man pages are in a place where terminal.app finds them when entering rthe name (eg enter grep into the help search box.

Describe how to reproduce the problem

Enter zfs or zpool in the help menu's search box

Include any warning/errors/backtraces from the system logs

zfs icon doesn't display on mounted volumes

In Finder there's just a blank space where the zfs volume icon should be. This at least affects Big Sur. I'm not sure if it affects earlier versions of macOS.

panic: casenorm/mixed_create_failure

This is a relatively new bug, since rebase with upstream around Aug. Unsure why it only affects macOS, when all the interaction is in the shared source files. (not macOS specific files).

It appears to add a zap, then grow the zap forever, until memory requirements break.

                 *1000  VNOP_CREATE + 95 (kernel.development + 5105071) [0xffffff80006de5af]
                   *1000  zfs_vnop_create + 222 (zfs_vnops_osx.c:1363,10 in zfs + 1681006) [0xffffff7f831ca66e]
                     *1000  zfs_create + 1744 (zfs_vnops.c:1437,11 in zfs + 1630432) [0xffffff7f831be0e0]
                       *1000  zfs_link_create + 541 (zfs_dir.c:815,10 in zfs + 1436829) [0xffffff7f8318ec9d]
                         *1000  zap_add + 155 (zap_micro.c:1274,8 in zfs + 1319387) [0xffffff7f831721db]
                           *1000  zap_add_impl + 159 (zap_micro.c:1238,9 in zfs + 1319567) [0xffffff7f8317228f]
                             *1000  fzap_add + 112 (zap.c:885,10 in zfs + 1284912) [0xffffff7f83169b30]
                               *1000  fzap_add_cd + 282 (zap.c:856,9 in zfs + 1283178) [0xffffff7f8316946a]
                                 *1000  zap_expand_leaf + 345 (zap.c:659,10 in zfs + 1283785) [0xffffff7f831696c9]
                                   *1000  zap_grow_ptrtbl + 439 (zap.c:391,11 in zfs + 1292455) [0xffffff7f8316b8a7]
                                     *565  zap_table_grow + 588 (zap.c:212,2 in zfs + 1293948) [0xffffff7f8316be7c]
                                       *541  dmu_buf_hold + 155 (dmu.c:243,9 in zfs + 222939) [0xffffff7f830666db]
                                         *541  dbuf_read + 977 (dbuf.c:1658,9 in zfs + 162177) [0xffffff7f83057981]

We essentially never leave fzap_add_cd()

The loop appears to be:

fzap_add_cd(zap_name_t *zn,
    uint64_t integer_size, uint64_t num_integers,
    const void *val, uint32_t cd, void *tag, dmu_tx_t *tx)
{
/* ... */
retry:
/* ... */
    err = zap_entry_create(l, zn, cd,
        integer_size, num_integers, val, &zeh);
/* ... */
    } else if (err == EAGAIN) {
        printf("EAGAIN\n");
        err = zap_expand_leaf(zn, l, tag, tx, &l);
        if (err == 0) {
            printf("err %d - retry\n", err);
            goto retry;

storing 815 at index 132ea3
returning 0 due to end
err 0 - retry
zap_leaf_phys(l)->l_hdr.lh_nfree 2 < numchunks 3
EAGAIN
finished; numblocks now 2048 (4096k entries)

and, the reason zap_entry_create() returns EAGAIN is:

    if (zap_leaf_phys(l)->l_hdr.lh_nfree < numchunks) {
	printf("zap_leaf_phys(l)->l_hdr.lh_nfree %d < numchunks %d\n",
            zap_leaf_phys(l)->l_hdr.lh_nfree, numchunks);
        return (SET_ERROR(EAGAIN));
    }
zap_leaf_phys(l)->l_hdr.lh_nfree 2 < numchunks 3 
zap_leaf_phys(l)->l_hdr.lh_nfree 2 < numchunks 3
zap_leaf_phys(l)->l_hdr.lh_nfree 2 < numchunks 3
zap_leaf_phys(l)->l_hdr.lh_nfree 2 < numchunks 3
These numbers don't appear to change, even as the leaf grows

and the reason zap_expand_leaf() returns 0, is from the end of the function. (Not the one earlier return 0).

recursion of visitbp can blow the stack

0xffffff80081566d0 : 0xffffff80082332af mach_kernel : _hndl_double_fault + 0xf
0xffffffc117028000 : 0xffffff7fa51bb639 org.openzfsonosx.zfs : _spl_mutex_tryenter + 0x19
0xffffffc117028020 : 0xffffff7fa51acbd3 org.openzfsonosx.zfs : _kmem_depot_alloc + 0x23
0xffffffc117028050 : 0xffffff7fa51ac533 org.openzfsonosx.zfs : _kmem_cache_alloc + 0x323
0xffffffc1170280e0 : 0xffffff7fa51c5aff org.openzfsonosx.zfs : _vmem_alloc + 0x6f
0xffffffc117028170 : 0xffffff7fa51c48b1 org.openzfsonosx.zfs : _vmem_xalloc + 0xa21
0xffffffc117028320 : 0xffffff7fa51c5d35 org.openzfsonosx.zfs : _vmem_alloc + 0x2a5
0xffffffc1170283b0 : 0xffffff7fa51b426f org.openzfsonosx.zfs : _kmem_slab_create + 0xaf
0xffffffc117028440 : 0xffffff7fa51acd60 org.openzfsonosx.zfs : _kmem_slab_alloc + 0x70
0xffffffc117028480 : 0xffffff7fa51ac5a3 org.openzfsonosx.zfs : _kmem_cache_alloc + 0x393
0xffffffc117028510 : 0xffffff7fa4ecb83b org.openzfsonosx.zfs : _sio_alloc + 0x2b
0xffffffc117028530 : 0xffffff7fa4ecb369 org.openzfsonosx.zfs : _scan_io_queue_insert + 0x1c9
0xffffffc117028590 : 0xffffff7fa4ecacde org.openzfsonosx.zfs : _dsl_scan_enqueue + 0x2de
0xffffffc117028610 : 0xffffff7fa4ec9e1d org.openzfsonosx.zfs : _dsl_scan_scrub_cb + 0x41d
0xffffffc1170286b0 : 0xffffff7fa4ecf50d org.openzfsonosx.zfs : _dsl_scan_visitbp + 0x163d
0xffffffc117028b40 : 0xffffff7fa4ece374 org.openzfsonosx.zfs : _dsl_scan_visitbp + 0x4a4
0xffffffc117028fd0 : 0xffffff7fa4ece374 org.openzfsonosx.zfs : _dsl_scan_visitbp + 0x4a4
0xffffffc117029460 : 0xffffff7fa4ece7fe org.openzfsonosx.zfs : _dsl_scan_visitbp + 0x92e
0xffffffc1170298f0 : 0xffffff7fa4ece374 org.openzfsonosx.zfs : _dsl_scan_visitbp + 0x4a4
0xffffffc117029d80 : 0xffffff7fa4ece374 org.openzfsonosx.zfs : _dsl_scan_visitbp + 0x4a4
0xffffffc11702a210 : 0xffffff7fa4ece374 org.openzfsonosx.zfs : _dsl_scan_visitbp + 0x4a4
0xffffffc11702a6a0 : 0xffffff7fa4ece374 org.openzfsonosx.zfs : _dsl_scan_visitbp + 0x4a4
0xffffffc11702ab30 : 0xffffff7fa4ece374 org.openzfsonosx.zfs : _dsl_scan_visitbp + 0x4a4
0xffffffc11702afc0 : 0xffffff7fa4ece374 org.openzfsonosx.zfs : _dsl_scan_visitbp + 0x4a4
0xffffffc11702b450 : 0xffffff7fa4eceb57 org.openzfsonosx.zfs : _dsl_scan_visitbp + 0xc87
0xffffffc11702b8e0 : 0xffffff7fa4ecd146 org.openzfsonosx.zfs : _dsl_scan_visit_rootbp + 0x196
0xffffffc11702b950 : 0xffffff7fa4ecd5a6 org.openzfsonosx.zfs : _dsl_scan_visitds + 0x2c6
0xffffffc11702bbd0 : 0xffffff7fa4ec8936 org.openzfsonosx.zfs : _dsl_scan_visit + 0x1f6
0xffffffc11702bc70 : 0xffffff7fa4ec6e81 org.openzfsonosx.zfs : _dsl_scan_sync + 0x7d1
0xffffffc11702bdb0 : 0xffffff7fa4f0f95f org.openzfsonosx.zfs : _spa_sync_iterate_to_convergence + 0x15f
0xffffffc11702be00 : 0xffffff7fa4f0f211 org.openzfsonosx.zfs : _spa_sync + 0x531
0xffffffc11702bef0 : 0xffffff7fa4f2e4b5 org.openzfsonosx.zfs : _txg_sync_thread + 0x3d5

kern.stack_depth_max: 22368

zil_commit_waiter can stall forever

System information

Type	Version/Name
Distribution Name	macOS
Distribution Version
Linux Kernel
Architecture
ZFS Version
SPL Version

git HEAD

Describe the problem you're observing

a call to fsync can stall forever in zil_commit_waiter.

Describe how to reproduce the problem

Easily repeatable, first test encountered with zfs-tests.

Include any warning/errors/backtraces from the system logs

The stalled command:

 9:04AM sh               sh -x ./scripts/zfs-tests.sh -r tiny
 9:05AM zfs create testpool/testfs

The main stack:
Duration: 9.98s
Steps: 999 (10ms sampling interval)

zfs stack

  Thread 0xea80             DispatchQueue 140735589485544                       999 samples (1-999)       priority 29-31 (base 31)  cpu time 3.113s
  999  start + 1 (libdyld.dylib + 91093) [0x7fff587253d5]
    999  main + 760 (zfs_main.c:8549,9 in zfs + 10984) [0x103731ae8]
      999  zfs_do_create + 3209 (zfs_main.c:1135,8 in zfs + 18201) [0x103733719]
        999  zfs_mount_and_share + 251 (zfs_main.c:766,14 in zfs + 56107) [0x10373cb2b]
          999  zfs_mount + 141 (libzfs_mount.c:375,10 in libzfs.2.dylib + 116349) [0x1037bc67d]
            999  zfs_mount_at + 1391 (libzfs_mount.c:503,7 in libzfs.2.dylib + 117807) [0x1037bcc2f]
              999  mount + 10 (libsystem_kernel.dylib + 38470) [0x7fff58862646]
               *999  hndl_unix_scall64 + 22 (kernel.development + 180998) [0xffffff800022c306]
                 *999  unix_syscall64 + 757 (kernel.development + 7721589) [0xffffff800095d275]
                   *999  mount + 54 (kernel.development + 3440038) [0xffffff8000547da6]
                     *999  __mac_mount + 1004 (kernel.development + 3441052) [0xffffff800054819c]
                       *999  mount_common + 1303 (kernel.development + 3433207) [0xffffff80005462f7]
                         *999  prepare_coveredvp + 117 (kernel.development + 3438229) [0xffffff8000547695]
                           *999  zfs_vnop_fsync + 168 (zfs_vnops_osx.c:1643,8 in zfs + 1637736) [0xffffff7f82bb8d68]
                             *999  zfs_fsync + 380 (zfs_vnops.c:2485,3 in zfs + 1596204) [0xffffff7f82baeb2c]
                               *999  zil_commit + 156 (zil.c:2946,2 in zfs + 1738620) [0xffffff7f82bd177c]
                                 *999  zil_commit_impl + 101 (zil.c:2984,2 in zfs + 1738741) [0xffffff7f82bd17f5]
                                   *998  zil_commit_waiter + 246 (zil.c:2690,23 in zfs + 1739622) [0xffffff7f82bd1b66]
                                     *983  cv_timedwait_hires + 379 (spl-condvar.c:223,14 in zfs + 2467019) [0xffffff7f82c834cb]
                                       *982  msleep + 98 (kernel.development + 6504274) [0xffffff8000833f52]
                                         *980  _sleep + 234 (kernel.development + 6502554) [0xffffff800083389a]
                                           *910  lck_mtx_sleep_deadline + 155 (kernel.development + 666779) [0xffffff80002a2c9b]
                                             *842  thread_block_reason + 332 (kernel.development + 726876) [0xffffff80002b175c]
                                               *842  thread_invoke + 1668 (kernel.development + 734164) [0xffffff80002b33d4]
                                                 *741  machine_switch_context + 245 (kernel.development + 2154805) [0xffffff800040e135] (runnable)
                                                 *101  machine_switch_context + 245 (kernel.development + 2154805) [0xffffff800040e135]
                                             *68   thread_block_reason + 345 (kernel.development + 726889) [0xffffff80002b1769]
                                               *67   ml_set_interrupts_enabled + 48 (kernel.development + 2145584) [0xffffff800040bd30] (running)
                                               *1    return_to_iret + 276 (kernel.development + 179716) [0xffffff800022be04]
                                                 *1    ast_taken_kernel + 113 (kernel.development + 548657) [0xffffff8000285f31]
                                                   *1    thread_block_reason + 332 (kernel.development + 726876) [0xffffff80002b175c]
                                                     *1    thread_invoke + 1668 (kernel.development + 734164) [0xffffff80002b33d4]
                                                       *1    machine_switch_context + 245 (kernel.development + 2154805) [0xffffff800040e135] (runnable)
                                           *69   lck_mtx_sleep_deadline + 105 (kernel.development + 666729) [0xffffff80002a2c69]
                                             *68   assert_wait_deadline + 152 (kernel.development + 725112) [0xffffff80002b1078]
                                               *61   ml_set_interrupts_enabled + 48 (kernel.development + 2145584) [0xffffff800040bd30] (running)
                                               *7    return_to_iret + 276 (kernel.development + 179716) [0xffffff800022be04]
                                                 *7    ast_taken_kernel + 113 (kernel.development + 548657) [0xffffff8000285f31]
                                                   *7    thread_block_reason + 332 (kernel.development + 726876) [0xffffff80002b175c]
                                                     *7    thread_invoke + 1668 (kernel.development + 734164) [0xffffff80002b33d4]
                                                       *7    machine_switch_context + 245 (kernel.development + 2154805) [0xffffff800040e135] (runnable)
                                             *1    _global_eventq + 1 (kernel.development + 960657) [0xffffff80002ea891] (running)
                                           *1    lck_mtx_sleep_deadline + 176 (kernel.development + 666800) [0xffffff80002a2cb0]
                                             *1    lck_mtx_lock + 129 (kernel.development + 2141521) [0xffffff800040ad51] (running)
                                         *1    _sleep + 695 (kernel.development + 6503015) [0xffffff8000833a67] (running)
                                         *1    _sleep + 98 (kernel.development + 6502418) [0xffffff8000833812] (running)
                                       *1    msleep + 34 (kernel.development + 6504210) [0xffffff8000833f12] (running)
                                     *4    cv_timedwait_hires + 411 (spl-condvar.c:227,31 in zfs + 2467051) [0xffffff7f82c834eb]
                                       *4    gethrestime_sec + 17 (spl-time.c:108,5 in zfs + 2564369) [0xffffff7f82c9b111]
                                         *4    microtime + 25 (kernel.development + 6585785) [0xffffff8000847db9]
                                           *4    ml_set_interrupts_enabled + 48 (kernel.development + 2145584) [0xffffff800040bd30] (running)
                                     *3    spl_wdlist_settime + 1 (spl-mutex.c:73 in zfs + 2536849) [0xffffff7f82c94591] (running)
                                     *3    cv_timedwait_hires + 120 (spl-condvar.c:184,21 in zfs + 2466760) [0xffffff7f82c833c8]
                                       *1    _rtc_nanotime_read + 16 (kernel.development + 176208) [0xffffff800022b050] (running)
                                       *1    _rtc_nanotime_read + 12 (kernel.development + 176204) [0xffffff800022b04c] (running)
                                       *1    _rtc_nanotime_read + 7 (kernel.development + 176199) [0xffffff800022b047] (running)
                                     *2    current_thread + 1 (kernel.development + 2148737) [0xffffff800040c981] (running)
                                     *1    cv_timedwait_hires + 391 (spl-condvar.c:225,17 in zfs + 2467031) [0xffffff7f82c834d7] (running)
                                     *1    cv_timedwait_hires + 382 (spl-condvar.c:225,19 in zfs + 2467022) [0xffffff7f82c834ce] (running)
                                     *1    cv_timedwait_hires + 139 (spl-condvar.c:189,7 in zfs + 2466779) [0xffffff7f82c833db] (running)
                                   *1    zil_commit_waiter + 250 (zil.c:2694,17 in zfs + 1739626) [0xffffff7f82bd1b6a] (running)

There does not appear to be any other threads in play, generally they are all idle. For example;

other stacks

  Thread 0xea40             999 samples (1-999)       priority 81 (base 81)
 *999  call_continuation + 46 (kernel.development + 176334) [0xffffff800022b0ce]
   *999  taskq_thread + 711 (spl-taskq.c:1768,11 in zfs + 2561511) [0xffffff7f82c9a5e7]
     *999  taskq_thread_wait + 99 (spl-taskq.c:1677,3 in zfs + 2562403) [0xffffff7f82c9a963]
       *999  spl_cv_wait + 130 (spl-condvar.c:75,14 in zfs + 2466034) [0xffffff7f82c830f2]
         *999  msleep + 98 (kernel.development + 6504274) [0xffffff8000833f52]
           *999  _sleep + 474 (kernel.development + 6502794) [0xffffff800083398a]
             *999  lck_mtx_sleep + 136 (kernel.development + 666168) [0xffffff80002a2a38]
               *999  thread_block_reason + 332 (kernel.development + 726876) [0xffffff80002b175c]
                 *999  thread_invoke + 1668 (kernel.development + 734164) [0xffffff80002b33d4]
                   *999  machine_switch_context + 245 (kernel.development + 2154805) [0xffffff800040e135]



  Thread 0xea41             999 samples (1-999)       priority 81 (base 81)     cpu time <0.001s
 *999  call_continuation + 46 (kernel.development + 176334) [0xffffff800022b0ce]
   *999  txg_quiesce_thread + 167 (txg.c:618,4 in zfs + 1030583) [0xffffff7f82b249b7]
     *999  txg_thread_wait + 132 (txg.c:254,3 in zfs + 1036404) [0xffffff7f82b26074]
       *999  spl_cv_wait + 130 (spl-condvar.c:75,14 in zfs + 2466034) [0xffffff7f82c830f2]
         *999  msleep + 98 (kernel.development + 6504274) [0xffffff8000833f52]
           *999  _sleep + 474 (kernel.development + 6502794) [0xffffff800083398a]
             *999  lck_mtx_sleep + 136 (kernel.development + 666168) [0xffffff80002a2a38]
               *999  thread_block_reason + 332 (kernel.development + 726876) [0xffffff80002b175c]
                 *999  thread_invoke + 1668 (kernel.development + 734164) [0xffffff80002b33d4]
                   *999  machine_switch_context + 245 (kernel.development + 2154805) [0xffffff800040e135]

  Thread 0xea42             999 samples (1-999)       priority 81 (base 81)     cpu time <0.001s
 *999  call_continuation + 46 (kernel.development + 176334) [0xffffff800022b0ce]
   *998  txg_sync_thread + 373 (txg.c:542,4 in zfs + 1031157) [0xffffff7f82b24bf5]
     *998  txg_thread_wait + 92 (txg.c:251,10 in zfs + 1036364) [0xffffff7f82b2604c]
       *998  spl_cv_timedwait + 293 (spl-condvar.c:135,14 in zfs + 2466405) [0xffffff7f82c83265]
         *998  msleep + 98 (kernel.development + 6504274) [0xffffff8000833f52]
           *998  _sleep + 234 (kernel.development + 6502554) [0xffffff800083389a]
             *998  lck_mtx_sleep_deadline + 155 (kernel.development + 666779) [0xffffff80002a2c9b]
               *998  thread_block_reason + 332 (kernel.development + 726876) [0xffffff80002b175c]
                 *998  thread_invoke + 1668 (kernel.development + 734164) [0xffffff80002b33d4]
                   *998  machine_switch_context + 245 (kernel.development + 2154805) [0xffffff800040e135]
   *1    txg_sync_thread + 760 (txg.c:579,3 in zfs + 1031544) [0xffffff7f82b24d78]
     *1    spa_sync + 1322 (spa.c:9023,2 in zfs + 918362) [0xffffff7f82b0935a]
       *1    spa_sync_iterate_to_convergence + 338 (spa.c:8826,3 in zfs + 920194) [0xffffff7f82b09a82]
         *1    ddt_sync + 211 (ddt.c:1154,9 in zfs + 207219) [0xffffff7f82a5b973]
           *1    zio_wait + 598 (zio.c:2203,11 in zfs + 1775062) [0xffffff7f82bda5d6]
             *1    spl_cv_timedwait + 293 (spl-condvar.c:135,14 in zfs + 2466405) [0xffffff7f82c83265]
               *1    msleep + 98 (kernel.development + 6504274) [0xffffff8000833f52]
                 *1    _sleep + 234 (kernel.development + 6502554) [0xffffff800083389a]
                   *1    lck_mtx_sleep_deadline + 155 (kernel.development + 666779) [0xffffff80002a2c9b]
                     *1    thread_block_reason + 332 (kernel.development + 726876) [0xffffff80002b175c]
                       *1    thread_invoke + 1668 (kernel.development + 734164) [0xffffff80002b33d4]
                         *1    machine_switch_context + 245 (kernel.development + 2154805) [0xffffff800040e135]

  Thread 0xea43             999 samples (1-999)       priority 81 (base 81)     cpu time <0.001s
 *999  call_continuation + 46 (kernel.development + 176334) [0xffffff800022b0ce]
   *999  mmp_thread + 1904 (mmp.c:674,10 in zfs + 800992) [0xffffff7f82aec8e0]
     *999  cv_timedwait_hires + 379 (spl-condvar.c:223,14 in zfs + 2467019) [0xffffff7f82c834cb]
       *999  msleep + 98 (kernel.development + 6504274) [0xffffff8000833f52]
         *999  _sleep + 234 (kernel.development + 6502554) [0xffffff800083389a]
           *999  lck_mtx_sleep_deadline + 155 (kernel.development + 666779) [0xffffff80002a2c9b]
             *999  thread_block_reason + 332 (kernel.development + 726876) [0xffffff80002b175c]
               *999  thread_invoke + 1668 (kernel.development + 734164) [0xffffff80002b33d4]
                 *999  machine_switch_context + 245 (kernel.development + 2154805) [0xffffff800040e135]

  Thread 0xec17             67 samples (933-999)      priority 81 (base 81)     cpu time 0.003s
 *67  call_continuation + 46 (kernel.development + 176334) [0xffffff800022b0ce]
   *67  tx_flush_thread + 121 (apfs + 797086) [0xffffff7f818bb99e]
     *67  msleep + 98 (kernel.development + 6504274) [0xffffff8000833f52]
       *67  _sleep + 234 (kernel.development + 6502554) [0xffffff800083389a]
         *67  lck_mtx_sleep_deadline + 155 (kernel.development + 666779) [0xffffff80002a2c9b]
           *67  thread_block_reason + 332 (kernel.development + 726876) [0xffffff80002b175c]
             *67  thread_invoke + 1668 (kernel.development + 734164) [0xffffff80002b33d4]
               *67  machine_switch_context + 245 (kernel.development + 2154805) [0xffffff800040e135]

The code that is stuck is here:

https://github.com/openzfsonosx/openzfs/blob/master/module/zfs/zil.c#L2690


		if (lwb != NULL && lwb->lwb_state == LWB_STATE_OPENED) {
			ASSERT3B(timedout, ==, B_FALSE);

			/*
			 * If the lwb hasn't been issued yet, then we
			 * need to wait with a timeout, in case this
			 * function needs to issue the lwb after the
			 * timeout is reached; responsibility (2) from
			 * the comment above this function.
===>		clock_t timeleft = cv_timedwait_hires(&zcw->zcw_cv,
			    &zcw->zcw_lock, wakeup, USEC2NSEC(1),
			    CALLOUT_FLAG_ABSOLUTE);

			if (timeleft >= 0 || zcw->zcw_done)
				continue;

			timedout = B_TRUE;
			zil_commit_waiter_timeout(zilog, zcw);

and it is spinning rather quickly, easily taking out a "core" on its own. But increasing the timeout does not fix the problem, just lowers the loadavg. It is simply not signalled, or if it is, it is missed.

10.15.7 Catalina failed to boot from ZVOL - OpenZFS 2.0

System information

Type	Version/Name
Distribution Name	macOS
Distribution Version	10.15.7
macOS Kernel	19.6.0
Architecture	amd64
ZFS Version	zfs-2.0.0-29_ge98d6f91d
kmod Version	zfs-kmod-macOS-2.0.0-1-ge98d6f91d

Describe the problem you're observing

10.15.7 Catalina can not boot from APFS container on ZVOL with OpenZFS 2.0 while it can boot normally with 1.9.4

Describe how to reproduce the problem

I have macOS Catalina installed on APFS container on ZVOL. The system will boot via a booter partition (HFS+). It runs well with OpenZFS 1.9.4 since 10.15.5 and earlier version. I have just updated it to 10.15.7.
OpenZFS 1.9.4 has been installed manually by extract and copy all the file/kext/scripts to their correct location and permission/owner.
I delete old 1.9.4 version and install 2.0 release (2nd release on forum) manually. Of course the prelinkedkernel has been updated on both the APFS partition and the external HFS+ booter.
I can confirm the macOS installer (10.15.7) can load the kext and import ZFS pool normally (I manually create an install partition, add the kext/file/scripts and update prelinkedkernel, this is also how I can install macOS on APFS container on ZVOL)

Include any warning/errors/backtraces from the system logs

It was at boot time so I could not capture the log, please check the photo instead. Sorry.

zdb does not translate xnu inode values to zfs

This causes zdb -dddddd $ds $objnum to fail for the "large" xnu inode values (16 reserved).

https://github.com/openzfsonosx/openzfs/blob/macOS_pure/tests/zfs-tests/include/blkdev.shlib#L609

@lundman suggests trying

if (rc == 0) {
    zor->zor_obj_start = INO_XNUTOZFS(zor->zor_obj_start, 2);
    zor->zor_obj_end = INO_XNUTOZFS(zor->zor_obj_end, 2);

(Issue discovered while fixing [tests/functional/checksum] test failures)

Recursive zfs send/receive isn't working to/from files

joe@joes-Mac ~ % zpool version
zfs-2.0.0-rc6
zfs-kmod-macos-2.0.0-rc6-14-g63b3fb4a6-dirty

Trying to run the zfs-tests/tests/functional/rsend tests doesn't get far because the initial setup is failing.

ASSERTION: zfs send -I sends all incrementals from fs@init to fs@final.
SUCCESS: eval zfs send -R testpool@final > /var/tmp/backdir-rsend/pool-R
ERROR: eval zfs receive -d -F testpool2 < /var/tmp/backdir-rsend/pool-R exited 1
cannot receive incremental stream: destination 'testpool2/testfs/vclone' does not exist

Both send to file and receive from file are failing:

joe@joes-Mac openzfs % export __ZFS_MAIN_MOUNTPOINT_DIR=/
joe@joes-Mac openzfs % sudo zpool destroy testpool2 && sudo zpool create testpool2 disk1
Volume testpool2 on disk9s1 unmounted
joe@joes-Mac openzfs % rm outf                               
joe@joes-Mac openzfs % sudo zfs send -R testpool@final > outf
joe@joes-Mac openzfs % cat outf | sudo zfs receive -d -F testpool2
cannot receive incremental stream: destination 'testpool2/testfs/vclone' does not exist
joe@joes-Mac openzfs % rm outf                                    
joe@joes-Mac openzfs % sudo zfs send -R testpool@final | tee outf &>/dev/null
joe@joes-Mac openzfs % cat outf | sudo zfs receive -d -F testpool2           
Volume testpool2 on disk9s1 unmounted
joe@joes-Mac openzfs % sudo zpool destroy testpool2 && sudo zpool create testpool2 disk1
Unmount successful for /Volumes/testpool2/testfs/fs1/fs2
Unmount successful for /Volumes/testpool2/testfs/fs1/fclone
Unmount successful for /Volumes/testpool2/testfs/fs1
Unmount successful for /Volumes/testpool2/testfs
Unmount successful for /Volumes/testpool2/pclone
Volume testpool2 on disk9s1 unmounted
Exporting 'testpool2/testfs/vclone'
... asking ZVOL to export 'disk16'
Unmount of all volumes on disk16 was successful
Exporting 'testpool2/testfs/vol'
... asking ZVOL to export 'disk14'
Unmount of all volumes on disk14 was successful
Exporting 'testpool2/vol'
... asking ZVOL to export 'disk15'
Unmount of all volumes on disk15 was successful
joe@joes-Mac openzfs % sudo zfs receive -d -F testpool2 < outf
Volume testpool2 on disk9s1 unmounted
cannot receive: invalid stream (bad magic number)
joe@joes-Mac openzfs % sudo zpool destroy testpool2 && sudo zpool create testpool2 disk1
Volume testpool2 on disk9s1 unmounted
joe@joes-Mac openzfs % cat outf | sudo zfs receive -d -F testpool2 
Volume testpool2 on disk9s1 unmounted
joe@joes-Mac openzfs %

Note that using tee and cat the issue can be avoided as shown above.

Panic while working with long paths

Don't know if it's related to openzfsonosx/zfs#300 or #16, but it doesn't have multibyte filenames or filenames longer than 255.

Panic logs

panic(cpu 0 caller 0xffffff800c003dbd): "__memmove_chk object size check failed: dst 0xffffff92910bb6f0, src 0xffffff802f408000, (256 < 261)"@/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-6153.141.28.1/osfmk/device/subrs.c:701
Backtrace (CPU 0), Frame : Return Address
0xffffff92910bb3a0 : 0xffffff800bf1763d mach_kernel : _handle_debugger_trap + 0x49d
0xffffff92910bb3f0 : 0xffffff800c051b25 mach_kernel : _kdp_i386_trap + 0x155
0xffffff92910bb430 : 0xffffff800c0436ae mach_kernel : _kernel_trap + 0x4ee
0xffffff92910bb480 : 0xffffff800bebda40 mach_kernel : _return_from_trap + 0xe0
0xffffff92910bb4a0 : 0xffffff800bf16d07 mach_kernel : _DebuggerTrapWithState + 0x17
0xffffff92910bb5a0 : 0xffffff800bf170f7 mach_kernel : _panic_trap_to_debugger + 0x227
0xffffff92910bb5f0 : 0xffffff800c6be82c mach_kernel : _panic + 0x54
0xffffff92910bb660 : 0xffffff800c003dbd mach_kernel : ___memmove_chk + 0x3d
0xffffff92910bb680 : 0xffffff7f8ccc9c2e org.openzfsonosx.zfs : _zfs_vnop_lookup + 0x6e
0xffffff92910bb820 : 0xffffff800c16bd0e mach_kernel : _lookup + 0x3ee
0xffffff92910bb980 : 0xffffff800c16aff1 mach_kernel : _namei + 0xdc1
0xffffff92910bbb80 : 0xffffff800c1848e5 mach_kernel : _nameiat + 0x75
0xffffff92910bbbd0 : 0xffffff800c18ab87 mach_kernel : _stat_extended + 0x187
0xffffff92910bbf10 : 0xffffff800c18b5af mach_kernel : _lstat64 + 0x2f
0xffffff92910bbf40 : 0xffffff800c5819a7 mach_kernel : _unix_syscall64 + 0x287
0xffffff92910bbfa0 : 0xffffff800bebe206 mach_kernel : _hndl_unix_scall64 + 0x16
      Kernel Extensions in backtrace:
         org.openzfsonosx.zfs(2.0.1)[C8900DAA-9C3B-36DD-8C6D-7869FC2DBF69]@0xffffff7f8cbb0000-0xffffff7f8e361fff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[F3D120E5-29D0-32C7-A09A-E88B84675F9C]@0xffffff7f8cb10000

Another one, don't know why this one is different, I might've hit an edge case for character length.

panic(cpu 0 caller 0xffffff80234be782): "Kernel stack memory corruption detected"@/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-6153.141.28.1/libkern/stack_protector.c:37
Backtrace (CPU 0), Frame : Return Address
0xffffff92257033a0 : 0xffffff8022d1763d mach_kernel : _handle_debugger_trap + 0x49d
0xffffff92257033f0 : 0xffffff8022e51b25 mach_kernel : _kdp_i386_trap + 0x155
0xffffff9225703430 : 0xffffff8022e436ae mach_kernel : _kernel_trap + 0x4ee
0xffffff9225703480 : 0xffffff8022cbda40 mach_kernel : _return_from_trap + 0xe0
0xffffff92257034a0 : 0xffffff8022d16d07 mach_kernel : _DebuggerTrapWithState + 0x17
0xffffff92257035a0 : 0xffffff8022d170f7 mach_kernel : _panic_trap_to_debugger + 0x227
0xffffff92257035f0 : 0xffffff80234be82c mach_kernel : _panic + 0x54
0xffffff9225703660 : 0xffffff80234be782 mach_kernel : _mac_exc_action_check_exception_send + 0x3a2
0xffffff9225703670 : 0xffffff8022cda8e9 mach_kernel : ___stack_chk_fail + 0x9
0xffffff9225703680 : 0xffffff7fa3ac9f95 org.openzfsonosx.zfs : _zfs_vnop_lookup + 0x3d5
0xffffff9225703820 : 0xffffff8022f6bd0e mach_kernel : _lookup + 0x3ee
0xffffff9225703980 : 0xffffff8022f6aff1 mach_kernel : _namei + 0xdc1
0xffffff9225703b80 : 0xffffff8022f848e5 mach_kernel : _nameiat + 0x75
0xffffff9225703bd0 : 0xffffff8022f8ab87 mach_kernel : _stat_extended + 0x187
0xffffff9225703f10 : 0xffffff8022f8b5af mach_kernel : _lstat64 + 0x2f
0xffffff9225703f40 : 0xffffff80233819a7 mach_kernel : _unix_syscall64 + 0x287
0xffffff9225703fa0 : 0xffffff8022cbe206 mach_kernel : _hndl_unix_scall64 + 0x16
      Kernel Extensions in backtrace:
         org.openzfsonosx.zfs(2.0.1)[C8900DAA-9C3B-36DD-8C6D-7869FC2DBF69]@0xffffff7fa39b0000->0xffffff7fa5161fff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[F3D120E5-29D0-32C7-A09A-E88B84675F9C]@0xffffff7fa3910000

Steps to reproduce:

# Unfortunately I don't know what mc does exactly, 
# I couldn't reproduce with standard utils like ls / stat
brew install midnight-commander

truncate -s 100M /tmp/ztest.img
sudo zpool create ztest /tmp/ztest.img
sudo chown $(whoami) /Volumes/ztest/

# the path needs to be long, remove 6-7 chars and everything's okay
# notice that it's many nested directories, not a single one in /Volumes/ztest/
mkdir -p /Volumes/ztest/123456789.123456789/123456789.123456789.123456789/123456789.123456789.123456789/123456789.123456789/123456789.12345689.123456789/123456789.123456789.123456789/
cd /Volumes/ztest/123456789.123456789/123456789.123456789.123456789/123456789.123456789.123456789/123456789.123456789/123456789.12345689.123456789/123456789.123456789.123456789/
mc
# go to the parent directory and back
# panic here

System info:

zfs-macOS-2.0.1-1
zfs-kmod-2.0.1-1

macOS 10.15.7 (19H1030)
Darwin mars.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Apr 12 20:57:45 PDT 2021; root:xnu-6153.141.28.1~1/RELEASE_X86_64 x86_64 i386 Macmini6,2 Darwin

panic: import_rewind_config_changed

testpool1: blkptr at 0xffffff88967d3cc0 DVA 0 has invalid VDEV 1

After zpool create, chown: /Volumes/mypool/.Spotlight-V100/Store-V2: No such file or directory

Immediately, after zpool create:

2.0-rc6

joe@joes-Mac-2 ~ % sudo zpool create mypool disk2 
joe@joes-Mac-2 ~ % sudo chown -R joe /Volumes/mypool 
chown: /Volumes/mypool/.Spotlight-V100/Store-V2: No such file or directory
chown: /Volumes/mypool/.Spotlight-V100/VolumeConfiguration.plist: No such file or directory
chown: /Volumes/mypool/.Spotlight-V100: No such file or directory
n1@joes-Mac-2 ~ % sudo chown -R joe /Volumes/mypool
chown: /Volumes/mypool/.Spotlight-V100/Store-V2: No such file or directory
chown: /Volumes/mypool/.Spotlight-V100/VolumeConfiguration.plist: No such file or directory
chown: /Volumes/mypool/.Spotlight-V100: No such file or directory
joe@joes-Mac-2 ~ % sudo zpool export mypool
Volume mypool on disk6s1 unmounted
joe@joes-Mac-2 ~ % sudo zpool import mypool
joe@joes-Mac-2 ~ % sudo chown -R joe /Volumes/mypool
joe@joes-Mac-2 ~ %

Fine in 1.9.4:

joe@joes-Mac-3 ~ % sudo zpool create mypool disk2 
joe@joes-Mac-3 ~ % sudo chown -R joe /Volumes/mypool 
joe@joes-Mac-3 ~ %

cmd/zfs is missing mount_zfs logic

Making it hard for boot to mount "/"

Why is zfs panicking? thanks

System information

Type	Version/Name
Distribution Name
Distribution Version
Linux Kernel
Architecture
ZFS Version
SPL Version

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

panic(cpu 2 caller 0xffffff800c04c32a): Kernel trap at 0xffffff7f8e993d26, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000000000000, CR3: 0x0000000012f0b000, CR4: 0x00000000000406e0
RAX: 0x0000000000000000, RBX: 0xffffff8119614a78, RCX: 0x0000000000000000, RDX: 0x0000000003000000
RSP: 0xffffff913d4bbf30, RBP: 0xffffff913d4bbfa0, RSI: 0xffffff8119614aa0, RDI: 0x0000000000000000
R8:  0x0000000000000000, R9:  0x0000000000989680, R10: 0x0000000000000000, R11: 0x0000007735c182d9
R12: 0x0000000000000a85, R13: 0xffffff8128146898, R14: 0x0000000000000000, R15: 0x00000e1dc19b333f
RFL: 0x0000000000010202, RIP: 0xffffff7f8e993d26, CS:  0x0000000000000008, SS:  0x0000000000000000
Fault CR2: 0x0000000000000000, Error code: 0x0000000000000002, Fault CPU: 0x2, PL: 0, VF: 1

Backtrace (CPU 2), Frame : Return Address
0xffffff913d4bb850 : 0xffffff800bf215cd mach_kernel : _handle_debugger_trap + 0x49d
0xffffff913d4bb8a0 : 0xffffff800c05a3c5 mach_kernel : _kdp_i386_trap + 0x155
0xffffff913d4bb8e0 : 0xffffff800c04bf7e mach_kernel : _kernel_trap + 0x4ee
0xffffff913d4bb930 : 0xffffff7f90273a7d as.vit9696.VirtualSMC : __ZN18VirtualSMCProvider10kernelTrapI22x86_saved_state_1010_tEEvPT_Pm + 0x46d
0xffffff913d4bb9d0 : 0xffffff800bec7a40 mach_kernel : _return_from_trap + 0xe0
0xffffff913d4bb9f0 : 0xffffff800bf20c97 mach_kernel : _DebuggerTrapWithState + 0x17
0xffffff913d4bbaf0 : 0xffffff800bf21087 mach_kernel : _panic_trap_to_debugger + 0x227
0xffffff913d4bbb40 : 0xffffff800c6c2c7c mach_kernel : _panic + 0x54
0xffffff913d4bbbb0 : 0xffffff800c04c32a mach_kernel : _sync_iss_to_iks + 0x2aa
0xffffff913d4bbd30 : 0xffffff800c04c028 mach_kernel : _kernel_trap + 0x598
0xffffff913d4bbd80 : 0xffffff7f90273a7d as.vit9696.VirtualSMC : __ZN18VirtualSMCProvider10kernelTrapI22x86_saved_state_1010_tEEvPT_Pm + 0x46d
0xffffff913d4bbe20 : 0xffffff800bec7a40 mach_kernel : _return_from_trap + 0xe0
0xffffff913d4bbe40 : 0xffffff7f8e993d26 net.lundman.spl : _taskq_thread + 0xf8
0xffffff913d4bbfa0 : 0xffffff800bec713e mach_kernel : _call_continuation + 0x2e
      Kernel Extensions in backtrace:
         as.vit9696.VirtualSMC(1.1.4)[D51ACAF0-8CAA-3DC2-8260-C51D354709DF]@0xffffff7f90264000->0xffffff7f9028afff
            dependency: as.vit9696.Lilu(1.4.5)[04010205-4C3D-3B04-BAF5-48AA69FE71F2]@0xffffff7f90081000
            dependency: com.apple.iokit.IOACPIFamily(1.4)[9D1FF279-C4A2-3344-902F-E0B22B508689]@0xffffff7f8cca1000
         net.lundman.spl(1.9.4)[F4C8FC99-7C45-3C5A-BE70-A23D3A03170D]@0xffffff7f8e986000->0xffffff7f8fb7bfff

BSD process name corresponding to current thread: kernel_task
Boot args: msgbuf=393216 -radvesa npci=0x2000 -v debug=0x100 keepsyms=1 

Mac OS version:
19E287

Kernel version:
Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64
Kernel UUID: AB0AA7EE-3D03-3C21-91AD-5719D79D7AF6
Kernel slide:     0x000000000bc00000
Kernel text base: 0xffffff800be00000
__HIB  text base: 0xffffff800bd00000
System model name: iMacPro1,1 (Mac-7BA5B2D9E42DDD94)
System shutdown begun: NO
Panic diags file available: YES (0x0)

System uptime in nanoseconds: 15773832300059
last loaded kext at 14463807127: org.virtualbox.kext.VBoxNetAdp	6.1.4 (addr 0xffffff7f90492000, size 28672)
last unloaded kext at 190856079786: >!AHPET	1.8 (addr 0xffffff7f8e3c0000, size 20480)
loaded kexts:
org.virtualbox.kext.VBoxNetAdp	6.1.4
org.virtualbox.kext.VBoxNetFlt	6.1.4
org.virtualbox.kext.VBoxUSB	6.1.4
org.virtualbox.kext.VBoxDrv	6.1.4
com.paragon-software.filesystems.extfs	30.3.11
com.nomachine.driver.nxau	4.1.b2
com.objective-see.lulu	1.2.3
net.lundman.zfs	1.9.4
net.lundman.spl	1.9.4
com.insanelymac.RealtekRTL8111	2.2.2
as.vit9696.VirtualSMC	1.1.4
as.vit9696.WhateverGreen	1.4.0
as.vit9696.!AALC	1.5.0
as.vit9696.Lilu	1.4.5
com.rehabman.driver.USBInjectAll	0.7.1
@fileutil	20.036.15
@filesystems.autofs	3.0
@AGDCPluginDisplayMetrics	5.1.16
>!AUpstreamUserClient	3.6.8
>!AMCCSControl	1.11
>!AHV	1
|IOUserEthernet	1.0.1
|IO!BSerialManager	7.0.4f6
>pmtelemetry	1
@Dont_Steal_Mac_OS_X	7.0.0
>!A!ISlowAdaptiveClocking	4.0.0
>ACPI_SMC_PlatformPlugin	1.0.0
>!A!IMCEReporter	115
|SCSITaskUserClient	422.101.1
@private.KextAudit	1.0
>!AVirtIO	1.0
@filesystems.hfs.kext	522.100.5
@!AFSCompression.!AFSCompressionTypeDataless	1.0.0d1
@BootCache	40
@!AFSCompression.!AFSCompressionTypeZlib	1.0.0
@filesystems.apfs	1412.101.1
>!AAHCIPort	341.0.2
>!AACPIButtons	6.1
>!ARTC	2.0
>!ASMBIOS	2.1
>!AAPIC	1.7
$!AImage4	1
@nke.applicationfirewall	303
$TMSafetyNet	8
@!ASystemPolicy	2.0.0
>!A!ICPUPowerManagement	222.0.0
|EndpointSecurity	1
|IOUSBUserClient	900.4.2
@kext.triggers	1.0
>usb.networking	5.0.0
>usb.!UHostCompositeDevice	1.2
>!ASMBus!C	1.0.18d1
|IOSMBus!F	1.1
|IOAVB!F	840.3
>!ASSE	1.0
|IO!B!F	7.0.4f6
|IO!BPacketLogger	7.0.4f6
@!AGPUWrangler	5.1.16
|IOSlowAdaptiveClocking!F	1.0.0
>IOPlatformPluginLegacy	1.0.0
>IOPlatformPlugin!F	6.0.0d8
|IONDRVSupport	575.1
@kext.AMDSupport	3.0.8
@!AGraphicsDeviceControl	5.1.16
|IOGraphics!F	575.1
@plugin.IOgPTPPlugin	840.3
|IOEthernetAVB!C	1.1.0
|IOSkywalk!F	1
|IOAHCISerialATAPI	268
|IOAudio!F	300.2
@vecLib.kext	1.2.0
|IOSerial!F	11
|IOSurface	269.11
@filesystems.hfs.encodings.kext	1
|IOAHCIBlock!S	316.100.5
>usb.!UOHCIPCI	1.2
>usb.!UOHCI	1.2
>usb.!UEHCI	1.2
|IOAHCI!F	290.0.1
>usb.!UHostPacketFilter	1.0
|IOUSB!F	900.4.2
>!AEFINVRAM	2.1
>!AEFIRuntime	2.1
|IOHID!F	2.0.0
$quarantine	4
$sandbox	300.0
@kext.!AMatch	1.0.0d1
>DiskImages	493.0.0
>!AFDEKeyStore	28.30
>!AEffaceable!S	1.0
>!AKeyStore	2
>!UTDM	489.101.1
|IOSCSIBlockCommandsDevice	422.101.1
>!ACredentialManager	1.0
>KernelRelayHost	1
>!ASEPManager	1.0.1
>IOSlaveProcessor	1
|IOUSBMass!SDriver	157.101.3
|IOSCSIArchitectureModel!F	422.101.1
|IO!S!F	2.1
|IOUSBHost!F	1.2
>!UHostMergeProperties	1.2
>usb.!UCommon	1.0
>!ABusPower!C	1.0
|CoreAnalytics!F	1
>!AMobileFileIntegrity	1.0.5
@kext.CoreTrust	1
|IOTimeSync!F	840.3
|IONetworking!F	3.4
|IOReport!F	47
>!AACPIPlatform	6.1
>!ASMC	3.1.9
>watchdog	1
|IOPCI!F	2.9
|IOACPI!F	1.4
@kec.pthread	1
@kec.corecrypto	1.0
@kec.Libm	1

Pool disappears after reboot, previous partition (whatever it is) reappears instead

Mojave 10.14.6, OpenZFS installed with brew (OpenZFS on OS X 1.9.4 Mojave.pkg)

By "disappears" here I don't mean "it's not imported", I mean actually, really disappears (no output from zpool import). The problem - on that particular machine at least (a macbook pro 13 unibody), and on that specific partition - is reproducible. I create, I reboot, nothing is there anymore.

It happens on a SSD partition (disk0s3). Previous partitions are: disk0s1 EFI, disk0s2 APFS with the OS. There's also another internal disk (disk1, HDD, entirely devoted to ZFS), plus a bunch of usb disks with ZFS, no problem there.

Whenever I reboot, disk0s3 reverts back to whatever partition was there before creating a ZFS pool in its place, be it a fat32 one or a HFS one or whatever.

I've tried to search here and on the internet, but I found nothing and I got no effin' clue to whatever's going on. My search results are obscured by people that cannot find their pool because it's not automatically imported, but this is not my case at all. This is not the only macOS machine on which I use ZFS - on one machine in particular I have exactly the same configuration, with a pool on a SSD partition and a pool on a whole magnetic disk. I use ZFS on Linux too, and it's the first time I encounter a situation like this.

If anyone has any ideas about how to proceed, I would be grateful. I don't have specific knowledge about ZFS internals, but I'm a software developer and I'm comfortable with the command line, so if there's debugging to do, I might try it if there are instructions to follow.

Error "no pools available to import" in Big Sur x86_64

System information

Type	Version/Name
Distribution Name	macOS
Distribution Version	11.4
Architecture	x86_64
ZFS Version	2.0.1
SPL Version

Describe the problem you're observing

Import the pool, but zpool show error no pools available to import

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

Log:

System Policy: zpool(1607) deny(1) file-read-data /dev/disk1s8
Violation:       deny(1) file-read-data /dev/disk1s8
Process:         zpool [1607]
Path:            /usr/local/zfs/bin/zpool
Load Address:    0x10f37e000
Identifier:      zpool
Version:         ??? (???)
Code Type:       x86_64 (Native)
Parent Process:  sudo [1603]
Responsible:     /System/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal
User ID:         0

Date/Time:       2021-06-18 22:24:10.392 GMT+8
OS Version:      macOS 11.4 (20F71)
Report Version:  8

MetaData: {"platform-policy":true,"errno":1,"signing-id":"zpool","primary-filter-value":"\/dev\/disk1s8","team-id":"735AM5QEU3","responsible-process-path":"\/System\/Applications\/Utilities\/Terminal.app\/Contents\/MacOS\/Terminal","profile":"platform","pid":1607,"normalized_target":["dev","disk1s8"],"path":"\/dev\/disk1s8","apple-internal":false,"matched-user-intent-extension":false,"rdev":16777224,"build":"macOS 11.4 (20F71)","operation":"file-read-data","profile-flags":0,"process":"zpool","platform_binary":"no","action":"deny","process-path":"\/usr\/local\/zfs\/bin\/zpool","primary-filter":"path","hardware":"Mac","target":"\/dev\/disk1s8","uid":0,"flags":21,"summary":"deny(1) file-read-data \/dev\/disk1s8","platform-binary":false,"vnode-type":"BLOCK-DEVICE","matched-extension":false}

Thread 0 (id: 21854):
0   libsystem_kernel.dylib        	0x00007fff202f3cde __psynch_cvwait + 10
1   libzfs.4.dylib                	0x000000010f400c12 tpool_wait + 98
2   libzfs.4.dylib                	0x000000010f3fc1d2 zpool_search_import + 1730
3   zpool                         	0x000000010f386973 zpool_do_import + 2067
4   zpool                         	0x000000010f380755 main + 453
5   libdyld.dylib                 	0x00007fff20341f5d start + 1
6                                 	0x0000000000000002

Thread 1 (id: 21863):
0   libsystem_kernel.dylib        	0x00007fff202f295e __workq_kernreturn + 10
1   libsystem_pthread.dylib       	0x00007fff2032242f start_wqthread + 15

Thread 2 (id: 21865):
0   libsystem_kernel.dylib        	0x00007fff202f1b52 __open + 10
1   libzfs.4.dylib                	0x000000010f3fef99 zpool_open_func + 825
2   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 3 (id: 21866):
0   libsystem_kernel.dylib        	0x00007fff202f1b52 __open + 10
1   libzfs.4.dylib                	0x000000010f3fef99 zpool_open_func + 825
2   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 4 (id: 21867):
0   libsystem_kernel.dylib        	0x00007fff202f1b52 __open + 10
1   libzfs.4.dylib                	0x000000010f3fef99 zpool_open_func + 825
2   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 5 (id: 21868):
0   libsystem_kernel.dylib        	0x00007fff202f1b52 __open + 10
1   libzfs.4.dylib                	0x000000010f3fef99 zpool_open_func + 825
2   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 6 (id: 21869):
0   libsystem_kernel.dylib        	0x00007fff202f35fe __psynch_mutexdrop + 10
1   libsystem_pthread.dylib       	0x00007fff203223c6 _pthread_mutex_firstfit_unlock_slow + 240
2   libzfs.4.dylib                	0x000000010f401074 tpool_worker + 612
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 7 (id: 21870):
0   libsystem_kernel.dylib        	0x00007fff202f34ca __psynch_mutexwait + 10
1   libsystem_pthread.dylib       	0x00007fff20322192 _pthread_mutex_firstfit_lock_slow + 204
2   libzfs.4.dylib                	0x000000010f400e30 tpool_worker + 32
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 8 (id: 21871):
0   libsystem_kernel.dylib        	0x00007fff202f27c6 stat$INODE64 + 10
1   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
2   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
3   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 9 (id: 21872):
0   libsystem_kernel.dylib        	0x00007fff202f1b52 __open + 10
1   libzfs.4.dylib                	0x000000010f3fef99 zpool_open_func + 825
2   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 10 (id: 21874):
0   libsystem_kernel.dylib        	0x00007fff202f1b52 __open + 10
1   libzfs.4.dylib                	0x000000010f3fef99 zpool_open_func + 825
2   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 11 (id: 21875):
0   libsystem_kernel.dylib        	0x00007fff202f1b52 __open + 10
1   libzfs.4.dylib                	0x000000010f3fef99 zpool_open_func + 825
2   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 12 (id: 21876):
0   libsystem_malloc.dylib        	0x00007fff2014fc08 tiny_malloc_should_clear + 1236
1   libsystem_malloc.dylib        	0x00007fff2014e727 szone_malloc_should_clear + 66
2   libsystem_malloc.dylib        	0x00007fff20167fe5 _malloc_zone_malloc + 118
3   libsystem_c.dylib             	0x00007fff202528e2 strdup + 32
4   libzfs.4.dylib                	0x000000010f3fb4b1 zutil_strdup + 17
5   libzfs.4.dylib                	0x000000010f3fec8e zpool_open_func + 46
6   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
7   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
8   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 13 (id: 21877):
0   libsystem_kernel.dylib        	0x00007fff202f27c6 stat$INODE64 + 10
1   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
2   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
3   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 14 (id: 21878):
0   libsystem_kernel.dylib        	0x00007fff202f27c6 stat$INODE64 + 10
1   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
2   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
3   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 15 (id: 21879):
0   libsystem_kernel.dylib        	0x00007fff202f27c6 stat$INODE64 + 10
1   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
2   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
3   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 16 (id: 21880):
0   libsystem_kernel.dylib        	0x00007fff202f34ca __psynch_mutexwait + 10
1   libsystem_pthread.dylib       	0x00007fff20322192 _pthread_mutex_firstfit_lock_slow + 204
2   libzfs.4.dylib                	0x000000010f400e30 tpool_worker + 32
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 17 (id: 21881):
0   libsystem_kernel.dylib        	0x00007fff202f29ee __ulock_wait + 10
1   libsystem_malloc.dylib        	0x00007fff2014f7c2 tiny_malloc_should_clear + 142
2   libsystem_malloc.dylib        	0x00007fff2014e727 szone_malloc_should_clear + 66
3   libsystem_malloc.dylib        	0x00007fff20167fe5 _malloc_zone_malloc + 118
4   libsystem_c.dylib             	0x00007fff202528e2 strdup + 32
5   libzfs.4.dylib                	0x000000010f3fb4b1 zutil_strdup + 17
6   libzfs.4.dylib                	0x000000010f3fec8e zpool_open_func + 46
7   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
8   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
9   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 18 (id: 21882):
0   libsystem_kernel.dylib        	0x00007fff202f29ee __ulock_wait + 10
1   libsystem_malloc.dylib        	0x00007fff2014f7c2 tiny_malloc_should_clear + 142
2   libsystem_malloc.dylib        	0x00007fff2014e727 szone_malloc_should_clear + 66
3   libsystem_malloc.dylib        	0x00007fff20167fe5 _malloc_zone_malloc + 118
4   libsystem_c.dylib             	0x00007fff202528e2 strdup + 32
5   libzfs.4.dylib                	0x000000010f3fb4b1 zutil_strdup + 17
6   libzfs.4.dylib                	0x000000010f3fec8e zpool_open_func + 46
7   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
8   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
9   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 19 (id: 21883):
0   libsystem_kernel.dylib        	0x00007fff202f34ca __psynch_mutexwait + 10
1   libsystem_pthread.dylib       	0x00007fff20322192 _pthread_mutex_firstfit_lock_slow + 204
2   libzfs.4.dylib                	0x000000010f400e30 tpool_worker + 32
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 20 (id: 21884):
0   libsystem_kernel.dylib        	0x00007fff202f35fe __psynch_mutexdrop + 10
1   libsystem_pthread.dylib       	0x00007fff203223c6 _pthread_mutex_firstfit_unlock_slow + 240
2   libzfs.4.dylib                	0x000000010f401074 tpool_worker + 612
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 21 (id: 21885):
0   libsystem_kernel.dylib        	0x00007fff202f34ca __psynch_mutexwait + 10
1   libsystem_pthread.dylib       	0x00007fff20322192 _pthread_mutex_firstfit_lock_slow + 204
2   libzfs.4.dylib                	0x000000010f400e30 tpool_worker + 32
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 22 (id: 21886):
0   libsystem_kernel.dylib        	0x00007fff202f34ca __psynch_mutexwait + 10
1   libsystem_pthread.dylib       	0x00007fff20322192 _pthread_mutex_firstfit_lock_slow + 204
2   libzfs.4.dylib                	0x000000010f400e30 tpool_worker + 32
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 23 (id: 21887):
0   libsystem_kernel.dylib        	0x00007fff202f34ca __psynch_mutexwait + 10
1   libsystem_pthread.dylib       	0x00007fff20322192 _pthread_mutex_firstfit_lock_slow + 204
2   libzfs.4.dylib                	0x000000010f400e30 tpool_worker + 32
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 24 (id: 21888):
0   libsystem_kernel.dylib        	0x00007fff202f34ca __psynch_mutexwait + 10
1   libsystem_pthread.dylib       	0x00007fff20322192 _pthread_mutex_firstfit_lock_slow + 204
2   libzfs.4.dylib                	0x000000010f400e30 tpool_worker + 32
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Thread 25 (id: 21889):
0   libsystem_kernel.dylib        	0x00007fff202f1b52 __open + 10
1   libzfs.4.dylib                	0x000000010f3fef99 zpool_open_func + 825
2   libzfs.4.dylib                	0x000000010f4010c0 tpool_worker + 688
3   libsystem_pthread.dylib       	0x00007fff203268fc _pthread_start + 224
4   libsystem_pthread.dylib       	0x00007fff20322443 thread_start + 15

Binary Images:
       0x10f37e000 -        0x10f3a1ff7  zpool (0) <cea734fc-9107-3a96-aea6-82b22620edb8> /usr/local/zfs/bin/zpool
       0x10f3bc000 -        0x10f413fcf  libzfs.4.dylib (0) <cfb3ec58-8aae-39a3-98be-18ee4b52f9c1> /usr/local/zfs/lib/libzfs.4.dylib
    0x7fff2014c000 -     0x7fff20178ff7  libsystem_malloc.dylib (317.121.1) <cad162a5-7367-3a30-9c15-5d036411aede> /usr/lib/system/libsystem_malloc.dylib
    0x7fff201fb000 -     0x7fff20283ff7  libsystem_c.dylib (1439.100.3) <38f8a126-c995-349a-b909-ff831914ed2e> /usr/lib/system/libsystem_c.dylib
    0x7fff202f0000 -     0x7fff2031ffff  libsystem_kernel.dylib (7195.121.3) <a4938cf5-abc0-397b-8a6e-b7beefa24d0a> /usr/lib/system/libsystem_kernel.dylib
    0x7fff20320000 -     0x7fff2032bfff  libsystem_pthread.dylib (454.120.2) <17482c9d-061e-3769-ac9e-be1239d33098> /usr/lib/system/libsystem_pthread.dylib
    0x7fff2032c000 -     0x7fff20367fff  libdyld.dylib (852) <c10cea28-d5a0-324f-8f07-8c7ce4805412> /usr/lib/system/libdyld.dylib

Thanks.

Non-root snapshot amnesia

joe@joes-Mac ~ % cd /Volumes/testpool/testfs/.zfs/snapshot
joe@joes-Mac snapshot % ls
snap
joe@joes-Mac snapshot % stat .
838860822 281474976710653 dr-xr-xr-x 0 root wheel 838860822 0 "Jun  5 04:34:54 2021" "Jun  5 04:34:54 2021" "Jun  5 04:34:54 2021" "Jun  5 04:34:54 2021" 512 1 0 .
joe@joes-Mac snapshot % ls    
.
joe@joes-Mac snapshot % ls
snap
joe@joes-Mac snapshot % ls
snap
joe@joes-Mac snapshot % stat .
838860822 281474976710653 dr-xr-xr-x 0 root wheel 838860822 0 "Jun  5 04:34:54 2021" "Jun  5 04:34:54 2021" "Jun  5 04:34:54 2021" "Jun  5 04:34:54 2021" 512 1 0 .
joe@joes-Mac snapshot % ls    
.
joe@joes-Mac snapshot % ls
snap
joe@joes-Mac snapshot % touch foo
touch: foo: Permission denied
joe@joes-Mac snapshot % ls       
snap
joe@joes-Mac snapshot % ls
snap
joe@joes-Mac snapshot % sudo touch foo
touch: foo: Operation not supported
joe@joes-Mac snapshot % ls
.
joe@joes-Mac snapshot % ls
snap
joe@joes-Mac snapshot % touch foo
touch: foo: Permission denied
joe@joes-Mac snapshot % ls       
snap
joe@joes-Mac snapshot % sudo touch foo
touch: foo: Operation not supported
joe@joes-Mac snapshot % sudo ls       
snap
joe@joes-Mac snapshot % sudo touch foo
touch: foo: Operation not supported
joe@joes-Mac snapshot % ls
.
joe@joes-Mac snapshot % touch foo
touch: foo: Permission denied
joe@joes-Mac snapshot % sudo ls       
snap
joe@joes-Mac snapshot % file .
.: directory
joe@joes-Mac snapshot % ls
snap
joe@joes-Mac snapshot % ls
snap
joe@joes-Mac snapshot % sudo file .
.: directory
joe@joes-Mac snapshot % ls         
.
joe@joes-Mac snapshot % ls
snap
joe@joes-Mac snapshot %

rw_lock goes negative

Died:

 frame #63: 0xffffff7f91158645 zfs`zvol_os_register_device_cb(param=0xffffff90ad3b7000) at zvol_os.c:126:2
   123
   124         rw_enter(&zv->zv_suspend_lock, RW_WRITER);
   125         zvolRegisterDevice(zv);
-> 126         rw_exit(&zv->zv_suspend_lock);
   127     }

(lldb) p *zv
  zv_suspend_lock = {
    rw_lock = ([0] = 553648128, [1] = 0, [2] = 0, [3] = 0)
    rw_owner = 0x0000000000000000
    rw_readers = -1
    rw_pad = 305419896
  }

triggered by test after:
zvol_misc/zvol_misc_snapdev (run as root) [00:16] [FAIL]

taskq_cancel_id() can return too quickly

Two panics related to this, one is vdev_deadman:

     : 0xffffff8004ebda40 mach_kernel : _return_from_trap + 0xe0
     : 0xffffff7f8942bbbf org.openzfsonosx.zfs : _vdev_deadman + 0x1f
     : 0xffffff7f8941149a org.openzfsonosx.zfs : _spa_deadman + 0xca
     : 0xffffff7f896a6246 org.openzfsonosx.zfs : _taskq_thread + 0x4a6

as it triggered after taskq_cancel_id() returned and resources were freed.

The other is
dmu_objset.c: if ((taskq_cancel_id(os->os_spa->spa_upgrade_taskq, id)) == 0) {

abd using wrong size on free

panic(cpu 0): VERIFY3(size == abd->abd_orig_size) failed (120 == 128)

abd_alloc_scatter_offset_chunkcnt: abd 0xffffff90ca886080 chunkcnt 4 abd_size 128
abd_free_struct: abd 0xffffff90ca886080 ->abd_size 8192 chunkcnt 2 size 120
VERIFY3(size == abd->abd_orig_size) failed (120 == 128)

Ie, abd_alloc_scatter_offset_chunkcnt() allocates with 4 chunkcnt, but abd_free_struct uses 2 when calling free.

as discussed in openzfs/zfs#10431

zfstests: zpool_upgrade

Long filename with multibyte

Max filename is 255 characters long on Mac.
This does not mean 255 bytes.

For multi-byte characters, it will exceed 255 bytes.
Attempting to copy a file with that long name to zfs, it will crash.

At least, I want no copy with error msg.

My environment:
macOS Big Sur 11.2.1
OpenZFSonOsX-2.0.0-Big.Sur-11.0

panic: zpool_wait_trim_flag

There are some mutex locks being held for unusually long times.

Here is tail end of the test running:

2020-10-05 17:57:09.777 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803bb2e5e0) held for 246s by '../../../zfs/spa.c':7283
2020-10-05 17:57:09.778 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff80402e1860) held for 118s by '../../../zfs/vdev_trim.c':475
2020-10-05 17:57:09.778 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803838ce00) held for 246s by '../../../zfs/zil.c':2453
2020-10-05 17:57:19.348 Df kernel.development[0:3f6d0] (zfs) <zfs`spl_mutex_exit (spl-mutex.c:361)> SPL: mutex (0xffffff80402e1860) finally released after 128s by '../../../zfs/vdev_trim.c':475
2020-10-05 17:57:19.776 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803bb2e5e0) held for 256s by '../../../zfs/spa.c':7283
2020-10-05 17:57:19.777 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803838ce00) held for 256s by '../../../zfs/zil.c':2453
2020-10-05 17:57:29.776 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803bb2e5e0) held for 266s by '../../../zfs/spa.c':7283
2020-10-05 17:57:29.776 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803838ce00) held for 266s by '../../../zfs/zil.c':2453
2020-10-05 17:57:39.776 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803bb2e5e0) held for 276s by '../../../zfs/spa.c':7283
2020-10-05 17:57:39.776 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803838ce00) held for 276s by '../../../zfs/zil.c':2453
2020-10-05 17:57:49.774 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803bb2e5e0) held for 286s by '../../../zfs/spa.c':7283
2020-10-05 17:57:49.775 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803838ce00) held for 286s by '../../../zfs/zil.c':2453
2020-10-05 17:57:59.774 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803bb2e5e0) held for 296s by '../../../zfs/spa.c':7283
2020-10-05 17:57:59.775 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803838ce00) held for 296s by '../../../zfs/zil.c':2453
2020-10-05 17:58:09.773 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803bb2e5e0) held for 306s by '../../../zfs/spa.c':7283
2020-10-05 17:58:09.774 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803838ce00) held for 306s by '../../../zfs/zil.c':2453
2020-10-05 17:58:19.773 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803bb2e5e0) held for 316s by '../../../zfs/spa.c':7283
2020-10-05 17:58:19.774 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff80402e1860) held for 60s by '../../../zfs/vdev_trim.c':475
2020-10-05 17:58:19.774 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803838ce00) held for 316s by '../../../zfs/zil.c':2453
2020-10-05 17:58:29.773 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803bb2e5e0) held for 326s by '../../../zfs/spa.c':7283
2020-10-05 17:58:29.774 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff80402e1860) held for 70s by '../../../zfs/vdev_trim.c':475
2020-10-05 17:58:29.774 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803838ce00) held for 326s by '../../../zfs/zil.c':2453
2020-10-05 17:58:39.772 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803bb2e5e0) held for 336s by '../../../zfs/spa.c':7283
2020-10-05 17:58:39.773 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff80402e1860) held for 80s by '../../../zfs/vdev_trim.c':475
2020-10-05 17:58:39.773 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803838ce00) held for 336s by '../../../zfs/zil.c':2453
2020-10-05 17:58:49.772 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803bb2e5e0) held for 346s by '../../../zfs/spa.c':7283
2020-10-05 17:58:49.772 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff80402e1860) held for 90s by '../../../zfs/vdev_trim.c':475
2020-10-05 17:58:49.772 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803838ce00) held for 346s by '../../../zfs/zil.c':2453
2020-10-05 17:58:59.771 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803bb2e5e0) held for 356s by '../../../zfs/spa.c':7283
2020-10-05 17:58:59.772 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff80402e1860) held for 100s by '../../../zfs/vdev_trim.c':475
2020-10-05 17:58:59.772 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803838ce00) held for 356s by '../../../zfs/zil.c':2453
2020-10-05 17:59:09.771 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803bb2e5e0) held for 366s by '../../../zfs/spa.c':7283
2020-10-05 17:59:09.772 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff80402e1860) held for 110s by '../../../zfs/vdev_trim.c':475
2020-10-05 17:59:09.772 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803838ce00) held for 366s by '../../../zfs/zil.c':2453
2020-10-05 17:59:19.770 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803bb2e5e0) held for 376s by '../../../zfs/spa.c':7283
2020-10-05 17:59:19.771 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff80402e1860) held for 120s by '../../../zfs/vdev_trim.c':475
2020-10-05 17:59:19.771 Df kernel.development[0:b9a] (zfs) <zfs`spl_wdlist_check (spl-mutex.c:97)> SPL: mutex (0xffffff803838ce00) held for 376s by '../../../zfs/zil.c':2453
2020-10-05 17:59:27.342 Df kernel.development[0:3f6d0] (zfs) <zfs`spl_mutex_exit (spl-mutex.c:361)> SPL: mutex (0xffffff80402e1860) finally released after 128s by '../../../zfs/vdev_trim.c':475
2020-10-05 17:59:27.345 Df kernel.development[0:3f641] (zfs) <zfs`__dprintf (zfs_debug.c:250)> spa_history.c:309:spa_history_log_sync(): txg 13 trim vdev=/var/tmp/testdir/file_vdev2 activated
2020-10-05 17:59:27.356 Df kernel.development[0:3f258] (zfs) <zfs`spl_mutex_exit (spl-mutex.c:361)> SPL: mutex (0xffffff803838ce00) finally released after 384s by '../../../zfs/zil.c':2453
client_loop: send disconnect: Broken pipe

Unfortunately, crashed without starting debugger.

Destroying mounted zvol fails

This is the same as upstream behavior, but given we can automatically eject zvols on zpool destroy and zpool export, it seems we should be able to do the same for zfs destroy.

joes-Mac:~ root# zpool create foo disk6 
joes-Mac:~ root# zfs create -V64M foo/vol
joes-Mac:~ root# diskutil partitionDisk $(readlink /var/run/zfs/zvol/dsk/foo/vol) JHFS+ myhfs R &>/dev/null
joes-Mac:~ root# mount | grep myhfs
/dev/disk9s1 on /Volumes/myhfs (hfs, local, nodev, nosuid, journaled, noowners)
joes-Mac:~ root# zfs destroy foo/vol
cannot destroy 'foo/vol': dataset is busy
joes-Mac:~ root# zpool destroy foo    
Volume foo on disk8s1 unmounted
Exporting 'foo/vol'
... asking apfs to eject 'disk9s1'
Unmount of all volumes on disk9 was successful
... asking ZVOL to export 'disk9'
Unmount of all volumes on disk9 was successful
joes-Mac:~ root#

Also, the error printed when trying to destroy a child dataset with a zvol inside it is confusing:

joes-Mac:~ root# zpool create foo disk6
joes-Mac:~ root# zfs create foo/fs
joes-Mac:~ root# zfs create -V64M foo/fs/vol 
joes-Mac:~ root# diskutil partitionDisk $(readlink /var/run/zfs/zvol/dsk/foo/fs/vol) JHFS+ myhfs R &>/dev/null
joes-Mac:~ root# mount | grep myhfs
/dev/disk9s1 on /Volumes/myhfs (hfs, local, nodev, nosuid, journaled, noowners)
joes-Mac:~ root# zfs destroy foo/fs
cannot destroy 'foo/fs': filesystem has children
use '-r' to destroy the following datasets:
foo/fs/vol
joes-Mac:~ root# zfs destroy -r foo/fs
cannot destroy 'foo/fs/vol': dataset is busy
Unmount successful for /Volumes/foo/fs
cannot destroy 'foo/fs': dataset already exists
joes-Mac:~ root# zfs destroy -r foo/fs
cannot destroy 'foo/fs/vol': dataset is busy
cannot destroy 'foo/fs': dataset already exists
joes-Mac:~ root#

Yes, we know foo/fs exists. That's why we're trying to destroy it.

arcstat not working

System information

Type	Version/Name
Distribution Name	macOS Big Sur
Distribution Version	11.4
Linux Kernel
Architecture	ARM M1
ZFS Version	2.1.0 rc1
SPL Version	-

Describe the problem you're observing

running arcstat is not successful

Describe how to reproduce the problem

execute arcstat

Include any warning/errors/backtraces from the system logs

Traceback (most recent call last):
  File "/usr/local/zfs/bin/arcstat", line 554, in <module>
    main()
  File "/usr/local/zfs/bin/arcstat", line 527, in main
    init()
  File "/usr/local/zfs/bin/arcstat", line 395, in init
    snap_stats()
  File "/usr/local/zfs/bin/arcstat", line 226, in snap_stats
    kstat_update()
NameError: name 'kstat_update' is not defined

Xcode 12.2: /usr/include/mach/task.h:236:1: error: too few arguments provided to function-like macro invocation );

System information

Type	Version/Name
Distribution Name	macOS
Distribution Version	Big Sur / 11.0.1 (20B29)
Linux Kernel	N/A
Architecture	x86_64
ZFS Version	current macOS branch
SPL Version	current macOS branch

Describe the problem you're observing

Can't compile on recent stable Xcode 12.2 @ Big Sur 11.0.1

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

In file included from zfs_util.c:61:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:39:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/LaunchServices.h:23:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/IconsCore.h:23:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/OSServices.h:29:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/CSIdentity.h:43:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:87:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/SecCode.h:35:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/xpc/xpc.h:337:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/launch.h:19:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/mach/mach.h:67:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/mach/mach_interface.h:49:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/mach/task.h:236:1: error: too few arguments provided to function-like
      macro invocation
);
^
../../../../include/sys/zfs_context.h:232:9: note: macro 'thread_create' defined here
#define thread_create(stk, stksize, func, arg, len, pp, state, pri)     \
        ^
1 error generated.

The snapshot directory of a destroyed snapshot lingers invisibly

Note: This may be related to #69 but I'm not sure.

Here is a reproducer:

joe@joes-Mac ~ % cat snapshot-ghost.sh 
#!/bin/sh -x
zpool create foo disk0
touch /Volumes/foo/hello
sleep 1 && sync && zpool sync
zfs snapshot foo@s0
zfs mount foo@s0
zfs unmount foo@s0
zfs destroy foo@s0
ls -al /Volumes/foo/.zfs/snapshot/s0
echo "Why no error? It was destroyed."
ls -al /Volumes/foo/.zfs/snapshot/
echo "Wait s0 is not even there!"
file /Volumes/foo/.zfs/snapshot/s0
echo "Or is it?"
ls -al /Volumes/foo/.zfs/snapshot/s1
echo "Correct because it never existed."
zpool destroy foo
joe@joes-Mac ~ %

Demo running it:

Last login: Thu Jun  3 07:58:06 on ttys006
joe@joes-Mac ~ % sudo ./snapshot-ghost.sh 
+ zpool create foo disk0
+ touch /Volumes/foo/hello
+ sleep 1
+ sync
+ zpool sync
+ zfs snapshot foo@s0
+ zfs mount foo@s0
ZFS: snapshot mountpoint '/Volumes/foo/.zfs/snapshot/s0/'
+ zfs unmount foo@s0
Unmount successful for /Volumes/foo/.zfs/snapshot/s0/
+ zfs destroy foo@s0
+ ls -al /Volumes/foo/.zfs/snapshot/s0
total 2
dr-xr-xr-x  0 root  wheel  0 Jun  3 08:06 .
dr-xr-xr-x  0 root  wheel  0 Jun  3 08:06 ..
+ echo 'Why no error? It was destroyed.'
Why no error? It was destroyed.
+ ls -al /Volumes/foo/.zfs/snapshot/
total 2
dr-xr-xr-x  0 root  wheel  0 Jun  3 08:06 .
dr-xr-xr-x  0 root  wheel  0 Jun  3 08:06 ..
+ echo 'Wait s0 is not even there!'
Wait s0 is not even there!
+ file /Volumes/foo/.zfs/snapshot/s0
/Volumes/foo/.zfs/snapshot/s0: directory
+ echo 'Or is it?'
Or is it?
+ ls -al /Volumes/foo/.zfs/snapshot/s1
ls: /Volumes/foo/.zfs/snapshot/s1: No such file or directory
+ echo 'Correct because it never existed.'
Correct because it never existed.
+ zpool destroy foo
Volume foo on disk8s1 unmounted
joe@joes-Mac ~ %

It seems that the snapshot must have been mounted and unmounted in order for its snapshot directory to linger invisibly as above.

Perhaps the s0 directory has been cached as existing and the cache isn't updated after the snapshot is unmounted and destroyed.

zfs destroy prints extraneous line saying "retry"

2.0.0-rc6:

joe@joes-Mac-2 ~ % sudo zfs destroy tank/fs
Unmount successful for /Volumes/tank/fs
retry

1.9.4

joe@joes-Mac ~ % sudo zfs destroy tank/fs
Running process: '/usr/sbin/diskutil' 'unmount' '/Volumes/tank/fs' 
Unmount successful for /Volumes/tank/fs

Destroying mounted snapshot fails

joe@joes-Mac openzfs % sudo zpool create foo disk6
joe@joes-Mac openzfs % sudo zfs snapshot foo@snappy
joe@joes-Mac openzfs % sudo zfs mount foo@snappy
ZFS: snapshot mountpoint '/Volumes/foo/.zfs/snapshot/snappy/'
joe@joes-Mac openzfs % sudo zfs destroy foo@snappy
cannot destroy snapshot foo@snappy: dataset is busy
joe@joes-Mac openzfs % sudo zfs destroy foo@snappy
joe@joes-Mac openzfs %

Missing some sysctls

sysctl: unknown oid 'kstat.zfs.darwin.tunable.l2arc_mfuonly'
sysctl: unknown oid 'kstat.zfs.darwin.tunable.zfs_multihost_history'
sysctl: unknown oid 'kstat.zfs.darwin.tunable.zfs_rebuild_scrub_enabled'
sysctl: unknown oid 'kstat.zfs.darwin.tunable.zfs_txg_history'
sysctl: unknown oid 'kstat.zfs.darwin.tunable.vdev_file_physical_ashift'
sysctl: unknown oid 'kstat.zfs.darwin.tunable.zvol_volmode'
sysctl: unknown oid 'kstat.zfs.darwin.tunable.zfs_zevent_retain_max'

Trying to create a pool on a zvol gets stuck

On 1.9.4 creating a pool on a zvol works fine.

But in 2.0-rc6 zfs gets stuck

Last login: Mon May  3 16:24:42 on ttys004
joe@joes-Mac ~ % sudo zfs create -V 500M goodbye/myzvol           
Password:
joe@joes-Mac ~ % ls -l /var/run/zfs/zvol/dsk/goodbye/myzvol 
lrwxr-xr-x  1 root  daemon  10 May  3 16:25 /var/run/zfs/zvol/dsk/goodbye/myzvol -> /dev/disk8
joe@joes-Mac ~ % sudo zpool create poolonmyzvol disk8
<command hangs here>

All other zfs commands are impossible at this point, and the system must be rebooted. Spindump attached.

spindump.txt

openzfsonosx / openzfs Goto Github PK

openzfs's Introduction

Official Resources

Installation

Contribute & Develop

Release

Supported Kernels

openzfs's People

Contributors

Stargazers

Watchers

Forkers

openzfs's Issues

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

Steps to reproduce:

System info:

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

Recommend Projects

Recommend Topics

Recommend Org