artasoftkey / zumastor Goto Github PK

We're planning to switch from a block-based
exception index to an extent-based index.
This is a tricky thing.  Once this is designed,
but before it's implemented, we probably want
to add a unit test for the btree code,
and use that to help bring up the new index.

This depends on the unit test framework (issue 29).
It probably will require some changes to better separate
the btree code from the rest of ddsnap;
see writeup in
http://zumastor.googlecode.com/svn/trunk/doc/Cleanup_roadmap.txt
The reason to defer writing the unit test until
after the extent-based index is designed is
to help make sure that when we hoist btree
out of ddsnap, we do it in a way that isn't
likely to interfere with the extent rewrite.

* What steps will reproduce the problem?

1. Define a master without --samba, for example: zumastor define master
zumatest --hourly 24
2. Create a file, take snapshot
3. Modify the file, take snapshot
4. Define the same master with --samba: zumastor define master zumatest
--hourly 24 --samba
5. Modify the file, take snapshot

* What is the expected output? What do you see instead?

Expected: all the snapshots are shown in the "Previous versions" tab in
Windows.
Actual output: only snapshots taken after defining the master with --samba
(step 4) are shown in "Previous versions"

* What version of the product are you using? On what operating system?

Zumastor 0.6
Ubuntu Server 7.10
Windows XP SP2

The openfiler project would like to try zumastor,
but they need us to support 2.6.22.10.
See
https://lists.openfiler.com/pipermail/openfiler-devel/2007-November/000665.html

"zumastor snapshot" currently returns before the snapshot is
taken (let alone mounted).  We probably need to 
have an option to say whether to wait, and might
want to change the default to wait.

Other commands might need this, too.

At the moment, we assume that RAM is cheap, and the snapshot exception
btree takes up (size of snapshot store / snapshot blocksize) * 16 bytes,
or 512MB for a 5OOGB snapshot store with 16KB snapshot blocks.

This is excessive for some customers, for instance, popular low cost
Linux-based NAS appliances tend to ship with only 128MB of RAM.
Even if they increase the amount of RAM they ship with,
it won't be a huge amount, and people would rather use that
precious RAM for caching data rather than fluffy metadata.

Dan P. estimates it would take about a month to switch to an
extent-based tree which would reduce RAM consumption significantly.

This is a subset of issue 20.

Users will want to do both offline and offline growing and shrinking
of Zumastor volumes and snapshot stores, but the most urgent of these
(and the easiest to implement) is probably offline growth.
So let's implement / document / test that soon,
even if the rest of it has to wait until later.

Under certain circumstances, calling sync() or the like
before taking a snapshot may not be enough to make sure the snapshot 
is safe.  We should provide an option to control whether ddsnapd
calls freeze_bdev() / thaw_bdev() around the snapshot, and consider
making this the default, as it makes XFS no longer a special case, 
and would let us remove some XFS-specific cruft from zumastor.

This is related to, but not exactly the same as,
issue 32 (Add hooks for application quiescing during snapshots).
That issue is about cooperating apps; this new issue
is about making the snapshot safe(r) in the presence of
noncooperating apps.  Both are probably needed.

See discussion on zumastor list, e.g.
http://groups.google.com/group/zumastor/msg/24d2b500f46ca2a1
which has the results of a test demonstrating the problem.

We seem to run on 64 bit systems, but compiling produces some
scary warnings that might mean there are bugs lurking.

Most file servers seem to run
either on a proprietary operating
system, on proprietary hardware, or
use proprietary software.

Linux should be capable of serving most
file server needs out of the box.

So what's keeping people from using Linux for file servers?
For one thing, Linux's native LVM snapshots
suffer from performance and stability problems.
For another, Linux provides no really good way to replicate
disk volumes (rsync is good, but not good enough for some users).

Let's fix at least those issues.

In setup_sb(), the size of the copyout buffer is hardcoded to 32 times the
chunk size.  This should at least be a #define and could even be a run-time
tunable.

According to Dan Phillips, one way to fix this is to add the value to the
on-disk superblock and then, if desired, tune it there.

Some users will object to a limit of 64 snapshots.
For instance, at least one LVM user expected 200
snapshots to work.

Dan P. says that it might not be too hard to switch
from absolute snapshot numbers to incremental
ones, such that the absolute snapshot number is
reconstructed as we scan for the appropriate
exception record.  This would remove the current arbitrary
limit.

We still don't have a unit test framework.
Chris wrote and merged one, but it was
reverted right before 0.4.  Let's bring
it back for 0.6.

We currently have a bogus volume size limitation of 2TB due to only calling
the 32 bit BLKGETSIZE ioctl().  To support larger volumes we need to call
the BLKGETSIZE64 ioctl().  The getsize() function can be lifted from
util-linux-ng.

The cbtb tests have been badly broken for over a week,
and strictly speaking, I don't think they've ever all
passed; at least, I don't see a "test success" message in my inbox.

This may be because there is currently
no way to run the cbtb tests by hand;
everybody checks in their changes and waits for
the continuous build and test box to run the tests.

So let's make a way to run the cbtb test suite
on UML as a quick smoke test before submission.
(This is like tests/uml, but using the same cbtb tests 
that will be run by the continuous build and test box.)

This should be fast so even people who aren't test enthusiasts
will be willing to run them. e.g. we'd like the time to build
and run everything to be something like
12 minutes when touching the kernel, 
4 minutes if touching only ddsnap,
3 minutes if only touching a test script
This is a bit of a stretch because the current uml tests
take 27 minutes to run when touching the kernel, but
perhaps we can shave that down (e.g. by automatically using -j2
when building uml if the system has multiple cores).

We'd also like this to be usable on non-debian systems.

We have not yet documented how to
do even offline resizing, especially
of snapshot stores.  Let's design, 
document, test, and implement that feature.

To allow service owners to run a read-only service continually off of older
snapshots (internal request), and to allow for the system tests to have
constant paths to snapshots, it would be useful if the zumastor script
would create stable symbolic symlinks to the current and previous snapshot
mountpoints and block devices.  For example:

/var/run/zumastor/mount/testvol-hourly -> /var/run/zumastor/mount/testvol(274)
/var/run/zumastor/mount/testvol-hourly-1 ->
/var/run/zumastor/mount/testvol(272)
/var/run/zumastor/devices/testvol-hourly -> /dev/mapper/testvol(274)

Symlinks can be established for each of the rotations (hourly, daily, etc)

With the recent change to datestamped mountpoints, this is even more
important as nearly all the filesystem tests are broken and have no simple
way to determine where the next snapshot and previous snapshots were
mounted.  The test scripts will then be able to readlink the symlink and
store that value for future reference.

The symlinks would be updated as the zumastor script removes mounts and
devices, and replaced with the new values as appropriate.

Priority set high as most of the regression tests are failing as a result
of recent changes.

-Drake

The snapshot store cache size is currently hardcoded at the lesser of 128MB
or 1/4 system RAM.  This should be tunable and/or the default should be the
lesser of (1/4 system ram) or (expected amount needed (roughly 1/2000th the
snapshot store size?))

What steps will reproduce the problem?
1. Try to build from source on Ubuntu as described by doc/install.html
2.
3.

What is the expected output? What do you see instead?

ddsnap/scripts/genallpatches.sh and ddsnap/scripts/genpatch.sh fail 
because on Ubuntu '/bin/sh' is not bash. First line should 
be '#!/bin/bash' instead of '#!/bin/sh'.

What version of the product are you using? On what operating system?

SVN checkout. Ubuntu Server 7.10.

Please provide any additional information below.

The current init_allocation code does bitmap initialization together for
metadata device and snapshot data device, although metadata and snapshot
may use separate bitmap blocks. This may cause problem when metadata and
snapshot use separate devices and the number of metadata bitmap blocks take
a fractional block.

It appears that an extra call to finish_copyout() crept in to copyout() at
some point, which effectively eliminates the minor optimization of
accumulating consecutive chunks before writing them.

Found via code inspection.

Inside the continuous test, sometimes the old tunbr-generated IP/MAC
allocation in /var/lib/misc/dnsmasq.leases is prematurely deleted and a new
IP address is allocated instead for the MAC address.  The test then fails
to run at all, as the test waits to ping and scp/ssh to the old address and
times out.  This is one of two major sources of sporadic test failures.

This will happen sometimes in the middle of a run, as happened in r1230. 
syslog on the host shows the test host requesting an address, and being
allocated a different one.  I've modified the test harness on the zumabuild
machine to report the contents of dnsmasq.leases and the environment as
each test is starting up.

With the current code, zumastor package installation will fail if the
structure of the ddsnap superblock changes and users have to re-initialize
the whole zumastor volume in order to perform the upgrade. This is quite
inconvenient and unnecessary. Zumastor upgrade should automatically detect
superblock version mismatch and we should allow users to set some
environment variable that tells zumastor upgrade to automatically delete
all of the snapshots and build a superblock based on the new structure.

There are a lot of redundant code on message handling in ddsnap.c and
ddsnapd.c. There are also some bugs on error message handling. In
ddsnapd.c, sometimes we send error-code with error-message but sometimes we
send error-message only, and in ddsnap.c, we don't handle the error-code
cases properly. We should change the code to always send error-code with
error-message and read error-code in ddsnap.c.

We may want to add a version variable in the messages sending between a
ddsnap kernel thread and the user-level ddsnap server. This allows us to
easily detect if a system has a kernel version that mis-matches its
user-level ddsnap program.

What steps will reproduce the problem?
1. Try to build Zumastor trunk or 0.4-branch
2.
3.

What is the expected output? What do you see instead?

It fails because of missing dependencies. The following packages are 
missing from doc/install.html: zlib1g-dev, libpopt-dev, debhelper, 
subversion, devscripts. I would replace the several "aptitude install" 
steps with a single step which installs all the packages at the very 
beginning, before attempting to Zumastor.

What version of the product are you using? On what operating system?

Ubuntu Server 7.10. Tried Zumastor trunk and 0.4 branch.

Please provide any additional information below.

What steps will reproduce the problem?
1. Install zumastor packages on vmware image of ubuntu dapper
2. Reboot

What is the expected output? What do you see instead?
Waiting for root filesystem...
Unable to find root filesystem on /dev/mapper/Ubuntu-root

Please use labels and text to provide additional information.
The initrd isn't able to find the lvm device because the scsi driver is not
supported by the r880 kernel.  The driver is in drivers/message/fusion (not
drivers/scsi)

Volumes containing live application datastores (e.g. SQL databases)
won't have consistent snapshots unless Zumastor provides
hooks so the apps can be quiesced during the snapshot.

See for example
http://www.netapp.com/library/tr/3166.pdf
There are also commercial products that do this for people who
can't script it themselves, e.g.
http://www.falconstor.com/en/solutions/?pg=Products&sb=VMware
So there does seem to be demand.

The ddsnap code assumes that the chunksize parsed during initialization is
always multiplies of 512-byte sector size. But the current ddsnap/zumastor
code does not do any checks on that. Neither have we documented this
limitation. If users use a smaller chunk size, like 64-byte, the code will
break. We should add such checks in ddsnap code and document this
requirement. DanP suggests using 1k as the smallest allowed chunksize for
efficient btree operations.

The replication-snapshots-zumastor.sh usually hangs at the zumastor master
command on the downstream host, and then times out and reports as a test
failure.  Hitting return during this pause though during a UML run of this
test lets it continue, so it appears that /bin/zumastor has paused for user
input at this point.  No reads in /bin/zumastor without stdin redirections
were seen.

We have never tested or documented
multilayer replication
(e.g. A replicates to B replicates to C)
so there are probably some bugs.
Let's document it, test it, and solve any problems this reveals.

"zumastor replicate <volume>" on the target should cause the source to
issue "zumastor replicate <volume> <target>", reproducing any command-line
options there as well.

Users need to be able to access snapshots 
from remote systems via NFS and CIFS.

This needs both code and documentation.

Drake says it's not clear that
we support periodic snapshots
on a system that's also doing
replication.  Let's add test and
doc for how to do this, and fix any problems
that reveals.

Ben posted some very nice patches in late November/early December
that, among other things, gets rid of the symbol exports 
in ddsnap.base.patch.  We should merge them.

Allow the usage of arbitrary snapshot names instead of "volumename(x)".
This might be difficult with the current codebase, but is a very nice
feature to have.

Something roughly like this lets you create a snapshot with any name you
want using NetApp Filers.
snap create volumename snapshotname

ZFS also seems to support something similar:
zfs snapshot volumename@snapshotname
http://docs.sun.com/app/docs/doc/819-5461/6n7ht6qsb?a=view

What steps will reproduce the problem?
1. Run `zumastor forget <volumename>`

What is the expected output? What do you see instead?

`zumastor forget` seems like a poor choice of verb. I tried destroy first,
to find that the verb was forget. I then tried `zumastor forget
<volumename>`, and got the following:
"""
% sudo zumastor forget zumatest       
usage: /bin/zumastor forget volume|target|source
"""

Based on that output, I assumed my syntax was correct, because I was
passing a volume name as an argument to forget. Looking back at the
installation docs, I see that I had to use `zumastor forget volume
<volumename>`.

Perhaps we could make this more clear?


Version Information:
% zumastor --version     
zumastor revision 1310 built on Thu Jan 24 18:25:35 PST 2008 by
build@unassigned
Linux zuma6.example.com 2.6.22.10-zumastor-r1306 #1 SMP PREEMPT Thu Jan 24
15:35:38 PST 2008 i686 GNU/Linux

ddsnap has nice --help and --version options,
but zumastor needs them, too.

Zumastor needs to provide an easy and efficient way for revert origin
volume back to a specified snapshot.

Nautilus is the file manager used by default in Ubuntu.
It provides convenient Python bindings which are
probably sufficient to implement a "Previous Versions"
tab (TimeVault has done something like this, see
https://wiki.ubuntu.com/TimeVault )
Ideally the user story here would be very similar
to the one for the "Previous Versions" tab in
Windows -- it would let you see the list of snapshots
of a file with a single right click, and let you restore
the file from a snapshot with another right click.

zumastor could probably add 'ls' and 'cp' verbs, or something
like them, to make seeing and copying from files
within snapshot easy for people who don't want to
think about where the snapshots are mounted
and who don't use Nautilus.

From Dan Phillips, Nov 22:

Shapor made an insightful observation: the only real reason for the
ddsnap snapshot auto-squashing feature to exist at all is so that a
device mapper ddsnap snapshot device may remain in existence in an
invalid state (still open by an application program perhaps) while the
underlying snapshot store data has been automatically recovered.  Well,
we now have the code to accomplish this merely by deleting the snapshot
in question, there is no need to leave the snapshot recorded in the
ddsnap superblock as "exists but is invalid".

We may entirely replace the notion of snapshot squashing by a request
that the server sends to the snapshot client, to put the client into an
error state and no longer reference the snapshot, thus allowing the
server to delete the snapshot using the normal delete mechanism.

The only new, interesting bit here is that the server has to defer any
write requests that were in progress until the snapshot client confirms
the invalidation request.  This is rather similar to the
defer-on-read-lock code that already exists, so there is not much new
to code, but all this is sensitive and has to be carefully tested.  
Races that need to be taken care of are slightly different.

Maybe make this simplification in 0.6 or later.  It is always nice to
throw out a significant number of lines of code. 

Admins want to zumastor-ize /home, they don't
want to make /home a symlink to /var/run/zumastor/mounts/bork/bork/bork

As described in
http://zumastor.googlecode.com/svn/trunk/doc/Cleanup_roadmap.txt

The large-volume copy test stopped during the weekend when it was nearly
finished. The ddsnap server log shows the following error
message:
Sat Jan 26 21:17:04 2008: [5533] probe: Failed assertion "((struct eleaf
*)nodebuf->data)->magic == 0x1eaf"

I found some suspicious error message in the server log that happened about
10 hours before the fatal failure, which may be the cause of the problem.
Sat Jan 26 10:59:53 2008: [5533] set_buffer_dirty_check: number of dirty
buffers 37 is too large for journal 32

I tried to restart zumastor after the failure, but 'ddsnap server' always
exited when it tried to load the superblock from disk. Running under gdb
shows that the recorded number of snapshots is some invalid
large value, so 'ddsnap server' crashes when it tries to read the list of
snapshots. So somehow the superblock recorded on disk is also corrupted
during the failure.

When a working directory is modified, smoke-etch will look for
linux-i386-nnn rather than linux-i386-nnnM, which is built by the build
script.  If the user build an unmodified UML for a particular revision, the
failure won't be apparent - the old UML kernel will be run instead.  Causes
both test failures and fails to allow testing of the kernel.

Trivial patch en route to use the same version number generator as
buildcurrent.sh.

Several things have gone wrong with the UML testing.

On dapper, the uppercase M was a problem for the kernel build in
svnversion.  Downcasing it is not yet fully successful, and the harness is
still malfunctioning at the moment.

Instances boot, but are not getting IP addresses.  The same userspace and
UML kernel, when booted manually using the same cmdline options under tunbr
*are* working.

The changes above have also broken installs under qemu, though this looks
like simple typos.

Most file servers seem to run
either on a proprietary operating
system, on proprietary hardware, or
use proprietary software.

Linux should be capable of serving most
file server needs out of the box.

So what's keeping people from using Linux for file servers?
For one thing, Linux's native LVM snapshots
suffer from performance and stability problems.
For another, Linux provides no really good way to replicate
disk volumes (rsync is good, but not good enough for some users).

Let's fix at least those issues.

Zumastor should provide an easy way to resize a volume on top of it. This
is an important feature that many customer expect.

zumastor is currently unable to support xfs or reiser4 as they do not have
the errors mount option, and this is specified in the zumastor script.

There is another request for user-specified mount options, which would also
permit per-filesystem options.

Right now bidirectional root authentication is required between the master
and slave.  To keep backups secure even when the origin has been
compromised, this requirement should be eliminated.

sed -i '/^172.23.32.153 .*$/d' /home/zbuild/.ssh/known_hosts
sed: couldn't edit /home/zbuild/.ssh/known_hosts: not a regular file
+ true

/home/zbuild/zuma-dapper-i386.sh: line 227: unexpected EOF while looking
for matching `"'

Cleaning out more of the install script to try to lose this error or pin it
down.