berolinux / xar Goto Github PK

One of the strengths of the xar format is its metadata capabilities.
However, it's rather difficult to expose particular pieces of metadata at
whim. (For instance, backup software would want the checksums for arbitrary
files.) A quick hack is below, enough to allow me to get metadata with a
bash util designed to keep archival backups.

Right now, when you archive a file, it will print out the file name once
that file has successfully been added to the archive.  Instead, it would be
much more useful for debugging purposes to print the filename before it is
archived.

[deleted issue]

Not so much a bug as a note to self.  The ._ file handling needs more exercise.

Specifying lists of files to include in an archive, exclude from an
archive, or skip compression of, can be problematic if these lists are
large as currently they have to go on the command line.

Options which match the functionality of tar's --files-from &
--exclude-from would be useful, plus an option to allow the same for
compression. 

Add an option with file extensions to use --compression=none on,
such as .gz or .bzip2 or uncompressable files like .mp3 or .jpg

The zip command uses the colon character as file suffix delimiter,
such as: "zip -rn .Z:.zip:.tiff:.gif:.snd  foo foo" for instance

[deleted issue]

in io.c (rev 188), line 420:

    bsize = def_bsize;
    inbuf = malloc(bsize);
    if( !inbuf ) {

def_bsize is not longer initialized (du to stream patch changed) so bsize is 
initialized with 
garbage.

I think you should change the line 395:
bsize = get_rsize(x);

into 
def_bsize = get_rsize(x);

as def_bsize is reused later in this function and is expected to contain the 
archive rsize default 
value.

Each archive should have some self descriptive entries, such as the time it
was created, and possibly a uuid.

When extracting an archive that contains hardlinks onto a filesystem that does 
not support 
hardlinks, the file should be duplicated.

The current behavior is to error extracting the hardlink, and it is not created.

archiving EAs with non-UTF8 characters in the name can cause a corrupted toc

We should be treating the EA names the same way we treat file names.  The
file name encoding code should be broken out and be modular for this purpose.

xar should use strmode(3) in xar_get_mode().  strmode() is not available on
linux or many other platforms, but it is available on Mac OS X and FreeBSD.
 For systems that don't have strmode(), a local version should be packaged
like asprintf().

xar should support adler32, crc32, and crc64 checksums.
In the case of adler32 and gzip compression is used, the adler32 for
uncompressed data is computed for "free" by zlib and is the last 4 bytes of
compressed data.

This is for systems with low compute abilities.

That's not really an error.
It just would be smarter if xar_extract_tobuffersz() did something like this:

int err = xar_arcmod_extract(x,f,NULL,*buffer,*size);
if (err != 0) {
  *size = 0;
  free(buffer);
  *buffer = NULL;
}
return err;

It would remove the need to to it in each caller.

darwinattr.c doesn't call xar_check_prop() to determine if EAs should be
archived or not.

What steps will reproduce the problem?
1. Create an archive, include file properites (esp. EAs such as resource forks 
on Mac OS X HFS(+))
2. Extract archives, desiring to not have said resource forks
3. There is no step 3

What is the expected output? What do you see instead?
Resource forks are extracted, want to be able to extract files without the 
resource forks

What version of the product are you using? On what operating system?


Please provide any additional information below.
--prop-include and --prop-exclude are the flags for archival, can they be used 
for extraction as 
well?

in io.c (rev188)
in the function xar_attrcopy_to_heap(), around the line 295

    tmpp = xar_prop_pget(p, "archived-checksum");
    if( tmpp )
        csum = xar_prop_getvalue(tmpp);
    tmpf = xmlHashLookup(XAR(x)->csum_hash, BAD_CAST(csum));

if tmpp is null, csum is never initialized, and the call to xmlHashLookup may 
crash.

You should maybe initialize csum to null, and before the call to xmlHashLookup 
(line 298), check 
for csum value, and if null, return an error, or do whatever you want.

Extracting the attached xar files causes a crash.  It doesn't appear to be
a security issue.

xar 1.4 and 1.5.1 on Leopard9A557, Macbook Pro

xar -xvf 00001537.xar

Process:         xar [316]
Path:            /usr/bin/xar
Identifier:      xar
Version:         ??? (???)
Code Type:       X86 (Native)
Parent Process:  bash [161]

Date/Time:       2007-09-20 13:19:00.566 -0700
OS Version:      Mac OS X 10.5 (9A557)
Report Version:  6

Exception Type:  EXC_BAD_ACCESS (SIGBUS)
Exception Codes: KERN_PROTECTION_FAILURE at 0x0000000000000000
Crashed Thread:  0

Thread 0 Crashed:
0   libSystem.B.dylib               0x94f488b0 strlen + 16
1   libxar.1.dylib                  0x0001b6f0 xar_get_path + 57
2   xar                             0x00002f26 0x1000 + 7974
3   xar                             0x0000169a 0x1000 + 1690

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffff  ebx: 0x0001b6c2  ecx: 0x00000000  edx: 0x00000000
  edi: 0x001051c0  esi: 0x001051c0  ebp: 0xbfffee48  esp: 0xbfffee2c
   ss: 0x0000001f  efl: 0x00010286  eip: 0x94f488b0   cs: 0x00000017
   ds: 0x0000001f   es: 0x0000001f   fs: 0x00000000   gs: 0x00000037
  cr2: 0x00000000

Binary Images:
    0x1000 -     0x3fff +xar ??? (???) /usr/bin/xar
   0x12000 -    0x1cfff  libxar.1.dylib ??? (???) /usr/lib/libxar.1.dylib
0x8fe00000 - 0x8fe2d883  dyld 95.1 (???) <cb0c4a65f873d94eb99efb94d7f69baf>
/usr/lib/dyld
0x90fdd000 - 0x90fe4fe9  libgcc_s.1.dylib ??? (???)
<8fce590327ad8ef6d56bfc797c40e435> /usr/lib/libgcc_s.1.dylib
0x916e5000 - 0x916f1fff  libbz2.1.0.dylib ??? (???)
<453492858aec09b3751a8281eef111ab> /usr/lib/libbz2.1.0.dylib
0x9438a000 - 0x943e7ffb  libstdc++.6.dylib ??? (???)
<d6534a13f6b56396aef884041dc53d3a> /usr/lib/libstdc++.6.dylib
0x943e8000 - 0x944c9ff7  libxml2.2.dylib ??? (???)
<f42cfd9dcb8c46c80f8a516ab46e5ec6> /usr/lib/libxml2.2.dylib
0x94e06000 - 0x94f3eff7  libicucore.A.dylib ??? (???)
<ca5f84815dc176a281e44321715d18c7> /usr/lib/libicucore.A.dylib
0x94f3f000 - 0x95096fe3  libSystem.B.dylib ??? (???)
<5aff1930a0bf4a96eb2c52b93a489092> /usr/lib/libSystem.B.dylib
0x9535f000 - 0x95411ffb  libcrypto.0.9.7.dylib ??? (???)
<6007bd3a529aaf328f0bcae022e0130f> /usr/lib/libcrypto.0.9.7.dylib
0x95fe7000 - 0x95ff5ffd  libz.1.dylib ??? (???)
<507fbeee432f55d0b9da49d08f2b9942> /usr/lib/libz.1.dylib
0x96113000 - 0x96114fef  libmathCommon.A.dylib ??? (???)
/usr/lib/system/libmathCommon.A.dylib
0xffff0000 - 0xffff1780  libSystem.B.dylib ??? (???) /usr/lib/libSystem.B.dylib

src/xar.c: In function 'dumptoc':
src/xar.c:455: warning: implicit declaration of function 'xar_serialize'

This is because xar_serialize isn't in xar.h

It's not clear it should be, but xar(1) uses it to serialize and dump a toc
for an archive.

The latest SVN snapshot of xar includes a xar_extract_tobuffer() function, 
which is very nice. 
However, if a very large file, such as one that was 1 GB in size, is stored 
inside a xar archive, it 
could be impractical to extract the entire file to a buffer, all at once. For 
this reason, it would be 
nice if there were a way to read a fixed number of bytes at once into a buffer, 
looping until one had 
read the entire file. This would make it much easier to pipe the data in a file 
to another task, for 
example.

Right now, xar(1) only allows addition of only 1 subdoc, however the
library interface allows for any number of subdocs to be added.  The xar(1)
tool should expose this.

What steps will reproduce the problem?
1. xar --help

What is the expected output? What do you see instead?

expected:
        --no-compress    POSIX regular expression of files
                              to archive, but not compress.

instead:
        --no-compress    POSIX regular expression of files
                              not to archive, but not compress.


What version of the product are you using? On what operating system?
1.5.2

Please provide any additional information below.
occured on Gentoo Linux, x86_64

What steps will reproduce the problem?
1. Make a new folder
2. Make four new folders inside this folder, and name them "naïve", "resumé", 
"el niño", and 
"bibliothèque"
3. Open the Terminal and cd to this folder
4. xar -cf ../test.xar .
5. xar -tf ../test.xar (notice how the filenames are chopped off)

What is the expected output? What do you see instead?

I expected to see the following output:

$ xar -tf test.xar
.DS_Store
bibliothèque
el niño
naïve
resumé

but instead I saw this:

$ xar -tf test.xar
.DS_Store
bibliothèq
el niño
naïv
resume

Only "el niño" survived. The rest had their names chopped off. Extracting the 
files from the 
archive gives the same results - you get the files back, with chopped filenames.

It gets worse if you use filenames with more unusual composed characters, like 
for example 
"Antonín Dvořák". In this case, the file will not even extract from the 
archive. This is what 
happens:

$ xar -tf test2.xar 
Antonín Dvořa?

$ xar -xf test2.xar
Error while extracting archive:(Antonín Dvořa?): stat: Could not create 
directory (Invalid 
argument) - ignored

If using the xar library with a C app, using xar_prop_get() with "name" as the 
key gets you the 
following:

"Antoni\314\201n Dvor\314\214a\314"

whereas the correct string should be:

"Antoni\314\201n Dvor\314\214a\314\201k"

Clearly, this is a problem as it will make it impossible to reliably store 
files with special 
characters in their names within a xar archive until it is fixed.

What version of the product are you using? On what operating system?

This occurs with xar 1.4 as well as with the latest snapshot from SVN.

Xar doesn't preserve the ownership information of symbolic links.  You can 
verify this by downloading backup bouncer (http://www.n8gray.org/code/
backup-bouncer) and running the test-suite.  Using ls to get details:

[n8gray@golux backup-bouncer]$ ls -l /Volumes/Src/15-symlink-ownership/
total 32
-rw-r--r--   1 n8gray  staff  14 May  1 16:59 some-file
lrwxr-xr-x   1 www     www    11 May  1 16:59 symlink1@ -> ./some-file
lrwxr-xr-x   1 root    wheel  11 May  1 16:59 symlink2@ -> ./some-file
lrwxr-xr-x   1 n8gray  staff  10 May  1 16:59 symlink3@ -> ./symlink1
[n8gray@golux backup-bouncer]$ ls -l /Volumes/Dst/70-xar/15-symlink-
ownership/
total 32
-rw-r--r--   1 n8gray  staff  14 May  1 16:59 some-file
lrwxr-xr-x   1 root    staff  11 May  2 01:15 symlink1@ -> ./some-file
lrwxr-xr-x   1 root    staff  11 May  2 01:15 symlink2@ -> ./some-file
lrwxr-xr-x   1 root    staff  10 May  2 01:15 symlink3@ -> ./symlink1

Note that symlink ownership can be set using lchown(2) on OS X 10.4 or 
later.  (I think it wasn't there in 10.3 but I could be wrong.)

What steps will reproduce the problem?

imd20:jberry ~/Projects/macports/users/jberry/test % ls
portpkg.xar xpkg
imd20:jberry ~/Projects/macports/users/jberry/test % xar -tf portpkg.xar 
portpkg
portpkg/files
portpkg/Portfile
imd20:jberry ~/Projects/macports/users/jberry/test % xar -xf portpkg.xar portpkg
imd20:jberry ~/Projects/macports/users/jberry/test % ls portpkg
imd20:jberry ~/Projects/macports/users/jberry/test % ls
portpkg     portpkg.xar xpkg
imd20:jberry ~/Projects/macports/users/jberry/test % 

What is the expected output? What do you see instead?
I expect the portpkg directory to have the contents as shown in the -t listing.

What version of the product are you using? On what operating system?
1.4 on Mac OS X 10.4.9

If you set the finder info creation time with /Developer/Tools/SetFile -d, that 
time is not preserved 
upon extraction.

We're preserving everything in the finder info returned by getattrlist(), 
including mod time.  
Something else may have to be done to preserve the different time field.

What steps will reproduce the problem?
1. create a file foo with ACLs
2. xar -c -f tmpfile foo
3. cd somewhere_else
3. xar -x -f tmpfile

or alternately, test with backup-bouncer and notice the failures.

What is the expected output? What do you see instead?
I expect the output of "ls -Plde foo" to be identical in both directories.  
Instead the foo in 
somewhere_else has no ACLs.

What version of the product are you using? On what operating system?
Today's SVN HEAD, Mac OS X 10.5.1.

Please provide any additional information below.
The last time I tested xar was on Tiger in May, and it did not have this issue. 
 You can verify this 
problem using backup-bouncer:  http://www.n8gray.org/code/backup-bouncer/

xar preserves the setuid bit when extracting from an archive.  This has
been considered a minor security issue in other products in the past.   See
for example 
http://nvd.nist.gov/nvd.cfm?cvename=CVE-2005-0602
http://nvd.nist.gov/nvd.cfm?cvename=CVE-2005-2541


If a local attacker can trick a user into extracting an archive, he can
potentially run code as that user(including root).

There should be a nondefault option to preserve the bits.   e.g. unzip -K,
tar -p

Steps to repro with xar 4.5.1 and 4.4 on Leopard 9A557:
 mkdir stuff
 touch stuff/1
 touch stuff/2
 chmod u+sx stuff/1
 chmod g+sx stuff/2
xar -cvf stuff.xar stuff
rm -r stuff
xar -xvf stuff.xar

1 and 2 still have the setuid/setgid bit set.  
sudo xar -xvf stuff.xar
1 and 2 still have the bit set and are owned by root.

Creating a xar archive with a bogus/missing filename for the -s option will 
generate a toc that 
looks malformed: the subdoc element encloses all other content.

For example:

% touch fff
% xar -cf foo.xar fff -s xxx
% xar -f foo.xar --dump-toc=/dev/stdout
<?xml version="1.0" encoding="UTF-8"?>
<xar>
 <subdoc subdoc_name="subdoc">
  <toc>
   <file id="1">
    <name>fff</name>
    <type>file</type>
    <mode>0664</mode>
    <uid>501</uid>
    <user>jberry</user>
    <gid>501</gid>
    <group>jberry</group>
    <atime>2007-04-24T17:11:50Z</atime>
    <mtime>2007-04-24T17:11:50Z</mtime>
    <ctime>2007-04-24T17:11:50Z</ctime>
   </file>
   <checksum style="sha1">
    <size>20</size>
    <offset>0</offset>
   </checksum>
  </toc>
 </subdoc>
 <toc/>
</xar>

xar archives more data than what people really want sometimes.  There should be 
a "distribution" 
mode that only archives "safe" file attributes.  Suggested initial list of 
properties are:
data
uid
gid
name
type

There is no simple way to get the returned buffer size when calling 
xar_extract_tobuffer();

It may be helpfull to do something like this:

int32_t xar_extract_tobuffer(xar_t x, xar_file_t f, char **buffer, size_t 
*size);

or di i miss something?

What steps will reproduce the problem?
1. archive a file with a creation date way in the past
2. restore the file
3. check the creation date

What is the expected output? What do you see instead?

While many archivers (pax and cpio come to mind) also reset the creation date, 
I would hazard 
that this is incorrect behavior. An archive shouldn't change fundamental 
metadata about the file. 
It was, in fact, created earlier.

What version of the product are you using? On what operating system?

The HEAD of SVN. Mac OS X 10.4.8 on a PPC

xar now archives too much information about files, and can result in 
information leakage for 
inattentive archivers.  There should be a way to specify which attributes to 
exclude, or a mode that 
defaults to nothing (basically not archiving the file), and specifies what to 
include.

In this case 'attributes' mean every property of the "file" tag in the toc, 
including atime, mtime, 
individual EAs, ACLs, etc.

The xar_file_t fspath is incorrectly initialized in xar_add_frombuffer (at 
least) when parent is not 
null.

the char*tmp var in never initialized and is then affected to 
XAR_FILE(ret)->fspath.
This cause a "Deallocation of a pointer not malloced" in xar_close() -> 
xar_file_free();

<<<<
xar_file_t xar_add_frombuffer(xar_t x, xar_file_t parent, const char *name, 
char *buffer, size_t 
length) {
    xar_file_t ret;
    char *tmp;
    char idstr[32];
<<<<    
    if( !parent ) {
           ...
    } else {
       ...
        XAR_FILE(ret)->fspath = tmp;
    }
<<<<

What steps will reproduce the problem?
1. Add "spotlight comments" to a file in The Finder
2. Xar that file into a new archive
3. Un-Xar it and "get info". No comments!

What is the expected output? What do you see instead?

I would hope that comments would be archived.

What version of the product are you using? On what operating system?

1.4 on MacOS X 10.4.8

Please provide any additional information below.

For what it's worth, I've found no command-line archiving programs that
maintain Finder comments.

When archiving a device node in linux, the fact that it is a character or
block device node is recorded, but the major/minor number is not recorded.

Additionally, when extracting, a regular zero-length file is created, not a
device node.

What steps will reproduce the problem?
1. Use the xattr utility to set a custom extended attribute (e.g. echo
"Hello world" > testfile.txt; xattr --set testxa testxavalue testfile.txt)

2. Archive the file in Xar (xar -cf testfile.xar testfile.txt)

3. De-archive the file (xar -xf testfile.xar)

4. The custom EA is missing. (xattr --list testfile.txt)

What is the expected output? What do you see instead?

I would expect for the extended attributes to be maintained. They are not.

What version of the product are you using? On what operating system?

1.4 (installed via Fink unstable) on MacOS X 10.4.8

Please provide any additional information below.

Resource forks appear to be maintained properly.

With certain compressors, which level you choose makes a lot of difference
(i.e. more than it does with e.g. gzip). Especially memory usage varies a
lot between different levels, and could be need to change between files.

Adding another option for compression level would make this flexible, where
the level varies from -1 (--fast) to -9 (--best) with the default level
being somewhere inbetween and up to the compressor - usually by passing "-1"

What steps will reproduce the problem?
1. xar -tvf foo.xar foo/bar

What is the expected output?
foo/bar

What do you see instead?
foo/foo
foo/bar
foo/baz


What version of the product are you using? On what operating system?
trunk/tiger

add -z and -j, as shortcuts for --compression=gzip and --compression=bzip2

I would like to see an option implemented, to allow for deletion of a 
(group of) file(s), after archival, in order to reclaim the disk space 
that was previously used by them.

Jesse Peterson reported a problem archiving acls on 10.4.  Below is his
diagnosis from email to xar-devel:
% uname -a
Darwin localhost 8.8.1 Darwin Kernel Version 8.8.1: Mon Sep 25
19:42:00 PDT 2006; root:xnu-792.13.8.obj~1/RELEASE_I386 i386 i386
% cd /
% sudo fsaclctl -p / -e
% touch foo
% chmod +a 'root allow read' foo
% chmod +a 'root allow write' foo
% ls -le foo
-rw-r--r-- + 1 jesse admin  0 Jan 17 09:41 foo
0: user:root allow read,write
% xar -v -c -f foo.xar foo
foo
% xar --dump-toc=- -f foo.xar
% xar --dump-toc=- -f foo.xar
<?xml version="1.0" encoding="UTF-8"?>
<xar>
<toc>
  <checksum style="sha1">
   <size>20</size>
   <offset>0</offset>
  </checksum>
  <file id="1">
   <name>foo</name>
   <type>file</type>
   <mode>0644</mode>
   <uid>1085</uid>
   <user>jesse</user>
   <gid>80</gid>
   <group>admin</group>
   <atime>2007-01-17T17:41:51Z</atime>
   <mtime>2007-01-17T17:41:51Z</mtime>
   <ctime>2007-01-17T17:53:06Z</ctime>
  </file>
</toc>
</xar>

Interestingly I've found that acl_get_entry appears to be returning 0
for my file in xar-1.4/lib/stat.c:130 which would seem to signify that,
well, there is no ACL entry for the file ;). However, acl_get_file
returns non-null which presumably would indicate that there is an ACL
entry on the file.

In:
http://darwinsource.opendarwin.org/10.4.2/file_cmds-116.9/ls/print.c  
on lines ~ 354 and the function printacl() (lines ~ 252 - 318) appears
to being acl_get_file and acl_get_entry in a very similar way. Which of 
course makes it interesting that "ls -le" prints the ACLs yet the above
is true.

(Note: this is a feature request, not an issue per se.)

According to the "Why xar" document:

"Files in xar are individually compressed. This allows for quick extraction
of individual files without the extra disk space requirements and CPU usage
of extracting the entire archive, as compared to a compressed tar archive.
This makes xar useful for quick restores of accidentally deleted or
overwritten files, from a backup archive."

I was looking at an efficient random-access format exactly for this backup
scenario.  I am looking at compressing and encrypting files with GPG when
storing them in the archive.  Right now, using tar, it means tar-ing it
all, than gpg-ing the whole archive.  So to get a file from the archive,
you need to un-gpg-it globally, then use tar to extract the file.  With xar
too, it would mean un-gpg-ing it globally before seeing the contents.

I was thus thinking it could be great to have something like
--compression=pipe and then a --compress-pipe=<command>.  The command would
be fed with the input file and output the compressed file, then stored in
the archive.

In my scenario, it would make it possible to do something like

xar ... --compress=pipe --compress-pipe='gpg -c --output -'

However, problems arise at decompression:

a) User needs to know how it was compressed
   Possible solution: store the 'compress-pipe=<...>' that was used in 
   the xar metadata to give a clue to the user

b) User needs to specify how to uncompress
   Again, at uncompress, use something like
   --compress=pipe --uncompress-pipe='gpg -d -'

   At compression, there could also be an optional 
   --suggested-uncompress-pipe=<> that would store metadata so that the 
   receiving end has a clue.  (But should never be executed at 
   decompression, as it can be a major security flaw...)


Having such a mechanism (most probably a little more refined) would make
xar a perfect choice for random access archives where file are stored
according to user specifications.

I hope these ideas can fuel some discussion.  Thanks.

Difference between usage info and man info for --extract-subdoc=name

usage mentions 
    --extract-subdoc=name Extracts the specified subdocument
                          to a document in cwd named <name>.xml

man doesn't mention this command.


xar 1.4, Mac OS X 10.5 9A581

There should be an option to not overwrite existing files, similar to tar -k or 
unzip -n

xar should record the inode # and device # for files.

What steps will reproduce the problem?
1. Compile trunk (r101) code on NetBSD 3.
2. Attempt to archive any file(s).

What is the expected output? What do you see instead?
Expect: file archival.
See: "Error adding file <[path/]filename>"

What version of the product are you using? On what operating system?
xar from Subversion trunk (revision 101).

% uname -a
NetBSD localhost 3.0 NetBSD 3.0 (GENERIC) #0: Mon Dec 19 01:04:02 UTC 2005
 [email protected]:/home/builds/ab/netbsd-3-0-RELEASE/i386/200512182024Z-obj/home/builds/ab/netbsd-3-0-RELEASE/src/sys/arch/i386/compile/GENERIC
i386

Please provide any additional information below.
It would appear that using extattr_list_link() using
EXTATTR_NAMESPACE_SYSTEM failes with errno as EOPNOTSUPP.
See attached patch for a (probably unacceptable) resolution.

The xar_subdoc_new() function allows multiple subdocs to be added with the
same name.  This probably shouldn't be allowed.

[deleted issue]

INSTALL notes to use ./configure when really it should be ./autogen.sh