support dynamic linkers about usercorn HOT 59 CLOSED

probably needs more syscalls

needs reading the dynamic linker section

dyld is WEIRD

x86_64 linux test currently fails to load symbols - it jumps to 0 from PLT

from usercorn.

currently static and dynamically linked glibc all crash on load for various reasons. haven't tried any other libc implementations, and osx dynamic linking isn't supported yet either.

from usercorn.

fixed glibc crashes (was bad interp entry point)

from usercorn.

fixed incorrect auxv for musl and glibc.

musl crashes in decode_vec
glibc is jumping to a random unmapped address

from usercorn.

glibc is crashing at elf_get_dynamic_info()

oh, I need to point to the program header inside the loaded section, not copy my own phdr in...

from usercorn.

@lunixbochs can you share your code to build x86.darwin.macho?

from usercorn.

It's just the test binary from https://github.com/lunixbochs/lib43 - if you want to build the 32-bit version specifically, do cmake -DCPU=i386

I'm assuming it works because lib43 doesn't care about macho auxv :)

from usercorn.

osx auxv equivalent is supported on x86_64 now. the primary blocker is now unimplemented mach syscalls

from usercorn.

This is what happens on i386 after trying to run a dynamically linked binary:

Inconsistency detected by ld.so: rtld.c: 1141: dl_main: Assertion `_rtld_local._dl_rtld_map.l_libname' failed!`

from usercorn.

That's a regression. I'm not sure why it's happening yet. Wouldn't be surprised if it's a problem in auxv.

If you want to crack open the glibc source and take a look, narrowing this down would be helpful.

from usercorn.

You meant this?

http://code.woboq.org/userspace/glibc/elf/rtld.c.html#1128

from usercorn.

from usercorn.

Would any of usercorn's tracing options be of any help here?

from usercorn.

Yeah. -trace should enable all of the useful tracing options. I don't recommend posting entire traces as they can be quite large (and I see the same error when I run one)

from usercorn.

No worries - I would've attached a file if you needed a trace. :)

-original message-
Subject: Re: [usercorn] support dynamic linkers (#21)
From: Ryan Hileman [email protected]
Date: 20.11.2015 22:29

Yeah. -trace should enable all of the useful tracing options. I don't recommend posting entire traces as they can be quite large (and I see the same error when I run one)

Reply to this email directly or view it on GitHub:
#21 (comment)

from usercorn.

I can offer this backtrace from gdb:

(gdb) catch syscall exit
(gdb) catch syscall exit_group
(gdb) run
Starting program: /home/petevine/unpacked/usercorn-unstable/usercorn /usr/bin/ls
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Inconsistency detected by ld.so: rtld.c: 1141: dl_main: Assertion `_rtld_local._dl_rtld_map.l_libname' failed!
[New LWP 2387]
[New Thread 0xb317fb40 (LWP 2386)]
[New Thread 0xb39bfb40 (LWP 2385)]
[Switching to Thread 0xb1608b40 (LWP 2387)]

Catchpoint 4 (call to syscall exit), 0xb3c9c729 in start_thread ()
   from /lib/libpthread.so.0
(gdb) bt
#0  0xb3c9c729 in start_thread () from /lib/libpthread.so.0
#1  0xb3bf758e in clone () from /lib/libc.so.6
(gdb) c
Continuing.
[Thread 0xb1608b40 (LWP 2387) exited]
[Switching to Thread 0xb3ab0700 (LWP 2384)]

Catchpoint 5 (call to syscall exit_group), 0xb7fd9b30 in __kernel_vsyscall ()
(gdb) bt
#0  0xb7fd9b30 in __kernel_vsyscall ()
#1  0x0811701f in syscall.Syscall ()
    at /home/petevine/go/src/syscall/asm_linux_386.s:23
#2  0x08114d49 in syscall.Exit (code=127)
    at /home/petevine/go/src/syscall/zsyscall_linux_386.go:376
#3  0x080b5c6f in os.Exit (code=127) at /home/petevine/go/src/os/proc.go:54
#4  0x0804b8db in main.main ()
    at /home/petevine/go/src/github.com/lunixbochs/usercorn/go/usercorn/cli.go:85

from usercorn.

gdb isn't interesting here because it's attached to Usercorn (which I know very well) and not the target.

Once I have #10 we'll be able to attach GDB to the target, but that shouldn't be necessary here (as Usercorn should be able to provide a symbolicated traceback if the binary has symbols). It might not work on dynamic symbols? (#79)

from usercorn.

okay, looks like phnum is zero for some reason, which makes phdr impossible to parse.
edit: was a regression, is fixed in 089b6fa - back to fixing TLS and weirdness

from usercorn.

Could you explain what's going on here? (I chose wc cause it seems to be working in busybox)

./usercorn /bin64/ldd /bin64/wc --help

panic: Could not identify file magic.

goroutine 1 [running]:
main.main()
        /home/petevine/go/src/github.com/lunixbochs/usercorn/go/usercorn/cli.go:55 +0x636

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /home/petevine/go/src/runtime/asm_386.s:1662 +0x1

or
./usercorn /bin64/wc --help

invalid read: @0x8, 0x8 = 0x0
Registers:
   fs 0x0000000000000000   rbx 0x000000000012a000   rsi 0x0000000000000000   r11 0x00000000607fe620 
   gs 0x0000000000000000   rcx 0x000000000040108d    r8 0x000000000012a000   r12 0x000000000012a000 
  rax 0x0000000000000000   rdi 0x00000000607fed5d    r9 0x0000000000000001   r13 0x0000000000000000 
  rbp 0x00000000607fe770   rdx 0x000000000d696913   r10 0x000000000101cdc1   r14 0x000000000d696913 
                                                                             r15 0x000000000040108d 

Thx!

from usercorn.

You can't run ldd under usercorn because it's a shell script that basically does LD_TRACE_LOADED_LIBRARIES=1 /bin64/wc --help

In the other one, that's the bug this issue is about. Look at the stacktrace after -etrace to see where it's breaking.

from usercorn.

Oh, the --help was a silly paste mistake from trying to run the binary; provided the bug wasn't there should invoking LD_TRACE_LOADED_LIBRARIES=1 /bin64/wc under usercorn work at present?
I'll be adding the trace later.

from usercorn.

Oh, it's actually LD_TRACE_LOADED_OBJECTS, and that doesn't seem to work because it's currently breaking before link finishes. LD_DEBUG=all works for me.

This is my stacktrace:

0x100fd10 match_symbol (dl-version.c:76)
0x10100f7 _dl_check_map_versions+0x27 (dl-version.c:191)
0x10100d0 _dl_check_map_versions (dl-version.c:174)
0x1010562 _dl_check_all_versions+0x12 (dl-version.c:384)
0x1010550 _dl_check_all_versions (dl-version.c:380)
0x1004790 version_check_doit
0x100eb10 _dl_receive_error (dl-error.c:205)
0x100173a dl_main+0xca
0x1001670 dl_main
0x1015c42 _dl_sysdep_start+0x42 (dl-sysdep.c:111)
0x1015c00 _dl_sysdep_start (dl-sysdep.c:86)
0x1004825 _dl_start+0x55
0x10047d0 _dl_start
0x1001180 _start

from usercorn.

Yeah, that's the trace. How do you pass vars of the guest environment to usercorn?

from usercorn.

Environment is inherited from host. It ends up on the stack above argv.

from usercorn.

I asked because on my 32-bit system all I'm getting with LD_DEBUG=all ./usercorn /bin64/wc is info about the host system.

from usercorn.

Oh, ld.so might be unsetting that env var. I'm testing on OS X so my host linker won't do things like that.

from usercorn.

Unless this tiny part is what I was after?

file=libc.so.6 [0];  needed by /bin64/wc [0]
     23137:     find library=libc.so.6 [0]; searching
     23137:      search cache=/etc/ld.so.cache
     23137:      search path=/lib64/tls/x86_64:/lib64/tls:/lib64/x86_64:/lib64:/usr/lib64/tls/x86_64:/usr/lib64/tls:/usr/lib64/x86_64:/usr/lib64           (system search path)
     23137:       trying file=/lib64/tls/x86_64/libc.so.6
     23137:
     23137:     file=libc.so.6 [0];  generating link map
     23137:       dynamic: 0x00000000009ceb60  base: 0x000000000060a000   size: 0x00000000003caec0
     23137:         entry: 0x000000000062a990  phdr: 0x000000000060a040  phnum:                 10
     23137:
invalid read: @0x8, 0x8 = 0x0

from usercorn.

This is my output: https://bochs.info/p/njj89z

from usercorn.

Is the usercorn env= part the answer to my original question or just the debug output?

from usercorn.

~/p/usercorn is my shell path. It's more obvious in a screenshot because it's colored.

from usercorn.

In that case I think a feature like this would be a good idea.
I got tons of debug info from usercorn (which honoured the variable) whereas I was merely interested in some info about the guest environment.

from usercorn.

For reference, qemu-user takes these arguments. What you're proposing is the -E / -U interface. This will be pretty easy, so I'm adding issue #106 for it.

Argument      Env-variable      Description
-h                              print this help
-g port       QEMU_GDB          wait gdb connection to 'port'
-L path       QEMU_LD_PREFIX    set the elf interpreter prefix to 'path'
-s size       QEMU_STACK_SIZE   set the stack size to 'size' bytes
-cpu model    QEMU_CPU          select CPU (-cpu help for list)
-E var=value  QEMU_SET_ENV      sets targets environment variable (see below)
-U var        QEMU_UNSET_ENV    unsets targets environment variable (see below)
-0 argv0      QEMU_ARGV0        forces target process argv[0] to be 'argv0'
-r uname      QEMU_UNAME        set qemu uname release string to 'uname'
-B address    QEMU_GUEST_BASE   set guest_base address to 'address'
-R size       QEMU_RESERVED_VA  reserve 'size' bytes for guest virtual address space
-d item[,...] QEMU_LOG          enable logging of specified items (use '-d help' for a list of items)
-D logfile    QEMU_LOG_FILENAME write logs to 'logfile' (default stderr)
-p pagesize   QEMU_PAGESIZE     set the host page size to 'pagesize'
-singlestep   QEMU_SINGLESTEP   run in singlestep mode
-strace       QEMU_STRACE       log system calls
-version      QEMU_VERSION      display version information and exit

from usercorn.

From the changelog:

elf/dl-version.c (match_symbol): Use D_PTR to access relocated entries in l_info.

and the source:

const char *strtab = (const void *) D_PTR (map, l_info[DT_STRTAB]);

So l_info is relocated. I did some research and "modern kernels apply relocations to ld.so".

This step of startup should be fixed by #107

from usercorn.

I'd emailed you about this thinking I broke something with a little hack but it wasn't my fault after all:

./usercorn /bin64/wc --help
hi 0
Inconsistency detected by ld.so: ../sysdeps/x86_64/dl-machine.h: 497: elf_machine_rela_relative: Assertion `((reloc->r_info) & 0xffffffff) == 8' failed!

I'm on 32-bit i386 linux.

from usercorn.

What's hi 0? I'm pretty sure the error we're getting on glibc is actually relocation related across the board, so I'm not sure your binary is anything special (besides the fact it prints an assert). If you do think something changed, can you try git bisecting?

from usercorn.

No idea about hi 0 (probably wc is confused and trying to count something in spite of being asked not to).

I don't have that binary any longer and rebuilt from the latest unstable source. That's what you meant, right?

from usercorn.

Can you git bisect to before this message was being printed, and see which usercorn -trace is longer? (fyi, make usercorn is way faster than make because it doesn't try to download deps)

from usercorn.

I can't do that now but bisecting from memory it would have to be this one:

d3f09a2

I'll try to provide an actual trace comparison from before that later.

from usercorn.

Alright, if that's the case can you get me a trace from before and after (well, before and current unstable) when you get a chance?

from usercorn.

Ok, I tried building @ 95806ec but it failed with:

# github.com/lunixbochs/usercorn/go/usercorn
go/usercorn/cli.go:53: cannot use absPrefix (type string) as type *usercorn.Config in argument to usercorn.NewUsercorn
go/usercorn/cli.go:57: corn.Verbose undefined (type *usercorn.Usercorn has no field or method Verbose)
go/usercorn/cli.go:58: corn.TraceSys undefined (type *usercorn.Usercorn has no field or method TraceSys)
go/usercorn/cli.go:59: corn.TraceMem undefined (type *usercorn.Usercorn has no field or method TraceMem)
go/usercorn/cli.go:60: corn.TraceMemBatch undefined (type *usercorn.Usercorn has no field or method TraceMemBatch)
go/usercorn/cli.go:61: corn.TraceReg undefined (type *usercorn.Usercorn has no field or method TraceReg)
go/usercorn/cli.go:62: corn.TraceExec undefined (type *usercorn.Usercorn has no field or method TraceExec)
go/usercorn/cli.go:67: corn.TraceMatchDepth undefined (type *usercorn.Usercorn has no field or method TraceMatchDepth)
go/usercorn/cli.go:69: corn.TraceMatchDepth undefined (type *usercorn.Usercorn has no field or method TraceMatchDepth)
go/usercorn/cli.go:72: corn.TraceMatch undefined (type *usercorn.Usercorn has no field or method TraceMatch)
go/usercorn/cli.go:72: too many errors
make: *** [usercorn] Error 2

Any use going back further?

from usercorn.

For now, just the latest binary's trace:

newer.txt

And an older one's @ 2b80f6a which just reads:

panic: Could not identify file magic.

goroutine 1 [running]:
main.main()
        /home/petevine/go/src/github.com/lunixbochs/usercorn/go/usercorn/cli.go:55 +0x636

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /home/petevine/go/src/runtime/asm_386.s:1662 +0x1

which, going in the opposite direction, was still the case @ d3f09a2

and the latest 3976cd6 brings back the same error.

from usercorn.

woah, my x86_64 dynamic musl-libc test works on the latest master, and my static glibc binary stopped working, so maybe it's fixed and I broke something else

bisect shows this broke in 089b6fa

from usercorn.

It seems the error above was caused by inadvertent truncation of the wc binary - the trace is the same invalid read as in the old one.

from usercorn.

Even though every trace shows:

Inconsistency detected by ld.so: do-rel.h: 116: elf_dynamic_do_Rela: Assertion `map->l_info[(34 + 0 + (0x6fffffff - (0x6ffffff0)))] != ((void *)0)' failed!

some binaries get stuff like this:

$ ./usercorn /bin64/ls
/bin64/ls: error while loading shared libraries: tls/x86_64/libcap.so.2: file too short

Is it about symlink handling?

from usercorn.

Symlinks are handled. I think there's actually a linux-specific bug. It's way more reliable on OS X.

from usercorn.

You probably mean darwin host; anyway could you point me to a minimal darwin lib/bin package for client testing? I have never tried the -prefix switch and a different kernel yet.

from usercorn.

maybe puredarwin http://www.puredarwin.org/downloads/

from usercorn.

I tried mounting the downloaded vmdk image with qemu-nbd, but all I got eventually was an hfsplus: invalid secondary volume header error. Maybe you could email me just a few essential libs and a test binary?

from usercorn.

I'm not sending you OS X binaries, and I don't have any other xnu/darwin setup right now.

from usercorn.

Here's a tarball of the HFS rootfs from PureDarwinNano_20091226.tar.xz: https://bochs.info/~aegis/PureDarwin.tar.gz

from usercorn.

I'd already tried that too - the mounted iso image contains just a few files for booting from cd. How's the actual data supposed to be stored?

from usercorn.

Huh? This is a working root filesystem, with dynamic binaries, dyld, libraries, etc. Should be more than sufficient for basic dynamic link tests.

from usercorn.

Yeah, sorry, lack of concentration! I'd already had a file named PureDarwinNano_20091226.tar.xz which you mentioned above unfortunately :)

Anyway, thanks a lot and getting back on topic:

$ ./usercorn -prefix PureDarwin/ PureDarwin/bin/ls -l


+ block @0x8fe1fbb8
                        0x8fe1fbb8:       5a pop edx
                        0x8fe1fbb9:     89e1 mov ecx, esp         edx 0x8fe1d33a 
                        0x8fe1fbbb:     0f34 sysenter             ecx 0x607fee2c 

                    + block @0x8fe1fbbd
                      0x8fe1fbbd:   0f1f00 nop dword ptr [eax]
                      0x8fe1fbc0: b806000000 mov eax, 6
                      0x8fe1fbc5:     cd82 int 0x82             eax 0x00000006 

panic: Syscall missing: 50331654

goroutine 17 [running, locked to thread]:
github.com/lunixbochs/usercorn/go.(*Usercorn).Syscall(0x106ca2a0, 0x3000006, 0x0, 0x0, 0x10856010, 0x0, 0x106122c0, 0x0, 0x0)
        /home/odroid/go/src/github.com/lunixbochs/usercorn/go/usercorn.go:695 +0x130
github.com/lunixbochs/usercorn/go/arch/x86.DarwinSyscall(0xab29e740, 0x106ca2a0, 0x3)
        /home/odroid/go/src/github.com/lunixbochs/usercorn/go/arch/x86/darwin.go:41 +0xfc
github.com/lunixbochs/usercorn/go/arch/x86.DarwinInterrupt(0xab29e740, 0x106ca2a0, 0x82)
        /home/odroid/go/src/github.com/lunixbochs/usercorn/go/arch/x86/darwin.go:53 +0x80
github.com/lunixbochs/usercorn/go.(*Usercorn).addHooks.func5(0xab29e6b0, 0x106e04d0, 0x82)
        /home/odroid/go/src/github.com/lunixbochs/usercorn/go/usercorn.go:553 +0x84
github.com/unicorn-engine/unicorn/bindings/go/unicorn.hookInterrupt(0x4c0148, 0x82, 0x108460b0)
        /home/odroid/go/src/github.com/unicorn-engine/unicorn/bindings/go/unicorn/hook.go:45 +0x80

And it seems it's still just i386 which might explain the problem.

from usercorn.

// osfmk/i386/machdep_call.c
// DDD: the last two are BSD_CALL instead of CALL...
//#define __NR_thread_get_cthread_self      VG_DARWIN_SYSCALL_CONSTRUCT_MDEP(0)
//#define __NR_thread_set_cthread_self      VG_DARWIN_SYSCALL_CONSTRUCT_MDEP(1)
// 2 is invalid
#define __NR_thread_fast_set_cthread_self VG_DARWIN_SYSCALL_CONSTRUCT_MDEP(3)
//#define __NR_thread_set_user_ldt          VG_DARWIN_SYSCALL_CONSTRUCT_MDEP(4)
//#define __NR_i386_set_ldt                 VG_DARWIN_SYSCALL_CONSTRUCT_MDEP(5)
//#define __NR_i386_get_ldt                 VG_DARWIN_SYSCALL_CONSTRUCT_MDEP(6)

So this is i386_get_ldt. I'll need to update the syscall table in ghostrace with these (I don't have any machine-dependent syscalls in there atm)

LDT/GDT support on x86_32 aren't done yet, but I think if you want a 64-bit darwin root you need to build it yourself or find a mac.

from usercorn.

Ok, I'll explore the former option :)

from usercorn.

http://web.mit.edu/darwin/src/modules/xnu/osfmk/man/i386_get_ldt.html
http://web.mit.edu/darwin/src/modules/xnu/osfmk/man/i386_set_ldt.html

There's a Unicorn register LDTR which points at the LDT memory address. I'm actually not sure what the LDT does in kernel mode (as I'm not doing a usermode transition). A workaround for now would be to actually set a GDT.

There's experimental code in x86/linux.go:gdtWrite() to write GDT entries, but I don't remember if the upstream GDT problems were fixed.

To read the userspace segment descriptors, you can do s := u.StrucAt(addr), then s.Unpack(&descriptor), where descriptor is a struct type like...

type Descriptor struct {
    Addr uint32
    Size uint32
}

func() DoStuff() {
    var desc Descriptor
    s := u.StrucAt(addr)
    for i := 0; i < count; i++ {
        err := s.Unpack(&desc)
        if err != nil {
            return UINT64_MAX
        }
    }
}

You can do the opposite with s.Pack(&desc) to write descriptors back to guest memory.

from usercorn.

gdtWrite is actually working now, so maybe PureDarwin could be supported with this.

from usercorn.

Thanks, I've just confirmed busybox-i686 is fully functional.

from usercorn.

Closing this. Will break out a couple of remaining tasks.

from usercorn.