a2liu / heaparray-rs Goto Github PK

All of the reads in AtomicPtrArray are only atomic on the reading of the pointer itself; if an interrupt occurs right after reading the pointer but right before dereferencing it, another thread can do a compare_and_swap and then deallocate memory that the other thread is currently pointing to. This is like a chicken and the egg problem honestly. Maybe CAS operations on array pointers are frivolous? Or at least we need a better way to approach it.

The semantics of the Rust AtomicPtr struct's API don't play well with the custom destructors that the array code needs to invoke. I need to figure out how to fix that.

Hi,

1. I need to allocate 2GB mem on the heap. For my usecase it doesn't need to be initialized. Initializing it with
   let mut buf = HeapArray::<u8, ()>::new(fsize as usize, |_| 0); however takes ages, minutes I mean.

2. I don't know the reason yet, but the following code executed with cargo test test_read_large_file_into_heaparray -- --test-threads=1 --nocapture --ignored never return. It seems to be stuck in an endless loop since a CPU core is fully utilized.

#[test]
#[ignore]
fn test_read_large_file_into_heaparray() -> Result<(), Box<dyn Error + 'static>> {
    let fname = "<my large file>";
    let mut file = File::open(&fname)?;
    let fsize = file.metadata()?.len();
    println!("Init heaparray: {}", fsize);
    let mut buf = HeapArray::<u8, ()>::new(fsize as usize, |_| 0);
    println!("Reading data: {}", fsize);
    let n = file.read(buf.as_slice_mut())?;
    assert_eq!(n as u64, fsize);
    println!("Done reading data: {}", n);
    std::thread::sleep(std::time::Duration::from_secs(10));
    println!("Data: {:?}", &buf.as_slice()[..100]);
    Ok(())
}

The following tests fail with invalid memory references (signal: 11, SIGSEGV: invalid memory reference):

• tests::thin_array_ptr::check_null
• tests::fat_array_ptr::check_null
• tests::array_ref::null_test
• tests::array_ref::ref_counting_test

These all invoke some variant of the is_null method as defined in the BaseArrayRef trait; ultimately, they all call this method defined in heaparray::mem_block:

impl<'a,E,L> MemBlock<'a,E,L> {

    // more code above...
    pub fn is_null(&self) -> bool {
        self as *const Self as usize == Self::NULL
    }
}

My guess is that on release builds, Rust assumes that references to objects cannot be null (or in this case, usize::MAX), and so it optimizes away my null check. Unfortunately, that assumption doesn't hold, so the rust code just blows past all the safety checks and seg-faults!

I think this is due to the number of abstractions in this crate; often the actual logic being called is nested beneath 3 or more layers of indirection; this results in wayyy too many function calls (when at a low optimization level). I think the only legitimate solution is to add #[inline] to the vast majority of the methods, and at the very least to all of the really short ones that act as wrappers.

The compare_and_swap doc test causes the following error:

---- src/api/mod.rs - api::ArcArray (line 12) stdout ----
thread 'src/api/mod.rs - api::ArcArray (line 12)' panicked at 'test executable failed:


', src/librustdoc/test.rs:372:17
stack backtrace:
   0:        0x1138196a3 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h2e1
a54aafa1209de
   1:        0x1138122e2 - std::sys_common::backtrace::_print::h670ed14fc4950210
   2:        0x113815c46 - std::panicking::default_hook::{{closure}}::h5d2e6205d52978de
   3:        0x1138157ae - std::panicking::default_hook::h8e6307de267f5f56
   4:        0x111f2cc91 - rustc::util::common::panic_hook::h073beb5665f10e50
   5:        0x1138164a0 - std::panicking::rust_panic_with_hook::h9bb34120d55c0ca9
   6:        0x113815eec - std::panicking::continue_panic_fmt::h6366fe4aa3f57700
   7:        0x113815e40 - std::panicking::begin_panic_fmt::h36b3a6a184dda489
   8:        0x10f383978 - rustdoc::test::run_test::ha9bf2e69c50c6752
   9:        0x10f652570 - syntax::with_globals::h333ac99b9e98188e
  10:        0x10f5288bd - std::sys_common::backtrace::__rust_begin_short_backtrace::hb115
eb7b14b42e9f
  11:        0x10f40e58a - std::panicking::try::do_call::h1a5027ca9338069a
  12:        0x1138260ee - __rust_maybe_catch_panic
  13:        0x10f4215f6 - <F as alloc::boxed::FnBox<A>>::call_box::h49f33130f2c8acd1
  14:        0x113824ebb - std::sys::unix::thread::Thread::new::thread_start::hf63bd47663b
691b4
  15:     0x7fff629662ea - _pthread_body
  16:     0x7fff62969248 - _pthread_start
query stack during panic:
end of query stack

I'm not sure what's going on here, but hopefully some tinkering around with some of the unsafe code will fix it.

Notes:

• This error isn't consistent, sometimes it happens and sometimes it doesn't
• This is also applicable to the doc test in src::naive_rc::TpArcArray
• Error happens less frequently when testing with cargo test --doc api::ArcArray, and even less frequently with cargo test --doc api::ArcArray -- --test-threads=1