When writing a large file, the function fails with the error message of overusing instruction limit.

The writing performance needs to the optimized to be able to create a file of 10Mb within the time limit.

Hey! I've been incorporating the ic-wasm-polyfill into Kybra (Python CDK), and I just ran into a problem here: https://github.com/wasm-forge/stable-fs/blob/main/src/storage/stable.rs#L18

We have a Python StableBTreeMap exposed and by default our examples start their memory ids from 0. I think this is very simple and nice, to allow people to be able to access these early memory ids, as it is very intuitive. But because this library takes those memory ids, no other stable structures in user code can use them.

In Kybra as we've used stable structures, we've started at 254 and started to work down. It would be nice if this library could do something similar.

Hi! As part of our efforts to verify that crates.io releases are reproducible ¹, we have discovered that one or more crates hosted in this repo are missing the repository field ² in Cargo.toml.

The repository field helps users and automated tools find the link to the git repository hosting the source code. While not required by crates.io, we believe it should always be set.

We suggest that the repository field be configured for every crate published to crates.io. Thank you very much.

I'm getting a strange error after attempting to open, read, and/or write many files. After about 20-25 writes I start getting the following error

Panicked at 'called `Result::unwrap()` on an `Err` value: Io(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })', stable-fs-0.1.11/src/storage/types.rs:85:51

Even after doing a redeploy I am still unable to open any additional files after I hit what feels like some sort of limit.

Here is a simple canister that illustrates the problem.

After running dfx cansiter call canister write_25_files once it cannot be called again without this error popping up. Nor can read_25_file be called without this same error.

use std::{
    cell::RefCell,
    io::{self, Read, Write},
};

use ic_stable_structures::{
    memory_manager::{MemoryId, MemoryManager},
    DefaultMemoryImpl,
};

thread_local! {
    static MEMORY_MANAGER_REF_CELL: RefCell<MemoryManager<DefaultMemoryImpl>> = RefCell::new(MemoryManager::init(DefaultMemoryImpl::default()));
}

#[ic_cdk::init]
fn my_init() {
    let polyfill_memory =
        MEMORY_MANAGER_REF_CELL.with(|manager| manager.borrow().get(MemoryId::new(0)));
    ic_wasi_polyfill::init_with_memory(&[], &[], polyfill_memory);
}

#[ic_cdk::post_upgrade]
fn my_post_upgrade() {
    let polyfill_memory =
        MEMORY_MANAGER_REF_CELL.with(|thing| thing.borrow().get(MemoryId::new(0)));
    ic_wasi_polyfill::init_with_memory(&[], &[], polyfill_memory);
}

#[ic_cdk::update]
pub fn write_25_files() {
    let dir_name = "auto";
    let file_count: u8 = 25;
    std::fs::create_dir_all(dir_name).unwrap();
    for i in 0..file_count {
        let path = format!("{}/my_file_{}.txt", dir_name, i);
        ic_cdk::println!("Writing to {}", path);

        let mut file = std::fs::OpenOptions::new()
            .create(true)
            .write(true)
            .open(&path)
            .unwrap();
        let write_buff = [i; SIZE_OF_FILE];
        file.write(&write_buff).unwrap();
        drop(file);
    }
}

#[ic_cdk::update]
pub fn read_25_files() {
    let dir_name = "auto";
    let file_count: u8 = 25;
    std::fs::create_dir_all(dir_name).unwrap();
    for i in 0..file_count {
        let path = format!("{}/my_file_{}.txt", dir_name, i);
        ic_cdk::println!("Writing to {}", path);

        let mut file = std::fs::OpenOptions::new()
            .create(true)
            .read(true)
            .open(&path)
            .unwrap();
        let mut read_buf = [1, 1, 1];
        let num_bytes = file.read(&mut read_buf).unwrap();
        ic_cdk::println!("Read {} bytes {}", num_bytes, path);
    }
}

Idea:
The new ic-polyfill function mounts some virtual memory in the file system. Whenever a user tries to read or write in this file, the read and write operations work directly on the virtual memory as if it was one of the files on the filesystem. Direct memory access is much faster because there is less inherent data copying, and no extra overhead related to the use of complex data structures such as btreemap. This creates a "fast file" at the cost of using one dedicated virtual memory storages. The user can mount several virtual memories, however, the number of virtual memories is limited.

The feature can have several purposes:

1. Fast file for reading and writing can be helpful for applications like sqlite or some application related to intensive reading and writing to some particular file.
2. A user can see their own virtual memory from the perspective of the file system or the wasi file system calls.
3. Comparison with the fast file can be a sort of a baseline for estimating performance of work with the usual files.