mrcroxx / foyer Goto Github PK

View Code? Open in Web Editor NEW

120.0 5.0 9.0 1.92 MB

Hybrid in-memory and disk cache in Rust

Home Page: https://github.com/foyer-rs/foyer

License: Apache License 2.0

Rust 99.49% Makefile 0.36% Shell 0.15%

hybrid-cache rust cache

foyer's Introduction

foyer

foyer aims to be an efficient and user-friendly hybrid cache lib in Rust.

foyer draws inspiration from Facebook/CacheLib, a highly-regarded hybrid cache library written in C++, and ben-manes/caffeine, a popular Java caching library, among other projects.

However, foyer is more than just a rewrite in Rust effort; it introduces a variety of new and optimized features.

Features

Hybrid Cache: Seamlessly integrates both in-memory and disk-based caching for optimal performance and flexibility.
Plug-and-Play Algorithms: Empowers users with easily replaceable caching algorithms, ensuring adaptability to diverse use cases.
Fearless Concurrency: Built to handle high concurrency with robust thread-safe mechanisms, guaranteeing reliable performance under heavy loads.
Zero-Copy In-Memory Cache Abstraction: Leveraging Rust's robust type system, the in-memory cache in foyer achieves a better performance with zero-copy abstraction.
User-Friendly Interface: Offers a simple and intuitive API, making cache integration effortless and accessible for developers of all levels.
Out-of-the-Box Observability: Integrate popular observation systems such as Prometheus, Grafana, Opentelemetry, and Jaeger in just ONE line.

Projects Using foyer

Feel free to open a PR and add your projects here:

RisingWave: SQL stream processing, analytics, and management.
Chroma: Embedding database for LLM apps.

Usage

To use foyer in your project, add this line to the dependencies section of Cargo.toml.

foyer = "0.11"

If your project is using the nightly rust toolchain, the nightly feature needs to be enabled.

foyer = { version = "0.11", features = ["nightly"] }

Out-of-the-box In-memory Cache

use foyer::{Cache, CacheBuilder};

fn main() {
    let cache: Cache<String, String> = CacheBuilder::new(16).build();

    let entry = cache.insert("hello".to_string(), "world".to_string());
    let e = cache.get("hello").unwrap();

    assert_eq!(entry.value(), e.value());
}

Easy-to-use Hybrid Cache

use foyer::{DirectFsDeviceOptionsBuilder, HybridCache, HybridCacheBuilder};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let dir = tempfile::tempdir()?;

    let hybrid: HybridCache<u64, String> = HybridCacheBuilder::new()
        .memory(64 * 1024 * 1024)
        .storage()
        .with_device_config(
            DirectFsDeviceOptionsBuilder::new(dir.path())
                .with_capacity(256 * 1024 * 1024)
                .build(),
        )
        .build()
        .await?;

    hybrid.insert(42, "The answer to life, the universe, and everything.".to_string());
    assert_eq!(
        hybrid.get(&42).await?.unwrap().value(),
        "The answer to life, the universe, and everything."
    );

    Ok(())
}

Fully Configured Hybrid Cache

use std::sync::Arc;

use anyhow::Result;
use chrono::Datelike;
use foyer::{
    DirectFsDeviceOptionsBuilder, FifoPicker, HybridCache, HybridCacheBuilder, LruConfig, RateLimitPicker, RecoverMode,
    RuntimeConfig, TokioRuntimeConfig, TombstoneLogConfigBuilder,
};
use tempfile::tempdir;

#[tokio::main]
async fn main() -> Result<()> {
    let dir = tempdir()?;

    let hybrid: HybridCache<u64, String> = HybridCacheBuilder::new()
        .memory(1024)
        .with_shards(4)
        .with_eviction_config(LruConfig {
            high_priority_pool_ratio: 0.1,
        })
        .with_object_pool_capacity(1024)
        .with_hash_builder(ahash::RandomState::default())
        .with_weighter(|_key, value: &String| value.len())
        .storage()
        .with_device_config(
            DirectFsDeviceOptionsBuilder::new(dir.path())
                .with_capacity(64 * 1024 * 1024)
                .with_file_size(4 * 1024 * 1024)
                .build(),
        )
        .with_flush(true)
        .with_indexer_shards(64)
        .with_recover_mode(RecoverMode::Quiet)
        .with_recover_concurrency(8)
        .with_flushers(2)
        .with_reclaimers(2)
        .with_buffer_threshold(256 * 1024 * 1024)
        .with_clean_region_threshold(4)
        .with_eviction_pickers(vec![Box::<FifoPicker>::default()])
        .with_admission_picker(Arc::new(RateLimitPicker::new(100 * 1024 * 1024)))
        .with_reinsertion_picker(Arc::new(RateLimitPicker::new(10 * 1024 * 1024)))
        .with_compression(foyer::Compression::Lz4)
        .with_tombstone_log_config(
            TombstoneLogConfigBuilder::new(dir.path().join("tombstone-log-file"))
                .with_flush(true)
                .build(),
        )
        .with_runtime_config(RuntimeConfig::Separated {
            read_runtime_config: TokioRuntimeConfig {
                worker_threads: 4,
                max_blocking_threads: 8,
            },
            write_runtime_config: TokioRuntimeConfig {
                worker_threads: 4,
                max_blocking_threads: 8,
            },
        })
        .build()
        .await?;

    hybrid.insert(42, "The answer to life, the universe, and everything.".to_string());
    assert_eq!(
        hybrid.get(&42).await?.unwrap().value(),
        "The answer to life, the universe, and everything."
    );

    let e = hybrid
        .fetch(20230512, || async {
            let value = mock().await?;
            Ok(value)
        })
        .await?;
    assert_eq!(e.key(), &20230512);
    assert_eq!(e.value(), "Hello, foyer.");

    hybrid.close().await.unwrap();

    Ok(())
}

async fn mock() -> Result<String> {
    let now = chrono::Utc::now();
    if format!("{}{}{}", now.year(), now.month(), now.day()) == "20230512" {
        return Err(anyhow::anyhow!("Hi, time traveler!"));
    }
    Ok("Hello, foyer.".to_string())
}

Other Examples

More examples and details can be found here.

Supported Rust Versions

foyer is built against the recent stable release. The minimum supported version is 1.77. The current foyer version is not guaranteed to build on Rust versions earlier than the minimum supported version.

Development State & Roadmap

Currently, foyer is still under heavy development.

The development state and the roadmap can be found in foyer - Development Roadmap.

Contributing

Contributions for foyer are warmly welcomed! 🥰

Don't forget to pass make fast (which means fast check & test) locally before submitting a PR. 🚀

If you want to run a broader range of checks locally, run make full. 🙌

Star History

foyer's People

Contributors

Stargazers

Watchers

Forkers

wenym1 tennyzhuang igxnon xiaguan ishiihara xuanwo andylokandy chacix wanghenshui

foyer's Issues

feat: new mechanism to limit memory usage by stall

The original implementation uses a ring buffer to manage and limit memory usage. The new one, without the ring buffer, has no method but a rate limit admission policy only to do it, which can be forgotten sometimes or misconfigured. A new mechanism is need to limit memory usage and stall writers.

ref: #230

bug: insert value size is zero

Reproduce commands(by fuzz)

        Input {
            capacity: 95,
            cache_type: S3Fifo,
            operations: [
                Insert(
                    0,
                    0,
                    0,
                ),
                Insert(
                    33,
                    33,
                    143,
                ),
            ],
        }

thread '<unnamed>' panicked at /home/susun/foyer/foyer-memory/src/eviction/s3fifo.rs:223:13:
assertion failed: self.is_empty()

feat: accumulate next buffer when flushing

Simplify the abstraction of evict policy

Regarding the abstraction of eviction in foryer-memory, i.e., the Eviction trait, I feel it is somewhat over-abstracted. We should simplify some of its interfaces and the use of generics to reduce the development and maintenance costs of the eviction algorithms.

Here are two references:

Looking forward to everyone's opinions."

tracking: compression

Roadmap:

~~Alter apply writer allocation size according to compressed size.~~ (write model modified with ring buffer, not planed)
Alter rate limit algorithm to handle actual rate != expected rate.
Add compression support. #198
Refine bench tool data generator to adapt to compression. #204

chore: use workspace cargo configuration

perf: `fs::read` dispatch io requests concurrently

Tracking: reduce insert tail latency

Related issue: #121

Roadmap:

Fair allocation. #140
~~[ ] Separated dirty buffer.~~
~~[ ] Pipeline dirty buffer flush.~~

bug: unit test in mod `storage_test` panic on macos target

log:

--- STDOUT:              foyer-storage::storage_test test_lazy_store ---

running 1 test
test test_lazy_store ... FAILED

failures:

failures:
    test_lazy_store

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 5 filtered out; finished in 2.49s


--- STDERR:              foyer-storage::storage_test test_lazy_store ---
thread 'test_lazy_store' panicked at foyer-storage/tests/storage_test.rs:[79](https://github.com/MrCroxx/foyer/actions/runs/8642338731/job/23693294946?pr=319#step:14:80):17:
assertion failed: store.lookup(&i).await.unwrap().is_none()
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

feat: statics and metrics

Statics and metrics are needed for better observation.

Eviction policy statics
Storage engine statics (for admission policy and reinsertion policy)
Storage engine metrics

feat: recovery

Recover storage engine after restart.

docs: architecture documents

foyer-storage architecture docs
foyer-intrusive architecture docs
foyer architecture docs (not planned yet, needed after foyer lib is complete)

feat: implement TTL

Implement TTL for foyer-memory and foyer-storage

Implement TTL for foyer-memory.
Implement TTL for foyer-storage.

bug: flush half-dirty region while closing store

feat: YSCB workload generator for benchmark

bug: get / insert IOPS and throughput analysis in bench

Currently, the op IOPS and throughput analysis are based on op call, which is not accurate if the admission/reinsertion policy is enabled.

e.g. (rated ticket admission policy is enabled)

Total:
disk total iops: 491.6
disk total throughput: 7.7 MiB/s
disk read iops: 0.0
disk read throughput: 0 B/s
disk write iops: 491.6
disk write throughput: 7.7 MiB/s
insert iops: 1327858.5/s # ***
insert throughput: 81.0 GiB/s # ***
insert lat p50: 0us
insert lat p90: 0us
insert lat p99: 1us
insert lat pmax: 5887us
get iops: 1819351.0/s # ***
get miss: 99.99%
get hit lat p50: 7us
get hit lat p90: 10us
get hit lat p99: 16us
get hit lat pmax: 209us
get miss lat p50: 0us
get miss lat p90: 0us
get miss lat p99: 0us
get miss lat pmax: 7583us

feat: poll mode flusher & reclaimer

Try poll mode flusher & reclaimer to simply race condition and resource control.

feat: write batch

feat: admission policy based on real flush throughput

bug: address saniziter report stack overflow with foyer-storage

tracking: make foyer compatible with stable rust

bug: abnormally high latency with gp3

configuration: ec2 c6id.2xlarge with EBS gp3 6000 IOPS 300MB/s 100GB

[3037s/7200s]
disk total iops: 3022.8
disk total throughput: 274.5 MiB/s
disk read iops: 969.0
disk read throughput: 45.5 MiB/s
disk write iops: 2053.8
disk write throughput: 229.0 MiB/s
insert iops: 2495.6/s
insert throughput: 224.1 MiB/s
insert lat p50: 78us
insert lat p90: 1564671us
insert lat p99: 1736703us
insert lat p999: 3506175us
insert lat p9999: 3883007us
insert lat p99999: 3997695us
insert lat pmax: 7405567us
get iops: 614.7/s
get miss: 5.28%
get throughput: 51.6 MiB/s
get hit lat p50: 2175us
get hit lat p90: 10879us
get hit lat p99: 14591us
get hit lat p999: 27775us
get hit lat p9999: 62719us
get hit lat p99999: 143359us
get hit lat pmax: 278527us
get miss lat p50: 54us
get miss lat p90: 435us
get miss lat p99: 2735us
get miss lat p999: 4767us
get miss lat p9999: 16319us
get miss lat p99999: 26879us
get miss lat pmax: 30591us

foyer-storage-bench --dir /ebs/foyer --capacity 81920 --region-size 64 --lookup-range 100000 --flushers 8 --reclaimers 4 --time 7200 --writers 4096 --w-rate 0.1 --ticket-insert-rate-limit 240 --readers 512 --r-rate 0.1 --runtime --entry-size-min 32768 --entry-size-max 153600 --metrics --io-size 65536 --ring-buffer-capacity 1500

Collaboration discuss: Rust Multi-level Cache Library

Hey there! 👋

As part of my work, I've been heavily involved in various cache-related projects and have read numerous papers on caching, including many issues surrounding Facebook's cacheLib. After much consideration, I feel it's time for me to contribute to the open-source community.

I've been planning to write a multi-level cache library in Rust for about a month now. However, just as I was about to start coding, someone informed me that this has already been done. Upon closer inspection, it appears that the existing project has made significant progress.

So, I was wondering if there's any possibility for collaboration between us. 🤝 I'd be more than happy to contribute my expertise and insights to help enhance the library further. Perhaps we could explore new features, optimize performance, or tackle any outstanding issues.

I'm open to any ideas you might have! Whether it's pair programming, code reviews, or simply discussing potential improvements, I'm all for it. 💡

Looking forward to hearing your thoughts on this! 😄

feat: direct i/o

foyer-storage is a FTL-like storage engine, which will be used directly on raw block device or file systems. For there should be upper layer memory cache system above foyer-storage, bypassing os page cache is reasonable for better memory control (especially for vm/container environments with few resources).

bug: github action macos-latest runner chooses wrong toolchain

The aarch64 target is expected.

FYI: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners

bug: add operaton overflow

        Input {
            capacity: 16140894441656287227,
            cache_type: S3Fifo,
            operations: [
                Insert(
                    0,
                    15,
                    1085102592571150095,
                ),
                Insert(
                    15,
                    15,
                    18379149462045917184,
                ),
                Insert(
                    0,
                    0,
                    0,
                ),
            ],
        }

thread '<unnamed>' panicked at /home/susun/foyer/foyer-memory/src/generic.rs:229:15:
attempt to add with overflow

tracking: improve the API to make foyer more user-friendly

Remove unnecessary generic type.
Better cache builder for in-memory cache. #350
Better cache builder for disk cache. #369
Refine and unify the Key Value abstraction of im-memory cache and disk cache. #366
Unify in-memory cache and disk cache build. #391

chore: CHANGELOG for version release

feat: light-weight single thread in-memory cache

Currently foyer-memory only provides a thread-safe sharded cache. A light-weight single thread in-memory cache can also be provided with some refactors to extend the use cases.

test: CI pipeline on more platforms.

Platforms:

Windows
MacOS #318

bug: core dump when cancelling foyer-storage-bench

When cancelling foyer-storage-bench with CTRL-C, the program will exit with core dump.

[1]    2360763 IOT instruction (core dumped)  RUST_BACKTRACE=1 RUST_LOG=info ./target/release/foyer-storage-bench --dir     |
       2360764 interrupt                      tee target/bench.log

feat: better intrusive index collections

Currently, foyer-storage and foyer-memory require that Key: Clone, ... because they may refer to the same key from multiple index collections. If there are intrusive index collections like an intrusive hashmap, Key can be embedded in the entry item. But the performance of the current intrusive index collections is poor, which needs optimization.

is support on windows?

error[E0432]: unresolved import std::os::fd
--> E:\rust.cargo\registry\src\rsproxy.cn-0dccff568467c15b\foyer-storage-0.3.0\src\device\fs.rs:17:9
|
17 | os::fd::{AsRawFd, BorrowedFd, RawFd},
| ^^ could not find fd in os

feat: integrate tracing framework

feat: unbounded eviction container

Currently, Eviction for in-memory cache requires capacity to set up pool size in some eviction algorithm implementations.

It is not reasonable for an unbounded eviction container, needs refactoring.

feat: support mvcc

tracking:

support tombstone.
support snapshot read api

bug: deadlock with 1 worker thread

Found deadlock with 1 worker thread, blocked CI.

Remove entry index when checksum mismatch

Currently, read_entry returns None when the checksum mismatch and outputs a WARN log. The entry can be removed from index to prevent from accessing again.

bug: flusher & reclaimer needs graceful shutdown

When foyer-storage-bench finished, the following panic log is printed because flusher is dropped but inner read slice is not properly destroyed.

thread 'thread 'tokio-runtime-workertokio-runtime-worker' panicked at '' panicked at 'future is not consumed: Slice { slice: Slice { ptr: 0x7fdcbbff5000, len: 67108864 }, allocator: Some(AlignedAllocator { align: 4096 }) }future is not consumed: Slice { slice: Slice { ptr: 0x7fdcb7ff3000, len: 67108864 }, allocator: Some(AlignedAllocator { align: 4096 }) }', ', /home/mrcroxx/github/mrcroxx/foyer/foyer-storage/src/region.rs/home/mrcroxx/github/mrcroxx/foyer/foyer-storage/src/region.rs::511511::1313

stack backtrace:
thread 'thread 'tokio-runtime-workertokio-runtime-worker' panicked at '' panicked at 'future is not consumed: Slice { slice: Slice { ptr: 0x7fdcc3ff9000, len: 67108864 }, allocator: Some(AlignedAllocator { align: 4096 }) }future is not consumed: Slice { slice: Slice { ptr: 0x7fdcbfff7000, len: 67108864 }, allocator: Some(AlignedAllocator { align: 4096 }) }', ', /home/mrcroxx/github/mrcroxx/foyer/foyer-storage/src/region.rs/home/mrcroxx/github/mrcroxx/foyer/foyer-storage/src/region.rs::511511::1313

   0: rust_begin_unwind
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/std/src/panicking.rs:577:5
   1: core::panicking::panic_fmt
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/core/src/panicking.rs:67:14
   2: core::ptr::drop_in_place<foyer_storage::region::ReadSlice<foyer_storage::device::io_buffer::AlignedAllocator>>
   3: core::ptr::drop_in_place<foyer_storage::flusher::Runner<foyer_storage::device::io_buffer::AlignedAllocator,foyer_storage::device::fs::FsDevice,foyer_intrusive::eviction::lfu::Lfu<foyer_storage::region_manager::RegionEpItemAdapter<foyer_intrusive::eviction::lfu::LfuLink>>,foyer_intrusive::eviction::lfu::LfuLink>::run::{{closure}}>
   4: std::panicking::try
   5: tokio::runtime::task::harness::Harness<T,S>::shutdown
   6: tokio::runtime::task::list::OwnedTasks<S>::close_and_shutdown_all
   7: tokio::runtime::scheduler::multi_thread::worker::Context::run
   8: std::thread::local::LocalKey<T>::with
   9: tokio::runtime::context::runtime::enter_runtime
  10: tokio::runtime::scheduler::multi_thread::worker::run
  11: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  12: tokio::runtime::task::core::Core<T,S>::poll
  13: tokio::runtime::task::raw::poll
  14: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: rust_begin_unwind
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/std/src/panicking.rs:577:5
   1: core::panicking::panic_fmt
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/core/src/panicking.rs:67:14
   2: core::ptr::drop_in_place<foyer_storage::region::ReadSlice<foyer_storage::device::io_buffer::AlignedAllocator>>
   3: core::ptr::drop_in_place<foyer_storage::flusher::Runner<foyer_storage::device::io_buffer::AlignedAllocator,foyer_storage::device::fs::FsDevice,foyer_intrusive::eviction::lfu::Lfu<foyer_storage::region_manager::RegionEpItemAdapter<foyer_intrusive::eviction::lfu::LfuLink>>,foyer_intrusive::eviction::lfu::LfuLink>::run::{{closure}}>
   4: std::panicking::try
   5: tokio::runtime::task::harness::Harness<T,S>::shutdown
   6: tokio::runtime::task::list::OwnedTasks<S>::close_and_shutdown_all
   7: tokio::runtime::scheduler::multi_thread::worker::Context::run
   8: std::thread::local::LocalKey<T>::with
   9: tokio::runtime::context::runtime::enter_runtime
  10: tokio::runtime::scheduler::multi_thread::worker::run
  11: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  12: tokio::runtime::task::core::Core<T,S>::poll
  13: tokio::runtime::task::raw::poll
  14: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: rust_begin_unwind
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/std/src/panicking.rs:577:5
   1: core::panicking::panic_fmt
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/core/src/panicking.rs:67:14
   2: core::ptr::drop_in_place<foyer_storage::region::ReadSlice<foyer_storage::device::io_buffer::AlignedAllocator>>
   3: core::ptr::drop_in_place<foyer_storage::flusher::Runner<foyer_storage::device::io_buffer::AlignedAllocator,foyer_storage::device::fs::FsDevice,foyer_intrusive::eviction::lfu::Lfu<foyer_storage::region_manager::RegionEpItemAdapter<foyer_intrusive::eviction::lfu::LfuLink>>,foyer_intrusive::eviction::lfu::LfuLink>::run::{{closure}}>
   4: std::panicking::try
   5: tokio::runtime::task::harness::Harness<T,S>::shutdown
   6: tokio::runtime::task::list::OwnedTasks<S>::close_and_shutdown_all
   7: tokio::runtime::scheduler::multi_thread::worker::Context::run
   8: std::thread::local::LocalKey<T>::with
   9: tokio::runtime::context::runtime::enter_runtime
  10: tokio::runtime::scheduler::multi_thread::worker::run
  11: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  12: tokio::runtime::task::core::Core<T,S>::poll
  13: tokio::runtime::task::raw::poll
  14: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: rust_begin_unwind
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/std/src/panicking.rs:577:5
   1: core::panicking::panic_fmt
             at /rustc/28a29282f6dde2e4aba6e1e4cfea5c9430a00217/library/core/src/panicking.rs:67:14
   2: core::ptr::drop_in_place<foyer_storage::region::ReadSlice<foyer_storage::device::io_buffer::AlignedAllocator>>
   3: core::ptr::drop_in_place<foyer_storage::flusher::Runner<foyer_storage::device::io_buffer::AlignedAllocator,foyer_storage::device::fs::FsDevice,foyer_intrusive::eviction::lfu::Lfu<foyer_storage::region_manager::RegionEpItemAdapter<foyer_intrusive::eviction::lfu::LfuLink>>,foyer_intrusive::eviction::lfu::LfuLink>::run::{{closure}}>
   4: std::panicking::try
   5: tokio::runtime::task::harness::Harness<T,S>::shutdown
   6: tokio::runtime::task::list::OwnedTasks<S>::close_and_shutdown_all
   7: tokio::runtime::scheduler::multi_thread::worker::Context::run
   8: std::thread::local::LocalKey<T>::with
   9: tokio::runtime::context::runtime::enter_runtime
  10: tokio::runtime::scheduler::multi_thread::worker::run
  11: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  12: tokio::runtime::task::core::Core<T,S>::poll
  13: tokio::runtime::task::raw::poll
  14: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

chore: fill lost unit tests and raise code coverage

bug: frozen file not found

risingwave panic after a while with:

thread 'risingwave-batch-tasks' panicked at 'frozen file not found', /root/.cargo/git/checkouts/foyer-e7ae575671bf01ff/8e80c6c/foyer/src/store/read_only_file_store.rs:331:22
stack backtrace:
   0:     0x55c8c29079fc - std::backtrace_rs::backtrace::libunwind::trace::h43d94404a3e24b5e
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x55c8c29079fc - std::backtrace_rs::backtrace::trace_unsynchronized::hb80ac700c030e86a
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55c8c29079fc - std::sys_common::backtrace::_print_fmt::h6b86e24d9aefa090
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x55c8c29079fc - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h49d07dd1f79a975a
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x55c8bd191e1f - core::fmt::rt::Argument::fmt::h1864a7d8f112957b
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/core/src/fmt/rt.rs:138:9
   5:     0x55c8bd191e1f - core::fmt::write::hd813644c20dd03f8
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/core/src/fmt/mod.rs:1094:21
   6:     0x55c8c2902d6f - std::io::Write::write_fmt::h6fea2a18781d1e2e
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/io/mod.rs:1712:15
   7:     0x55c8c2907805 - std::sys_common::backtrace::_print::h9d3d888fc0c72876
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x55c8c2907805 - std::sys_common::backtrace::print::h04abcd5816949a30
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x55c8c2908d67 - std::panicking::default_hook::{{closure}}::ha75ea1b8d759bad4
  10:     0x55c8c2908b74 - std::panicking::default_hook::he2c557a4cfe41c90
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/panicking.rs:288:9
  11:     0x55c8c228a23b - call<(&core::panic::panic_info::PanicInfo), (dyn core::ops::function::Fn<(&core::panic::panic_info::PanicInfo), Output=()> + core::marker::Send + core::marker::Sync), alloc::alloc::Global>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/alloc/src/boxed.rs:1999:9
  12:     0x55c8c228a23b - {closure#0}
                               at /risingwave/src/utils/runtime/src/lib.rs:119:9
  13:     0x55c8c228a23b - {closure#0}<risingwave_rt::set_panic_hook::{closure_env#0}>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/panicking.rs:234:47
  14:     0x55c8c2909384 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hcf9192dbf6344206
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/alloc/src/boxed.rs:1999:9
  15:     0x55c8c2909384 - std::panicking::rust_panic_with_hook::hb4b975f86bb49bbc
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/panicking.rs:695:13
  16:     0x55c8c2909114 - std::panicking::begin_panic_handler::{{closure}}::h381a1086a75e4820
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/panicking.rs:582:13
  17:     0x55c8c2907dd6 - std::sys_common::backtrace::__rust_end_short_backtrace::h2f950c06980ecb26
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/sys_common/backtrace.rs:151:18
  18:     0x55c8c2908eb2 - rust_begin_unwind
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/panicking.rs:578:5
  19:     0x55c8bd18eb23 - core::panicking::panic_fmt::he9fd35b0a19637c8
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/core/src/panicking.rs:67:14
  20:     0x55c8bd18e983 - core::panicking::panic_display::ha29c9cb28ca203a7
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/core/src/panicking.rs:150:5
  21:     0x55c8bd18e983 - core::panicking::panic_str::ha5696582f17c5eba
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/core/src/panicking.rs:134:5
  22:     0x55c8bd18e983 - core::option::expect_failed::h2745ad5925c1c797
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/core/src/option.rs:1932:5
  23:     0x55c8be065f7d - expect<&foyer::store::read_only_file_store::Frozen>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/core/src/option.rs:898:21
  24:     0x55c8be065f7d - {async_block#0}<risingwave_storage::hummock::tiered_cache::SstableBlockIndex, alloc::vec::Vec<u8, alloc::alloc::Global>>
                               at /root/.cargo/git/checkouts/foyer-e7ae575671bf01ff/8e80c6c/foyer/src/store/read_only_file_store.rs:328:17
  25:     0x55c8bdf06990 - poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<(), foyer::store::error::Error>> + core::marker::Send), alloc::alloc::Global>>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/core/src/future/future.rs:125:9
  26:     0x55c8bdf06990 - {async_fn#0}<risingwave_storage::hummock::tiered_cache::SstableBlockIndex, foyer::policies::tinylfu::TinyLfu<risingwave_storage::hummock::tiered_cache::SstableBlockIndex>, foyer::policies::tinylfu::Handle<risingwave_storage::hummock::tiered_cache::SstableBlockIndex>, alloc::vec::Vec<u8, alloc::alloc::Global>, foyer::store::read_only_file_store::ReadOnlyFileStore<risingwave_storage::hummock::tiered_cache::SstableBlockIndex, alloc::vec::Vec<u8, alloc::alloc::Global>>>
                               at /root/.cargo/git/checkouts/foyer-e7ae575671bf01ff/8e80c6c/foyer/src/container.rs:236:38
  27:     0x55c8bdf06990 - {async_fn#0}<risingwave_storage::hummock::tiered_cache::SstableBlockIndex, foyer::policies::tinylfu::TinyLfu<risingwave_storage::hummock::tiered_cache::SstableBlockIndex>, foyer::policies::tinylfu::Handle<risingwave_storage::hummock::tiered_cache::SstableBlockIndex>, alloc::vec::Vec<u8, alloc::alloc::Global>, foyer::store::read_only_file_store::ReadOnlyFileStore<risingwave_storage::hummock::tiered_cache::SstableBlockIndex, alloc::vec::Vec<u8, alloc::alloc::Global>>>
                               at /root/.cargo/git/checkouts/foyer-e7ae575671bf01ff/8e80c6c/foyer/src/container.rs:142:32
  28:     0x55c8bdf06990 - {async_fn#0}
                               at /risingwave/src/storage/src/hummock/tiered_cache.rs:120:18
  29:     0x55c8bdf06990 - {async_block#0}
                               at /risingwave/src/storage/src/hummock/sstable_store.rs:206:69
  30:     0x55c8bdf5cc59 - {async_block#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>
                               at /risingwave/src/storage/src/hummock/block_cache.rs:205:35
  31:     0x55c8bdf5cc59 - {async_block#0}<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>, risingwave_storage::hummock::error::HummockError, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure#0}::{async_block_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>>
                               at /risingwave/src/common/src/cache.rs:967:55
  32:     0x55c8bdf5cc59 - poll<risingwave_common::cache::{impl#14}::lookup_with_request_dedup::{async_block_env#0}<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>, risingwave_storage::hummock::error::HummockError, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure#0}::{async_block_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>>>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  33:     0x55c8bdf87987 - {closure#0}<tracing::instrument::Instrumented<risingwave_common::cache::{impl#14}::lookup_with_request_dedup::{async_block_env#0}<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>, risingwave_storage::hummock::error::HummockError, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure#0}::{async_block_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:223:17
  34:     0x55c8bdf87987 - with_mut<tokio::runtime::task::core::Stage<tracing::instrument::Instrumented<risingwave_common::cache::{impl#14}::lookup_with_request_dedup::{async_block_env#0}<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>, risingwave_storage::hummock::error::HummockError, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure#0}::{async_block_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>>>>, core::task::poll::Poll<core::result::Result<risingwave_common::cache::CacheableEntry<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, risingwave_storage::hummock::error::HummockError>>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<tracing::instrument::Instrumented<risingwave_common::cache::{impl#14}::lookup_with_request_dedup::{async_block_env#0}<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>, risingwave_storage::hummock::error::HummockError, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure#0}::{async_block_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle>>>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/loom/std/unsafe_cell.rs:14:9
  35:     0x55c8bdf48d74 - poll<tracing::instrument::Instrumented<risingwave_common::cache::{impl#14}::lookup_with_request_dedup::{async_block_env#0}<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>, risingwave_storage::hummock::error::HummockError, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure#0}::{async_block_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:212:13
  36:     0x55c8be04585a - {closure#0}<tracing::instrument::Instrumented<risingwave_common::cache::{impl#14}::lookup_with_request_dedup::{async_block_env#0}<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>, risingwave_storage::hummock::error::HummockError, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure#0}::{async_block_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:476:19
  37:     0x55c8be04585a - call_once<core::task::poll::Poll<core::result::Result<risingwave_common::cache::CacheableEntry<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, risingwave_storage::hummock::error::HummockError>>, tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<risingwave_common::cache::{impl#14}::lookup_with_request_dedup::{async_block_env#0}<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>, risingwave_storage::hummock::error::HummockError, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure#0}::{async_block_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/core/src/panic/unwind_safe.rs:271:9
  38:     0x55c8be04585a - do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<risingwave_common::cache::{impl#14}::lookup_with_request_dedup::{async_block_env#0}<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>, risingwave_storage::hummock::error::HummockError, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure#0}::{async_block_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>, core::task::poll::Poll<core::result::Result<risingwave_common::cache::CacheableEntry<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, risingwave_storage::hummock::error::HummockError>>>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/panicking.rs:485:40
  39:     0x55c8be04585a - try<core::task::poll::Poll<core::result::Result<risingwave_common::cache::CacheableEntry<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, risingwave_storage::hummock::error::HummockError>>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<risingwave_common::cache::{impl#14}::lookup_with_request_dedup::{async_block_env#0}<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>, risingwave_storage::hummock::error::HummockError, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure#0}::{async_block_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/panicking.rs:449:19
  40:     0x55c8be04585a - catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<risingwave_common::cache::{impl#14}::lookup_with_request_dedup::{async_block_env#0}<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>, risingwave_storage::hummock::error::HummockError, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure#0}::{async_block_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>>, core::task::poll::Poll<core::result::Result<risingwave_common::cache::CacheableEntry<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, risingwave_storage::hummock::error::HummockError>>>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/panic.rs:142:14
  41:     0x55c8be04585a - poll_future<tracing::instrument::Instrumented<risingwave_common::cache::{impl#14}::lookup_with_request_dedup::{async_block_env#0}<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>, risingwave_storage::hummock::error::HummockError, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure#0}::{async_block_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:464:18
  42:     0x55c8be04585a - poll_inner<tracing::instrument::Instrumented<risingwave_common::cache::{impl#14}::lookup_with_request_dedup::{async_block_env#0}<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>, risingwave_storage::hummock::error::HummockError, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure#0}::{async_block_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:198:27
  43:     0x55c8be04585a - poll<tracing::instrument::Instrumented<risingwave_common::cache::{impl#14}::lookup_with_request_dedup::{async_block_env#0}<(u64, u64), alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>, risingwave_storage::hummock::error::HummockError, risingwave_storage::hummock::block_cache::{impl#5}::get_or_insert_with::{closure#0}::{async_block_env#0}<risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure_env#0}, risingwave_storage::hummock::sstable_store::{impl#1}::get_block_response::{async_fn#0}::{closure#0}::{async_block_env#0}>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:152:15
  44:     0x55c8c29a516b - poll
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/raw.rs:200:18
  45:     0x55c8c29a516b - run<alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/mod.rs:394:9
  46:     0x55c8c29a516b - {closure#0}
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:464:18
  47:     0x55c8c29a516b - with_budget<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core, alloc::alloc::Global>, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/coop.rs:107:5
  48:     0x55c8c29a516b - budget<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core, alloc::alloc::Global>, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/coop.rs:73:5
  49:     0x55c8c29a516b - run_task
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:463:9
  50:     0x55c8c29a47c3 - run
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:426:24
  51:     0x55c8c2997f2d - {closure#0}
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:406:17
  52:     0x55c8c2997f2d - set<tokio::runtime::scheduler::multi_thread::worker::Context, tokio::runtime::scheduler::multi_thread::worker::run::{closure_env#0}, ()>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/macros/scoped_tls.rs:61:9
  53:     0x55c8c29a4238 - run
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:403:5
  54:     0x55c8c2986dfc - {closure#0}
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/scheduler/multi_thread/worker.rs:365:45
  55:     0x55c8c2986dfc - poll<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}, ()>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/task.rs:42:21
  56:     0x55c8c2986dfc - poll<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  57:     0x55c8c298c29d - {closure#0}<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:223:17
  58:     0x55c8c298c29d - with_mut<tokio::runtime::task::core::Stage<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/loom/std/unsafe_cell.rs:14:9
  59:     0x55c8c298c29d - poll<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/core.rs:212:13
  60:     0x55c8c296ed2f - {closure#0}<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:476:19
  61:     0x55c8c296ed2f - call_once<core::task::poll::Poll<()>, tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/core/src/panic/unwind_safe.rs:271:9
  62:     0x55c8c296ed2f - do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>, core::task::poll::Poll<()>>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/panicking.rs:485:40
  63:     0x55c8c296ed2f - try<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/panicking.rs:449:19
  64:     0x55c8c296ed2f - catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>, core::task::poll::Poll<()>>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/panic.rs:142:14
  65:     0x55c8c296ed2f - poll_future<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:464:18
  66:     0x55c8c296ed2f - poll_inner<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:198:27
  67:     0x55c8c296ed2f - poll<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/harness.rs:152:15
  68:     0x55c8c296d910 - poll
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/raw.rs:200:18
  69:     0x55c8c296d910 - run<tokio::runtime::blocking::schedule::BlockingSchedule>
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/task/mod.rs:431:9
  70:     0x55c8c296d910 - run
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/pool.rs:159:9
  71:     0x55c8c296d910 - run
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/pool.rs:513:17
  72:     0x55c8c2995e45 - {closure#0}
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.0/src/runtime/blocking/pool.rs:471:13
  73:     0x55c8c2995e45 - __rust_begin_short_backtrace<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/sys_common/backtrace.rs:135:18
  74:     0x55c8c297a692 - {closure#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/thread/mod.rs:529:17
  75:     0x55c8c297a692 - call_once<(), std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/core/src/panic/unwind_safe.rs:271:9
  76:     0x55c8c297a692 - do_call<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>>, ()>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/panicking.rs:485:40
  77:     0x55c8c297a692 - try<(), core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>>>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/panicking.rs:449:19
  78:     0x55c8c297a692 - catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>>, ()>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/panic.rs:142:14
  79:     0x55c8c297a692 - {closure#1}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/thread/mod.rs:528:30
  80:     0x55c8c297a692 - call_once<std::thread::{impl#0}::spawn_unchecked_::{closure_env#1}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>, ()>
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/core/src/ops/function.rs:250:5
  81:     0x55c8c290f035 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h05bc36bea856dce6
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/alloc/src/boxed.rs:1985:9
  82:     0x55c8c290f035 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h76ae250201c440c9
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/alloc/src/boxed.rs:1985:9
  83:     0x55c8c290f035 - std::sys::unix::thread::Thread::new::thread_start::h032990071e345e22
                               at /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/library/std/src/sys/unix/thread.rs:108:17
  84:     0x7f8b5a2c3b43 - <unknown>
  85:     0x7f8b5a355a00 - <unknown>
  86:                0x0 - <unknown>

introduce error level

Currently Error is only identified by kind, which is not convenient for distinguishing if the error is recoverable (like fetch value error or checksum mismatch, etc). Give a level to each error may make it easier.

refactor: remove `serialized_len` requirement of Key/Value

perf: insert tail latency issue

Sometimes insert tail latency is abnormally high. Needs investigation.

bug: unstable recovery test

broken CI: https://github.com/MrCroxx/foyer/actions/runs/5538840433/jobs/10109184240

feat: prometheus metrics support

Add new feature prometheus to export prometheus metrics.

feat: allow key/value impl std::io::Read to avoid buffer copy

Currently key/value needs to implement fn write(buffer:&mut [u8]) to write the ring buffer, then the data in the ring buffer will be compressed and written to the flusher buffer. With key/value implementing std::io::Read, writing the ring buffer can be avoided.

test: add unit test case with concurrent ops

case should be avoided:

t1: lookup index =>             [  scheduled ]            => load data from device
t2:             reclaimer drop index => reclaimer wait reads finish

perf: introduce ring buffer

Previously, foyer used the write model of navy, which is the storage engine of cachelib. When the buffer pool is full, the allocator in navy waits for a dirty buffer fully flushed and reuses it, which means the worst latency equals the flushing duration of the whole region, which can be extremely high.

Foyer plans to introduce a ring buffer for fast allocation. The uncompressed data will be first written to the ring buffer, then compressed and copied to the region dirty buffer.

Pros:

Worst latency equals the flush duration of an entry with similar size.
Friendly to compression (cachelib doesn't have the problem because it compress data in upper layer).

Cons:

One more buffer copy if data doesn't need to be compressed.
Complexity.

refactor: remove old intrusive dlist usage

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.