open-cas / ocf Goto Github PK
View Code? Open in Web Editor NEWOpen CAS Framework
License: BSD 3-Clause "New" or "Revised" License
Open CAS Framework
License: BSD 3-Clause "New" or "Revised" License
The ocf example only demonstrate the IO path by a synchronize call of submit_io
and complete callback function. But it still remains unclear that how we should manager the synchronization in a real scenario. How many threads should there be? Do we need to create a new thread(coroutine) in the complete callback to achieve synchronization? Which worker thread is responsible for executing the volume-defined ocf_volume_properties.ops.submit_io
interface?
The CAS itself is indeed a good example, however it requires the developers to have basic knowledge of the kernel. Otherwise you won't realize how and when each function is being called by specific kernel threads, just by reading the code.
So please add some documents in terms of worker threads management.
Hello
I am getting the following error when I try to build SPDK with ocf (--with-ocf option). I have tried to build with the ocf module in SPDK and also downloading the framework from the ocf git repository and I got the same error for both. I also did a build with ocf 18.12 as suggested in one of the other issues posted here and got the same error.
In file included from src/ocf/cleaning/nop.c:7:0:
src/ocf/cleaning/nop.h:12:55: error: unknown type name ‘ocf_cleaner_end_t’; did you mean ‘ocf_cleaner_t’?
void cleaning_nop_perform_cleaning(ocf_cache_t cache, ocf_cleaner_end_t cmpl);
^~~~~~~~~~~~~~~~~
ocf_cleaner_t
src/ocf/cleaning/nop.c:10:55: error: unknown type name ‘ocf_cleaner_end_t’; did you mean ‘ocf_cleaner_t’?
void cleaning_nop_perform_cleaning(ocf_cache_t cache, ocf_cleaner_end_t cmpl)
^~~~~~~~~~~~~~~~~
ocf_cleaner_t
/home/john/spdk/mk/spdk.common.mk:330: recipe for target 'src/ocf/cleaning/nop.o' failed
make[3]: *** [src/ocf/cleaning/nop.o] Error 1
/home/john/spdk/mk/spdk.subdirs.mk:44: recipe for target 'ocf/env' failed
make[2]: *** [ocf/env] Error 2
/home/john/spdk/mk/spdk.subdirs.mk:44: recipe for target 'bdev' failed
make[1]: *** [bdev] Error 2
/home/john/spdk/mk/spdk.subdirs.mk:44: recipe for target 'lib' failed
make: *** [lib] Error 2
There's a possible race condition between pass-through write and cache insert. When PT write and read insert are submitted to the same LBA at the same time it's possible all further read hits to given LBA would return old data instead of data written by PT write request.
Race condition scenario:
Time Pass-through Write Cache insert read
|
| Traversation (->miss)
| : Traversation (->miss)
| : Mapping
| : Read from core
| : :
| Data written to the device :
|
|
v
Key conditions:
It's easily reproducible by artificially slowing down writes to core device in bottom adapter.
Test script (for open-cas-linux):
#!/bin/bash
set -ex
CACHE=/dev/nvme0n1p1
CORE=/dev/sdb1
S=50000
dd if=/dev/urandom of=base1.bin bs=4k count=1
dd if=/dev/urandom of=base2.bin bs=4k count=1
# 1. Write base pattern to core (base1.bin)
dd if=base1.bin of=${CORE} seek=${S} bs=4k count=1 oflag=direct
casadm -S -i 1 -d ${CACHE} -c wa -f
casadm -A -i 1 -d ${CORE}
# 2. Overwrite pattern on cas dev. Goes in PT mode. (base2.bin)
dd if=base2.bin of=/dev/cas1-1 seek=${S} bs=4k count=1 oflag=direct &
# 3. Read the same LBA in parallel while Open-CAS/open-cas-linux#2 is in progress (out1.bin)
sleep 3
dd if=/dev/cas1-1 of=out1.bin skip=${S} bs=4k count=1 iflag=direct
wait
# 4. Reread the same LBA after write is finished (out2.bin)
dd if=/dev/cas1-1 of=out2.bin skip=${S} bs=4k count=1 iflag=direct
casadm -T -i 1 -n
# 5. Reread the same LBA after cache stop, directly from core (out3.bin)
dd if=${CORE} of=out3.bin skip=${S} bs=4k count=1 iflag=direct
md5sum base1.bin base2.bin out1.bin out2.bin out3.bin
# Expected checksums:
# base1 - A
# base2 - B
# out1 - A or B
# out2 - B
# out3 - B
# Actual checksums:
# base1 - A
# base2 - B
# out1 - A
# out2 - A -> wrong
# out3 - B
Tried compiling SPDK with ocf and it failed with below:
[root@Target spdk]# ./configure --with-rdma --with-ocf=/home/ocf
Using default SPDK env in /home/spdk/lib/env_dpdk
Using default DPDK in /home/spdk/dpdk/build
Configuring ISA-L (logfile: /tmp/spdk-isal.log)...done.
configuring OCF...
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_stats.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_utilities.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_io_class.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_def.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_cfg.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_logger.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_metadata_updater.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_io.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_queue.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_ctx.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_trace.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_cleaner.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_cache.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_stats_builder.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_core.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/cleaning/alru.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/cleaning/acp.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_metadata.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_types.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_err.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_data_obj.h
INSTALL /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_mngt.h
make[1]: Entering directory '/home/ocf'
make[1]: Nothing to be done for 'distcleandir'.
make[1]: Leaving directory '/home/ocf'
done configuring OCF
Creating mk/config.mk...done.
Type 'make' to build.
[root@Target spdk]# make
Configuration done using x86_64-native-linuxapp-gcc
make[3]: Entering directory '/home/spdk/dpdk'
== Build lib
== Build lib/librte_compat
SYMLINK-FILE include/rte_compat.h
== Build lib/librte_kvargs
SYMLINK-FILE include/rte_kvargs.h
CC rte_kvargs.o
AR librte_kvargs.a
INSTALL-LIB librte_kvargs.a
== Build lib/librte_eal
== Build lib/librte_eal/common
SYMLINK-FILE include/rte_branch_prediction.h
SYMLINK-FILE include/rte_common.h
SYMLINK-FILE include/rte_debug.h
SYMLINK-FILE include/rte_eal.h
SYMLINK-FILE include/rte_eal_interrupts.h
SYMLINK-FILE include/rte_errno.h
SYMLINK-FILE include/rte_launch.h
SYMLINK-FILE include/rte_lcore.h
SYMLINK-FILE include/rte_log.h
SYMLINK-FILE include/rte_memory.h
SYMLINK-FILE include/rte_memzone.h
SYMLINK-FILE include/rte_per_lcore.h
SYMLINK-FILE include/rte_random.h
SYMLINK-FILE include/rte_tailq.h
SYMLINK-FILE include/rte_interrupts.h
SYMLINK-FILE include/rte_alarm.h
SYMLINK-FILE include/rte_string_fns.h
SYMLINK-FILE include/rte_version.h
SYMLINK-FILE include/rte_eal_memconfig.h
SYMLINK-FILE include/rte_malloc_heap.h
SYMLINK-FILE include/rte_hexdump.h
SYMLINK-FILE include/rte_devargs.h
SYMLINK-FILE include/rte_bus.h
SYMLINK-FILE include/rte_dev.h
SYMLINK-FILE include/rte_class.h
SYMLINK-FILE include/rte_option.h
SYMLINK-FILE include/rte_pci_dev_feature_defs.h
SYMLINK-FILE include/rte_pci_dev_features.h
SYMLINK-FILE include/rte_malloc.h
SYMLINK-FILE include/rte_keepalive.h
SYMLINK-FILE include/rte_time.h
SYMLINK-FILE include/rte_service_component.h
SYMLINK-FILE include/rte_service.h
SYMLINK-FILE include/rte_bitmap.h
SYMLINK-FILE include/rte_vfio.h
SYMLINK-FILE include/rte_hypervisor.h
SYMLINK-FILE include/rte_test.h
SYMLINK-FILE include/rte_reciprocal.h
SYMLINK-FILE include/rte_fbarray.h
SYMLINK-FILE include/rte_uuid.h
SYMLINK-FILE include/rte_atomic.h
SYMLINK-FILE include/rte_atomic_32.h
SYMLINK-FILE include/rte_atomic_64.h
SYMLINK-FILE include/rte_byteorder.h
SYMLINK-FILE include/rte_byteorder_32.h
SYMLINK-FILE include/rte_byteorder_64.h
SYMLINK-FILE include/rte_cpuflags.h
SYMLINK-FILE include/rte_cycles.h
SYMLINK-FILE include/rte_io.h
SYMLINK-FILE include/rte_memcpy.h
SYMLINK-FILE include/rte_pause.h
SYMLINK-FILE include/rte_prefetch.h
SYMLINK-FILE include/rte_rtm.h
SYMLINK-FILE include/rte_rwlock.h
SYMLINK-FILE include/rte_spinlock.h
SYMLINK-FILE include/rte_vect.h
SYMLINK-FILE include/generic/rte_atomic.h
SYMLINK-FILE include/generic/rte_byteorder.h
SYMLINK-FILE include/generic/rte_cycles.h
SYMLINK-FILE include/generic/rte_prefetch.h
SYMLINK-FILE include/generic/rte_spinlock.h
SYMLINK-FILE include/generic/rte_memcpy.h
SYMLINK-FILE include/generic/rte_cpuflags.h
SYMLINK-FILE include/generic/rte_rwlock.h
SYMLINK-FILE include/generic/rte_vect.h
SYMLINK-FILE include/generic/rte_io.h
SYMLINK-FILE include/generic/rte_pause.h
== Build lib/librte_eal/linuxapp
== Build lib/librte_eal/linuxapp/eal
SYMLINK-FILE include/exec-env/rte_kni_common.h
CC eal.o
CC eal_cpuflags.o
CC eal_hugepage_info.o
CC eal_memory.o
CC eal_thread.o
CC eal_log.o
CC eal_vfio.o
CC eal_vfio_mp_sync.o
CC eal_memalloc.o
CC eal_lcore.o
CC eal_debug.o
CC eal_timer.o
CC eal_interrupts.o
CC eal_alarm.o
CC eal_dev.o
CC eal_common_lcore.o
CC eal_common_timer.o
CC eal_common_memzone.o
CC eal_common_log.o
CC eal_common_launch.o
CC eal_common_memalloc.o
CC eal_common_memory.o
CC eal_common_tailqs.o
CC eal_common_errno.o
CC eal_common_cpuflags.o
CC eal_common_hypervisor.o
CC eal_common_string_fns.o
CC eal_common_hexdump.o
CC eal_common_devargs.o
CC eal_common_class.o
CC eal_common_bus.o
CC eal_common_dev.o
CC eal_common_options.o
CC eal_common_thread.o
CC eal_common_proc.o
CC eal_common_fbarray.o
CC eal_common_uuid.o
CC rte_malloc.o
CC hotplug_mp.o
CC malloc_elem.o
CC malloc_heap.o
CC malloc_mp.o
CC rte_keepalive.o
CC rte_option.o
CC rte_service.o
CC rte_reciprocal.o
CC rte_cpuflags.o
CC rte_hypervisor.o
CC rte_spinlock.o
CC rte_cycles.o
AR librte_eal.a
INSTALL-LIB librte_eal.a
== Build lib/librte_pci
== Build lib/librte_ring
== Build lib/librte_cmdline
SYMLINK-FILE include/rte_pci.h
CC rte_pci.o
SYMLINK-FILE include/rte_ring.h
SYMLINK-FILE include/rte_ring_generic.h
SYMLINK-FILE include/rte_ring_c11_mem.h
CC rte_ring.o
SYMLINK-FILE include/cmdline.h
SYMLINK-FILE include/cmdline_parse_num.h
SYMLINK-FILE include/cmdline_parse.h
SYMLINK-FILE include/cmdline_parse_ipaddr.h
SYMLINK-FILE include/cmdline_parse_string.h
SYMLINK-FILE include/cmdline_parse_etheraddr.h
SYMLINK-FILE include/cmdline_rdline.h
SYMLINK-FILE include/cmdline_vt100.h
SYMLINK-FILE include/cmdline_socket.h
SYMLINK-FILE include/cmdline_cirbuf.h
SYMLINK-FILE include/cmdline_parse_portlist.h
CC cmdline.o
CC cmdline_cirbuf.o
CC cmdline_parse.o
CC cmdline_parse_etheraddr.o
CC cmdline_parse_ipaddr.o
CC cmdline_parse_num.o
CC cmdline_parse_string.o
CC cmdline_rdline.o
CC cmdline_vt100.o
CC cmdline_socket.o
CC cmdline_parse_portlist.o
AR librte_pci.a
INSTALL-LIB librte_pci.a
AR librte_ring.a
INSTALL-LIB librte_ring.a
== Build lib/librte_mempool
AR librte_cmdline.a
INSTALL-LIB librte_cmdline.a
SYMLINK-FILE include/rte_mempool.h
CC rte_mempool.o
CC rte_mempool_ops.o
CC rte_mempool_ops_default.o
AR librte_mempool.a
INSTALL-LIB librte_mempool.a
== Build lib/librte_mbuf
SYMLINK-FILE include/rte_mbuf.h
SYMLINK-FILE include/rte_mbuf_ptype.h
SYMLINK-FILE include/rte_mbuf_pool_ops.h
CC rte_mbuf.o
CC rte_mbuf_pool_ops.o
CC rte_mbuf_ptype.o
AR librte_mbuf.a
INSTALL-LIB librte_mbuf.a
== Build lib/librte_net
== Build lib/librte_cryptodev
== Build lib/librte_compressdev
SYMLINK-FILE include/rte_ip.h
SYMLINK-FILE include/rte_udp.h
SYMLINK-FILE include/rte_tcp.h
SYMLINK-FILE include/rte_sctp.h
SYMLINK-FILE include/rte_esp.h
SYMLINK-FILE include/rte_crypto_sym.h
SYMLINK-FILE include/rte_icmp.h
SYMLINK-FILE include/rte_cryptodev_pmd.h
SYMLINK-FILE include/rte_arp.h
SYMLINK-FILE include/rte_crypto.h
SYMLINK-FILE include/rte_gre.h
SYMLINK-FILE include/rte_crypto_asym.h
SYMLINK-FILE include/rte_cryptodev.h
SYMLINK-FILE include/rte_ether.h
SYMLINK-FILE include/rte_net.h
SYMLINK-FILE include/rte_comp.h
SYMLINK-FILE include/rte_net_crc.h
SYMLINK-FILE include/rte_compressdev.h
SYMLINK-FILE include/rte_compressdev_pmd.h
SYMLINK-FILE include/rte_compressdev_internal.h
CC rte_cryptodev.o
SYMLINK-FILE include/rte_mpls.h
CC rte_cryptodev_pmd.o
CC rte_net.o
CC rte_net_crc.o
CC rte_compressdev.o
CC rte_arp.o
CC rte_compressdev_pmd.o
CC rte_comp.o
AR librte_net.a
INSTALL-LIB librte_net.a
== Build lib/librte_ethdev
AR librte_compressdev.a
SYMLINK-FILE include/rte_ethdev_core.h
SYMLINK-FILE include/rte_ethdev.h
SYMLINK-FILE include/rte_ethdev_driver.h
SYMLINK-FILE include/rte_ethdev_pci.h
SYMLINK-FILE include/rte_ethdev_vdev.h
SYMLINK-FILE include/rte_eth_ctrl.h
SYMLINK-FILE include/rte_tm.h
SYMLINK-FILE include/rte_flow_driver.h
SYMLINK-FILE include/rte_flow.h
SYMLINK-FILE include/rte_dev_info.h
SYMLINK-FILE include/rte_tm_driver.h
SYMLINK-FILE include/rte_mtr.h
INSTALL-LIB librte_compressdev.a
SYMLINK-FILE include/rte_mtr_driver.h
CC ethdev_private.o
CC rte_ethdev.o
CC rte_class_eth.o
CC rte_flow.o
CC rte_tm.o
CC rte_mtr.o
CC ethdev_profile.o
AR librte_cryptodev.a
INSTALL-LIB librte_cryptodev.a
AR librte_ethdev.a
INSTALL-LIB librte_ethdev.a
== Build buildtools
== Build kernel
== Build buildtools/pmdinfogen
== Build kernel/linux
HOSTCC pmdinfogen.o
HOSTLD dpdk-pmdinfogen
INSTALL-HOSTAPP dpdk-pmdinfogen
== Build drivers
== Build drivers/common
== Build drivers/bus
== Build drivers/bus/pci
== Build drivers/bus/vdev
SYMLINK-FILE include/rte_bus_vdev.h
CC vdev.o
CC vdev_params.o
SYMLINK-FILE include/rte_bus_pci.h
CC pci_params.o
CC linux/pci.o
CC pci_common_uio.o
CC pci_common.o
CC linux/pci_uio.o
CC linux/pci_vfio.o
AR librte_bus_vdev.a
INSTALL-LIB librte_bus_vdev.a
AR librte_bus_pci.a
INSTALL-LIB librte_bus_pci.a
== Build drivers/mempool
== Build drivers/mempool/bucket
== Build drivers/mempool/ring
CC rte_mempool_bucket.o
CC rte_mempool_ring.o
AR librte_mempool_ring.a
INSTALL-LIB librte_mempool_ring.a
AR librte_mempool_bucket.a
INSTALL-LIB librte_mempool_bucket.a
== Build drivers/net
== Build drivers/crypto
== Build drivers/compress
== Build app
Build complete [x86_64-native-linuxapp-gcc]
make[3]: Leaving directory '/home/spdk/dpdk'
CC lib/bdev/bdev.o
CC lib/bdev/part.o
CC lib/bdev/scsi_nvme.o
LIB libspdk_bdev.a
CC lib/bdev/error/vbdev_error.o
CC lib/bdev/error/vbdev_error_rpc.o
LIB libspdk_bdev_error.a
CC lib/bdev/gpt/gpt.o
CC lib/bdev/gpt/vbdev_gpt.o
LIB libspdk_bdev_gpt.a
CC lib/bdev/lvol/vbdev_lvol.o
CC lib/bdev/lvol/vbdev_lvol_rpc.o
LIB libspdk_bdev_lvol.a
CC lib/bdev/malloc/bdev_malloc.o
CC lib/bdev/malloc/bdev_malloc_rpc.o
LIB libspdk_bdev_malloc.a
CC lib/bdev/null/bdev_null.o
CC lib/bdev/null/bdev_null_rpc.o
LIB libspdk_bdev_null.a
CC lib/bdev/nvme/bdev_nvme.o
CC lib/bdev/nvme/bdev_nvme_rpc.o
CC lib/bdev/nvme/nvme_rpc.o
LIB libspdk_bdev_nvme.a
CC lib/bdev/passthru/vbdev_passthru.o
CC lib/bdev/passthru/vbdev_passthru_rpc.o
LIB libspdk_bdev_passthru.a
CC lib/bdev/raid/bdev_raid.o
CC lib/bdev/raid/bdev_raid_rpc.o
LIB libspdk_bdev_raid.a
CC lib/bdev/rpc/bdev_rpc.o
LIB libspdk_bdev_rpc.a
CC lib/bdev/split/vbdev_split.o
CC lib/bdev/split/vbdev_split_rpc.o
LIB libspdk_bdev_split.a
CC lib/bdev/ocf/env/src/ocf/ocf_queue.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_wi.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_wt.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_wb.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_fast.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_common.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_rd.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_bf.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_pt.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_discard.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_d2c.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_wa.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_zero.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_inv.o
CC lib/bdev/ocf/env/src/ocf/engine/engine_ops.o
CC lib/bdev/ocf/env/src/ocf/engine/cache_engine.o
CC lib/bdev/ocf/env/src/ocf/ocf_utils.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_hash.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_partition.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_misc.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_raw_dynamic.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_io.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_updater.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_raw.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_raw_volatile.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_raw_atomic.o
CC lib/bdev/ocf/env/src/ocf/metadata/metadata_collision.o
CC lib/bdev/ocf/env/src/ocf/ocf_ctx.o
CC lib/bdev/ocf/env/src/ocf/ocf_stats.o
CC lib/bdev/ocf/env/src/ocf/concurrency/ocf_concurrency.o
CC lib/bdev/ocf/env/src/ocf/concurrency/ocf_cache_concurrency.o
CC lib/bdev/ocf/env/src/ocf/mngt/ocf_mngt_misc.o
CC lib/bdev/ocf/env/src/ocf/mngt/ocf_mngt_core_pool.o
CC lib/bdev/ocf/env/src/ocf/mngt/ocf_mngt_cache.o
CC lib/bdev/ocf/env/src/ocf/mngt/ocf_mngt_core.o
CC lib/bdev/ocf/env/src/ocf/mngt/ocf_mngt_flush.o
CC lib/bdev/ocf/env/src/ocf/mngt/ocf_mngt_io_class.o
CC lib/bdev/ocf/env/src/ocf/mngt/ocf_mngt_common.o
CC lib/bdev/ocf/env/src/ocf/ocf_core.o
CC lib/bdev/ocf/env/src/ocf/ocf_data_obj.o
CC lib/bdev/ocf/env/src/ocf/ocf_cache.o
CC lib/bdev/ocf/env/src/ocf/eviction/eviction.o
CC lib/bdev/ocf/env/src/ocf/eviction/lru.o
CC lib/bdev/ocf/env/src/ocf/cleaning/alru.o
CC lib/bdev/ocf/env/src/ocf/cleaning/cleaning.o
CC lib/bdev/ocf/env/src/ocf/cleaning/acp.o
CC lib/bdev/ocf/env/src/ocf/ocf_logger.o
CC lib/bdev/ocf/env/src/ocf/ocf_trace.o
CC lib/bdev/ocf/env/src/ocf/ocf_stats_builder.o
CC lib/bdev/ocf/env/src/ocf/ocf_io.o
CC lib/bdev/ocf/env/src/ocf/ocf_io_class.o
CC lib/bdev/ocf/env/src/ocf/utils/utils_io.o
CC lib/bdev/ocf/env/src/ocf/utils/utils_cache_line.o
CC lib/bdev/ocf/env/src/ocf/utils/utils_list.o
CC lib/bdev/ocf/env/src/ocf/utils/utils_allocator.o
CC lib/bdev/ocf/env/src/ocf/utils/utils_cleaner.o
CC lib/bdev/ocf/env/src/ocf/utils/utils_part.o
CC lib/bdev/ocf/env/src/ocf/utils/utils_req.o
CC lib/bdev/ocf/env/src/ocf/ocf_metadata.o
CC lib/bdev/ocf/env/ocf_env.o
LIB libspdk_ocfenv.a
CC lib/bdev/ocf/ctx.o
ctx.c:340:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘name’
.name = "OCF SPDK",
^~~~
ctx.c:340:10: warning: initialization of ‘ctx_data_t * ()(uint32_t)’ {aka ‘void * ()(unsigned int)’} from incompatible pointer type ‘char ’ [-Wincompatible-pointer-types]
.name = "OCF SPDK",
^~~~~~~~~~
ctx.c:340:10: note: (near initialization for ‘vbdev_ocf_ctx_ops.data.alloc’)
ctx.c:342:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_alloc’
.data_alloc = vbdev_ocf_ctx_data_alloc,
^~~~~~~~~~
ctx.c:342:16: warning: initialization of ‘int ()(struct ocf_queue )’ from incompatible pointer type ‘ctx_data_t * ()(uint32_t)’ {aka ‘void * ()(unsigned int)’} [-Wincompatible-pointer-types]
.data_alloc = vbdev_ocf_ctx_data_alloc,
^~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:342:16: note: (near initialization for ‘vbdev_ocf_ctx_ops.queue.init’)
ctx.c:343:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_free’
.data_free = vbdev_ocf_ctx_data_free,
^~~~~~~~~
ctx.c:343:15: warning: initialization of ‘int ()(struct ocf_cleaner )’ from incompatible pointer type ‘void ()(ctx_data_t )’ {aka ‘void ()(void )’} [-Wincompatible-pointer-types]
.data_free = vbdev_ocf_ctx_data_free,
^~~~~~~~~~~~~~~~~~~~~~~
ctx.c:343:15: note: (near initialization for ‘vbdev_ocf_ctx_ops.cleaner.init’)
ctx.c:344:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_mlock’
.data_mlock = vbdev_ocf_ctx_data_mlock,
^~~~~~~~~~
ctx.c:344:16: warning: initialization of ‘int ()(struct ocf_metadata_updater )’ from incompatible pointer type ‘int ()(ctx_data_t )’ {aka ‘int ()(void )’} [-Wincompatible-pointer-types]
.data_mlock = vbdev_ocf_ctx_data_mlock,
^~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:344:16: note: (near initialization for ‘vbdev_ocf_ctx_ops.metadata_updater.init’)
ctx.c:345:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_munlock’
.data_munlock = vbdev_ocf_ctx_data_munlock,
^~~~~~~~~~~~
ctx.c:345:18: warning: initialization of ‘int ()(struct ocf_logger )’ from incompatible pointer type ‘void ()(ctx_data_t )’ {aka ‘void ()(void *)’} [-Wincompatible-pointer-types]
.data_munlock = vbdev_ocf_ctx_data_munlock,
^~~~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:345:18: note: (near initialization for ‘vbdev_ocf_ctx_ops.logger.open’)
ctx.c:346:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_rd’; did you mean ‘data’?
.data_rd = vbdev_ocf_ctx_data_rd,
^~~~~~~
data
ctx.c:346:13: warning: excess elements in struct initializer
.data_rd = vbdev_ocf_ctx_data_rd,
^~~~~~~~~~~~~~~~~~~~~
ctx.c:346:13: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:347:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_wr’; did you mean ‘data’?
.data_wr = vbdev_ocf_ctx_data_wr,
^~~~~~~
data
ctx.c:347:13: warning: excess elements in struct initializer
.data_wr = vbdev_ocf_ctx_data_wr,
^~~~~~~~~~~~~~~~~~~~~
ctx.c:347:13: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:348:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_zero’
.data_zero = vbdev_ocf_ctx_data_zero,
^~~~~~~~~
ctx.c:348:15: warning: excess elements in struct initializer
.data_zero = vbdev_ocf_ctx_data_zero,
^~~~~~~~~~~~~~~~~~~~~~~
ctx.c:348:15: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:349:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_seek’
.data_seek = vbdev_ocf_ctx_data_seek,
^~~~~~~~~
ctx.c:349:15: warning: excess elements in struct initializer
.data_seek = vbdev_ocf_ctx_data_seek,
^~~~~~~~~~~~~~~~~~~~~~~
ctx.c:349:15: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:350:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_cpy’; did you mean ‘data’?
.data_cpy = vbdev_ocf_ctx_data_cpy,
^~~~~~~~
data
ctx.c:350:14: warning: excess elements in struct initializer
.data_cpy = vbdev_ocf_ctx_data_cpy,
^~~~~~~~~~~~~~~~~~~~~~
ctx.c:350:14: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:351:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘data_secure_erase’
.data_secure_erase = vbdev_ocf_ctx_data_secure_erase,
^~~~~~~~~~~~~~~~~
ctx.c:351:23: warning: excess elements in struct initializer
.data_secure_erase = vbdev_ocf_ctx_data_secure_erase,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:351:23: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:353:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘queue_init’; did you mean ‘queue’?
.queue_init = vbdev_ocf_ctx_queue_init,
^~~~~~~~~~
queue
ctx.c:353:16: warning: excess elements in struct initializer
.queue_init = vbdev_ocf_ctx_queue_init,
^~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:353:16: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:354:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘queue_kick’; did you mean ‘queue’?
.queue_kick = vbdev_ocf_ctx_queue_kick,
^~~~~~~~~~
queue
ctx.c:354:16: warning: excess elements in struct initializer
.queue_kick = vbdev_ocf_ctx_queue_kick,
^~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:354:16: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:355:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘queue_stop’; did you mean ‘queue’?
.queue_stop = vbdev_ocf_ctx_queue_stop,
^~~~~~~~~~
queue
ctx.c:355:16: warning: excess elements in struct initializer
.queue_stop = vbdev_ocf_ctx_queue_stop,
^~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:355:16: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:357:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘cleaner_init’; did you mean ‘cleaner’?
.cleaner_init = vbdev_ocf_ctx_cleaner_init,
^~~~~~~~~~~~
cleaner
ctx.c:357:18: warning: excess elements in struct initializer
.cleaner_init = vbdev_ocf_ctx_cleaner_init,
^~~~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:357:18: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:358:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘cleaner_stop’; did you mean ‘cleaner’?
.cleaner_stop = vbdev_ocf_ctx_cleaner_stop,
^~~~~~~~~~~~
cleaner
ctx.c:358:18: warning: excess elements in struct initializer
.cleaner_stop = vbdev_ocf_ctx_cleaner_stop,
^~~~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:358:18: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:360:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘metadata_updater_init’; did you mean ‘metadata_updater’?
.metadata_updater_init = vbdev_ocf_dobj_updater_init,
^~~~~~~~~~~~~~~~~~~~~
metadata_updater
ctx.c:360:27: warning: excess elements in struct initializer
.metadata_updater_init = vbdev_ocf_dobj_updater_init,
^~~~~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:360:27: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:361:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘metadata_updater_stop’; did you mean ‘metadata_updater’?
.metadata_updater_stop = vbdev_ocf_dobj_updater_stop,
^~~~~~~~~~~~~~~~~~~~~
metadata_updater
ctx.c:361:27: warning: excess elements in struct initializer
.metadata_updater_stop = vbdev_ocf_dobj_updater_stop,
^~~~~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:361:27: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:362:3: error: ‘const struct ocf_ctx_ops’ has no member named ‘metadata_updater_kick’; did you mean ‘metadata_updater’?
.metadata_updater_kick = vbdev_ocf_dobj_updater_kick,
^~~~~~~~~~~~~~~~~~~~~
metadata_updater
ctx.c:362:27: warning: excess elements in struct initializer
.metadata_updater_kick = vbdev_ocf_dobj_updater_kick,
^~~~~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:362:27: note: (near initialization for ‘vbdev_ocf_ctx_ops’)
ctx.c:339:53: warning: missing braces around initializer [-Wmissing-braces]
static const struct ocf_ctx_ops vbdev_ocf_ctx_ops = {
^
.name = "OCF SPDK",
{ }
ctx.c:342:16:
.data_alloc = vbdev_ocf_ctx_data_alloc,
{ }
.data_free = vbdev_ocf_ctx_data_free,
{ }
.data_mlock = vbdev_ocf_ctx_data_mlock,
{ }
.data_munlock = vbdev_ocf_ctx_data_munlock,
{ }
ctx.c:386:21: error: variable ‘logger’ has initializer but incomplete type
static const struct ocf_logger logger = {
^~~~~~~~~~
ctx.c:387:3: error: ‘const struct ocf_logger’ has no member named ‘printf’
.printf = vbdev_ocf_ctx_log_printf,
^~~~~~
ctx.c:387:12: warning: excess elements in struct initializer
.printf = vbdev_ocf_ctx_log_printf,
^~~~~~~~~~~~~~~~~~~~~~~~
ctx.c:387:12: note: (near initialization for ‘logger’)
ctx.c:388:3: error: ‘const struct ocf_logger’ has no member named ‘dump_stack’
.dump_stack = NULL,
^~~~~~~~~~
ctx.c:388:16: warning: excess elements in struct initializer
.dump_stack = NULL,
^~~~
ctx.c:388:16: note: (near initialization for ‘logger’)
ctx.c: In function ‘vbdev_ocf_ctx_init’:
ctx.c:396:37: warning: passing argument 2 of ‘ocf_ctx_init’ from incompatible pointer type [-Wincompatible-pointer-types]
ret = ocf_ctx_init(&vbdev_ocf_ctx, &vbdev_ocf_ctx_ops);
^~~~~~~~~~~~~~~~~~
In file included from /home/spdk/lib/bdev/ocf/env/include/ocf/ocf_cache.h:17,
from /home/spdk/lib/bdev/ocf/env/include/ocf/ocf.h:22,
from ctx.c:33:
/home/spdk/lib/bdev/ocf/env/include/ocf/ocf_ctx.h:339:63: note: expected ‘const struct ocf_ctx_config *’ but argument is of type ‘const struct ocf_ctx_ops *’
int ocf_ctx_init(ocf_ctx_t *ctx, const struct ocf_ctx_config *cfg);
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~
ctx.c:401:2: warning: implicit declaration of function ‘ocf_ctx_set_logger’; did you mean ‘ocf_core_set_name’? [-Wimplicit-function-declaration]
ocf_ctx_set_logger(vbdev_ocf_ctx, &logger);
^~~~~~~~~~~~~~~~~~
ocf_core_set_name
ctx.c: At top level:
ctx.c:386:32: error: storage size of ‘logger’ isn’t known
static const struct ocf_logger logger = {
^~~~~~
make[3]: *** [/home/spdk/mk/spdk.common.mk:320: ctx.o] Error 1
make[2]: *** [/home/spdk/mk/spdk.subdirs.mk:44: ocf] Error 2
make[1]: *** [/home/spdk/mk/spdk.subdirs.mk:44: bdev] Error 2
make: *** [/home/spdk/mk/spdk.subdirs.mk:44: lib] Error 2
[root@Target spdk]#
===if I change OCF_METADATA_HASH_DEBUG as 1.
#define OCF_METADATA_HASH_DEBUG 0
#if 1 == OCF_METADATA_HASH_DEBUG
#define OCF_DEBUG_TRACE(cache)
ocf_cache_log(cache, log_info, "[Metadata][Hash] %s\n", func)
#define OCF_DEBUG_PARAM(cache, format, ...)
ocf_cache_log(cache, log_info, "[Metadata][Hash] %s - "format"\n",
func, ##VA_ARGS)
#else
===== then I suffer below build error ==== could you please help?
===== or can you show me to how to understand and debug metadata hash map easily?
and also I see there is RB-Tree in the code, what is it for?
gcc -g -Wall -Iinclude/ -Isrc//ocf/env/ -c -o src/ocf/eviction/eviction.o src/ocf/eviction/eviction.c
gcc -g -Wall -Iinclude/ -Isrc//ocf/env/ -c -o src/ocf/eviction/lru.o src/ocf/eviction/lru.c
gcc -g -Wall -Iinclude/ -Isrc//ocf/env/ -c -o src/ocf/metadata/metadata.o src/ocf/metadata/metadata.c
gcc -g -Wall -Iinclude/ -Isrc//ocf/env/ -c -o src/ocf/metadata/metadata_collision.o src/ocf/metadata/metadata_collision.c
gcc -g -Wall -Iinclude/ -Isrc//ocf/env/ -c -o src/ocf/metadata/metadata_hash.o src/ocf/metadata/metadata_hash.c
src/ocf/metadata/metadata_hash.c:2:8: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘-’ token
Connection closed by foreign host.
Disconnected from remote host(10.239.70.152-lab) at 20:40:05.
Type `help' to learn how to use Xshell prompt.
[C:~]$
Allocation is an option about the IO Class Configuration in the enterprise edition Intel CAS.
But I can't find the same configure option in the open-cas.
/**
* @brief IO class configuration
*/
struct ocf_mngt_io_class_config {
/**
* @brief IO class ID
*/
uint32_t class_id;
/**
* @brief IO class name
*/
const char *name;
/**
* @brief IO class eviction priority
*/
int16_t prio;
/**
* @brief IO class cache mode
*/
ocf_cache_mode_t cache_mode;
/**
* @brief IO class minimum size
*/
uint32_t min_size;
/**
* @brief IO class maximum size
*/
uint32_t max_size;
};
Suppose I had a core device /dev/core that contains some metadata at 0-1GB, and I wanted this 1GB metadata to be always resident in the cache, how should I make it happen?
I've thought about using IO classification, but it seemed that the only available IO class name for this scenario was “lba”, so I don't know if it is the answer to my problem, and if it is, how should I write the classification configuration file?
I am getting NULL in io->volume->cache
during ocf_core_validate_io()
when sending reads immediately after initialization (in callback of ocf_mngt_cache_add_core_finish()
)
Full backtrace:
(gdb) bt
#0 0x00005555555e1d95 in ocf_core_validate_io (io=0x200027228190) at src/ocf/ocf_core.c:202
#1 0x00005555555e1ec6 in ocf_core_submit_io_mode (io=0x200027228190, cache_mode=ocf_cache_mode_none) at src/ocf/ocf_core.c:233
#2 0x00005555555e2515 in ocf_core_volume_submit_io (io=0x200027228190) at src/ocf/ocf_core.c:395
#3 0x00005555555e0a5d in ocf_volume_submit_io (io=0x200027228190) at src/ocf/ocf_volume.c:237
#4 0x00005555555b701b in ocf_core_submit_io (io=0x200027228190) at /home/vmysak/samba_share0/cas/hyb/lib/bdev/ocf/env/include/ocf/ocf_core.h:156
#5 0x00005555555b7e7c in io_submit_to_ocf (bdev_io=0x200007043c00, io=0x200027228190) at vbdev_ocf.c:576
#6 0x00005555555b8026 in io_handle (ch=0x555555dc6cd0, bdev_io=0x200007043c00) at vbdev_ocf.c:625
#7 0x00005555555b80ba in vbdev_ocf_get_buf_cb (ch=0x555555dc6cd0, bdev_io=0x200007043c00, success=true) at vbdev_ocf.c:653
#8 0x0000555555621c36 in spdk_bdev_io_get_buf (bdev_io=0x200007043c00, cb=0x5555555b8074 <vbdev_ocf_get_buf_cb>, len=32768) at bdev.c:596
#9 0x00005555555b8116 in vbdev_ocf_submit_request (ch=0x555555dc6cd0, bdev_io=0x200007043c00) at vbdev_ocf.c:664
#10 0x00005555556242e3 in _spdk_bdev_io_submit (ctx=0x200007043c00) at bdev.c:1693
#11 0x000055555562462e in spdk_bdev_io_submit (bdev_io=0x200007043c00) at bdev.c:1741
#12 0x000055555562646b in spdk_bdev_read_blocks (desc=0x555555dc6bc0, ch=0x555555dc6c00, buf=0x200027002b80, offset_blocks=0, num_blocks=64, cb=0x5555555ae364 <spdk_gpt_bdev_complete>, cb_arg=0x555555dc6af0) at bdev.c:2595
#13 0x0000555555626326 in spdk_bdev_read (desc=0x555555dc6bc0, ch=0x555555dc6c00, buf=0x200027002b80, offset=0, nbytes=32768, cb=0x5555555ae364 <spdk_gpt_bdev_complete>, cb_arg=0x555555dc6af0) at bdev.c:2563
#14 0x00005555555ae699 in vbdev_gpt_read_gpt (bdev=0x555555dc3ea0) at vbdev_gpt.c:477
#15 0x00005555555ae810 in vbdev_gpt_examine (bdev=0x555555dc3ea0) at vbdev_gpt.c:528
#16 0x0000555555629192 in spdk_bdev_start (bdev=0x555555dc3ea0) at bdev.c:3740
#17 0x00005555556291d7 in spdk_bdev_register (bdev=0x555555dc3ea0) at bdev.c:3751
#18 0x00005555555b8ce9 in register_ocf_bdev (vbdev=0x555555dc3d20) at vbdev_ocf.c:1051
#19 0x00005555555b6e94 in vbdev_ocf_mngt_continue (vbdev=0x555555dc3d20, status=0) at utils.c:128
#20 0x00005555555b8905 in add_core_cmpl (cache=0x2000142273c0, core=0x200014227b80, priv=0x555555dc3d20, error=0) at vbdev_ocf.c:913
#21 0x00005555555d7e26 in ocf_mngt_cache_add_core_finish (pipeline=0x20000fc1fa00, priv=0x20000fc1fa28, error=0) at src/ocf/mngt/ocf_mngt_core.c:525
#22 0x00005555555f8512 in _ocf_pipeline_run_step (req=0x200014628180) at src/ocf/utils/utils_pipeline.c:47
#23 0x00005555555e2f3d in ocf_io_handle (io=0x0, opaque=0x200014628180) at src/ocf/ocf_queue.c:74
#24 0x00005555555e303b in ocf_queue_run_single (q=0x20001c2001c0) at src/ocf/ocf_queue.c:94
#25 0x00005555555b84a8 in mngt_queue_poll (opaque=0x20001c2001c0) at vbdev_ocf.c:796
#26 0x0000555555657b9e in spdk_thread_poll (thread=0x555555dc2800, max_msgs=0, now=6274227700624692) at thread.c:456
#27 0x00005555556151d4 in _spdk_reactor_run (arg=0x555555dc25c0) at reactor.c:244
#28 0x00005555556154f6 in spdk_reactors_start () at reactor.c:333
#29 0x0000555555613d79 in spdk_app_start (opts=0x7fffffffe270, start_fn=0x555555564450 <spdk_startup>, arg1=0x0) at app.c:677
#30 0x00005555555645e5 in main (argc=3, argv=0x7fffffffe408) at iscsi_tgt.c:112
Possible cause:
Address of io->volume in ocf_core_validate_io()
is greater by 1 byte than address of core->volume in ocf_mngt_cache_add_core_finish()
Also, removing check at #202 in ocf_core_validate_io()
silenced this error and everything worked OK
Kernel warning during nhit promotion policy operations:
[21563.552417] WARNING: CPU: 15 PID: 0 at block/blk-core.c:3255 blk_end_bidi_request+0x68/0x70
[21563.561406] Modules linked in: cas_cache(O) cas_disk(O) xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink bpfilter nls_utf8 rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs lockd grace fscache sunrpc intel_rapl iosf_mbi iTCO_wdt ioatdma mxm_wmi iTCO_vendor_support sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel kvm_intel i2c_i801 lpc_ich intel_pch_thermal ipmi_si ipmi_devintf ipmi_msghandler wmi pcc_cpufreq acpi_pad ixgbe igb mdio i2c_algo_bit ptp crc32c_intel nvme i2c_core pps_core nvme_core dca xhci_pci xhci_hcd
[21563.629400] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G O 4.19.56-t2.el7.twitter.x86_64 #1
[21563.639501] Hardware name: Supermicro SYS-5018D-FN4T/X10SDV-8C-TLN4F, BIOS 1.1c 10/03/2016
[21563.648390] RIP: 0010:blk_end_bidi_request+0x68/0x70
[21563.653675] Code: 43 00 89 de 48 89 ef 49 89 c5 e8 63 fe ff ff 49 8b bc 24 c0 01 00 00 4c 89 ee e8 33 38 43 00 31 d2 5b 89 d0 5d 41 5c 41 5d c3 <0f> 0b eb b0 0f 1f 40 00 0f 1f 44 00 00 48 8b 07 48 83 b8 00 01 00
[21563.673317] RSP: 0018:ffff9c95e7bc3d68 EFLAGS: 00010286
[21563.678859] RAX: 0000000000000000 RBX: ffff9c95e44892d8 RCX: 0000000000000000
[21563.686341] RDX: 0000000000001000 RSI: 0000000000000000 RDI: ffff9c95a1b417c0
[21563.693795] RBP: ffff9c95a1b417c0 R08: ffff9c95e1316a00 R09: 0000000000001000
[21563.701293] R10: ffff9c95e051ff98 R11: ffff9c95e1316450 R12: ffff9c95e3935658
[21563.708745] R13: ffff9c95e13163f8 R14: 0000000000000000 R15: 0000000000000000
[21563.716217] FS: 0000000000000000(0000) GS:ffff9c95e7bc0000(0000) knlGS:0000000000000000
[21563.724885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[21563.730946] CR2: 00007fc2c8894630 CR3: 000000015a20a004 CR4: 00000000003606e0
[21563.738426] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[21563.745874] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[21563.753359] Call Trace:
[21563.756135]
[21563.758498] blk_end_request_all+0x1f/0x40
[21563.762919] block_dev_complete_rq+0x6f/0x90 [cas_cache]
[21563.768592] ? ocf_req_complete+0xeb/0x150 [cas_cache]
[21563.774070] ? _ocf_read_generic_miss_complete+0x75/0xd0 [cas_cache]
[21563.780744] ? ocf_submit_volume_req_cmpl+0x16/0x20 [cas_cache]
[21563.787000] ? cas_bd_io_end+0x37/0x50 [cas_cache]
[21563.792128] ? cas_bd_io_end_callback+0x5e/0xb0 [cas_cache]
[21563.798037] ? blk_update_request+0x78/0x2c0
[21563.802626] ? scsi_end_request+0x2c/0x240
[21563.807054] ? scsi_io_completion+0x88/0x610
[21563.811640] ? blk_done_softirq+0x84/0xa0
[21563.815980] ? __do_softirq+0xe2/0x2dd
[21563.820064] ? irq_exit+0xc5/0xd0
[21563.823691] ? do_IRQ+0x85/0xd0
[21563.827169] ? common_interrupt+0xf/0xf
[21563.831350]
[21563.833764] ? cpuidle_enter_state+0xa2/0x310
[21563.838468] ? cpuidle_enter_state+0x93/0x310
[21563.843163] ? do_idle+0x1b6/0x210
[21563.846885] ? cpu_startup_entry+0x5f/0x70
[21563.851339] ? start_secondary+0x182/0x1c0
[21563.855751] ? secondary_startup_64+0xa4/0xb0
[21563.860455] ---[ end trace 650fe3afc3dad17c ]---
[21563.865452] WARNING: CPU: 15 PID: 0 at block/blk-core.c:3200 blk_finish_request+0x11c/0x150
[21563.874422] Modules linked in: cas_cache(O) cas_disk(O) xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink bpfilter nls_utf8 rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs lockd grace fscache sunrpc intel_rapl iosf_mbi iTCO_wdt ioatdma mxm_wmi iTCO_vendor_support sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel kvm_intel i2c_i801 lpc_ich intel_pch_thermal ipmi_si ipmi_devintf ipmi_msghandler wmi pcc_cpufreq acpi_pad ixgbe igb mdio i2c_algo_bit ptp crc32c_intel nvme i2c_core pps_core nvme_core dca xhci_pci xhci_hcd
[21563.942349] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G W O 4.19.56-t2.el7.twitter.x86_64 #1
[21563.952448] Hardware name: Supermicro SYS-5018D-FN4T/X10SDV-8C-TLN4F, BIOS 1.1c 10/03/2016
[21563.961327] RIP: 0010:blk_finish_request+0x11c/0x150
[21563.966608] Code: 6a 1a 00 00 48 8b 43 40 48 8d 53 40 48 39 c2 0f 84 4a ff ff ff 0f 0b 4c 89 ee 48 89 df e8 7c eb 00 00 8b 43 18 e9 1c ff ff ff <0f> 0b e9 07 ff ff ff 0f b6 43 14 83 e8 20 83 f8 03 0f 86 2c ff ff
[21563.986225] RSP: 0018:ffff9c95e7bc3d40 EFLAGS: 00010086
[21563.991768] RAX: 0000139c70051421 RBX: ffff9c95a1b417c0 RCX: 0000000000000018
[21563.999235] RDX: 000008a5cc704076 RSI: 002327a315fb43d7 RDI: ffffffff85234040
[21564.006687] RBP: ffff9c95e3935658 R08: 0000000000000001 R09: 0000000000000000
[21564.014157] R10: ffff9c95e051ea00 R11: 0000000000006500 R12: 0000000000000000
[21564.021610] R13: 0000139c70051421 R14: 0000000000000000 R15: 0000000000000000
[21564.029081] FS: 0000000000000000(0000) GS:ffff9c95e7bc0000(0000) knlGS:0000000000000000
[21564.037776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[21564.043837] CR2: 00007fc2c8894630 CR3: 000000015a20a004 CR4: 00000000003606e0
[21564.051319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[21564.058773] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[21564.066244] Call Trace:
[21564.069042]
[21564.071388] blk_end_bidi_request+0x4d/0x70
[21564.075884] blk_end_request_all+0x1f/0x40
[21564.080336] block_dev_complete_rq+0x6f/0x90 [cas_cache]
[21564.085990] ? ocf_req_complete+0xeb/0x150 [cas_cache]
[21564.091464] ? _ocf_read_generic_miss_complete+0x75/0xd0 [cas_cache]
[21564.098159] ? ocf_submit_volume_req_cmpl+0x16/0x20 [cas_cache]
[21564.104427] ? cas_bd_io_end+0x37/0x50 [cas_cache]
[21564.109573] ? cas_bd_io_end_callback+0x5e/0xb0 [cas_cache]
[21564.115491] ? blk_update_request+0x78/0x2c0
[21564.120092] ? scsi_end_request+0x2c/0x240
[21564.124535] ? scsi_io_completion+0x88/0x610
[21564.129142] ? blk_done_softirq+0x84/0xa0
[21564.133479] ? __do_softirq+0xe2/0x2dd
[21564.137560] ? irq_exit+0xc5/0xd0
[21564.141212] ? do_IRQ+0x85/0xd0
[21564.144673] ? common_interrupt+0xf/0xf
[21564.148818]
[21564.151271] ? cpuidle_enter_state+0xa2/0x310
[21564.155980] ? cpuidle_enter_state+0x93/0x310
[21564.160654] ? do_idle+0x1b6/0x210
[21564.164400] ? cpu_startup_entry+0x5f/0x70
[21564.168816] ? start_secondary+0x182/0x1c0
[21564.173244] ? secondary_startup_64+0xa4/0xb0
[21564.177915] ---[ end trace 650fe3afc3dad17d ]---
In Open CAS there is a feature called "promotion" which didn't existed in Intel CAS, I've went through the documentation and it said "This parameter specifies the desired promotion policy of core lines", but still I don't know what it is and how this parameter would affect core line. Maybe Adding some little extra description of such concept to the "Basic concepts" section of this page : [https://open-cas.github.io/ocf_intro.html](OpenCAS Introduction) would be very helpful?
The initial start after configuring the opencas.conf file is ok; all devices are created as expected;
We are using raid0 via hardware, and the we identify the device with a consistent name across reboots;
We defined 1 raid device per cache; and we cannot get a clean reboot;
Also we tried to add core and cache devices to lvm global filter rules;
checking the journal, there are mentions that the core device was busy and will be attached as inactive; but it remains also as detached
The interesting part is that the failed initialization conducts to an emergency boot, in which we can see that some cache/core are assembled;
checking with casadm -L, shows that the same core device that failed is observed twice, once as detached, and again under its respective cache device as inactive;
using casctl stop; casctl init --force solve the issue;
also removing the detached cores, doing casctl stop, casctl init, also fix the issue;
but, casctl stop and casctl init without removing hte detached cores, prevents those to initiate correctly;
The fastest route is rename the opencas.conf file to avoid the initialization, let the machine boot, rename back, we can start the cache without any issue, but it has to be manually performed, and do not allow to map the drives at /etc/fstab
another good option is to type the root password to get into the emergency mode; casctl stop; casadm -L; casadm --remove-detached -d /dev/xxxx; casctl start;
LOGS from pyocf
Test name: test_start_read_first_and_check_mode
I encountered this several times, however it is very sporadic. Seems like it isn't related to hardware configuration.
When using nhit promotion policy some requests are never completed (hung requests).
When using default promotion policy everything works fine.
Steps to reproduce:
Fio config:
[JESD219]
ioengine=libaio
direct=1
rw=randread
norandommap
randrepeat=0
iodepth=256
numjobs=4
bssplit=512/4:1024/1:1536/1:2048/1:2560/1:3072/1:3584/1:4k/67:8k/10:16k/7:32k/3:64k/3
blockalign=4k
random_distribution=zoned:50/5:30/15:20/80
filename=${FILE}
group_reporting=1
iostat output at the end of workload:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util cas1-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 622.00 0.00 0.00 0.00 0.00 100.00
root@localhost:~# uname -a
Linux localhost 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@localhost:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial
root@localhost:~# casadm --version
+=========================+===================+
| Name | Version |
+=========================+===================+
| CAS Cache Kernel Module | 19.09.00.00001216 |
| CAS Disk Kernel Module | 19.09.00.00001216 |
| CAS CLI Utility | 19.09.00.00001216 |
+=========================+===================+
root@localhost:~/open-cas-linux# git show
commit 40eaa3bd78121db778fc8dfae7b1a71072d52b10
root@localhost:~/open-cas-linux/ocf# git show
commit 4e5530851b0e7d15a4c5883c5d96c4065f0aa34f
root@localhost:~# casadm -P -i 1 -f conf
Cache Id 1
Cache Size 477657 [4KiB Blocks] / 1.82 [GiB]
Cache Device /dev/nvme0n1
Core Devices 1
Inactive Core Devices 0
Write Policy wt
Eviction Policy lru
Cleaning Policy alru
Promotion Policy nhit
Cache line size 4 [KiB]
Metadata Memory Footprint 42.5 [MiB]
Dirty for 0 [s] / Cache clean
Metadata Mode normal
Status Running
root@localhost:~# casadm -P -i 1 -j 1 -f conf
Core Id 1
Core Device /dev/sdc1
Exported Object /dev/cas1-1
Core Size 1952768 [4KiB Blocks] / 7.45 [GiB]
Dirty for 0 [s] / Cache clean
Status Active
Seq cutoff threshold 1024 [KiB]
Seq cutoff policy full
It might be related to not aligned sizes.
When power failure occurs during management operation like adding/removing core or changing I/O class configuration, it's possible that on next cache load invalid checksum error would happen.
It's possible that ocf_medatata_hash_flush_segment() step in ocf_metadata_hash_flush_sb_pipeline_props pipeline would flush metadata_segment_sb_config with checksums calculated based on new metadata content, but won't be able to flush actual segments content because of power failure. In this case on disk metadata would contain new checksums for all segments from this pipeline, but old content for one or more of them.
This seems to occur in random configurations once a while. Couldn't get a local reproduction.
SIGSEGV is returned when trying to set promotion policy to invalid value.
Example test is in #264 test_neg_set_promotion_policy
There is a clear mention in the documentation of how control evection for folders and file extension.
Is it possible to use wildcards and files too? so, we can effectively create exclusion/inclusions in the best fashion as possible?
OCF does not issue FLUSH request after flushing metadata to cache device. This could lead to lost of recent metadata state on drives that don't have PLI protection and do have write buffering enabled.
In some scenarios, we can know ahead that a high usage of the cache will be required.
Hence, in this situation, would be very interesting the capability to trigger the eviction in the same way the flush can be triggered.
With that, high performance can be achieved. We suggest a command to evict up a cache percentage defined by while casting the command.
Eviction is triggered upon writing to cache with request bigger than cache size.
IO should go directly to core and eviction should not occur.
Test for this particular case can be found in #199
They allocate the numbers instead of creating a generator. This might be the cause of a serious bottleneck while running full security tests suite. We would also like to have the ability to exclude certain ranges from the generator.
This occurs repeatedly in test_secure_erase_simple_io_read_misses
Tests included in #264
ocf_core_submit_io
ocf_volume_submit_io
volume->type->properties->ops.submit_io
ocf_core_volume_submit_io
ocf_engine_hndl_req
ocf_queue_run
ocf_io_handle
ocf_write_wt // req->io_if->write
_ocf_write_wt_do
_ocf_write_wt_submit
ocf_submit_cache_reqs
ocf_volume_submit_io
Looks like the final ocf_volume_submit_io
traverses back to the beginning.
So I guess the CAS IO is a run-to-completion design? And where should it block if there is no incoming IO?
When trying to load dirty shutdown wb cache it is possible to use force
flag, which bypasses the dirty_shutdown
bit check in attach and then fails the CRC check for collision
segment.
Reject attempt to load the cache with force
flag set, or silently ignore it - it doesn't make sense in this context.
I wanted to use cleaning-alru policy and achieve the following goals:
The cache line size was 8KB so I thought the flush_max_buffers should be 40 * 1024 / 8 which is 5120
so I set the cleaning-alru policy with the following command:
casadm -X -n cleaning-alru -i 1 -w 30 -b 5120 -s 120
and then I dd-ed about 1GB data into /dev/cas1-1
But it turned out that after the cleaning thread woke up, instead of flushing 40MiB every 30 seconds, it flushed the whole 1GB data in nearly 14 seconds, so:
I test the IO pattern for direct sync IO in fio, the writeback cache device seems to be very slow. can anyone help to explain why? and how to optimaze it?
fio configuration is below
[global]
name=fio-rand-write
filename=/dev/cas1-12:/dev/cas1-1:/dev/cas1-2:/dev/cas1-3:/dev/cas1-4:/dev/cas1-5:/dev/cas1-6:/dev/cas1-7:/dev/cas1-8:/dev/cas1-9:/dev/cas1-10:/dev/cas1-11
rw=randwrite
#rate_iops=40000
bs=8K
direct=1
fdatasync=1
numjobs=1
time_based=1
runtime=1000000000
norandommap
random_distribution=zipf:1.2
[file1]
ioengine=libaio
iodepth=32
ocf_mngt_cache_lock() increments cache reference counter. When cache is being stopped, outstanding waiters are woken up if cache reference count drops to 0. This will actually never happen as the waiters are holding reference to cache object.
To fix this it should probably be enough to wake up outstanding waiters unconditionally during cache stop.
We are consistently facing soft lockups in two machines that are near identical (one is v4, another is v2, one is LRDIM another RDIM); the third machine is doing ok with the very same cache device. All have a private LAN with Infiniband, Ubuntu 18, Intel 3605 1.5TB as cache;
The main diference is: the 2 machines with issue are x10qbi 4 x cpus, 1TB RAM DDR3; the one that is fine is x9dri (2 x cpus) 386 GB RAM DDR3, all on Ubuntu 18, another difference, the machine that is ok is running raid0 (4 disks x 2 TB), the 2 with issues, faced problems with RAID10 and RAID5, all adaptec raid cards (8 disks x 3 TB);
watchdog: BUG: soft lockup - CPU#51 stuck for 22s! [pacd:140354]
Modules linked in: macvlan cfg80211 ceph libceph fscache aufs overlay rdma_ucm(OE) ib_ucm(OE) ib_ipoib(OE) ib_umad(OE) esp6_offload esp6 esp4_offload esp4 xfrm_algo mlx5_fpga_
mdev vfio_iommu_type1 vfio mdev(OE) mlx4_en(OE) bonding xfs nls_iso8859_1 intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass ipmi_ssif i
k(OE) intel_rapl_perf mei_me lpc_ich mei shpchp ioatdma ipmi_si mac_hid ipmi_devintf ipmi_msghandler acpi_pad sch_fq_codel nf_nat_pptp nf_nat_proto_gre nf_conntrack_pptp nf_co
dma_cm(OE) iw_cm(OE) ib_cm(OE) iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi knem(OE) ip_tables x_tables autofs4
btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_ib(OE) ib_uverbs(OE) ib
f_pclmul crc32_pclmul ghash_clmulni_intel ast hid_generic i2c_algo_bit pcbc ttm usbhid aesni_intel hid drm_kms_helper aes_x86_64 crypto_simd syscopyarea glue_helper ixgbe sysf
sys_fops ptp nvme_core devlink aacraid ahci pps_core drm mlx_compat(OE) libahci mdio wmi
CPU: 51 PID: 140354 Comm: pacd Tainted: G OE 4.15.0-29-generic #31-Ubuntu
Hardware name: Supermicro PIO-848B-TRF4T-ST031/X10QBI, BIOS 3.2a 08/08/2019
RIP: 0010:_raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8d09bf2c3d68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff11
RAX: 00000000ff5e4a5d RBX: ffff8d49ab049b68 RCX: ffff8d09bf2c3d98
RDX: ffff8d49ab049b70 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff8d09bf2c3d68 R08: 000000000000000e R09: 0000000000000000
R10: ffff8d09bf2c3c70 R11: 0000000000000073 R12: 00000000ff5e4a5d
R13: 0000000000000202 R14: 0000000000000000 R15: 0000000000000000
FS: 00007ff6c3e9e700(0000) GS:ffff8d09bf2c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000152b4038 CR3: 0000003dc5766001 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<IRQ>
__wake_up_common_lock+0x8e/0xc0
__wake_up+0x13/0x20
rwb_wake_all+0x2c/0x60
scale_up.part.20+0x28/0x40
wb_timer_fn+0x21b/0x3d0
? blk_mq_tag_update_depth+0x110/0x110
blk_stat_timer_fn+0x147/0x150
call_timer_fn+0x30/0x130
run_timer_softirq+0x3fb/0x450
? ktime_get+0x43/0xa0
? lapic_next_deadline+0x26/0x30
__do_softirq+0xdf/0x2b2
irq_exit+0xb6/0xc0
smp_apic_timer_interrupt+0x71/0x130
apic_timer_interrupt+0x84/0x90
</IRQ>
RIP: 0010:_raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9a6f5aa7fa68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff11
RAX: 0000000000000202 RBX: ffff9a6f5aa7fac0 RCX: ffff8c8805690000
RDX: 0000000000000000 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff9a6f5aa7fa68 R08: 0000000000d00000 R09: ffff8d09bf663440
R10: 0000000000000000 R11: 0000000000000082 R12: ffff8d49ab049b68
R13: 0000000000000002 R14: 0000000000000060 R15: ffff8d49ab049b00
prepare_to_wait_exclusive+0x72/0x80
wbt_wait+0x137/0x350
? wait_woken+0x80/0x80
blk_mq_make_request+0xe0/0x570
generic_make_request+0x124/0x300
submit_bio+0x73/0x150
? submit_bio+0x73/0x150
? xfs_setfilesize_trans_alloc.isra.13+0x3e/0x90 [xfs]
xfs_submit_ioend+0x87/0x1c0 [xfs]
xfs_vm_writepages+0xd1/0xf0 [xfs]
do_writepages+0x4b/0xe0
? iomap_write_begin.constprop.18+0x140/0x140
? iomap_file_buffered_write+0x6e/0xa0
? iomap_write_begin.constprop.18+0x140/0x140
? xfs_iunlock+0xf8/0x100 [xfs]
__filemap_fdatawrite_range+0xc1/0x100
? __filemap_fdatawrite_range+0xc1/0x100
file_write_and_wait_range+0x5a/0xb0
xfs_file_fsync+0x5f/0x230 [xfs]
vfs_fsync_range+0x51/0xb0
do_fsync+0x3d/0x70
SyS_fdatasync+0x13/0x20
do_syscall_64+0x73/0x130
entry_SYSCALL_64_after_hwframe+0x3d/0xa2
RIP: 0033:0x7ff729d273e7
RSP: 002b:00007ff6c3d9ec50 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
RAX: ffffffffffffffda RBX: 0000000000000027 RCX: 00007ff729d273e7
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000027
RBP: 00007ff6c3d9ed30 R08: 0000000000000000 R09: 000000000000002c
R10: 00007ff6bc0008d0 R11: 0000000000000293 R12: 00007ff6c3d9ece0
R13: 00007ff6bc010965 R14: 00007ff6c3d9ed60 R15: 00007ff6bc00e3f0
Code: 47 76 ff ff ff 7f 89 d0 5d c3 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f
Line 107 in 7f7e8bc
Hi,
I have an issue that using ocf_submit_discard
ends up in some allocator objects not being deallocated
After doing ocf_cache_stop
-> ocf_ctx_exit
I see in env_allocator_destroy
that allocated object counter is > 0.
Same is not the case for reads, writes and flushes - all objects are getting freed by the end of applictation as it expected
I use Write-Through mode with one cache device and one core device
My environment for OCF is this implementation in SPDK: https://review.gerrithub.io/#/c/spdk/spdk/+/435708/
Where can I find docs about the detail design? For example, metadata, IO engine, cache-line, etc.
It's very difficult to understand the code barely by comments.
From code inspection, I found that env/postix/ocf_env.h
isn't quite right.
#include <stdio.h>
#include <string.h>
#define min(a, b) ((a < b) ? (a) : (b))
#define env_strncpy(dest, dmax, src, slen) ({ \
strncpy(dest, src, min(dmax - 1, slen)); \
dest[dmax - 1] = '\0'; \
0; \
})
int
main(void)
{
char buf[16] = "0123456789abcde";
char *s = "###";
(void) printf(" before: '%s'\n", buf);
env_strncpy(buf, sizeof (buf), s, strlen(s));
(void) printf(" after: '%s'\n", buf);
return (0);
}
$ ./t
before: '0123456789abcde'
after: '###3456789abcde'
My example program is fixed with the following:
@@ -4,7 +4,7 @@
#define min(a, b) ((a < b) ? (a) : (b))
#define env_strncpy(dest, dmax, src, slen) ({ \
strncpy(dest, src, min(dmax - 1, slen)); \
- dest[dmax - 1] = '\0'; \
+ dest[min(dmax - 1, slen)] = '\0'; \
0; \
})
I believe a similar issue exists with env_strncmp()
, but I have not analyzed callers to understand their intent.
I have one question about alru cleaning while reading codes about alru cleaning recently. strcut alru_cleaning_policy_meta could be accessed concurrently when doing alru cleaning and when also a write request hits cache and ocf_cleaning_set_hot_cache_line is called to update alru list. So it seems possible that when cleaning thread traverse alru list, the lru_prev or lru_next could be changed in case of cache line hit. I think there should be a lock or something to handle this race condition. But I could not find it in ofc codes. Am I missing anything? Could you please help to explain that?
Hi there,
I have a question related to the OCF codebase performance/ scalability. (not related to cache behaviors)
Specifically, Im using some simple code to evaluate OCF's performance. The context logic is same as the example context, except:
However when we experiment with different number of threads, OCF looks like not scalable, e.g.:
1 thread: 700K requests/s
2 threads: 500K requests/s
4 threads: 360K requests/s
Im wondering:
Thanks,
Kan
Multiple problems occurred. High reproduction rate
test_neg_write_too_long_data
- SIGABRTtest_neg_read_too_long_data
- SIGSEGVtest_neg_write_too_far
- SIGABRTtest_neg_read_too_far
- SIGSEGVtest_neg_write_offset_outside_of_device
- fails for any input (TypeError)test_neg_read_offset_outside_of_device
- fails for any input (TypeError)Sorting may take long time on bigger caches, which unnecessarily elongates flush time on core drives that perform well under random workload.
For ACP cleaning policy, there are two parameters which are "-w" and "-b", it means:
[-w, --wake-up <NUM>]: Period of time between awakenings of flushing thread in milliseconds. MIN: 0, MAX: 10000
(inclusive), DEFAULT: 10.
[-b, --flush-max-buffers <NUM>]: Number of dirty cache blocks to be flushed in one cleaning cycle. MIN: 1, MAX:
10000 (inclusive), DEFAULT: 128.
But seems there is another factor which may determine the flush speed of ACP:
void cleaning_policy_acp_perform_cleaning(ocf_cache_t cache,
ocf_cleaner_end_t cmpl)
{
struct acp_cleaning_policy_config *config;
struct acp_context *acp = _acp_get_ctx_from_cache(cache);
struct acp_state *state = &acp->state;
acp->cmpl = cmpl;
if (!state->in_progress) {
/* get next chunk to clean */
state->chunk = _acp_get_cleaning_candidate(cache);
if (!state->chunk) {
/* nothing co clean */
cmpl(&cache->cleaner, ACP_BACKOFF_TIME_MS);
return;
}
/* new cleaning cycle - reset state */
state->iter = 0;
state->in_progress = true;
}
ACP_DEBUG_INIT(acp);
config = (void *)&cache->conf_meta->cleaning[ocf_cleaning_acp].data;
if (_acp_prepare_flush_data(acp, config->flush_max_buffers))
_acp_flush(acp);
else
_acp_flush_end(acp, 0);
}
Or in other words, with our current coding, here is how ACP works:
Dear experts,
this is Yale Wang. my partner AiSHu reports an issue which can reproduce frequently.
once they make server lose power with unplugging power cord, the OpenCAS will fail to initialize.
i have a pdf include detail server configuration, steps to reproduce the issue, and the screen snapshot showing log.
thanks
after unexpected power lose.pdf
Seems to occur on low end instances - AWS EC2 t3.medium (2vCPU, 4GB RAM) in this case
Command executed: pytest --ignore=tests/security
Gdb output:
tests/basic/test_pyocf.py::test_ctx_fixture PASSED [ 0%]
tests/basic/test_pyocf.py::test_simple_wt_write [New Thread 0x7fffe6113700 (LWP 2327)]
[New Thread 0x7fffe57ec700 (LWP 2328)]
[New Thread 0x7fffe4feb700 (LWP 2329)]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe4feb700 (LWP 2329)]
0x00007fffea5b0605 in env_atomic64_add (i=17, a=0x19)
at /home/ec2-user/jenkins/workspace/OCF/pyocf/Execute_tests/tests/functional/pyocf/ocf/src/ocf/env/ocf_env.h:453
453 __sync_add_and_fetch(&a->counter, i);
This problem occurred on OCF v19.3. Not checked on current master.
Now, in case of allocation error during start cache ocf returns OCF_ERR_START_CACHE_FAIL instead of ENOMEM. There is no also message about allocation error so user can be confused.
When power failure occurs during cache stopping, it's possible that there would be invalid metadata checksum on next cache load.
If in cache shutdown sequence power failure occurs after sb_runtime and/or part_runtime segments are flushed, but before final sb_config flush, there might be metadata checksum error for those segments on next cache load.
Checksums for those segments are verified in load sequence regardless of shutdown status - dirty or clean. They should be checked only after clean shutdown. For dirty shutdown there is no guarantee that checksums for those segments are up to date.
In current implementation of metadata_io_i_asynch function there is no mechanism to throttle queue depth. This mechanism is needed in environments based on model without interruptions - like SPDK, now in the worth case scenario ocf can double memory consumption (can consume double size of metadata stored on cache device) during flushing metadata to cache device.
In case when we have enough memory to handle situation described above, starting cache still can fail due to io submit fail caused by running out of space in submission queue.
Starting cache with configure option metadata_volatile = true fails with:
MalCache: Metadata Error
MalCache: Metadata Flush ERROR
MalCache: ERROR: Cannot save cache state
MalCache: Attaching cache device failed
Full log
Inserting cache MalCache
MalCache: Successfully added
MalCache: Cache mode : wb
MalCache: Super block config offset : 0 kiB
MalCache: Super block config size : 3544 B
MalCache: Super block runtime offset : 0 kiB
MalCache: Super block runtime size : 1512 B
MalCache: Reserved offset : 0 kiB
MalCache: Reserved size : 128 kiB
MalCache: Core config offset : 0 kiB
MalCache: Core config size : 360 kiB
MalCache: Core runtime offset : 0 kiB
MalCache: Core runtime size : 1172 kiB
MalCache: Core UUID offset : 0 kiB
MalCache: Core UUID size : 16384 kiB
MalCache: Cleaning offset : 0 kiB
MalCache: Cleaning size : 604 kiB
MalCache: Eviction offset : 0 kiB
MalCache: Eviction size : 400 kiB
MalCache: Collision offset : 0 kiB
MalCache: Collision size : 604 kiB
MalCache: List info offset : 0 kiB
MalCache: List info size : 856 kiB
MalCache: Hash offset : 0 kiB
MalCache: Hash size : 68 kiB
MalCache: Cache line size: 4 kiB
MalCache: Metadata capacity: 36 MiB
MalCache: OCF metadata self-test PASSED
MalCache: Metadata Error
MalCache: Metadata Flush ERROR
MalCache: ERROR: Cannot save cache state
MalCache: Attaching cache device failed
vbdev_ocf.c: 767:start_cache_cmpl: *ERROR*: Failed to attach cache device: -1000017
MalCache: Metadata Error
MalCache: Adding core core1 failed
vbdev_ocf.c: 906:add_core_cmpl: *ERROR*: Failed to add core device to cache instance
Messages printed during ocf_metadata_hash_load_superblock
operation
pipeline->error gets set in ocf_metadata_hash_generic_complete
to -95
Backtrace:
(gdb) bt
#0 ocf_pipeline_finish (pipeline=0x200027c0b780, error=-95) at src/ocf/utils/utils_pipeline.c:123
#1 0x00005555555e9c37 in ocf_metadata_hash_generic_complete (priv=0x200027c0b7a8, error=-95) at src/ocf/metadata/metadata_hash.c:892
#2 0x00005555555f25d0 in raw_volatile_flush_all (cache=0x2000142273c0, raw=0x20001c2002d8, cmpl=0x5555555e9c05 <ocf_metadata_hash_generic_complete>, priv=0x200027c0b7a8) at src/ocf/metadata/metadata_raw_volatile.c:45
#3 0x00005555555e81e7 in ocf_metadata_raw_flush_all (cache=0x2000142273c0, raw=0x20001c2002d8, cmpl=0x5555555e9c05 <ocf_metadata_hash_generic_complete>, priv=0x200027c0b7a8) at src/ocf/metadata/metadata_raw.h:293
#4 0x00005555555ea5b2 in ocf_medatata_hash_flush_segment (pipeline=0x200027c0b780, priv=0x200027c0b7a8, arg=0x55555594b620 <ocf_metadata_hash_flush_sb_flush_segment_args>) at src/ocf/metadata/metadata_hash.c:1161
#5 0x00005555555f8227 in _ocf_pipeline_run_step (req=0x200014627f00) at src/ocf/utils/utils_pipeline.c:44
#6 0x00005555555e2c7f in ocf_io_handle (io=0x0, opaque=0x200014627f00) at src/ocf/ocf_queue.c:74
#7 0x00005555555e2d7d in ocf_queue_run_single (q=0x20001c2001c0) at src/ocf/ocf_queue.c:94
#8 0x00005555555b84a8 in mngt_queue_poll (opaque=0x20001c2001c0) at vbdev_ocf.c:796
#9 0x0000555555657520 in spdk_thread_poll (thread=0x555555dc2800, max_msgs=0, now=6263853433141732) at thread.c:456
#10 0x0000555555614b56 in _spdk_reactor_run (arg=0x555555dc25c0) at reactor.c:244
#11 0x0000555555614e78 in spdk_reactors_start () at reactor.c:333
#12 0x00005555556136fb in spdk_app_start (opts=0x7fffffffe270, start_fn=0x555555564450 <spdk_startup>, arg1=0x0) at app.c:677
#13 0x00005555555645e5 in main (argc=3, argv=0x7fffffffe408) at iscsi_tgt.c:112
Tested on d88a4ac and previus,
the issue reproduces precisely after 6cd8447
SPDK version (devel): https://review.gerrithub.io/c/spdk/spdk/+/448537/9
Bottom devices are RAM regions (malloc bdevs)
Kernel bug occured during cache stop:
[ 4799.934072] Cache cache1 successfully stopped
[ 4799.939197] [Open-CAS] [Classifier] Deinitialized IO classifier
[ 4799.946112] Thread cas_io_cache1_-1 stopped
[ 4799.951269] Thread ocf_metadata_updater_cache1 stopped
[ 4799.957307] ------------[ cut here ]------------
[ 4799.962688] kernel BUG at /root/open-cas-linux/modules/cas_cache/src/ocf/utils/utils_refcnt.c:19!
[ 4799.973045] invalid opcode: 0000 [#1] SMP
[ 4799.977873] Modules linked in: cas_cache(OE) cas_disk(OE) brd nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache loop sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper iTCO_wdt cryptd iTCO_vendor_support mei_me joydev sg pcspkr mei lpc_ich wmi i2c_i801 ioatdma ipmi_si ipmi_devintf ipmi_msghandler ip_tables ext4 mbcache jbd2 sd_mod
[ 4800.060336] crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm isci crc32c_intel igb ahci libsas nvme drm scsi_transport_sas libahci ptp nvme_core libata pps_core drm_panel_orientation_quirks dca i2c_algo_bit [last unloaded: cas_disk]
[ 4800.095851] CPU: 0 PID: 103074 Comm: casadm Kdump: loaded Tainted: G W OE ------------ 3.10.0-957.el7.x86_64 #1
[ 4800.108634] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013
[ 4800.121629] task: ffff9608cffa1040 ti: ffff960760804000 task.ti: ffff960760804000
[ 4800.130406] RIP: 0010:[] [] ocf_refcnt_dec+0x45/0x50 [cas_cache]
[ 4800.141184] RSP: 0018:ffff960760807c18 EFLAGS: 00010297
[ 4800.147350] RAX: 00000000ffffffff RBX: ffffa4acc49dc000 RCX: dead000000000200
[ 4800.155552] RDX: ffff960760807ca0 RSI: 00000000fff0bdbb RDI: ffffa4acc49dc5d0
[ 4800.163748] RBP: ffff960760807c30 R08: ffffa4accc420000 R09: 00000001802a0016
[ 4800.171958] R10: 00000000f6158001 R11: ffffe732c0d85600 R12: 00000000fff0bdbb
[ 4800.180156] R13: ffff960760807ca0 R14: ffffa4accc420000 R15: ffffa4acc4ae4b18
[ 4800.188374] FS: 00007f3a2e465740(0000) GS:ffff960aee000000(0000) knlGS:0000000000000000
[ 4800.197843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4800.204475] CR2: 00007f37133dc0a0 CR3: 00000000ae0c6000 CR4: 00000000000607f0
[ 4800.212663] Call Trace:
[ 4800.215618] [] ? ocf_mngt_cache_put+0x20/0x60 [cas_cache]
[ 4800.223716] [] _ocf_mngt_cache_lock_complete+0x85/0x90 [cas_cache]
[ 4800.232887] [] _ocf_async_lock_run_waiters+0x4b/0x70 [cas_cache]
[ 4800.241865] [] ocf_async_lock_deinit+0x8d/0xb0 [cas_cache]
[ 4800.250060] [] ocf_mngt_cache_put+0x3b/0x60 [cas_cache]
[ 4800.257974] [] ocf_queue_put+0x49/0x60 [cas_cache]
[ 4800.265398] [] cache_mngt_exit_instance+0x177/0x260 [cas_cache]
[ 4800.274274] [] cas_service_ioctl_ctrl+0x1039/0x1e50 [cas_cache]
[ 4800.283166] [] ? sched_slice.isra.58+0x66/0xc0
[ 4800.290205] [] ? __enqueue_entity+0x78/0x80
[ 4800.296945] [] ? enqueue_entity+0x2ef/0xbe0
[ 4800.303690] [] ? cpumask_next_and+0x35/0x50
[ 4800.310423] [] ? enqueue_task_fair+0x208/0x6c0
[ 4800.317455] [] ? sched_clock_cpu+0x85/0xc0
[ 4800.324106] [] do_vfs_ioctl+0x3a0/0x5a0
[ 4800.330455] [] ? wake_up_new_task+0x131/0x1a0
[ 4800.337389] [] ? do_fork+0x107/0x320
[ 4800.343463] [] SyS_ioctl+0xa1/0xc0
[ 4800.349348] [] ? system_call_after_swapgs+0xa2/0x146
[ 4800.356957] [] system_call_fastpath+0x22/0x27
[ 4800.363882] [] ? system_call_after_swapgs+0xae/0x146
[ 4800.371487] Code: 75 0e b8 01 00 00 00 f0 0f b1 5f 08 85 c0 75 06 89 d8 5b 5d c3 90 48 8b 57 18 48 8b 47 10 48 89 d7 e8 d0 50 6e ca 89 d8 5b 5d c3 <0f> 0b 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 53
[ 4800.394090] RIP [] ocf_refcnt_dec+0x45/0x50 [cas_cache]
[ 4800.402005] RSP
I specified a io class file for a cache but when I decided to disable it there was not option that would allow me to do so, and I had to stop/start the cache to achieve it, so how can I disable currently used io class file, without having to restart the cache?
root@node195:/etc/opencas# casadm -C -H
Usage: casadm --io-class {--load-config|--list}Manage IO classes
Loads configuration for IO classes:
Usage: casadm --io-class --load-config --cache-id --file
Options that are valid with --load-config (-C) are:
-i --cache-id Identifier of cache instance <1-16384>
-f --file Configuration file containing IO class definitionLists currently configured IO classes:
Usage: casadm --io-class --list --cache-id [option...]
Options that are valid with --list (-L) are:
-i --cache-id Identifier of cache instance <1-16384>
-o --output-format Output format: {table|csv}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.