Write-error on swap-device about zram-config HOT 46 CLOSED

MEM_FACTOR=50
seems good.

from zram-config.

No.Several error with zram is reported.

from zram-config.

@StuartIanNaylor
I agree with you.
and I leave it a while.

from zram-config.

@StuartIanNaylor
Okay.
I think that you did great job some japanese rely on your script.
:)

from zram-config.

and I should make post and no spam anymore :P

from zram-config.

Same error 3B on raspbian without adding diskswap.
I add diskswap , and no error happens.

pi@raspberrypi:~ $ swapon
NAME TYPE SIZE USED PRIO
/var/swap file 1.8G 0B -2
/dev/zram0 partition 1.3G 519.8M 75

from zram-config.

Wasn't that obvious at first, this issue happens when mem_limit is full.

How to avoid this? What's the point of disk_size?

from zram-config.

from zram-config.

Is this still happening?

from zram-config.

Aug 12 05:20:19 raspberrypi kernel: Read-error on swap-device (254:0:573072)

[17526.179037] zram: Decompression failed! err=-22, page=71634
[17526.179062] zram: Decompression failed! err=-22, page=71634
[17526.179071] Read-error on swap-device (254:0:573072)

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.4.51-v7+ #1327 SMP Thu Jul 23 10:58:46 BST 2020 armv7l GNU/Linux

from zram-config.

Post you ztab as likely your disk_size is to high a ratio to mem_limit or you have idle pages that are already compressed maybe or its a bug with lz4/zram.

To be honest the Ztab defaults are extremely optimistic as the original focus was logs and text logs are highly compressible.
Try 2x disk_size:mem_limit and if works maybe up to 2.5x disk_size:mem_limit or you have found a bug and should via Raspberry.
The actual resultant size of a compression ratio is just guess work and varies drastically on input and sometimes you can just get large compressed idle chunks that just eat up all available space.

from zram-config.

MEM_FACTOR=75
DRIVE_FACTOR=210
COMP_ALG=lzo-rle
SWAP_DEVICES=1
PAGE_CLUSTER=0
SWAPPINESS=90

sorry, mine is zram-swap-config.

from zram-config.

I changed DRIVE_FACTOR=210 DRIVE_FACTOR=250.

from zram-config.

NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lzo-rle 1.6G 828K 74B 4K 4 [SWAP]

This works well,sometimes error occurs like above.
I want to use 75% of memory as zram device.

from zram-config.

I was going to say your swapping repos's.

Yeah its your drive factor probably as even if 210% seems like its struggling to compress.
Unless its lzo-rle as was going to say now we are on 5.4 its in LZO-RLE is in the kernel.

There is always deflate but boy that is slow (zlib) zstd is a trade off that does give higher compression with not so bad compress and decompress speed.

I have been trying for a while to get Raspberry to enable the write back cache which is a safe guard against this where very infrequently we will dump to SD if idle in zram for a long time.
Yeah I know it is writes but very low infrequent writes that you have control over, but for now that doesn't solve things.

Because resultant compression is just pure guesswork and that the compression level of input is totally dependent on what your software is doing, you are going to have to drop the drive_factor to as low as 125% as lower it becomes a bit pointless.
Do it in 20% steps and see how you go.

It is interesting you are seeing decompress errors though and maybe this is something introduced in 5.4 and lzo-rle as I have only had problems compressing the uncompressible, aka already compressed.

Post your error messages on the next one and also if I can ask what else are you running?

from zram-config.

MEM_FACTOR=75
DRIVE_FACTOR=125

root@raspberrypi:/home/pi# zramctl
NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd 820.5M 4K 58B 4K 4 [SWAP]

Maybe I was too greedy.

So far there are no errors with this setup.

from zram-config.

MEM_FACTOR=50
DRIVE_FACTOR=125

root@raspberrypi:/home/pi# zramctl
NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd 547M 4K 58B 4K 4 [SWAP]

from zram-config.

OK but that is only a 125% compression rate so build up.
The amount of ram you allocate prob should run in 75%
free -h should give you a good idea.

Its not greedy its just the unknown that a random app might hold a large chunk of compressed data in memory and be idle.
Its not that usual but it can happen and you just have to tune a bit by getting to a working the work up.

Is zstd working ok as never did try it as its not blazing fast like lzo-rle but guess maybe a good compromise.
If it was just a huge compressed idle lump of memory then doesn't really matter as no alg will compress it much further.
Actually maybe it was just a chunk of mem that lzo-rle struggled to compress and zstd does actually manage.

Post your results long term as others will prob find it useful.

I have only ever really found this problematic when installing or compiling

from zram-config.

root@raspberrypi:/home/pi# free -h
total used free shared buff/cache available
Mem: 875Mi 324Mi 77Mi 16Mi 473Mi 475Mi
Swap: 546Mi 18Mi 528Mi

As router, It seems fine.
I will try it and to increase mem_factor and ratio as desktop.

root@raspberrypi:/home/pi# zramctl
NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd 547M 17.9M 3.9M 4.4M 4 [SWAP]

from zram-config.

root@raspberrypi:/home/pi# zramctl
NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd 1.5G 2.1M 622.9K 1.1M 4 [SWAP]

root@raspberrypi:/home/pi# free -h
total used free shared buff/cache available
Mem: 875Mi 345Mi 84Mi 22Mi 445Mi 449Mi
Swap: 1.4Gi 2.0Mi 1.4Gi

MEM_FACTOR=75
DRIVE_FACTOR=225

zstd compression ratio is 2.8 , so I set DRIVE_FACTOR around 2.8.

from zram-config.

I misunderstood.

compression ratio is <3. so DRIVE_FACTOR should be <200% .

from zram-config.

Google on there Chromebooks ran 50% memory 200% compression (drive factor) which should be achievable.
Only thing now is load as presume on boot things rattle a bit but should calm down.
Slower compression is usually because its more intensive (load) but guess you can lower the SWAPPINESS but the we we are employing zram is because we wish to use it :)

See how you go but yeah drive_factor is likely the important one but also check mem usage with free -h.
Usually no tuning is needed but seems you might be running something quite load and memory hungry.

from zram-config.

Raspbian Desktop Use.
It seems no problem.
But dmesg left some lines.
I have never seen those with lh4/lzo.

root@raspberrypi:/home/pi# free -h
total used free shared buff/cache available
Mem: 874Mi 542Mi 75Mi 86Mi 255Mi 189Mi
Swap: 1.3Gi 463Mi 847Mi

[58257.470689] kswapd0: page allocation failure: order:0, mode:0xa20(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0
[58257.470727] CPU: 0 PID: 60 Comm: kswapd0 Tainted: G C 5.4.51-v7+ #1333
[58257.470730] Hardware name: BCM2835
[58257.470735] Backtrace:
[58257.470756] [<8010d480>] (dump_backtrace) from [<8010d778>] (show_stack+0x20/0x24)
[58257.470763] r6:b33d6000 r5:00000000 r4:80d95cb8 r3:baec4fd4
[58257.470774] [<8010d758>] (show_stack) from [<808cdbe4>] (dump_stack+0xe0/0x124)
[58257.470788] [<808cdb04>] (dump_stack) from [<802c7324>] (warn_alloc+0xd4/0x15c)
[58257.470795] r8:80d05684 r7:00000a00 r6:80ab60e4 r5:00000000 r4:80d04f48 r3:baec4fd4
[58257.470803] [<802c7250>] (warn_alloc) from [<802c84f4>] (__alloc_pages_nodemask+0x1090/0x123c)
[58257.470807] r3:00000000 r2:80ab60e4
[58257.470813] r6:00000000 r5:00000000 r4:00000200
[58257.470821] [<802c7464>] (__alloc_pages_nodemask) from [<802c8940>] (page_frag_alloc+0x180/0x1a0)
[58257.470828] r10:00000000 r9:00000000 r8:00000000 r7:00000a00 r6:00000a20 r5:00000b40
[58257.470831] r4:b37a0e48
[58257.470844] [<802c87c0>] (page_frag_alloc) from [<8077b35c>] (__netdev_alloc_skb+0x150/0x184)
[58257.470849] r7:00000a00 r6:b37a0e48 r5:b2b6e000 r4:00000b40
[58257.470857] [<8077b20c>] (__netdev_alloc_skb) from [<80694084>] (rx_submit+0x10c/0x23c)
[58257.470863] r7:00000a00 r6:a916fcc0 r5:ad83cb80 r4:b2b6e540
[58257.470868] [<80693f78>] (rx_submit) from [<80694ba4>] (rx_complete+0x200/0x220)
[58257.470875] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:a916fcc0 r5:ad83cb80
[58257.470879] r4:b2b6e540 r3:00000001
[58257.470887] [<806949a4>] (rx_complete) from [<8069ecec>] (__usb_hcd_giveback_urb+0x74/0x118)
[58257.470892] r7:00000000 r6:00000000 r5:b335ee00 r4:ad83cb80
[58257.470900] [<8069ec78>] (__usb_hcd_giveback_urb) from [<8069eddc>] (usb_hcd_giveback_urb+0x4c/0xfc)
[58257.470906] r7:80d04f48 r6:ad83cb80 r5:ad83cb80 r4:b0e26ec0
[58257.470916] [<8069ed90>] (usb_hcd_giveback_urb) from [<806ca854>] (completion_tasklet_func+0x90/0xd0)
[58257.470922] r6:ad83cb80 r5:b324b800 r4:b0e26ec0 r3:60000113
[58257.470930] [<806ca7c4>] (completion_tasklet_func) from [<806d9fec>] (tasklet_callback+0x20/0x24)
[58257.470935] r7:b37962c4 r6:00000000 r5:b29fd7c4 r4:b29fd7c0
[58257.470946] [<806d9fcc>] (tasklet_callback) from [<8012628c>] (tasklet_action_common+0x64/0xec)
[58257.470955] [<80126228>] (tasklet_action_common) from [<8012633c>] (tasklet_hi_action+0x28/0x30)
[58257.470961] r9:00000001 r8:b305e000 r7:00000100 r6:b33d6000 r5:00000000 r4:80d03080
[58257.470972] [<80126314>] (tasklet_hi_action) from [<8010249c>] (__do_softirq+0x1a4/0x418)
[58257.470979] [<801022f8>] (__do_softirq) from [<80126130>] (irq_exit+0x100/0x110)
[58257.470986] r10:b925173c r9:b33d6000 r8:b305e000 r7:00000001 r6:00000000 r5:00000000
[58257.470989] r4:80ca42e4
[58257.471000] [<80126030>] (irq_exit) from [<80181200>] (__handle_domain_irq+0x70/0xc4)
[58257.471007] [<80181190>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
[58257.471016] r8:ac4d800d r7:b33d76f4 r6:ffffffff r5:20000013 r4:00000000 r3:b33d76c0
[58257.471022] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[58257.471026] Exception stack(0xb33d76c0 to 0xb33d7708)
[58257.471033] 76c0: ac4d800d 00001ff3 00000000 0000000b 80d04f48 00000553 b918ff38 b33d7840
[58257.471040] 76e0: ac4d800d b92a9b3c b925173c b33d7774 00001ff3 b33d7710 000000ff 7f20d298
[58257.471043] 7700: 20000013 ffffffff
[58257.471047] r4:7f20d298 r3:baec4fd4
[58257.471084] [<7f20d1ec>] (HUF_compress4X_repeat [zstd_compress]) from [<7f229a58>] (ZSTD_compressBlock_internal+0x260/0x12d0 [zstd_compress])
[58257.471091] r10:7f22f2a0 r9:b92a9b3c r8:00000000 r7:00000004 r6:00000553 r5:80d04f48
[58257.471093] r4:b918f008
[58257.471119] [<7f2297f8>] (ZSTD_compressBlock_internal [zstd_compress]) from [<7f22acb0>] (ZSTD_compressContinue_internal+0x1e8/0x43c [zstd_compress])
[58257.471126] r10:ac4d8006 r9:00001ffa r8:953e1000 r7:00001000 r6:00001000 r5:b918f008
[58257.471129] r4:953e2000
[58257.471154] [<7f22aac8>] (ZSTD_compressContinue_internal [zstd_compress]) from [<7f22b938>] (ZSTD_compressCCtx+0x2f8/0x628 [zstd_compress])
[58257.471161] r10:32c30001 r9:00000004 r8:00000001 r7:00000008 r6:b33d7a00 r5:b918f074
[58257.471164] r4:b918f008
[58257.471184] [<7f22b640>] (ZSTD_compressCCtx [zstd_compress]) from [<7f1b312c>] (__zstd_compress+0x88/0xa0 [zstd])
[58257.471190] r10:00002000 r9:b918f008 r8:00001000 r7:953e1000 r6:ac4d8000 r5:b33d7b28
[58257.471193] r4:00000000
[58257.471203] [<7f1b30a4>] (__zstd_compress [zstd]) from [<7f1b31b4>] (zstd_compress+0x34/0x3c [zstd])
[58257.471210] r10:b33d6000 r9:00000000 r8:80e24fb8 r7:80d07d44 r6:b33d6000 r5:7f1d5400
[58257.471213] r4:b0e3aa00
[58257.471226] [<7f1b3180>] (zstd_compress [zstd]) from [<8051f970>] (crypto_compress+0x2c/0x34)
[58257.471244] [<8051f944>] (crypto_compress) from [<7f1d1218>] (zcomp_compress+0x40/0x48 [zram])
[58257.471264] [<7f1d11d8>] (zcomp_compress [zram]) from [<7f1d2a3c>] (zram_bvec_rw+0x3e8/0xa50 [zram])
[58257.471283] [<7f1d2654>] (zram_bvec_rw [zram]) from [<7f1d3168>] (zram_rw_page+0xc4/0x15c [zram])
[58257.471290] r10:00000001 r9:00000000 r8:0002aff8 r7:b3af5ba4 r6:80d04f48 r5:00000000
[58257.471293] r4:00000000
[58257.471310] [<7f1d30a4>] (zram_rw_page [zram]) from [<8033caf0>] (bdev_write_page+0x8c/0xc8)
[58257.471317] r10:b33d7cb0 r9:b33d7d68 r8:b2c446c0 r7:00000000 r6:0002aff0 r5:b2c446c0
[58257.471319] r4:7f1d4038
[58257.471329] [<8033ca64>] (bdev_write_page) from [<802cd9ac>] (__swap_writepage+0x1d8/0x3d4)
[58257.471335] r7:802cd3cc r6:b33d7cc0 r5:80d04f48 r4:b3af5ba4
[58257.471341] [<802cd7d4>] (__swap_writepage) from [<802cdbe8>] (swap_writepage+0x40/0x88)
[58257.471347] r8:00000000 r7:b33d7f10 r6:b33d7cc0 r5:b3af5ba4 r4:ffffffed
[58257.471356] [<802cdba8>] (swap_writepage) from [<8028b888>] (shrink_page_list+0x9a4/0xf34)
[58257.471361] r6:b33d7d60 r5:b3af5ba4 r4:b3af5ba8 r3:802cdba8
[58257.471368] [<8028aee4>] (shrink_page_list) from [<8028c6f8>] (shrink_inactive_list+0x1c8/0x454)
[58257.471375] r10:00000019 r9:80d97a00 r8:80d04f48 r7:b33d7d60 r6:80d97a04 r5:00000000
[58257.471378] r4:b33d7f10
[58257.471385] [<8028c530>] (shrink_inactive_list) from [<8028d1f0>] (shrink_node_memcg+0x3ac/0x7cc)
[58257.471392] r10:00000019 r9:00000000 r8:80d97a04 r7:b33d7f10 r6:00000000 r5:b33d7dec
[58257.471395] r4:00000000
[58257.471402] [<8028ce44>] (shrink_node_memcg) from [<8028d6ec>] (shrink_node+0xdc/0x4f8)
[58257.471409] r10:80d97280 r9:000000b6 r8:80d057dc r7:00000000 r6:0000025f r5:00000000
[58257.471412] r4:b33d7f10
[58257.471419] [<8028d610>] (shrink_node) from [<8028ec88>] (kswapd+0x3dc/0x954)
[58257.471425] r10:80d97280 r9:00000000 r8:00000003 r7:00000000 r6:00000001 r5:00000001
[58257.471428] r4:00003c00
[58257.471438] [<8028e8ac>] (kswapd) from [<80142f30>] (kthread+0x13c/0x168)
[58257.471445] r10:b3309ddc r9:b30fbe0c r8:8028e8ac r7:80d97280 r6:00000000 r5:b3309d80
[58257.471447] r4:b3309dc0
[58257.471456] [<80142df4>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[58257.471459] Exception stack(0xb33d7fb0 to 0xb33d7ff8)
[58257.471464] 7fa0: 00000000 00000000 00000000 00000000
[58257.471470] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[58257.471476] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[58257.471482] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80142df4
[58257.471486] r4:b3309d80 r3:80104648
[58257.471512] Mem-Info:
[58257.471531] active_anon:62598 inactive_anon:48378 isolated_anon:25
active_file:4078 inactive_file:2735 isolated_file:0
unevictable:4 dirty:0 writeback:0 unstable:0
slab_reclaimable:4710 slab_unreclaimable:7520
mapped:11266 shmem:5488 pagetables:2594 bounce:0
free:4632 free_pcp:651 free_cma:0
[58257.471542] Node 0 active_anon:250392kB inactive_anon:193512kB active_file:16312kB inactive_file:10940kB unevictable:16kB isolated(anon):100kB isolated(file):0kB mapped:45064kB dirty:0kB writeback:0kB shmem:21952kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[58257.471563] Normal free:18528kB min:53248kB low:57344kB high:61440kB active_anon:250624kB inactive_anon:193448kB active_file:16312kB inactive_file:11148kB unevictable:16kB writepending:0kB present:917504kB managed:895296kB mlocked:16kB kernel_stack:3568kB pagetables:10376kB bounce:0kB free_pcp:2604kB local_pcp:1440kB free_cma:0kB
[58257.471570] lowmem_reserve[]: 0 0
[58257.471585] Normal: 10504kB (UMEHC) 6918kB (UEH) 32016kB (UEH) 8532kB (UE) 1564kB (UE) 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 18528kB
[58257.471667] 12324 total pagecache pages
[58257.471678] 22 pages in swap cache
[58257.471688] Swap cache stats: add 2050607, delete 2050690, find 8606/1407398
[58257.471694] Free swap = 447308kB
[58257.471701] Total swap = 1342940kB
[58257.471708] 229376 pages RAM
[58257.471715] 0 pages HighMem/MovableOnly
[58257.471722] 5552 pages reserved
[58257.471727] 16384 pages cma reserved

from zram-config.

journalctl -xe

Aug 17 23:09:01 raspberrypi kernel: [<8051f944>] (crypto_compress) from [<7f1d1218>] (zcomp_compress+0x40/0x48 [zram])
Aug 17 23:09:01 raspberrypi kernel: [<7f1d11d8>] (zcomp_compress [zram]) from [<7f1d2a3c>] (zram_bvec_rw+0x3e8/0xa50 [zram])
Aug 17 23:09:01 raspberrypi kernel: [<7f1d2654>] (zram_bvec_rw [zram]) from [<7f1d3168>] (zram_rw_page+0xc4/0x15c [zram])
Aug 17 23:09:01 raspberrypi kernel: [<7f1d30a4>] (zram_rw_page [zram]) from [<8033caf0>] (bdev_write_page+0x8c/0xc8)

from zram-config.

I chenged swappiness=90 to 100.
and trying to use.

from zram-config.

swapiness is just pref of use, so higher means swap will be used even if more mem is avail.

As to above dunno? But looking like zstd is favoured either.

from zram-config.

I didn't know.
I got it.
I got swappiness=80.

from zram-config.

I set it back to 100; 100 is less error-prone than 80.
pi@raspberrypi:~ $ free -h
total used free shared buff/cache available
Mem: 874Mi 546Mi 134Mi 46Mi 193Mi 228Mi
Swap: 1.3Gi 553Mi 758Mi

from zram-config.

I found this article.
https://haydenjames.io/linux-performance-almost-always-add-swap-part2-zram/

we shuld set sysctl for zram.

from zram-config.

Its already done in zram-swap-config.conf
Have you looked at the code?

	sysctl vm.page-cluster=${PAGE_CLUSTER}
	sysctl vm.swappiness=${SWAPPINESS}

Also just so others know we are talking about zram-swap-config.

How 100 can be less error prone than 80 is very confusing as your forcing more use the higher the number.
But the problem seems to be zram that you have, all zram-swap-config does is config the zram.
You will have to ask the zram maintainers about zram errors.

from zram-config.

I alerady changed sysctl.conf.
I restored it.
and I trust you :)

from zram-config.

Failure to match algorithm compression and disk size results in an error.
For example, if you select zstd, double the disk size is not enough, so you need to make it nearly triple.

from zram-config.

With raspbian desktop, It seems only lz4 is certain choice.
Because of something error with others.
Maybe it depends on its speed.

from zram-config.

There is no disk and double the disk size is basically guessing low compression, its mem_factor that allocates ram usage and smaller disks are just a lower expectation on overall compression.
There are certain operations that just fail as already compressed huge chunks that can not be compressed fill past the available ram.
Its rare but it does happen seems from experience only when compiling certain apps.
If something is not working with zram and has changed file a bug on the raspbian desktop repo version.

from zram-config.

This is my memo.
System looks at disk_size , not original mem_limit.

from zram-config.

So you have a 8gb Pi 4 ?

from zram-config.

Me ? no .
This is pi kernel problem.

from zram-config.

That problem is only with the 8gb Pi4

from zram-config.

That is armbian and a kernel option any absolutely nothing to do with this project. Choose or compile your own kernel but those choices are not part of this repo.

from zram-config.

@StuartIanNaylor
Excuse me , I thought that I showed you related info.
But here is github.

from zram-config.

Post on the kernels or https://www.kernel.org/ is likely to have better effect. This is just an app that obs need a kernel and uses zram but isn't the source...

from zram-config.

I don't know what is happening but seems its a kernel option that needs to change like you posted. Guess its just time where new kernels will adopt and disable the malloc method that is the source.

from zram-config.

@StuartIanNaylor
I see.
and Have you never seen these decompression error around your pi?

from zram-config.

Yes and the open source code that creates it is not in this repo but others and where you should be posting rather than spamming.

I have handed over the repo, so no longer have to be polite.

from zram-config.

@StuartIanNaylor
Another repo you mean is raspberrypi kernel repo?

from zram-config.

its ecdye/zram-config so you can post as you wish.

from zram-config.