Comments (15)
mjansson
commented on May 18, 2024
Have you tried the latest master? I fixed a few issues regarding order of constructors/destructors during init/fini, and for me it works fine now:
mattias@lindev:~/projects/rpmalloc$ LD_PRELOAD=/home/mattias/projects/rpmalloc/bin/linux/release/x86-64/librpmalloc.so python ps_mem.py -p 4977
Private + Shared = RAM used Program
1.9 MiB + 581.0 KiB = 2.5 MiB bash
---------------------------------
2.5 MiB
=================================
from rpmalloc.
justanerd
commented on May 18, 2024
I'm running the latest master still happening.
But I have more proccesses maybe that is triggering it.
I try to find a another way to trigger it every time.
from rpmalloc.
psrok1
commented on May 18, 2024
Same for this simple benchmark:
https://gist.github.com/psrok1/fa0cf1b5ee589bd316fc96709253ec2b
I've also tried latest master and got segfault for bigger strings.
from rpmalloc.
mjansson
commented on May 18, 2024
Which python version and OS are you running? Tried it on my MacBook with python 2.7.10 and it works as expected
TinyBook:rpmalloc mattias$ DYLD_FORCE_FLAT_NAMESPACE=1 DYLD_INSERT_LIBRARIES=/Users/mattias/projects/rpmalloc/bin/macosx/release/librpmalloc.dylib python rpmalloc-bench.py
avg 564.628500938 ms
and running without injecting rpmalloc
TinyBook:rpmalloc mattias$ python rpmalloc-bench.py
avg 650.517408848 ms
I'll try on a Linux machine as well
from rpmalloc.
justanerd
commented on May 18, 2024
I have python 2.7.13.
The code seems to be using realloc.
I built rpmalloc just with ninja tried clang and gcc no difference.
Also it seems to be alot slower with rpmalloc
glibc:
avg 227.386951447 ms
rpmalloc:
avg 966.183185577 ms
CPU is a Haswell 5960X
from rpmalloc.
mjansson
commented on May 18, 2024
Hopefully fixed now, was a copy and paste mistake causing the array for adaptive cache control of the large blocks to be undersized -> freeing a large block > 256KiB would potentially overwrite the heap link list causing abort on exit cleanup.
Also introduced realloc hysteresis bounds to speed up similar use cases.
from rpmalloc.
justanerd
commented on May 18, 2024
That worked thanks.
But I got a new one this time with git pull
I will write the segfault here but I can create a new issue if needed should be another issue:
Thread 3.2 "git-remote-http" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 150754]
_memory_deallocate (p=0x10002c0320) at rpmalloc/rpmalloc.c:1160
1160 if (heap_id == heap->id) {
(gdb) bt
#0 _memory_deallocate (p=0x10002c0320) at rpmalloc/rpmalloc.c:1160
#1 0x00007ffff794481d in __res_iclose () from /lib64/libc.so.6
#2 0x00007ffff7983c42 in ?? () from /lib64/libc.so.6
#3 0x00007ffff7983bf2 in __libc_thread_freeres () from /lib64/libc.so.6
#4 0x00007ffff718d169 in start_thread () from /lib64/libpthread.so.0
#5 0x00007ffff7926b2f in clone () from /lib64/libc.so.6
git version 2.12.2
from rpmalloc.
mjansson
commented on May 18, 2024
I worked out a couple of corner cases in the malloc wrapper init/fini handling, try it and see if it solves your remaining segfaults
from rpmalloc.
justanerd
commented on May 18, 2024
That issue is also fixed thanks.
Now I'm getting other kinds of bugs:
*** Error in `php': double free or corruption (out): 0x0000001000852ba0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x713ab)[0x7f30e86543ab]
/lib64/libc.so.6(+0x77846)[0x7f30e865a846]
/lib64/libc.so.6(+0x78046)[0x7f30e865b046]
php(zend_hash_destroy+0x45)[0x727025]
php(ts_free_id+0xaf)[0x780cef]
/usr/lib64/php7.1/lib64/opcache.so(accel_shutdown+0xb2)[0x7f30e6c42f52]
/usr/lib64/php7.1/lib64/opcache.so(+0x63e90)[0x7f30e6c42e90]
php(module_destructor+0x47)[0x72f1a7]
php[0x73445c]
php(zend_hash_graceful_reverse_destroy+0x107)[0x726887]
php(zend_shutdown+0x5c)[0x733bac]
php(php_module_shutdown+0x26)[0x77da36]
php(main+0x46a)[0x6ef0ba]
/lib64/libc.so.6(__libc_start_main+0xf1)[0x7f30e86031e1]
php(_start+0x2a)[0x6ee0ca]
Thats php 7.1.3 with enabled opcache modul for cli
mpv has also a segfault but I have no debug symbols on this system:
#0 0x00007ffff15fbc4c in ?? () from /lib64/libc.so.6
#1 0x00007ffff5bea004 in ?? () from /usr/lib64/libavcodec.so.57
#2 0x00007ffff5c35fe2 in ?? () from /usr/lib64/libavcodec.so.57
#3 0x00007ffff5c36ab6 in ?? () from /usr/lib64/libavcodec.so.57
#4 0x00007ffff5ee83b3 in avcodec_decode_video2 () from /usr/lib64/libavcodec.so.57
#5 0x00007ffff5ee8dfc in ?? () from /usr/lib64/libavcodec.so.57
#6 0x00007ffff5eec9e8 in avcodec_send_packet () from /usr/lib64/libavcodec.so.57
#7 0x00007ffff575c0c6 in ?? () from /usr/lib64/libavformat.so.57
#8 0x00007ffff57637d0 in avformat_find_stream_info () from /usr/lib64/libavformat.so.57
#9 0x0000000000438dc5 in ?? ()
#10 0x00000000004335a1 in ?? ()
#11 0x0000000000433a1e in ?? ()
#12 0x0000000000433bac in ?? ()
#13 0x0000000000473797 in ?? ()
#14 0x00007ffff7bd7e9f in thread_starter (argptr=0x1000060030) at rpmalloc/malloc.c:87
#15 0x00007ffff22b4144 in start_thread () from /lib64/libpthread.so.0
#16 0x00007ffff15bfb2f in clone () from /lib64/libc.so.6
from rpmalloc.
justanerd
commented on May 18, 2024
Ok, now I'm at my home machine here is a better mpv backtrace just triggered it by playing a x264 video:
Thread 5 "mpv" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 25254]
0x00007ffff7bd6797 in _memory_deallocate (p=0x100b170020) at rpmalloc/rpmalloc.c:1160
1160 if (heap_id == heap->id) {
(gdb) bt
#0 0x00007ffff7bd6797 in _memory_deallocate (p=0x100b170020) at rpmalloc/rpmalloc.c:1160
#1 0x00007ffff7bd7790 in rpfree (ptr=0x100b170020) at rpmalloc/rpmalloc.c:1614
#2 0x00007ffff7bd7e85 in free (ptr=0x100b170020) at rpmalloc/malloc.c:161
#3 0x00007ffff43d21c1 in ?? () from /lib64/libdl.so.2
#4 0x00007ffff79be251 in __nptl_deallocate_tsd.part.4 () from /lib64/libpthread.so.0
#5 0x00007ffff79bf2f8 in start_thread () from /lib64/libpthread.so.0
#6 0x00007ffff770ab2f in clone () from /lib64/libc.so.6
from rpmalloc.
mjansson
commented on May 18, 2024
php seems to have it's own malloc wrapping, I wouldn't count on it being compatible with a LD_PRELOAD scenario. For example, it aborts with tcmalloc as well:
mattias@lindev:~/projects/rpmalloc$ LD_PRELOAD=/home/mattias/external/gperftools/.libs/libtcmalloc_minimal.so php
*** Error in `php': free(): invalid pointer: 0x000055db01d9e230 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x790cb)[0x7f78604010cb]
/lib/x86_64-linux-gnu/libc.so.6(+0x8275a)[0x7f786040a75a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f786040e18c]
php(zend_hash_destroy+0x146)[0x55daffc2d896]
php(module_destructor+0x5d)[0x55daffc22e7d]
php(+0x26101c)[0x55daffc1b01c]
php(zend_hash_graceful_reverse_destroy+0x111)[0x55daffc2e391]
php(zend_shutdown+0x111)[0x55daffc1c0f1]
php(php_module_shutdown+0x2b)[0x55daffbbba5b]
php(main+0x3eb)[0x55daffa9e13b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f78603a83f1]
php(_start+0x2a)[0x55daffa9e30a]
mpv on the other hand does seem to run with tcmalloc preloaded, but not rpmalloc. Looking into this.
from rpmalloc.
alex3d
commented on May 18, 2024
Yet one segfault in ffmpeg (ubuntu 16.04):
$ LD_PRELOAD=/path/to/rpmalloc/bin/linux/debug/x86-64/librpmalloc.so ffmpeg -hide_banner -i /tmp/1.ts
Program received signal SIGSEGV, Segmentation fault.
__memset_sse2 () at ../sysdeps/x86_64/multiarch/../memset.S:78
78 ../sysdeps/x86_64/multiarch/../memset.S: No such file or directory.
(gdb) bt
#0 __memset_sse2 () at ../sysdeps/x86_64/multiarch/../memset.S:78
#1 0x00007ffff529e4a3 in memset (__len=80, __ch=0, __dest=0x1000705e60) at /usr/include/x86_64-linux-gnu/bits/string3.h:90
#2 av_mallocz (size=size@entry=80) at libavutil/mem.c:256
#3 0x00007ffff52947ca in av_expr_parse (expr=expr@entry=0x7fffffffc0c0, s=0x1000070361 "", s@entry=0x1000070360 "1", const_names=const_names@entry=0x7fffffffc390,
func1_names=func1_names@entry=0x0, funcs1=funcs1@entry=0x0, func2_names=func2_names@entry=0x0, funcs2=0x0, log_offset=0, log_ctx=0x10001e2020)
at libavutil/eval.c:695
#4 0x00007ffff52955ff in av_expr_parse_and_eval (d=d@entry=0x7fffffffc188, s=s@entry=0x1000070360 "1", const_names=const_names@entry=0x7fffffffc390,
const_values=const_values@entry=0x7fffffffc190, func1_names=func1_names@entry=0x0, funcs1=funcs1@entry=0x0, func2_names=0x0, funcs2=0x0, opaque=0x0, log_offset=0,
log_ctx=0x10001e2020) at libavutil/eval.c:725
#5 0x00007ffff52a1928 in set_string_number (obj=obj@entry=0x10001e2020, target_obj=0x10001e2020, o=0x7ffff6b1b020 <avcodec_options+15360>,
val=val@entry=0x1000070360 "1", dst=dst@entry=0x10001e23a8) at libavutil/opt.c:240
#6 0x00007ffff52a2449 in av_opt_set (obj=obj@entry=0x10001e2020, name=<optimized out>, val=0x1000070360 "1", search_flags=search_flags@entry=0) at libavutil/opt.c:395
#7 0x00007ffff52a2c26 in av_opt_set_dict2 (obj=obj@entry=0x10001e2020, options=options@entry=0x7fffffffc788, search_flags=search_flags@entry=0) at libavutil/opt.c:1462
#8 0x00007ffff52a2cef in av_opt_set_dict (obj=obj@entry=0x10001e2020, options=options@entry=0x7fffffffc788) at libavutil/opt.c:1479
#9 0x00007ffff63adf3e in avcodec_open2 (avctx=0x10001e2020, codec=codec@entry=0x7ffff6b34720 <ff_dvbsub_decoder>, options=options@entry=0x10001f0058)
at libavcodec/utils.c:1423
#10 0x00007ffff73673d5 in avformat_find_stream_info (ic=0x1000140020, options=0x10001f0020) at libavformat/utils.c:3180
#11 0x00000000004159a5 in open_input_file (o=o@entry=0x7fffffffcef0, filename=<optimized out>) at ffmpeg_opt.c:945
#12 0x0000000000418c7b in open_files (l=0x1000130098, l=0x1000130098, open_file=0x4154b0 <open_input_file>, inout=0x42eaa5 "input") at ffmpeg_opt.c:2910
#13 ffmpeg_parse_options (argc=argc@entry=4, argv=argv@entry=0x7fffffffd9d8) at ffmpeg_opt.c:2947
#14 0x0000000000407a97 in main (argc=4, argv=0x7fffffffd9d8) at ffmpeg.c:4150
$ ltrace -fe '*' env LD_PRELOAD=./bin/linux/debug/x86-64/librpmalloc.so ffmpeg -hide_banner -i /tmp/1.ts
...
[pid 28139] librpmalloc.so->rpposix_memalign(0x7ffd2f028cb0, 32, 88, 0 <unfinished ...>
[pid 28139] librpmalloc.so->rpaligned_alloc(32, 88, 88, 0 <unfinished ...>
[pid 28139] librpmalloc.so->rpmalloc(120, 88, 88, 0) = 0x1000120ba0
[pid 28139] <... rpaligned_alloc resumed> ) = 0x1000120ba0
[pid 28139] <... rpposix_memalign resumed> ) = 0
[pid 28139] librpmalloc.so->rpmalloc_thread_initialize(0, 8, 0xe880, 800) = 0x1000000000
[pid 28139] librpmalloc.so->rprealloc(0, 8, 0xe880, 800) = 0x1000070350
[pid 28139] librpmalloc.so->rpmalloc_thread_initialize(0, 2, 0xe8ee, 816) = 0x1000000000
[pid 28139] librpmalloc.so->rprealloc(0, 2, 0xe8ee, 816) = 0x1000070360
[pid 28139] librpmalloc.so->rpmalloc_thread_initialize(0x7ffd2f028bf0, 32, 16, 0x1000070360) = 0x1000000000
[pid 28139] librpmalloc.so->rpposix_memalign(0x7ffd2f028bf0, 32, 16, 0x1000070360 <unfinished ...>
[pid 28139] librpmalloc.so->rpaligned_alloc(32, 16, 16, 0x1000070360 <unfinished ...>
[pid 28139] librpmalloc.so->rpmalloc(48, 16, 16, 0x1000070360) = 0x10001a2640
[pid 28139] <... rpaligned_alloc resumed> ) = 0x10001a2640
[pid 28139] <... rpposix_memalign resumed> ) = 0
[pid 28139] librpmalloc.so->rpmalloc_thread_initialize(0, 16, 16, 0) = 0x1000000000
[pid 28139] librpmalloc.so->rprealloc(0, 16, 16, 0) = 0x1000070370
[pid 28139] librpmalloc.so->rpmalloc_is_thread_initialized(0x1000070320, 0x7f6cfe46f0cb, 0, 2) = 1
[pid 28139] librpmalloc.so->rpfree(0x1000070320, 0x7f6cfe46f0cb, 0x1000000000, 1) = 48
[pid 28139] librpmalloc.so->rpmalloc_is_thread_initialized(0x1000070330, 0x1000000010, 0, 48) = 1
[pid 28139] librpmalloc.so->rpfree(0x1000070330, 0x1000000010, 0x1000000000, 1) = 49
[pid 28139] librpmalloc.so->rpmalloc_is_thread_initialized(0x1000070340, 0x1000000010, 0, 49) = 1
[pid 28139] librpmalloc.so->rpfree(0x1000070340, 0x1000000010, 0x1000000000, 1) = 50
[pid 28139] librpmalloc.so->rpmalloc_is_thread_initialized(0x10001a2620, 0x1000000010, 0, 50) = 1
[pid 28139] librpmalloc.so->rpfree(0x10001a2620, 0x1000000010, 0x1000000000, 1) = 0x10001a0000
[pid 28139] librpmalloc.so->rpmalloc_thread_initialize(0x7ffd2f028510, 32, 2, 864) = 0x1000000000
[pid 28139] librpmalloc.so->rpposix_memalign(0x7ffd2f028510, 32, 2, 864 <unfinished ...>
[pid 28139] librpmalloc.so->rpaligned_alloc(32, 2, 2, 864 <unfinished ...>
[pid 28139] librpmalloc.so->rpmalloc(34, 2, 2, 864) = 0x10001a2670
[pid 28139] <... rpaligned_alloc resumed> ) = 0x10001a2680
[pid 28139] <... rpposix_memalign resumed> ) = 0
[pid 28139] librpmalloc.so->rpmalloc_thread_initialize(0x7ffd2f028310, 32, 56, 0) = 0x1000000000
[pid 28139] librpmalloc.so->rpposix_memalign(0x7ffd2f028310, 32, 56, 0 <unfinished ...>
[pid 28139] librpmalloc.so->rpaligned_alloc(32, 56, 56, 0 <unfinished ...>
[pid 28139] librpmalloc.so->rpmalloc(88, 56, 56, 0) = 0x10001f0140
[pid 28139] <... rpaligned_alloc resumed> ) = 0x10001f0140
[pid 28139] <... rpposix_memalign resumed> ) = 0
[pid 28139] librpmalloc.so->rpmalloc_thread_initialize(0x7ffd2f0284e0, 32, 80, 0) = 0x1000000000
[pid 28139] librpmalloc.so->rpposix_memalign(0x7ffd2f0284e0, 32, 80, 0 <unfinished ...>
[pid 28139] librpmalloc.so->rpaligned_alloc(32, 80, 80, 0 <unfinished ...>
[pid 28139] librpmalloc.so->rpmalloc(112, 80, 80, 0) = 0x1000075e50
[pid 28139] <... rpaligned_alloc resumed> ) = 0x1000075e60
[pid 28139] <... rpposix_memalign resumed> ) = 0
[pid 28139] --- SIGSEGV (Segmentation fault) ---
[pid 28139] +++ killed by SIGSEGV +++
from rpmalloc.
mjansson
commented on May 18, 2024
Thanks for the additional report, I got it reproduced and I'm looking into it.
from rpmalloc.
mjansson
commented on May 18, 2024
Found an issue when freeing memory which had been realigned during allocation if requested alignment was greater than 16. Please try the latest master branch and see if this resolves your issues.
from rpmalloc.
mjansson
commented on May 18, 2024
Closing this for now, let me know if you still have issues.
from rpmalloc.
Related Issues (20)
- Compilation fails on latest Alpine Linux HOT 1
- Potential bug: get_thread_id / rpmalloc_heap_thread_set_current HOT 10
- Warnings: "... specifies less restrictive attributes than its target" HOT 4
- Segmentation fault when building with ENABLE_OVERRIDE=1 and linking it statically HOT 15
- Segmentation fault when revising parameters(span_size and span_map_count) HOT 1
- undefined `false` when compiling HOT 1
- Linking error: unresolved external symbol _rpmalloc_module_init HOT 5
- `SIGABRT` when building w/ `-DENABLE_ASSERTS=1 -DENABLE_OVERRIDE=1 -DENABLE_PRELOAD=1` HOT 1
- Memory usage seems large compared with alternatives and is unaffected by cache settings. HOT 7
- rpmalloc_heap_free - thread safe? HOT 1
- Segmentation fault with the rewrite branch in certain benchmarks HOT 5
- enable_unlimited_thread_cache HOT 1
- Build fails using clang-16 HOT 3
- rpmalloc_heap_free_all has some bug HOT 3
- rpmalloc and msvc asan HOT 4
- Add version information
- _rpmalloc_span_extract_free_list_deferred is not lock-free HOT 1
- get_thread_id() implementation throws SIGILL on my ARM CPU HOT 1
- Android dlopen failed due to TLS_MODEL attribute HOT 1
- Building `develop` as part of LLVM with the built-in option fails on Windows HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rpmalloc.