atilaneves / automem Goto Github PK
View Code? Open in Web Editor NEWC++-style automatic memory management smart pointers for D
License: BSD 3-Clause "New" or "Revised" License
C++-style automatic memory management smart pointers for D
License: BSD 3-Clause "New" or "Revised" License
So we could use ref-counted classes?
RefCounted seems to have issues when used together with zip
: The refcounted payload is not destroyed. I stumpled upon this when using valgrind to check why my program was not freeing its memory. It works fine with lockstep
.
Below is a short reproduction of the error. Runnable version: https://run.dlang.io/is/eVIP3r
/+dub.sdl:
dependency "automem" version="~>0.6.4"
+/
import automem.ref_counted;
import std.range;
import std.stdio;
int created, destroyed;
void main() {
{
RefCounted!S s = RefCounted!S(1);
auto r1 = repeat(s, 2);
auto r2 = repeat(s, 2);
foreach(s1, s2; lockstep(r1, r2))
assert(s1.val == s2.val);
}
writeln("after lockstep: created ", created, " vs destroyed ", destroyed);
assert(created == 1);
assert(destroyed == 1);
{
RefCounted!S s = RefCounted!S(1);
auto r1 = repeat(s, 2);
auto r2 = repeat(s, 2);
foreach(s1, s2; zip(r1, r2))
assert(s1.val == s2.val);
}
writeln("after zip: created ", created, " vs destroyed ", destroyed);
assert(created == 2);
assert(destroyed == 2); // fails!
}
struct S {
@disable this();
@disable this(this);
this(int val) {
this.val = val;
created++;
}
~this() {
destroyed++;
}
int val;
}
Thanks for looking into this!
Vector.free is annotated scope
on the this
parameter. It then passes a member of this
to alloc.dispose in @trusted
lambda, but does not verify that alloc.dispose
has a scope
attribute on the argument. This compiles because @trusted
turns off escape analysis.
Test case:
/+ dub.sdl:
dependency "automem" version="~>0.5.1"
dflags "-dip1000" "-dip25"
+/
void main() @safe
{
import std.stdio : writefln;
import std.experimental.allocator.mallocator: Mallocator;
import automem.vector;
auto vec1 = vector(1, 2, 3);
int[] slice1 = vec1[];
vec1.reserve(4096);
int[] slice2 = vec1[];
"slice1.ptr: %x, slice2.ptr: %x \nslice1: %s, slice2: %s"
.writefln(&slice1[0], &slice2[0], slice1, slice2);
}
Sample output:
slice1.ptr: 7fc6d4d21000, slice2.ptr: 7fc6d3800000
slice1: [-724430832, 32710, -820732144], slice2: [1, 2, 3]
As far as I understand, languages with support for substructural type systems usually solve this by allowing either a single mutable reference or zero or more const references.
Compiling
/+dub.sdl:
dependency "automem" version="~>0.6.1"
+/
import automem;
void main() @nogc
{
StringM str;
}
by dub test.d --single
gives
onlineapp.d:8: error: undefined reference to '_D7automem6vector__T6VectorTyaTS3std12experimental9allocator10mallocator10MallocatorZQCn6__dtorMFNaNbNiNfZv'
collect2: error: ld returned 1 exit status
Error: linker exited with status 1
dmd failed with exit code 1.
Currently, this code snippet below does not compile:
import automem;
class Foo
{
this() @nogc
{
}
~this() @nogc
{
}
}
void main() @nogc
{
auto x = Unique!Foo();
}
Error: @nogc function 'D main' cannot call non-@nogc destructor 'automem.unique.Unique!(Foo, IAllocator).Unique.~this'
Would it be possible to have the code above compile? It would be great!
Ali has a nice trick here that shows how to make the compiler assume something is @nogc
:
https://forum.dlang.org/post/[email protected]
We are trying to remove the long-deprecated std.c
namespace from phobos, but there are a few packages still depending on it indirectly due to https://github.com/dlang-community/stdx-allocator. stdx-allocator was recently updated to remove the dependency, but DMD needs projects to update to the latest release (2.77.2) to keep its CI functioning.
See https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fphobos/detail/PR-6515/5/pipeline/206
This errors:
import std.experimental.allocator.mallocator;
UniqueArray!(int, Mallocator) a;
a ~= [0,1];
I'm interested in trying this library but unfortunately when I add it to my project I can't compile it:
[1/8] Compiling D object 'subprojects/automem/83545ae@@automem@sha/automem_source_automem_array.d.o'.
FAILED: subprojects/automem/83545ae@@automem@sha/automem_source_automem_array.d.o
gdc -Isubprojects/automem/83545ae@@automem@sha -Isubprojects/automem -I../../../code/d/snake/subprojects/automem -I../../../code/d/snake/subprojects/automem/automem/source -fdiagnostics-color=always -Wall -Wdeprecated -g -fdebug -fPIC -MD -MQ 'subprojects/automem/83545ae@@automem@sha/automem_source_automem_array.d.o' -MF 'subprojects/automem/83545ae@@automem@sha/automem_source_automem_array.d.o.deps' -o 'subprojects/automem/83545ae@@automem@sha/automem_source_automem_array.d.o' -c ../../../code/d/snake/subprojects/automem/automem/source/automem/array.d
/home/michele/dev/code/d/snake/subprojects/automem/automem/source/automem/vector.d:308:11: error: statement expected to be { }, not (
308 | in(start >= 0)
| ^
/home/michele/dev/code/d/snake/subprojects/automem/automem/source/automem/vector.d:310:9: error: found 'do' when expecting ';' following statement
310 | do
| ^
/home/michele/dev/code/d/snake/subprojects/automem/automem/source/automem/vector.d:311:5: error: missing body { ... } after in or out
311 | {
| ^
ninja: build stopped: subcommand failed.
I'm quite new to D and I'm not sure what that syntax means but it looks like gdc doesn't understand it either.
source/automem/utils.d(8,6): Error: shared method automem.ref_counted.TestUtils!().SharedStruct.~this is not callable using a non-shared object
source/automem/utils.d(12,6): Error: shared method automem.ref_counted.TestUtils!().SharedStruct.~this is not callable using a non-shared object
source/automem/ref_counted.d(370,20): Error: template instance automem.ref_counted.RefCounted!(shared(SharedStruct), IAllocator) error instantiating
source/automem/utils.d(8,6): Error: shared method automem.ref_counted.TestUtils!().SharedStruct.~this is not callable using a non-shared object
source/automem/utils.d(12,6): Error: shared method automem.ref_counted.TestUtils!().SharedStruct.~this is not callable using a non-shared object
source/automem/ref_counted.d(461,20): Error: template instance automem.ref_counted.RefCounted!(shared(SharedStruct), TestAllocator*) error instantiating
source/automem/ref_counted.d(513,5): Error: static assert __traits(compiles, sendRefCounted(Allocator, Args...)(Args args)(7)) is false
vector[0 .. vector.length]
throws an error, when it clearly should be equivalent to vector[]
. The culprit: https://github.com/atilaneves/automem/blob/master/source/automem/vector.d#L365
After adding automem 0.0.9 to dub dependencies my -unittest build failed with error message:
../../.dub/packages/automem-0.0.9/automem/source/automem/ref_counted.d(9,12): Error: module test_allocator is in file 'test_allocator.d' which cannot be read
import path[0] = source/
import path[1] = ../../.dub/packages/automem-0.0.9/automem/source/
import path[2] = ../diet-ng/source/
import path[3] = ../../.dub/packages/mysql-native-1.1.0/mysql-native/source/
....
RefCounted.release method is not atomic.
void release() {
/*
...
*/
dec;
if(_impl._count == 0) {
/*
...
*/
}
}
calling "dec" is atomic and "_impl._count == 0" is atomic but together they are not atomic.
Something like this is necessary:
if(atomicFetchSub!(MemoryOrder.acq_rel)(_impl._count , 1) == 1){ //... }
Clang use it too:
https://github.com/llvm-mirror/libcxx/blob/master/include/memory#L3398
/+dub.sdl:
dependency "automem" version="~>0.6.1"
+/
import automem;
StringM str;
void main() @nogc
{
str ~= "a";
str.popBack;
str ~= "a"; // generates the error "Assigning to non default initialised non mutable member" in vector.d:297
assert(str.length == 1);
assert(str[0] == 'a');
}
due to the fact that after popBack
call and adding new element again the last element of the storage has already been initialized so the check fails.
automem/source/automem/vector.d
Lines 208 to 213 in 8f61747
struct Vector(E, Allocator = typeof(theAllocator)) if(isAllocator!Allocator) {
...
E[] _elements;
...
/// Access the ith element. Can throw RangeError.
ref inout(E) opIndex(long i) scope return inout {
if(i < 0 || i >= length)
mixin(throwBoundsException);
return _elements[i.toSizeT];
}
...
The signature is incorrect. Because this
is passed by ref
and the function returns by ref
, the function is return-ref
so it may return the address of this
(this._elements
), but not the value (this._elements[i]
).
Blocking dlang/dmd#12665 (comment)
This @trusted
block of code:
automem/source/automem/vector.d
Lines 313 to 316 in 98d7b1b
allows a pointer to the vector to escape:
/+ dub.sdl:
dependency "automem" version="==0.6.0"
dflags "-preview=dip1000" "-preview=dip25"
+/
import automem.vector;
@safe:
typeof(vector(1).range()) global;
void main()
{
auto vec1 = vector(1, 2, 3);
global = vec1.range; // compiles, but it shouldn't
}
Stack corruption PoC:
/+ dub.sdl:
dependency "automem" version="==0.6.0"
dflags "-preview=dip1000" "-preview=dip25"
+/
import std.stdio : writeln;
import automem.vector;
@safe:
typeof(vector(1).range()) global;
void main()
{
escape;
global.length.writeln; // 0
stackSmash;
global.length.writeln; // 42
}
void escape()
{
auto vec1 = vector(1, 2, 3);
global = vec1.range;
}
void stackSmash()
{
long[4096] arr = 42;
}
$DC --version
DMD64 D Compiler v2.085.0
Copyright (C) 1999-2019 by The D Language Foundation, All Rights Reserved written by Walter Bright
dub automem_test.d
0
42
Seems as if the issue is here:
automem/source/automem/allocator.d
Line 45 in a806e1f
You can find a proof of concept of this here (comment only the for block out),
https://gist.github.com/hatf0/9de09fd8e78e21e42f89199312852858
Example:
/+ dub.sdl:
dependency "automem" version="~>0.6.1"
+/
import automem;
void main()
{
auto r = S();
auto rc = RefCounted!S();
auto un = Unique!S();
foreach(i; r) {} //works
foreach(i; rc) {}
foreach(i; un) {}
}
struct S
{
@property int front() {return 0;}
void popFront(){}
@property bool empty(){return false;}
}
dub test.d :(
test.d(13,5): Error: invalid foreach aggregate rc, define opApply(), range primitives, or use .tupleof
test.d(14,5): Error: invalid foreach aggregate un, define opApply(), range primitives, or use .tupleof
Same thing happens if front is a data member. For unique it's probably not possible to fix this, because of this DMD problem: https://issues.dlang.org/show_bug.cgi?id=15413
But I don't know why it doesn't work for RefCounted.
Probably a user error, but shouldn't this code be correct?
/+ dub.sdl:
dependency "automem" version="~>0.6.1"
+/
import automem;
void main()
{
auto rc = RefCounted!S();
for (auto r2 = rc; !r2.empty; r2.popFront)
{
}
}
struct S
{
int front = 0;
void popFront(){front++;}
@property bool empty(){return front > 100;}
}
Currently one gets error message like Error: incompatible types for (serverURI) == (""): Vector!(immutable(char), Mallocator) and string
Obvious workaround: vec[] == arr
Please tag a new release. The latest tag v0.6.9 is from Feb 17, 2023, and is still used in dmd/phobos's buildkite, blocking dlang/dmd#14364.
not a Vector of immutable elements mind you - but a totally immutable vector.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.