cysharp / mastermemory Goto Github PK

like Grpc.Tools
https://github.com/grpc/grpc/tree/master/src/csharp/Grpc.Tools

refer ProtoCompile
https://github.com/grpc/grpc/blob/master/src/csharp/Grpc.Tools/ProtoCompile.cs

How to add these attributes to the types from other assembly, which could not modify their sources?
Or, how to add these attributes to type types generated by other code generator? Class attributes are ok for generated partial class, but it's hard to add attribute to properties.

Severity Code Description Project File Line Suppression State
Error CS0115 'MemoryDatabase.Init(Dictionary<string, (int offset, int count)>, ReadOnlyMemory, MessagePackSerializerOptions)': no suitable method found to override AspNetCoreWebApp C:\Codes\AspNetCoreWebApp\MasterMemory\MemoryDatabase.cs 33 Active

Severity Code Description Project File Line Suppression State
Error CS8138 Cannot reference 'System.Runtime.CompilerServices.TupleElementNamesAttribute' explicitly. Use the tuple syntax to define tuple names. AspNetCoreWebApp C:\Codes\AspNetCoreWebApp\MasterMemory\MemoryDatabase.cs 89 Active

Currently bin data is always lz4 compressed.
Add uncompression option for some reasons.

I have started testing MasterMemory and while testing different modification of my models structures - to ensure it supports all possible modification, I came across a null string being read and sent to string.Intern making building the database impossible.

In InternStringResolver, there is no null check to the string that is sent to string.Intern.
Adding the null check didn't seem to cause any problem.

Hi,

I just created fresh project in Unity 5.5, with empty GO and MB attached (in zip). This results in exception:

ArgumentException: does not implement right interface
System.Collections.Generic.Comparer1+DefaultComparer[ZeroFormatter.KeyTuple2[Gender,System.Int32]].Compare (KeyTuple2 x, KeyTuple2 y) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/Dictionary.cs:771)
System.Array.compare[KeyTuple2] (KeyTuple2 value1, KeyTuple2 value2, IComparer1 comparer) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/Dictionary.cs:319)
System.Array.qsort[KeyTuple2,Person] (ZeroFormatter.KeyTuple2[] keys, .Person[] items, Int32 low0, Int32 high0, IComparer1 comparer) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/Dictionary.cs:619) System.Array.Sort[KeyTuple2,Person] (ZeroFormatter.KeyTuple2[] keys, .Person[] items, Int32 index, Int32 length, IComparer1 comparer) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/Dictionary.cs:303)
Rethrow as InvalidOperationException: The comparer threw an exception.
System.Array.Sort[KeyTuple2,Person] (ZeroFormatter.KeyTuple2[] keys, .Person[] items, Int32 index, Int32 length, IComparer1 comparer) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/Dictionary.cs:303) MasterMemory.Memory2[ZeroFormatter.KeyTuple2[Gender,System.Int32],Person].FastSort (IEnumerable1 datasource, System.Func2 indexSelector, IComparer1 comparer) (at Assets/MasterMemory/Memory.cs:360)
MasterMemory.Memory2[ZeroFormatter.KeyTuple2[Gender,System.Int32],Person]..ctor (IEnumerable1 datasource, System.Func2 indexSelector, Boolean rootMemory) (at Assets/MasterMemory/Memory.cs:87)
MasterMemory.Memory2[ZeroFormatter.KeyTuple2[Gender,System.Int32],Person]..ctor (IEnumerable1 datasource, System.Func2 indexSelector) (at Assets/MasterMemory/Memory.cs:77)
MemoryTesting.Start () (at Assets/MemoryTesting.cs:71)

I am using unitypackage from releases for MasterMemory (0.1.0) and binary (dll) releases for ZeroFormatter (1.6.0)

MemoryTesting.zip

Since C# 9 we have a great new type supporting immutability: records.
I saw that MessagePack now supports records, would be cool is MasterMemory supports it too.

I'm using MasterMemory, but I intend to have multiple databases. One database will represent gameplay data and should be synchronized to cloud storage, but another will represent configuration data and should stay on one device. These databases will each be saved to separate files; how can I do this with MasterMemory?

Hi! I found minor issue. That issue not problem for me, but I think that report can be helpfull for someone.

Fail in console app running on MessagepackCompiler.RunAsync
MessagePackCompiler.CodeAnalysis.MessagePackGeneratorResolveFailedException: can't find matched constructor. type:global::_Game.DataModel.Stats
   at MessagePackCompiler.CodeAnalysis.TypeCollector.CollectObject(INamedTypeSymbol type) in d:\a\1\s\src\MessagePack.GeneratorCore\CodeAnalysis\TypeCollector.cs:line 541
   at MessagePackCompiler.CodeAnalysis.TypeCollector.CollectCore(ITypeSymbol typeSymbol) in d:\a\1\s\src\MessagePack.GeneratorCore\CodeAnalysis\TypeCollector.cs:line 384
   at MessagePackCompiler.CodeAnalysis.TypeCollector.CollectObject(INamedTypeSymbol type) in d:\a\1\s\src\MessagePack.GeneratorCore\CodeAnalysis\TypeCollector.cs:line 589
   at MessagePackCompiler.CodeAnalysis.TypeCollector.CollectCore(ITypeSymbol typeSymbol) in d:\a\1\s\src\MessagePack.GeneratorCore\CodeAnalysis\TypeCollector.cs:line 384
   at MessagePackCompiler.CodeAnalysis.TypeCollector.Collect() in d:\a\1\s\src\MessagePack.GeneratorCore\CodeAnalysis\TypeCollector.cs:line 319
   at MessagePackCompiler.CodeGenerator.GenerateFileAsync(String input, String output, String conditionalSymbol, String resolverName, String namespace, Boolean useMapMode, String multipleIfDirectiveOutputSymbols) in d:\a\1\s\src\MessagePack.GeneratorCore\CodeGenerator.cs:line 58
   at MessagePack.Generator.MessagepackCompiler.RunAsync(String input, String output, String conditionalSymbol, String resolverName, String namespace, Boolean useMapMode, String multipleIfDirectiveOutputSymbols) in d:\a\1\s\src\MessagePack.Generator\MessagepackCompiler.cs:line 30
   at ConsoleAppFramework.ConsoleAppEngine.RunCore(ConsoleAppContext ctx, Type type, MethodInfo methodInfo, String[] args, Int32 argsOffset)

    [MessagePackObject(true)]
    public class Stats
    {
        public int Str { get; }
        public int Con { get; }
        public int Dex { get; }
        public int Int { get; }
        public int Men { get; }
        public int Wit { get; }
        public int Cha { get; }
        public int Lck { get; }

        public Stats(int str, int con, int dex, int intelligence, int men, int wit, int cha, int lck)
        {
            Str = str;
            Con = con;
            Dex = dex;
            Int = intelligence; // <== Problem here. Compiler doesn't like 'Int' name 
            Men = men;
            Wit = wit;
            Cha = cha;
            Lck = lck;
        }
    } 

I can not find the method builder.AppendDynamic from generated code files. How can I fix it?

Try the following test (added to tests\MasterMemory.Tests\DatabaseTest.cs)

Currently the db.SampleTable.FindRangeByAge( min: 2, max: 2) returns a range with Count 1 and pointing at the first element in the sorted list - which has age 9.

The problem is due to First / Last being used on empty element in FindManyRangeCore.

I have a suggested fix; which also improves the efficiency of FindManyRangeCore (in TableBase.cs) which I will attempt to upload. I am not very familiar with git however.

If the above behavior is by design - please let me know.

Steps to create:

        Sample[] CreateData()
        {
            // Id = Unique, PK
            // FirstName + LastName = Unique
            var data = new[]
            {
                new Sample { Id = 5, Age = 19, FirstName = "aaa", LastName = "foo" },
                new Sample { Id = 6, Age = 29, FirstName = "bbb", LastName = "foo" },
                new Sample { Id = 7, Age = 39, FirstName = "ccc", LastName = "foo" },
                new Sample { Id = 8, Age = 49, FirstName = "ddd", LastName = "foo" },
                new Sample { Id = 1, Age = 59, FirstName = "eee", LastName = "foo" },
                new Sample { Id = 2, Age = 89, FirstName = "aaa", LastName = "bar" },
                new Sample { Id = 3, Age = 79, FirstName = "be", LastName = "de" },
                new Sample { Id = 4, Age = 89, FirstName = "aaa", LastName = "tako" },
                new Sample { Id = 9, Age = 99, FirstName = "aaa", LastName = "ika" },
                new Sample { Id = 10, Age = 9, FirstName = "eee", LastName = "baz" },
            };
            return data;
        }

         [Fact]
        public void Ranges()
        {
            var builder = new DatabaseBuilder();
            builder.Append(CreateData());

            var bin = builder.Build();
            var db = new MemoryDatabase(bin);

            db.SampleTable.FindRangeByAge(2,2).Select(x=>x.Id).ToArray().Should().BeEquivalentTo( new int[] {} );     
            db.SampleTable.FindRangeByAge(30,50).Select(x=>x.Id).ToArray().Should().BeEquivalentTo( new int[] { 7, 8 } );     
            db.SampleTable.FindRangeByAge(100,100).Select(x=>x.Id).ToArray().Should().BeEquivalentTo( new int[] {} );     
        }

as title says,i have test store HashSet and it seems works fine,but I didn't see the information of the storage collection type in the sample, so I want to get some confirmation information here before using it in the production environment: Can the storage collection type be read normally when the table is not used as a query key?
And if possible, I also want to ask whether the collection type can be used as a query key by customizing the comparator?

If you run dotnet-mmgen on this class...

using MasterMemory;
using MessagePack;

namespace CorundumGames
{
    [MessagePackObject, MemoryTable("BugTable")]
    public sealed class BugEntry
    {
        private const int SecondaryKeyIndex = 0;

        [Key(0)]
        [PrimaryKey]
        public int Primary { get; }

        [Key(1)]
        [SecondaryKey(SecondaryKeyIndex)]
        public string Secondary { get; }
    }
}

...then it will fail with an error message that looks something like this:

Fail in console app running on Program.Execute
System.NullReferenceException: Object reference not set to an instance of an object.
   at MasterMemory.GeneratorCore.CodeGenerator.ExtractPropertyAttribute(PropertyDeclarationSyntax property) in /home/runner/work/MasterMemory/MasterMemory/src/MasterMemor
y.GeneratorCore/CodeGenerator.cs:line 205
   at MasterMemory.GeneratorCore.CodeGenerator.<CreateGenerationContext>b__4_5(PropertyDeclarationSyntax x) in /home/runner/work/MasterMemory/MasterMemory/src/MasterMemor
y.GeneratorCore/CodeGenerator.cs:line 165
   at System.Linq.Enumerable.SelectEnumerableIterator`2.ToArray()
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at MasterMemory.GeneratorCore.CodeGenerator.CreateGenerationContext(String filePath)+MoveNext()
   at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
   at MasterMemory.GeneratorCore.CodeGenerator.GenerateFile(String usingNamespace, String inputDirectory, String outputDirectory, String prefixClassName, Boolean addImmut
ableConstructor, Boolean throwIfKeyNotFound, Boolean forceOverwrite, Action`1 logger) in /home/runner/work/MasterMemory/MasterMemory/src/MasterMemory.GeneratorCore/CodeGe
nerator.cs:line 30
   at MasterMemory.Generator.Program.Execute(String inputDirectory, String outputDirectory, String usingNamespace, String prefixClassName, Boolean addImmutableConstructor
, Boolean returnNullIfKeyNotFound, Boolean forceOverwrite) in /home/runner/work/MasterMemory/MasterMemory/src/MasterMemory.Generator/Program.cs:line 31

However, running it on this (look at the [SecondaryKey])...

using MasterMemory;
using MessagePack;

namespace CorundumGames
{
    [MessagePackObject, MemoryTable("BugTable")]
    public sealed class BugEntry
    {
        [Key(0)]
        [PrimaryKey]
        public int Primary { get; }

        [Key(1)]
        [SecondaryKey(0)]
        public string Secondary { get; }
    }
}

...works as expected.

OverflowException occurs when trying to add large data to DatabaseBuilder.
Add large data using the ImmutableBuilder's Diff method does not throw OverflowException .

Is it possible to avoid this Exception by changing some settings?

Here is the source code I ran

var builder = new DatabaseBuilder();
var list = new List<Person>();
for (int i = 0; i < 100000000; i++)
{
    list.Add(new Person { PersonId = i, Age = i, Gender = (Gender)(i % 2), Name = "AAAAAAAAAAA" });
}
builder.Append(list);

ImmutableBuilder's Diff method does not throw OverflowException

var builder = new DatabaseBuilder();
var list = new List<Person>();

for (int i = 0; i < 100000000; i++)
{
    list.Add(new Person { PersonId = i, Age = i, Gender = (Gender)(i % 2), Name = "AAAAAAAAAAA" });
}

var firstData = list.First();
builder.Append(new List<Person>() { firstData });
byte[] dataA = builder.Build();
var db = new MemoryDatabase(dataA);

var builder1 = db.ToImmutableBuilder();
builder1.Diff(list.Skip(1).ToArray());
db = builder1.Build();

builder.Append(list);

MessagePack.MessagePackSerializationException   HResult=0x80131500   Message=Failed to serialize MasterMemorytest3.Model.Person[] value.   Source=MessagePack   スタック トレース:    場所 MessagePack.MessagePackSerializer.Serialize[T](MessagePackWriter& writer, T value, MessagePackSerializerOptions options)    場所 MessagePack.MessagePackSerializer.Serialize[T](IBufferWriter`1 writer, T value, MessagePackSerializerOptions options, CancellationToken cancellationToken)    場所 MasterMemory.DatabaseBuilderBase.AppendCore[T,TKey](IEnumerable`1 datasource, Func`2 indexSelector, IComparer`1 comparer)    場所 MasterMemorytest3.DatabaseBuilder.Append(IEnumerable`1 dataSource) (F:\\src\\source\\repos\\MasterMemorytest\\MasterMemorytest3\\MasterMemory\\DatabaseBuilder.cs):行 29    場所 MasterMemorytest3.Program.Test1() (F:\\src\\source\\repos\\MasterMemorytest\\MasterMemorytest3\\Program.cs):行 39    場所 MasterMemorytest3.Program.Main(String[] args) (F:\\src\\source\\repos\\MasterMemorytest\\MasterMemorytest3\\Program.cs):行 21    この例外は、最初にこの呼び出し履歴      [外部コード] でスローされました  内部例外 1: OverflowException: Arithmetic operation resulted in an overflow.

Using Unity 2018.3.7. Importing MasterMemory as-is results in 58 different "type or namespace cannot be found" errors for the following: TypeBuilder, ModuleBuilder, AssemblyBuilder, LocalBuilder, FieldBuilder, MethodBuilder and ILGenerator.

When I delete the MessagePack folder and import the MessagePack 1.7.3.5, all those errors go away and a new one appears:

Assets\Plugins\MasterMemory\Scripts\Loader.cs(57,13): error CS0103: The name 'DefaultResolver' does not exist in the current context

<PackAsTool>true</PackAsTool>
refer https://github.com/Cysharp/Ulid/blob/master/src/Ulid.Cli/Ulid.Cli.csproj

I do not know if this is a good idea, but wouldn't it be better for RangeView to implement ICollection ? It already has a count method. This would improve performance when using Linq on RangeView as a IEnumerable.
This is useful for framework that needs abstraction.

This package has no Assembly Definition. so, I cannot add MasterMemory attributes to classes in other assemblies in Unity.
Is supporting Assembly Definition planned?

https://github.com/dotnet/csharplang/blob/master/proposals/records.md

Hi,

How can we add items if the database is initialized?
Would like to use your db as a secondary Index for searching where results get updated once per hour appx.

Thanks

for example, define by attribute

[ForeignKey(0, typeof(ItemMaster), nameof(ItemMaster.ItemId))]
public int RewardItemId { get; }

automatically generate Exists validation and automatically generate navigation property.

Currently can not implements navigation proeprty so pending implementation.

MasterMemory's design, MemoryTableAttribute always have to use MessagePackObjectAttirbute.
Implements two attributes is slightly painful.
If MemoryTableAttribute itself includes MessagePackObjectAttribute, we can use only one MemoryTableAttribute.

Semantic Versioning is a widely used convention in software development, distribution, and deployment.
I propose that MasterMemory adopt the semantic versioning conventions for future releases. If there are good reasons for not adopting this convention, MasterMemory should adopt a release labelling scheme that cannot be mistaken for semantic versioning.

MasterMemory.Generator CLI(dotnet-mmgen) does not work on .NET6.

It was not possible to find any compatible framework version
The framework 'Microsoft.NETCore.App', version '5.0.0' (arm64) was not found.
  - The following frameworks were found:
      6.0.5 at [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

and .NET5 SDK is End of support May 10, 2022.

https://github.com/tcnksm/ghr

Do you have plans to support Microsoft Entity Framawork Core?

• Memory
  • Query Single Key
    • Find
    • FindNearest
    • FindRange
    • FindAll
  • Query Multiple Key(KeyTuple)
    • Find
    • FindNearest
    • FindRange
    • FindAll
  • Secondary Key
  • Index KeyTuple Deconstruction
• View
  • RangeView
  • LookupView
  • DictionaryView
• Database
  • Serialize
  • Deserialize
• .NET Core
• Unity
  • Create Project
  • Setup Cloudbuild
  • How Create KeyTupleComparer?
• NuGet Upload
• ReadMe

Hey, thanks for your amazing work as usual. I wanted to test if the library might work out for my use case, but right now I'm not getting the result returned that I expected. Basically I expected it to find the person object with firstname == "realname" but I got the different one. Is this expected behavior? Thanks for your help!

  [MemoryTable("people"), MessagePackObject(true)]
  public class PersonModel
  {
      [SecondaryKey(0), NonUnique]
      [SecondaryKey(1, keyOrder: 1), NonUnique]
      public string LastName { get; set; }

      [SecondaryKey(2), NonUnique]
      [SecondaryKey(1, keyOrder: 0), NonUnique]
      public string FirstName { get; set; }

      [PrimaryKey] public string RandomId { get; set; }
  }

I'm considering using MasterMemory for some of my Unity game's save data, but I'd like to ask about my use case to make sure I understand what this library is designed for.

Here are some facts about my game's data:

• I am using MessagePack-CSharp for serialization.
• I do not intend to use MasterMemory for game design assets (e.g. difficulty, level design, game mechanics); for that I will just use standard ScriptableObjects and prefabs.
• Most of my game's save data is only loaded once at the start of the game. This data is used every couple of minutes.
• Part of my game's save data consists of local leaderboards for each game mode that are updated with one entry at a time whenever the player finishes a session. When that happens (usually once every few minutes), the data is saved to disk.
• My game will have online leaderboards, but I haven't yet decided how it will be managed or structured.
• Another part of my game's save data consists of input configuration, which is fairly complicated due to the nature of the input manager I use. This data is referenced whenever a controller is connected, disconnected, or assigned to a player. It is modified whenever the player customizes their controls (very rare) or when the player is assigned a controller (more common).

Given this information, do you believe MasterMemory would be well-suited for my game?

Hi team, firstly this is awesome!

I have a question: I want to share database between 2 or more application, can I do that

I guess maybe I need a connectionstring or an address to make my applications understand where database is.

Have we support it yet or any road map for it

Thank you

should return empty

The following code is not compilable when using the following code to initialize and query MasterMemory under .NET 7:

using MasterMemory;
using MasterMemoryDemo;
using MessagePack;

var builder = new DatabaseBuilder();
byte[] data = builder.Build();

// -----------------------

var db = new MemoryDatabase(data);


[MemoryTable("PortedMsisdns"), MessagePackObject(true)]
public class MsisdnInfo
{
    [PrimaryKey]
    public string Msisdn { get; set; }
    public string Operator { get; set; }
}

The error message from the compiler is

  MemoryDatabase.cs(32, 33): [CS0115] 'MemoryDatabase.Init(Dictionary<string, (int offset, int count)>, ReadOnlyMemory<byte>, MessagePackSerializerOptions)': no suitable method found to override
  MemoryDatabase.cs(16, 24): [CS0534] 'MemoryDatabase' does not implement inherited abstract member 'MemoryDatabaseBase.Init(Dictionary<string, (int offset, int count)>, ReadOnlyMemory<byte>, MessagePackSerializerOptions, int)'

I guess you must have considered this :)
Issue would be compatibility between version. Or maybe create a new package with the MemoryPack implementation: MasterMemory.MemoryPack?

I want to visually edit data in Unity (using odininspector), but this tool does not support serialization property.

I've started join query support.
a40278c