grijjy / grijjyfoundation Goto Github PK

Hello.
using TgoBsonSerializer.Serialize into the following record:
RInfo = record Usages: TArray<Byte>; end;

Will serialize into:
[
{
"Usages":{
"$binary":"DwoRCxANERM=",
"$type":"00"
},
}
]

instead of:
[
{
"Usages":[
11,
15,
17,
5,
17,
15,
28,
46
]
}
]

if i switch to shortint the result will output correctly.

Thank you

this event can be implemented like:

TOnSent = procedure(Sender: TObject; const ABuffer: Pointer; const ASize: Integer) of object;

  TgoHttpClient = class(TObject)
..
   { Fired when the data has been sent by the socket }
    property OnSent: TOnSent read FOnSent write FOnSent;

        FConnection := _HttpClientSocketManager.Request(FURI.Host, FURI.Port);
..
        FConnection.OnSent := OnSocketSent; // <<< added
        FConnection.OnRecv := OnSocketRecv;

procedure TgoHttpClient.OnSocketSent(const ABuffer: Pointer;
  const ASize: Integer);
begin
  FLastSent := Now;
  if Assigned(FOnSent) then FOnSent(Self, ABuffer, ASize); //added
end;

Currently, all records field are always cleared on de-serialization. It would be positive to be able to have excessive fields which could be preset and kept during de-serialization.
(protobuf) (enhancement)

In unit Grijjy.Http; TThreadSafeBuffer.Read Line #684 I found;
Move(FBuffer[Size], FBuffer[0], FSize - ALength);
obviously this reads behind Buffer Size. Should be:
Move(FBuffer[ALength], FBuffer[0], FSize - ALength);

It may be positive to introduce skipping of default values on de/serialization, either via pre-defined list of type defaults, either via an additional attribute(?).
(enhancement)

Can TgoHttpClientManager add a Request Http Client Object method?

While testing json/bson library with records for serialize/de-serialize I noticed that any non-US characters are converted to their binary representations.

For example "Yazılım" text contain some national characters (small i letters without dot at top) converted as "Yaz\u0131l\u0131m"

There is no problem that library de-serialize this representation. I also tested that initial text "Yazılım" is also successfully being de-serialized.

My question is: Is it possible to serialize the text as it is?

Thanks.

Hi,

Is there a possibility to create a Tvalue from a TgoBsonDocument?

This one is not working and results in a empty TgoBsonDocument record:
TValue.Make(@BsonDoc,TypeInfo(TgoBsonDocument),RttiValue);
Or
RttiValue.From<TgoBsonDocument>(BsonDoc);

The other way is working:
RttiValue.AsType<TgoBsonDocument>

Best regards
Dennis

Hello,
I'm currently trying to implement the ProtocolBuffer classes to serialize objects as protobuffers in Delphi. When comparing the byte results to the same protobuffer in .NET Core, I noticed that the raw bytes are not the same when serializing the same object in .NET Core.

Delphi

Person record

  TPerson = record
    [Serialize(1)] Id: Integer;
  end;

Raw bytes

SerializedData		(8, 242, 192, 1)
	[0]	8
	[1]	242
	[2]	192
	[3]	1

.NET Core 3.1 (C#)

Person class

    [ProtoContract]
    class Person
    {
        [ProtoMember(1)]
        public int Id { get; set; }
    }

Raw bytes

SerializedData		(8, 185, 96)
	[0]	8
	[1]	185
	[2]	96

Versions of the used libraries in .NET Core:

• protobuf-net (2.4.4)

Conclusion

Due to differences in the raw byte structure of the serialized data of the object, it's not possible to serialize an object in Delphi and deserialize it in .NET Core.
It would be nice if you could provide additional information about the version of the protobuffer standard you implemented, so I could make tests myself. The best option would be an update to the newest standard of protobuffers.

Thank you in advance,

Philipp

Hello,
I might have missed an option to set. If that is the case, just let me know what I should do, please.

Assume that you have following record definition:

  TSubReturn = record
  public
    code: Integer;
    desc: string;
  end;

  TReturn = record
  public
    code: Integer;
    desc: string;
    subReturn: TSubReturn;
  end;

  TEODBatchUploadResponse = record
  public
    &message: string;
    requiredFileSequence: Integer;
    ret: TReturn;
  end;

My regular json string is:
{"EodBatchUploadResult":{"ResponseCode":"0","ResponseDescription":"","ResponseSubCode":null,"message":null,"requiredFileSequence":0,"ret":{"code":0,"desc":null,"subReturn":{"code":0,"desc":null}}}}

However, from time to time, server returns following response:
{"ResultCode":2,"ResultMessage":"Error"}

Both of these example json strings are parsed as OK. Meaning, no exception during de-serialization operation raised.

I do not need to parse second example. On the other hand, I would like an exception or a reference like json is not parsed because none of the record fields exists in it.

Let me know if this is a feature, of Grijjy, please.

TgoHttpClient uses TMessageManager that is defined since XE6 in System.Messaging.
it would be great, if this class had some workaround for compiling this unit with preXE6 versions

I noticed that I get an error that serializer can serialize only class and records,.
The problem is found in Grijjy.Bson.Serialization file and it is caused by the ATypeInfo.Kind that returns the record I am using as tkmrecord.
This record is show as tkmrecord because I am using a "class operator Initialize(out rec: Depvis);". I remove it, it is again a tkrecord.
I add tkmrecord in every "case of" that tkrecord existed, and it works. Is this a problem?

Hello,
Using protocol buffers when i try to (de)serialize a record from another unit it throws an error:
Unable register record type MyRecord. It does not contain any [Serialize] fields

Placing the record into the unit i perform the (de)serialize works!

Any ideas ? Attached sample project.

Thank you

according to Delphinus.Info.json, these platforms are supported: "Win32;Win64;OSX32;Android;iOSDevice32;iOSDevice64;Linux64", but Grijjy.Http.pas can be compiled under Windows and Linux only.

workaround: replace {$IFDEF LINUX} with {$IFDEF POSIX}

When you have record as

TTest = record
  [Serialize(01)] Array: TArray<anytype>;  
end;

If Array item is nil (empty) then code in Grijjy.ProtocolBuffers at line 1090 crashes due to Range Check Error because RecBytes has no [0] item.

Correct implementation should be

...
if (Len > 0) then begin
  AReader.ReadBytes(RecBytes[0], Len);
  RecReader := TReader.Create(@RecBytes[0], Len);
  try
    Info.Deserialize(RecReader, P);
  finally
    RecReader.Free;
  end;
end;

it will be usable for debugging.

it can be something like

unit Grijjy.System.Logging;

interface

type
  TgoLog = (ToFile, ToConsole, ToDefault);
  TgoLogs = set of TgoLog;

  TgoLogging = class
  public
    constructor Create(aMode: TgoLogs; aName: String);
    procedure Send(aMessage: String);
  end;

implementation
uses
  Winapi.Windows;

constructor TgoLogging.Create(aMode: TgoLogs; aName: String);
begin
end;

procedure TgoLogging.Send(aMessage: String);
begin
{$IFDEF MSWINDOWS}
  OutputDebugString(PChar(aMessage));
{$ENDIF}
end;

end.

nowadays, it is accepted only as property:

• TgoOpenSSL:

    { Password for private key }
    property Password: UnicodeString read FPassword write FPassword;

• TgoSocketConnection:

    { Password for private key }
    property Password: String read GetPassword write SetPassword;
...
function TgoSocketConnection.GetPassword: String;
begin
  Result := OpenSSL.Password;
end;
procedure TgoSocketConnection.SetPassword(const Value: String);
begin
  OpenSSL.Password := Value;
end;

[dcc32 Hint] Grijjy.Collections.pas(909): H2077 Value assigned to 'TgoReadOnlySet.Contains' never used

and

[dcc32 Hint] Grijjy.Collections.pas(1807): H2077 Value assigned to 'TgoValueDictionary<TKey,TValue>.ContainsKey' never used

Propose changing them into:

function TgoReadOnlySet.Contains(const AItem: T): Boolean;
var
Mask, Index, HashCode, HC: Integer;
begin
Result := False;

if FCount = 0 then
Exit;

HashCode := FComparer.GetHashCode(AItem) and $7FFFFFFF;
Mask := Length(FItems) - 1;
Index := HashCode and Mask;

while True do
begin
HC := FItems[Index].HashCode;
if (HC = EMPTY_HASH) then
Exit(False);

if (HC = HashCode) and FComparer.Equals(FItems[Index].Item, AItem) then
  Exit(True);

Index := (Index + 1) and Mask;

end;
end;

and

function TgoValueDictionary<TKey, TValue>.ContainsKey(
const AKey: TKey): Boolean;
var
Mask, Index, HashCode, HC: Integer;
begin
Result := False;

if FCount = 0 then
Exit;

HashCode := FComparer.GetHashCode(AKey) and $7FFFFFFF;
Mask := Length(FItems) - 1;
Index := HashCode and Mask;

while True do
begin
HC := FItems[Index].HashCode;
if (HC = EMPTY_HASH) then
Exit(False);

if (HC = HashCode) and FComparer.Equals(FItems[Index].Key, AKey) then
  Exit(True);

Index := (Index + 1) and Mask;

end;
end;

-Tee-

Current code in Grijjy.OpenSSL.API does not include the name of the DLL:

function GetProc(AModule: HMODULE; const AProcName: String): Pointer;
begin
  Result := GetProcAddress(AModule, PChar(AProcName));
  if (Result = nil) then
    raise Exception.CreateFmt('%s is not found', [AProcName]);
end;

Even the short name of the DLL will make it hard to find why it fails: only the full name will give enough context on why it does not load.

(I will make a pull request for this later)

Hello

I got a bit bored and I consider adding Your ZeroMQ wrapper as another Delphinus package but...

...to add "Grijjy foundation" as a dependency it have to contain at least one revision marked as "release" (https://github.com/grijjy/GrijjyFoundation/releases) 😛

It's not any crucial issue but could You guys add some tag / tags for current / future releases in this repository?
[Thanks!]

Regards
P.

nowadays TgoHttpClient has

    { Certificate in PEM format }
    property Certificate: TBytes read FCertificate write FCertificate;
    { Private key in PEM format }
    property PrivateKey: TBytes read FPrivateKey write FPrivateKey;

but hasn't

    { Password for private key }
    property Password: String read GetPassword write SetPassword;

workaround:
add this property to TgoHttpClient or give access to FConnection: TgoSocketConnection;

Sample showing issue: https://services.odata.org/TripPinRESTierService/People
FURI := TURI.Create(FURL); doesn't understand context of original request, therefore can't determine http/https and dies with
if PosScheme = -1 then
if ARaiseNoSchema then
raise ENetURIException.CreateResFmt(@SNetUriInvalid, [AURIStr])
else
Exit;

Pls make Host/Port properties as writable in TgoSocketConnection.

by other hand, you can create a new public method

    function Connect(const AHostname: String;
      const APort: Word; const AUseNagle: Boolean = False): Boolean;

that will call private PostConnect method internally.

Is it possible? I didn't found anything about it.

Nowadays Grijjy.SocketPool.Win.TgoSocketConnection has 2 Send methods:

    { Sends the bytes to the socket }
    function Send(const ABuffer: Pointer; const ASize: Integer): Boolean; overload;
    function Send(const ABytes: TBytes): Boolean; overload;

but Grijjy.SocketPool.Linux.TgoSocketConnection has one Send method:

function Send(const ABytes: TBytes): Boolean;

his implementation is

function TgoSocketConnection.Send(const ABytes: TBytes): Boolean;
begin
  Result := Write(ABytes, Length(ABytes));
end;

Nowadays for sending data via linux version you have to create TBytes and move your data to it.
it is bottleneck if you pass big-size data

For example: PObject(@item)^.DisposeOf;

[dcc32 Warning] Grijjy.Collections.pas(1140): W1000 Symbol 'DisposeOf' is deprecated: 'Use Free instead'

q. How to serialize TObjectList using TgoBsonSerializer?

In some REST API (i.e. Activiti) there are cases in witch a boolean value is used as a three state value.
I.e. in Activiti there are some boolean values used as a query criteria with this behavior (i.e the "suspended" parameter) :

• if the value is true then only the suspended process instances are returned
• if the value is false only the not suspended process instances are returned
• if non value is specified all instances are returned.

This behavior cannot be obtained using a record like this :
MyRecord = record
suspended: boolean;
end;

I've solved declaring suspended as string and using the [BsonIgnoreIfDefault] but this not the right way to solve the problem.

There should be a specific type to support this use case (quite frequent). Or, as alternative, add support for the Variant type which can handle null values

Hi,

In Grijjy.Collections Unit, IsTypeManaged() function is used.
But this function is only for newer delphiXE7.

Your GrijjyFoundation is support only for newer XE7?
Or is any option to use GrijjyFoundation unit in older XE5?

as I see, only TCP/IP v4 sockets are supported nowadays

Hello everyone AddRJavaToClassesDex returns an error:
[Exec Error] The command "D:\Delphi\Isxodniki\Android\Gocha\NewVersion\Client\Enigma\Components\DelphiSocialFrameworks\trunk\LoginWithFacebook\Resources\Android\AddRJavaToClassesDex.bat D:\Delphi\Isxodniki\Android\Gocha\NewVersion\Client\Enigma\Components\DelphiSocialFrameworks\trunk\LoginWithFacebook Debug "C:\Users\Public\Documents\Embarcadero\Studio\21.0\CatalogRepository\AndroidSDK-2525-21.0.38860.1461\build-tools\29.0.3\Aapt.exe" "C:\Users\Public\Documents\Embarcadero\Studio\21.0\CatalogRepository\AndroidSDK-2525-21.0.38860.1461\platforms\android-29\android.jar" "C:\Users\Public\Documents\Embarcadero\Studio\21.0\CatalogRepository\AndroidSDK-2525-21.0.38860.1461\build-tools\29.0.3\dx.bat"" exited with code 1.

how to fix it? thanks in advance

I'm unable to serialize TDateTime member. The deserialization works fine i.e if I try to deserialize a value like this one "createTime": "2017-09-05T15:58:39.286+02:00" create a correct DateTime value.

But, when I try to serialize a TDateTime member then the result is something like this : {createDate":{"$date" : 150465600000}}
I'm doing something wrongs ?

After the client connects to the server successfully, the state becomes the Connected. At this time, the server is closed, and the state is still connected at the time of the request. Restart the service program, in the request, no return value. If the status is connected, can you tell if the socket is valid?

Allow serializing not only nested records, but also nested pointers to records (which are interpreted as normal records) to allow flexible internal structure changes.
This may allow support of class se/de-realization with just one additional "content" record.
(suggestion)

Hello,
Attached a sample project!

Deserialize.Memory.Leak.zip

say, if TgoSocketConnection.SSL is set to true.
nowadays, they are always loaded.

when adding
uses Grijjy.CloudLogging;
in blank multi-device application on Delphi 11.3, compiling for android-32, i get an error

[DCC Error] E2597 C:\Users\Public\Documents\Embarcadero\Studio\22.0\CatalogRepository\AndroidNDK-21-22.0.48361.3236\android-ndk-r21\toolchains\arm-linux-androideabi-4.9\prebuilt\windows-x86_64\bin\arm-linux-androideabi-ld.exe: error: cannot find -lgnustl_static
C:/Grijjy/Projects/ThirdParty/ZeroMQ/Android/zmq//jni/../../zeromq/src/zmq.cpp:163: error: undefined reference to 'operator new(unsigned int, std::nothrow_t const&)'
C:/Grijjy/Projects/ThirdParty/ZeroMQ/Android/zmq//jni/../../zeromq/src/zmq.cpp:166: error: undefined reference to 'std::nothrow'
...
...

Any Idea how to fix this?
Thanks a lot

The JSON reading uses US format settings, but writing uses the default.

This for instance fails on systems having a decimal comma.

add possibility to validate peer's certificate

If I have a property of TStringList or TStrings type I got an exception
If I decorated with [BsonIgnore] attribute nothing happen, serialization dont work.
If I remove this TStringList property all work as expected.

if you call this method just after creation this component, you will receive AV at FConnection.Disconnect; line because FConnection is nil here

workaround - just add simple checking like:

procedure TgoHttpClient.Close;
begin
  FRecvAbort := True;
  FRecv.SetEvent;
  if FConnection <> nil then FConnection.Disconnect;
end;

Hello team Grijjy,

I create record type with TgoBsonArray but TgoBsonSerializer.Serialize not parse it. Please fix it for me?

My code here:

procedure TestJson;
type
  TMyType = record
  public
    values: TgoBsonArray;
  end;
var
  T, T2: TMyType;
  E: TgoBsonArray;
  Json: string;
begin
  T.values := TgoBsonArray.Create;
  E := TgoBsonArray.Create(['A', 1]);
  T.values.Add(E);
  E := TgoBsonArray.Create(['B', 2]);
  T.values.Add(E);

  TgoBsonSerializer.Serialize(T, Json);
  //Value of T is '{ "values" : { } }' . Not parse TgoBsonArray?
  TgoBsonSerializer.Deserialize(Json, T2);
end;

Thank you!

I am using TgoHttpClient with http2 as a client to connect to a gRPC server.
The connection works fine, but if I do a a bad request there is no error. The statuscode is still 200.
gRPC uses a grpc-status and grpc-message header, but I am not able to use this header since it is filtered out for some reason.

I found that if I change frame.headers.cat = NGHTTP2_HCAT_RESPONSE to (frame.headers.cat in [NGHTTP2_HCAT_RESPONSE,3]) I do get the correct headers. on https://nghttp2.org/documentation/enums.html#c.nghttp2_headers_category I found what 3 means and
on https://github.com/ultraware/DelphiGrpc/tree/master/ngHttp2 I found there is constant declared for this.

function TgoHttpClient.nghttp2_on_header_callback(session: pnghttp2_session; const frame: pnghttp2_frame;
  const name: puint8; namelen: size_t; const value: puint8; valuelen: size_t;
  flags: uint8; user_data: Pointer): Integer;
var
  AName, AValue: String;
  Index: Integer;
begin
  {$IFDEF LOGGING}
  grLog('on_header_callback');
  {$ENDIF}
  if frame.hd.&type = _NGHTTP2_HEADERS then
    if (frame.headers.cat in [NGHTTP2_HCAT_RESPONSE,3]) then
    begin
      .....
    end;
end;

Example from https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md.

HEADERS (flags = END_HEADERS)
:status = 200
grpc-encoding = gzip
content-type = application/grpc+proto

DATA
<Length-Prefixed Message>

HEADERS (flags = END_STREAM, END_HEADERS)
grpc-status = 0 # OK
trace-proto-bin = jher831yy13JHy3hc

Is the solution to update https://github.com/grijjy/DelphiRemotePushSender with https://github.com/ultraware/DelphiGrpc/tree/master/ngHttp2 and use NGHTTP2_HCAT_HEADERS instead of the hardcoded 3 ?

There is a memory leak when using TgoBsonSerializer.Deserialize (AJson, AValue) and AJosn has invalid or badly formatted json content.

I realize this when using:

ReportMemoryLeaksOnShutdown: = True;

at application startup,

call the procedure Deserialize with an invalid string.

When closing the application I receive the message:

Unexpected Memory Leak

An unexpected memory leak has occurred. The unexpected small block leaks are:

37 - 44 bytes: UnicodeString x 1
53 - 60 bytes: UnicodeString x 1
61 - 68 bytes: UnicodeString x 2
69 - 76 bytes: UnicodeString x 1
77 - 84 bytes: UnicodeString x 2
109 - 116 bytes: UnicodeString x 2
117 - 124 bytes: Unknown x 1