cross-language-cpp / djinni-generator Goto Github PK

Currently the header is generated into --objc-out , but it should go into--objc-header-out , or a user defined location

The naming is also something that could also be checked, if you choose foo-umbrella for the header name (note, no extension) fooumbrellaVersionNumber is generated.
Not optimal, but the -umbrella part seems to be what some people use.

We also need to check if the code that is generated is fit for 2021 and laster

And when looking over that, maybe other things will be found.
So ideas and suggestions are welcome, if possible they will be integrated.

Hi,

I talked to "Li Feng", the engineer from Snapchat that is responsible for Djinni on Snapchat to understand all modifications that they do in Djinni source and all the new features added.

I think that it is too much relevant, because will speed up somethings (like strings), have some general fixes and will add support for more field types (like transfer pointers/ownership between languages).

I will bring the source code and the cross-language-cpp guys can check the pull-request slowly as i bring it to our repository.

The Snapchat repository is here:
https://github.com/Snapchat/djinni

The Snapchat article is here:
https://eng.snap.com/improving_djinni

Some modifications that i collect from their README:

• Replaced sbt and gyp with Bazel
• Added move assigment operator to GlobalRef in all djinni_support.hpp files
• Made JniClassInitializer constructor public to allow arbitrary additional initialization in JNI_OnLoad()
• Speed up string passing between Java and C++ (about 5-10x faster depending on string size)
• Eliminate CppProxy finalizers (better stability)
• Injecting code with DJINNI_FUNCTION_PROLOGUE
• Option to generate ObjC protocols
• Option to generate function prologue
• Option to omit objc helper methods
• Option to disable exception translation in ObjC
• array<> type support
• outcome<> type support
• Protobuf type support
• Local flags with @flag directive
• DataView for copy free data passing
• DateRef for copy free data passing with ownership
• Bug fixes

What do you think?

As a learning from #96

Since we plan to clean up the cli, we should have a way of making cli options and flag deprecated, so if they are used there will be a warning, and users have time to react on that situation, contacting us, explain use cases and let us find solutions together.

It would be nice if this could be a simple deprecate_option decorator for an option, so it's easy to reuse, but let's see

The shebang in the generated packed djinni file got lost during the last changes.
That means, for cmake's execute_process , the file is not executable anymore without adding a java -jar in front

Add it back

As a developer I want github to automatically attach the djinni binary to a release, when it is created.
This way users can easily find & download the latest pre-build binary from the github page.

In case of defining flags which later are translated to 32 bit unsigned enum, for the 32th enum value issue is occurred as '1<<31' is considered signed i32 operation and is overflow.
Instead, by generating with unsigned postfix 'u', i.e. '1u<<31' will work fine.

Proposal, use '1u<<shift' instead of '1 << shift'. Both C++ and ObjC support it fine

Hi,

The C Wrapper Generator still depends on python something:
https://github.com/cross-language-cpp/djinni-generator/blob/main/src/main/scala/djinni/CWrapperGenerator.scala

Can it be removed?

Parts:

Recently, our team ran into a Djinni issue that started with bumping the target SDK version of our Android library to 30. On ARM devices, this triggered crashes which we could successfully backtrace to Android's new pointer tagging functionality: https://source.android.com/devices/tech/debug/tagged-pointers

Disclaimer: We're not yet using this version of Djinni, but are using an older fork of the original project.

Do you know if pointer tagging is already working on this project, or do you plan to support it in future versions?

The current generator implementation has no tests. There is just global integration-testing, that involves building a working program with the generated gluecode.
Once the generator is extracted from the uber-repo, it will need its own testing, to allow at least some QA.
I want to write some basic integration-tests, that make sure that input A produced the expected output B. This should be enough as a starting point.

When the generator is run like this:

djinni --objc-out ... --cpp-out ... --objc-swift-bridging-header bridging-header --list-out-files list-out-files.txt

The file list-out-files.txt does not include the generated bridging header bridging-header.h.

As a user I'd like the bridging header to be part of the generated files list because I rely on the list as input for the PUBLIC_HEADER CMake property. For that I take the content of list-out-files.txt, parse it to a CMake list and FILTER it for objc-headers. In the current state I have to manually add bridging-header.h to this list afterwards.

Is there some interest in adding Rust support ? I have some part of my C++ code bases that I'd like to migrate to Rust and I think that Djinni and its interfaces would make a good and easy target for interop. What do you think ?

Just some random idea, I'm not an expert in Djinni and even less in Rust but maybe first adding a C generator that creates an FFI api. Then use that code to present the C++ implementation to the equivalent Rust interface and vice-versa. There is the problem of somehow respecting the std::shared_ptr<> lifetimes and probably a tons of other things I did not think about but I just want to spark a discussion and check for interest in this feature.

Thanks for maintaining this nice project!

With the implementation of #28 , that puts the default values of various id styles into the command line help output, we created some redundancy in definition of default values.
Now the default is described in the help text (Main.scala) , but defined in generator.scala

It would be nice to get rid of this redundancy and have only 1 location in the code that defined the default style.
This should very likely be in Main.scala, where the command line arguments are created and the default is set as an extra arguement

When a djinni interface is defined with comments:

# comment
Foo = interface +c +j +o +s {
  bar();
}

The comments are not copied to the generated C++/CLI class.

As a user I'd like the comments to be copied to the C++/CLI class like it happens with all other languages as well. That way I can generate a documentation of the generated C++/CLI code that documents the interfaces with the comments from the djinni IDL file.

If the C wrapper generation is enabled when trying to consume a record defined in another module the generator will run into an exception

Exception in thread "main" scala.NotImplementedError: an implementation is missing
	at djinni.CWrapperMarshal.references(CWrapperMarshal.scala:91)
	at djinni.CWrapperGenerator$CRefs.collect(CWrapperGenerator.scala:1097)
	at djinni.CWrapperGenerator$CRefs.collect(CWrapperGenerator.scala:1070)
	at djinni.CWrapperGenerator$CRefs.collect(CWrapperGenerator.scala:1066)
	at djinni.CWrapperGenerator.$anonfun$generateInterface$3(CWrapperGenerator.scala:2121)
	at djinni.CWrapperGenerator.$anonfun$generateInterface$3$adapted(CWrapperGenerator.scala:2121)
	at scala.Option.foreach(Option.scala:437)
	at djinni.CWrapperGenerator.$anonfun$generateInterface$1(CWrapperGenerator.scala:2121)
	at djinni.CWrapperGenerator.$anonfun$generateInterface$1$adapted(CWrapperGenerator.scala:2119)
	at scala.collection.immutable.List.map(List.scala:250)
	at scala.collection.immutable.List.map(List.scala:79)
	at djinni.CWrapperGenerator.generateInterface(CWrapperGenerator.scala:2119)
	at djinni.Generator.$anonfun$generate$2(generator.scala:630)
	at djinni.Generator.$anonfun$generate$2$adapted(generator.scala:623)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at djinni.Generator.generate(generator.scala:623)
	at djinni.generatorTools.package$.generate(generator.scala:396)
	at djinni.Main$.main(Main.scala:947)
	at djinni.Main.main(Main.scala)

Looking into the source for CWrapperMarshal.scala it looks like this feature is not implemented.
Looking at the doc Modularization and Library Support it clearly states that it is not supported for Python yet, though it was not obvious just from the docs that this meant it was not supported in the C Wrapper as well.

When can we expect this to be implemented? Are there challenges or obstacles preventing this from being implemented?

As a user, I want that my enums, records and other entities are generated as specified in the djinni idl file, and not be converted into uppercase letters.

Origin:
cross-language-cpp/djinni-intellij-plugin#18

There are at least 2 emacs djinni.el modes available

• https://github.com/danielmartin/djinni-mode
• https://github.com/finalpatch/djinni-mode/

differences as described by @finalpatch

Other than minor differences in syntax highlighting:

His mode provides a shortcut to run the compilation (by invoking .run_djinni.sh under the same dir by default) and highlight errors (no sure how well that works).  my mode doesn't have this.

His indentation is more simplistic (increase indent on { and decrease on }) and is a lot slower (takes several seconds to re-indent a large djinni file).  my mode has more sophisticated indentation (eg.  can indent parameters on different lines) and works much faster.

My mode provides Imenu support for quickly jump to interface and records

As a mac user I think it would be super cool if I could just install the djinni-generator from homebrew.

Tests depend on what a test run generated into the it/resources/result folder.

But it is possible that there are files from previous runs, so tests on local machine require human intervention before running tests.
It would be better to invalidate or remove all already existing files before a test is running

scopt 3.7.x is only compatible with scala <= 2.13

In order to potentially be able to update to scala 3 somewhere in the future, we'll have to update scopt to the lastest major version in the first place. This involves some refactoring of the CLI parsing.

While doing so, it may be interesting to discover if there is a way to deprecate cli options in the new scopt version. (#97 )

Hi,

This is the list of Snapchat features and fixes:

1. Added move assigment operator to GlobalRef in all djinni_support.hpp files
2. Made JniClassInitializer constructor public to allow arbitrary additional initialization in JNI_OnLoad()
3. Speed up string passing between Java and C++ (about 5-10x faster depending on string size)
4. Eliminate CppProxy finalizers (better stability)
5. Injecting code with DJINNI_FUNCTION_PROLOGUE
6. Option to generate ObjC protocols
7. Option to generate function prologue
8. Option to omit objc helper methods
9. Option to disable exception translation in ObjC
10. array<> type support
11. outcome<> type support
12. Protobuf type support
13. Local flags with @Flag directive
14. DataView for copy free data passing
15. DateRef for copy free data passing with ownership
16. Generating string names for C++ enums

The question is:

What of this we can bring to "djinni-generator"?

Thanks.

The documentation & --help command should document the default values for the text-transformation ident styles.

This make it easier to understand what happens during target language code generation.

Djinni generator currently fails to parse a YAML file unless all of the supported languages are specified; there's an unhandled exception java.util.NoSuchElementException: key not found: <key> for any missing field (as can be seen in this run). This issue becomes more relevant as more languages get supported by the generator.

As a user I want to get a quick overview in the Readme on how the CLI works & what the djinni-IDL is.

To be defined: Should the in-detail djinni-IDL documentation be in the main Readme like in the old repo, or should it be in a separate document, e.g. in a subfolder /doc?

@prsolucoes has mentioned this fork of djinni:
https://github.com/hiennguyenle/finn

A a first glance it seems very active & miles ahead in terms of features.

As a maintainer I want to evaluate the improvements. To list some awesome features that have been added to the generator:

• Deprecated Directive
• Extend Record
• a swift generator

Given that the advancements match with our idea of djinni, I'd like to get in contact with the maintainer if . Maybe we can collaborate in some way.

The documentation still says:

Optionals (optional). This is std::experimental::optional in C++11

https://djinni.xlcpp.dev/djinni-generator/idl/#data-types

C++11 is not supported any more & the default is std::optional now. The documentation needs to be updated.

For the upcoming release we need docs for

• #14 , the applied changes in #22
• #20 and #21
• and better docu for #11 , so this issue can be seen as done (for now)

My use case is that I'm trying to pass a callback as a an argument of a method within my interface, something like:

InitializeCallback = function {
    state: MyState;
    error: MyError;
}

my_interface = interface +c {
    initialize(callback: InitializeCallback);
}

If not, is there a plan to support it?

See best practices document from Apple here: https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/ProgrammingWithObjectiveC/WorkingwithProtocols/WorkingwithProtocols.html

As an example not conforming to NSObject protocol results in build errors if we attempt to get the class from the protocol e.g.:

for (id thing in things) {
NSString *className = NSStringFromClass([thing class]); // build error
...
}

It seems the python generator does not make use of the --cpp-include-prefix when writeing including files in the generated C wrapper, but only includes the base file name.
This breaks builds where public generated cpp files are expected to be found based on a base include directory.

I understand the approach according to documentation is to:

1. define you djinni file
2. generate hpp/objc/java
3. add cpp implementation

But for my use case, I'd love to be able to:

1. given an existing C++ .h file
2. generate objc/java glue code

Is what I'm explaining currently possible?

Using non-null pointer types with the --cpp-nn-header argument together with modularization does not work properly.

Here’s a toy example:
a.djinni

a = interface +c {
    const test();
}
b = interface +c {
    const test(a: a): a;
}

c.djinni

@extern "a.yaml"
c = interface +c {
    const test(a: a): a;
}

run-djinni.sh

#! /usr/bin/env bash
set -e
module="$1"
base_dir=$(cd "`dirname "0"`" && pwd)
cpp_header_out="$base_dir/cpp/include/abc/$module"
cpp_src_out="$base_dir/cpp/src/$module"
namespace="abc"
djinni_file="$module.djinni"
djinni \
  --cpp-out $cpp_src_out \
  --cpp-header-out $cpp_header_out \
  --cpp-namespace $namespace \
  --cpp-nn-header "\"nn.hpp\"" \
  --cpp-nn-type "dropbox::oxygen::nn_shared_ptr" \
  --cpp-nn-check-expression "NN_CHECK_ASSERT" \
  --yaml-out . \
  --yaml-out-file ${module}.yaml \
  --idl $djinni_file

Executing run-djinni.sh for both modules a and c generates the three headers a.hpp, b.hpp and c.hpp:

./run-djinni.sh a
./run-djinni.sh c

cpp/include/abc/a/a.hpp

// AUTOGENERATED FILE - DO NOT MODIFY!
// This file was generated by Djinni from a.djinni
#pragma once
namespace abc {
class A {
public:
    virtual ~A() {}
    virtual void test() const = 0;
};
}  // namespace abc

cpp/include/abc/a/b.hpp

// AUTOGENERATED FILE - DO NOT MODIFY!
// This file was generated by Djinni from a.djinni
#pragma once
#include "nn.hpp"
#include <memory>
namespace abc {
class A;
class B {
public:
    virtual ~B() {}
    virtual dropbox::oxygen::nn_shared_ptr<A> test(const dropbox::oxygen::nn_shared_ptr<A> & a) const = 0;
};
}  // namespace abc

cpp/include/abc/c/c.hpp

// AUTOGENERATED FILE - DO NOT MODIFY!
// This file was generated by Djinni from c.djinni
#pragma once
#include "a.hpp"
#include <memory>
namespace abc {
class C {
public:
    virtual ~C() {}
    virtual std::shared_ptr<::abc::A> test(const std::shared_ptr<::abc::A> & a) const = 0;
};
}  // namespace abc

I think C::test should take and return a shared_ptr<A> rather than a nn_shared_ptr<A> just like B::test does.

The use of finalize is a big can of worms (see StackOverflow Why is the finalize() method deprecated in Java 9?), but it would be nice to at least have a djinni option to have the generated Java classes annotate the finalize generated methods with @SuppressWarnings("deprecation") so I don't get Java compile warnings.

I was very pleased to discover that Djinni generated Javadoc / Doxygen friendly documentation from the .djinni file comments. In fact, this was a big "selling point" in favor of using Djinni.

For me, [scoop]https://scoop.sh is the best package manager on Windows

If we had the djinni generator there and the support lib on vcpkg, this would greatly simplify the installation for people developing on Windows

Hi,

My suggestion is to use a scala format tool, like this:

• https://scalameta.org/scalafmt/
• https://github.com/scalameta/scalafmt

It will make the code more clear and easy to do maintenance.

Thanks.

In #43 a big new feature is introduced: Python support for djinni.

Because djinni has been split into components, we do now face a problem while introducing the big new feature that includes big changes in both generator & support-lib:

The changes in the generator need to be released, otherwise it cannot be pulled as dependency in the support-lib to execute the unit tests. Once the generator is released, the unit-tests in the support-lib repository can test if the generated code compiles & works.
If theses test fails because of an error in the generator implementation, a hotfix needs to be made in the generator and the fix has to be released again.

From my point of view this should not be a problem in general, as all things should be tested locally before being pushed to the generator/support-lib repository anyways.The tests in github action should just be there to verify that everything is fine.

But in reality things go wrong, and situations like this may get nasty, if multiple patches need to be made in the generator for some unexpected reason. As a maintainer I think we should discuss options on how to reduce this hard dependency when introducing big features that require changes in both components.

I (@richey-v) posted this question in the Slack forum:

has anyone tried running their Djinni-generated code with Google sanitizers or something similar? I am just now doing that and getting failures that include djinni:: calls in the stack trace... the lines above the djinni:: calls are from java-11-openjdk-amd64 so I'm not sure that the failure isn't in the JVM... I also see leaks that do not include djinni:: calls, so I would think that would point back to the JVM.

This Issue is a recommendation to add run-time QA instrumentation, such as the Google sanitizers mentioned above, to the existing QA processes.

I may be able to post a small example and more information at some point... please ask me for those

This PR #118 updated the genreated @protocol for

callback = interface +o { 
   ...
}

from this

@protocol callback {
   ....
}

to this

@protocol callback <NSObject> {
   ....
}

When using djinni to integrate with objc its not much of an issue. But, when the target consumer is Swift, it adds an additional requirement that the swift objects be NSObjects, which limits what swift objects can implement the protocol.

Previously we were using generator 1.1.0 to generate an interface consumed exclusively by swift code. Once we updated to 1.3.0, we had to go change the derivation of every swift object implementing an interface from djinni to derive from NSObject. In some cases we had to rework the code as the class originally implementing the interface was derived from an object that we don't control the source for, so we had to switch to a light object that implemented the interface an delegated to the actual object that had the implementation.

I would propose that there is a way to disable the requirement.

Maybe its auto-disabled when --objc-swift-bridging-header is defined, as the purpose of that switch is to generate an interface importable by swift.

Or a new switch --objc-strict-protocols that when ON enables this functionality.

As a user I'd like to be able to run djinni --version to find out the currently installed version. This makes it easier to verify what version has been installed on a system.

One idea on how to achieve this is that on release creation the name of the version tag is somehow injected into the build action.

The stalebot recently implemented needs to be replaced
See probot/stale#385 for details

Hi,

We need investigate is the keyword "null" can be a correct value for optional.

Example:

foo_data = record {
     label: optional<string>;
     const empty: foo_data = {
          label = null // obviously error as null Const null does not exist. but what can I use?
          //label = “” // I can only do this
     }
}

The expected result is generate a null value depending of the language (the generator).

• In "C++" the "null" keyword need be "nullptr"
• In "C" the "null" keyword need be "NULL"
• In "Java" the "null" keyword need be "null"
• In "ObjC" the "null" keyword need be "nil"
• In "Python" the "null" keyword need be "None"

currently, generated code has, for example in case of jni bindings, this include statement

#include "djinni_support.hpp"

but I think it should be

#include <djinni/jni/djinni_support.hpp>

so it is possible to not having to add various include directives, or just use standard include if the support lib is installed into the file system / root system

alternative opinions are welcome !

As a user, I would like to be able to define where all the public interface headers of as library using djinni are

--

The output location for Objective C header can not be set like those for C++ and JNI header.

For cpp and jni header and source files, there are those options that play together

• --cpp-out
• --cpp-header-out
• --cpp-include-prefix

• --jni-out
• --jni-header-out
• --jni-include-prefix
• --jni-include-cpp-prefix

There is an option for --objc-include-prefix , but no one for --objc-header-out

Since the objective C header are similar a part of the public interface like the other headers, it would be nice being able to tell the generator where to put the objective C header files to.

Since the new update from conan 1.X -> 2.X, the recipe of djinni-generator and djinni-support-lib are not working anymore.

Are you aware about it ?

Hi,

What is the project status?

What it have more than the oficial and abandoned project?

Im using this fork "https://github.com/hiennguyenle/finn" that you can get some fixes and changes.

Thanks.

Right now one can have a list<T>, set<T> or optional<T> but nothing else.
It would be really useful to be able to define user specified generic types!

Right now the only solution is to duplicate the djinni API for each types...

e.g. we have a generic property<T> type in C++, but we have to expose it as property_i32, property_float, property_foo, property_bar... in djinni which is really annoying

Hi,

I think that the test directory has wrong name.

Today it's name is "it", but need be by default "test".

The problem is that commands like this sbt test didn't found the tests.

One solution can be add this line in build.sbt file:

scalaSource in Test := baseDirectory.value / "src/it/scala",

Example:

lazy val djinni = (project in file("."))
  .configs(IntegrationTest)
  .settings(
    name := "djinni",
    Defaults.itSettings,
    scalaSource in Test := baseDirectory.value / "src/it/scala",
    libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % "it",
    libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2",
    libraryDependencies += "org.yaml" % "snakeyaml" % "1.26",
    libraryDependencies += "com.github.scopt" %% "scopt" % "4.0.1",
    libraryDependencies += "commons-io" % "commons-io" % "2.11.0",
    assemblyOutputPath in assembly := { file("target/bin") / (assemblyJarName in assembly).value },
    assemblyJarName in assembly := s"${name.value}",
    assemblyOption in assembly := (assemblyOption in assembly).value.copy(prependShellScript = Some(defaultUniversalScript(shebang = false))),
    test in assembly := {}
  )

But when SBT find the tests, it has problem with dependencies:

sbt test
[info] welcome to sbt 1.4.7 (Oracle Corporation Java 11.0.12)
[info] loading settings for project djinni-generator-build from plugins.sbt ...
[info] loading project definition from /Users/paulo/Developer/workspaces/java/djinni-generator/project
[info] loading settings for project djinni from build.sbt ...
[info] set current project to djinni (in build file:/Users/paulo/Developer/workspaces/java/djinni-generator/)
[info] compiling 3 Scala sources to /Users/paulo/Developer/workspaces/java/djinni-generator/target/scala-2.12/test-classes ...
[error] /Users/paulo/Developer/workspaces/java/djinni-generator/src/it/scala/djinni/GeneratorIntegrationTest.scala:3:12: object scalatest is not a member of package org
[error] import org.scalatest.GivenWhenThen
[...]

What do you think in solve it and move to correct place and solve dependencies problem?

Currently --objc-type-prefix is an optional objc argument. If this is left off the generated objc code will not compile as there are type collisions.

Even if you move the c++ types into a namespace (--cpp-namespace test) the collision still exists, as the generated code does not specify an explicit scope when referencing the objc classes.

user = record {
    id: i64;
    name: string;
}

this will generate the following

// AUTOGENERATED FILE - DO NOT MODIFY!
// This file was generated by Djinni from core.djinni

#import "User.h"
#include "user.hpp"

static_assert(__has_feature(objc_arc), "Djinni requires ARC to be enabled for this file");

@class User;

namespace djinni_generated {

struct User
{
    using CppType = ::test::User;
    using ObjcType = User*;

    using Boxed = User;

    static CppType toCpp(ObjcType objc);
    static ObjcType fromCpp(const CppType& cpp);
};

}  // namespace djinni_generated

// AUTOGENERATED FILE - DO NOT MODIFY!
// This file was generated by Djinni from core.djinni

#import "User+Private.h"
#import "djinni/objc/DJIMarshal+Private.h"
#include <cassert>

namespace djinni_generated {

auto User::toCpp(ObjcType obj) -> CppType
{
    assert(obj);
    return {::djinni::I64::toCpp(obj.id),
            ::djinni::String::toCpp(obj.name)};
}

auto User::fromCpp(const CppType& cpp) -> ObjcType
{
    return [[User alloc] initWithId:(::djinni::I64::fromCpp(cpp.id))
                                   name:(::djinni::String::fromCpp(cpp.name))];
}

}  // namespace djinni_generated

As you can see using ObjcType = User*; references User in the current scope (which is the struct), the generated code should be using ObjcType = ::User*; to ensure that it references the objc class (which are always in the global scope)
The same issue occurs with using Boxed = User; and the 'fromCpp' implementation.

Additionally in the generated implementation file the 'fromCpp' references the Objc class directly instead of using the type alias (ObjcType)

The generator should be updated to
a) reference objc types via the global scope
b) ::fromCpp should use the type alias (ObjcType) instead of the type directly
c) require at least one of --objc-type-prefix --cpp-namespace to ensure the generated code is compilable.

Additional suggestion:
use a decorator suffix for the generated converter classes (UserType, UserConverter) to further disambiguate the name collisions.

The addition of --objc-header-out in Release v0.3.0 is great! I would be helped greatly if there was a way to split out the placement of the private Objective-C headers (the +Private.h ones).

The motivation for this is to avoid exposing internal (private) symbols. As-is, when creating a Framework with the Djinni-generated headers as the public headers, I have to do a lot of shenanigans to filter the Objective-C headers and to create multiple build targets. I believe that adding an additional option (such as the proposed --objc-private-header-out from the title of this issue) or some other mechanism would be very helpful.

the generated C++ code for JNI contains stray ";" characters in places where the C++ syntax does not allow them.

the fix is trivial, we will upload a pull request to fix this in a minute