ricosjp / ruststep Goto Github PK

STEP files are sometimes very large, and may take a long time and memory for loading them. Micro and integrated benchmarks should be easily accessible to detect/evaluate speed down/up of load process.

criterion.rs will be a good way.

Sometimes STEP file is too large (>1GB) to expand it on memory. Since it is ASCII file, the size of files almost x10 larger than what we actually need. See ARCHITECTURE.md for detail.

ENTITY
  (´・ω・｀)  -- invalid syntax
ENDENTITY

EXPRESS language has END* tokens, which makes recovery from invalid syntax easily.

STEP Record A((1.0, 2.0)) should be parsed into a Rust struct A, but this has some imcompletion issue.

In current master e463fef it checks based on std::any::type_name::<T>. However, I've found in #22 that this cannot works if we use #[serde(name = "A")] attributes. We have to consider another way, and current checking system will be dropped in #22.

As far as I read ISO-10303-11, user defined identifiers should be snake_case, but some identifier copied into schema/ does contains non snake_case identifier like degree_Celsius

Objective

Split rust code generation phases into

• Generate abstract Rust code with proc-macro
• Generate concrete Rust code by proc-macro

This is motivated by #83, which requires us to generate original impl serde::Deserialize for *Holder due to *Any type handling because serde_derive::Deserialize proc-macro does not match our requirements. We have to generate large lines of codes, but most of them are based on a few information provided by esprc. The first step of codegen is responsible for passing necessary information from espr to proc-macro, and the second is responsible for generate actual code.

TODO

• Introduce ruststep-derive #75
• Generate Rust code using ruststep-derive by esprc #79
• Drop ruststep::ap000 #93

Split from #12

cargo has -Z timings feature in nightly. It shows how long compile spent times like following:

   Completed espr v0.1.0 lib (test) in 6.1s
   Completed ruststep v0.1.0 lib (test) in 10.1s
   Completed ruststep v0.1.0 in 18.7s
   Completed espr-derive v0.1.0 lib (test) in 0.5s
   Completed ruststep-derive v0.1.0 test "try_build" (test) in 1.4s
   Completed ruststep-derive v0.1.0 lib (test) in 2.3s
   Completed ruststep v0.1.0 test "abc_dataset" (test) in 3.4s
   Completed ruststep v0.1.0 test "enum_deserialize" (test) in 3.5s
   Completed ruststep v0.1.0 test "simple" (test) in 3.5s
   Completed ruststep v0.1.0 test "list" (test) in 3.6s
   Completed ruststep v0.1.0 test "any_deserialize" (test) in 3.7s

https://doc.rust-lang.org/cargo/reference/unstable.html#timings

Split from #1

I prefer to update CHANGELOG manually, but I always forget to update it in each Pull Request. It should be checked in each PR if the CHANGELOG.md has been updated.

Translate from Japanese slide (internal) and add to README:

• STEP は CAD だけの為のものではない
  • Protocol Buffers のように汎用なスキーマ＋シリアライズフォーマット
  • 後継の ISO （例えば ISO 13399 “Cutting tool data representation and exchange”）も STEP で（EXPRESS schema で）定義されている
    • computer-aided design (CAD)
    • computer-aided manufacturing (CAM)
    • computer-aided engineering (CAE)
    • tool management software
    • product data management (PDM/EDM)
    • manufacturing resource planning (MRP) or enterprise resource planning (ERP)
    • and other computer-aided technologies (CAx) and systems.
• スキーマから様々なものを生成する事になる
  • SQL や他のデータベース向けのテーブル定義・Object-Record Mapper
  • 通信用 (on-wire) メモリ表現
    • ASCII / XML にシリアライズするのは重いので
    • 将来的には Apache Arrow, Parquet 形式をサポートするのがいいと思う
  • これらの生成に EXPRESS Compiler が必要になる
    • Protocol Buffers が protoc plugin で gRPC を生成しているような拡張性を実現したい

Tokenize String from STEP will failed.

I rewrote the test case so that this would cause it to fail.
https://github.com/ricosjp/ruststep/blob/53e9baeb987955a4955f1fcbcba81e107d4f0d7f/ruststep/tests/simple.rs

The third milestone for #148.

The link http://www.steptools.com/stds/archive/ at https://github.com/ricosjp/ruststep/tree/master/schemas is already broken.

Attempting to parse AP242 or AP203-long using esprc leads to a stack overflow error on my Windows machine. We should probably switch from stack based to heap based allocations for the parser.

The second milestone for #148.

Before creating tables automatically by espr as described in ARCHITECTURE.md, we must try to create them by hand to determine its specification.

TODO

• Create tables manually #15
• Convert Exchange AST to manually generated tables #22
  • Basic serde integration (for HEADER section) #18
  • Nested record support #21
  • PlaceHolder for reference e.g. A((1.0, #11)) #20
• Documents for tables submodule #24
• Generate tables definitions by espr
• Add about serde in ARCHITECTURE.md

Should be

enum PetAny {
  Pet(Box<Pet>),
  Cat(Box<Cat>),
  Rabbit(Box<Rabbit>),
  Dog(Box<Dog>)
}
struct Pet {
  name: String,
}
struct Cat {
  pet: Pet,
  // other field if exists
}

Current status

enum PetAny {
  Pet(Box<Pet>),
  Cat(Box<Cat>),
  Rabbit(Box<Rabbit>),
  Dog(Box<Dog>)
}
struct Pet {
  name: String,
}
struct Cat {
  // do not have pet
  // other field if exists
}

I pushed any_in_struct for add pet: Pet at Cat. However, this fix is a bit ad-hoc in the is_supertype fix, and the build of ap201.rs did not pass due to a problem with the derive macro in the deserializer.

This issue is the first milestone of #130 .

Create tuple struct for each TYPE declaration.

Start logging from 0.1.0 release.

Rules

• Based on Keep a Changelog
• Manually edit CHANGELOG.md file
• Each line of CHANGELOG.md should corresponds to a Pull Request, not a commit.
  • Allow two or more lines refer same Pull Request

TODO

• Create CHANGELOG.md file #154
• Add CI action to check CHANGELOG.md is updated in Pull Request #155

JSON schema would be more popular than protocol buffers

STEP file input (load) has been specified in ARCHITECTURE.md and implemented partially in ruststep::parser with later process by #12 . But, its output spec is not determined.

Duplication Elimination

For example, let us consider the entities discussed in ARCHITECTURE.md:

ENTITY a;
  x: INTEGER;
  y: INTEGER;
END_ENTITY;

ENTITY b;
  z: INTEGER;
  w: a;
END_ENTITY;

If a user has two instances of b

let b1 = B(1, A(1, 2));
let b2 = B(2, A(1, 2));

The problem here is how to eliminate the duplicated instance A(1, 2):

DATA;
#1 = A(1, 2);
#2 = B(1, @1);
#3 = B(2, @1);
ENDSEC;

We can also emit duplicated instances without elimination process, and it would be somewhat easy to implement:

DATA;
#1 = A(1, 2);
#2 = B(1, @1);
#3 = A(1, 2);
#4 = B(2, @3);
ENDSEC;

Rust API

Rust struct like

struct A {
  x: f64,
  y: f64,
}

struct B {
  z: f64,
  a: A,
}

into a STEP Record B((1.0, A((2.0, 3.0)) will be done by serde. The data mapping from Rust struct to serde data model has introduced in #18, #20, #21, and the responsibility of this issue is to implement serde::Serialize compatible for these existing implementations.

TODO

• Implement STEP output using serde::Serialize
  • Serialize a single struct into a record #30
• Duplication elimination
• Update ARCHITECTURE.md

Currently, they are parsed by espr, but code generation is not implemented.

esprc output should be work only with ruststep

From #130 and #144

std::mem::size_of<Tables>() will be size_of<HashMap<usize, XxxHolder>() * number_of_tables, but number_of_tables will be large for large EXPRESS schemas. It should be allocated on heap instead of stack.

Objective

Represent EXPRESS's super/sub types as Rust's enum instead of trait object Box<dyn *Any>. This is mainly due to serde does not supports trait object.

• Definition #82, #84
• Codegen #80
• Serialize, Deserialize #70
• Update ARCHITECTURE.md #112

Subtype-Supertype Graph

Revival of #40. Since we have to gather all subtypes to define supertype enum, SubSuperGraph will be constructed after Namespace creation and before Legalize. Though the ENTITY in EXPRESS schema has SUPERTYPE OF declaration, it does not list up all subtypes correctly #80 (comment)

• Create SubSuperGraph #86, use for ABSTRACT SUPERTYPE case
• Use sub-super graph always #88

From #133 (comment)

https://github.com/ricosjp/ruststep/runs/2263884269?check_suite_focus=true

Implement a macro include_express, which compiles the express file with esprc and embeds the result in the code.

This project is developed at private GitLab, but plans to move GitHub entirely.

TODO

• Licenses
• Mirror GitLab to GitHub (as git repository)
• Setup GitHub Action #3
• Import issues
• Setup GitHub Pages to deploy master document
  • https://github.com/marketplace/actions/github-pages-action will be helpful.
• CONTRIBUTION.md

IR should not include functions for Rust code generation. It should be separated.

This is the third milestone of #130.

• Merge the branches of previous milestone.
• Add implementation for tuple to TableInit.
• Add HashMap field to macro generated Tables.

While parsing one of the APs I noticed that a line following empty tail remarks is ignored by the parser.

An example would be:

--
A

In this case A would be treated as a remark. I'm currently working on a fix for this.

EXPRESS has "remark tag"

ENTITY ent;
attr: INTEGER;
END_ENTITY;
(*"ent.attr" The attr attribute ... *)

(from ISO-10303-11(2004) p.13) It will helpful to convert it into Rust document

Since the all parsers in espr are public, the parse test should be out of submodules. Current tests are not in-homogeneous, i.e. some test "parser can parse it", and others test "parser parses it correctly". The correctness of parsed values should be tested after semantic analysis phase, and they should only test "parse can parse it".

This is the second milestone of #130.

More specifically, if one replaces the relevant file in the master branch with the following, the build will not pass.

https://github.com/ricosjp/ruststep/blob/vec-vec-entity/ruststep/tests/list.rs

HEADER;
  FILE_NAME('name');
ENDSEC;
DATA;
  #1 = A(1.0, 2.0);
ENDSEC;

should be mapped into serde data model like

{
  "HEADER": {
    "FILENAME": ["name"],
  },
  "DATA": [
    { "ENTITY_ID": 1, "VALUE": { "NAME": "A", "PARAMETERS": [1.0, 2.0] } }
  ]
}

esprc fails to parse the following EXPRESS code.

SCHEMA test_schemal;

TYPE generic_hoge = STRING;
END_TYPE;

TYPE decl_hoge = generic_hoge;
END_TYPE; -- success

ENTITY have_decl_hoge;
  a: decl_hoge;
END_ENTITY; -- success

ENTITY have_hoge;
  a: generic_hoge;
END_ENTITY; -- fail

END_SCHEMA;

This issue must be fixed for esprc parsing ap214.

https://easycla.lfx.linuxfoundation.org/

CLA of RICOS Co. Ltd. is here: https://github.com/ricosjp/CLA/blob/master/icla.md

In the current implementation, the implementation by TYPE declaration is translated into a type alias. However, the intention is to treat the TYPE declaration in the express language as a separate type rather than a simple type alias. In particular, constraints can be given in the where declaration. Therefore, the TYPE declaration should normally be translated as a tuple structure in Rust code.

Current nom::error::Error only shows the last &str which is not matched.

Links

• nom::error::VerboseError
• nom-greedyerror
  • Blog (in Japanese)

I tried to use esprc transpiler for IFC schema (ver 2x3) https://standards.buildingsmart.org/IFC/RELEASE/IFC2x3/FINAL/EXPRESS/IFC2X3_Final.exp as input, using the below terminal command:

./target/release/esprc ./schemas/ifc/IFC2x3.exp | rustfmt > ./ruststep/src/ifc2x3.rs

It results in following error:

`Syntax Error occurred while parsing following line [Nom(Tag)]:

FUNCTION IfcCorrectObjectAssignment
(Constraint: IfcObjectTypeEnum; Objects : SET[1:?] OF IfcObjectDefinition)
: LOGICAL;

LOCAL
Count : INTEGER := 0;
END_LOCAL;

  IF NOT(EXISTS(Constraint)) THEN 
    RETURN(TRUE);

Syntax Error occurred while parsing following line [Nom(Many1)]:

SCHEMA IFC2X3;

TYPE IfcAbsorbedDoseMeasure = REAL;
END_TYPE;

TYPE IfcAccelerationMeasure = REAL;
END_TYPE;

TYPE IfcAmountOfSubstanceMeasure = REAL;
END_TYPE;

thread 'main' panicked at 'Syntax Error', espr/src/bin/esprc.rs:35:13`

If I increase the number of error lines in the command (like 10000), I see that it basically dumps the entire content of the schema file as syntax error ([Nom(Tag)] and [Nom(Many1)].

The code is too complex for me to debug and understand. Please advise.
Does esprc support IFC schemas? Is it planned for later releases?

ISO 10303-11 9.2.3 Subtypes and supertypes

A subtype/supertype graph is a multiply rooted directed acyclic graph where the nodes represent the entity types, and the edges represent the subtype/supertype relationships.

Should be generate a graph after Namespace is created, and before Legalize