Problem
Currently, we can specify 'attribute' and 'command' in ZAP cluster xml definition file, but not 'event'

Proposed Solution
Need to support to specify event defined in CHIP spec in ZAP xml file.

The same cluster is appearing twice in the cluster UI. We probably need to check the UI queries for package id

Steps to reproduce:

1. Ensure your Matter repo includes project-chip/connectedhomeip#8839 (which fixes some similar problems in Matter).
2. Open src/app/zap-templates/common/ClustersHelper.js in the Matter repo.
3. In the Clusters.init function, right after this.ready.running = true; add throw new Error("FAIL");
4. Run ./scripts/tools/zap/generate.py examples/all-clusters-app/all-clusters-common/all-clusters-app.zap in the Matter repo

Observe that the "FAIL" exception gets logged, but generation claims to complete successfully....

If I add some asynchronicity (e.g. awaiting on a setTimeout) to the guts of Markers.prototype.resolve in the handlebars code that seems to cause this exception to fail generation, for reasons that are totally unclear to me.

@tecimovic

"Shrink" the .zap files, essentially eliminating duplication. Go back to the model of "only record deviations, not the whole duplication". So if ZAP file is using the default Basic cluster, for example, it will NOT record duplications of all attributes that are mandatory anyway. It will only record the data BEYOND what the spec already lists.

Problem
Thermostat.xml file contains the min and max values for
Temperature setpoints and limits but they do not appear in the generated .zap file
These were identified in #8878

Also the zap tool displays warnings about signed ints but does generate the correct code.

Matter spec seems to have removed the ranges

Proposed Solution
Follow up on these

Add a global attribute per-cluster choice.
Currently we only have per-app or global choice.

Problem

ZAP can be used directly from source via various scripts that do NOT require DISPLAY, because they bypass 'electron' and just use plain 'node' engine to perform generation and such.

However, once you use a binary package of zap (.deb, or .rpm) as downloaded from here, you loose this ability.

Binary package ALWAYS uses Electron, so the moment you start zap in a headless environment, without DISPLAY set, you get a core dump and nothing happens.

Workaround is easy: just wrap the execution using xvfb-run.
However that is a bit sad and should be fixed.

Solution

Make the binary build produce 2 executables:

• zap, which is what we know now: the full zap tool including ui and all.
• zap-headless, which is the generation and conversion and all the hadless parts of zap, and NO UI bundled in, so it does not require electron.

A user will then be able to run zap for EVERYTHING, or zap-headless just for headless purpose.

ZCL has a concept of "feature bits" in the spec, which are the user-conveninent shortcuts to enable/disable sets of attributes or other entities tagged with that feature bit.

Neither ZAP tool, nor the XML does not support that currently. We need to implement it.

ZAP contains "IDE integration" feature.

What it does is the following:
a.) ZAP may run standalone, without any IDE integration.
or
b.) ZAP may run in "IDE integration" mode.

When running in IDE integration mode ZAP is aware of an IDE it lives in so it can possibly communicate with other parts of the system. Canonical example:
if ZAP runs inside Simplicity Studio, then when you enable a cluster inside ZAP, zap informs Simplicity Studio to turn on the C component that actually IMPLEMENTS that cluster.

This functionality needs to be abstracted, so that this works also in other IDEs, most notably VSCode.

So the goal of this issue is to provide a clean abstraction point, where different IDE integration implementations can plug in cleanly.

This already builds upon existing IDE integration with Simplicity Studio, however the point is to clean this up, after some code review revealed that the integration point is currently not cleanly abstracted.

Our UI quasar/Vue implementation is holding way too much state with way too much complexity, so maintaining the UI has pretty significant overhead. We need to clean this up.

Zap is doing the following:
For attributes, we are matching DEFINE column in ATTRIBUTE table.
For commands, we are matching NAME column in the COMMAND table.
The camel-case vs snakecase inside XML files reflects that. As it is right now, if you use camelcase for attribute it most likely won't even work.
We need to fix this, so that you will be allowed to put either name or define OR the actual code of the attribute or command.

This is all legacy ugliness that needs to be cleaned up.

When I edit OTA Server Cluster in all-clusters-app.zap, other Clusters are also changed.

Steps to reproduce:

1. checkout connectedhomeip master branch
2. run ./scripts/tools/zap/configure.sh
3. open all-clusters-app.zap, remove OTA Server Cluster from Endpoint 1, and save
4. Observe diff: Clusters other than OTA Server should have been edited (this is unintended)

There is this length= inside the XML that is implementation specific. We need to find a generic way to do this to deal with all SDKs.

Custom helpers do things like this:

const zapPath = "../../../../../third_party/zap/repo/dist/src-electron/";
const templateUtil = require(zapPath + "generator/template-util.js");
const zclHelper = require(zapPath + "generator/helper-zcl.js");
const iteratorUtil = require(zapPath + "util/iterator-util.js");

They should not. This doesn't work in a binary pre-built environment. It also doesn't work if you use zap from a different location.
So we need to create a way to include custom helpers by feeding them API on the context, instead of the helpers reaching into the source repo for it themselves.

I have enabled all attributes from the Basic Cluster, because they are required by the TE2 test spec.

This is the resulting zap file: config.zap.txt

When compiling I get the following error:

../src/gen/endpoint_config.h:271:29: error: exponent has no digits
  271 | #define ATTRIBUTE_MAX_SIZE (273ERROR : PartNumber, invalid size, char_stringERROR : ProductURL, invalid size, char_string643212)
      | 

And in endpoint_config.h.

// Total size of singleton attributes
#define ATTRIBUTE_SINGLETONS_SIZE                                                                                                  \
    (270ERROR : PartNumber, invalid size, char_stringERROR : ProductURL, invalid size, char_string64321)

// Total size of attribute storage
#define ATTRIBUTE_MAX_SIZE (273ERROR : PartNumber, invalid size, char_stringERROR : ProductURL, invalid size, char_string643212)

currently asUnderlyingZclType can do some transformation for basic type, for example, from UINT16U to uint16_t, for unknown type, it convert it to uint8,

is that possible to have similar API, that API can return unknown string or false if that input type is not known.

I used ZAP to add a few attributes to the lighting clusters, this is a snippet from the lighting.zap:

            {
              "name": "manufacturer name",
              "code": 4,
              "mfgCode": null,
              "side": "server",
              "included": 1,
              "storageOption": "RAM",
              "singleton": 0,
              "bounded": 0,
              "defaultValue": "Tridonic GmbH & Co KG",
              "reportable": 0,
              "minInterval": 0,
              "maxInterval": 65344,
              "reportableChange": 0
            },
            {
              "name": "model identifier",
              "code": 5,
              "mfgCode": null,
              "side": "server",
              "included": 1,
              "storageOption": "RAM",
              "singleton": 0,
              "bounded": 0,
              "defaultValue": "Opportune",
              "reportable": 0,
              "minInterval": 0,
              "maxInterval": 65344,
              "reportableChange": 0
            },

Then ran ./scripts/tools/zap_generate.sh ./examples/lighting-app/lighting-common/lighting-app.zap, which generated:

#define GENERATED_DEFAULTS { \
  21, 'T', 'r', 'i', 'd', 'o', 'n', 'i', 'c', ' ', 'G', 'm', 'b', 'H', ' ', '&', ' ', 'C', 'o', ' ', 'K', 'G', 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   /* Default for cluster: "Basic", attribute: "manufacturer name". side: server, little-endian */ \
  9, 'O', 'p', 'p', 'o', 'r', 't', 'u', 'n', 'e', 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   /* Default for cluster: "Basic", attribute: "model identifier". side: server, little-endian */ \
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   /* Default for cluster: "Basic", attribute: "date code". side: server, little-endian */ \
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   /* Default for cluster: "Basic", attribute: "product code". side: server, little-endian */ \
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   /* Default for cluster: "Basic", attribute: "product url". side: server, little-endian */ \
}

There seems to be a length tag in the beginning, which is not being accounted for in

#define GENERATED_ATTRIBUTES { \
  { 0xFFFD, ZAP_TYPE(INT16U), 2, 0, { (uint8_t *) 3 } }, /* Basic (server): cluster revision */  \
  { 0x0000, ZAP_TYPE(INT8U), 1, 0, { (uint8_t *) 0x08 } }, /* Basic (server): ZCL version */  \
  { 0x0001, ZAP_TYPE(INT8U), 1, ZAP_ATTRIBUTE_MASK(SINGLETON), { (uint8_t *) 0x00 } }, /* Basic (server): application version */  \
  { 0x0002, ZAP_TYPE(INT8U), 1, ZAP_ATTRIBUTE_MASK(SINGLETON), { (uint8_t *) 0x00 } }, /* Basic (server): stack version */  \
  { 0x0003, ZAP_TYPE(INT8U), 1, 0, { (uint8_t *) 0x01 } }, /* Basic (server): hardware version */  \
  { 0x0004, ZAP_TYPE(CHAR_STRING), 32, 0, { (uint8_t *) ZAP_LONG_DEFAULTS_INDEX(0) } }, /* Basic (server): manufacturer name */  \
  { 0x0005, ZAP_TYPE(CHAR_STRING), 32, 0, { (uint8_t *) ZAP_LONG_DEFAULTS_INDEX(32) } }, /* Basic (server): model identifier */  \
  { 0x0006, ZAP_TYPE(CHAR_STRING), 16, 0, { (uint8_t *) ZAP_LONG_DEFAULTS_INDEX(64) } }, /* Basic (server): date code */  \

Changing it to

  { 0x0005, ZAP_TYPE(CHAR_STRING), 32, 0, { (uint8_t *) ZAP_LONG_DEFAULTS_INDEX(32+1) } }, /* Basic (server): model identifier */  \
  { 0x0006, ZAP_TYPE(CHAR_STRING), 16, 0, { (uint8_t *) ZAP_LONG_DEFAULTS_INDEX(64+2) } }, /* Basic (server): date code */  \

allows to be able to read not only the first, but also the following attributes.

Is the length tag supposed to be there? If yes, please account for it when indexing into the array.

FYI: @jepenven-silabs @vivien-apple

<struct name="SimpleStruct">
        <cluster code="0x050F"/>
        <item name="a" type="UINT8U" optional="false"/>
        <item name="b" type="BOOLEAN" optional="false"/>
        <item name="c" define="SimpleEnum" type="ENUM8" optional="false"/>

DROP TABLE IF EXISTS "STRUCT_ITEM";
CREATE TABLE IF NOT EXISTS "STRUCT_ITEM" (
  "STRUCT_REF" integer,
  "FIELD_IDENTIFIER" integer,
  "NAME" text,
  "TYPE" text,
  "IS_ARRAY" integer,
  "MIN_LENGTH" integer,
  "MAX_LENGTH" integer,
  "IS_WRITABLE" integer,
  foreign key (STRUCT_REF) references STRUCT(STRUCT_ID)
);

looks like currently zap don't support struct item with define ?

for enum type, we would like to get "define"

then I can write {{asType label}} = static_cast<{{define}}>(v);

System:
Debian 10
node v14.16.1 and v16.1.0 (installed via nvm)

On my machine installation failed due to missing dependencies of the canvas package, the log file did not contain anything useful.

I fixed that by installing the relevant system dependencies for canvas, see instructions here: https://www.npmjs.com/package/canvas

janus@janus-latitude:~/grundfos/zap$ npm install
npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but package-lock.json was generated for lockfileVersion@2. I'll try to do my best with it!

> [email protected] postinstall /home/janus/grundfos/zap
> electron-builder install-app-deps && husky install

  • electron-builder  version=22.10.5
  • loaded configuration  file=package.json ("build" field)
  • rebuilding native dependencies  [email protected], [email protected], [email protected], [email protected] platform=linux arch=x64
  ⨯ cannot execute  cause=exit status 1
                    out=
    > [email protected] install /home/janus/grundfos/zap/node_modules/bufferutil
    > node-gyp-build
    
    
    > [email protected] install /home/janus/grundfos/zap/node_modules/canvas
    > node-pre-gyp install --fallback-to-build
    
    Failed to execute '/home/janus/.nvm/versions/node/v14.16.1/bin/node /home/janus/.nvm/versions/node/v14.16.1/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/home/janus/grundfos/zap/node_modules/canvas/build/Release/canvas.node --module_name=canvas --module_path=/home/janus/grundfos/zap/node_modules/canvas/build/Release --napi_version=7 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=electron-v12.0' (1)
    
                    errorOut=node-pre-gyp WARN Using request for node-pre-gyp https download 
    node-pre-gyp WARN Tried to download(404): https://github.com/Automattic/node-canvas/releases/download/v2.7.0/canvas-v2.7.0-electron-v12.0-linux-glibc-x64.tar.gz 
    node-pre-gyp WARN Pre-built binaries not found for [email protected] and [email protected] (electron-v12.0 ABI, glibc) (falling back to source compile with node-gyp) 
    Package pixman-1 was not found in the pkg-config search path.
    Perhaps you should add the directory containing `pixman-1.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'pixman-1' found
    gyp: Call to 'pkg-config pixman-1 --libs' returned exit status 1 while in binding.gyp. while trying to load binding.gyp
    gyp ERR! configure error 
    gyp ERR! stack Error: `gyp` failed with exit code: 1
    gyp ERR! stack     at ChildProcess.onCpExit (/home/janus/.nvm/versions/node/v14.16.1/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:351:16)
    gyp ERR! stack     at ChildProcess.emit (events.js:315:20)
    gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)
    gyp ERR! System Linux 5.10.0-5-amd64
    gyp ERR! command "/home/janus/.nvm/versions/node/v14.16.1/bin/node" "/home/janus/.nvm/versions/node/v14.16.1/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure" "--fallback-to-build" "--module=/home/janus/grundfos/zap/node_modules/canvas/build/Release/canvas.node" "--module_name=canvas" "--module_path=/home/janus/grundfos/zap/node_modules/canvas/build/Release" "--napi_version=7" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=electron-v12.0"
    gyp ERR! cwd /home/janus/grundfos/zap/node_modules/canvas
    gyp ERR! node -v v14.16.1
    gyp ERR! node-gyp -v v5.1.0
    gyp ERR! not ok 
    node-pre-gyp ERR! build error 
    node-pre-gyp ERR! stack Error: Failed to execute '/home/janus/.nvm/versions/node/v14.16.1/bin/node /home/janus/.nvm/versions/node/v14.16.1/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/home/janus/grundfos/zap/node_modules/canvas/build/Release/canvas.node --module_name=canvas --module_path=/home/janus/grundfos/zap/node_modules/canvas/build/Release --napi_version=7 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=electron-v12.0' (1)
    node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/home/janus/grundfos/zap/node_modules/canvas/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
    node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:315:20)
    node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:1048:16)
    node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)
    node-pre-gyp ERR! System Linux 5.10.0-5-amd64
    node-pre-gyp ERR! command "/home/janus/.nvm/versions/node/v14.16.1/bin/node" "/home/janus/grundfos/zap/node_modules/canvas/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
    node-pre-gyp ERR! cwd /home/janus/grundfos/zap/node_modules/canvas

```    node-pre-gyp ERR! node -v v14.16.1
    node-pre-gyp ERR! node-pre-gyp -v v0.15.0
    node-pre-gyp ERR! not ok 
    npm ERR! code ELIFECYCLE
    npm ERR! errno 1
    npm ERR! [email protected] install: `node-pre-gyp install --fallback-to-build`
    npm ERR! Exit status 1
    npm ERR! 
    npm ERR! Failed at the [email protected] install script.
    npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
    
    npm ERR! A complete log of this run can be found in:
    npm ERR!     /home/janus/.npm/_logs/2021-05-06T10_07_44_845Z-debug.log
    
                    command=/home/janus/.nvm/versions/node/v14.16.1/bin/node /home/janus/.nvm/versions/node/v14.16.1/lib/node_modules/npm/bin/npm-cli.js rebuild [email protected] [email protected] [email protected] [email protected]
                    workingDir=
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] postinstall: `electron-builder install-app-deps && husky install`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/janus/.npm/_logs/2021-05-06T10_07_44_904Z-debug.log

When setting any of the thermostat setpoint limits (int16s) a warning is always generated for value being out of range. Correct code is generated though.

ZAP is generaing incorrect default value for attribute of type="INT64U" if I put it as the first attribute member, such as :
Breadcrumb
Generated value:
/* 1644 - Breadcrumb, /
0x6F,
0x00
But if I put attribute of type="INT64U" not on the fist member, such as:
FabricId
Breadcrumb
ZAP could generate expected default value as:
/ 1652 - Breadcrumb, */
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,

Currently zcl_enums use selectAllEnums, which retrieve all enums for all clusters, for chip complicated type representation, we expect zcl_enums is able to retriever enums per cluster, which similar to functionality selectCommandsByClusterId API

The function endpoint_attribute_list should support the generation of default values for big endian systems.
The rationale is that the current generated values are only working in little endian systems.

See also:
project-chip/connectedhomeip#6175

This work covers the discussion we had with @andy31415 on 6/8/2021.

To-do items:

• Add the support for TLV-based structs:

  • extend XML shema for Silabs XML files (no need to do anything with dotdot)
  • extend SQL schema
  • fix the queries
  • provide basic helpers.
  • prevent loading of mixed case: some fields have ID some don't.
  • allow initial loading of structs without IDs, gated by zcl.json preference.
  • Call these "FIELD_ID" not "ORDINAL", or "TAG".
    NOTE: MIND THE MANUFACTURER SPECIFIC STUFF! (Read up on spec.)

• Add the support for CHIP "events" (attributes with history?)

  • understand the spec
  • extend XML shema for Silabs XML files (no need to do anything with dotdot)
  • extend SQL schema
  • fix the queries
  • provide basic helpers

• Add the support request/response metainfo (or "return value" for the requests, however you see it).

  • extend XML shema for Silabs XML files (no need to do anything with dotdot)
  • extend SQL schema
  • fix the queries
  • provide basic helpers.

• Feature map support.

Currently zcl_structs use selectAllStructsWithItems, which retrieve all structs for all clusters, for chip complicated type representation, we expect zcl_structs is able to retriever structs per cluster, which similar to functionality selectCommandsByClusterId API

Zap and the silabs XML is lacking support for complex data types. We need to add and implement that.

Minimum required list of features:

• use type 'ARRAY' to specify list type within 'attribute' or 'command', with a user defined struct being an element of the array.
• allow nested structs: a struct that contains another struct.
• allow support for the maximum length of lists and arrays, and generation of code that enforces these maximums.
• allows support for attributes of the type "struct". Provide ability to specify complex default values and ability for the UI to edit them.

Support arbitrary nesting of structs, etc.

Access-control qualities (read spec).

We have added a lot of new XML elements. We need UI for them. This issue is a placeholder for all that.

We have a feature where end-user can add "custom XML" which adds entities to the DB so that they can use them in the UI and zap files and such.

However, at this point we don't have an ability to add "package extensions" or even additional .zapt templates from the end-user.

We need to modify this feature in a following way:
1.) Toplevel file read in is no longer just a ZCL XML file, but some kind of "custom.json" file or something.
2.) This file can point to any number of the ZCL XML files.
3.) This file can point to any number of the gen-templates.json files (or just contain data directly.)
4.) Through 3, the package can also contain package extension JSON files or additional .zapt templates.

It would be nice if the code would be able to just take a .zip file as an input and disentangle all this from a single zip file.

We would like to have one API to check if struct has array, since we need to generate two version for struct with list
We have the below struct xm;

<struct name="NestedStructList">
    <cluster code="0x050F"/>
    <item name="a" type="UINT8U" optional="false"/>
    <item name="b" type="BOOLEAN" optional="false"/>
     <!--
    <item name="c" type="SimpleStruct" optional="false"/>
     -->
    <item name="d" type="ARRAY" entryType="SimpleStruct" optional="false"/>
    <item name="e" type="ARRAY" entryType="UINT32U" optional="false"/>
    <item name="f" type="ARRAY" entryType="OCTET_STRING" optional="false"/>
   </struct>

I write the below is pesudo code, assume I have a API zcl_struct_check_array_items_inside to check if there is array inside struct

{{#zcl_structs}}
{{#if ( zcl_struct_check_array_items_inside)}}
namespace Decodable{{asType label}} {
}
{{/if}}
namespace {{asType label}} {
}
{{/zcl_structs}}

looking at current zap code, I cannot find the API to check if there is list inside struct, I would like to check if there is list inside struct when looping inside {{#zcl_structs}}, then generate two version

SDK needs to provide rules and logic for validation:

• validation of XML files.
• validation of live user session data

This validation at minimum has to provide some kind of API that returns "severity + message" validation tokens back to the requester and shows them in UI or in the console when being triggered.

Use case is as follows:
1.) You create a .zap file for an application using some set of XML files.
2.) You then at some later point, open the .zap file against a DIFFERENT, usually NEWER set of static ZCL XML files.
3.) At this point zap need to do the following:
a.) Detect that this is the case.
b.) Walk the user through some kind of "upgrade" process upgrading things that need upgrading.
c.) There are things that can happen automatically (new mandatory attributes, cluster versions.)
d.) There are things that have to happen with user assistance, so they need to get some warnings and notifications, etc.
e.) In either case, the final step of the "upgrade" process is telling user what has changed and allowing him to say: "Ooops, sorry, no! Just keep the file as it was! It was a mistake all long, I just loaded wrong XML files...." so user can back-out of the whole upgrade process.

ZAP does not generate code for a command if it has the same name of an already generated command even if they are part of different clusters.

for example, the command resetcounts exist for clusters:

ethernet-network-diagnostics-cluster
thread-network-diagnostics-cluster
wifi-network-diagnostics-cluster

The commands only get generated for the first cluster.

The way metafiles load is like this:
1.) Read top file, read CRC, compare to loaded file. If different load it.
2.) Read individual file, read CRC, compare to individual loaded CRC in DB, if different load it.

This runs into problems of structural integrity because XML files are dependent on each other.

Solution is to load entire pack of metafiles, starting as zcl.json as an "atomic" package

It should work like this:
1.) Read zcl.json, read all the XML files that it lists.
2.) Collect CRC of all of these.
3.) Compare ALL of these CRCs against the CRCs in the database.
4.) If AT LEAST ONE fails to match, drop entire pack from the DB and reload ALL the files.,

This way, if ANY file changes, ALL files will get reloaded.

Note: this goes for BOTH the gen-template.json as well as for zcl.json.

If I try to use asUnderlyingZclType type under zcl_struct_items
it prints TYPE WARNING: array array defaults to uint8_t, which looks not right.

Device types don't have a unique ID anywhere in the XML files, so they are ambiguous across multiple-spec versions of XML file.
It's a real problem when loading two separate specs of the XML file that contain same devices.

There is no single identifier to disambiguate.

Hi, This is Jeonghwan Kim.
I tried to install ZAP refer to FAQ then succeed to install ZAP on my ubuntu machine.

after run zap, I meet the below issue?
Would you help me?

` ✘ ⚙  ~/work/zap   master ±  npm run zap

[email protected] zap
node src-script/zap-start.js --logToStdout --gen ./test/gen-template/zigbee/gen-templates.json

🚀 Executing: git log -1 --format={"hash": "%H","timestamp": %ct}
🔍 Git commit: 742ebb7 from Tue Mar 02 2021 22:10:54 GMT+0900 (대한민국 표준시)
🔍 Current hash: jJRPde7u50RsCrLfJaRztyBEFmk=
🔍 Previous hash: jJRPde7u50RsCrLfJaRztyBEFmk=
👍 There were no changes to front-end code, so we don't have to rebuild the SPA.
🚀 Executing: electron src-electron/main-process/electron-main.dev.js --logToStdout --gen ./test/gen-template/zigbee/gen-templates.json
⇝ (node:21240) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
The futex facility returned an unexpected error code.😎 All done: 1s, 462ms.`

#155 addressed a similar problem. After some discussion with @tecimovic we have identified the problem:

When opening the zap file in the following manner:

node ./src-script/zap-start.js --logToStdout -z $ZCL_JSON -g $APP_TEMPLATES_JSON -i "$OPT_IN_FILE"

and editing the zap file and pressing ctl+s the zap tool still presents an error:

{"level":50,"time":1622040654554,"pid":85940,"hostname":"meow","name":"zap","msg":"Unable to save project with sessionId(1)"}
{"level":50,"time":1622040654554,"pid":85940,"hostname":"meow","name":"zap","msg":{"errno":-2,"code":"ENOENT","syscall":"open","path":""}}

Currently there is a workaround for this:

In the UI use the File -> Save As... dialog to save the changes.

This was some early prototype how to use jsdoc2md for API documentation. It has by now clearly outlived its usefulness, so we need to take a look how to regen this better.

It is unclear to me if the current result is due to the tool itself being a bit feeble, or do we have to maintain our jsdoc tags better. We need to investigate.

Roughly do this:
1.) Add a CLI -loadScript <file.js> or something like that.
2.) Upon loading a .zap file, execute code in this script, using the APIs that zap will provide. Things like: enableCluster(0x000), disableCluster(0x0001) or setAttributeDefault(0x0000, 0x0123, 'true'), and similar, so a basic DSL to provide functionality to update data.
3.) After .zap file is loaded, the data is modified according to the script.
4.) You can after that proceed normally given to what you flow is: export .zap file back out, generate, etc. Whatever you are doing.

By default zcl.json and gen-templates.json points to what appears the files in the zap repo, the paths are:

"path": "../../../../../../zap/zcl-builtin/silabs/zcl.json",
"path": "../../../../../../zap/test/gen-template/zigbee/gen-templates.json",

You cannot add the correct files using zap, it complains (in the terminal) about a wrong file extension, and every time you save it overwrites whatever you set yourselves manually in the zap file.

In https://github.com/project-chip/connectedhomeip/pull/9246/files#diff-da5ff4a2da0ce6846cc2ab57f8c45892c2457a2dc89195dd20f5a40cbc94d0ca a 32-bit bitmap is being output like so in the long defaults:

            /* Endpoint: 1, Cluster: Window Covering (server), big-endian */                                                       \
                                                                                                                                   \
            /* 2246 - FeatureMap, */                                                                                               \
            0x00, 0x00, 0x00, 0x01,                                                                                                \
...
            /* Endpoint: 1, Cluster: Window Covering (server), little-endian */                                                    \
                                                                                                                                   \
            /* 2246 - FeatureMap, */                                                                                               \
            0x00, 0x00, 0x00, 0x01,                                                                                                \

This is in a situation in which only bit 0 (i.e. the lowest bit) is set. For little-endian this should mean that the value is 0x01, 0x00, 0x00, 0x00, I would think.

This is blocking that PR from landing at the moment, because the tests fail.

@tecimovic @vivien-apple

After editing a .zap file through the ui I get the following error message:

{"level":40,"time":1621490551426,"pid":55329,"hostname":"meows-iMac.local","name":"zap","msg":"HTTP server exception.","err":{"type":"TypeError","message":"browserApi.executeSave is not a function","stack":"TypeError: browserApi.executeSave is not a function\n    at fileSave (/Users/meow/Prog/chip/eve-ncs-chip/external/zap/src-electron/ui/menu.js:313:14)\n    at doSave (/Users/meow/Prog/chip/eve-ncs-chip/external/zap/src-electron/ui/menu.js:252:5)\n    at click (/Users/meow/Prog/chip/eve-ncs-chip/external/zap/src-electron/ui/menu.js:52:11)\n    at MenuItem.click (electron/js2c/browser_init.js:73:1742)\n    at Object.a._executeCommand (electron/js2c/browser_init.js:81:2492)","alert":"⛔"}}
{"level":50,"time":1621490551426,"pid":55329,"hostname":"meows-iMac.local","name":"zap","msg":{"alert":"⛔"}}

And the file is not updated.

When schema changes, zap will wipe out the DB and reload it from scratch.

This looses the SETTINGS table, which are local per-install settings.

Add following to the process of changing the scheme:

• before wiping out old schema, do a full SELECT on the SETTINGS table.
• perform the wipe-out as now.
• load back the data from earlier SELECT query into the new SETTINGS table.

Hi,

I'm trying to generate a cluster with:

• Binary Input
• Percentage Remaining
• Identify

The generated code for the binary input is not compiling. It also looks very strange:

In callback.h:

/** @brief Binary Input (Basic) Cluster Init
 *
 * Cluster Init
 *
 * @param endpoint    Endpoint that is being initialized
 */
void emberAfBinaryInput(Basic) ClusterInitCallback(chip::EndpointId endpoint);

and in gen_config.h:

// Use this macro to check if the server side of the Binary Input (Basic) cluster is included
#define ZCL_USING_BINARY_INPUT_BASIC_CLUSTER_SERVER
#define EMBER_AF_PLUGIN_BINARY_INPUT_(_BASIC_) _SERVER
#define EMBER_AF_PLUGIN_BINARY_INPUT_(_BASIC_)

Some compiler error output related to this code snippets:

../src/gen/gen_config.h:48: warning: "EMBER_AF_PLUGIN_BINARY_INPUT_" redefined
   48 | #define EMBER_AF_PLUGIN_BINARY_INPUT_(_BASIC_)
      | 
../src/gen/gen_config.h:47: note: this is the location of the previous definition
   47 | #define EMBER_AF_PLUGIN_BINARY_INPUT_(_BASIC_) _SERVER

and

In file included from ../src/gen/call-command-handler.cpp:25:
../src/gen/callback.h:56:30: error: variable or field 'emberAfBinaryInput' declared void
   56 | void emberAfBinaryInput(Basic) ClusterInitCallback(chip::EndpointId endpoint);
      |                              ^
../src/gen/callback.h:56:25: error: 'Basic' was not declared in this scope
   56 | void emberAfBinaryInput(Basic) ClusterInitCallback(chip::EndpointId endpoint);

Here is also the used file for the generation: config.zap.txt

Zap is currently doing the following wrong thing:
1.) When starting up, you specify the ZCL and generation meta-info files.
2.) Zap will physically load those two bits of data into the database upon startup.
3.) When we load the .zap file, it will attach to these.

This model completely breaks-down when we load in the multiple .zap files that are based on different SDK.

The required flow is the following:
1.) We specify ZCL and generation meta-file at startup but those are just DEFAULT values.
2.) We do NOT load them at startup.
3.) When we open a "new configuration", we only then load in the DEFAULT metafiles, unless previously loaded.
4.) When we are loading in a .zap file, we take the values of the meta-file from the .zap files and:
a.) we load them in at this point, if not yet loaded.
b.) if we can't load them, we ask the user if it's ok to use DEFAULTS.
c.) user may also choose to point to a whole-different set of files which we load in at this point.

There was an issue with user_cluster_has_enabled_command helper. It is returning non-deterministic results because the GROUP BY in the query queryCommand.selectCommandDetailsFromAllEndpointTypesAndClusters() has a GROUP BY but no aggregates, so essentially it's simply picking the first value from the rows which have duplicates.

Given that the sorting order there is irrelevant, we should not be doing that. I suspect this query comes from assumption that the configuration of a command across multiple endpoints MUST be the same (as is the case in Silabs ZIgbee Pro implementation).

In case of Matter, they may be different, so we need to fix this.

I've temporarily patched the matter by making GROUP BY optional in the query, that but that just patches the problem for now.

could not npm install successfully.
Environment

• Windows 10
• node v14.15.1

commit : e43bd37

error on the postinstall step : electron-builder install-app-deps && husky install

I installed GTK manually, Note that the GTK bundles could not be downloaded from here https://github.com/Automattic/node-canvas/wiki/Installation%3A-Windows, the download is broken

I searched for cairo, and in this link the official GTK lin is also broken : https://www.cairographics.org/download/

after searching for a while from here : http://gladewin32.sourceforge.net/

but I still have missing dlls as not clear from where to get which version of GTK,
Warning: Missing input files:
C:\GTK\bin\libpng14-14.dll

I think then the npm install fallback on building and the dependencies errors start:
D:\Projects\zap\node_modules\canvas\src\register_font.h(3,10): fatal error C1083: Cannot open include file: 'pango/pango.h': No such file or directory

If possible to provide a hint if this is an electron install issue or which dependency do I have to focus on, the post install step is not clear to me.

Otherwise if this project is not currently building on all platforms if possible to provide the exact platform and dependencies versions built with so that I try to reproduce.

Any help would be appreciated.