Splitting transport layer out of library? about zigbee4java HOT 18 CLOSED

Hi Chris, Tommi,

Yes, I am currently interested in abstracting the transport method from the
current zigbee4java library. Actually I've already completed a simple
abstraction in my fork, and privately have a second transport
implementation for Android USB host (CC2531 plugged into Android USB
port). I'm happy to share this implementation if either of you are
interested, but I didn't want to include Android code in an otherwise
standard Java project, so kept it separate.

My Android implementation isn't quite working yet, because of another
issue: the Android USB host really requires working with byte streams and
not single ints at a time. I played with buffering these ints and
converting back to byte streams in the USB implementation, but it all seems
quite inefficient so I would rather go a little deeper into the
ZPacketParser etc and convert it to use bytes. Unsigned byte handling is
always a thorny issue in Java, and I saw comments in the zigbee4java code
that suggest a struggle with this, unsure which way to go. Actually to
transform between a 0-255 int and a binary-same 0-255 unsigned byte is not
too difficult and I have introduced a function to go both ways which I am
applying in the appropriate places.

Without going into further detail, with the limited time I have I'm
continuing to look at a refactor throughout ZPacketParser layer and make it
all byte based, converting where needed. I also see theres a lot of places
where shorts and longs are needed and the conversion is done manually. Did
you consider using Java ByteBuffer? I need to confirm it produces the
binary-same results but this could be a more concise Java-way of doing the
conversion.

After this I hope I'll be able to run the Zigbee Console again (testing
just with the CC2531 plugged into my desktop first).

It will not be easy to avoid some initial regression with this change but I
am continually running the Unit Test cases that have already been created
and they are still passing so far :-)

Anyway, to answer your question Chris I am all for collaborating, it makes
sense as there are only a few of us. Tommi this looks like a great effort
and my initial thoughts of writing something from scratch were quickly
forgotten when I realised how much work was already put in here. It looks
like it could do with a little TLC in places though, so as long as we have
compatible purposes I think it really makes sense to work on improving the
main branch together. Tommi do you see the internal conversion that I am
doing as valuable or is this something you have been through before and
would warn me against attempting? Great, Chris, that you are taking a look
at the current implementation vs. ZigBee spec, which I suspect may have
changed slightly between the reference CC2531 firmware that Tommi developed
with, and the more recent 'Z-Stack Home' firmware that I am trying to use
at the moment. Tommi do you have any insight into the difference between
the older firmware from your documentation, and Ti's newer Z-Stack Home
http://www.ti.com/tool/z-stack firmware, which I think is essentially a
later version.

The hex file I flashed my CC2531 with, from the Z-Stack Home package, is
called *CC2531ZNP-Pro.hex *.
I've hosted it on Google Drive
https://drive.google.com/file/d/0Bx7-OG-TjZUqMG13bEFJam9xeHc/view?usp=sharing
for convenience.

Thanks,

Chris

On 21 February 2015 at 07:33, Chris Jackson [email protected]
wrote:

I'm interested in discussing splitting out the COM port from the library
to allow other coordinators to be used. I see @chris-hatton
https://github.com/chris-hatton is doing something similar in his fork.

Is this something that is of interest? I think it makes sense to define an
interface, and then (somehow) pass this as an argument into the API
initialisation. The question is what is the most appropriate place to put
this interface such that it will work with 'any' coordinator.

I'm not familiar enough (yet!) with ZigBee to know what other interfaces
are available, and therefore where is best to define such an interface. I'm
guessing it's at a higher level than the COM port level.

I know ZB4O has a similar concept with their OSGi interfaces, so maybe the
interface can easily be defined at the same level

I think this would be a good route for the library but I'm interested in
your thoughts. Since there's only a few people working on this it would be
good to try and maintain a common codebase if possible so we can benefit
from each others bug fixes.

—
Reply to this email directly or view it on GitHub
#4.

from zigbee4java.

Hi Chris,

I'm happy to share this implementation if either of you are
interested, but I didn't want to include Android code in an otherwise
standard Java project, so kept it separate.
I would be interested to see it, more out of interest at the moment. However, I would advocate keeping it separate, but also to remove the current serial driver from the library and treat it as a separate entity (e.g. maybe a separate repository?). This would make it easier for people to provide different transport layers (e.g. using an Ethernet coordinator, or maybe something like an XBee, or…). It would mean the main library remains independent of the transport - we just need to make sure the interface is defined at the correct point to allow a wide variety of hardware. I haven’t looked too closely at this layer yet, but I hope that’s the network.serial package?

I’ve got an Ethernet coordinator here that I want to support at some stage, and I know others want to use things like XBee and other USB coordinators - I don’t have any intention of writing interfaces for all (or maybe any) of these, but I’d like to be able to define an interface where support can be added without changes to the library.

From what I can see, and from what little I currently know about ZigBee interfaces, your work looks like the right direction - Tommi - what do you think?

Anyway, to answer your question Chris I am all for collaborating, it makes
sense as there are only a few of us.
Great :) I’ve had a quick look at what you’re up to and it looks good (similarly aligned to what I think). If we can keep the core part of the library separate from our individual usage (which ought to be easy enough) then it should benefit us all to keep a single master repo :)

Great, Chris, that you are taking a look
at the current implementation vs. ZigBee spec,
At the moment, I’m still learning. I’ve spent the past 18 months working on a zwave stack (which is ongoing, but working well) and now find myself with some ZigBee hardware so am turning some of my attention this way. I’m basically playing with some hardware, and messing with the console example app to get my head around the library and the zigbee layers. I’m also trying to refer to the protocol docs where I can to help my understanding before I jump in too deep and mess stuff :)

Cheers
Chris

from zigbee4java.

Hi

It is fairly straightforward to add different serial port implementations as long as the dongle / controller remains the same or at least implements the same serial command interface. The serial command interface is a proprietary interface from Texas Instruments.

ZigBee for OSGI and subsequently ZigBee for Java have tight integration between classes implementing the API and the Texas Instruments serial interface. Decoupling would mean quite extensive rewrite. One way to achieve this would be to implement another hardware integration against same API and refactor all intermediate classes between API and serial interface encode / decode to achieve better decoupling.

I would personally favor developing custom serial command interface which would be open source and could be freely adopted to be used with any hardware.

It is a good idea to separate the different serial port and serial command interface implementations to separate maven projects.

Regards,
Tommi

from zigbee4java.

Hi Tommi,
Thanks for the feedback.

The problem with developing a new serial interface is that it doesn’t allow use of existing dongles/ethernet coordinators/whatever. I’d like to be able to support different (commercially available) coordinators rather than requiring someone to build/buy a special piece of hardware.

I thought that ZB4O had this concept? It seems possible to develop an interface bundle that provides the software API to the dongle, and I thought they had produced 2 different ones (both for TI chips, but I think different). Maybe I need to look a bit deeper into this…

Chris

from zigbee4java.

Hi

The idea is inspiring and certainly doable. The question is how much code one can reuse and if with some refactoring the level of reuse can be increased.

You are correct that custom serial command interface would work only with custom firmwares. For devices offering existing serial command interface one would need software plugins. For example clusters do have separated interfaces and implementations so it is possible to switch the implementation but there is no plugin framework as of now in place.

Regards,
Tommi

from zigbee4java.

The idea is inspiring and certainly doable. The question is how much code one can reuse and if with some refactoring the level of reuse can be increased.

Do you mean re-use between ‘drivers’, or reuse from the existing zigbee4java code in order to support this abstraction? I’m not interested (right now) in writing lots of drivers to support the different hardware, just in having this available as I think with the project I work on really needs the option of implementing other dongles. That said, I also have an ethernet coordinator here which I’d like to look at using if I can, so having this transport layer abstraction separated from the main library is something I’d be interested in.

In ZB4O, they have what they call ZIC (ZigBee Interface Controller). I can never find anything on their site, so here’s the link -:
http://zb4osgi.aaloa.org/redmine/projects/zb4osgi/wiki/ZIC_Deve http://zb4osgi.aaloa.org/redmine/projects/zb4osgi/wiki/ZIC_Deve

I’ve not looked at where they’ve split the interface, but I would hope that it’s ‘simple’ to do the same here?

I’ve not thought this through properly yet, but I hope it should be possible to have a separate class/package that implements the interface. We then instantiate that, and pass it as a parameter to the library initialisation? Something like that anyway…

More thought needed obviously :)

from zigbee4java.

Hi

zigbee4java does have this same setup but the SimpleDriver interface is renamed to ZigBeeNetworkManager interface. It is a good plugin point but my observation is that the page about ZIC in zigbee4osgi site gives tad optimistic picture of the situation. The message objects like ZDO_NODE_DESC_REQ used in the interface are not pure value objects but inherit from ZToolPacket and consist mostly of serialization code specific to the Texas Instruments serial interface. I think that we would need to refactor these messages to be pure value objects and place the serialization code somewhere else. There may be other points as well where the Texas Instruments serial interface dependency is leaking outside the SimpleDriver.

Regards,
Tommi

from zigbee4java.

Good evening Tommi, Chris,

I’ve been out of touch for a few days, being busy with the day job, but read your exchanges about continuing work on zigbee4java with interest.

As you previously noticed Chris, I had an experimental branch of my own fork, where I had abstracted out the Serial port.
In this branch I had also replaced Maven with Gradle, and propagated the use of byte rather than int, inside ZPacketParser.
These changes were done in a somewhat experimental fashion and not consistently enough to contribute.

So, this evening I restarted development from Tommi’s current master position, and manually re-played only the minimal changes required for the port abstraction.
At both your suggestion, I also prepared two new repositories, zigbee4java-serialPort and zigbee4java-androidUsbSerialPort, where two concrete port implementations reside.
zigbee4java-serialPort is simply the reference Serial Port implementation, as removed from the main library.
zigbee4java-androidUsbSerialPort is intended only for use in an Android build environment, and drives a CDC-ACM USB/Serial converter (which the CC2531 presents itself as) using Android’s USB-Host feature. This means that on most Android devices with a USB port or suitable ‘OTG’ USB adapter, zigbee4java may drive the CC2531 dongle with no root access or Android kernel changes.

My fork with new changes made in master branch: https://github.com/Arvis-Home/zigbee4java https://github.com/Arvis-Home/zigbee4java
zigbee4java-serialPort repository: https://github.com/Arvis-Home/zigbee4java-serialPort https://github.com/Arvis-Home/zigbee4java-serialPort
zigbee4java-androidUsbSerialPort repository: https://github.com/chris-hatton/zigbee4java-androidUsbSerialPort https://github.com/chris-hatton/zigbee4java-androidUsbSerialPort

I feel these changes are ready to integrate and, if you are interested in doing so, please review them in my fork.
If you approve them I will happily create a Pull Request.

As part of this contribution I’d transfer you ownership of the zigbee4java-serialPort repository.

I’m keen to continue contributing to the project, with a personal aim of establishing full functionality on Android, using the CC2531 flashed with the latest Z-Stack Home firmware?

Towards this aim, do you have any insight into whether zigbee4java’s protocol has drifted from that expected by the latest Z-Stack Home? This does appear to be the direct successor to the earlier, firmware advised by the zigbee4java documentation. Since this is older and quite convoluted to compile, and Z-Stack Home is made readily available as a binary hex file, it seems to make sense to me, if we aim to establish base-line compatibility and re-focus on Z-Stack Home as zigbee4java's target firmware.

What are your thoughts on this?

Thanks & regards,

Chris Hatton

On 23 Feb 2015, at 01:24, Tommi S.E. Laukkanen [email protected] wrote:

Hi

zigbee4java does have this same setup but the SimpleDriver interface is renamed to ZigBeeNetworkManager interface. It is a good plugin point but my observation is that the page about ZIC in zigbee4osgi site gives tad optimistic picture of the situation. The message objects like ZDO_NODE_DESC_REQ used in the interface are not pure value objects but inherit from ZToolPacket and consist mostly of serialization code specific to the Texas Instruments serial interface. I think that we would need to refactor these messages to be pure value objects and place the serialization code somewhere else. There may be other points as well where the Texas Instruments serial interface dependency is leaking outside the SimpleDriver.

Regards,
Tommi

—
Reply to this email directly or view it on GitHub #4 (comment).

from zigbee4java.

Hi

Great work Chris. I am a bit busy myself during this week but if you make pull request I will try to review and pull it latest this weekend. I have not yet looked nor tested Z-Stack Home firmware but if it works well (as it should :) and has security enabled version then I think it is really good thing to have as official firmware for us. What do you think Chris J?

Chris H, unless you specifically want to keep android serial port version as separate project we could include it as well as maven sub module. Android projects can be compiled (these days) with maven so there is no technical reason as such to keep it separate? It would be easier to publish the entire packet to maven repository and make sure compilations pass if interfaces are modified.

I can share access to my repository for regular contributors or alternatively we can work with forks and I can do reviews when processing pull requests.

With sub module naming I would prefer convention along these lines:

• zigbee4java-default-serial
• zigbee4java-android-serial

Your thoughts on the naming Chris H an Chris J?

Best regards,
Tommi

from zigbee4java.

Hi Tommi, ChrisJ,

Briefly:

• Yes, Z-Stack Home does have a security enabled version as well.
• No particular need to keep the Android port module separate; I will
  have a play with bringing the two external port repo's back into the
  main zigbee4java repo. as Maven modules, then defining Maven profiles to
  selectively compile with port(s).
• I'm fine with your proposed module naming, Tommi, and will rename
  accordingly - ChrisJ any opinion on this?
• I'd prefer if you review pull requests for the first couple of
  contributions, gives us an opportunity to make sure we're aligned in
  direction and style etc. maybe we can get direct repo. access when we
  get comfortable.

Cheers,
ChrisH

On 26/02/2015 6:34 am, Tommi S.E. Laukkanen wrote:

Hi

Great work Chris. I am a bit busy myself during this week but if you
make pull request I will try to review and pull it latest this
weekend. I have not yet looked nor tested Z-Stack Home firmware but if
it works well (as it should :) and has security enabled version then I
think it is really good thing to have as official firmware for us.
What do you think Chris J?

Chris H, unless you specifically want to keep android serial port
version as separate project we could include it as well as maven sub
module. Android projects can be compiled (these days) with maven so
there is no technical reason as such to keep it separate? It would be
easier to publish the entire packet to maven repository and make sure
compilations pass if interfaces are modified.

I can share access to my repository for regular contributors or
alternatively we can work with forks and I can do reviews when
processing pull requests.

With sub module naming I would prefer convention along these lines:

• zigbee4java-default-serial
• zigbee4java-android-serial

Your thoughts on the naming Chris H an Chris J?

Best regards,
Tommi

—
Reply to this email directly or view it on GitHub
#4 (comment).

from zigbee4java.

Hi guess,
Just home after a couple of days travelling for the day job :(

I think it is really good thing to have as official firmware for us. What do you think Chris J?
Absolutely - this is great :)

This is definitely a step in the right direction so thanks for the work :)

Chris

from zigbee4java.

• I'm fine with your proposed module naming, Tommi, and will rename
  accordingly - ChrisJ any opinion on this?
  No - that’s fine by me
• I'd prefer if you review pull requests for the first couple of
  contributions, gives us an opportunity to make sure we're aligned in
  direction and style etc. maybe we can get direct repo. access when we
  get comfortable.
  It’s always nice to have someone look over code I think. Well - at least for me anyway as I’m still getting to grips with zigbee :)

Chris

from zigbee4java.

Hi

I just pulled the pull request from Chris H. Looks good and I am working through the rest of them next. Just need to check if we get merge conflicts.

2-3 pull requests to get aligned and then direct repo access sounds like a good principle.

-Tommi

from zigbee4java.

Hi

I have pulled your pull requests Chris H and Chris J. Thank you for the improvements. There are plenty of code from both in them so I have added you as collaborators. Lets keep our code base clean and functional. I suggest the following conventions to achieve this but everything is open to discussion:

1. Push one thing at a time.
2. Where possible and convenient collect all changes related to same thing in the single commit,
3. but on the other hand commit often to avoid merge problems.
4. If you develop something large changing a lot of existing classes then please develop it in your fork and lets look at it together before pulling it to master repository.
5. Unit test and explorative test before committing.
6. Verify that maven build works before committing.
7. Lets try to explorative test each others changes before release.
8. I will try to do maven release each Sunday if there is something to release.

We can also add checkstyle definition to project root if we can agree on something reasonable. I usually prefer something originating from Sun convention with certain rules relaxed.

Having integration test project built against selected cheap devices everyone can quite easily purchase would be great addition. What do you think?

Cheers,
Tommi

from zigbee4java.

Great stuff - thanks guys.

I’ve just moved my project over to the latest codebase - thanks for the work - it’s looking good.

I’m happy with all your points Tommi - thanks.

Having integration test project built against selected cheap devices everyone can quite easily purchase would be great addition. What do you think?

This is definitely a good idea and I’m happy to take a look at this. The question is ‘what devices’. I was doing a bit of a scout around for devices last week and at least in Europe it’s not clear to me what people use - there doesn’t seem to be an obvious ‘cheap device’ around. I found a few, but they all have different interfaces - I was going to ‘randomly’ buy one to see if I could get it working, mainly to check that we’d split the interface at the right point, but I’ve not done so yet…

I also have an ethernet coordinator here which I’d like to have a play with, but information on it is a bit minimal so it might take a little effort to sort out…

Cheers
Chris

from zigbee4java.

Hi guys,

Tommi - thank you for the repository permissions and accepting my pull request, all your points from previous email noted and agreed.

Just wanted to run a proposal by you both, re: progressing Android support. We currently have zigbee-console as the test-bed application for desktop, which currently depends upon zigbee-serial-default.

What I propose is that we support zigbee-console on both Desktop and Android platforms; by introducing two new submodules: zigbee-console-default and zigbee-console-android.
The platform-independent console engine would remain in zigbee-console, possible renamed to zigbee-console-common/shared (opinions?), and the smaller platform specific front-ends would be created in the new modules.

You may have noticed that the file ZigBeeConsole.java has already been split into ZigBeeConsole.java and ZigBeeSerialConsole.java, inside the console module.
I would begin by moving ZigBeeSerialConsole.java out into zigbee-console-default and then start work on an Android UI for the core console in the other new module.

I am developing the Android parts using Android Studio, which unfortunately cannot work with Maven modules, so I am building Gradle support files which mirror the structure of Maven as closely as possible. I don’t believe there will be any problems in having both Gradle and Maven files alongside each other in the same project but let me know if you foresee any issues.

Tommi, ChrisJ, please let me know if you are both happy with the above proposal or would like to consider another strategy for Android support.

Thanks,
ChrisH

On 1 Mar 2015, at 21:14, Chris Jackson [email protected] wrote:

Great stuff - thanks guys.

I’ve just moved my project over to the latest codebase - thanks for the work - it’s looking good.

I’m happy with all your points Tommi - thanks.

Having integration test project built against selected cheap devices everyone can quite easily purchase would be great addition. What do you think?

This is definitely a good idea and I’m happy to take a look at this. The question is ‘what devices’. I was doing a bit of a scout around for devices last week and at least in Europe it’s not clear to me what people use - there doesn’t seem to be an obvious ‘cheap device’ around. I found a few, but they all have different interfaces - I was going to ‘randomly’ buy one to see if I could get it working, mainly to check that we’d split the interface at the right point, but I’ve not done so yet…

I also have an ethernet coordinator here which I’d like to have a play with, but information on it is a bit minimal so it might take a little effort to sort out…

Cheers
Chris

—
Reply to this email directly or view it on GitHub #4 (comment).

from zigbee4java.

Hi

I am all for having android console module in the project.

It would be nice if all the modules would be built by default so compilation problems would be detected in the entire code base. We could have a default maven profile which builds everything?

Just a side note: The maven support for android is fairly recent but I think you can import the maven projects to at least IntelliJ Idea based android studio and have the android functions enabled. I do like the idea of supporting both maven and gradle to be friendly for both kind of projects though.

For the integration test device set I would like to include at minimum 1 hue bulb or similar and a temperature sensor. This would allow us to include tests for both wired and battery devices, control and reporting features. It would be nice to have both subscribe based and broadcasting sensors in the set so this would mean two sensors. I am mostly interested in the security enabled mode but if there is business case for non secure network then we should have test setup for both. I would personally use have the secure one I think. We should build the integration test set so that you can easily choose which devices you actually have even if we agree on some base set.

My hope is that one could purchase all the sensors for maximum $100 total but that may be optimistic at this date. If you have ideas for devices we could create another issue for the integration test harness and discuss things further there.

-Tommi

from zigbee4java.

Closing this as this is done. We can continue discussions in discussion group. Please let me know if you have better forum / group services in mind.

from zigbee4java.