matthijskooijman / arduino-dsmr Goto Github PK

Hi,
I'm been changing from an ESP32/Arduino environment into an ESP32/esp-idf environment that has support for Arduino-code, and so far everything has been smooth, except for this library.
Somehow, the compiler trips over the following code:

#define DEFINE_FIELD(fieldname, value_t, obis, field_t, field_args...) \
  struct fieldname : field_t<fieldname, ##field_args> { \
    value_t fieldname; \
    bool fieldname ## _present = false; \
    static constexpr ObisId id = obis; \
    static constexpr char name_progmem[] DSMR_PROGMEM = #fieldname; \
    static constexpr const __FlashStringHelper *name = reinterpret_cast<const __FlashStringHelper*>(&name_progmem); \
    value_t& val() { return fieldname; } \
    bool& present() { return fieldname ## _present; } \
  }

at the reinterpret_cast<.... part it gives me this error:

.pio/libdeps/switch-1/Dsmr/src/dsmr/fields.h:185:56: error: a reinterpret_cast is not a constant expression
     static constexpr const __FlashStringHelper *name = reinterpret_cast<const __FlashStringHelper*>(&name_progmem); \
                                                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.pio/libdeps/switch-1/Dsmr/src/dsmr/fields.h:231:1: note: in expansion of macro 'DEFINE_FIELD'
 DEFINE_FIELD(electricity_long_failures, uint32_t, ObisId(0, 0, 96, 7, 9), IntField, units::none);

What i've tried/checked

• The build flags for the GCC compiler (C++11) for the Arduino environment and for the esp-idf environment are the same: gcc 8.2.0
• the __FlashStringHelper is present, and is the same as before
• The DEFINE_FIELD() functions where the DEFINE_FIELD macro is used, expand to the same code in both environments.

Could you give me any hints where i should look to solve this issue?

Hi there,

I am a user in Germany and I can use the P1-port of my smart meter as well (thx to my power provider).
I have an ISKRA MT382 which seems to be widely used in your country as well.
I also seems that the same OBIS codes are used, but I am missing some of them.

I have a tariff that uses four counters, so 1.8.1 (20:00 - 06:00 Uhr/Uur), 1.8.2 (06:00 - 14:00 Uhr/Uur), 1.8.3 (14:00 - 17:00 Uhr/Uur) and 1.8.4 (17:00 - 20:00 Uhr/Uur) is used. In you parser only 2 of them are available.
Because the total counter (1.8.0) is not available as well, as it seems, I cannot use the parser to get all power consumption as I want to.

Maybe you could be so nice and add other delivery counters (1.8.3 and 1.8.4, maybe more till 1.8.8) and the totals 1.8.0 (and 2.8.0, because there is only one tariff in Germany for delivery, so only 2.8.0 is used).
Because this wouldn't be a breaking change and is not against the logic of the existing code, I hope you can help me to get all the data I need with your great parser...

btw. I found some reference, that 1.8.0 and 2.8.0 is used in Luxembourg as well:
https://electris.lu/files/Dokumente_und_Formulare/Netz_Tech_Dokumente/SPEC_-_E-Meter_P1_specification_20210305.pdf

Regards,

Sebastian

Here is an example of the output of my meter.

/ISK5\2MT382-1008<\r>
0-0:96.1.1(314xxxxxxxxxxxxxxxxxxxxx37)<\r><\n>  --> available 
0-0:128.20.0(016)<\r><\n>
1-0:0.9.2(210528)<\r><\n>
1-0:0.9.1(215658)<\r><\n>
1-0:1.7.0(001.322kW)<\r><\n> --> available 
1-0:2.7.0(000.000kW)<\r><\n> --> available 
**1-0:1.8.0(010192.57kWh)<\r><\n> --> Total, not available
1-0:1.8.1(006094.31kWh)<\r><\n> --> Tariff 1, available
1-0:1.8.2(001410.19kWh)<\r><\n> --> Tariff 2, available
**1-0:1.8.3(000981.30kWh)<\r><\n> --> Tariff 3, not available
**1-0:1.8.4(001706.76kWh)<\r><\n> --> Tariff 4, not available
1-0:1.8.5(000000.00kWh)<\r><\n> --> Tariff 5 (not used in my case), not available
1-0:1.8.6(000000.00kWh)<\r><\n> --> Tariff 6 (not used in my case), not available
1-0:1.8.7(000000.00kWh)<\r><\n> --> Tariff 7 (not used in my case), not available
1-0:1.8.8(000000.00kWh)<\r><\n> --> Tariff 8 (not used in my case), not available
1-0:1.2.0(106.732kW)<\r><\n>
1-0:1.6.0(007.072kW)<\r><\n>
**1-0:2.8.0(006715.38kWh)<\r><\n> --> Total, not available
1-0:3.8.0(000122.04kvarh)<\r><\n>
1-0:4.8.0(006255.99kvarh)<\r><\n>
1-0:0.2.2(Smart001)<\r><\n>
1-0:0.3.0(00500)<\r><\n>
1-0:0.3.1(00500)<\r><\n>
1-0:0.3.3(250)<\r><\n>
0-0:97.97.0(00000000)<\r>

When the Arduino runs low on memory, malloc starts failing and the String class starts silently dropping its contents. In practice, this means that the parser.ino example running on a Uno gives empty values for all String fields. I suspect that memory is just enough, since no other random behaviour happens (which is typical when memory is exhausted and the stack starts to overwrite the heap and global variables), but malloc starts to fail a bit earlier (to leave some space for the stack).

After compilation, there are still 800 bytes of free RAM. About 300 are taken up by the huge data struct, but the other 500 are possibly taken up by stack variables? This might warrant some investigation to try and reduce the memory usage.

Additionally, it would be good if memory errors could be handled more gracefully. I'm not sure if the String class properly returns this error condition (it might have some kind of "isvalid" method), but worst case we can just check the String length value and return an error if it does not match.

On the Belgian smart meters, linked to the introduction of the so called capacity tariff as of 1 Jan '23, the smart meter will keep a log of max 13 values, each representing a month (timestamped) and the corresponding maximum quarter hourly power consumption in kW.
This is how it will look for 3 months :
0-0:98.1.0(3)(1-0:1.6.0)(1-0:1.6.0)(200501000000S)(200423192538S)(03.695kW)(2
00401000000S)(200305122139S)(05.980kW)(200301000000S)(200210035421W)(04.318*k
W)
Is there a way to have this parsed other than a RawField ?

Hi Matthijs
Thank you for this library. I am having problems getting it to work at the labtable. I installed the DSMRlogger program from Aandewiel and that parses data. However I am trying to get a labtable setup with a PIC controller emulating the DSMR smart meter, and I do not get any result, not even an error..... The Aandewiel program lets me see the raw data from the smart meter and -when connected to the PIC controller- indeed outputs the sent data. However it is not parsed, nothing, not even an error, just nothing.

The PIC sends out a telegram. Unlike the smart meter it throws out the data in one blow, no pauses in between. Every line is ended with a 0x0D, 0x0AD:

/KFM5KAIFA-METER                                                                
                                                                                
1-3:0.2.8(42)                                                                   
0-0:1.0.0(190805204420S)                                                        
0-0:96.1.1(4530303236303030303237313334363135)                                  
1-0:1.8.1(004029.456*kWh)                                                       
1-0:1.8.2(002611.355*kWh)                                                       
1-0:2.8.1(000000.000*kWh)                                                       
1-0:2.8.2(000000.000*kWh)                                                       
0-0:96.14.0(0002)                                                               
1-0:1.7.0(00.163*kW)                                                            
1-0:2.7.0(00.000*kW)                                                            
0-0:96.7.21(00012)                                                              
0-0:96.7.9(00005)                                                               
1-0:99.97.0(6)(0-0:96.7.19)(190508111811S)(0000008169*s)(190406211046S)(0000011)
1-0:32.32.0(00001)                                                              
1-0:52.32.0(00003)                                                              
1-0:72.32.0(00004)                                                              
1-0:32.36.0(00000)                                                              
1-0:52.36.0(00000)                                                              
1-0:72.36.0(00000)                                                              
0-0:96.13.1()                                                                   
0-0:96.13.0()                                                                   
1-0:31.7.0(000*A)                                                               
1-0:51.7.0(000*A)                                                               
1-0:71.7.0(000*A)                                                               
1-0:21.7.0(00.157*kW)                                                           
1-0:41.7.0(00.000*kW)                                                           
1-0:61.7.0(00.011*kW)                                                           
1-0:22.7.0(00.000*kW)                                                           
1-0:42.7.0(00.000*kW)                                                           
1-0:62.7.0(00.000*kW)                                                           
0-1:24.1.0(003)                                                                 
0-1:96.1.0(4730303332353631323533373030343135)                                  
0-1:24.2.1(190805200000S)(03293.167*m3)                                         
!5A99 

I changed some of your example program to get this:

/*
 * Permission is hereby granted, free of charge, to anyone
 * obtaining a copy of this document and accompanying files,
 * to do whatever they want with them without any restriction,
 * including, but not limited to, copying, modification and redistribution.
 * NO WARRANTY OF ANY KIND IS PROVIDED.
 *
 * Example that shows how to periodically read a P1 message from a
 * serial port and automatically print the result.
*/

#include "dsmr.h"

/**
 * Define the data we're interested in, as well as the datastructure to
 * hold the parsed data. This list shows all supported fields, remove
 * any fields you are not using from the below list to make the parsing
 * and printing code smaller.
 * Each template argument below results in a field of the same name.
 */
using MyData = ParsedData<
  /* String */ identification,
  /* String */ p1_version,
  /* String */ timestamp,
  /* String */ equipment_id,
  /* FixedValue */ energy_delivered_tariff1,
  /* FixedValue */ energy_delivered_tariff2,
  /* FixedValue */ energy_returned_tariff1,
  /* FixedValue */ energy_returned_tariff2,
  /* String */ electricity_tariff,
  /* FixedValue */ power_delivered,
  /* FixedValue */ power_returned,
  /* FixedValue */ electricity_threshold,
  /* uint8_t */ electricity_switch_position,
  /* uint32_t */ electricity_failures,
  /* uint32_t */ electricity_long_failures,
  /* String */ electricity_failure_log,
  /* uint32_t */ electricity_sags_l1,
  /* uint32_t */ electricity_sags_l2,
  /* uint32_t */ electricity_sags_l3,
  /* uint32_t */ electricity_swells_l1,
  /* uint32_t */ electricity_swells_l2,
  /* uint32_t */ electricity_swells_l3,
  /* String */ message_short,
  /* String */ message_long,
  /* FixedValue */ voltage_l1,
  /* FixedValue */ voltage_l2,
  /* FixedValue */ voltage_l3,
  /* FixedValue */ current_l1,
  /* FixedValue */ current_l2,
  /* FixedValue */ current_l3,
  /* FixedValue */ power_delivered_l1,
  /* FixedValue */ power_delivered_l2,
  /* FixedValue */ power_delivered_l3,
  /* FixedValue */ power_returned_l1,
  /* FixedValue */ power_returned_l2,
  /* FixedValue */ power_returned_l3,
  /* uint16_t */ gas_device_type,
  /* String */ gas_equipment_id,
  /* uint8_t */ gas_valve_position,
  /* TimestampedFixedValue */ gas_delivered,
  /* uint16_t */ thermal_device_type,
  /* String */ thermal_equipment_id,
  /* uint8_t */ thermal_valve_position,
  /* TimestampedFixedValue */ thermal_delivered,
  /* uint16_t */ water_device_type,
  /* String */ water_equipment_id,
  /* uint8_t */ water_valve_position,
  /* TimestampedFixedValue */ water_delivered,
  /* uint16_t */ slave_device_type,
  /* String */ slave_equipment_id,
  /* uint8_t */ slave_valve_position,
  /* TimestampedFixedValue */ slave_delivered
>;

/**
 * This illustrates looping over all parsed fields using the
 * ParsedData::applyEach method.
 *
 * When passed an instance of this Printer object, applyEach will loop
 * over each field and call Printer::apply, passing a reference to each
 * field in turn. This passes the actual field object, not the field
 * value, so each call to Printer::apply will have a differently typed
 * parameter.
 *
 * For this reason, Printer::apply is a template, resulting in one
 * distinct apply method for each field used. This allows looking up
 * things like Item::name, which is different for every field type,
 * without having to resort to virtual method calls (which result in
 * extra storage usage). The tradeoff is here that there is more code
 * generated (but due to compiler inlining, it's pretty much the same as
 * if you just manually printed all field names and values (with no
 * cost at all if you don't use the Printer).
 */
struct Printer {
  template<typename Item>
  void apply(Item &i) {
    if (i.present()) {
      Serial.print(Item::name);
      Serial.print(F(": "));
      Serial.print(i.val());
      Serial.print(Item::unit());
      Serial.println();
    }
  }
};

// Set up to read from the second serial port, and use D2 as the request
// pin. On boards with only one (USB) serial port, you can also use
// SoftwareSerial.
#ifdef ARDUINO_ARCH_ESP32
// Create Serial1 connected to UART 1
//HardwareSerial Serial1(1);
#endif
P1Reader reader(&Serial, 0);

unsigned long last;

void setup() {
  Serial.begin(9600);
//  Serial1.begin(115200);
  #ifdef VCC_ENABLE
  // This is needed on Pinoccio Scout boards to enable the 3V3 pin.
  pinMode(VCC_ENABLE, OUTPUT);
  digitalWrite(VCC_ENABLE, HIGH);
  #endif
Serial.print("testing parser.............\n\r");
  // start a read right away
  reader.enable(true);
  last = millis();
}

void loop () {
  // Allow the reader to check the serial buffer regularly
  reader.loop();

  // Every minute, fire off a one-off reading
  unsigned long now = millis();
  if (now - last > 60000) {
    reader.enable(true);
    Serial.print("trying to read a telegram....\n\r");
    last = now;
  }

  if (reader.available()) {
    Serial.print("now here\n\r");
    MyData data;
    String err;
    if (reader.parse(&data, &err)) {
      // Parse succesful, print result
      data.applyEach(Printer());
    } else {
      // Parser error, print error
      Serial.println(err);
    }
  }
}

I do program in C, not in C++ it is very hard for me to read and understand. What could be wrong? Could you explain the reader.loop function?
Thanks

Hi Matthijs,

I build a wireless logger with your great library and everything works great..

But now I'm trying to use a working logger with a different Smart Meter and everything goes haywire!

It works with this meter Landis Gyr+ (SMR 5.0)

but not with this meter ISKRA (AM550)
With the ISKRA only part of the telegram is parsed correct (the Energy Delivered/Returned records) but the Power records give strange results. But the parser does not report any errors(?).

Have you heard of this problem before and is there a fix for it?

If you need more information, just let me know!

Regards

I have upgraded my Arduino environment and EPS8266 libraries. I did run in the Error: a reinterpret_cast is not a constant expression and applied the fix a mentioned.

With the old version I was able to compile and run all my code. Since this upgrade I get error
read:98:12: error: conversion from 'const dsmr::fields::NameConverterdsmr::fields::identification' to non-scalar type 'String' requested
98 | String Name = Item::name;

I have been able to reduce the code in read.ino example to produce the error. I am lost, dont know how to fix this , I hope you can help.

This is the code in read.ino that fails during compile. Again this did work before the reinterpret_cast problem.
struct Printer { template<typename Item> void apply(Item &i) { String Name = Item::name; // DO WHAT WE WANT WITH String ..... } };

Detailed error during compile :
/Users/xxx/Documents/Arduino/read/read.ino: In instantiation of 'void Printer::apply(Item&) [with Item = dsmr::fields::identification]': /Users/xxx/Documents/Arduino/libraries/arduino-dsmr-master/src/dsmr/fields.h:46:12: required from 'void dsmr::ParsedField<T>::apply(F&) [with F = Printer; T = dsmr::fields::identification]' /Users/xxx/Documents/Arduino/libraries/arduino-dsmr-master/src/dsmr/parser.h:129:13: required from 'void dsmr::ParsedData<T, Ts ...>::applyEach_inlined(F&&) [with F = Printer&; T = dsmr::fields::identification; Ts = {dsmr::fields::p1_version, dsmr::fields::timestamp, dsmr::fields::equipment_id, dsmr::fields::energy_delivered_tariff1, dsmr::fields::energy_delivered_tariff2, dsmr::fields::energy_returned_tariff1, dsmr::fields::energy_returned_tariff2, dsmr::fields::electricity_tariff, dsmr::fields::power_delivered, dsmr::fields::power_returned, dsmr::fields::electricity_threshold, dsmr::fields::electricity_switch_position, dsmr::fields::electricity_failures, dsmr::fields::electricity_long_failures, dsmr::fields::electricity_failure_log, dsmr::fields::electricity_sags_l1, dsmr::fields::electricity_sags_l2, dsmr::fields::electricity_sags_l3, dsmr::fields::electricity_swells_l1, dsmr::fields::electricity_swells_l2, dsmr::fields::electricity_swells_l3, dsmr::fields::message_short, dsmr::fields::message_long, dsmr::fields::voltage_l1, dsmr::fields::voltage_l2, dsmr::fields::voltage_l3, dsmr::fields::current_l1, dsmr::fields::current_l2, dsmr::fields::current_l3, dsmr::fields::power_delivered_l1, dsmr::fields::power_delivered_l2, dsmr::fields::power_delivered_l3, dsmr::fields::power_returned_l1, dsmr::fields::power_returned_l2, dsmr::fields::power_returned_l3, dsmr::fields::gas_device_type, dsmr::fields::gas_equipment_id, dsmr::fields::gas_valve_position, dsmr::fields::gas_delivered, dsmr::fields::thermal_device_type, dsmr::fields::thermal_equipment_id, dsmr::fields::thermal_valve_position, dsmr::fields::thermal_delivered, dsmr::fields::water_device_type, dsmr::fields::water_equipment_id, dsmr::fields::water_valve_position, dsmr::fields::water_delivered, dsmr::fields::slave_device_type, dsmr::fields::slave_equipment_id, dsmr::fields::slave_valve_position, dsmr::fields::slave_delivered}]' /Users/xxx/Documents/Arduino/libraries/arduino-dsmr-master/src/dsmr/parser.h:124:22: required from 'void dsmr::ParsedData<T, Ts ...>::applyEach(F&&) [with F = Printer; T = dsmr::fields::identification; Ts = {dsmr::fields::p1_version, dsmr::fields::timestamp, dsmr::fields::equipment_id, dsmr::fields::energy_delivered_tariff1, dsmr::fields::energy_delivered_tariff2, dsmr::fields::energy_returned_tariff1, dsmr::fields::energy_returned_tariff2, dsmr::fields::electricity_tariff, dsmr::fields::power_delivered, dsmr::fields::power_returned, dsmr::fields::electricity_threshold, dsmr::fields::electricity_switch_position, dsmr::fields::electricity_failures, dsmr::fields::electricity_long_failures, dsmr::fields::electricity_failure_log, dsmr::fields::electricity_sags_l1, dsmr::fields::electricity_sags_l2, dsmr::fields::electricity_sags_l3, dsmr::fields::electricity_swells_l1, dsmr::fields::electricity_swells_l2, dsmr::fields::electricity_swells_l3, dsmr::fields::message_short, dsmr::fields::message_long, dsmr::fields::voltage_l1, dsmr::fields::voltage_l2, dsmr::fields::voltage_l3, dsmr::fields::current_l1, dsmr::fields::current_l2, dsmr::fields::current_l3, dsmr::fields::power_delivered_l1, dsmr::fields::power_delivered_l2, dsmr::fields::power_delivered_l3, dsmr::fields::power_returned_l1, dsmr::fields::power_returned_l2, dsmr::fields::power_returned_l3, dsmr::fields::gas_device_type, dsmr::fields::gas_equipment_id, dsmr::fields::gas_valve_position, dsmr::fields::gas_delivered, dsmr::fields::thermal_device_type, dsmr::fields::thermal_equipment_id, dsmr::fields::thermal_valve_position, dsmr::fields::thermal_delivered, dsmr::fields::water_device_type, dsmr::fields::water_equipment_id, dsmr::fields::water_valve_position, dsmr::fields::water_delivered, dsmr::fields::slave_device_type, dsmr::fields::slave_equipment_id, dsmr::fields::slave_valve_position, dsmr::fields::slave_delivered}]' /Users/xxx/Documents/Arduino/read/read.ino:207:31: required from here read:98:12: error: conversion from 'const dsmr::fields::NameConverter<dsmr::fields::identification>' to non-scalar type 'String' requested 98 | String Name = Item::name; | ^~~~

Hope you can point me in right direction,
Regards,
Rob

Hero!

Does this handle the baudrate increase aswell when using a Serial interface?

Hi Matthijs,
I have seen reports that belgium smartmeters do not parse correctly using the 5.0.2 dsmr parsing. This is due to the extention made by the Belgium netbeheerder.

The document for Belgium can be found here

I will be looking into making changes to the library to enable parsing of the DSMR of Belgium.
Robert

Probably a user error but i'm trying to use the parser in a custom component for esphome:

https://github.com/nldroid/DsmrP1CustomSensor/blob/master/dsmr_p1_sensor.h

I get these errors during linking. Any idea how to fix this?

.pioenvs\esp_dsmr\src\main.cpp.o:(.text._ZN4dsmr10ParsedDataIINS_6fields10p1_versionENS1_24energy_delivered_tariff1EEE10parse_lineERKNS_6ObisIdEPKcS9_[_ZN4dsmr10ParsedDataIINS_6fields10p1_versionENS1_24energy_delivered_tariff1EEE10parse_lineERKNS_6ObisIdEPKcS9_]+0x4): undefined reference to `dsmr::fields::p1_version::id'
.pioenvs\esp_dsmr\src\main.cpp.o:(.text._ZN4dsmr10ParsedDataIINS_6fields10p1_versionENS1_24energy_delivered_tariff1EEE10parse_lineERKNS_6ObisIdEPKcS9_[_ZN4dsmr10ParsedDataIINS_6fields10p1_versionENS1_24energy_delivered_tariff1EEE10parse_lineERKNS_6ObisIdEPKcS9_]+0x8): undefined reference to `dsmr::fields::energy_delivered_tariff1::id'
.pioenvs\esp_dsmr\src\main.cpp.o:(.text._ZN4dsmr10ParsedDataIINS_6fields10p1_versionENS1_24energy_delivered_tariff1EEE10parse_lineERKNS_6ObisIdEPKcS9_[_ZN4dsmr10ParsedDataIINS_6fields10p1_versionENS1_24energy_delivered_tariff1EEE10parse_lineERKNS_6ObisIdEPKcS9_]+0xc): undefined reference to `dsmr::fields::units::kWh'

Hi,

I've been working with this library for a few days now and I feel a bit stuck. That has probably a lot to do with the way I'm using the library. I created a small little shield to connect to my P1 port on the smart meter. This shield has the request pin always at high. When initializing I just use a dummy pin to set "high" when I call reader.enable() from your library to trigger a read, in conjunction with a SoftwareSerial object. It is starting to look like something isn't flushed totally and because of that, I sometimes get a lot of "OBIS id Empty" messages, because there were multiple DSMR messages being read at once (?). Also I frequently experience "Invalid unit" messages but that looks more like a corrupt message to me (which doesn't bother me that much to be honest).

I've tried using reader.enable(false), removing the millis() check in the loop and still running reader.loop inside the arduino loop but to no avail.

I was wondering if you've experienced these kind of errors (well duh, you made the library) and if you have any suggestion on how to work around / fix this?

With kind regards,

Thomas

Sinds kort is mijn Raspberry pi 4 extreem traag. Na wat onderzoek lijkt het erop dat de python en gunicorn van dsmr reader bijna 100% van de cpu opgebruikt.

Ik gebruik de remote datalogger en op de server zelf de laatste docker stack. De sever zelf is waar het allemaal traag loopt. De remote datalogger lijkt niets aan de hand.

Kan hier iets aan gedaan worden?

Dear Matthijs,

I have this piece of code:

      slimmeMeter.loop();
.
.
.
      if (slimmeMeter.available()) {
        Debugf("read telegram [%d]\r\n", ++telegramCount);

        if (slimmeMeter.parse(&DSMRdata, &DSMRerror)) {  // Parse succesful, print result
          processData();
          if (Verbose1) {
            DSMRdata.applyEach(showValues());
            printData();
          }
          
        } else {                                    // Parser error, print error
          Debugf("Parse error %s\r\n", DSMRerror.c_str());
          telegramErrors++;
        }
        slimmeMeter.clear();  // reset DSMRdata ???
        
      } // if (slimmeMeter.available()) 

DSMRdata and DSMRerror are declaired globaly:

// Set up to read from the Serial port, and use VCC_ENABLE as the
// request pin. 
#ifdef VCC_ENABLE
  P1Reader slimmeMeter(&Serial, VCC_ENABLE);
#else
  P1Reader slimmeMeter(&Serial, 0);
#endif
MyData    DSMRdata;
String    DSMRerror;

but, after processing the first telegram, I keep getting this error for every next telegram:

Parse error XMX5LGBBLB2410065887
                    ^
                    Duplicate field

I was under the impression that slimmeMeter.clear() would clear (empty) the DSMRdata area .. but it seems not to.

Can/will you please elaborate how to clear DSMRdata in my setup (I need DSMRdata to be globally declared to fix/find some heap problems).

My ISKRA AM550 sends messages like this according to Serial.println(reader.raw())

ISK5\2M550E-1012

1-3:0.2.8(50)
0-0:1.0.0(181006151740S)
0-0:96.1.1(4530303433303037313330393836393138)
1-0:1.8.1(000045.342*kWh)
1-0:1.8.2(000088.857*kWh)
1-0:2.8.1(000000.000*kWh)
1-0:2.8.2(000000.000*kWh)
0-0:96.14.0(0001)
1-0:1.7.0(00.252*kW)
1-0:2.7.0(00.000*kW)
0-0:96.7.21(00008)
0-0:96.7.9(00002)
1-0:99.97.0()
1-0:32.32.0(00005)
1-0:32.36.0(00001)
0-0:96.13.0()
1-0:32.7.0(238.8*V)
1-0:31.7.0(001*A)
1-0:21.7.0(00.258*kW)
1-0:22.7.0(00.000*kW)
0-1:24.1.0(003)
0-1:96.1.0(4730303339303031383134313338343138)
0-1:24.2.1(181006151505S)(00032.834*m3)

The first telegram is parsed correctly like this

identification: ISK5\2M550E-1012
p1_version: 50
timestamp: 181006151740S
equipment_id: 4530303433303037313330393836393138
energy_delivered_tariff1: 45.34kWh
energy_delivered_tariff2: 88.86kWh
energy_returned_tariff1: 0.00kWh
energy_returned_tariff2: 0.00kWh
electricity_tariff: 0001
power_delivered: 0.25kW
power_returned: 0.00kW
electricity_failures: 8
electricity_long_failures: 2
electricity_failure_log: ()
electricity_sags_l1: 5
electricity_swells_l1: 1
message_long: 
voltage_l1: 238.80V
current_l1: 1A
power_delivered_l1: 0.26kW
power_returned_l1: 0.00kW
gas_device_type: 3
gas_equipment_id: 4730303339303031383134313338343138
gas_delivered: 32.83m3

The subsequent telegrams result in "Duplicate field" parse errors pointing to the very first ISK5\2M550E-1012 line. To work around this problem I commended out line 114 and 115 in parser.h. After that the parsing works (most of the times, sometimes there is some obvious junk in the message). However, I see that the identification field is growing in subsequent telegrams like this:

identification: ISK5\2M550E-1012
identification: ISK5\2M550E-1012ISK5\2M550E-1012
identification: ISK5\2M550E-1012ISK5\2M550E-1012ISK5\2M550E-1012

The same happens for

electricity_failure_log: ()
electricity_failure_log: ()()
electricity_failure_log: ()()()

I suspect that this is might be the underlying cause of the "Duplicate field" parse errors, since it seems that the previous values are not cleared.

My C++ template skills are not good enough to figure out why this would happen. Right now my workaround is to ignore the parse errors by keeping line 114 and 155 commented out and by not including identification and electricity_failure_log in the ParsedData template.

Hello,

Can you tell me what the readout (rX) pin is of those sketches?

best regards,
Justinn

The parser implementation tightly depends on Arduino framework.
I want to use the parser on a different microcontroller platform without Arduino libraries.
Also, I would like to be able to run the parser on my development machine for unit testing purposes.

Has anyone tried to do it before? What steps would I need to take to achieve that?

Dag Matthijs,
Heb je een suggestie voor me om de arduino-dsmr
draaien te maken voor de laatst arduino-esp8266 lib?
Alvast bedankt.

Mvg
tthandroid

Hi,

I was testing the Parse example but my code is haning at ParseResult.
ParseResult<void> res = P1Parser::parse(&data, raw, lengthof(raw), true);
I have a print before that line and after but i only get the print before that line.

I use the Arduino Nano, have you any idea's?

Hi Matthijs,

I want to use this library but I get a compile error:

Arduino: 1.8.3 (Mac OS X), Board: "NodeMCU 1.0 (ESP-12E Module), 80 MHz, 115200, 4M (3M SPIFFS)"

../Documents/com~apple~CloudDocs/ArduinoProjects/libraries/arduino-dsmr-master/examples/parse/parse.ino:12:
../Documents/com~apple~CloudDocs/ArduinoProjects/libraries/arduino-dsmr-master/src/dsmr/parser.h:34:24: fatal error: util/crc16.h: No such file or directory
 #include <util/crc16.h>
                        ^
compilation terminated.
exit status 1
Error compiling for board NodeMCU 1.0 (ESP-12E Module).

Can you point me in the right direction on how to obtain this crc16 library??

Hm.. I was under the impression that this library was also for the ESP8266 .. but it seems not :-(
Any change you will make this library compatible for the ESP8266 soon??

your code compiles but when comes to execution it stops at the line:
if (reader.available()) {
which means no reading happens and no data received!

would appreciate your insight what could the problem be? my hardware is Arduino Uno with inverted RX and TX through using :
SoftwareSerial mySerial(8,9,1);
and a 10k resistor to boost the data voltage.
So don't know if the problem is in the hardware or somewhere else!

(insert code does not seem to work.. not with ticks and not with code)
Given this code example:

struct Printer {

template<typename Item>

void apply(Item &i) {

  if (i.present()) {

    Serial.print(Item::name);

    Serial.print(F(": "));

    Serial.print(i.val());

    Serial.print(Item::unit());

    Serial.println();

  }

}

};


.. how do I get the int_val()???
tried everything I can think off ..

i.int_val()


Thanks in advance!

I'm trying to figuring out how to use your library with SoftwareSerial. I have configured it like this:

#include "dsmr.h"
#include <SoftwareSerial.h>
#define SERIAL_RX D5 // pin for SoftwareSerial RX
SoftwareSerial mySerial(SERIAL_RX, -1, true); 

And tried each following:

P1Reader mySerial;
P1Reader reader(mySerial);
P1Reader reader(&mySerial);

Is it posible to put an example sketch in your source for how to configure this?

Hi Matthijs,

Could you please update the corresponding Library on PlatformIO for this repo? I'm getting the reinterpret cast issue that you've fixed with a previous update when using PlatformIO, like with ESPHome.

Thank you in advance and with kind regards,
Thomas

Dear,
The timestamp is not parsed by the sketch, even with the latest library.
With the SerialPassthrough script did i grab attached example data from my Belgian smart meter.
DSMR example.txt

The code for reading date/time is:
// Format YYMMDDHHMMSS (example: 210314123511)
TIMESTAMP = data.timestamp;
// Put timestamp in readable format
TIME = TIMESTAMP.substring(6, 8) + ":" + TIMESTAMP.substring(8, 10) + ":" + TIMESTAMP.substring(10, 12)
+ " " +
TIMESTAMP.substring(4, 6) + "/" + TIMESTAMP.substring(2, 4);

I can't build minimal_parse in de arduino 1.8.19 IDE but also platformio gives the same error. For now i have no clue how to fix this.

error: 'identification' was not declared in this scope

How do I use this :

    /**
     * Returns the data read so far.
     */
    const String &raw() {
      return buffer;
    }

If it should do what I think...
It would be very nice to be able to print the raw format of the telegram!

Thanks for making this!

I spent about an hour staring at messages from the compiler similar to

undefined reference to `dsmr::fields::timestamp::id'

Mostly caused by my inexperience I guess. It seems that for (my installation?) of the Arduino compiler to recognize your code, I need to have dsmr.h and all other files from the src directory in the same directory (and editing dsmr.h to 'not refer to the dsmr directory')

Or is there some other way to make this work?

Hello

I tried to compile the demo code and got the following error:

.../.arduino15/packages/esp8266/tools/xtensa-lx106-elf-gcc/2.5.0-4-b40a506/bin/../lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld: sketch/minimal_parse.ino.cpp.o:(.text._ZN4dsmr10ParsedDataIINS_6fields14identificationENS1_15power_deliveredEEE10parse_lineERKNS_6ObisIdEPKcS9_[dsmr::ParsedData<dsmr::fields::identification, dsmr::fields::power_delivered>::parse_line(dsmr::ObisId const&, char const*, char const*)]+0x4): undefined reference to dsmr::fields::identification::id'`

Am I doing something wrong? If so, can you please point me in the right direction?

Hi, I want to bring a bug to your attention. I found this while fuzzing the library. The issue appears when an invalid input with more than 3 'decimals' is given.

The issue is that the max_decimals value is post-decremented, which means that, if the max_decimals value hits 0, the loop stops, but the max_decimals value is then wrapped around to the max uint value. This will trigger a very long process that will decrement the max_decimals value back to 0.

An example is shown below. Note that, for fuzzing purposes, I've disabled checksum checking in the build that generated this. To provoke the actual behavior, you might need to update the checksum/disable checksum checking.

/Ene5\T210-D ESMR5.0

1-3:0.2.8(50)
0-0:1.0.0(200112213629W)
0-0:96.1.1(3330300333833330303332323639333333)
1-0:1.8.1(.00385.6
00900)
0-1*m3)
!B1C3

/Ene5\T210-D ESMR5.0

1-3:0.2.8(50)
0-0:1.0.0(200112213629W)
0-0:96.1.1(3330300333833330303332323639333333)
1-0:1.8.1(0.0385.6
00900)
0-1*m3)
!B1C3

By changing line 201 to the following, it can be fixed.

-      while(num_end < end && !strchr("*)", *num_end) && max_decimals--) {
+      while(num_end < end && !strchr("*)", *num_end) && max_decimals) {
+        --max_decimals;

Because I wanted to fuzz on a x86-64 target, I've also updated the code to support running on linux. This might have helped triggering this bug, but I'm not sure about that.

I could share the adapted version, but it's mostly the same with basically all String usage changed with std::string.

After applying this fix, the code correctly reports an error in the data instead of hanging/looping for a very long time.

1-0:1.8.1(.00385.6
              ^
Missing unit

I'm getting a lot of errors when I try to use the read example.

Note that I'm using an Arduino UNO so I'm using Software Serial.

This is what I have edited in the file:
#include <SoftwareSerial.h>
SoftwareSerial mySerial(8,9); // RX, TX
mySerial.begin(115200);

Hi,

I have a new gasmeter that is not installed on MBUS channel 1, but on channel 2. As a result I did not get any gasmeter values.

Is it possible to add some sort of configuration to the library so users of this lib don't need to make changes in the library to get, for example, gasmeter data from the correct channel?

In other words... Make is easier to change the following values without the need to make library changes:

const uint8_t GAS_MBUS_ID = 1;
const uint8_t WATER_MBUS_ID = 2;
const uint8_t THERMAL_MBUS_ID = 3;
const uint8_t SLAVE_MBUS_ID = 4;

In the end a solution would be a nice addition for this project:
https://github.com/mrWheel/DSMRlogger2HTTP

Your lib works like a charm, i have been using it for a while.
Today I made a slight modification to my sketch and I realised that it does not compile anymore with the newest ESP package.
I worked around the issue by downgrading, but I thought to notify you...

Hello!

I bought a Slimmelezer+ from Marcel Zuidwijk and it seems to use this parser in the firmware built upon ESPHome. It connects to the electricity meter just fine but can't read the data:

19:04:32 | [E] | [dsmr:265] | /ADN9 6534  
^ 
Invalid identification string

Now this seems to be because of this line in this library:

// The first identification line looks like:
// XXX5<id string>
// The DSMR spec is vague on details, but in 62056-21, the X's
// are a three-leter (registerd) manufacturer ID, the id
// string is up to 16 chars of arbitrary characters and the
// '5' is a baud rate indication. 5 apparently means 9600,
// which DSMR 3.x and below used. It seems that DSMR 2.x
// passed '3' here (which is mandatory for "mode D"
// communication according to 62956-21), so we also allow
// that.
if (line_start + 3 >= line_end || (line_start[3] != '5' && line_start[3] != '3'))
return res.fail(F("Invalid identification string"), line_start);

I.e. unless the fourth char (after the /, which is read before) is 3 or 5, the lib stops reading data. But, that fourth char is the baud rate according to IEC62056-21 (and your comment) and could be many things. In Sweden the standard is 115200 which is the number 9. My Aidon meter spits out data like this:

/ADN9 6534

0-0:1.0.0(213112235959W)
1-0:1.8.0(12345678.123*kWh)
1-0:2.8.0(12345678.123*kWh)
1-0:3.8.0(12345678.123*kVarh) 
....
1-0:71.7.0(123.1*A)!

It seems like this check is far too strict? Baud rate can be left to ESPHome or whatever other device uses this lib? It looks like without this header verification the device could parse "my" format as well, and 9 seems to be a valid number according to the spec.

Hi,

I'm trying to use this library in combination with a Wemos d1 mini v3 and hardwareserial. I live in Belgium, and got a sagemcom t211 installed. I'm aware that Belgium uses a slightly different OBIS syntax, but this is not the problem.

I have managed to connect and get a read by applying this scheme from silvanverschuur , together with his script (i.e. a manual parsing of the telegram). However, given the several benefits from using your library, I'd like to start using it. I started off by using the read.ino from the examples folder.

I have tried a lot of setups to connect the Wemos to the P1 port, but can't seem to get past the if(reader.available()) statement: it never becomes true. My gut feeling tells me that it is due to the lack of the request pin, but I can't seem to find out how to properly connect it. The most simple solution, making a direct connection from P1's pin 3 to Wemos pin D2, isn't doing the trick.

I don't know how to proceed from this point on, and my stubbornness is already causing quite some sleep deprivation. Any help on this matter would be incredibly appreciated.

Edit

Oh my, apparently there is a difference in pin numbers between the Wemos d1 mini and the Arduino IDE: https://chewett.co.uk/blog/1066/pin-numbering-for-wemos-d1-mini-esp8266/

I now have to try everything again, and will report back. In the mean time, please don't hesistate if you have suggestions.

Thank you in advance,
Erwin

I may have an older unit, as seems the lib does not properly parse my messages

see here what my meter outputs:

/KMP5 ZABF000000000000
0-0:96.1.1(205C4D246333034353537383234323121)
1-0:1.8.1(00185.110*kWh)
1-0:1.8.2(00084.020*kWh)
1-0:2.8.1(00013.030*kWh)
1-0:2.8.2(00019.040*kWh)
0-0:96.14.0(0001)
1-0:1.7.0(0000.98*kW)
1-0:2.7.0(0000.03*kW)
0-0:17.0.0(999*A)
0-0:96.3.10(1)
0-0:96.13.1()
0-0:96.13.0()
0-1:24.1.0(3)
0-1:96.1.0(3238313031453631373038389930337131)
0-1:24.3.0(120517020000)(08)(60)(1)(0-1:24.2.1)(m3)
(00124.477)
0-1:24.4.0(1)
!

The most obvious missing from my meter is the CRC...
The other parsing issues:

0-0:17.0.0(999*A)
               ^
Invalid unit

and

(00124.477)
^
OBIS id Empty

I worked around the CRC issue, but the gasmeter value on separate line seems more tricky

Trying to get this library to work on my Arduino Wifi Rev2 with above-mentioned environment version.

I get compile errors mentioning incompatible types 'char *' and '__FlashStringHelper *'.

Could you point me in the direction of a solution?

Is it not a good idea to rename the gas_, thermal_ and water_ fields to mbus1_, mbus2_ and mbus3_ respectively?

People can than dynamically name these fields in there programs depending on the device connected.

I think it would solve lots of the confusion as it would not assume the type of device connected.

Maybe you could even supply a function to “connect” for instance “gas” to “mbus3” and so on ...

This issue was reported to esphome at esphome/issues#3886. The T211 meter is used by Enexis on 3-phase service. When no gas meter is present, or when it is not linked yet by the service provider, the meter reports a gas reading without m3 unit. This causes the library to throw a 'missing unit' error, which should probably be handled more gracefully.

Example:

0-1:24.2.1(632525252525S)(00000.000)
          ^
Missing unit

The timestamp of the gas reading in this example line also seems to be a placeholder. Possibly this identifier (632525252525S) can be used to detect a gas meter that is not properly linked yet, allowing an empty value without unit to be returned.

Example correct line:

0-1:24.2.1(xxxx25070000S)(14336.324*m3)

This is where the error originates: https://github.com/matthijskooijman/arduino-dsmr/blob/master/src/dsmr/parser.h#L219

The rest of the datagram is perfectly valid, so we most likely don't want to error the entire parse action, causing no data to be returned at all.

Suggestion: Change TimestampedFixedField to handle the 632525252525S timestamp, and pass an empty unit to the FixedField
-OR-
As the value 0 is dimensionless, allow a missing unit for a 0 value

Hi,

Since yesterday I'm the happy owner of the Slimme Meter uitlezer (https://opencircuit.nl/Blog/Slimme-meter-uitlezer). It works perfectly for electricity, but for some reason the gas values (usage and equipment id) of my telegram are rendered to 0 m3 and and unknown equipment ID.

Willem, the creator of the Slimme Meter uitlezer, advised me to make changes to your lib or contact the creator of the DSMR parser lib. Maybe my meter is sending other info than other meters... Since I have no clue where to start looking I hope that the brains behind this lib has an idea why my gas meter values are not detected :)

My telegram look lik this:

/ISK5\2M550E-1012                                                               
                                                                                
1-3:0.2.8(50)                                                                   
0-0:1.0.0(181202194148W)                                                        
0-0:96.1.1(453030***************9323137)                                  
1-0:1.8.1(000617.195*kWh)                                                       
1-0:1.8.2(000730.512*kWh)                                                       
1-0:2.8.1(000000.000*kWh)                                                       
1-0:2.8.2(000000.000*kWh)                                                       
0-0:96.14.0(0001)                                                               
1-0:1.7.0(00.192*kW)                                                            
1-0:2.7.0(00.000*kW)                                                            
0-0:96.7.21(00009)                                                              
0-0:96.7.9(00004)                                                               
1-0:99.97.0(2)(0-0:96.7.19)(180116115413W)(0000002996*s)(181103104217W)(00000042
71*s)                                                                           
1-0:32.32.0(00009)                                                              
1-0:32.36.0(00001)                                                              
0-0:96.13.0()                                                                   
1-0:32.7.0(225.3*V)                                                             
1-0:31.7.0(000*A)                                                               
1-0:21.7.0(00.193*kW)                                                           
1-0:22.7.0(00.000*kW)                                                           
0-2:24.1.0(003)                                                                                                                                       
0-2:96.1.0(47303033**************31333137)                                  
0-2:24.2.1(181202194007W)(00283.687*m3)                                         
!9F30    

Do you have any idea why I get an empty value for the equipment id and the gas usage?

Looking forward to your reaction.

Matthijs,

I can see this is a very neat and smart written library, but the whole concept of "templates" is far beyond my "C++" knowledge.

Can (will) you please explain (preferable with a short piece of code) how I can obtain certain values?

For example:
How can I obtain the value of "timestamp" or "energy_delivered_tariff1"?

My solution "so far" is:

String pdel1Strng;

struct collectValues {
    template<typename Item>
    void apply(Item &i) {
      if (i.present()) {
        String ItemName = String(Item::name);
        if (ItemName == "power_delivered_l1") {
          pdel1Strng = String(i.val());
        }
    }
}  // collectValue()
.
.
  pdel1 += pdel1Strng.toFloat();
  Serial.print("Total : ");
  Serial.print(pdel1);
  Serial.println("kWh");

Thanks!

Hi, thanks for this awesome library! Works great!

I have an issue accessing the timestamp corresponding to the gas_delivered value (TimestampedFixedValue type). The readme states that there is an additional timestamp() function which would return a timestamp string. The compiler however throws an error:

DSMR:376:65: error: no match for call to '(String) ()'
       mqttGasStruct["timestamp"] = data.gas_delivered.timestamp();
exit status 1
no match for call to '(String) ()'

I am not that familiar with the C++ template structures used, but I cannot find any function named timestamp() in fields.h. I can directly access gas_delivered.timestamp (the variable, not the function), but that is not how it is documented and it seems that this value is not updated after receiving new telegrams.

Am I doing something wrong? Unfortunately, the examples do not cover these TimestampedFixedValue timestamps.

Hi,
I'm trying to include the library in a project, but I seem unable to make it work.
I'm using platormio with an ESP32 dev-board.
This is my platformio.ini :

[env:az-delivery-devkit-v4]
platform = espressif32
board = az-delivery-devkit-v4
framework = arduino
monitor_speed = 115200
monitor_port = /dev/ttyUSB0
upload_port = /dev/ttyUSB0
lib_deps = 
 https://github.com/matthijskooijman/arduino-dsmr.git

I copied the reader example to my main.cpp, modified the HWserial and compiled. That worked fine, so I flashed my board with it.
But when I send the data from the parse example to the device, via an extra 3.3V USB-to-serial adapter, nothing happens.
Is there a way to test what's happening ?
I made a test-application to read from HWserial and transmit the data to Serial, even "detecting" the end of the telegram and that works, so the hardware is OK.

P.S. this is my modification for HWserial, because if I leave the original text I get : multiple definition of `Serial1'

#ifdef ARDUINO_ARCH_ESP32
// Create HWSerial connected to UART 2
HardwareSerial HWSerial(2);
#endif
P1Reader reader(&HWSerial, 2);

and

  Serial.begin(115200);
  HWSerial.begin(115200);