volkszaehler / vzlogger Goto Github PK

Reason is this line: https://github.com/volkszaehler/vzlogger/blame/master/src/vzlogger.cpp#L248 any idea why that is?

If the local(1) call is removed, vzlogger no longer daemonizes, just as expected. Only now, the following error starts appearing:

error: unknown type in sml_value_to_double

Something is very wrong here...

We already have a log file parser which is essentially:

• a file transport to read the file line by line plus
• a pattern matcher for that line (which seems to be broken but alas)

A JSON logger would consist of:

• an HTTP(S) transport
• a JSON parser
• a search expression

Logfile and JSON parser might/should use the same infrastructure and config settings in that regard.

Features I would be looking for are:

• search expression for target value:
  • properties: parent.child
  • arrays: parent[1]
  • arrays with access from end: parent[-1]
  • child property matching for arrays: parent.channels[@uuid=abc]
    • option to skip duplicates if found value == last value

The latter option might also make general sense for other meters.

What do you think?

Compiling vzlogger from source according to http://wiki.volkszaehler.org/software/controller/vzlogger/installation_cpp-version#building_from_source fails for me on openSUSE 13.2 (i586) with the following error:

linux-xlde:~/vzlogger/vzlogger # cmake .
-- The C compiler identification is GNU 4.8.3
-- The CXX compiler identification is GNU 4.8.3
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
Compiling for target ''
-- checking if -Wno-ignored-qualifiers works
-- Performing Test W_NO_IGNORED_QUALIFIERS
-- Performing Test W_NO_IGNORED_QUALIFIERS - Success
-- FindSml check
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.28")
-- checking for module 'sml>=0.9'
-- package 'sml>=0.9' not found
-- SML_HOME env is not set, setting it to /usr/local
-- Looking for sml in /usr/local
-- FindMicrohttpd check
-- checking for module 'microhttpd>=0.9'
-- package 'microhttpd>=0.9' not found
-- MICROHTTPD_HOME env is not set, setting it to /usr/local
-- Looking for microhttpd in /usr/local
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- FindJson check
-- checking for module 'json>=0.9'
-- found json, version 0.10
-- JSON_HOME env is not set, setting it to /usr/local
-- Looking for json in /usr/local
Jsoon search: '/usr/local/include;/usr/local/include;/usr/local/include;/usr/include'
CMake Error at CMakeLists.txt:85 (message):
libjson ist required.

Install json or call cmake -DJSON_HOME=path_to_json_install

-- Configuring incomplete, errors occurred!
See also "/root/vzlogger/vzlogger/CMakeFiles/CMakeOutput.log".
See also "/root/vzlogger/vzlogger/CMakeFiles/CMakeError.log".

using autogen.sh, configure.sh, make works - at least with the source before commit, 9a3a8c5 - as that one seems to introduce the following problem:

g++ -g -O2 -lpthread -lm -lstdc++ -ljson -lcurl -lssl -lcrypto -o vzlogger vzlogger.o Channel.o Config_Options.o threads.o Buffer.o Meter.o ltqnorm.o Obis.o Options.o Reading.o exception.o local.o MeterMap.o MeterS0.o MeterD0.o MeterSML.o MeterFluksoV2.o MeterFile.o MeterExec.o MeterRandom.o Volkszaehler.o MySmartGrid.o CurlIF.o CurlCallback.o CurlResponse.o -lsml -luuid -lmicrohttpd
threads.o: In function logging_thread(void*)': /root/vzlogger/vzlogger/src/threads.cpp:197: undefined reference tovz::api::Null::Null(std::tr1::shared_ptr, std::list<Option, std::allocator >)'
MeterMap.o: In function MeterMap::registration()': /root/vzlogger/vzlogger/src/MeterMap.cpp:132: undefined reference tovz::api::Null::Null(std::tr1::shared_ptr, std::list<Option, std::allocator >)'

when adding calculated instant power readings to the output of the s0 meter,
it was made to consume two pulses in MeterS0::read to calculate power.
this is not necessary, we can simply remember the timestamp of the previous reading's pulse.

Solved as part of #110, this is just a reminder.

Ich nutze den YPORT+ Node (mit Erweiterung) von Udo um meinen Stromzähler mit dem USB-Lesekopf und den Gaszähler mit S0 auszuwerten.
Der YPORT+ läuft unter OpenWrt, vzlogger übersetze ich mit der OpenWrt Buildroot. Das hat bisher problemlos funktioniert. Seit kurzem bricht der Übersetzungsprozess mit Fehlern ab.

Compile Log:

vzlogger.o: In function config_parse_cli(int, char**, Config_Options*)': vzlogger.cpp:(.text+0x394): undefined reference tog_GIT_SHALONG'
threads.o: In function logging_thread(void*)': threads.cpp:(.text+0x6f8): undefined reference tovz::api::Null::Null(std::tr1::shared_ptr, std::list<Option, std::allocator >)'
MeterMap.o: In function MeterMap::registration()': MeterMap.cpp:(.text+0x362): undefined reference tovz::api::Null::Null(std::tr1::shared_ptr, std::list<Option, std::allocator >)'
collect2: error: ld returned 1 exit status
make[5]: *** [vzlogger] Error 1
make[5]: Leaving directory /home/dt/udoopenwrt/build_dir/target-mips_34kc_uClibc-0.9.33.2/vzlogger-1d272282c0948c329d54fd705ab189ca60d4c4d3/src' make[4]: *** [all-recursive] Error 1 make[4]: Leaving directory/home/dt/udoopenwrt/build_dir/target-mips_34kc_uClibc-0.9.33.2/vzlogger-1d272282c0948c329d54fd705ab189ca60d4c4d3'
make[3]: *** [all] Error 2
make[3]: Leaving directory /home/dt/udoopenwrt/build_dir/target-mips_34kc_uClibc-0.9.33.2/vzlogger-1d272282c0948c329d54fd705ab189ca60d4c4d3' make[2]: *** [/home/dt/udoopenwrt/build_dir/target-mips_34kc_uClibc-0.9.33.2/vzlogger-1d272282c0948c329d54fd705ab189ca60d4c4d3/.built] Error 2 make[2]: Leaving directory/home/dt/custom-feed/vzlogger'
make[1]: *** [package/feeds/custom/vzlogger/compile] Error 2
make[1]: Leaving directory `/home/dt/udoopenwrt'
make: *** [package/vzlogger/compile] Fehler 2

Es hängt anscheinend an folgenden Änderungen:
9a3a8c: Added null api for not pushing readings to any con...
3e59e4: add git version info

Das compilieren geht, wenn ich:
a) das null api in threads.cpp und MeterMap.cpp auskommentire
b) in vzlogger.cpp das # include gitSha1.h und g_GIT_SHALONG auskommentiere

See #88 for details. Issues will occur if middleware returns strange or partial results/error codes.

E.g. the version info is missing in the log file. There should always be a "vzlogger V... based on ... from ... started" within each start visible. But it's not.

This issue applies to Git commit d16c0c4 of the source (latest as of 20141228T1118Z):
If I query a vzlogger instance compiled with microhttpd support for the current meter values I get something like follows:

{ "version": "0.4.0", "generator": "vzlogger", "data": [ { "uuid": "72b84f00-78cf-11e4-828e-b5e03ccbd224", "last": 1419764667.134391, "interval": -1, "protocol": "sml" }, { "uuid": "442811a0-7882-11e4-8cbb-4fd2114e6edc", "last": 1419764667.134441, "interval": -1, "protocol": "sml" }, { "uuid": "8d571aa0-7882-11e4-9755-1fd3ed3c5076", "last": 1419764667.134452, "interval": -1, "protocol": "sml" }, { "uuid": "9468cfb0-7882-11e4-8021-c54e0227b4a7", "last": 1419764667.134461, "interval": -1, "protocol": "sml" }, { "uuid": "d0bfd3e0-7882-11e4-8eb8-c32561418f40", "last": 1419764667.134432, "interval": -1, "protocol": "sml" } ] }

The above includes

• UUID of the channel,
• timestamp of the reading/accumulation,
• interval (-1 in this case as I don't use interval, but I use aggtime/aggfixedinterval instead), and
• protocol pertaining to the current channel.

Note that the current value of the channel's reading is missing -- which is exactly what I expect when querying the internal web server.

Even if I explicitly query a specific channel only, I don't get the current value:

$ curl localhost:8080/442811a0-7882-11e4-8cbb-4fd2114e6edc
{ "version": "0.4.0", "generator": "vzlogger", "data": [ { "uuid": "442811a0-7882-11e4-8cbb-4fd2114e6edc", "last": 1419766166.622873, "interval": -1, "protocol": "sml" } ] }

Hi,

vzlogger claims about a Duplicate Primary Key Entry. See http://pastebin.com/v31nLdVR how this can be solved?

Very often the http response result is filled up with outdated data. It is possible to return only current values every time?

Example response:

{ 
"version": "0.4.0", 
"generator": "vzlogger", 
"data": [ { 
  "uuid": "1",
   "last": 1430649171260, 
  "interval": -1, 
  "protocol": "d0", 
  "tuples": [
    [ 1430649169437, 6172.3514360999998 ], 
    [ 1430508780497, 6162.6050561000002 ], 
    [ 1430508782456, 6162.6053058999996 ], 
    [ 1430508784456, 6162.6055554000004 ], 
    [ 1430508786456, 6162.6058050000001 ], 
    [ 1430508788456, 6162.6060545999999 ], [ 1430508790457, 6162.6063045000001 ], [ 1430508792457, 6162.6065540999998 ], [ 1430508794456, 6162.6068039000002 ], [ 1430508796456, 6162.6070541999998 ], [ 1430508798455, 6162.6073046000001 ], [ 1430508800456, 6162.6075546000002 ], [ 1430508802456, 6162.6078045000004 ], [ 1430508804455, 6162.6080540000003 ], [ 1430508806456, 6162.6083036 ], [ 1430508808456, 6162.6085528000003 ], [ 1430508810455, 6162.6088018 ], [ 1430508812456, 6162.6090514999996 ], [ 1430508814456, 6162.6093012000001 ], [ 1430508816455, 6162.6095512000002 ], [ 1430508818456, 6162.6098011000004 ], [ 1430508820455, 6162.6100509999997 ], [ 1430508822455, 6162.6103018000003 ], [ 1430508824454, 6162.6105519000002 ], [ 1430508826455, 6162.6108021 ], [ 1430508828454, 6162.6110521999999 ], [ 1430508830455, 6162.6113025000004 ], [ 1430508832455, 6162.6115530999996 ], [ 1430508834454, 6162.6118036999997 ], [ 1430508836455, 6162.6120540000002 ], [ 1430508838454, 6162.6123049999997 ], [ 1430508840455, 6162.6125554999999 ], [ 1430508842455, 6162.6128054000001 ], [ 1430508844454, 6162.6130549999998 ], [ 1430508846455, 6162.6133053000003 ], [ 1430508848455, 6162.6135553000004 ], [ 1430508850454, 6162.6138054000003 ], [ 1430508852455, 6162.6140552999996 ], [ 1430508854455, 6162.6143052999996 ], [ 1430508856454, 6162.6145556000001 ], [ 1430508858454, 6162.6148058999997 ], [ 1430508860454, 6162.6150556000002 ], [ 1430508862454, 6162.6153056000003 ], [ 1430508864454, 6162.6155560999996 ], [ 1430508866454, 6162.6158059999998 ], [ 1430508868454, 6162.6160559999998 ], [ 1430508870454, 6162.6163060999997 ], [ 1430508872454, 6162.6165565000001 ], [ 1430508874454, 6162.6168067999997 ], [ 1430508876454, 6162.6170566000001 ], [ 1430508878454, 6162.6173061 ], [ 1430508880453, 6162.6175549 ], [ 1430508882453, 6162.6178006999999 ], [ 1430508884454, 6162.6180455000003 ], [ 1430508886453, 6162.6182901000002 ], [ 1430508888454, 6162.6185353000001 ], [ 1430508890453, 6162.6187836999998 ], [ 1430508892453, 6162.6190306999997 ], [ 1430508894453, 6162.6192758999996 ], [ 1430508896454, 6162.6195212000002 ], [ 1430508898454, 6162.6197668000004 ], [ 1430508900453, 6162.6200122999999 ], [ 1430508902454, 6162.6202567 ], [ 1430508904453, 6162.6205011000002 ], [ 1430508906453, 6162.6207459999996 ], [ 1430508908453, 6162.6209912000004 ], [ 1430508910452, 6162.6212361999997 ], [ 1430508912453, 6162.6214806999997 ], [ 1430508914453, 6162.6217255000001 ], [ 1430508916452, 6162.6219705000003 ], [ 1430508918452, 6162.6222154999996 ], [ 1430508920453, 6162.6224602000002 ], [ 1430508922453, 6162.6227050999996 ], [ 1430508924452, 6162.6229499999999 ], [ 1430508926453, 6162.6231951999998 ], [ 1430508928453, 6162.6234396 ], [ 1430508930452, 6162.6236841999998 ], [ 1430508932452, 6162.6239292 ], [ 1430508934453, 6162.6241742000002 ], [ 1430508936452, 6162.6244189999998 ], [ 1430508938452, 6162.6246677999998 ], [ 1430508940452, 6162.6249303000004 ], [ 1430508942452, 6162.6252096999997 ], [ 1430508944452, 6162.6254898999996 ], [ 1430508946452, 6162.6257691000001 ], [ 1430508948452, 6162.6260492000001 ], [ 1430508950452, 6162.6263299000002 ], [ 1430508952452, 6162.6266101000001 ], [ 1430508954451, 6162.6268842999998 ], [ 1430508956452, 6162.6271543000003 ], [ 1430508958452, 6162.6274241000001 ], [ 1430508960451, 6162.6276928999996 ], [ 1430508962452, 6162.6279610000001 ], [ 1430508964452, 6162.6282288000002 ], [ 1430508966451, 6162.6284967000001 ], [ 1430508968452, 6162.6287647999998 ], [ 1430508970451, 6162.6290324000001 ], [ 1430508972451, 6162.6293015000001 ], [ 1430508974452, 6162.6295726999997 ], [ 1430508976451, 6162.6298446000001 ], [ 1430508978451, 6162.6301160000003 ], [ 1430508980452, 6162.6303870000002 ], ...

My configuration:

{
"retry" : 10,
"daemon": false,
"verbosity" : 5,
"log" : "/home/pi/vzlogger.log",

"local" : {
        "enabled" : true,
        "port" : 8080,
        "index" : true,
        "timeout" : 0,
        "buffer" : 3
},

"meters" : [{
        "enabled" : true,
        "protocol" : "d0",
        "device" : "/dev/ttyUSB0",
        "channels": [
        {
                "uuid": "1",
                "identifier": "1-0:1.8.0"
        },
        {
                "uuid": "2",
                "identifier": "1-0:2.8.0"
        },
        {
                "uuid": "3",
                "identifier": "1-0:1.7.0"
        }
        ]
}
]}

previously, it was possible to configure vzlogger to run continously, but not send readings to an api.
the feature was probably broken before, and removed entirely in #40 .
this feature was useful when the "local"-mode internal webserver was being used.

this was achieved by setting daemon: false in the configuration.

the feature has been mewhat re-instantiated by adding the "Null" api: #70

• do we want to restore this feature or not?
• in either case, update any documentation

• pointless
• bad practive to release
  • without discussion
  • with open issues

Need to check what happens if the daemonize() is omitted- does everything still work?

any signalling only results in
"[<timestamp>] Closing connections to terminate"

this behaviour is produced with the current master and the example etc/vzlogger.conf,
which has all meters disabled.

According to vzlogger.conf:

"aggmode: "AVG", /* add all s0 intervals in the aggregation. Possible Modes: SUM, AVG, MAXIMUM and NONE*/

However, with MAXIMUM vzlogger complains on startup.

Just a reminder...

Do we have a definition of the log levels anywhere?

@justinotherguy any idea what that is? Obsolete/ remove?

As more and more people struggle with the complexity of the configuration file I suggest to add a config-file editor like linux kernel "make menuconfig", raspi-config, opensips/makemenuconfig,...

The editor should support:

• taking the etc/vzlogger.conf as a template (incl. help text from comments)
• asking the customer which options to choose in a structure way (global options, adding meters? (add->d0->ask options)
• writing only the needed/configured options into a conf file
• reading of existing conf files and using this as preset input
• ...

Using this approach we could make the etc/vzlogger.conf much more complex/unreadable by adding more detailed comments, descriptions,... (instead of trying to keep the documentation in sync in the wiki or source or conf file...) and generate a specific config file easy to read, maintain and understand.

What do you think about it? suggestions/comments welcome/requested.

Is anybody aware of such an editor or some other tool/source to start with? (opensips/makemenuconfig might be a starting point providing a simple ncurses based implementation, json parsing/stuff to be added,...)

WIP here: https://github.com/r00t-/vzlogger/tree/raspis0
see also: "request for comments: s0 meter, supporting different interfaces" #85

Aus der ML:

2015-01-02 15:09 GMT+01:00 Martin Heinze [email protected]:

Gerade bei Zählerständen ist es nicht wirklich notwendig zu jeder Auslesung einen Wert in der DB zu speichern

Sehe ich auch so, ist allerdings vom Anwendungfall abhängig.

Es sollte relativ einfach sein einen Konfigparameter in der Art von ignoreDuplicateReadings oder besser minReadingDelta (flexible) zu bauen den der vzlogger mit dem vorhergegenden Absolut(!)wert vergleicht und nur schreibt wenn größer. Damit kann man sich Roundtrips in die Middleware/DB ersparen.

Since the introduction of #117 (curl single session) the shutdown is sometimes not properly working, i.e. a hang occurs and process needs to be killed with -9.
This is due to pthread_mutex_lock functions are no thread cancellation points. If one thread gets killed that owns the mutex it doesn't get released. So the other threads still keep on waiting for the mutex but as pthread_mutex_lock is no cancellation point it never gets killed.

(I'll provide a fix using either robust mutexes or using _timed_wait and pthread_testcancel.)

see also #105

https://github.com/volkszaehler/vzlogger/blob/master/debian/

i´m getting several of these compile warnings (below). it seems, with more recent json libs one should now use json_object_object_get_ex() instead

according to the json changelog, this was introduced with 0.10

https://github.com/json-c/json-c/blob/master/ChangeLog:

0.10
Add json_object_object_get_ex(), a NULL-safe get object method, to be able
to distinguish between a key not present and the value being NULL.

since libjson 0.10 release is dated 2,5yrs back, i´d curious if it`s a problem to change this and bump vzlogger libjson requirement from 0.9 to 0.10

api/MySmartGrid.cpp: In member function ‘void vz::api::MySmartGrid::api_parse_exception(char*, size_t)’:
api/MySmartGrid.cpp:335:34: warning: ‘json_object* json_object_object_get(json_object*, const char*)’ is deprecated (declared at /usr/include/json/json_object.h:217) [-Wdeprecated-declarations]
   struct json_object *json_obj = json_object_object_get(json_obj_in, "exception");
                  ^
api/MySmartGrid.cpp:335:81: warning: ‘json_object* json_object_object_get(json_object*, const char*)’ is deprecated (declared at /usr/include/json/json_object.h:217) [-Wdeprecated-declarations]
   struct json_object *json_obj = json_object_object_get(json_obj_in, "exception");
                                         ^
api/MySmartGrid.cpp:339:32: warning: ‘json_object* json_object_object_get(json_object*, const char*)’ is deprecated (declared at /usr/include/json/json_object.h:217) [-Wdeprecated-declarations]
     json_object_get_string(json_object_object_get(json_obj,  "type")),
                ^
api/MySmartGrid.cpp:339:72: warning: ‘json_object* json_object_object_get(json_object*, const char*)’ is deprecated (declared at /usr/include/json/json_object.h:217) [-Wdeprecated-declarations]
     json_object_get_string(json_object_object_get(json_obj,  "type")),
                                    ^
api/MySmartGrid.cpp:340:32: warning: ‘json_object* json_object_object_get(json_object*, const char*)’ is deprecated (declared at /usr/include/json/json_object.h:217) [-Wdeprecated-declarations]
     json_object_get_string(json_object_object_get(json_obj,  "message"))
                ^
api/MySmartGrid.cpp:340:75: warning: ‘json_object* json_object_object_get(json_object*, const char*)’ is deprecated (declared at /usr/include/json/json_object.h:217) [-Wdeprecated-declarations]
     json_object_get_string(json_object_object_get(json_obj,  "message"))
                                       ^
api/MySmartGrid.cpp:343:15: warning: ‘json_object* json_object_object_get(json_object*, const char*)’ is deprecated (declared at /usr/include/json/json_object.h:217) [-Wdeprecated-declarations]
    json_obj = json_object_object_get(json_obj_in, "response");
           ^
api/MySmartGrid.cpp:343:61: warning: ‘json_object* json_object_object_get(json_object*, const char*)’ is deprecated (declared at /usr/include/json/json_object.h:217) [-Wdeprecated-declarations]
    json_obj = json_object_object_get(json_obj_in, "response");
                                 ^

linux-xlde:~/vzlogger/vzlogger # rpm -q --list libjson-devel-0.10-3.6.1.i586
/usr/include/json
/usr/include/json/arraylist.h
/usr/include/json/bits.h
/usr/include/json/debug.h
/usr/include/json/json.h
/usr/include/json/json_config.h
/usr/include/json/json_inttypes.h
/usr/include/json/json_object.h
/usr/include/json/json_object_iterator.h
/usr/include/json/json_object_private.h
/usr/include/json/json_tokener.h
/usr/include/json/json_util.h
/usr/include/json/linkhash.h
/usr/include/json/printbuf.h
/usr/include/json/random_seed.h
/usr/lib/libjson.so
/usr/lib/pkgconfig/json.pc

vzlogger -v shows commit hash. Unfortunately commit hash does not give direct indication how old the build or rather the commit is.

Would it be possible to add the commit date as well?

silly bug that i debugged two years ago, still not fixed.
http://volkszaehler.org/pipermail/volkszaehler-dev/2012-December/002244.html

previously, it was possible to configure vzlogger to only get a single reading from each meter and then terminate.
the feature was probably broken before, and removed entirely in #40 .
this feature is also still advertised in the documentation: "can run as a daemon in background or via cron." (wiki and manpage)
(with a cron invocation, one could get meter readings at fixed intervalls without having vzlogger running continously)

this was achieved by setting daemon: false (and local: off, setting for foreground: not relevant) in the configuration.

this only really makes sense with getting absolute energy readings (or maybe a meter that internally averages power over the timeframe since the last read. it's completely useless for S0 meters for example.

• do we want to restore this feature or not?
• in either case, update any documentation

the current aggmode=AVG simply calculates the average of the readings in the buffer.
this is only correct if the readings are all valid for identical time periods.

if we assume power readings from an s0-meter,
we get less readings/time while power is lower, and more while it's higher (because we get more pulses),
thus the averaging will be biased towards higher power if the aggregation period contains periods of both low and high power.

(assume we aggregate over 5 minutes... for the first four minutes, we have very low usage, one impulse per minute, for the rest we have higher usage, one impulse per second.
now have have four readings of the low power value, but sixty of the high value,
a distribution of 4:60, while the actual distribution was 4:1. leading to an incorrect high average.)

the same would be true for other meter types, if the time between readings was not constant.
(this might also affect tools like vzcompress)

current workaround would be not to use the Power readings from s0 (with aggregation),
but to use Impulse readings (and aggmode=SUM) instead.

    {
     "enabled" : true,
     "protocol" : "file",
     "path" : "/sys/class/thermal/thermal_zone0/temp",
     "format" : "$v",        /* a format string for parsing complex logfiles */
                             /* arbitrary text and whitespaces are  allowed, see 'scanf()' */
                             /* at least $v has to be used */
                             /* $i => identifier, $v => value, $t => timestamp */
     "rewind" : true,        /* reset file pointer each interval to  the beginning of the file */
     "interval" : 2,         /* of ommitted, we will try to listen on changes with inotify */
     "identifier" : "dummy",
     "channel" : {
             "uuid" : "f6ff48d0-aeac-11e3-b2ed-3bbb8f66a68d",
             "middleware" : "http://localhost/middleware.php"
             }
     } // meter

result:
[Mar 18 16:20:06][] MeterFile::read: 32, 32
[Mar 18 16:20:06][] MeterFile::read: '32552'
[Mar 18 16:20:06][] MeterFile::read: -0.000000, ▒, 0
[Mar 18 16:20:06][mtr1] Got 1 new readings from meter:
[Mar 18 16:20:06][mtr1] Reading: id=▒/StringItentifier: value=-0.00 ts=0.000
[Mar 18 16:20:06][chn1] ==> number of tuples: 0

The file contains "32552".

I expected that $v would contain "32552".

I have "sml" meters. I don't want to send a measurement every time the meter sends one so i configured a interval of 10 seconds. The vzlogger then sends a measurement every 10 seconds to the middleware... everything seems fine ...

The Problem now is that the shown value in the frontent is too low (i calculated a factor of 6) compared to the real time reading on the meter.

I think this factor is somewhat in the interval the meter sends values to vzlogger (something around 2 seconds) and the configured interval of 10. With an interval of 20 seconds the factor is 12.

With no interval set in the configuration of vzlogger the values are correct but the values are sent about every 2 seconds to the Middleware ...

From http://volkszaehler.org/pipermail/volkszaehler-dev/2015-March/004302.html

lib is required or build or build fails, yet not mentioned in wiki.

Suggesting to fix build first but add to wiki anyway afterwards.

@mbehr1 any chance you could take a look?

Hello,

has anyone ported code for arduino to read dlms protocol, yet?

I would like to use IR-TTL to read my Landis Gyr ZMD120.

Your help will be appreciated!

best regards

Thorsten

i think it is since #97 was merged, that i could never get vzlogger to build cleanly on my debian sid systems,
and it's been taking any fun out of developing.
i am running into a few issues:

• for inexplicable reason, all depedancies of libcurl are linked statically.
  is there any reason for this?
  it means one has to install some 20 -dev packages of libraries to have the static libs around,
  and which are not direct dependancies or curl-dev, because they are not needed for dynamic against it.
  i most of the time eneded up manuall removing all static declarations from the linker invocation
• /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/libmicrohttpd.a(libmicrohttpd_la-daemon.o): undefined reference to symbol 'gnutls_certificate_set_retrieve_function2@@GNUTLS_DEBIAN_0_3_0_0'
  //usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28: error adding symbols: DSO missing from command line
• it somehow seems that curl and microhttpd are linking two different versions of gnutls?

I just tried to figure out which values are allowed for an Obis string. According to e.g.
http://www.mayor.de/lian98/doc.de/html/g_iec62056_struct.htm

are the parameters not hexadecimal. But for C the special characters: C F L P and for D the special character F is defined.

So the current isxdigit solution is not sufficient (anyhow it would be strange as there would nothing like „0x3a“=(3_16+10) vs. „3a“ (3_10+10???). Shall I add an implementation according to the spec mentioned above?

make[1]: Leaving directory '/root/vzlogger/libs/libsml/test'
~/vzlogger/libs ~/vzlogger
~/vzlogger

building and installing vzlogger

building vzlogger
CMake Error: The source directory "/root/vzlogger" does not appear to contain CMakeLists.txt.

There is a CMakeLists.txt in /root/vzlogger/vzlogger

From the ml:

[ [ 1426414635584.1191, 1658272.1000000001 ] ]
vzlogger sendet die timestamps als float? nicht soo sinnvoll...

file src/protocols/MeterD0.cpp, line 310:

print(log_debug, "Parsed reading (OBIS code=%s, value=%s, unit=%s)", name().c_str(), obis_code, value, unit);
rds[number_of_tuples].value(strtof(value, NULL));

function strtof converts char* to float. The result will be automatically converted to the correct class member type double. As result, a number with 9 digits looses 2 digits precision.

Parts of my log:
the correct value 18670.3919 will be sent as 18670.392578

[Mar 23 10:33:14][d0] Parsed reading (OBIS code=1-0:0.0.0_255, value=17157728, unit=)
[Mar 23 10:33:14][d0] Parsed reading (OBIS code=1-0:1.8.1_255, value=018670.3919, unit=)
[Mar 23 10:33:14][d0] Parsed reading (OBIS code=1-0:96.5.5_255, value=80, unit=)
[Mar 23 10:33:14][d0] Parsed reading (OBIS code=0-0:96.1.255_255, value=38868046, unit=)
[Mar 23 10:33:14][d0] Read package with 4 tuples (vendor=ISk, baudrate=5, identification=MT671-0001)
[Mar 23 10:33:14][mtr0] Got 4 new readings from meter:
[Mar 23 10:33:14][mtr0] Reading: id=1-0:0.0.0_255/ObisItentifier:1-0:0.0.0_255 value=17157728.00 ts=1395567194.629
[Mar 23 10:33:14][mtr0] Reading: id=1-0:1.8.1_255/ObisItentifier:1-0:1.8.1_255 value=18670.39 ts=1395567194.658
[Mar 23 10:33:14][mtr0] Reading: id=1-0:96.5.5_255/ObisItentifier:1-0:96.5.5_255 value=80.00 ts=1395567194.679
[Mar 23 10:33:14][mtr0] Reading: id=0-0:96.1.255_255/ObisItentifier:0-0:96.1.255_255 value=38868048.00 ts=1395567194.708
[Mar 23 10:33:14][chn0] Adding reading to queue (value=18670.39 ts=1395567194.658)
[Mar 23 10:33:14][chn0] ==> number of tuples: 1
[Mar 23 10:33:14][chn0] compare: 1395567192648 1395567194658 1395567194658.167969
[Mar 23 10:33:14][chn0] JSON request body: [ [ 1395567194658.167969, 18670.392578 ] ]
[Mar 23 10:33:14][chn0] CURL: Re-using existing connection! (#0) with host localhost
[Mar 23 10:33:14][chn0] CURL: Connected to localhost (127.0.0.1) port 80 (#0)
[Mar 23 10:33:14][chn0] CURL: Sent 42 bytes..
[Mar 23 10:33:14][chn0] CURL: Sent '[ [ 1395567194658.167969, 18670.392578 ] ]' bytes
[Mar 23 10:33:14][chn0] Buffer dump (size=0 keep=32): {}
[Mar 23 10:33:14][chn0] CURL: Received 17 bytes
[Mar 23 10:33:14][chn0] CURL: Received '{"version":"0.3"}' bytes
[Mar 23 10:33:14][chn0] CURL: Connection #0 to host localhost left intact
[Mar 23 10:33:14][chn0] CURL Request succeeded with code: 200

The solution is to use strtod, not strtof

If I set my "timeout" to 30 the vzlooger still keeps the fixed 10s timeout

logfile:
[Jan 01 23:49:00][d0] nothing received for more than 10 seconds
[Jan 01 23:49:00][d0] read timed out!, context: 0, bytes read: 0, last byte 0x7f

I wanted to have microhttpd included in my build, so I used the following command to configure my build:

cmake -DENABLE_LOCAL=on .

Still in the resulting binary microhttpd is disabled. :-(

sml and d0 meters use obis identifiers to identify readings,
vzlogger stores both the identifier specified in the configuration and
those returned my meters in an Obis object.
the obis identifier is essentially a 6-byte binary string.

the comparison operator used to check the configured identifier against the
ones returned by a meter considers 0xff bytes wildcards,
in BOTH directions, so:

• if a meter returns a reading with "too many" 0xff bytes in it's identifier, it ends up in too many channels.
  see: http://volkszaehler.org/pipermail/volkszaehler-dev/2013-January/002321.html
• a meter reading that has a 0xff byte cannot be explicitly matched.
  see: http://volkszaehler.org/pipermail/volkszaehler-users/2014-December/005054.html

i am not sure what the correct behaviour is.
the current one seems to work for MOST users, but is clearly problematic.

suggestions:

• remove wildcard logic alltogether
• make it one-way to fix at least one of the issues mentioned (means we get a non-commutative equals operator)
• extend the obis identifer object to differentiate between wildcard and explicit 0xff values

most changes WILL break existing configs,
unless we leave 255 as wildcard and use SOMETHING ELSE for an explicit 0xff
(or introduce a new config flag to switch to the new matching mode)

vzlogger/src/Obis.cpp:
bool Obis::operator==(const Obis &rhs) const {
        for (int i = 0; i < 6; i++) {
                if (_obisId._raw[i] == rhs._obisId._raw[i] || _obisId._raw[i] == 0xff || rhs._obisId._raw[i] == 0xff ) {
                        continue; /* skip on wildcard or equal */

As I'm playing around with different version of vzlogger I find that it would be helpful if "somehow" a unique string would be included as part of the version string, in order to distinguish between different source levels used to build the binary.

Would it be possible to include something like the Git commit hash into the source, e. g. as an include file (.h or .hpp) when invoking "make" to build the tool? My idea is to query the hash using the git client (git log?) and then write that info into the include file.

Sorry if this sounds fuzzy, but I'm pretty clueless WRT use of Git.

while making an initscript for debian, i realised that vzlogger gives vzlogger.conf priority over command line switches, that is very confusing, cause normally it is the other way arround.

i`m trying to use vzlogger in foreground and i don´t need httpd. local is disabled, so is daemon. i would expect that vzlogger starts in foreground and does not start httpd - but vzlogger starts as daemon instead and also starts httpd, which is a major issue - as starting a tool which opens a listening port when explicitly being told not to do so imposes a security risk.

linux-xlde:~/vzlogger # vzlogger
[Dec 22 04:02:59][mtr0] Creating new meter with protocol sml.
[Dec 22 04:02:59][mtr0] Meter configured, enabled.
[Dec 22 04:02:59] New meter initialized (protocol=sml)
[Dec 22 04:02:59] Have 1 meters.
[Dec 22 04:02:59][main] daemon=0, local=1
[Dec 22 04:02:59] Daemonize process...
[Dec 22 04:02:59] Opened logfile /var/log/vzlogger.log
[Dec 22 04:02:59][] ===> Start meters
[Dec 22 04:02:59][mtr0] Meter connection established
[Dec 22 04:02:59][mtr0] Meter thread started
[Dec 22 04:02:59][mtr0] Meter is opened. Starting channels.
[Dec 22 04:02:59][http] Starting local interface HTTPd on port 8080
[Dec 22 04:02:59][] Startup done.
[Dec 22 04:02:59][mtr0] Number of readers: 32
[Dec 22 04:02:59][mtr0] Config.daemon: 0
[Dec 22 04:02:59][mtr0] Config.local: 1

this is my configuration -

/**

• vzlogger configuration
  *
• use proper encoded JSON with javascript comments
  *
• take a look at the wiki for detailed information:
• http://wiki.volkszaehler.org/software/controller/vzlogger#configuration
  */

{
"retry": 30, // how long to sleep between failed requests, in seconds
"daemon": false, // run periodically
"verbosity": 15, // between 0 and 15
"log": "/var/log/vzlogger.log", // path to logfile, optional

"local": {
    "enabled": false,   // should we start the local HTTPd for serving live readings?
    "port": 8080,       // the TCP port for the local HTTPd
    "index": true,      // should we provide a index listing of available channels if no UUID was requested?
    "timeout": 30,      // timeout for long polling comet requests, 0 disables comet, in seconds
    "buffer": 600       // how long to buffer readings for the local interface, in seconds
},

"meters": [
    {
        "enabled": true,               // disabled meters will be ignored
        "skip": false,                  // if enabled, errors when opening meter will lead to meter being ignored
        "protocol": "sml",              // see 'vzlogger -h' for list of available protocols
        "device": "/dev/ttyUSB1",
        "interval": 2,
    }
]

}

https://www.mail-archive.com/search?l=volkszaehler-dev%40lists.volkszaehler.org&q=from%3A%22Sebastian+Michel%22&o=newest&f=1

Hallo zusammen,

also es funktioniert nun sowohl mit dem ISKRAemeco MT174 als auch dem
Siemens TD3511. Anbei findet ihr den Patch. Ich hab versucht sowenig
Änderungen zu machen wie es geht. Hier nochmal die Zusammenfassungen:

1. Das Device wird im blocking mode geöffnet. Das führte bei mir dazu,
   dass die read() Funktion mit 0 zurückkehrt. Das heißt normalerweise ein
   eof oder derartiges. Sowas sollte bei einem seriellen Device niemals
   passieren. Ich weiß auch nicht warum das der Fall ist.

• Ich hab den Code dahingehend geändert, dass das Device im
  non-blocking mode geöffnet wird. Damit funktioniert's nun.

2. Die Statemachine sollte niemals unendlich warten und irgendwo hängen
   bleiben.

• Ich hab ein Timeout von 10s eingebaut. Also wenn innerhalb 10s keine
  Daten empfangen werden bzw. beim Start kein Sync-byte kommt, wird dieser
  Lesevorgang abgebrochen.

3. Die Statemaching wird immer beim Empfang eines '/' zurückgesetzt. Da
   mein Zähler ein '/' im Identification String sendet, hängt sich die
   Statemachine auf.

• Das habe ich rausgenommen.

4. Mein Zähler sendet ca. 330 obis-codes. Da ist dann sowas dabei wie
   2.8.1_16, 2.8.1_15 usw. Ich interessiere mich eigentlich nur für die
   Einträge von 2.8.0. Das ist insofern ein Problem, dass der D0-Meter nur
   maximal 32 Einträge zulässt. Das heißt bis die interessanten codes
   ankommen, sind die 32 Einträge schon längst belegt.

• Ich filtere jetzt nur noch obis-codes heraus die ein value und eine
  Einheit besitzen. Damit kommen bei meinem Zähler genau 32 zusammen.

5. Der Zähler sendet ungültige Obis-Codes. Zumindest ist der string für
   die Klasse Obis.cpp nicht auswertbar. Das führt dazu, dass vzlogger
   abstürzt. Vermutlich sind es die obis-codes die unter anderem Buchstaben
   statt Zahlen haben.

• Ich hab die Stellen mit einem try-except Block abgefangen und
  überspringe die betroffenen obis-codes.

6. Fehler beim Setzen der Parity

• Das hab ich korrigiert.

Weiterhin habe ich mal die Ausgaben der beiden Zähler angehangen, mit
denen das getestet wurde.

Gruß
Sebastian

ISKRAemeco_MT174.txt
Description: Binary data


Siemens_TD-3511.txt
Description: Binary data
diff --git a/src/Obis.cpp b/src/Obis.cpp
index f918797..eebf5f8 100644
--- a/src/Obis.cpp
+++ b/src/Obis.cpp
@@ -192,7 +192,7 @@ int Obis::parse(const char *str) {
 }

 int Obis::lookup_alias(const char *alias) {
-   for (const obis_alias_t *it = aliases; it != NULL; it++) {
+   for (const obis_alias_t *it = aliases; !it-id.isNull(); it++) {
        if (strcmp(it-name, alias) == 0) {
            *this = it-id;
            return SUCCESS;
diff --git a/src/protocols/MeterD0.cpp b/src/protocols/MeterD0.cpp
index f48f864..cc74146 100644
--- a/src/protocols/MeterD0.cpp
+++ b/src/protocols/MeterD0.cpp
@@ -204,6 +204,13 @@ ssize_t MeterD0::read(std::vectorReadingrds, size_t max_readings) {
    char byte;          /* we parse our input byte wise */
    int byte_iterator; 
    size_t number_of_tuples;
+int bytes_read;
+
+time_t start_time;
+time_t end_time;
+
+// get current time
+time(start_time);

    if(_pull.size()) {
        int wlen=write(_fd,_pull.c_str(),_pull.size());
@@ -215,12 +222,41 @@ ssize_t MeterD0::read(std::vectorReadingrds, size_t max_readings) {

    context = START;/* start with context START */

-   while (::read(_fd, byte, 1)) {
-       if (byte == '/') context = START;   /* reset to START if / reoccurs */
-       else if (byte == '!') context = END;    /* ! is the identifier for the END */
+   while (1) {
+
+// check if timed out
+time(end_time);
+if (difftime(end_time, start_time)  10)
+{
+ print(log_error, nothing received for more than 10 seconds, name().c_str());
+ break;
+}
+
+// now read a single byte
+bytes_read = ::read(_fd, byte, 1);
+if (bytes_read == 0 || bytes_read == -1  errno == EAGAIN)
+{
+// wait and read again
+usleep(5000);  // 5 ms
+continue;
+}
+else if (bytes_read == -1)
+{
+// an error occured reading a byte
+print(log_error, error reading a byte (%d), name().c_str(), errno);
+break;
+}
+
+// reset timeout
+if (context != START)
+{
+time(start_time);
+}
+
+       if (byte == '!') context = END; /* ! is the identifier for the END */
        switch (context) {
 case START:            /* strip the initial / */
-if  (byte != '\r'   byte != '\n') { /*allow extra new line at the start */
+if  (byte == '/') {
   byte_iterator = number_of_tuples = 0;/* start */
   context = VENDOR;/* set new context: START - VENDOR */
 }
@@ 

it's not great to clone the whole S0 meter code for the raspberry pi version
there's lots of common code that would have to be maintained redunantly.
what would be better?

• implement a unified s0-meter and handle differences internally?
• implement an abstract s0-meter class and have others inherit from it?
  (it's C++ after all but not sure if i know enough C++)

(for reference, my current code is here: https://github.com/r00t-/vzlogger/tree/raspis0 )

Documentation for install is v.good, just missing microhttpd dependancy,

sudo apt-get install libmicrohttpd-dev

I would also make note to copy files to /usr/local/lib as it seems to be a more usual location

sudo cp -R sml/include/* /usr/local/include/

Cheers

Dave

Even if I have "daemon": false in vzlogger.conf the program still daemonizes:

[Nov 30 12:59:57][main] daemon=0, local=1
[Nov 30 12:59:57] Daemonize process...

This applies to vzlogger 0.3.9, compiled and run on Raspbian 2014-09-09.

andig recommended to use 0.4.0 -- where can I get it? I pulled the latest source from GitHub yesterday...

And yes, I am sure that I'm touching the right config file. I'm changing it all the time, and my changes are picked up by vzlogger (apart from this issue I'm reporting here).

Aus der Mailing List:

On 05/25/2014 09:43 PM, Andreas Götz wrote:

Gibts jemanden der Kontakt zu TheCount hat um zu verstehen was bzgl libsml eigentlich "richtig" wäre?
der Fork stammt von mir, behebt aber bei weitem nicht alle Fehler. Ihr
könnt ihn gerne benutzen und mir auch Pull Requests schicken, aber:

• der Fork erfordert einen C99-Compiler, während die alte Version
  (wahrscheinlich) mit C89 auskam,
• der Fork ist wahrscheinlich nicht ABI-kompatibel zum Original (API
  passt glaube ich noch). Ich war auch zu faul, die Versionsnummer der
  Bibliothek entsprechend anzupassen.
  "Richtig" wäre, der libsml eine komplette Frischzellenkur zu verpassen
  (wenn man sie nicht gleich neu schreibt), insbesondere:
• ein paar kleinere, aber wichtige Anpassungen der API,
• komplettes Audit der Implementation; insbesondere die Fehlerbehandlung
  fehlt öfters,
• vernünftige Build Comprehension – k.a. ob die libsml derzeit bspw.
  unter Windows kompilierbar ist,
• verbesserte Dokumentation,
• weitere Tests (und sich dabei überlegen, wie man das Unity-Framework
  vernünftig ins Repository einbindet).
  Mangels Zeit kann ich das leider nicht machen. Auf konkrete Anfrage
  erläutere ich die Punkte oben aber gerne näher, und auch auf Pull
  Requests werde ich reagieren.
  Viele Grüße
  Alexander

Since #97 build fails even if libcurl4-gnutls-dev is installed: http://pastebin.com/Z00nyPwe

Likely introduced by #97. Should we back out #97 as MySmartGrid isn't used anyway?

A prereq check would be helpful.