MQTT timestamps and protocol improvements (#140)

* add TS_INPUT, TS_GRID and TS_TOTAL

* prepare MQTT timestamps

- add _set_mqtt_timestamp method
- fix hexdump printing

* push dev and debug images to docker.io

* add unix epoche timestamp for MQTT pakets

* set timezone for unit tests

* set name für setting timezone step

* trigger new action

* GEN3 and GEN3PLUS: handle multiple message

- read: iterate over the receive buffer
- forward: append messages to the forward buffer
- _update_header: iterate over the forward buffer

* GEN3: optimize timeout handling

- longer timeout in state init and reveived
- got to state pending only from state up

* update changelog

* cleanup

.github/FUNDING.yml

@@ -0,0 +1,3 @@
+# These are supported funding model platforms
+
+ko_fi: sallius

.github/workflows/python-app.yml

@@ -0,0 +1,56 @@
+# This workflow will install Python dependencies, run tests and lint with a single version of Python
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
+
+name: Python application
+
+on:
+  push:
+    branches: [ "main", "dev-*", "*/issue*" ]
+    paths-ignore:
+    - '**.md' # Do no build on *.md changes
+    - '**.yml' # Do no build on *.yml changes
+    - '**.yaml' # Do no build on *.yaml changes
+    - '**.yuml' # Do no build on *.yuml changes
+    - '**.svg' # Do no build on *.svg changes
+    - '**.json' # Do no build on *.json changes
+    - '**.cfg' # Do no build on *.cfg changes
+    - '**.gitignore' # Do no build on *.gitignore changes
+    - '**.dockerfile' # Do no build on *.dockerfile changes
+    - '**.sh' # Do no build on *.sh changes     
+  pull_request:
+    branches: [ "main" ]
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Set timezone
+      uses: szenius/set-timezone@v2.0
+      with:
+        timezoneLinux: "Europe/Berlin"
+        timezoneMacos: "Europe/Berlin"
+        timezoneWindows: "Europe/Berlin"
+    - uses: actions/checkout@v4
+    - name: Set up Python 3.12
+      uses: actions/setup-python@v5
+      with:
+        python-version: "3.12"
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install flake8 pytest pytest-asyncio
+        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+    - name: Lint with flake8
+      run: |
+        # stop the build if there are Python syntax errors or undefined names
+        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+    - name: Test with pytest
+      run: |
+        python -m pytest app

.gitignore

@@ -1,9 +1,11 @@
 __pycache__
 .pytest_cache
+bin/**
 mosquitto/**
 homeassistant/**
 tsun_proxy/**
 Doku/**
 .DS_Store
 .coverage
+.env
 coverage.xml

.markdownlint.json

@@ -0,0 +1,4 @@
+{
+    "MD013": false,
+    "MD033": false
+  }

.vscode/launch.json

@@ -6,7 +6,7 @@
     "configurations": [
         {
             "name": "Python: Aktuelle Datei",
-            "type": "python",
+            "type": "debugpy",
             "request": "launch",
             "program": "${file}",
             "console": "integratedTerminal",

.vscode/settings.json

@@ -8,5 +8,8 @@
         "system_tests"
     ],
     "python.testing.unittestEnabled": false,
-    "python.testing.pytestEnabled": true
+    "python.testing.pytestEnabled": true,
+    "flake8.args": [
+        "--extend-exclude=app/tests/*.py system_tests/*.py"   
+    ]
 }

CHANGELOG.md

@@ -5,15 +5,178 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [unreleased]
+
+- add timestamp to MQTT topics [#138](https://github.com/s-allius/tsun-gen3-proxy/issues/138)
+- improve the message handling, to avoid hangs
+- GEN3: allow long timeouts until we received first inverter data (not only device data)
+- bump aiomqtt to version 2.2.0
+- bump schema to version 0.7.7
+- Home Assistant: improve inverter status value texts
+- GEN3: add inverter status
+- fix flapping registers [#128](https://github.com/s-allius/tsun-gen3-proxy/issues/128)
+- register OUTPUT_COEFFICIENT at HA
+- GEN3: INVERTER_STATUS,
+- add config option to disable the MODBUS polling [#120](https://github.com/s-allius/tsun-gen3-proxy/issues/120)
+- make the maximum output coefficient configurable [#123](https://github.com/s-allius/tsun-gen3-proxy/issues/123)
+- cleanup shutdown
+- add preview build
+- MODBUS: the last digit of the inverter version is a hexadecimal number [#119](https://github.com/s-allius/tsun-gen3-proxy/issues/119)
+- GEN3PLUS: add client_mode connection on port 8899 [#117](https://github.com/s-allius/tsun-gen3-proxy/issues/117)
+
+## [0.9.0] - 2024-07-01
+
+- fix exception in MODBUS timeout callback
+
+## [0.9.0-RC1] - 2024-06-29
+
+- add asyncio log and debug mode
+- stop the HTTP server on shutdown gracefully
+- Synchronize regular MODBUS commands with the status of the inverter to prevent the inverter from crashing due to
+  unexpected packets. [#111](https://github.com/s-allius/tsun-gen3-proxy/issues/111)
+- GEN3: avoid sending MODBUS commands to the inverter during the inverter's reporting phase
+- GEN3: determine the connection timeout based on the connection state
+- GEN3: support more data encodings for DSP version V5.0.17 [#108](https://github.com/s-allius/tsun-gen3-proxy/issues/108)
+- detect dead connections [#100](https://github.com/s-allius/tsun-gen3-proxy/issues/100)
+- improve connection logging wirt a unique connection id
+- Add healthcheck, readiness and liveness checks [#91](https://github.com/s-allius/tsun-gen3-proxy/issues/91)
+- MODBUS close handler releases internal resource [#93](https://github.com/s-allius/tsun-gen3-proxy/issues/93)
+- add exception handling for message forwarding [#94](https://github.com/s-allius/tsun-gen3-proxy/issues/94)
+- GEN3: make timestamp handling stateless, to avoid blocking when the TSUN cloud is down [#56](https://github.com/s-allius/tsun-gen3-proxy/issues/56)
+- GEN3PLUS: dump invalid packages with wrong start or stop byte
+- label debug imagages als `debug`
+- print imgae build time during proxy start
+- add type annotations
+- improve async unit test and fix pytest warnings
+- run github tests even for pulls on issue branches
+
+## [0.8.1] - 2024-06-21
+
+- Fix MODBUS responses are dropped and not forwarded to the TSUN cloud [#104](https://github.com/s-allius/tsun-gen3-proxy/issues/104)
+- GEN3: Fix connections losts due MODBUS requests [#102](https://github.com/s-allius/tsun-gen3-proxy/issues/102)
+
+## [0.8.0] - 2024-06-07
+
+- improve logging: add protocol or node_id to connection logs
+- improve logging: log ignored AT+ or MODBUS commands
+- improve tracelog: log level depends on message type and source
+- fix typo in docker-compose.yaml and remove the external network definition
+- trace heartbeat and regular modbus pakets witl log level DEBUG
+- GEN3PLUS: don't forward ack paket from tsun to the inverter
+- GEN3PLUS: add allow and block filter for AT+ commands
+- catch all OSError errors in the read loop
+- log Modbus traces with different log levels
+- add Modbus fifo and timeout handler
+- build version string in the same format as TSUN for GEN3 inverters
+- add graceful shutdown
+- parse Modbus values and store them in the database
+- add cron task to request the output power every minute
+- GEN3PLUS: add MQTT topics to send AT commands to the inverter
+- add MQTT topics to send Modbus commands to the inverter
+- convert data collect interval to minutes
+- add postfix for rc and dev versions to the version number
+- change logging level to DEBUG for some logs
+- remove experimental value Register.VALUE_1
+- format Register.POWER_ON_TIME as integer
+- ignore catch-up values from the inverters for now
+
+## [0.7.0] - 2024-04-20
+
+- GEN3PLUS: fix temperature values
+- GEN3PLUS: read corect firmware and logger version
+- GEN3PLUS: add inverter status
+- GEN3PLUS: fix encoding of `power on time` value
+- GEN3PLUS: fix glitches in inverter data after connection establishment
+  see: [#53](https://github.com/s-allius/tsun-gen3-proxy/issues/53)
+- improve docker container labels
+- GEN3PLUS: add timestamp of inverter data into log
+- config linter for *.md files
+- switch to aiomqtt version 2.0.1
+- refactor unittest and increase testcoverage
+- GEN3PLUS: add experimental handler for `ÀT` commands
+- GEN3PLUS: implement self-sufficient island support
+  see: [#42](https://github.com/s-allius/tsun-gen3-proxy/issues/42)
+- Improve error messages on config errors
+  see: [#46](https://github.com/s-allius/tsun-gen3-proxy/issues/46)
+- Prepare support of inverters with 6 MTPPs
+- Clear `Daily Generation` values at midnigth
+  see: [#32](https://github.com/s-allius/tsun-gen3-proxy/issues/32)
+- Read pv module details from config file and use it for the Home Assistant registration
+  see: [#43](https://github.com/s-allius/tsun-gen3-proxy/issues/43)
+- migrate to aiomqtt version 2.0.0
+  see: [#44](https://github.com/s-allius/tsun-gen3-proxy/issues/44)
+
+## [0.6.0] - 2024-04-02
+
+- Refactoring to support Solarman V5 protocol
+- Add unittest for Solarman V5 implementation
+- Handle checksum errors
+- Handle wrong start or Stop bytes
+- Watch for AT commands and signal their occurrence to HA
+- Build inverter type names for MS-1600 .. MS-2000
+- Build device name for Solarman logger module  
+
+## [0.5.5] - 2023-12-31
+
+- Fixed [#33](https://github.com/s-allius/tsun-gen3-proxy/issues/33)
+- Fixed detection of the connected inputs/MPPTs
+- Preparation for overwriting received data
+- home assistant improvements:
+  - Add unit 'W' to the `Rated Power` value for home assistant
+  - `Collect_Interval`, `Connect_Count` and `Data_Up_Interval` as diagnostic value and not as graph
+  - Add data acquisition interval
+  - Add number of connections
+  - Add communication type
+  - Add 'Internal SW Exception' counter
+
+## [0.5.4] - 2023-11-22
+
+- hardening remove dangerous commands from busybox
+- add OTA start message counter
+- add message handler for over the air updates
+- add unit tests for ota messages
+- add unit test for int64 data type
+- cleanup msg_get_time_handler
+- remove python packages setuptools, wheel, pip from final image to reduce the attack surface
+
+## [0.5.3] - 2023-11-12
+
+- remove apk packet manager from the final image
+- send contact info every time a client connection is established
+- use TSUN timestamp instead of local time, as TSUN also expects Central European Summer Time in winter
+
+## [0.5.2] - 2023-11-09
+
+- add int64 data type to info parser
+- allow multiple calls to Message.close()
+- check for race cond. on closing and establishing client connections
+
+## [0.5.1] - 2023-11-05
+
+- fixes f-string by limes007
+- add description for dns settings by limes007
+
+## [0.5.0] - 2023-11-04
+
+- fix issue [#21](https://github.com/s-allius/tsun-gen3-proxy/issues/21)
+- register proxy dev as soon as the MQTT connection is established
+- increase test coverage of the Messages class
+- add error counter for unknown control bytes
+- lint code with flake8
+  
+## [0.4.3] - 2023-10-26
+
+- fix typos by Lenz Grimmer
+- catch mqtt errors, so we can forward messages to tsun even if the mqtt broker is not reachable
+- avoid resetting the daily generation counters even if the inverter sends zero values after reconnection
 
 ## [0.4.2] - 2023-10-21
 
 - count unknown data types in received messages
-- count defintion errors in our internal tables
+- count definition errors in our internal tables
 - increase test coverage of the Infos class to 100%
-- avoids resetting the daily generation counters even if the inverter sends zero values at sunset
-   
+- avoid resetting the daily generation counters even if the inverter sends zero values at sunset
+
 ## [0.4.1] - 2023-10-20
 
 - fix issue [#18](https://github.com/s-allius/tsun-gen3-proxy/issues/18)
@@ -39,13 +202,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - optimize and reduce logging
 - switch to pathon 3.12
-- classify some values for diagnostics 
+- classify some values for diagnostics
   
 ## [0.2.0] - 2023-10-07
 
 This version halves the size of the Docker image and reduces the attack surface for security vulnerabilities, by omitting unneeded code. The feature set is exactly the same as the previous release version 0.1.0.
 
-### Changes
+### Changes in 0.2.0
 
 - move from slim-bookworm to an alpine base image
 - install python requirements with pip wheel
@@ -78,31 +241,31 @@ This version halves the size of the Docker image and reduces the attack surface
   
 ❗Due to the change from one device to multiple devices in the Home Assistant, the previous MQTT device should be deleted in the Home Assistant after the update to pre-release '0.0.4'. Afterwards, the proxy must be restarted again to ensure that the sub-devices are created completely.
 
-### Added
+### Added in 0.0.4
 
 - Register multiple devices at home-assistant instead of one for all measurements.
   Now we register: a Controller, the inverter and up to 4 input devices to home-assistant.
   
 ## [0.0.3] - 2023-09-28
 
-### Added
+### Added in 0.0.3
 
 - Fixes Running Proxy with host UID and GUID #2
 
 ## [0.0.2] - 2023-09-27
 
-### Added
+### Added in 0.0.2
 
 - Dockerfile opencontainer labels
 - Send voltage and current of inputs to mqtt
 
 ## [0.0.1] - 2023-09-25
 
-### Added
+### Added in 0.0.1
 
 - Logger for inverter packets
 - SIGTERM handler for fast docker restarts
-- Proxy as non-root docker application 
+- Proxy as non-root docker application
 - Unit- and system tests
 - Home asssistant auto configuration
 - Self-sufficient island operation without internet
@@ -111,4 +274,4 @@ This version halves the size of the Docker image and reduces the attack surface
 
 ### Added
 
-- First checkin, the project was born
+- First checkin, the project was born

CODE_OF_CONDUCT.md

@@ -60,7 +60,7 @@ representative at an online or offline event.
 
 Instances of abusive, harassing, or otherwise unacceptable behavior may be
 reported to the community leaders responsible for enforcement at
-compliance@allius.de.
+<compliance@allius.de>.
 All complaints will be reviewed and investigated promptly and fairly.
 
 All community leaders are obligated to respect the privacy and security of the
@@ -116,7 +116,7 @@ the community.
 
 This Code of Conduct is adapted from the [Contributor Covenant][homepage],
 version 2.0, available at
-https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+<https://www.contributor-covenant.org/version/2/0/code_of_conduct.html>.
 
 Community Impact Guidelines were inspired by [Mozilla's code of conduct
 enforcement ladder](https://github.com/mozilla/diversity).
@@ -124,5 +124,5 @@ enforcement ladder](https://github.com/mozilla/diversity).
 [homepage]: https://www.contributor-covenant.org
 
 For answers to common questions about this code of conduct, see the FAQ at
-https://www.contributor-covenant.org/faq. Translations are available at
-https://www.contributor-covenant.org/translations.
+<https://www.contributor-covenant.org/faq>. Translations are available at
+<https://www.contributor-covenant.org/translations>.

CONTRIBUTING.md

@@ -7,6 +7,7 @@ The project aims to bring TSUN third generation inverters (with WiFi support) in
 The code base of the proxy was created in a few weeks after work and offers many possibilities for collaboration.
 
 Especially in the area of
+
 - docker compose
 - packaging
 - test automation

LICENSE.md

@@ -1,4 +1,4 @@
-Copyright (c) 2023 Stefan Allius. 
+# Copyright © 2023 Stefan Allius
 
 Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
 

README.md

@@ -7,13 +7,11 @@
 <p align="center">
     <a href="https://opensource.org/licenses/BSD-3-Clause"><img alt="License: BSD-3-Clause" src="https://img.shields.io/badge/License-BSD_3--Clause-green.svg"></a>
     <a href="https://www.python.org/downloads/release/python-3120/"><img alt="Supported Python versions" src="https://img.shields.io/badge/python-3.12-blue.svg"></a>
-    <a href="https://sbtinstruments.github.io/aiomqtt/introduction.html"><img alt="Supported aiomqtt versions" src="https://img.shields.io/badge/aiomqtt-1.2.1-lightblue.svg"></a>
+    <a href="https://sbtinstruments.github.io/aiomqtt/introduction.html"><img alt="Supported aiomqtt versions" src="https://img.shields.io/badge/aiomqtt-2.2.0-lightblue.svg"></a>
+    <a href="https://libraries.io/pypi/aiocron"><img alt="Supported aiocron versions" src="https://img.shields.io/badge/aiocron-1.8-lightblue.svg"></a>
     <a href="https://toml.io/en/v1.0.0"><img alt="Supported toml versions" src="https://img.shields.io/badge/toml-1.0.0-lightblue.svg"></a>
-
 </p>
 
-
-###
 # Overview
 
 This proxy enables a reliable connection between TSUN third generation inverters and an MQTT broker. With the proxy, you can easily retrieve real-time values such as power, current and daily energy and integrate the inverter into typical home automations. This works even without an internet connection. The optional connection to the TSUN Cloud can be disabled!
@@ -23,11 +21,12 @@ In detail, the inverter establishes a TCP connection to the TSUN cloud to transm
 Through this, the inverter then establishes a connection to the proxy and the proxy establishes another connection to the TSUN Cloud. The transmitted data is interpreted by the proxy and then passed on to both the TSUN Cloud and the MQTT broker. The connection to the TSUN Cloud is optional and can be switched off in the configuration (default is on). Then no more data is sent to the Internet, but no more remote updates of firmware and operating parameters (e.g. rated power, grid parameters) are possible.
 
 By means of `docker` a simple installation and operation is possible. By using `docker-composer`, a complete stack of proxy, `MQTT-brocker` and `home-assistant` can be started easily.
-###
-ℹ️ This project is not related to the company TSUN. It is a private initiative that aims to connect TSUN inverters with an MQTT broker. There is no support and no warranty from TSUN.
-###
 
-```
+<br>
+ℹ️ This project is not related to the company TSUN. It is a private initiative that aims to connect TSUN inverters with an MQTT broker. There is no support and no warranty from TSUN.
+<br><br>
+
+```txt
 ❗An essential requirement is that the proxy can be looped into the connection
 between the inverter and TSUN Cloud.
 
@@ -40,64 +39,93 @@ If you use a Pi-hole, you can also store the host entry in the Pi-hole.
 
 ## Features
 
-- supports TSOL MS300, MS350, MS400, MS600, MS700 and MS800 inverters from TSUN
+- Supports TSUN GEN3 PLUS inverters: TSOL-MS2000, MS1800 and MS1600
+- Supports TSUN GEN3 inverters: TSOL-MS800, MS700, MS600, MS400, MS350 and MS300
 - `MQTT` support
 - `Home-Assistant` auto-discovery support
+- `MODBUS` support via MQTT topics
+- `AT-Command` support via MQTT topics (GEN3PLUS only)
+- Faster DataUp interval sends measurement data to the MQTT broker every minute
 - Self-sufficient island operation without internet
-- non-root Docker Container
+- Security-Features:
+  - control access via `AT-commands`
+  - Runs in a non-root Docker Container
 
 ## Home Assistant Screenshots
 
 Here are some screenshots of how the inverter is displayed in the Home Assistant:
 
-https://github.com/s-allius/tsun-gen3-proxy/wiki/home-assistant#home-assistant-screenshots
+<https://github.com/s-allius/tsun-gen3-proxy/wiki/home-assistant#home-assistant-screenshots>
+
 ## Requirements
 
 - A running Docker engine to host the container
 - Ability to loop the proxy into the connection between the inverter and the TSUN cloud
 
-
-###
 # Getting Started
 
 To run the proxy, you first need to create the image. You can do this quite simply as follows:
+
 ```sh
 docker build https://github.com/s-allius/tsun-gen3-proxy.git#main:app -t tsun-proxy
 ```
+
 after that you can run the image:
+
 ```sh
-docker run  --dns '8.8.8.8' --env 'UID=1000' -p '5005:5005'  -v ./config:/home/tsun-proxy/config -v ./log:/home/tsun-proxy/log tsun-proxy
+docker run  --dns '8.8.8.8' --env 'UID=1000' -p '5005:5005' -p '10000:10000' -v ./config:/home/tsun-proxy/config -v ./log:/home/tsun-proxy/log tsun-proxy
 ```
-You will surely see a message that the configuration file was not found. So that we can create this without admin rights, the `uid` must still be adapted. To do this, simply stop the proxy with ctrl-c and use the `id` command to determine your own UserId: 
+
+You will surely see a message that the configuration file was not found. So that we can create this without admin rights, the `uid` must still be adapted. To do this, simply stop the proxy with ctrl-c and use the `id` command to determine your own UserId:
+
 ```sh
 % id 
 uid=1050(sallius) gid=20(staff) ...
 ```
+
 With this information we can customize the `docker run`` statement:
+
 ```sh
-docker run  --dns '8.8.8.8' --env 'UID=1050' -p '5005:5005'  -v ./config:/home/tsun-proxy/config -v ./log:/home/tsun-proxy/log tsun-proxy
+docker run  --dns '8.8.8.8' --env 'UID=1050' -p '5005:5005' -p '10000:10000' -v ./config:/home/tsun-proxy/config -v ./log:/home/tsun-proxy/log tsun-proxy
 ```
 
-###
 # Configuration
-The Docker container does not require any special configuration. 
+
+The configuration consists of several parts. First, the container and the proxy itself must be configured, and then the connection of the inverter to the proxy must be set up, which is done differently depending on the inverter generation
+
+For GEN3PLUS inverters, this can be done easily via the web interface of the inverter. The GEN3 inverters do not have a web interface, so the proxy is integrated via a modified DNS resolution.
+
+  1. [Container Setup](#container-setup)
+  2. [Proxy Configuration](#proxy-configuration)
+  3. [Inverter Configuration](#inverter-configuration) (only GEN3PLUS)
+  4. [DNS Settings](#dns-settings) (Mandatory for GEN3)
+
+## Container Setup
+
+No special configuration is required for the Docker container if it is built and started as described above. It is recommended to start the container with docker-compose. The configuration is then specified in a docker-compose.yaml file. An example of a stack consisting of the proxy, MQTT broker and home assistant can be found [here](https://github.com/s-allius/tsun-gen3-proxy/blob/main/docker-compose.yaml).
+
 On the host, two directories (for log files and for config files) must be mapped. If necessary, the UID of the proxy process can be adjusted, which is also the owner of the log and configuration files.
 
-The proxy can be configured via the file 'config.toml'. When the proxy is started, a file 'config.example.toml' is copied into the config directory. This file shows all possible parameters and their default values. Changes in the example file itself are not evaluated. To configure the proxy, the config.example.toml file should be renamed to config.toml. After that the corresponding values can be adjusted. To load the new configuration, the proxy must be restarted. 
-
+A description of the configuration parameters can be found [here](https://github.com/s-allius/tsun-gen3-proxy/wiki/Configuration-details#docker-compose-environment-variables).
 
 ## Proxy Configuration
+
+The proxy can be configured via the file 'config.toml'. When the proxy is started, a file 'config.example.toml' is copied into the config directory. This file shows all possible parameters and their default values. Changes in the example file itself are not evaluated. To configure the proxy, the config.example.toml file should be renamed to config.toml. After that the corresponding values can be adjusted. To load the new configuration, the proxy must be restarted.
+
 The configration uses the TOML format, which aims to be easy to read due to obvious semantics.
-You find more details here: https://toml.io/en/v1.0.0
-
-
+You find more details here: <https://toml.io/en/v1.0.0>
 
 ```toml
-# configuration to reach tsun cloud
+# configuration for tsun cloud for 'GEN3' inverters
 tsun.enabled = true   # false: disables connecting to the tsun cloud, and avoids updates
 tsun.host    = 'logger.talent-monitoring.com'
 tsun.port    = 5005
 
+# configuration for solarman cloud  for 'GEN3 PLUS' inverters
+solarman.enabled = true   # false: disables connecting to the tsun cloud, and avoids updates
+solarman.host    = 'iot.talent-monitoring.com'
+solarman.port    = 10000
+
 
 # mqtt broker configuration
 mqtt.host    = 'mqtt'   # URL or IP address of the mqtt broker
@@ -110,6 +138,8 @@ mqtt.passwd  = ''
 ha.auto_conf_prefix = 'homeassistant'       # MQTT prefix for subscribing for homeassistant status updates
 ha.discovery_prefix = 'homeassistant'       # MQTT prefix for discovery topic 
 ha.entity_prefix    = 'tsun'                # MQTT topic prefix for publishing inverter values
+ha.proxy_node_id    = 'proxy'               # MQTT node id, for the proxy_node_id
+ha.proxy_unique_id  = 'P170000000000001'    # MQTT unique id, to identify a proxy instance
 
 
 # microinverters
@@ -118,17 +148,118 @@ inverters.allow_all = false   # True: allow inverters, even if we have no invert
 # inverter mapping, maps a `serial_no* to a `node_id` and defines an optional `suggested_area` for `home-assistant`
 #
 # for each inverter add a block starting with [inverters."<16-digit serial numbeer>"]
+
 [inverters."R17xxxxxxxxxxxx1"]
 node_id = 'inv1'              # Optional, MQTT replacement for inverters serial number  
 suggested_area = 'roof'       # Optional, suggested installation area for home-assistant
+pv1 = {type = 'RSM40-8-395M', manufacturer = 'Risen'}   # Optional, PV module descr
+pv2 = {type = 'RSM40-8-395M', manufacturer = 'Risen'}   # Optional, PV module descr
 
 [inverters."R17xxxxxxxxxxxx2"]
 node_id = 'inv2'              # Optional, MQTT replacement for inverters serial number  
 suggested_area = 'balcony'    # Optional, suggested installation area for home-assistant
+pv1 = {type = 'RSM40-8-405M', manufacturer = 'Risen'}   # Optional, PV module descr
+pv2 = {type = 'RSM40-8-405M', manufacturer = 'Risen'}   # Optional, PV module descr
 
+[inverters."Y17xxxxxxxxxxxx1"]  # This block is also for inverters with a Y47 serial no
+monitor_sn = 2000000000       # The "Monitoring SN:" can be found on a sticker enclosed with the inverter
+node_id = 'inv_3'             # MQTT replacement for inverters serial number  
+suggested_area = 'garage'     # suggested installation place for home-assistant
+# if your inverter supports SSL connections you must use the client_mode. Pls, uncomment
+# the next line and configure the fixed IP of your inverter
+#client_mode = {host = '192.168.0.1', port = 8899}       
+pv1 = {type = 'RSM40-8-410M', manufacturer = 'Risen'}   # Optional, PV module descr
+pv2 = {type = 'RSM40-8-410M', manufacturer = 'Risen'}   # Optional, PV module descr
+pv3 = {type = 'RSM40-8-410M', manufacturer = 'Risen'}   # Optional, PV module descr
+pv4 = {type = 'RSM40-8-410M', manufacturer = 'Risen'}   # Optional, PV module descr
+
+[gen3plus.at_acl]
+tsun.allow = ['AT+Z', 'AT+UPURL', 'AT+SUPDATE']   # allow this for TSUN access
+tsun.block = []                                   
+mqtt.allow = ['AT+']                              # allow all via mqtt
+mqtt.block = []
 
 ```
 
+## Inverter Configuration
+
+GEN3PLUS inverters offer a web interface that can be used to configure the inverter. This is very practical for sending the data directly to the proxy. On the one hand, the inverter broadcasts its own SSID on 2.4GHz. This can be recognized because it is broadcast with `AP_<Montoring SN>`. You will find the `Monitor SN` and the password for the WLAN connection on a small sticker enclosed with the inverter.
+
+If you have already connected the inverter to the cloud via the TSUN app, you can also address the inverter directly via WiFi. In the first case, the inverter uses the fixed IP address `10.10.100.254`, in the second case you have to look up the IP address in your router.
+
+The standard web interface of the inverter can be accessed at `http://<ip-adress>/index_cn.html`. Here you can set up the WLAN connection or change the password. The default user and password is `admin`/`admin`.
+
+For our purpose, the hidden URL `http://<ip-adress>/config_hide.html` should be called. There you can see and modify the parameters for accessing the cloud. Here we enter the IP address of our proxy and the IP port `10000` for the `Server A Setting` and for `Optional Server Setting`. The second entry is used as a backup in the event of connection problems.
+
+❗If the IP port is set to 10443 in the inverter configuration, you probably have a firmware with SSL support. In this case, you must use the client-mode configuration.
+
+If access to the web interface does not work, it can also be redirected via DNS redirection, as is necessary for the GEN3 inverters.
+
+## Client Mode (GEN3PLUS only)
+
+Newer GEN3PLUS inverters support SSL encrypted connections over port 10443 to the TSUN cloud. In this case you can't loop the proxy into this connection, since the certicate verification of the inverter don't allow this. You can configure the proxy in client-mode to establish an unencrypted connection to the inverter. For this porpuse the inverter listen on port `8899`.
+
+There are some requirements to be met:
+
+- the inverter should have a fixed IP
+- the proxy must be able to reach the inverter. You must configure a corresponding route in your router if the inverter and the proxy are in different IP networks
+- add a 'client_mode' line to your config.toml file, to specify the inverter's ip address
+
+## DNS Settings
+
+### Loop the proxy into the connection
+
+To include the proxy in the connection between the inverter and the TSUN Cloud, you must adapt the DNS record of *logger.talent-monitoring.com* within the network that your inverter uses. You need a mapping from logger.talent-monitoring.com to the IP address of the host running the Docker engine.
+
+The new GEN3 PLUS inverters use a different URL. Here, *iot.talent-monitoring.com* must be redirected.
+
+This can be done, for example, by adding a local DNS record to the Pi-hole if you are using it.
+
+### DNS Rebind Protection
+
+If you are using a router as local DNS server, the router may have DNS rebind protection that needs to be adjusted. For security reasons, DNS rebind protection blocks DNS queries that refer to an IP address on the local network.
+
+If you are using a FRITZ!Box, you can do this in the Network Settings tab under Home Network / Network. Add logger.talent-monitoring.com as a hostname exception in DNS rebind protection.
+
+### DNS server of proxy
+
+The proxy itself must use a different DNS server to connect to the TSUN Cloud. If you use the DNS server with the adapted record, you will end up in an endless loop as soon as the proxy tries to send data to the TSUN Cloud.
+
+As described above, set a DNS sever in the Docker command or Docker compose file.
+
+### Over The Air (OTA) firmware update
+
+Even if the proxy is connected between the inverter and the TSUN Cloud, an OTA update is supported. To do this, the inverter must be able to reach the website <http://www.talent-monitoring.com:9002/> in order to download images from there.
+
+It must be ensured that this address is not mapped to the proxy!
+
+# General Information
+
+## Compatibility
+
+In the following table you will find an overview of which inverter model has been tested for compatibility with which firmware version.
+A combination with a red question mark should work, but I have not checked it in detail.
+
+<table align="center">
+  <tr><th align="center">Micro Inverter Model</th><th align="center">Fw. 1.00.06</th><th align="center">Fw. 1.00.17</th><th align="center">Fw. 1.00.20</th><th align="center">Fw. 4.0.10</th><th align="center">Fw. 4.0.20</th></tr>
+  <tr><td>GEN3 micro inverters (single MPPT):<br>MS300, MS350, MS400<br>MS400-D</td><td align="center">❓</td><td align="center">❓</td><td align="center">❓</td><td align="center">➖</td><td align="center">➖</td></tr>
+  <tr><td>GEN3 micro inverters (dual MPPT):<br>MS600, MS700, MS800<br>MS600-D, MS800-D</td><td align="center">✔️</td><td align="center">✔️</td><td align="center">✔️</td><td align="center">➖</td><td align="center">➖</td></tr>
+  <tr><td>GEN3 PLUS micro inverters:<br>MS1600, MS1800, MS2000<br>MS2000-D</td><td align="center">➖</td><td align="center">➖</td><td align="center">➖</td><td align="center">✔️</td><td align="center">✔️</td></tr>
+  <tr><td>TITAN micro inverters:<br>TSOL-MP3000, MP2250, MS3000</td><td align="center">❓</td><td align="center">❓</td><td align="center">❓</td><td align="center">❓</td><td align="center">❓</td></tr>
+</table>
+
+```txt
+Legend
+➖: Firmware not available for this devices
+✔️: proxy support testet
+❓: proxy support possible but not testet
+🚧: Proxy support in preparation
+```
+
+❗The new inverters of the GEN3 Plus generation (e.g. MS-2000) use a completely different protocol for data transmission to the TSUN server. These inverters are supported from proxy version 0.6. The serial numbers of these inverters start with `Y17E` or `Y47E` instead of `R17E`
+
+If you have one of these combinations with a red question mark, it would be very nice if you could send me a proxy trace so that I can carry out the detailed checks and adjust the device and system tests. [Ask here how to send a trace](https://github.com/s-allius/tsun-gen3-proxy/discussions/categories/traces-for-compatibility-check)
+
 ## License
 
 This project is licensed under the [BSD 3-clause License](https://opensource.org/licenses/BSD-3-Clause).
@@ -138,7 +269,6 @@ Note the aiomqtt library used is based on the paho-mqtt library, which has a dua
 - One use of "COPYRIGHT OWNER" (EDL) instead of "COPYRIGHT HOLDER" (BSD)
 - One use of "Eclipse Foundation, Inc." (EDL) instead of "copyright holder" (BSD)
 
-
 ## Versioning
 
 This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). Breaking changes will only occur in major `X.0.0` releases.
@@ -150,4 +280,3 @@ We're very happy to receive contributions to this project! You can get started b
 ## Changelog
 
 The changelog lives in [CHANGELOG.md](https://github.com/s-allius/tsun-gen3-proxy/blob/main/CHANGELOG.md). It follows the principles of [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
-

app/.dockerignore

@@ -1,4 +1,5 @@
 tests/
 **/__pycache__
 *.pyc
-.DS_Store
+.DS_Store
+build.sh

app/Dockerfile

@@ -7,65 +7,73 @@ ARG GID=1000
 FROM python:3.12-alpine AS base
 USER root
 
-RUN apk update && \
-    apk upgrade
-RUN apk add --no-cache su-exec	
+COPY --chmod=0700 ./hardening_base.sh .
+RUN apk upgrade --no-cache && \
+    apk add --no-cache su-exec	&& \
+    ./hardening_base.sh && \
+    rm ./hardening_base.sh
 
 #
 # second stage for building wheels packages
-FROM base as builder
-
-RUN apk add --no-cache build-base && \
-    python -m pip install --no-cache-dir -U pip wheel
+FROM base AS builder
 
 # copy the dependencies file to the root dir and install requirements
 COPY ./requirements.txt /root/
-RUN python -OO -m pip wheel --no-cache-dir --wheel-dir=/root/wheels -r /root/requirements.txt
+RUN apk add --no-cache build-base && \
+    python -m pip install --no-cache-dir -U pip wheel && \
+    python -OO -m pip wheel --no-cache-dir --wheel-dir=/root/wheels -r /root/requirements.txt
 
 
 #
 # third stage for our runtime image
-FROM base as runtime
+FROM base AS runtime
 ARG SERVICE_NAME
 ARG VERSION
 ARG UID
 ARG GID
 ARG LOG_LVL
+ARG environment
 
 ENV VERSION=$VERSION
 ENV SERVICE_NAME=$SERVICE_NAME
 ENV UID=$UID
 ENV GID=$GID 
 ENV LOG_LVL=$LOG_LVL
+ENV HOME=/home/$SERVICE_NAME
 
     
 # set the working directory in the container
 WORKDIR /home/$SERVICE_NAME
 
-# update PATH environment variable
-ENV HOME=/home/$SERVICE_NAME
-
 VOLUME ["/home/$SERVICE_NAME/log", "/home/$SERVICE_NAME/config"]
 
-# install the requirements from the wheels packages from the builder stage
+# install the requirements from the wheels packages from the builder stage 
+# and unistall python packages and alpine package manger to reduce attack surface
 COPY --from=builder /root/wheels /root/wheels
+COPY --chmod=0700 ./hardening_final.sh .
 RUN python -m pip install --no-cache --no-index /root/wheels/* && \
-    rm -rf /root/wheels
+    rm -rf /root/wheels && \
+    python -m pip uninstall --yes setuptools wheel pip && \
+    apk --purge del apk-tools && \
+    ./hardening_final.sh && \
+    rm ./hardening_final.sh
+
 
 # copy the content of the local src and config directory to the working directory
 COPY --chmod=0700 entrypoint.sh /root/entrypoint.sh
 COPY config .
 COPY src .
-
-EXPOSE 5005
+RUN date > /build-date.txt
+EXPOSE 5005 8127 10000
 
 # command to run on container start
 ENTRYPOINT ["/root/entrypoint.sh"]
 CMD [ "python3", "./server.py" ]
 
 
+LABEL org.opencontainers.image.title="TSUN Gen3 Proxy"
 LABEL org.opencontainers.image.authors="Stefan Allius"
-LABEL org.opencontainers.image.source https://github.com/s-allius/tsun-gen3-proxy
-LABEL org.opencontainers.image.description 'The "TSUN Gen3 Micro-Inverter" proxy enables a reliable connection between TSUN third generation inverters and an MQTT broker to integrate the inverter into typical home automations'
+LABEL org.opencontainers.image.source=https://github.com/s-allius/tsun-gen3-proxy
+LABEL org.opencontainers.image.description='This proxy enables a reliable connection between TSUN third generation inverters (eg. TSOL MS600, MS800, MS2000) and an MQTT broker to integrate the inverter into typical home automations.'
 LABEL org.opencontainers.image.licenses="BSD-3-Clause"
 LABEL org.opencontainers.image.vendor="Stefan Allius"

app/build.sh

@@ -1,33 +1,75 @@
 #!/bin/bash
+# Usage: ./build.sh [dev|rc|rel]
+# dev: development build
+# rc: release candidate build
+# rel: release build and push to ghcr.io
+# Note: for release build, you need to set GHCR_TOKEN
+# export GHCR_TOKEN=<YOUR_GITHUB_TOKEN> in your .zprofile
+# see also: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry
+
 
 set -e
 
 BUILD_DATE=$(date -Iminutes)
+BRANCH=$(git rev-parse --abbrev-ref HEAD)
 VERSION=$(git describe --tags --abbrev=0)
 VERSION="${VERSION:1}"
 arr=(${VERSION//./ })
 MAJOR=${arr[0]}
 IMAGE=tsun-gen3-proxy
 
-if [[ $1 == dev ]] || [[ $1 == rc ]] ;then
+if [[ $1 == debug ]] || [[ $1 == dev ]] ;then
 IMAGE=docker.io/sallius/${IMAGE}
 VERSION=${VERSION}-$1
-elif [[ $1 == rel ]];then
+elif [[ $1 == rc ]] || [[ $1 == rel ]] || [[ $1 == preview ]] ;then
 IMAGE=ghcr.io/s-allius/${IMAGE}
 else
 echo argument missing!
-echo try: $0 '[dev|rc|rel]'
+echo try: $0 '[debug|dev|preview|rc|rel]'
 exit 1
 fi
 
 echo version: $VERSION  build-date: $BUILD_DATE   image: $IMAGE
-if [[ $1 == dev ]];then
-docker build --build-arg "VERSION=${VERSION}" --build-arg "LOG_LVL=DEBUG" --label "org.label-schema.build-date=${BUILD_DATE}" --label "org.opencontainers.image.version=${VERSION}" -t ${IMAGE}:latest app
+if [[ $1 == debug ]];then
+docker build --build-arg "VERSION=${VERSION}" --build-arg environment=dev --build-arg "LOG_LVL=DEBUG" --label "org.opencontainers.image.created=${BUILD_DATE}" --label "org.opencontainers.image.version=${VERSION}" --label "org.opencontainers.image.revision=${BRANCH}" -t ${IMAGE}:debug app
+echo " => pushing ${IMAGE}:debug"
+docker push -q ${IMAGE}:debug
+
+elif [[ $1 == dev ]];then
+docker build --build-arg "VERSION=${VERSION}" --build-arg environment=production --label "org.opencontainers.image.created=${BUILD_DATE}" --label "org.opencontainers.image.version=${VERSION}" --label "org.opencontainers.image.revision=${BRANCH}" -t ${IMAGE}:dev app
+echo " => pushing ${IMAGE}:dev"
+docker push -q ${IMAGE}:dev
+
+elif [[ $1 == preview ]];then
+docker build --build-arg "VERSION=${VERSION}" --build-arg environment=production --label "org.opencontainers.image.created=${BUILD_DATE}" --label "org.opencontainers.image.version=${VERSION}" --label "org.opencontainers.image.revision=${BRANCH}" -t ${IMAGE}:preview -t ${IMAGE}:${VERSION} app
+echo 'login to ghcr.io'    
+echo $GHCR_TOKEN | docker login ghcr.io -u s-allius --password-stdin
+echo " => pushing ${IMAGE}:preview"
+docker push -q ${IMAGE}:preview
+echo " => pushing ${IMAGE}:${VERSION}"
+docker push -q ${IMAGE}:${VERSION}
+
 elif [[ $1 == rc ]];then
-docker build --build-arg "VERSION=${VERSION}" --label "org.label-schema.build-date=${BUILD_DATE}" --label "org.opencontainers.image.version=${VERSION}" -t ${IMAGE}:latest app
+docker build --build-arg "VERSION=${VERSION}" --build-arg environment=production --label "org.opencontainers.image.created=${BUILD_DATE}" --label "org.opencontainers.image.version=${VERSION}" --label "org.opencontainers.image.revision=${BRANCH}" -t ${IMAGE}:rc -t ${IMAGE}:${VERSION} app
+echo 'login to ghcr.io'    
+echo $GHCR_TOKEN | docker login ghcr.io -u s-allius --password-stdin
+echo " => pushing ${IMAGE}:rc"
+docker push -q ${IMAGE}:rc
+echo " => pushing ${IMAGE}:${VERSION}"
+docker push -q ${IMAGE}:${VERSION}
+
 elif [[ $1 == rel ]];then
-docker build --no-cache --build-arg "VERSION=${VERSION}" --label "org.label-schema.build-date=${BUILD_DATE}" --label "org.opencontainers.image.version=${VERSION}" -t ${IMAGE}:latest -t ${IMAGE}:${MAJOR} -t ${IMAGE}:${VERSION} app
-docker push ghcr.io/s-allius/tsun-gen3-proxy:latest
-docker push ghcr.io/s-allius/tsun-gen3-proxy:${MAJOR}
-docker push ghcr.io/s-allius/tsun-gen3-proxy:${VERSION}
-fi
+docker build --no-cache --build-arg "VERSION=${VERSION}" --build-arg environment=production --label "org.opencontainers.image.created=${BUILD_DATE}" --label "org.opencontainers.image.version=${VERSION}" --label "org.opencontainers.image.revision=${BRANCH}" -t ${IMAGE}:latest -t ${IMAGE}:${MAJOR} -t ${IMAGE}:${VERSION} app
+echo 'login to ghcr.io'    
+echo $GHCR_TOKEN | docker login ghcr.io -u s-allius --password-stdin
+echo " => pushing ${IMAGE}:latest"
+docker push -q ${IMAGE}:latest
+echo " => pushing ${IMAGE}:${MAJOR}"
+docker push -q ${IMAGE}:${MAJOR}
+echo " => pushing ${IMAGE}:${VERSION}"
+docker push -q ${IMAGE}:${VERSION}
+fi
+
+echo ' => checking docker-compose.yaml file'
+docker-compose config -q
+echo 'done'

app/config/default_config.toml

@@ -3,6 +3,10 @@ tsun.enabled = true   # false: disables connecting to the tsun cloud, and avoids
 tsun.host    = 'logger.talent-monitoring.com'
 tsun.port    = 5005
 
+# configuration to reach the new tsun cloud for G3 Plus inverters
+solarman.enabled = true   # false: disables connecting to the tsun cloud, and avoids updates
+solarman.host    = 'iot.talent-monitoring.com'
+solarman.port    = 10000
 
 # mqtt broker configuration
 mqtt.host    = 'mqtt'   # URL or IP address of the mqtt broker
@@ -25,12 +29,33 @@ inverters.allow_all = true   # allow inverters, even if we have no inverter mapp
 #
 # for each inverter add a block starting with [inverters."<16-digit serial numbeer>"]
 [inverters."R170000000000001"]
-#node_id = ''          # Optional, MQTT replacement for inverters serial number  
-#suggested_area = ''  # Optional, suggested installation area for home-assistant
+#node_id = ''                 # Optional, MQTT replacement for inverters serial number  
+#suggested_area = ''          # Optional, suggested installation area for home-assistant
+#pv1 = {type = 'RSM40-8-395M', manufacturer = 'Risen'}   # Optional, PV module descr
+#pv2 = {type = 'RSM40-8-395M', manufacturer = 'Risen'}   # Optional, PV module descr
 
 #[inverters."R17xxxxxxxxxxxx2"]
-#node_id = ''          # Optional, MQTT replacement for inverters serial number  
-#suggested_area = ''  # Optional, suggested installation area for home-assistant
+#node_id = ''                 # Optional, MQTT replacement for inverters serial number  
+#suggested_area = ''          # Optional, suggested installation area for home-assistant
+#pv1 = {type = 'RSM40-8-405M', manufacturer = 'Risen'}   # Optional, PV module descr
+#pv2 = {type = 'RSM40-8-405M', manufacturer = 'Risen'}   # Optional, PV module descr
 
+[inverters."Y170000000000001"]
+monitor_sn = 2000000000       # The "Monitoring SN:" can be found on a sticker enclosed with the inverter
+#node_id = ''                 # Optional, MQTT replacement for inverters serial number  
+#suggested_area = ''          # Optional, suggested installation place for home-assistant
 
+# if your inverter supports SSL connections you must use the client_mode. Pls, uncomment
+# the next line and configure the fixed IP of your inverter
+#client_mode = {host = '192.168.0.1', port = 8899}  
 
+#pv1 = {type = 'RSM40-8-410M', manufacturer = 'Risen'}   # Optional, PV module descr
+#pv2 = {type = 'RSM40-8-410M', manufacturer = 'Risen'}   # Optional, PV module descr
+#pv3 = {type = 'RSM40-8-410M', manufacturer = 'Risen'}   # Optional, PV module descr
+#pv4 = {type = 'RSM40-8-410M', manufacturer = 'Risen'}   # Optional, PV module descr
+
+[gen3plus.at_acl]
+tsun.allow = ['AT+Z', 'AT+UPURL', 'AT+SUPDATE']
+tsun.block = []
+mqtt.allow = ['AT+']
+mqtt.block = []

app/entrypoint.sh

@@ -5,16 +5,19 @@ user="$(id -u)"
 echo "######################################################"
 echo "# prepare: '$SERVICE_NAME' Version:$VERSION"
 echo "# for running with UserID:$UID, GroupID:$GID"
+echo "# Image built: $(cat /build-date.txt) "
 echo "#"
 
 if [ "$user" = '0' ]; then
     mkdir -p /home/$SERVICE_NAME/log /home/$SERVICE_NAME/config
     
     if ! id $SERVICE_NAME &> /dev/null; then
+        echo "# create user"
         addgroup --gid $GID $SERVICE_NAME 2> /dev/null
         adduser -G $SERVICE_NAME -s /bin/false -D -H -g "" -u $UID $SERVICE_NAME
+    	chown -R $SERVICE_NAME:$SERVICE_NAME /home/$SERVICE_NAME || true
+        rm -fr /usr/sbin/addgroup /usr/sbin/adduser /bin/chown
     fi    
-	chown -R $SERVICE_NAME:$SERVICE_NAME /home/$SERVICE_NAME || true
     echo "######################################################"
     echo "#"
 

app/hardening_base.sh

@@ -0,0 +1,19 @@
+#!/bin/sh
+
+rm -fr /var/spool/cron
+rm -fr /etc/crontabs
+rm -fr /etc/periodic
+
+# Remove every user and group but root
+sed -i -r '/^(root)/!d' /etc/group
+sed -i -r '/^(root)/!d' /etc/passwd    
+
+# Remove init scripts since we do not use them.
+rm -fr /etc/inittab
+
+# Remove kernel tunables since we do not need them.
+rm -fr /etc/sysctl*
+rm -fr /etc/modprobe.d
+
+# Remove fstab since we do not need it.
+rm -f /etc/fstab

app/hardening_final.sh

@@ -0,0 +1,21 @@
+#!/bin/sh
+# For production images delete all uneeded admin commands and remove dangerous commands.
+# addgroup, adduser and chmod will be removed in entrypoint.sh during first start
+# su-exec will be needed for ever restart of the cotainer
+if [ "$environment" = "production" ] ; then \
+  find /sbin /usr/sbin  ! -type d \
+  -a ! -name addgroup \
+  -a ! -name adduser \
+  -a ! -name nologin \
+  -a ! -name su-exec \
+   -delete; \
+  find /bin /usr/bin -xdev \( \
+  -name chgrp -o \
+  -name chmod -o \
+  -name hexdump -o \
+  -name ln -o \
+  -name od -o \
+  -name strings -o \
+  -name su -o \
+  \) -delete  \
+; fi

app/proxy.svg

@@ -0,0 +1,411 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
+ "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by graphviz version 2.40.1 (20161225.0304)
+ -->
+<!-- Title: G Pages: 1 -->
+<svg width="720pt" height="1360pt"
+ viewBox="0.00 0.00 719.50 1360.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
+<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 1356)">
+<title>G</title>
+<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-1356 715.5,-1356 715.5,4 -4,4"/>
+<!-- A0 -->
+<g id="node1" class="node">
+<title>A0</title>
+<polygon fill="#fff8dc" stroke="#000000" points="153.6964,-1250 45.3036,-1250 45.3036,-1214 159.6964,-1214 159.6964,-1244 153.6964,-1250"/>
+<polyline fill="none" stroke="#000000" points="153.6964,-1250 153.6964,-1244 "/>
+<polyline fill="none" stroke="#000000" points="159.6964,-1244 153.6964,-1244 "/>
+<text text-anchor="middle" x="102.5" y="-1235" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">You can stick notes</text>
+<text text-anchor="middle" x="102.5" y="-1223" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">on diagrams too!</text>
+</g>
+<!-- A1 -->
+<g id="node2" class="node">
+<title>A1</title>
+<polygon fill="none" stroke="#000000" points="685.1817,-942 615.8183,-942 615.8183,-906 685.1817,-906 685.1817,-942"/>
+<text text-anchor="middle" x="650.5" y="-921" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">Singleton</text>
+</g>
+<!-- A2 -->
+<g id="node3" class="node">
+<title>A2</title>
+<polygon fill="none" stroke="#000000" points="589.5,-644 589.5,-676 711.5,-676 711.5,-644 589.5,-644"/>
+<text text-anchor="start" x="640.777" y="-657" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">Mqtt</text>
+<polygon fill="none" stroke="#000000" points="589.5,-588 589.5,-644 711.5,-644 711.5,-588 589.5,-588"/>
+<text text-anchor="start" x="607.9875" y="-625" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">&lt;static&gt;ha_restarts</text>
+<text text-anchor="start" x="615.7665" y="-613" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">&lt;static&gt;__client</text>
+<text text-anchor="start" x="599.3735" y="-601" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">&lt;static&gt;__cb_MqttIsUp</text>
+<polygon fill="none" stroke="#000000" points="589.5,-544 589.5,-588 711.5,-588 711.5,-544 589.5,-544"/>
+<text text-anchor="start" x="612.436" y="-569" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">&lt;async&gt;publish()</text>
+<text text-anchor="start" x="616.6045" y="-557" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">&lt;async&gt;close()</text>
+</g>
+<!-- A1&#45;&gt;A2 -->
+<g id="edge1" class="edge">
+<title>A1&#45;&gt;A2</title>
+<path fill="none" stroke="#000000" d="M650.5,-895.5395C650.5,-846.311 650.5,-744.0351 650.5,-676.2069"/>
+<polygon fill="none" stroke="#000000" points="647.0001,-895.7608 650.5,-905.7608 654.0001,-895.7608 647.0001,-895.7608"/>
+</g>
+<!-- A11 -->
+<g id="node12" class="node">
+<title>A11</title>
+<polygon fill="none" stroke="#000000" points="596.5,-348 596.5,-380 704.5,-380 704.5,-348 596.5,-348"/>
+<text text-anchor="start" x="633.5535" y="-361" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">Inverter</text>
+<polygon fill="none" stroke="#000000" points="596.5,-256 596.5,-348 704.5,-348 704.5,-256 596.5,-256"/>
+<text text-anchor="start" x="626.604" y="-329" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">cls.db_stat</text>
+<text text-anchor="start" x="619.9405" y="-317" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">cls.entity_prfx</text>
+<text text-anchor="start" x="610.7755" y="-305" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">cls.discovery_prfx</text>
+<text text-anchor="start" x="610.2115" y="-293" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">cls.proxy_node_id</text>
+<text text-anchor="start" x="606.3225" y="-281" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">cls.proxy_unique_id</text>
+<text text-anchor="start" x="622.1655" y="-269" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">cls.mqtt:Mqtt</text>
+<polygon fill="none" stroke="#000000" points="596.5,-236 596.5,-256 704.5,-256 704.5,-236 596.5,-236"/>
+</g>
+<!-- A2&#45;&gt;A11 -->
+<g id="edge13" class="edge">
+<title>A2&#45;&gt;A11</title>
+<path fill="none" stroke="#000000" d="M650.5,-543.7248C650.5,-495.3688 650.5,-429.8734 650.5,-380.1918"/>
+</g>
+<!-- A3 -->
+<g id="node4" class="node">
+<title>A3</title>
+<polygon fill="none" stroke="#000000" points="318.5,-402 318.5,-434 393.5,-434 393.5,-402 318.5,-402"/>
+<text text-anchor="start" x="338.2175" y="-415" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">Modbus</text>
+<polygon fill="none" stroke="#000000" points="318.5,-250 318.5,-402 393.5,-402 393.5,-250 318.5,-250"/>
+<text text-anchor="start" x="347.6615" y="-383" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">que</text>
+<text text-anchor="start" x="328.49" y="-359" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">snd_handler</text>
+<text text-anchor="start" x="329.605" y="-347" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">rsp_handler</text>
+<text text-anchor="start" x="339.6085" y="-335" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">timeout</text>
+<text text-anchor="start" x="329.8895" y="-323" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">max_retires</text>
+<text text-anchor="start" x="337.942" y="-311" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">last_xxx</text>
+<text text-anchor="start" x="349.8915" y="-299" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">err</text>
+<text text-anchor="start" x="336.5535" y="-287" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">retry_cnt</text>
+<text text-anchor="start" x="334.879" y="-275" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">req_pend</text>
+<text text-anchor="start" x="349.3365" y="-263" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">tim</text>
+<polygon fill="none" stroke="#000000" points="318.5,-182 318.5,-250 393.5,-250 393.5,-182 318.5,-182"/>
+<text text-anchor="start" x="329.89" y="-231" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">build_msg()</text>
+<text text-anchor="start" x="333.224" y="-219" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">recv_req()</text>
+<text text-anchor="start" x="330.724" y="-207" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">recv_resp()</text>
+<text text-anchor="start" x="341.0025" y="-195" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">close()</text>
+</g>
+<!-- A4 -->
+<g id="node5" class="node">
+<title>A4</title>
+<polygon fill="none" stroke="#000000" points="308.5,-1242 308.5,-1274 379.5,-1274 379.5,-1242 308.5,-1242"/>
+<text text-anchor="start" x="318.445" y="-1255" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">IterRegistry</text>
+<polygon fill="none" stroke="#000000" points="308.5,-1222 308.5,-1242 379.5,-1242 379.5,-1222 308.5,-1222"/>
+<polygon fill="none" stroke="#000000" points="308.5,-1190 308.5,-1222 379.5,-1222 379.5,-1190 308.5,-1190"/>
+<text text-anchor="start" x="325.939" y="-1203" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">__iter__</text>
+</g>
+<!-- A5 -->
+<g id="node6" class="node">
+<title>A5</title>
+<polygon fill="none" stroke="#000000" points="276.5,-1030 276.5,-1062 410.5,-1062 410.5,-1030 276.5,-1030"/>
+<text text-anchor="start" x="323.2175" y="-1043" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">Message</text>
+<polygon fill="none" stroke="#000000" points="276.5,-854 276.5,-1030 410.5,-1030 410.5,-854 276.5,-854"/>
+<text text-anchor="start" x="306.8265" y="-1011" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">server_side:bool</text>
+<text text-anchor="start" x="304.043" y="-999" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">header_valid:bool</text>
+<text text-anchor="start" x="296.814" y="-987" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">header_len:unsigned</text>
+<text text-anchor="start" x="302.648" y="-975" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">data_len:unsigned</text>
+<text text-anchor="start" x="321.8245" y="-963" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">unique_id</text>
+<text text-anchor="start" x="325.7135" y="-951" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">node_id</text>
+<text text-anchor="start" x="322.6585" y="-939" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">sug_area</text>
+<text text-anchor="start" x="293.489" y="-927" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">_recv_buffer:bytearray</text>
+<text text-anchor="start" x="292.0945" y="-915" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">_send_buffer:bytearray</text>
+<text text-anchor="start" x="286.2665" y="-903" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">_forward_buffer:bytearray</text>
+<text text-anchor="start" x="325.7135" y="-891" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">db:Infos</text>
+<text text-anchor="start" x="314.326" y="-879" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">new_data:list</text>
+<text text-anchor="start" x="332.662" y="-867" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">state</text>
+<polygon fill="none" stroke="#000000" points="276.5,-786 276.5,-854 410.5,-854 410.5,-786 276.5,-786"/>
+<text text-anchor="start" x="293.2095" y="-835" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">_read():void&lt;abstract&gt;</text>
+<text text-anchor="start" x="317.9445" y="-823" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">close():void</text>
+<text text-anchor="start" x="303.7725" y="-811" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">inc_counter():void</text>
+<text text-anchor="start" x="302.1025" y="-799" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">dec_counter():void</text>
+</g>
+<!-- A4&#45;&gt;A5 -->
+<g id="edge2" class="edge">
+<title>A4&#45;&gt;A5</title>
+<path fill="none" stroke="#000000" d="M343.5,-1179.6793C343.5,-1147.2188 343.5,-1103.8616 343.5,-1062.0836"/>
+<polygon fill="none" stroke="#000000" points="340.0001,-1179.8197 343.5,-1189.8197 347.0001,-1179.8198 340.0001,-1179.8197"/>
+</g>
+<!-- A6 -->
+<g id="node7" class="node">
+<title>A6</title>
+<polygon fill="none" stroke="#000000" points="415.5,-704 415.5,-736 529.5,-736 529.5,-704 415.5,-704"/>
+<text text-anchor="start" x="458.608" y="-717" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">Talent</text>
+<polygon fill="none" stroke="#000000" points="415.5,-600 415.5,-704 529.5,-704 529.5,-600 415.5,-600"/>
+<text text-anchor="start" x="425.263" y="-685" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">await_conn_resp_cnt</text>
+<text text-anchor="start" x="460.2775" y="-673" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">id_str</text>
+<text text-anchor="start" x="441.1" y="-661" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">contact_name</text>
+<text text-anchor="start" x="444.44" y="-649" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">contact_mail</text>
+<text text-anchor="start" x="448.0445" y="-637" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">db:InfosG3</text>
+<text text-anchor="start" x="446.384" y="-625" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">mb:Modbus</text>
+<text text-anchor="start" x="458.612" y="-613" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">switch</text>
+<polygon fill="none" stroke="#000000" points="415.5,-484 415.5,-600 529.5,-600 529.5,-484 415.5,-484"/>
+<text text-anchor="start" x="429.9925" y="-581" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">msg_contact_info()</text>
+<text text-anchor="start" x="431.9325" y="-569" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">msg_ota_update()</text>
+<text text-anchor="start" x="437.7765" y="-557" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">msg_get_time()</text>
+<text text-anchor="start" x="425.8285" y="-545" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">msg_collector_data()</text>
+<text text-anchor="start" x="427.7735" y="-533" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">msg_inverter_data()</text>
+<text text-anchor="start" x="436.9405" y="-521" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">msg_unknown()</text>
+<text text-anchor="start" x="457.5025" y="-497" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">close()</text>
+</g>
+<!-- A5&#45;&gt;A6 -->
+<g id="edge3" class="edge">
+<title>A5&#45;&gt;A6</title>
+<path fill="none" stroke="#000000" d="M404.0814,-776.5383C409.5999,-763.1056 415.1569,-749.5794 420.5898,-736.355"/>
+<polygon fill="none" stroke="#000000" points="400.8317,-775.2382 400.269,-785.8181 407.3066,-777.8983 400.8317,-775.2382"/>
+</g>
+<!-- A7 -->
+<g id="node8" class="node">
+<title>A7</title>
+<polygon fill="none" stroke="#000000" points="172.5,-668 172.5,-700 263.5,-700 263.5,-668 172.5,-668"/>
+<text text-anchor="start" x="190.495" y="-681" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">SolarmanV5</text>
+<polygon fill="none" stroke="#000000" points="172.5,-576 172.5,-668 263.5,-668 263.5,-576 172.5,-576"/>
+<text text-anchor="start" x="202.998" y="-649" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">control</text>
+<text text-anchor="start" x="206.0575" y="-637" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">serial</text>
+<text text-anchor="start" x="211.056" y="-625" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">snr</text>
+<text text-anchor="start" x="190.21" y="-613" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">db:InfosG3P</text>
+<text text-anchor="start" x="191.884" y="-601" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">mb:Modbus</text>
+<text text-anchor="start" x="204.112" y="-589" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">switch</text>
+<polygon fill="none" stroke="#000000" points="172.5,-520 172.5,-576 263.5,-576 263.5,-520 172.5,-520"/>
+<text text-anchor="start" x="182.4405" y="-557" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">msg_unknown()</text>
+<text text-anchor="start" x="203.0025" y="-533" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">close()</text>
+</g>
+<!-- A5&#45;&gt;A7 -->
+<g id="edge4" class="edge">
+<title>A5&#45;&gt;A7</title>
+<path fill="none" stroke="#000000" d="M284.228,-776.2903C273.8281,-750.3733 263.2923,-724.1174 253.7595,-700.3611"/>
+<polygon fill="none" stroke="#000000" points="281.0788,-777.8409 288.0512,-785.8181 287.5753,-775.2339 281.0788,-777.8409"/>
+</g>
+<!-- A6&#45;&gt;A3 -->
+<g id="edge6" class="edge">
+<title>A6&#45;&gt;A3</title>
+<path fill="none" stroke="#000000" d="M420.9917,-483.781C414.3472,-467.074 407.7026,-450.1475 401.5,-434 399.8828,-429.7898 398.247,-425.4956 396.6047,-421.154"/>
+<polygon fill="#000000" stroke="#000000" points="393.0037,-411.5882 400.7383,-419.3616 394.7652,-416.2676 396.5268,-420.947 396.5268,-420.947 396.5268,-420.947 394.7652,-416.2676 392.3154,-422.5325 393.0037,-411.5882 393.0037,-411.5882"/>
+<text text-anchor="middle" x="407.3001" y="-422.5743" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">1</text>
+<text text-anchor="middle" x="406.4454" y="-467.0549" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">has</text>
+</g>
+<!-- A8 -->
+<g id="node9" class="node">
+<title>A8</title>
+<polygon fill="none" stroke="#000000" points="410.5,-330 410.5,-362 560.5,-362 560.5,-330 410.5,-330"/>
+<text text-anchor="start" x="453.5455" y="-343" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">ConnectionG3</text>
+<polygon fill="none" stroke="#000000" points="410.5,-298 410.5,-330 560.5,-330 560.5,-298 410.5,-298"/>
+<text text-anchor="start" x="420.487" y="-311" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">remoteStream:ConnectionG3</text>
+<polygon fill="none" stroke="#000000" points="410.5,-254 410.5,-298 560.5,-298 560.5,-254 410.5,-254"/>
+<text text-anchor="start" x="466.054" y="-279" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">healthy()</text>
+<text text-anchor="start" x="470.5025" y="-267" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">close()</text>
+</g>
+<!-- A6&#45;&gt;A8 -->
+<g id="edge5" class="edge">
+<title>A6&#45;&gt;A8</title>
+<path fill="none" stroke="#000000" d="M478.3685,-473.6691C480.0687,-434.1731 481.827,-393.3258 483.1723,-362.0732"/>
+<polygon fill="none" stroke="#000000" points="474.8712,-473.5333 477.9378,-483.6747 481.8648,-473.8345 474.8712,-473.5333"/>
+</g>
+<!-- A7&#45;&gt;A3 -->
+<g id="edge8" class="edge">
+<title>A7&#45;&gt;A3</title>
+<path fill="none" stroke="#000000" d="M254.1161,-519.7083C259.8714,-507.5039 266.0613,-495.3029 272.5,-484 286.0537,-460.2067 295.6001,-458.1541 308.5,-434 310.2529,-430.7178 311.9697,-427.3559 313.6482,-423.937"/>
+<polygon fill="#000000" stroke="#000000" points="317.9998,-414.7692 317.7769,-425.7328 315.8557,-419.2862 313.7116,-423.8032 313.7116,-423.8032 313.7116,-423.8032 315.8557,-419.2862 309.6463,-421.8735 317.9998,-414.7692 317.9998,-414.7692"/>
+<text text-anchor="middle" x="317.8629" y="-431.7687" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">1</text>
+<text text-anchor="middle" x="254.262" y="-496.7088" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">has</text>
+</g>
+<!-- A9 -->
+<g id="node10" class="node">
+<title>A9</title>
+<polygon fill="none" stroke="#000000" points="125.5,-330 125.5,-362 281.5,-362 281.5,-330 125.5,-330"/>
+<text text-anchor="start" x="168.211" y="-343" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">ConnectionG3P</text>
+<polygon fill="none" stroke="#000000" points="125.5,-298 125.5,-330 281.5,-330 281.5,-298 125.5,-298"/>
+<text text-anchor="start" x="135.1525" y="-311" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">remoteStream:ConnectionG3P</text>
+<polygon fill="none" stroke="#000000" points="125.5,-254 125.5,-298 281.5,-298 281.5,-254 125.5,-254"/>
+<text text-anchor="start" x="184.054" y="-279" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">healthy()</text>
+<text text-anchor="start" x="188.5025" y="-267" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">close()</text>
+</g>
+<!-- A7&#45;&gt;A9 -->
+<g id="edge7" class="edge">
+<title>A7&#45;&gt;A9</title>
+<path fill="none" stroke="#000000" d="M212.8528,-509.7531C210.5717,-460.5471 207.9134,-403.2043 206.0152,-362.2565"/>
+<polygon fill="none" stroke="#000000" points="209.3591,-509.972 213.3185,-519.7991 216.3516,-509.6477 209.3591,-509.972"/>
+</g>
+<!-- A8&#45;&gt;A8 -->
+<g id="edge15" class="edge">
+<title>A8&#45;&gt;A8</title>
+<path fill="none" stroke="#000000" d="M560.6684,-348.7195C571.3394,-342.1337 578.5,-328.5605 578.5,-308 578.5,-292.9008 574.6382,-281.57 568.3604,-274.0076"/>
+<polygon fill="#000000" stroke="#000000" points="560.6684,-267.2805 571.1583,-270.4763 564.4321,-270.5721 568.1958,-273.8637 568.1958,-273.8637 568.1958,-273.8637 564.4321,-270.5721 565.2334,-277.251 560.6684,-267.2805 560.6684,-267.2805"/>
+<text text-anchor="middle" x="579.877" y="-269.8507" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">0..1</text>
+<text text-anchor="middle" x="570.593" y="-328.3557" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">has</text>
+</g>
+<!-- A12 -->
+<g id="node13" class="node">
+<title>A12</title>
+<polygon fill="none" stroke="#000000" points="506.5,-100 506.5,-132 628.5,-132 628.5,-100 506.5,-100"/>
+<text text-anchor="start" x="543.8845" y="-113" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">InverterG3</text>
+<polygon fill="none" stroke="#000000" points="506.5,-68 506.5,-100 628.5,-100 628.5,-68 506.5,-68"/>
+<text text-anchor="start" x="536.9355" y="-81" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">__ha_restarts</text>
+<polygon fill="none" stroke="#000000" points="506.5,0 506.5,-68 628.5,-68 628.5,0 506.5,0"/>
+<text text-anchor="start" x="516.1035" y="-49" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">async_create_remote()</text>
+<text text-anchor="start" x="526.382" y="-37" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">async_publ_mqtt()</text>
+<text text-anchor="start" x="552.5025" y="-13" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">close()</text>
+</g>
+<!-- A8&#45;&gt;A12 -->
+<g id="edge14" class="edge">
+<title>A8&#45;&gt;A12</title>
+<path fill="none" stroke="#000000" d="M507.022,-244.4839C518.719,-209.9635 533.1714,-167.3112 545.0148,-132.3588"/>
+<polygon fill="none" stroke="#000000" points="503.6947,-243.3974 503.8003,-253.9917 510.3245,-245.6439 503.6947,-243.3974"/>
+</g>
+<!-- A9&#45;&gt;A9 -->
+<g id="edge17" class="edge">
+<title>A9&#45;&gt;A9</title>
+<path fill="none" stroke="#000000" d="M281.8471,-348.2542C292.4443,-341.506 299.5,-328.0879 299.5,-308 299.5,-293.248 295.6948,-282.093 289.4763,-274.5351"/>
+<polygon fill="#000000" stroke="#000000" points="281.8471,-267.7458 292.3089,-271.0321 285.5822,-271.0697 289.3174,-274.3937 289.3174,-274.3937 289.3174,-274.3937 285.5822,-271.0697 286.3258,-277.7553 281.8471,-267.7458 281.8471,-267.7458"/>
+<text text-anchor="middle" x="301.0069" y="-270.4817" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">0..1</text>
+<text text-anchor="middle" x="291.5637" y="-327.7732" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">has</text>
+</g>
+<!-- A13 -->
+<g id="node14" class="node">
+<title>A13</title>
+<polygon fill="none" stroke="#000000" points="144.5,-94 144.5,-126 263.5,-126 263.5,-94 144.5,-94"/>
+<text text-anchor="start" x="177.05" y="-107" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">InverterG3P</text>
+<polygon fill="none" stroke="#000000" points="144.5,-62 144.5,-94 263.5,-94 263.5,-62 144.5,-62"/>
+<text text-anchor="start" x="173.4355" y="-75" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">__ha_restarts</text>
+<polygon fill="none" stroke="#000000" points="144.5,-6 144.5,-62 263.5,-62 263.5,-6 144.5,-6"/>
+<text text-anchor="start" x="154.268" y="-43" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">async_create_remote(</text>
+<text text-anchor="start" x="161.2175" y="-31" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">)async_publ_mqtt()</text>
+<text text-anchor="start" x="189.0025" y="-19" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">close()</text>
+</g>
+<!-- A9&#45;&gt;A13 -->
+<g id="edge16" class="edge">
+<title>A9&#45;&gt;A13</title>
+<path fill="none" stroke="#000000" d="M203.5,-243.955C203.5,-207.4743 203.5,-162.045 203.5,-126.2187"/>
+<polygon fill="none" stroke="#000000" points="200.0001,-243.9917 203.5,-253.9917 207.0001,-243.9917 200.0001,-243.9917"/>
+</g>
+<!-- A10 -->
+<g id="node11" class="node">
+<title>A10</title>
+<polygon fill="none" stroke="#000000" points="281.5,-698 281.5,-730 397.5,-730 397.5,-698 281.5,-698"/>
+<text text-anchor="start" x="309.774" y="-711" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">AsyncStream</text>
+<polygon fill="none" stroke="#000000" points="281.5,-618 281.5,-698 397.5,-698 397.5,-618 281.5,-618"/>
+<text text-anchor="start" x="325.053" y="-679" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">reader</text>
+<text text-anchor="start" x="327.283" y="-667" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">writer</text>
+<text text-anchor="start" x="329.497" y="-655" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">addr</text>
+<text text-anchor="start" x="325.053" y="-643" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">r_addr</text>
+<text text-anchor="start" x="325.608" y="-631" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">l_addr</text>
+<polygon fill="none" stroke="#000000" points="281.5,-490 281.5,-618 397.5,-618 397.5,-490 281.5,-490"/>
+<text text-anchor="start" x="291.1575" y="-599" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">&lt;async&gt;server_loop()</text>
+<text text-anchor="start" x="293.378" y="-587" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">&lt;async&gt;client_loop()</text>
+<text text-anchor="start" x="311.154" y="-575" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">&lt;async&gt;loop</text>
+<text text-anchor="start" x="327.282" y="-563" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">disc()</text>
+<text text-anchor="start" x="324.5025" y="-551" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">close()</text>
+<text text-anchor="start" x="304.7705" y="-527" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">__async_read()</text>
+<text text-anchor="start" x="309.78" y="-515" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">async_write()</text>
+<text text-anchor="start" x="298.107" y="-503" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">__async_forward()</text>
+</g>
+<!-- A10&#45;&gt;A8 -->
+<g id="edge9" class="edge">
+<title>A10&#45;&gt;A8</title>
+<path fill="none" stroke="#000000" d="M402.0319,-480.6532C422.1536,-439.0316 443.3588,-395.1687 459.3606,-362.0691"/>
+<polygon fill="none" stroke="#000000" points="398.824,-479.2474 397.6226,-489.7739 405.1262,-482.2941 398.824,-479.2474"/>
+</g>
+<!-- A10&#45;&gt;A9 -->
+<g id="edge10" class="edge">
+<title>A10&#45;&gt;A9</title>
+<path fill="none" stroke="#000000" d="M281.2511,-480.6532C262.5076,-439.0316 242.7548,-395.1687 227.849,-362.0691"/>
+<polygon fill="none" stroke="#000000" points="278.0609,-482.0929 285.3584,-489.7739 284.4435,-479.2186 278.0609,-482.0929"/>
+</g>
+<!-- A11&#45;&gt;A12 -->
+<g id="edge11" class="edge">
+<title>A11&#45;&gt;A12</title>
+<path fill="none" stroke="#000000" d="M622.3544,-225.9369C611.8702,-195.3687 600.1133,-161.0894 590.181,-132.1301"/>
+<polygon fill="none" stroke="#000000" points="619.1547,-227.3962 625.7097,-235.7198 625.7761,-225.1252 619.1547,-227.3962"/>
+</g>
+<!-- A11&#45;&gt;A13 -->
+<g id="edge12" class="edge">
+<title>A11&#45;&gt;A13</title>
+<path fill="none" stroke="#000000" d="M622.0673,-226.7211C613.147,-210.1001 601.7805,-194.0346 587.5,-182 538.0407,-140.3192 358.189,-98.0533 263.2057,-77.9953"/>
+<polygon fill="none" stroke="#000000" points="619.1257,-228.6587 626.7743,-235.9901 625.367,-225.4892 619.1257,-228.6587"/>
+</g>
+<!-- A14 -->
+<g id="node15" class="node">
+<title>A14</title>
+<polygon fill="none" stroke="#000000" points="178.5,-1320 178.5,-1352 281.5,-1352 281.5,-1320 178.5,-1320"/>
+<text text-anchor="start" x="219.162" y="-1333" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">Infos</text>
+<polygon fill="none" stroke="#000000" points="178.5,-1264 178.5,-1320 281.5,-1320 281.5,-1264 178.5,-1264"/>
+<text text-anchor="start" x="221.9415" y="-1301" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">stat</text>
+<text text-anchor="start" x="197.486" y="-1289" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">new_stat_data</text>
+<text text-anchor="start" x="211.1035" y="-1277" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">info_dev</text>
+<polygon fill="none" stroke="#000000" points="178.5,-1112 178.5,-1264 281.5,-1264 281.5,-1112 178.5,-1112"/>
+<text text-anchor="start" x="205.8355" y="-1245" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">static_init()</text>
+<text text-anchor="start" x="203.8845" y="-1233" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">dev_value()</text>
+<text text-anchor="start" x="200.8305" y="-1221" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">inc_counter()</text>
+<text text-anchor="start" x="199.1605" y="-1209" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">dec_counter()</text>
+<text text-anchor="start" x="197.21" y="-1197" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">ha_proxy_conf</text>
+<text text-anchor="start" x="212.213" y="-1185" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">ha_conf</text>
+<text text-anchor="start" x="204.994" y="-1173" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">ha_remove</text>
+<text text-anchor="start" x="206.3745" y="-1161" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">update_db</text>
+<text text-anchor="start" x="190.537" y="-1149" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">set_db_def_value</text>
+<text text-anchor="start" x="199.9855" y="-1137" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">get_db_value</text>
+<text text-anchor="start" x="188.3225" y="-1125" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">ignore_this_device</text>
+</g>
+<!-- A15 -->
+<g id="node16" class="node">
+<title>A15</title>
+<polygon fill="none" stroke="#000000" points="431.5,-940 431.5,-972 498.5,-972 498.5,-940 431.5,-940"/>
+<text text-anchor="start" x="447.493" y="-953" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">InfosG3</text>
+<polygon fill="none" stroke="#000000" points="431.5,-920 431.5,-940 498.5,-940 498.5,-920 431.5,-920"/>
+<polygon fill="none" stroke="#000000" points="431.5,-876 431.5,-920 498.5,-920 498.5,-876 431.5,-876"/>
+<text text-anchor="start" x="441.384" y="-901" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">ha_confs()</text>
+<text text-anchor="start" x="449.168" y="-889" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">parse()</text>
+</g>
+<!-- A14&#45;&gt;A15 -->
+<g id="edge18" class="edge">
+<title>A14&#45;&gt;A15</title>
+<path fill="none" stroke="#000000" d="M287.6238,-1123.7067C291.4001,-1119.5529 295.359,-1115.6229 299.5,-1112 342.985,-1073.9563 380.3024,-1104.4478 419.5,-1062 442.1524,-1037.4693 453.4109,-1001.3633 459.0018,-972.2357"/>
+<polygon fill="none" stroke="#000000" points="284.8741,-1121.5366 281.0238,-1131.4071 290.1891,-1126.0921 284.8741,-1121.5366"/>
+</g>
+<!-- A16 -->
+<g id="node17" class="node">
+<title>A16</title>
+<polygon fill="none" stroke="#000000" points="187.5,-940 187.5,-972 254.5,-972 254.5,-940 187.5,-940"/>
+<text text-anchor="start" x="200.1585" y="-953" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">InfosG3P</text>
+<polygon fill="none" stroke="#000000" points="187.5,-920 187.5,-940 254.5,-940 254.5,-920 187.5,-920"/>
+<polygon fill="none" stroke="#000000" points="187.5,-876 187.5,-920 254.5,-920 254.5,-876 187.5,-876"/>
+<text text-anchor="start" x="197.384" y="-901" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">ha_confs()</text>
+<text text-anchor="start" x="205.168" y="-889" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">parse()</text>
+</g>
+<!-- A14&#45;&gt;A16 -->
+<g id="edge19" class="edge">
+<title>A14&#45;&gt;A16</title>
+<path fill="none" stroke="#000000" d="M225.6878,-1101.5366C224.3454,-1055.5988 222.9195,-1006.7991 221.9029,-972.012"/>
+<polygon fill="none" stroke="#000000" points="222.191,-1101.7024 225.9817,-1111.5959 229.188,-1101.4979 222.191,-1101.7024"/>
+</g>
+<!-- A15&#45;&gt;A6 -->
+<g id="edge21" class="edge">
+<title>A15&#45;&gt;A6</title>
+<path fill="none" stroke="#000000" d="M465.7237,-875.9684C466.6086,-841.2366 467.8512,-792.4655 469.031,-746.1572"/>
+<polygon fill="#000000" stroke="#000000" points="469.2896,-736.0098 473.5333,-746.1212 469.1622,-741.0082 469.0348,-746.0066 469.0348,-746.0066 469.0348,-746.0066 469.1622,-741.0082 464.5362,-745.8919 469.2896,-736.0098 469.2896,-736.0098"/>
+</g>
+<!-- A16&#45;&gt;A7 -->
+<g id="edge20" class="edge">
+<title>A16&#45;&gt;A7</title>
+<path fill="none" stroke="#000000" d="M220.0411,-875.9684C219.6216,-832.0581 218.9877,-765.7079 218.4579,-710.2644"/>
+<polygon fill="#000000" stroke="#000000" points="218.3603,-700.0467 222.9557,-710.0032 218.4081,-705.0465 218.456,-710.0463 218.456,-710.0463 218.456,-710.0463 218.4081,-705.0465 213.9562,-710.0893 218.3603,-700.0467 218.3603,-700.0467"/>
+</g>
+<!-- A17 -->
+<g id="node18" class="node">
+<title>A17</title>
+<polygon fill="none" stroke="#000000" points=".5,-336 .5,-368 107.5,-368 107.5,-336 .5,-336"/>
+<text text-anchor="start" x="24.2695" y="-349" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">ModbusConn</text>
+<polygon fill="none" stroke="#000000" points=".5,-268 .5,-336 107.5,-336 107.5,-268 .5,-268"/>
+<text text-anchor="start" x="44.5515" y="-317" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">host</text>
+<text text-anchor="start" x="45.387" y="-305" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">port</text>
+<text text-anchor="start" x="43.997" y="-293" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">addr</text>
+<text text-anchor="start" x="10.383" y="-281" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">stream:InverterG3P</text>
+<polygon fill="none" stroke="#000000" points=".5,-248 .5,-268 107.5,-268 107.5,-248 .5,-248"/>
+</g>
+<!-- A17&#45;&gt;A13 -->
+<g id="edge22" class="edge">
+<title>A17&#45;&gt;A13</title>
+<path fill="none" stroke="#000000" d="M80.8473,-247.8342C91.2165,-226.5814 103.6422,-202.8044 116.5,-182 126.2708,-166.1905 137.6417,-149.852 148.8772,-134.6044"/>
+<polygon fill="#000000" stroke="#000000" points="155.0942,-126.2561 152.7306,-136.9642 152.1078,-130.2663 149.1214,-134.2765 149.1214,-134.2765 149.1214,-134.2765 152.1078,-130.2663 145.5123,-131.5887 155.0942,-126.2561 155.0942,-126.2561"/>
+<text text-anchor="middle" x="151.047" y="-142.8423" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">1</text>
+<text text-anchor="middle" x="81.2636" y="-224.8385" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">has</text>
+</g>
+</g>
+</svg>

app/proxy.yuml

@@ -0,0 +1,30 @@
+// {type:class}
+// {direction:topDown}
+// {generate:true}
+
+[note: You can stick notes on diagrams too!{bg:cornsilk}]
+[Singleton]^[Mqtt|<static>ha_restarts;<static>__client;<static>__cb_MqttIsUp|<async>publish();<async>close()]
+[Modbus|que;;snd_handler;rsp_handler;timeout;max_retires;last_xxx;err;retry_cnt;req_pend;tim|build_msg();recv_req();recv_resp();close()]
+[IterRegistry||__iter__]^[Message|server_side:bool;header_valid:bool;header_len:unsigned;data_len:unsigned;unique_id;node_id;sug_area;_recv_buffer:bytearray;_send_buffer:bytearray;_forward_buffer:bytearray;db:Infos;new_data:list;state|_read():void<abstract>;close():void;inc_counter():void;dec_counter():void]
+[Message]^[Talent|await_conn_resp_cnt;id_str;contact_name;contact_mail;db:InfosG3;mb:Modbus;switch|msg_contact_info();msg_ota_update();msg_get_time();msg_collector_data();msg_inverter_data();msg_unknown();;close()]
+[Message]^[SolarmanV5|control;serial;snr;db:InfosG3P;mb:Modbus;switch|msg_unknown();;close()]
+[Talent]^[ConnectionG3|remoteStream:ConnectionG3|healthy();close()]
+[Talent]has-1>[Modbus]
+[SolarmanV5]^[ConnectionG3P|remoteStream:ConnectionG3P|healthy();close()]
+[SolarmanV5]has-1>[Modbus]
+[AsyncStream|reader;writer;addr;r_addr;l_addr|<async>server_loop();<async>client_loop();<async>loop;disc();close();;__async_read();async_write();__async_forward()]^[ConnectionG3]
+[AsyncStream]^[ConnectionG3P]
+[Inverter|cls.db_stat;cls.entity_prfx;cls.discovery_prfx;cls.proxy_node_id;cls.proxy_unique_id;cls.mqtt:Mqtt|]^[InverterG3|__ha_restarts|async_create_remote();async_publ_mqtt();;close()]
+[Inverter]^[InverterG3P|__ha_restarts|async_create_remote(;)async_publ_mqtt();close()]
+[Mqtt]-[Inverter]
+[ConnectionG3]^[InverterG3]
+[ConnectionG3]has-0..1>[ConnectionG3]
+[ConnectionG3P]^[InverterG3P]
+[ConnectionG3P]has-0..1>[ConnectionG3P]
+
+[Infos|stat;new_stat_data;info_dev|static_init();dev_value();inc_counter();dec_counter();ha_proxy_conf;ha_conf;ha_remove;update_db;set_db_def_value;get_db_value;ignore_this_device]^[InfosG3||ha_confs();parse()]
+[Infos]^[InfosG3P||ha_confs();parse()]
+[InfosG3P]->[SolarmanV5]
+[InfosG3]->[Talent]
+[ModbusConn|host;port;addr;stream:InverterG3P;|]has-1>[InverterG3P]
+

app/requirements.txt

@@ -1,2 +1,4 @@
-    aiomqtt==1.2.1
-    schema==0.7.5
+    aiomqtt==2.2.0
+    schema==0.7.7
+    aiocron==1.8
+    aiohttp==3.9.5

app/src/async_stream.py

@@ -1,98 +1,248 @@
-import logging, traceback
-from config import Config
-#import gc
-from messages import Message, hex_dump_memory
+import asyncio
+import logging
+import traceback
+import time
+from asyncio import StreamReader, StreamWriter
+from messages import hex_dump_memory, State
+from typing import Self
+from itertools import count
 
+import gc
 logger = logging.getLogger('conn')
 
-class AsyncStream(Message):
 
-    def __init__(self, reader, writer, addr, remote_stream, server_side: bool) -> None:
-        super().__init__()
+class AsyncStream():
+    _ids = count(0)
+    MAX_PROC_TIME = 2
+    '''maximum processing time for a received msg in sec'''
+    MAX_START_TIME = 400
+    '''maximum time without a received msg in sec'''
+    MAX_INV_IDLE_TIME = 90
+    '''maximum time without a received msg from the inverter in sec'''
+    MAX_CLOUD_IDLE_TIME = 360
+    '''maximum time without a received msg from cloud side in sec'''
+
+    def __init__(self, reader: StreamReader, writer: StreamWriter,
+                 addr) -> None:
+        logger.debug('AsyncStream.__init__')
         self.reader = reader
         self.writer = writer
-        self.remoteStream = remote_stream
-        self.server_side = server_side
         self.addr = addr
-        
-    '''
-    Our puplic methods
-    '''
-    async def loop(self) -> None:
-        
+        self.r_addr = ''
+        self.l_addr = ''
+        self.conn_no = next(self._ids)
+        self.proc_start = None  # start processing start timestamp
+        self.proc_max = 0
+
+    def __timeout(self) -> int:
+        if self.state == State.init or self.state == State.received:
+            to = self.MAX_START_TIME
+        else:
+            if self.server_side and self.modbus_polling:
+                to = self.MAX_INV_IDLE_TIME
+            else:
+                to = self.MAX_CLOUD_IDLE_TIME
+        return to
+
+    async def publish_outstanding_mqtt(self):
+        '''Publish all outstanding MQTT topics'''
+        try:
+            if self.unique_id:
+                await self.async_publ_mqtt()
+            await self._async_publ_mqtt_proxy_stat('proxy')
+        except Exception:
+            pass
+
+    async def server_loop(self, addr: str) -> None:
+        '''Loop for receiving messages from the inverter (server-side)'''
+        logger.info(f'[{self.node_id}:{self.conn_no}] '
+                    f'Accept connection from {addr}')
+        self.inc_counter('Inverter_Cnt')
+        await self.publish_outstanding_mqtt()
+        await self.loop()
+        self.dec_counter('Inverter_Cnt')
+        await self.publish_outstanding_mqtt()
+        logger.info(f'[{self.node_id}:{self.conn_no}] Server loop stopped for'
+                    f' r{self.r_addr}')
+
+        # if the server connection closes, we also have to disconnect
+        # the connection to te TSUN cloud
+        if self.remoteStream:
+            logger.info(f'[{self.node_id}:{self.conn_no}] disc client '
+                        f'connection: [{self.remoteStream.node_id}:'
+                        f'{self.remoteStream.conn_no}]')
+            await self.remoteStream.disc()
+
+    async def client_loop(self, addr: str) -> None:
+        '''Loop for receiving messages from the TSUN cloud (client-side)'''
+        clientStream = await self.remoteStream.loop()
+        logger.info(f'[{clientStream.node_id}:{clientStream.conn_no}] '
+                    'Client loop stopped for'
+                    f' l{clientStream.l_addr}')
+
+        # if the client connection closes, we don't touch the server
+        # connection. Instead we erase the client connection stream,
+        # thus on the next received packet from the inverter, we can
+        # establish a new connection to the TSUN cloud
+
+        # erase backlink to inverter
+        clientStream.remoteStream = None
+
+        if self.remoteStream == clientStream:
+            # logging.debug(f'Client l{clientStream.l_addr} refs:'
+            #               f' {gc.get_referrers(clientStream)}')
+            # than erase client connection
+            self.remoteStream = None
+
+    async def loop(self) -> Self:
+        """Async loop handler for precessing all received messages"""
+        self.r_addr = self.writer.get_extra_info('peername')
+        self.l_addr = self.writer.get_extra_info('sockname')
+        self.proc_start = time.time()
         while True:
             try:
-                await self.__async_read()     
-                            
-                if self.unique_id: 
-                    await self.__async_write()     
+                proc = time.time() - self.proc_start
+                if proc > self.proc_max:
+                    self.proc_max = proc
+                self.proc_start = None
+                dead_conn_to = self.__timeout()
+                await asyncio.wait_for(self.__async_read(),
+                                       dead_conn_to)
+
+                if self.unique_id:
+                    await self.async_write()
                     await self.__async_forward()
                     await self.async_publ_mqtt()
-                
-      
-            except (ConnectionResetError,
-                    ConnectionAbortedError,
-                    RuntimeError) as error:
-                logger.warning(f'In loop for {self.addr}: {error}')
+
+            except asyncio.TimeoutError:
+                logger.warning(f'[{self.node_id}:{self.conn_no}] Dead '
+                               f'connection timeout ({dead_conn_to}s) '
+                               f'for {self.l_addr}')
+                await self.disc()
                 self.close()
-                return
+                return self
+
+            except OSError as error:
+                logger.error(f'[{self.node_id}:{self.conn_no}] '
+                             f'{error} for l{self.l_addr} | '
+                             f'r{self.r_addr}')
+                await self.disc()
+                self.close()
+                return self
+
+            except RuntimeError as error:
+                logger.info(f'[{self.node_id}:{self.conn_no}] '
+                            f'{error} for {self.l_addr}')
+                await self.disc()
+                self.close()
+                return self
+
             except Exception:
+                self.inc_counter('SW_Exception')
                 logger.error(
                     f"Exception for {self.addr}:\n"
                     f"{traceback.format_exc()}")
-                self.close()
-                return
-            
-    def disc(self) -> None:
-        logger.debug(f'in AsyncStream.disc() {self.addr}')
-        self.writer.close()
-            
-        
-    def close(self):
-        logger.debug(f'in AsyncStream.close() {self.addr}')
-        self.writer.close()
-        super().close()         # call close handler in the parent class
 
-#        logger.info (f'AsyncStream refs: {gc.get_referrers(self)}')
+    async def async_write(self, headline: str = 'Transmit to ') -> None:
+        """Async write handler to transmit the send_buffer"""
+        if self._send_buffer:
+            hex_dump_memory(logging.INFO, f'{headline}{self.addr}:',
+                            self._send_buffer, len(self._send_buffer))
+            self.writer.write(self._send_buffer)
+            await self.writer.drain()
+            self._send_buffer = bytearray(0)  # self._send_buffer[sent:]
+
+    async def disc(self) -> None:
+        """Async disc handler for graceful disconnect"""
+        if self.writer.is_closing():
+            return
+        logger.debug(f'AsyncStream.disc() l{self.l_addr} | r{self.r_addr}')
+        self.writer.close()
+        await self.writer.wait_closed()
+
+    def close(self) -> None:
+        """close handler for a no waiting disconnect
+
+           hint: must be called before releasing the connection instance
+        """
+        self.reader.feed_eof()          # abort awaited read
+        if self.writer.is_closing():
+            return
+        logger.debug(f'AsyncStream.close() l{self.l_addr} | r{self.r_addr}')
+        self.writer.close()
+
+    def healthy(self) -> bool:
+        elapsed = 0
+        if self.proc_start is not None:
+            elapsed = time.time() - self.proc_start
+        if self.state == State.closed or elapsed > self.MAX_PROC_TIME:
+            logging.debug(f'[{self.node_id}:{self.conn_no}:'
+                          f'{type(self).__name__}]'
+                          f' act:{round(1000*elapsed)}ms'
+                          f' max:{round(1000*self.proc_max)}ms')
+            logging.debug(f'Healthy()) refs: {gc.get_referrers(self)}')
+        return elapsed < 5
 
-    
     '''
     Our private methods
     '''
     async def __async_read(self) -> None:
+        """Async read handler to read received data from TCP stream"""
         data = await self.reader.read(4096)
         if data:
+            self.proc_start = time.time()
             self._recv_buffer += data
-            self.read()                # call read in parent class
+            wait = self.read()                # call read in parent class
+            if wait > 0:
+                await asyncio.sleep(wait)
         else:
             raise RuntimeError("Peer closed.")
-            
-    async def __async_write(self) -> None:
-        if self._send_buffer:
-            hex_dump_memory(logging.INFO, f'Transmit to {self.addr}:', self._send_buffer, len(self._send_buffer))
-            self.writer.write(self._send_buffer)
-            await self.writer.drain()
-            self._send_buffer = bytearray(0)  #self._send_buffer[sent:]
-            
+
     async def __async_forward(self) -> None:
-        if self._forward_buffer:
+        """forward handler transmits data over the remote connection"""
+        if not self._forward_buffer:
+            return
+        try:
             if not self.remoteStream:
-                await self.async_create_remote()  # only implmeneted for server side => syncServerStream
-                
+                await self.async_create_remote()
+                if self.remoteStream:
+                    if self.remoteStream._init_new_client_conn():
+                        await self.remoteStream.async_write()
+
             if self.remoteStream:
-                hex_dump_memory(logging.INFO, f'Forward to {self.remoteStream.addr}:', self._forward_buffer, len(self._forward_buffer))
-                self.remoteStream.writer.write (self._forward_buffer)
-                await self.remoteStream.writer.drain()                    
+                self.remoteStream._update_header(self._forward_buffer)
+                hex_dump_memory(logging.INFO,
+                                f'Forward to {self.remoteStream.addr}:',
+                                self._forward_buffer,
+                                len(self._forward_buffer))
+                self.remoteStream.writer.write(self._forward_buffer)
+                await self.remoteStream.writer.drain()
                 self._forward_buffer = bytearray(0)
 
-    async def async_create_remote(self) -> None:
-        pass
+        except OSError as error:
+            if self.remoteStream:
+                rmt = self.remoteStream
+                self.remoteStream = None
+                logger.error(f'[{rmt.node_id}:{rmt.conn_no}] Fwd: {error} for '
+                             f'l{rmt.l_addr} | r{rmt.r_addr}')
+                await rmt.disc()
+                rmt.close()
 
-    async def async_publ_mqtt(self) -> None:
-        pass       
-
-
-    def __del__ (self):
-        logging.debug (f"AsyncStream.__del__  {self.addr}")     
+        except RuntimeError as error:
+            if self.remoteStream:
+                rmt = self.remoteStream
+                self.remoteStream = None
+                logger.info(f'[{rmt.node_id}:{rmt.conn_no}] '
+                            f'Fwd: {error} for {rmt.l_addr}')
+                await rmt.disc()
+                rmt.close()
 
+        except Exception:
+            self.inc_counter('SW_Exception')
+            logger.error(
+                f"Fwd Exception for {self.addr}:\n"
+                f"{traceback.format_exc()}")
 
+    def __del__(self):
+        logger.debug(
+            f"AsyncStream.__del__  l{self.l_addr} | r{self.r_addr}")

app/src/config.py

@@ -1,78 +1,177 @@
 '''Config module handles the proxy configuration in the config.toml file'''
 
-import shutil, tomllib, logging
-from schema import Schema, And, Use, Optional
+import shutil
+import tomllib
+import logging
+from schema import Schema, And, Or, Use, Optional
+
 
 class Config():
-    '''Static class Config is reads and sanitize the config. 
-    
-    Read config.toml file and sanitize it with read(). 
+    '''Static class Config is reads and sanitize the config.
+
+    Read config.toml file and sanitize it with read().
     Get named parts of the config with get()'''
 
     config = {}
-    conf_schema = Schema({  'tsun': {
-                                    'enabled': Use(bool),
-                                    'host':    Use(str), 
-                                    'port':    And(Use(int), lambda n: 1024 <= n <= 65535)}, 
-
-                            'mqtt': {
-                                    'host':    Use(str), 
-                                    'port':    And(Use(int), lambda n: 1024 <= n <= 65535), 
-                                    'user':    And(Use(str), Use(lambda s: s if len(s) >0 else None)), 
-                                    'passwd':  And(Use(str), Use(lambda s: s if len(s) >0 else None))},
-
-                            
-                            'ha': {
-                                    'auto_conf_prefix': Use(str),
-                                    'discovery_prefix': Use(str),
-                                    'entity_prefix':    Use(str),
-                                    'proxy_node_id':    Use(str),
-                                    'proxy_unique_id':  Use(str)},
-                         
-                            'inverters': {
-                                    'allow_all' : Use(bool),
-                                    And(Use(str), lambda s: len(s) == 16 ): { 
-                                           Optional('node_id',         default=""): And(Use(str),Use(lambda s: s +'/' if len(s)> 0 and s[-1] != '/' else s)), 
-                                           Optional('suggested_area',  default=""): Use(str)
-                                           }}
-                            }, ignore_extra_keys=True)
+    def_config = {}
+    conf_schema = Schema({
+        'tsun': {
+            'enabled': Use(bool),
+            'host':    Use(str),
+            'port':    And(Use(int), lambda n: 1024 <= n <= 65535)
+            },
+        'solarman': {
+            'enabled': Use(bool),
+            'host':    Use(str),
+            'port':    And(Use(int), lambda n: 1024 <= n <= 65535)
+            },
+        'mqtt': {
+            'host':    Use(str),
+            'port':    And(Use(int), lambda n: 1024 <= n <= 65535),
+            'user':    And(Use(str), Use(lambda s: s if len(s) > 0 else None)),
+            'passwd':  And(Use(str), Use(lambda s: s if len(s) > 0 else None))
+            },
+        'ha': {
+            'auto_conf_prefix': Use(str),
+            'discovery_prefix': Use(str),
+            'entity_prefix':    Use(str),
+            'proxy_node_id':    Use(str),
+            'proxy_unique_id':  Use(str)
+            },
+        'gen3plus': {
+            'at_acl': {
+                Or('mqtt', 'tsun'): {
+                    'allow': [str],
+                    Optional('block', default=[]): [str]
+                    }
+                }
+            },
+        'inverters': {
+            'allow_all': Use(bool), And(Use(str), lambda s: len(s) == 16): {
+                Optional('monitor_sn', default=0): Use(int),
+                Optional('node_id', default=""): And(Use(str),
+                                                     Use(lambda s: s + '/'
+                                                         if len(s) > 0 and
+                                                         s[-1] != '/' else s)),
+                Optional('client_mode'): {
+                    'host': Use(str),
+                    Optional('port', default=8899):
+                        And(Use(int), lambda n: 1024 <= n <= 65535)
+                    },
+                Optional('modbus_polling', default=True): Use(bool),
+                Optional('suggested_area',  default=""): Use(str),
+                Optional('pv1'): {
+                    Optional('type'): Use(str),
+                    Optional('manufacturer'): Use(str),
+                    },
+                Optional('pv2'): {
+                    Optional('type'): Use(str),
+                    Optional('manufacturer'): Use(str),
+                    },
+                Optional('pv3'): {
+                    Optional('type'): Use(str),
+                    Optional('manufacturer'): Use(str),
+                    },
+                Optional('pv4'): {
+                    Optional('type'): Use(str),
+                    Optional('manufacturer'): Use(str),
+                    },
+                Optional('pv5'): {
+                    Optional('type'): Use(str),
+                    Optional('manufacturer'): Use(str),
+                    },
+                Optional('pv6'): {
+                    Optional('type'): Use(str),
+                    Optional('manufacturer'): Use(str),
+                    }
+                }}
+            }, ignore_extra_keys=True
+        )
 
     @classmethod
-    def read(cls) -> None:
-        '''Read config file, merge it with the default config and sanitize the result'''
+    def class_init(cls) -> None | str:  # pragma: no cover
+        try:
+            # make the default config transparaent by copying it
+            # in the config.example file
+            logging.debug('Copy Default Config to config.example.toml')
 
+            shutil.copy2("default_config.toml",
+                         "config/config.example.toml")
+        except Exception:
+            pass
+        err_str = cls.read()
+        del cls.conf_schema
+        return err_str
+
+    @classmethod
+    def _read_config_file(cls) -> dict:  # pragma: no cover
+        usr_config = {}
+
+        try:
+            with open("config/config.toml", "rb") as f:
+                usr_config = tomllib.load(f)
+        except Exception as error:
+            err = f'Config.read: {error}'
+            logging.error(err)
+            logging.info(
+                '\n  To create the missing config.toml file, '
+                'you can rename the template config.example.toml\n'
+                '  and customize it for your scenario.\n')
+        return usr_config
+
+    @classmethod
+    def read(cls, path='') -> None | str:
+        '''Read config file, merge it with the default config
+        and sanitize the result'''
+        err = None
         config = {}
         logger = logging.getLogger('data')
 
         try:
-            # make the default config transparaent by copying it in the config.example file
-            shutil.copy2("default_config.toml", "config/config.example.toml")
-
             # read example config file as default configuration
-            with open("default_config.toml", "rb") as f:
+            cls.def_config = {}
+            with open(f"{path}default_config.toml", "rb") as f:
                 def_config = tomllib.load(f)
+                cls.def_config = cls.conf_schema.validate(def_config)
 
-            # overwrite the default values, with values from the config.toml file
-            with open("config/config.toml", "rb") as f:
-                usr_config = tomllib.load(f)
- 
-            config['tsun']      = def_config['tsun']      | usr_config['tsun']
-            config['mqtt']      = def_config['mqtt']      | usr_config['mqtt']
-            config['ha']        = def_config['ha']        | usr_config['ha']
-            config['inverters'] = def_config['inverters'] | usr_config['inverters']
+            # overwrite the default values, with values from
+            # the config.toml file
+            usr_config = cls._read_config_file()
+
+            # merge the default and the user config
+            config = def_config.copy()
+            for key in ['tsun', 'solarman', 'mqtt', 'ha', 'inverters',
+                        'gen3plus']:
+                if key in usr_config:
+                    config[key] |= usr_config[key]
+
+            try:
+                cls.config = cls.conf_schema.validate(config)
+            except Exception as error:
+                err = f'Config.read: {error}'
+                logging.error(err)
+
+            # logging.debug(f'Readed config: "{cls.config}" ')
 
-            cls.config = cls.conf_schema.validate(config)
-            #logging.debug(f'Readed config: "{cls.config}" ')
- 
         except Exception as error:
-            logger.error(f'Config.read: {error}')
+            err = f'Config.read: {error}'
+            logger.error(err)
             cls.config = {}
 
+        return err
+
     @classmethod
-    def get(cls, member:str = None):
-        '''Get a named attribute from the proxy config. If member == None it returns the complete config dict'''
+    def get(cls, member: str = None):
+        '''Get a named attribute from the proxy config. If member ==
+          None it returns the complete config dict'''
 
         if member:
             return cls.config.get(member, {})
         else:
-            return cls.config
+            return cls.config
+
+    @classmethod
+    def is_default(cls, member: str) -> bool:
+        '''Check if the member is the default value'''
+
+        return cls.config.get(member) == cls.def_config.get(member)

app/src/gen3/connection_g3.py

@@ -0,0 +1,42 @@
+import logging
+# import gc
+from asyncio import StreamReader, StreamWriter
+from async_stream import AsyncStream
+from gen3.talent import Talent
+
+logger = logging.getLogger('conn')
+
+
+class ConnectionG3(AsyncStream, Talent):
+
+    def __init__(self, reader: StreamReader, writer: StreamWriter,
+                 addr, remote_stream: 'ConnectionG3', server_side: bool,
+                 id_str=b'') -> None:
+        AsyncStream.__init__(self, reader, writer, addr)
+        Talent.__init__(self, server_side, id_str)
+
+        self.remoteStream: 'ConnectionG3' = remote_stream
+
+    '''
+    Our puplic methods
+    '''
+    def close(self):
+        AsyncStream.close(self)
+        Talent.close(self)
+        # logger.info(f'AsyncStream refs: {gc.get_referrers(self)}')
+
+    async def async_create_remote(self) -> None:
+        pass
+
+    async def async_publ_mqtt(self) -> None:
+        pass
+
+    def healthy(self) -> bool:
+        logger.debug('ConnectionG3 healthy()')
+        return AsyncStream.healthy(self)
+
+    '''
+    Our private methods
+    '''
+    def __del__(self):
+        super().__del__()

app/src/gen3/infos_g3.py

@@ -0,0 +1,187 @@
+
+import struct
+import logging
+from typing import Generator
+
+if __name__ == "app.src.gen3.infos_g3":
+    from app.src.infos import Infos, Register
+else:  # pragma: no cover
+    from infos import Infos, Register
+
+
+class RegisterMap:
+    map = {
+        0x00092ba8: Register.COLLECTOR_FW_VERSION,
+        0x000927c0: Register.CHIP_TYPE,
+        0x00092f90: Register.CHIP_MODEL,
+        0x00095a88: Register.TRACE_URL,
+        0x00095aec: Register.LOGGER_URL,
+        0x0000000a: Register.PRODUCT_NAME,
+        0x00000014: Register.MANUFACTURER,
+        0x0000001e: Register.VERSION,
+        0x00000028: Register.SERIAL_NUMBER,
+        0x00000032: Register.EQUIPMENT_MODEL,
+        0x00013880: Register.NO_INPUTS,
+        0xffffff00: Register.INVERTER_CNT,
+        0xffffff01: Register.UNKNOWN_SNR,
+        0xffffff02: Register.UNKNOWN_MSG,
+        0xffffff03: Register.INVALID_DATA_TYPE,
+        0xffffff04: Register.INTERNAL_ERROR,
+        0xffffff05: Register.UNKNOWN_CTRL,
+        0xffffff06: Register.OTA_START_MSG,
+        0xffffff07: Register.SW_EXCEPTION,
+        0xffffff08: Register.MAX_DESIGNED_POWER,
+        0xffffff09: Register.OUTPUT_COEFFICIENT,
+        0xffffff0a: Register.INVERTER_STATUS,
+        0xffffff0b: Register.POLLING_INTERVAL,
+        0xfffffffe: Register.TEST_REG1,
+        0xffffffff: Register.TEST_REG2,
+        0x00000640: Register.OUTPUT_POWER,
+        0x000005dc: Register.RATED_POWER,
+        0x00000514: Register.INVERTER_TEMP,
+        0x000006a4: Register.PV1_VOLTAGE,
+        0x00000708: Register.PV1_CURRENT,
+        0x0000076c: Register.PV1_POWER,
+        0x000007d0: Register.PV2_VOLTAGE,
+        0x00000834: Register.PV2_CURRENT,
+        0x00000898: Register.PV2_POWER,
+        0x000008fc: Register.PV3_VOLTAGE,
+        0x00000960: Register.PV3_CURRENT,
+        0x000009c4: Register.PV3_POWER,
+        0x00000a28: Register.PV4_VOLTAGE,
+        0x00000a8c: Register.PV4_CURRENT,
+        0x00000af0: Register.PV4_POWER,
+        0x00000c1c: Register.PV1_DAILY_GENERATION,
+        0x00000c80: Register.PV1_TOTAL_GENERATION,
+        0x00000ce4: Register.PV2_DAILY_GENERATION,
+        0x00000d48: Register.PV2_TOTAL_GENERATION,
+        0x00000dac: Register.PV3_DAILY_GENERATION,
+        0x00000e10: Register.PV3_TOTAL_GENERATION,
+        0x00000e74: Register.PV4_DAILY_GENERATION,
+        0x00000ed8: Register.PV4_TOTAL_GENERATION,
+        0x00000b54: Register.DAILY_GENERATION,
+        0x00000bb8: Register.TOTAL_GENERATION,
+        0x000003e8: Register.GRID_VOLTAGE,
+        0x0000044c: Register.GRID_CURRENT,
+        0x000004b0: Register.GRID_FREQUENCY,
+        0x000cfc38: Register.CONNECT_COUNT,
+        0x000c3500: Register.SIGNAL_STRENGTH,
+        0x000c96a8: Register.POWER_ON_TIME,
+        0x000d0020: Register.COLLECT_INTERVAL,
+        0x000cf850: Register.DATA_UP_INTERVAL,
+        0x000c7f38: Register.COMMUNICATION_TYPE,
+        0x00000191: Register.EVENT_401,
+        0x00000192: Register.EVENT_402,
+        0x00000193: Register.EVENT_403,
+        0x00000194: Register.EVENT_404,
+        0x00000195: Register.EVENT_405,
+        0x00000196: Register.EVENT_406,
+        0x00000197: Register.EVENT_407,
+        0x00000198: Register.EVENT_408,
+        0x00000199: Register.EVENT_409,
+        0x0000019a: Register.EVENT_410,
+        0x0000019b: Register.EVENT_411,
+        0x0000019c: Register.EVENT_412,
+        0x0000019d: Register.EVENT_413,
+        0x0000019e: Register.EVENT_414,
+        0x0000019f: Register.EVENT_415,
+        0x000001a0: Register.EVENT_416,
+    }
+
+
+class InfosG3(Infos):
+
+    def ha_confs(self, ha_prfx: str, node_id: str, snr: str,
+                 sug_area: str = '') \
+            -> Generator[tuple[dict, str], None, None]:
+        '''Generator function yields a json register struct for home-assistant
+        auto configuration and a unique entity string
+
+        arguments:
+        prfx:str     ==> MQTT prefix for the home assistant 'stat_t string
+        snr:str      ==> serial number of the inverter, used to build unique
+                         entity strings
+        sug_area:str ==> suggested area string from the config file'''
+        # iterate over RegisterMap.map and get the register values
+        for reg in RegisterMap.map.values():
+            res = self.ha_conf(reg, ha_prfx, node_id, snr, False, sug_area)  # noqa: E501
+            if res:
+                yield res
+
+    def parse(self, buf, ind=0, node_id: str = '') -> \
+            Generator[tuple[str, bool], None, None]:
+        '''parse a data sequence received from the inverter and
+        stores the values in Infos.db
+
+        buf: buffer of the sequence to parse'''
+        result = struct.unpack_from('!l', buf, ind)
+        elms = result[0]
+        i = 0
+        ind += 4
+        while i < elms:
+            result = struct.unpack_from('!lB', buf, ind)
+            addr = result[0]
+            if addr not in RegisterMap.map:
+                info_id = -1
+            else:
+                info_id = RegisterMap.map[addr]
+            data_type = result[1]
+            ind += 5
+
+            if data_type == 0x54:   # 'T' -> Pascal-String
+                str_len = buf[ind]
+                result = struct.unpack_from(f'!{str_len+1}p', buf,
+                                            ind)[0].decode(encoding='ascii',
+                                                           errors='replace')
+                ind += str_len+1
+
+            elif data_type == 0x00:  # 'Nul' -> end
+                i = elms  # abort the loop
+
+            elif data_type == 0x41:  # 'A' -> Nop ??
+                # result = struct.unpack_from('!l', buf, ind)[0]
+                ind += 0
+                i += 1
+                continue
+
+            elif data_type == 0x42:  # 'B' -> byte, int8
+                result = struct.unpack_from('!B', buf, ind)[0]
+                ind += 1
+
+            elif data_type == 0x49:  # 'I' -> int32
+                result = struct.unpack_from('!l', buf, ind)[0]
+                ind += 4
+
+            elif data_type == 0x53:  # 'S' -> short, int16
+                result = struct.unpack_from('!h', buf, ind)[0]
+                ind += 2
+
+            elif data_type == 0x46:  # 'F' -> float32
+                result = round(struct.unpack_from('!f', buf, ind)[0], 2)
+                ind += 4
+
+            elif data_type == 0x4c:  # 'L' -> long, int64
+                result = struct.unpack_from('!q', buf, ind)[0]
+                ind += 8
+
+            else:
+                self.inc_counter('Invalid_Data_Type')
+                logging.error(f"Infos.parse: data_type: {data_type}"
+                              f" @0x{addr:04x} No:{i}"
+                              " not supported")
+                return
+
+            keys, level, unit, must_incr = self._key_obj(info_id)
+
+            if keys:
+                name, update = self.update_db(keys, must_incr, result)
+                yield keys[0], update
+            else:
+                update = False
+                name = str(f'info-id.0x{addr:x}')
+
+            if update:
+                self.tracer.log(level, f'[{node_id}] GEN3: {name} :'
+                                       f' {result}{unit}')
+
+            i += 1

app/src/gen3/inverter_g3.py

@@ -0,0 +1,132 @@
+import logging
+import traceback
+import json
+import asyncio
+from asyncio import StreamReader, StreamWriter
+from config import Config
+from inverter import Inverter
+from gen3.connection_g3 import ConnectionG3
+from aiomqtt import MqttCodeError
+from infos import Infos
+
+# import gc
+
+# logger = logging.getLogger('conn')
+logger_mqtt = logging.getLogger('mqtt')
+
+
+class InverterG3(Inverter, ConnectionG3):
+    '''class Inverter is a derivation of an Async_Stream
+
+    The class has some class method for managing common resources like a
+    connection to the MQTT broker or proxy error counter which are common
+    for all inverter connection
+
+    Instances of the class are connections to an inverter and can have an
+    optional link to an remote connection to the TSUN cloud. A remote
+    connection dies with the inverter connection.
+
+    class methods:
+        class_init():  initialize the common resources of the proxy (MQTT
+                       broker, Proxy DB, etc). Must be called before the
+                       first inverter instance can be created
+        class_close(): release the common resources of the proxy. Should not
+                       be called before any instances of the class are
+                       destroyed
+
+    methods:
+        server_loop(addr): Async loop method for receiving messages from the
+                           inverter (server-side)
+        client_loop(addr): Async loop method for receiving messages from the
+                           TSUN cloud (client-side)
+        async_create_remote(): Establish a client connection to the TSUN cloud
+        async_publ_mqtt(): Publish data to MQTT broker
+        close(): Release method which must be called before a instance can be
+                 destroyed
+    '''
+
+    def __init__(self, reader: StreamReader, writer: StreamWriter, addr):
+        super().__init__(reader, writer, addr, None, True)
+        self.__ha_restarts = -1
+
+    async def async_create_remote(self) -> None:
+        '''Establish a client connection to the TSUN cloud'''
+        tsun = Config.get('tsun')
+        host = tsun['host']
+        port = tsun['port']
+        addr = (host, port)
+
+        try:
+            logging.info(f'[{self.node_id}] Connect to {addr}')
+            connect = asyncio.open_connection(host, port)
+            reader, writer = await connect
+            self.remoteStream = ConnectionG3(reader, writer, addr, self,
+                                             False, self.id_str)
+            logging.info(f'[{self.remoteStream.node_id}:'
+                         f'{self.remoteStream.conn_no}] '
+                         f'Connected to {addr}')
+            asyncio.create_task(self.client_loop(addr))
+
+        except (ConnectionRefusedError, TimeoutError) as error:
+            logging.info(f'{error}')
+        except Exception:
+            self.inc_counter('SW_Exception')
+            logging.error(
+                f"Inverter: Exception for {addr}:\n"
+                f"{traceback.format_exc()}")
+
+    async def async_publ_mqtt(self) -> None:
+        '''publish data to MQTT broker'''
+        # check if new inverter or collector infos are available or when the
+        #  home assistant has changed the status back to online
+        try:
+            if (('inverter' in self.new_data and self.new_data['inverter'])
+                    or ('collector' in self.new_data and
+                        self.new_data['collector'])
+                    or self.mqtt.ha_restarts != self.__ha_restarts):
+                await self._register_proxy_stat_home_assistant()
+                await self.__register_home_assistant()
+                self.__ha_restarts = self.mqtt.ha_restarts
+
+            for key in self.new_data:
+                await self.__async_publ_mqtt_packet(key)
+            for key in Infos.new_stat_data:
+                await self._async_publ_mqtt_proxy_stat(key)
+
+        except MqttCodeError as error:
+            logging.error(f'Mqtt except: {error}')
+        except Exception:
+            self.inc_counter('SW_Exception')
+            logging.error(
+                f"Inverter: Exception:\n"
+                f"{traceback.format_exc()}")
+
+    async def __async_publ_mqtt_packet(self, key):
+        db = self.db.db
+        if key in db and self.new_data[key]:
+            data_json = json.dumps(db[key])
+            node_id = self.node_id
+            logger_mqtt.debug(f'{key}: {data_json}')
+            await self.mqtt.publish(f'{self.entity_prfx}{node_id}{key}', data_json)  # noqa: E501
+            self.new_data[key] = False
+
+    async def __register_home_assistant(self) -> None:
+        '''register all our topics at home assistant'''
+        for data_json, component, node_id, id in self.db.ha_confs(
+                self.entity_prfx, self.node_id, self.unique_id,
+                self.sug_area):
+            logger_mqtt.debug(f"MQTT Register: cmp:'{component}'"
+                              f" node_id:'{node_id}' {data_json}")
+            await self.mqtt.publish(f"{self.discovery_prfx}{component}"
+                                    f"/{node_id}{id}/config", data_json)
+
+        self.db.reg_clr_at_midnight(f'{self.entity_prfx}{self.node_id}')
+
+    def close(self) -> None:
+        logging.debug(f'InverterG3.close() l{self.l_addr} | r{self.r_addr}')
+        super().close()         # call close handler in the parent class
+#         logging.info(f'Inverter refs: {gc.get_referrers(self)}')
+
+    def __del__(self):
+        logging.debug("InverterG3.__del__")
+        super().__del__()

app/src/gen3/talent.py

@@ -0,0 +1,543 @@
+import struct
+import logging
+import time
+import pytz
+from datetime import datetime
+from tzlocal import get_localzone
+
+if __name__ == "app.src.gen3.talent":
+    from app.src.messages import hex_dump_memory, Message, State
+    from app.src.modbus import Modbus
+    from app.src.my_timer import Timer
+    from app.src.config import Config
+    from app.src.gen3.infos_g3 import InfosG3
+    from app.src.infos import Register
+else:  # pragma: no cover
+    from messages import hex_dump_memory, Message, State
+    from modbus import Modbus
+    from my_timer import Timer
+    from config import Config
+    from gen3.infos_g3 import InfosG3
+    from infos import Register
+
+logger = logging.getLogger('msg')
+
+
+class Control:
+    def __init__(self, ctrl: int):
+        self.ctrl = ctrl
+
+    def __int__(self) -> int:
+        return self.ctrl
+
+    def is_ind(self) -> bool:
+        return (self.ctrl == 0x91)
+
+    def is_req(self) -> bool:
+        return (self.ctrl == 0x70)
+
+    def is_resp(self) -> bool:
+        return (self.ctrl == 0x99)
+
+
+class Talent(Message):
+    MB_START_TIMEOUT = 40
+    MB_REGULAR_TIMEOUT = 60
+
+    def __init__(self, server_side: bool, id_str=b''):
+        super().__init__(server_side, self.send_modbus_cb, mb_timeout=11)
+        self.await_conn_resp_cnt = 0
+        self.id_str = id_str
+        self.contact_name = b''
+        self.contact_mail = b''
+        self.ts_offset = 0        # time offset between tsun cloud and local
+        self.db = InfosG3()
+        self.switch = {
+            0x00: self.msg_contact_info,
+            0x13: self.msg_ota_update,
+            0x22: self.msg_get_time,
+            0x71: self.msg_collector_data,
+            # 0x76:
+            0x77: self.msg_modbus,
+            # 0x78:
+            0x04: self.msg_inverter_data,
+        }
+        self.log_lvl = {
+            0x00: logging.INFO,
+            0x13: logging.INFO,
+            0x22: logging.INFO,
+            0x71: logging.INFO,
+            # 0x76:
+            0x77: self.get_modbus_log_lvl,
+            # 0x78:
+            0x04: logging.INFO,
+        }
+        self.modbus_elms = 0    # for unit tests
+        self.node_id = 'G3'     # will be overwritten in __set_serial_no
+        self.mb_timer = Timer(self.mb_timout_cb, self.node_id)
+        self.mb_timeout = self.MB_REGULAR_TIMEOUT
+        self.mb_start_timeout = self.MB_START_TIMEOUT
+        self.modbus_polling = False
+
+    '''
+    Our puplic methods
+    '''
+    def close(self) -> None:
+        logging.debug('Talent.close()')
+        if self.server_side:
+            # set inverter state to offline, if output power is very low
+            logging.debug('close power: '
+                          f'{self.db.get_db_value(Register.OUTPUT_POWER, -1)}')
+            if self.db.get_db_value(Register.OUTPUT_POWER, 999) < 2:
+                self.db.set_db_def_value(Register.INVERTER_STATUS, 0)
+                self.new_data['env'] = True
+
+        # we have references to methods of this class in self.switch
+        # so we have to erase self.switch, otherwise this instance can't be
+        # deallocated by the garbage collector ==> we get a memory leak
+        self.switch.clear()
+        self.log_lvl.clear()
+        self.state = State.closed
+        self.mb_timer.close()
+        super().close()
+
+    def __set_serial_no(self, serial_no: str):
+
+        if self.unique_id == serial_no:
+            logger.debug(f'SerialNo: {serial_no}')
+        else:
+            inverters = Config.get('inverters')
+            # logger.debug(f'Inverters: {inverters}')
+
+            if serial_no in inverters:
+                inv = inverters[serial_no]
+                self.node_id = inv['node_id']
+                self.sug_area = inv['suggested_area']
+                self.modbus_polling = inv['modbus_polling']
+                logger.debug(f'SerialNo {serial_no} allowed! area:{self.sug_area}')  # noqa: E501
+                self.db.set_pv_module_details(inv)
+            else:
+                self.node_id = ''
+                self.sug_area = ''
+                if 'allow_all' not in inverters or not inverters['allow_all']:
+                    self.inc_counter('Unknown_SNR')
+                    self.unique_id = None
+                    logger.warning(f'ignore message from unknow inverter! (SerialNo: {serial_no})')  # noqa: E501
+                    return
+                logger.debug(f'SerialNo {serial_no} not known but accepted!')
+
+            self.unique_id = serial_no
+
+    def read(self) -> float:
+        '''process all received messages in the _recv_buffer'''
+        self._read()
+        while True:
+            if not self.header_valid:
+                self.__parse_header(self._recv_buffer, len(self._recv_buffer))
+
+            if self.header_valid and \
+               len(self._recv_buffer) >= (self.header_len + self.data_len):
+                if self.state == State.init:
+                    self.state = State.received     # received 1st package
+
+                log_lvl = self.log_lvl.get(self.msg_id, logging.WARNING)
+                if callable(log_lvl):
+                    log_lvl = log_lvl()
+
+                hex_dump_memory(log_lvl, f'Received from {self.addr}:'
+                                f' BufLen: {len(self._recv_buffer)}'
+                                f' HdrLen: {self.header_len}'
+                                f' DtaLen: {self.data_len}',
+                                self._recv_buffer, len(self._recv_buffer))
+
+                self.__set_serial_no(self.id_str.decode("utf-8"))
+                self.__dispatch_msg()
+                self.__flush_recv_msg()
+            else:
+                return 0  # don not wait before sending a response
+
+    def forward(self) -> None:
+        '''add the actual receive msg to the forwarding queue'''
+        tsun = Config.get('tsun')
+        if tsun['enabled']:
+            buffer = self._recv_buffer
+            buflen = self.header_len+self.data_len
+            self._forward_buffer += buffer[:buflen]
+            hex_dump_memory(logging.DEBUG, 'Store for forwarding:',
+                            buffer, buflen)
+
+            fnc = self.switch.get(self.msg_id, self.msg_unknown)
+            logger.info(self.__flow_str(self.server_side, 'forwrd') +
+                        f' Ctl: {int(self.ctrl):#02x} Msg: {fnc.__name__!r}')
+        return
+
+    def send_modbus_cb(self, modbus_pdu: bytearray, log_lvl: int, state: str):
+        if self.state != State.up:
+            logger.warning(f'[{self.node_id}] ignore MODBUS cmd,'
+                           ' cause the state is not UP anymore')
+            return
+
+        self.__build_header(0x70, 0x77)
+        self._send_buffer += b'\x00\x01\xa3\x28'   # fixme
+        self._send_buffer += struct.pack('!B', len(modbus_pdu))
+        self._send_buffer += modbus_pdu
+        self.__finish_send_msg()
+
+        hex_dump_memory(log_lvl, f'Send Modbus {state}:{self.addr}:',
+                        self._send_buffer, len(self._send_buffer))
+        self.writer.write(self._send_buffer)
+        self._send_buffer = bytearray(0)  # self._send_buffer[sent:]
+
+    def _send_modbus_cmd(self, func, addr, val, log_lvl) -> None:
+        if self.state != State.up:
+            logger.log(log_lvl, f'[{self.node_id}] ignore MODBUS cmd,'
+                       ' as the state is not UP')
+            return
+        self.mb.build_msg(Modbus.INV_ADDR, func, addr, val, log_lvl)
+
+    async def send_modbus_cmd(self, func, addr, val, log_lvl) -> None:
+        self._send_modbus_cmd(func, addr, val, log_lvl)
+
+    def mb_timout_cb(self, exp_cnt):
+        self.mb_timer.start(self.mb_timeout)
+
+        if 2 == (exp_cnt % 30):
+            # logging.info("Regular Modbus Status request")
+            self._send_modbus_cmd(Modbus.READ_REGS, 0x2000, 96, logging.DEBUG)
+        else:
+            self._send_modbus_cmd(Modbus.READ_REGS, 0x3000, 48, logging.DEBUG)
+
+    def _init_new_client_conn(self) -> bool:
+        contact_name = self.contact_name
+        contact_mail = self.contact_mail
+        logger.info(f'name: {contact_name} mail: {contact_mail}')
+        self.msg_id = 0
+        self.await_conn_resp_cnt += 1
+        self.__build_header(0x91)
+        self._send_buffer += struct.pack(f'!{len(contact_name)+1}p'
+                                         f'{len(contact_mail)+1}p',
+                                         contact_name, contact_mail)
+
+        self.__finish_send_msg()
+        return True
+
+    '''
+    Our private methods
+    '''
+    def __flow_str(self, server_side: bool, type: str):  # noqa: F821
+        switch = {
+            'rx':      '  <',
+            'tx':      '  >',
+            'forwrd':  '<< ',
+            'drop':    ' xx',
+            'rxS':     '>  ',
+            'txS':     '<  ',
+            'forwrdS': ' >>',
+            'dropS':   'xx ',
+        }
+        if server_side:
+            type += 'S'
+        return switch.get(type, '???')
+
+    def _timestamp(self):   # pragma: no cover
+        '''returns timestamp fo the inverter as localtime
+        since 1.1.1970 in msec'''
+        if False:
+            # utc as epoche
+            ts = time.time()
+        else:
+            # convert localtime in epoche
+            ts = (datetime.now() - datetime(1970, 1, 1)).total_seconds()
+        return round(ts*1000)
+
+    def _utcfromts(self, ts: float):
+        '''converts inverter timestamp into unix time (epoche)'''
+        dt = datetime.fromtimestamp(ts/1000, pytz.UTC). \
+            replace(tzinfo=get_localzone())
+        return dt.timestamp()
+
+    def _utc(self):   # pragma: no cover
+        '''returns unix time (epoche)'''
+        return datetime.now().timestamp()
+
+    def _update_header(self, _forward_buffer):
+        '''update header for message before forwarding,
+        add time offset to timestamp'''
+        _len = len(_forward_buffer)
+        ofs = 0
+        while ofs < _len:
+            result = struct.unpack_from('!lB', _forward_buffer, 0)
+            msg_len = 4 + result[0]
+            id_len = result[1]    # len of variable id string
+            if _len < 2*id_len + 21:
+                return
+
+            result = struct.unpack_from('!B', _forward_buffer, id_len+6)
+            msg_code = result[0]
+            if msg_code == 0x71 or msg_code == 0x04:
+                result = struct.unpack_from('!q', _forward_buffer, 13+2*id_len)
+                ts = result[0] + self.ts_offset
+                logger.debug(f'offset: {self.ts_offset:08x}'
+                             f'  proxy-time: {ts:08x}')
+                struct.pack_into('!q', _forward_buffer, 13+2*id_len, ts)
+            ofs += msg_len
+
+    # check if there is a complete header in the buffer, parse it
+    # and set
+    #   self.header_len
+    #   self.data_len
+    #   self.id_str
+    #   self.ctrl
+    #   self.msg_id
+    #
+    # if the header is incomplete, than self.header_len is still 0
+    #
+    def __parse_header(self, buf: bytes, buf_len: int) -> None:
+
+        if (buf_len < 5):      # enough bytes to read len and id_len?
+            return
+        result = struct.unpack_from('!lB', buf, 0)
+        len = result[0]    # len of complete message
+        id_len = result[1]    # len of variable id string
+
+        hdr_len = 5+id_len+2
+
+        if (buf_len < hdr_len):  # enough bytes for complete header?
+            return
+
+        result = struct.unpack_from(f'!{id_len+1}pBB', buf, 4)
+
+        # store parsed header values in the class
+        self.id_str = result[0]
+        self.ctrl = Control(result[1])
+        self.msg_id = result[2]
+        self.data_len = len-id_len-3
+        self.header_len = hdr_len
+        self.header_valid = True
+        return
+
+    def __build_header(self, ctrl, msg_id=None) -> None:
+        if not msg_id:
+            msg_id = self.msg_id
+        self.send_msg_ofs = len(self._send_buffer)
+        self._send_buffer += struct.pack(f'!l{len(self.id_str)+1}pBB',
+                                         0, self.id_str, ctrl, msg_id)
+        fnc = self.switch.get(msg_id, self.msg_unknown)
+        logger.info(self.__flow_str(self.server_side, 'tx') +
+                    f' Ctl: {int(ctrl):#02x} Msg: {fnc.__name__!r}')
+
+    def __finish_send_msg(self) -> None:
+        _len = len(self._send_buffer) - self.send_msg_ofs
+        struct.pack_into('!l', self._send_buffer, self.send_msg_ofs, _len-4)
+
+    def __dispatch_msg(self) -> None:
+        fnc = self.switch.get(self.msg_id, self.msg_unknown)
+        if self.unique_id:
+            logger.info(self.__flow_str(self.server_side, 'rx') +
+                        f' Ctl: {int(self.ctrl):#02x} ({self.state}) '
+                        f'Msg: {fnc.__name__!r}')
+            fnc()
+        else:
+            logger.info(self.__flow_str(self.server_side, 'drop') +
+                        f' Ctl: {int(self.ctrl):#02x} Msg: {fnc.__name__!r}')
+
+    def __flush_recv_msg(self) -> None:
+        self._recv_buffer = self._recv_buffer[(self.header_len+self.data_len):]
+        self.header_valid = False
+
+    '''
+    Message handler methods
+    '''
+    def msg_contact_info(self):
+        if self.ctrl.is_ind():
+            if self.server_side and self.__process_contact_info():
+                self.__build_header(0x91)
+                self._send_buffer += b'\x01'
+                self.__finish_send_msg()
+            # don't forward this contact info here, we will build one
+            # when the remote connection is established
+            elif self.await_conn_resp_cnt > 0:
+                self.await_conn_resp_cnt -= 1
+            else:
+                self.forward()
+            return
+        else:
+            logger.warning('Unknown Ctrl')
+            self.inc_counter('Unknown_Ctrl')
+            self.forward()
+
+    def __process_contact_info(self) -> bool:
+        result = struct.unpack_from('!B', self._recv_buffer, self.header_len)
+        name_len = result[0]
+        if self.data_len < name_len+2:
+            return False
+        result = struct.unpack_from(f'!{name_len+1}pB', self._recv_buffer,
+                                    self.header_len)
+        self.contact_name = result[0]
+        mail_len = result[1]
+        logger.info(f'name: {self.contact_name}')
+
+        result = struct.unpack_from(f'!{mail_len+1}p', self._recv_buffer,
+                                    self.header_len+name_len+1)
+        self.contact_mail = result[0]
+        logger.info(f'mail: {self.contact_mail}')
+        return True
+
+    def msg_get_time(self):
+        if self.ctrl.is_ind():
+            if self.data_len == 0:
+                if self.state == State.up:
+                    self.state = State.pend     # block MODBUS cmds
+                if (self.modbus_polling):
+                    self.mb_timer.start(self.mb_start_timeout)
+                    self.db.set_db_def_value(Register.POLLING_INTERVAL,
+                                             self.mb_timeout)
+
+                ts = self._timestamp()
+                logger.debug(f'time: {ts:08x}')
+                self.__build_header(0x91)
+                self._send_buffer += struct.pack('!q', ts)
+                self.__finish_send_msg()
+
+            elif self.data_len >= 8:
+                ts = self._timestamp()
+                result = struct.unpack_from('!q', self._recv_buffer,
+                                            self.header_len)
+                self.ts_offset = result[0]-ts
+                logger.debug(f'tsun-time: {int(result[0]):08x}'
+                             f'  proxy-time: {ts:08x}'
+                             f'  offset: {self.ts_offset}')
+                return  # ignore received response
+        else:
+            logger.warning('Unknown Ctrl')
+            self.inc_counter('Unknown_Ctrl')
+
+        self.forward()
+
+    def parse_msg_header(self):
+        result = struct.unpack_from('!lB', self._recv_buffer, self.header_len)
+
+        data_id = result[0]    # len of complete message
+        id_len = result[1]     # len of variable id string
+        logger.debug(f'Data_ID: 0x{data_id:08x}   id_len:  {id_len}')
+
+        msg_hdr_len = 5+id_len+9
+
+        result = struct.unpack_from(f'!{id_len+1}pBq', self._recv_buffer,
+                                    self.header_len + 4)
+
+        timestamp = result[2]
+        logger.debug(f'ID: {result[0]}  B: {result[1]}')
+        logger.debug(f'time: {timestamp:08x}')
+        # logger.info(f'time: {datetime.utcfromtimestamp(result[2]).strftime(
+        # "%Y-%m-%d %H:%M:%S")}')
+        return msg_hdr_len, timestamp
+
+    def msg_collector_data(self):
+        if self.ctrl.is_ind():
+            self.__build_header(0x99)
+            self._send_buffer += b'\x01'
+            self.__finish_send_msg()
+            self.__process_data()
+
+        elif self.ctrl.is_resp():
+            return  # ignore received response
+        else:
+            logger.warning('Unknown Ctrl')
+            self.inc_counter('Unknown_Ctrl')
+
+        self.forward()
+
+    def msg_inverter_data(self):
+        if self.ctrl.is_ind():
+            self.__build_header(0x99)
+            self._send_buffer += b'\x01'
+            self.__finish_send_msg()
+            self.__process_data()
+            self.state = State.up  # allow MODBUS cmds
+
+        elif self.ctrl.is_resp():
+            return  # ignore received response
+        else:
+            logger.warning('Unknown Ctrl')
+            self.inc_counter('Unknown_Ctrl')
+
+        self.forward()
+
+    def __process_data(self):
+        msg_hdr_len, ts = self.parse_msg_header()
+
+        for key, update in self.db.parse(self._recv_buffer, self.header_len
+                                         + msg_hdr_len, self.node_id):
+            if update:
+                self._set_mqtt_timestamp(key, self._utcfromts(ts))
+                self.new_data[key] = True
+
+    def msg_ota_update(self):
+        if self.ctrl.is_req():
+            self.inc_counter('OTA_Start_Msg')
+        elif self.ctrl.is_ind():
+            pass
+        else:
+            logger.warning('Unknown Ctrl')
+            self.inc_counter('Unknown_Ctrl')
+        self.forward()
+
+    def parse_modbus_header(self):
+
+        msg_hdr_len = 5
+
+        result = struct.unpack_from('!lBB', self._recv_buffer,
+                                    self.header_len)
+        modbus_len = result[1]
+        # logger.debug(f'Ref: {result[0]}')
+        # logger.debug(f'Modbus MsgLen: {modbus_len} Func:{result[2]}')
+        return msg_hdr_len, modbus_len
+
+    def get_modbus_log_lvl(self) -> int:
+        if self.ctrl.is_req():
+            return logging.INFO
+        elif self.ctrl.is_ind():
+            if self.server_side:
+                return self.mb.last_log_lvl
+        return logging.WARNING
+
+    def msg_modbus(self):
+        hdr_len, modbus_len = self.parse_modbus_header()
+        data = self._recv_buffer[self.header_len:
+                                 self.header_len+self.data_len]
+
+        if self.ctrl.is_req():
+            if self.remoteStream.mb.recv_req(data[hdr_len:],
+                                             self.remoteStream.
+                                             msg_forward):
+                self.inc_counter('Modbus_Command')
+            else:
+                self.inc_counter('Invalid_Msg_Format')
+        elif self.ctrl.is_ind():
+            self.modbus_elms = 0
+            # logger.debug(f'Modbus Ind  MsgLen: {modbus_len}')
+            if not self.server_side:
+                logger.warning('Unknown Message')
+                self.inc_counter('Unknown_Msg')
+                return
+
+            for key, update, _ in self.mb.recv_resp(self.db, data[
+                    hdr_len:],
+                    self.node_id):
+                if update:
+                    self._set_mqtt_timestamp(key, self._utc())
+                    self.new_data[key] = True
+                self.modbus_elms += 1          # count for unit tests
+        else:
+            logger.warning('Unknown Ctrl')
+            self.inc_counter('Unknown_Ctrl')
+            self.forward()
+
+    def msg_forward(self):
+        self.forward()
+
+    def msg_unknown(self):
+        logger.warning(f"Unknow Msg: ID:{self.msg_id}")
+        self.inc_counter('Unknown_Msg')
+        self.forward()

app/src/gen3plus/connection_g3p.py

@@ -0,0 +1,43 @@
+import logging
+# import gc
+from asyncio import StreamReader, StreamWriter
+from async_stream import AsyncStream
+from gen3plus.solarman_v5 import SolarmanV5
+
+logger = logging.getLogger('conn')
+
+
+class ConnectionG3P(AsyncStream, SolarmanV5):
+
+    def __init__(self, reader: StreamReader, writer: StreamWriter,
+                 addr, remote_stream: 'ConnectionG3P',
+                 server_side: bool,
+                 client_mode: bool) -> None:
+        AsyncStream.__init__(self, reader, writer, addr)
+        SolarmanV5.__init__(self, server_side, client_mode)
+
+        self.remoteStream: 'ConnectionG3P' = remote_stream
+
+    '''
+    Our puplic methods
+    '''
+    def close(self):
+        AsyncStream.close(self)
+        SolarmanV5.close(self)
+        #  logger.info(f'AsyncStream refs: {gc.get_referrers(self)}')
+
+    async def async_create_remote(self) -> None:
+        pass
+
+    async def async_publ_mqtt(self) -> None:
+        pass
+
+    def healthy(self) -> bool:
+        logger.debug('ConnectionG3P healthy()')
+        return AsyncStream.healthy(self)
+
+    '''
+    Our private methods
+    '''
+    def __del__(self):
+        super().__del__()

app/src/gen3plus/infos_g3p.py

@@ -0,0 +1,142 @@
+
+import struct
+from typing import Generator
+
+if __name__ == "app.src.gen3plus.infos_g3p":
+    from app.src.infos import Infos, Register, ProxyMode
+else:  # pragma: no cover
+    from infos import Infos, Register, ProxyMode
+
+
+class RegisterMap:
+    # make the class read/only by using __slots__
+
+    __slots__ = ()
+    map = {
+        # 0x41020007: {'reg': Register.DEVICE_SNR,           'fmt': '<L'},                 # noqa: E501
+        0x41020018: {'reg': Register.DATA_UP_INTERVAL,     'fmt': '<B', 'ratio':   60, 'dep': ProxyMode.SERVER},  # noqa: E501
+        0x41020019: {'reg': Register.COLLECT_INTERVAL,     'fmt': '<B', 'eval': 'round(result/60)', 'dep': ProxyMode.SERVER},  # noqa: E501
+        0x4102001a: {'reg': Register.HEARTBEAT_INTERVAL,   'fmt': '<B', 'ratio':    1},  # noqa: E501
+        0x4102001c: {'reg': Register.SIGNAL_STRENGTH,      'fmt': '<B', 'ratio':    1, 'dep': ProxyMode.SERVER},  # noqa: E501
+        0x4102001e: {'reg': Register.CHIP_MODEL,           'fmt': '!40s'},               # noqa: E501
+        0x4102004c: {'reg': Register.IP_ADDRESS,           'fmt': '!16s'},               # noqa: E501
+        0x41020064: {'reg': Register.COLLECTOR_FW_VERSION, 'fmt': '!40s'},               # noqa: E501
+
+        0x4201001c: {'reg': Register.POWER_ON_TIME,        'fmt': '<H', 'ratio':    1, 'dep': ProxyMode.SERVER},  # noqa: E501
+        0x42010020: {'reg': Register.SERIAL_NUMBER,        'fmt': '!16s'},               # noqa: E501
+        0x420100c0: {'reg': Register.INVERTER_STATUS,      'fmt': '!H'},                 # noqa: E501
+        0x420100d0: {'reg': Register.VERSION,              'fmt': '!H', 'eval': "f'V{(result>>12)}.{(result>>8)&0xf}.{(result>>4)&0xf}{result&0xf}'"},  # noqa: E501
+        0x420100d2: {'reg': Register.GRID_VOLTAGE,         'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x420100d4: {'reg': Register.GRID_CURRENT,         'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x420100d6: {'reg': Register.GRID_FREQUENCY,       'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x420100d8: {'reg': Register.INVERTER_TEMP,        'fmt': '!H', 'eval': 'result-40'},  # noqa: E501
+        # 0x420100d8: {'reg': Register.INVERTER_TEMP,        'fmt': '!H'},                 # noqa: E501
+        0x420100dc: {'reg': Register.RATED_POWER,          'fmt': '!H', 'ratio':    1},  # noqa: E501
+        0x420100de: {'reg': Register.OUTPUT_POWER,         'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x420100e0: {'reg': Register.PV1_VOLTAGE,          'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x420100e2: {'reg': Register.PV1_CURRENT,          'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x420100e4: {'reg': Register.PV1_POWER,            'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x420100e6: {'reg': Register.PV2_VOLTAGE,          'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x420100e8: {'reg': Register.PV2_CURRENT,          'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x420100ea: {'reg': Register.PV2_POWER,            'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x420100ec: {'reg': Register.PV3_VOLTAGE,          'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x420100ee: {'reg': Register.PV3_CURRENT,          'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x420100f0: {'reg': Register.PV3_POWER,            'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x420100f2: {'reg': Register.PV4_VOLTAGE,          'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x420100f4: {'reg': Register.PV4_CURRENT,          'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x420100f6: {'reg': Register.PV4_POWER,            'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x420100f8: {'reg': Register.DAILY_GENERATION,     'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x420100fa: {'reg': Register.TOTAL_GENERATION,     'fmt': '!L', 'ratio': 0.01},  # noqa: E501
+        0x420100fe: {'reg': Register.PV1_DAILY_GENERATION, 'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x42010100: {'reg': Register.PV1_TOTAL_GENERATION, 'fmt': '!L', 'ratio': 0.01},  # noqa: E501
+        0x42010104: {'reg': Register.PV2_DAILY_GENERATION, 'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x42010106: {'reg': Register.PV2_TOTAL_GENERATION, 'fmt': '!L', 'ratio': 0.01},  # noqa: E501
+        0x4201010a: {'reg': Register.PV3_DAILY_GENERATION, 'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x4201010c: {'reg': Register.PV3_TOTAL_GENERATION, 'fmt': '!L', 'ratio': 0.01},  # noqa: E501
+        0x42010110: {'reg': Register.PV4_DAILY_GENERATION, 'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x42010112: {'reg': Register.PV4_TOTAL_GENERATION, 'fmt': '!L', 'ratio': 0.01},  # noqa: E501
+        0x42010126: {'reg': Register.MAX_DESIGNED_POWER,   'fmt': '!H', 'ratio':    1},  # noqa: E501
+
+        0xffffff01: {'reg': Register.OUTPUT_COEFFICIENT},
+        0xffffff02: {'reg': Register.POLLING_INTERVAL},
+        # 0x4281001c: {'reg': Register.POWER_ON_TIME,        'fmt': '<H', 'ratio':    1},  # noqa: E501
+
+    }
+
+
+class InfosG3P(Infos):
+    def __init__(self, client_mode: bool):
+        super().__init__()
+        self.client_mode = client_mode
+        self.set_db_def_value(Register.MANUFACTURER, 'TSUN')
+        self.set_db_def_value(Register.EQUIPMENT_MODEL, 'TSOL-MSxx00')
+        self.set_db_def_value(Register.CHIP_TYPE, 'IGEN TECH')
+        self.set_db_def_value(Register.NO_INPUTS, 4)
+
+    def __hide_topic(self, row: dict) -> bool:
+        if 'dep' in row:
+            mode = row['dep']
+            if self.client_mode:
+                return mode != ProxyMode.CLIENT
+            else:
+                return mode != ProxyMode.SERVER
+        return False
+
+    def ha_confs(self, ha_prfx: str, node_id: str, snr: str,
+                 sug_area: str = '') \
+            -> Generator[tuple[dict, str], None, None]:
+        '''Generator function yields a json register struct for home-assistant
+        auto configuration and a unique entity string
+
+        arguments:
+        prfx:str     ==> MQTT prefix for the home assistant 'stat_t string
+        snr:str      ==> serial number of the inverter, used to build unique
+                         entity strings
+        sug_area:str ==> suggested area string from the config file'''
+        # iterate over RegisterMap.map and get the register values
+        for row in RegisterMap.map.values():
+            info_id = row['reg']
+            if self.__hide_topic(row):
+                res = self.ha_remove(info_id, node_id, snr)  # noqa: E501
+            else:
+                res = self.ha_conf(info_id, ha_prfx, node_id, snr, False, sug_area)  # noqa: E501
+            if res:
+                yield res
+
+    def parse(self, buf, msg_type: int, rcv_ftype: int, node_id: str = '') \
+            -> Generator[tuple[str, bool], None, None]:
+        '''parse a data sequence received from the inverter and
+        stores the values in Infos.db
+
+        buf: buffer of the sequence to parse'''
+        for idx, row in RegisterMap.map.items():
+            addr = idx & 0xffff
+            ftype = (idx >> 16) & 0xff
+            mtype = (idx >> 24) & 0xff
+            if ftype != rcv_ftype or mtype != msg_type:
+                continue
+            if not isinstance(row, dict):
+                continue
+            info_id = row['reg']
+            fmt = row['fmt']
+            res = struct.unpack_from(fmt, buf, addr)
+            result = res[0]
+            if isinstance(result, (bytearray, bytes)):
+                result = result.decode().split('\x00')[0]
+            if 'eval' in row:
+                result = eval(row['eval'])
+            if 'ratio' in row:
+                result = round(result * row['ratio'], 2)
+
+            keys, level, unit, must_incr = self._key_obj(info_id)
+
+            if keys:
+                name, update = self.update_db(keys, must_incr, result)
+                yield keys[0], update
+            else:
+                name = str(f'info-id.0x{addr:x}')
+                update = False
+
+            if update:
+                self.tracer.log(level, f'[{node_id}] GEN3PLUS: {name}'
+                                       f' : {result}{unit}')

app/src/gen3plus/inverter_g3p.py

@@ -0,0 +1,135 @@
+import logging
+import traceback
+import json
+import asyncio
+from asyncio import StreamReader, StreamWriter
+from config import Config
+from inverter import Inverter
+from gen3plus.connection_g3p import ConnectionG3P
+from aiomqtt import MqttCodeError
+from infos import Infos
+
+# import gc
+
+# logger = logging.getLogger('conn')
+logger_mqtt = logging.getLogger('mqtt')
+
+
+class InverterG3P(Inverter, ConnectionG3P):
+    '''class Inverter is a derivation of an Async_Stream
+
+    The class has some class method for managing common resources like a
+    connection to the MQTT broker or proxy error counter which are common
+    for all inverter connection
+
+    Instances of the class are connections to an inverter and can have an
+    optional link to an remote connection to the TSUN cloud. A remote
+    connection dies with the inverter connection.
+
+    class methods:
+        class_init():  initialize the common resources of the proxy (MQTT
+                       broker, Proxy DB, etc). Must be called before the
+                       first inverter instance can be created
+        class_close(): release the common resources of the proxy. Should not
+                       be called before any instances of the class are
+                       destroyed
+
+    methods:
+        server_loop(addr): Async loop method for receiving messages from the
+                           inverter (server-side)
+        client_loop(addr): Async loop method for receiving messages from the
+                           TSUN cloud (client-side)
+        async_create_remote(): Establish a client connection to the TSUN cloud
+        async_publ_mqtt(): Publish data to MQTT broker
+        close(): Release method which must be called before a instance can be
+                 destroyed
+    '''
+
+    def __init__(self, reader: StreamReader, writer: StreamWriter, addr,
+                 client_mode: bool = False):
+        super().__init__(reader, writer, addr, None,
+                         server_side=True, client_mode=client_mode)
+        self.__ha_restarts = -1
+
+    async def async_create_remote(self) -> None:
+        '''Establish a client connection to the TSUN cloud'''
+        tsun = Config.get('solarman')
+        host = tsun['host']
+        port = tsun['port']
+        addr = (host, port)
+
+        try:
+            logging.info(f'[{self.node_id}] Connect to {addr}')
+            connect = asyncio.open_connection(host, port)
+            reader, writer = await connect
+            self.remoteStream = ConnectionG3P(reader, writer, addr, self,
+                                              server_side=False,
+                                              client_mode=False)
+            logging.info(f'[{self.remoteStream.node_id}:'
+                         f'{self.remoteStream.conn_no}] '
+                         f'Connected to {addr}')
+            asyncio.create_task(self.client_loop(addr))
+
+        except (ConnectionRefusedError, TimeoutError) as error:
+            logging.info(f'{error}')
+        except Exception:
+            self.inc_counter('SW_Exception')
+            logging.error(
+                f"Inverter: Exception for {addr}:\n"
+                f"{traceback.format_exc()}")
+
+    async def async_publ_mqtt(self) -> None:
+        '''publish data to MQTT broker'''
+        # check if new inverter or collector infos are available or when the
+        #  home assistant has changed the status back to online
+        try:
+            if (('inverter' in self.new_data and self.new_data['inverter'])
+                    or ('collector' in self.new_data and
+                        self.new_data['collector'])
+                    or self.mqtt.ha_restarts != self.__ha_restarts):
+                await self._register_proxy_stat_home_assistant()
+                await self.__register_home_assistant()
+                self.__ha_restarts = self.mqtt.ha_restarts
+
+            for key in self.new_data:
+                await self.__async_publ_mqtt_packet(key)
+            for key in Infos.new_stat_data:
+                await self._async_publ_mqtt_proxy_stat(key)
+
+        except MqttCodeError as error:
+            logging.error(f'Mqtt except: {error}')
+        except Exception:
+            self.inc_counter('SW_Exception')
+            logging.error(
+                f"Inverter: Exception:\n"
+                f"{traceback.format_exc()}")
+
+    async def __async_publ_mqtt_packet(self, key):
+        db = self.db.db
+        if key in db and self.new_data[key]:
+            data_json = json.dumps(db[key])
+            node_id = self.node_id
+            logger_mqtt.debug(f'{key}: {data_json}')
+            await self.mqtt.publish(f'{self.entity_prfx}{node_id}{key}', data_json)  # noqa: E501
+            self.new_data[key] = False
+
+    async def __register_home_assistant(self) -> None:
+        '''register all our topics at home assistant'''
+        for data_json, component, node_id, id in self.db.ha_confs(
+                self.entity_prfx, self.node_id, self.unique_id,
+                self.sug_area):
+            logger_mqtt.debug(f"MQTT Register: cmp:'{component}'"
+                              f" node_id:'{node_id}' {data_json}")
+            await self.mqtt.publish(f"{self.discovery_prfx}{component}"
+                                    f"/{node_id}{id}/config", data_json)
+
+        self.db.reg_clr_at_midnight(f'{self.entity_prfx}{self.node_id}')
+
+    def close(self) -> None:
+        logging.debug(f'InverterG3P.close() l{self.l_addr} | r{self.r_addr}')
+        super().close()         # call close handler in the parent class
+#        logger.debug (f'Inverter refs: {gc.get_referrers(self)}')
+
+    def __del__(self):
+        logging.debug("InverterG3P.__del__")
+        super().__del__()

app/src/gen3plus/solarman_v5.py

@@ -0,0 +1,670 @@
+import struct
+# import json
+import logging
+import time
+import asyncio
+from datetime import datetime
+
+if __name__ == "app.src.gen3plus.solarman_v5":
+    from app.src.messages import hex_dump_memory, Message, State
+    from app.src.modbus import Modbus
+    from app.src.my_timer import Timer
+    from app.src.config import Config
+    from app.src.gen3plus.infos_g3p import InfosG3P
+    from app.src.infos import Register
+else:  # pragma: no cover
+    from messages import hex_dump_memory, Message, State
+    from config import Config
+    from modbus import Modbus
+    from my_timer import Timer
+    from gen3plus.infos_g3p import InfosG3P
+    from infos import Register
+# import traceback
+
+logger = logging.getLogger('msg')
+
+
+class Sequence():
+    def __init__(self, server_side: bool):
+        self.rcv_idx = 0
+        self.snd_idx = 0
+        self.server_side = server_side
+
+    def set_recv(self, val: int):
+        if self.server_side:
+            self.rcv_idx = val >> 8
+            self.snd_idx = val & 0xff
+        else:
+            self.rcv_idx = val & 0xff
+            self.snd_idx = val >> 8
+
+    def get_send(self):
+        self.snd_idx += 1
+        self.snd_idx &= 0xff
+        if self.server_side:
+            return (self.rcv_idx << 8) | self.snd_idx
+        else:
+            return (self.snd_idx << 8) | self.rcv_idx
+
+    def __str__(self):
+        return f'{self.rcv_idx:02x}:{self.snd_idx:02x}'
+
+
+class SolarmanV5(Message):
+    AT_CMD = 1
+    MB_RTU_CMD = 2
+    MB_START_TIMEOUT = 40
+    '''start delay for Modbus polling in server mode'''
+    MB_REGULAR_TIMEOUT = 60
+    '''regular Modbus polling time in server mode'''
+    MB_CLIENT_DATA_UP = 30
+    '''Data up time in client mode'''
+
+    def __init__(self, server_side: bool, client_mode: bool):
+        super().__init__(server_side, self.send_modbus_cb, mb_timeout=5)
+
+        self.header_len = 11  # overwrite construcor in class Message
+        self.control = 0
+        self.seq = Sequence(server_side)
+        self.snr = 0
+        self.db = InfosG3P(client_mode)
+        self.time_ofs = 0
+        self.forward_at_cmd_resp = False
+        self.no_forwarding = False
+        '''not allowed to connect to TSUN cloud by connection type'''
+        self.switch = {
+
+            0x4210: self.msg_data_ind,   # real time data
+            0x1210: self.msg_response,   # at least every 5 minutes
+
+            0x4710: self.msg_hbeat_ind,  # heatbeat
+            0x1710: self.msg_response,   # every 2 minutes
+
+            # every 3 hours comes a sync seuqence:
+            # 00:00:00  0x4110   device data     ftype: 0x02
+            # 00:00:02  0x4210   real time data  ftype: 0x01
+            # 00:00:03  0x4210   real time data  ftype: 0x81
+            # 00:00:05  0x4310   wifi data       ftype: 0x81    sub-id 0x0018: 0c   # noqa: E501
+            # 00:00:06  0x4310   wifi data       ftype: 0x81    sub-id 0x0018: 1c   # noqa: E501
+            # 00:00:07  0x4310   wifi data       ftype: 0x01    sub-id 0x0018: 0c   # noqa: E501
+            # 00:00:08  0x4810   options?        ftype: 0x01
+
+            0x4110: self.msg_dev_ind,     # device data, sync start
+            0x1110: self.msg_response,    # every 3 hours
+
+            0x4310: self.msg_sync_start,  # regulary after 3-6 hours
+            0x1310: self.msg_response,
+            0x4810: self.msg_sync_end,    # sync end
+            0x1810: self.msg_response,
+
+            #
+            # MODbus or AT cmd
+            0x4510: self.msg_command_req,  # from server
+            0x1510: self.msg_command_rsp,     # from inverter
+            # 0x0510: self.msg_command_rsp,     # from inverter
+        }
+
+        self.log_lvl = {
+
+            0x4210: logging.INFO,   # real time data
+            0x1210: logging.INFO,   # at least every 5 minutes
+
+            0x4710: logging.DEBUG,  # heatbeat
+            0x1710: logging.DEBUG,  # every 2 minutes
+
+            0x4110: logging.INFO,   # device data, sync start
+            0x1110: logging.INFO,   # every 3 hours
+
+            0x4310: logging.INFO,   # regulary after 3-6 hours
+            0x1310: logging.INFO,
+
+            0x4810: logging.INFO,   # sync end
+            0x1810: logging.INFO,
+
+            #
+            # MODbus or AT cmd
+            0x4510: logging.INFO,  # from server
+            0x1510: self.get_cmd_rsp_log_lvl,
+        }
+        self.modbus_elms = 0    # for unit tests
+        g3p_cnf = Config.get('gen3plus')
+
+        if 'at_acl' in g3p_cnf:  # pragma: no cover
+            self.at_acl = g3p_cnf['at_acl']
+
+        self.node_id = 'G3P'  # will be overwritten in __set_serial_no
+        self.mb_timer = Timer(self.mb_timout_cb, self.node_id)
+        self.mb_timeout = self.MB_REGULAR_TIMEOUT
+        self.mb_start_timeout = self.MB_START_TIMEOUT
+        '''timer value for next Modbus polling request'''
+        self.modbus_polling = False
+
+    '''
+    Our puplic methods
+    '''
+    def close(self) -> None:
+        logging.debug('Solarman.close()')
+        if self.server_side:
+            # set inverter state to offline, if output power is very low
+            logging.debug('close power: '
+                          f'{self.db.get_db_value(Register.OUTPUT_POWER, -1)}')
+            if self.db.get_db_value(Register.OUTPUT_POWER, 999) < 2:
+                self.db.set_db_def_value(Register.INVERTER_STATUS, 0)
+                self.new_data['env'] = True
+
+        # we have references to methods of this class in self.switch
+        # so we have to erase self.switch, otherwise this instance can't be
+        # deallocated by the garbage collector ==> we get a memory leak
+        self.switch.clear()
+        self.log_lvl.clear()
+        self.state = State.closed
+        self.mb_timer.close()
+        super().close()
+
+    async def send_start_cmd(self, snr: int, host: str,
+                             start_timeout=MB_CLIENT_DATA_UP):
+        self.no_forwarding = True
+        self.snr = snr
+        self.__set_serial_no(snr)
+        self.mb_timeout = start_timeout
+        self.db.set_db_def_value(Register.IP_ADDRESS, host)
+        self.db.set_db_def_value(Register.POLLING_INTERVAL,
+                                 self.mb_timeout)
+        self.db.set_db_def_value(Register.HEARTBEAT_INTERVAL,
+                                 120)  # fixme
+        self.new_data['controller'] = True
+
+        self.state = State.up
+        self._send_modbus_cmd(Modbus.READ_REGS, 0x3000, 48, logging.DEBUG)
+        self.mb_timer.start(self.mb_timeout)
+
+    def new_state_up(self):
+        if self.state is not State.up:
+            self.state = State.up
+            if (self.modbus_polling):
+                self.mb_timer.start(self.mb_start_timeout)
+                self.db.set_db_def_value(Register.POLLING_INTERVAL,
+                                         self.mb_timeout)
+
+    def __set_serial_no(self, snr: int):
+        serial_no = str(snr)
+        if self.unique_id == serial_no:
+            logger.debug(f'SerialNo: {serial_no}')
+        else:
+            found = False
+            inverters = Config.get('inverters')
+            # logger.debug(f'Inverters: {inverters}')
+
+            for inv in inverters.values():
+                # logger.debug(f'key: {key} -> {inv}')
+                if (type(inv) is dict and 'monitor_sn' in inv
+                   and inv['monitor_sn'] == snr):
+                    found = True
+                    self.node_id = inv['node_id']
+                    self.sug_area = inv['suggested_area']
+                    self.modbus_polling = inv['modbus_polling']
+                    logger.debug(f'SerialNo {serial_no} allowed! area:{self.sug_area}')  # noqa: E501
+                    self.db.set_pv_module_details(inv)
+
+            if not found:
+                self.node_id = ''
+                self.sug_area = ''
+                if 'allow_all' not in inverters or not inverters['allow_all']:
+                    self.inc_counter('Unknown_SNR')
+                    self.unique_id = None
+                    logger.warning(f'ignore message from unknow inverter! (SerialNo: {serial_no})')  # noqa: E501
+                    return
+                logger.debug(f'SerialNo {serial_no} not known but accepted!')
+
+            self.unique_id = serial_no
+
+    def read(self) -> float:
+        '''process all received messages in the _recv_buffer'''
+        self._read()
+        while True:
+            if not self.header_valid:
+                self.__parse_header(self._recv_buffer, len(self._recv_buffer))
+
+            if self.header_valid and len(self._recv_buffer) >= \
+               (self.header_len + self.data_len+2):
+                log_lvl = self.log_lvl.get(self.control, logging.WARNING)
+                if callable(log_lvl):
+                    log_lvl = log_lvl()
+                hex_dump_memory(log_lvl, f'Received from {self.addr}:',
+                                self._recv_buffer, self.header_len +
+                                self.data_len+2)
+                if self.__trailer_is_ok(self._recv_buffer, self.header_len
+                                        + self.data_len + 2):
+                    if self.state == State.init:
+                        self.state = State.received
+
+                    self.__set_serial_no(self.snr)
+                    self.__dispatch_msg()
+                self.__flush_recv_msg()
+            else:
+                return 0  # wait 0s before sending a response
+
+    def forward(self, buffer, buflen) -> None:
+        '''add the actual receive msg to the forwarding queue'''
+        if self.no_forwarding:
+            return
+        tsun = Config.get('solarman')
+        if tsun['enabled']:
+            self._forward_buffer += buffer[:buflen]
+            hex_dump_memory(logging.DEBUG, 'Store for forwarding:',
+                            buffer, buflen)
+
+            fnc = self.switch.get(self.control, self.msg_unknown)
+            logger.info(self.__flow_str(self.server_side, 'forwrd') +
+                        f' Ctl: {int(self.control):#04x}'
+                        f' Msg: {fnc.__name__!r}')
+        return
+
+    def _init_new_client_conn(self) -> bool:
+        return False
+
+    '''
+    Our private methods
+    '''
+    def __flow_str(self, server_side: bool, type: str):  # noqa: F821
+        switch = {
+            'rx':      '  <',
+            'tx':      '  >',
+            'forwrd':  '<< ',
+            'drop':    ' xx',
+            'rxS':     '>  ',
+            'txS':     '<  ',
+            'forwrdS': ' >>',
+            'dropS':   'xx ',
+        }
+        if server_side:
+            type += 'S'
+        return switch.get(type, '???')
+
+    def _timestamp(self):
+        # utc as epoche
+        return int(time.time())    # pragma: no cover
+
+    def _heartbeat(self) -> int:
+        return 60                  # pragma: no cover
+
+    def __parse_header(self, buf: bytes, buf_len: int) -> None:
+
+        if (buf_len < self.header_len):  # enough bytes for complete header?
+            return
+
+        result = struct.unpack_from('<BHHHL', buf, 0)
+
+        # store parsed header values in the class
+        start = result[0]            # start byte
+        self.data_len = result[1]    # len of variable id string
+        self.control = result[2]
+        self.seq.set_recv(result[3])
+        self.snr = result[4]
+
+        if start != 0xA5:
+            hex_dump_memory(logging.ERROR,
+                            'Drop packet w invalid start byte from'
+                            f' {self.addr}:', buf, buf_len)
+
+            self.inc_counter('Invalid_Msg_Format')
+            # erase broken recv buffer
+            self._recv_buffer = bytearray()
+            return
+        self.header_valid = True
+        return
+
+    def __trailer_is_ok(self, buf: bytes, buf_len: int) -> bool:
+        crc = buf[self.data_len+11]
+        stop = buf[self.data_len+12]
+        if stop != 0x15:
+            hex_dump_memory(logging.ERROR,
+                            'Drop packet w invalid stop byte from '
+                            f'{self.addr}:', buf, buf_len)
+            self.inc_counter('Invalid_Msg_Format')
+            if len(self._recv_buffer) > (self.data_len+13):
+                next_start = buf[self.data_len+13]
+                if next_start != 0xa5:
+                    # erase broken recv buffer
+                    self._recv_buffer = bytearray()
+
+            return False
+
+        check = sum(buf[1:buf_len-2]) & 0xff
+        if check != crc:
+            self.inc_counter('Invalid_Msg_Format')
+            logger.debug(f'CRC {int(crc):#02x} {int(check):#08x}'
+                         f' Stop:{int(stop):#02x}')
+            # start & stop byte are valid, discard only this message
+            return False
+
+        return True
+
+    def __build_header(self, ctrl) -> None:
+        '''build header for new transmit message'''
+        self.send_msg_ofs = len(self._send_buffer)
+
+        self._send_buffer += struct.pack(
+            '<BHHHL', 0xA5, 0, ctrl, self.seq.get_send(), self.snr)
+        fnc = self.switch.get(ctrl, self.msg_unknown)
+        logger.info(self.__flow_str(self.server_side, 'tx') +
+                    f' Ctl: {int(ctrl):#04x} Msg: {fnc.__name__!r}')
+
+    def __finish_send_msg(self) -> None:
+        '''finish the transmit message, set lenght and checksum'''
+        _len = len(self._send_buffer) - self.send_msg_ofs
+        struct.pack_into('<H', self._send_buffer, self.send_msg_ofs+1, _len-11)
+        check = sum(self._send_buffer[self.send_msg_ofs+1:self.send_msg_ofs +
+                                      _len]) & 0xff
+        self._send_buffer += struct.pack('<BB', check, 0x15)    # crc & stop
+
+    def _update_header(self, _forward_buffer):
+        '''update header for message before forwarding,
+        set sequence and checksum'''
+        _len = len(_forward_buffer)
+        ofs = 0
+        while ofs < _len:
+            result = struct.unpack_from('<BH', _forward_buffer, ofs)
+            data_len = result[1]    # len of variable id string
+
+            struct.pack_into('<H', _forward_buffer, ofs+5,
+                             self.seq.get_send())
+
+            check = sum(_forward_buffer[ofs+1:ofs+data_len+11]) & 0xff
+            struct.pack_into('<B', _forward_buffer, ofs+data_len+11, check)
+            ofs += (13 + data_len)
+
+    def __dispatch_msg(self) -> None:
+        fnc = self.switch.get(self.control, self.msg_unknown)
+        if self.unique_id:
+            logger.info(self.__flow_str(self.server_side, 'rx') +
+                        f' Ctl: {int(self.control):#04x}' +
+                        f' Msg: {fnc.__name__!r}')
+            fnc()
+        else:
+            logger.info(self.__flow_str(self.server_side, 'drop') +
+                        f' Ctl: {int(self.control):#04x}' +
+                        f' Msg: {fnc.__name__!r}')
+
+    def __flush_recv_msg(self) -> None:
+        self._recv_buffer = self._recv_buffer[(self.header_len +
+                                               self.data_len+2):]
+        self.header_valid = False
+
+    def __send_ack_rsp(self, msgtype, ftype, ack=1):
+        self.__build_header(msgtype)
+        self._send_buffer += struct.pack('<BBLL', ftype, ack,
+                                         self._timestamp(),
+                                         self._heartbeat())
+        self.__finish_send_msg()
+
+    def send_modbus_cb(self, pdu: bytearray, log_lvl: int, state: str):
+        if self.state != State.up:
+            logger.warning(f'[{self.node_id}] ignore MODBUS cmd,'
+                           ' cause the state is not UP anymore')
+            return
+        self.__build_header(0x4510)
+        self._send_buffer += struct.pack('<BHLLL', self.MB_RTU_CMD,
+                                         0x2b0, 0, 0, 0)
+        self._send_buffer += pdu
+        self.__finish_send_msg()
+        hex_dump_memory(log_lvl, f'Send Modbus {state}:{self.addr}:',
+                        self._send_buffer, len(self._send_buffer))
+        self.writer.write(self._send_buffer)
+        self._send_buffer = bytearray(0)  # self._send_buffer[sent:]
+
+    def _send_modbus_cmd(self, func, addr, val, log_lvl) -> None:
+        if self.state != State.up:
+            logger.log(log_lvl, f'[{self.node_id}] ignore MODBUS cmd,'
+                       ' as the state is not UP')
+            return
+        self.mb.build_msg(Modbus.INV_ADDR, func, addr, val, log_lvl)
+
+    async def send_modbus_cmd(self, func, addr, val, log_lvl) -> None:
+        self._send_modbus_cmd(func, addr, val, log_lvl)
+
+    def mb_timout_cb(self, exp_cnt):
+        self.mb_timer.start(self.mb_timeout)
+
+        self._send_modbus_cmd(Modbus.READ_REGS, 0x3000, 48, logging.DEBUG)
+
+        if 1 == (exp_cnt % 30):
+            # logging.info("Regular Modbus Status request")
+            self._send_modbus_cmd(Modbus.READ_REGS, 0x2000, 96, logging.DEBUG)
+
+    def at_cmd_forbidden(self, cmd: str, connection: str) -> bool:
+        return not cmd.startswith(tuple(self.at_acl[connection]['allow'])) or \
+                cmd.startswith(tuple(self.at_acl[connection]['block']))
+
+    async def send_at_cmd(self, AT_cmd: str) -> None:
+        if self.state != State.up:
+            logger.warning(f'[{self.node_id}] ignore AT+ cmd,'
+                           ' as the state is not UP')
+            return
+        AT_cmd = AT_cmd.strip()
+
+        if self.at_cmd_forbidden(cmd=AT_cmd, connection='mqtt'):
+            data_json = f'\'{AT_cmd}\' is forbidden'
+            node_id = self.node_id
+            key = 'at_resp'
+            logger.info(f'{key}: {data_json}')
+            await self.mqtt.publish(f'{self.entity_prfx}{node_id}{key}', data_json)  # noqa: E501
+            return
+
+        self.forward_at_cmd_resp = False
+        self.__build_header(0x4510)
+        self._send_buffer += struct.pack(f'<BHLLL{len(AT_cmd)}sc', self.AT_CMD,
+                                         2, 0, 0, 0, AT_cmd.encode('utf-8'),
+                                         b'\r')
+        self.__finish_send_msg()
+        try:
+            await self.async_write('Send AT Command:')
+        except Exception:
+            self._send_buffer = bytearray(0)
+
+    def __forward_msg(self):
+        self.forward(self._recv_buffer, self.header_len+self.data_len+2)
+
+    def __build_model_name(self):
+        db = self.db
+        MaxPow = db.get_db_value(Register.MAX_DESIGNED_POWER, 0)
+        Rated = db.get_db_value(Register.RATED_POWER, 0)
+        Model = None
+        if MaxPow == 2000:
+            if Rated == 800 or Rated == 600:
+                Model = f'TSOL-MS{MaxPow}({Rated})'
+            else:
+                Model = f'TSOL-MS{MaxPow}'
+        elif MaxPow == 1800 or MaxPow == 1600:
+            Model = f'TSOL-MS{MaxPow}'
+        if Model:
+            logger.info(f'Model: {Model}')
+            self.db.set_db_def_value(Register.EQUIPMENT_MODEL, Model)
+
+    def __process_data(self, ftype, ts):
+        inv_update = False
+        msg_type = self.control >> 8
+        for key, update in self.db.parse(self._recv_buffer, msg_type, ftype,
+                                         self.node_id):
+            if update:
+                if key == 'inverter':
+                    inv_update = True
+                self._set_mqtt_timestamp(key, ts)
+                self.new_data[key] = True
+
+        if inv_update:
+            self.__build_model_name()
+    '''
+    Message handler methods
+    '''
+    def msg_unknown(self):
+        logger.warning(f"Unknow Msg: ID:{int(self.control):#04x}")
+        self.inc_counter('Unknown_Msg')
+        self.__forward_msg()
+
+    def msg_dev_ind(self):
+        data = self._recv_buffer[self.header_len:]
+        result = struct.unpack_from('<BLLL', data, 0)
+        ftype = result[0]  # always 2
+        total = result[1]
+        tim = result[2]
+        res = result[3]  # always zero
+        logger.info(f'frame type:{ftype:02x}'
+                    f' timer:{tim:08x}s  null:{res}')
+        if self.time_ofs:
+            #     dt = datetime.fromtimestamp(total + self.time_ofs)
+            #     logger.info(f'ts: {dt.strftime("%Y-%m-%d %H:%M:%S")}')
+            ts = total + self.time_ofs
+        else:
+            ts = None
+        self.__process_data(ftype, ts)
+        self.__forward_msg()
+        self.__send_ack_rsp(0x1110, ftype)
+
+    def msg_data_ind(self):
+        data = self._recv_buffer
+        result = struct.unpack_from('<BHLLLHL', data, self.header_len)
+        ftype = result[0]  # 1 or 0x81
+        total = result[2]
+        tim = result[3]
+        if 1 == ftype:
+            self.time_ofs = result[4]
+        unkn = result[5]
+        cnt = result[6]
+        logger.info(f'ftype:{ftype:02x} timer:{tim:08x}s'
+                    f' ??: {unkn:04x} cnt:{cnt}')
+        if self.time_ofs:
+            #     dt = datetime.fromtimestamp(total + self.time_ofs)
+            #     logger.info(f'ts: {dt.strftime("%Y-%m-%d %H:%M:%S")}')
+            ts = total + self.time_ofs
+        else:
+            ts = None
+
+        self.__process_data(ftype, ts)
+        self.__forward_msg()
+        self.__send_ack_rsp(0x1210, ftype)
+        self.new_state_up()
+
+    def msg_sync_start(self):
+        data = self._recv_buffer[self.header_len:]
+        result = struct.unpack_from('<BLLL', data, 0)
+        ftype = result[0]
+        total = result[1]
+        self.time_ofs = result[3]
+
+        dt = datetime.fromtimestamp(total + self.time_ofs)
+        logger.info(f'ts: {dt.strftime("%Y-%m-%d %H:%M:%S")}')
+
+        self.__forward_msg()
+        self.__send_ack_rsp(0x1310, ftype)
+
+    def msg_command_req(self):
+        data = self._recv_buffer[self.header_len:
+                                 self.header_len+self.data_len]
+        result = struct.unpack_from('<B', data, 0)
+        ftype = result[0]
+        if ftype == self.AT_CMD:
+            AT_cmd = data[15:].decode()
+            if self.at_cmd_forbidden(cmd=AT_cmd, connection='tsun'):
+                self.inc_counter('AT_Command_Blocked')
+                return
+            self.inc_counter('AT_Command')
+            self.forward_at_cmd_resp = True
+
+        elif ftype == self.MB_RTU_CMD:
+            if self.remoteStream.mb.recv_req(data[15:],
+                                             self.remoteStream.
+                                             __forward_msg):
+                self.inc_counter('Modbus_Command')
+            else:
+                logger.error('Invalid Modbus Msg')
+                self.inc_counter('Invalid_Msg_Format')
+            return
+
+        self.__forward_msg()
+
+    def publish_mqtt(self, key, data):  # pragma: no cover
+        asyncio.ensure_future(
+            self.mqtt.publish(key, data))
+
+    def get_cmd_rsp_log_lvl(self) -> int:
+        ftype = self._recv_buffer[self.header_len]
+        if ftype == self.AT_CMD:
+            if self.forward_at_cmd_resp:
+                return logging.INFO
+            return logging.DEBUG
+        elif ftype == self.MB_RTU_CMD:
+            if self.server_side:
+                return self.mb.last_log_lvl
+
+        return logging.WARNING
+
+    def msg_command_rsp(self):
+        data = self._recv_buffer[self.header_len:
+                                 self.header_len+self.data_len]
+        ftype = data[0]
+        if ftype == self.AT_CMD:
+            if not self.forward_at_cmd_resp:
+                data_json = data[14:].decode("utf-8")
+                node_id = self.node_id
+                key = 'at_resp'
+                logger.info(f'{key}: {data_json}')
+                self.publish_mqtt(f'{self.entity_prfx}{node_id}{key}', data_json)  # noqa: E501
+                return
+        elif ftype == self.MB_RTU_CMD:
+            valid = data[1]
+            modbus_msg_len = self.data_len - 14
+            # logger.debug(f'modbus_len:{modbus_msg_len} accepted:{valid}')
+            if valid == 1 and modbus_msg_len > 4:
+                # logger.info(f'first byte modbus:{data[14]}')
+                inv_update = False
+                self.modbus_elms = 0
+
+                for key, update, _ in self.mb.recv_resp(self.db, data[14:],
+                                                        self.node_id):
+                    self.modbus_elms += 1
+                    if update:
+                        if key == 'inverter':
+                            inv_update = True
+                        self._set_mqtt_timestamp(key, self._timestamp())
+                        self.new_data[key] = True
+
+                if inv_update:
+                    self.__build_model_name()
+            return
+        self.__forward_msg()
+
+    def msg_hbeat_ind(self):
+        data = self._recv_buffer[self.header_len:]
+        result = struct.unpack_from('<B', data, 0)
+        ftype = result[0]
+
+        self.__forward_msg()
+        self.__send_ack_rsp(0x1710, ftype)
+        self.new_state_up()
+
+    def msg_sync_end(self):
+        data = self._recv_buffer[self.header_len:]
+        result = struct.unpack_from('<BLLL', data, 0)
+        ftype = result[0]
+        total = result[1]
+        self.time_ofs = result[3]
+
+        dt = datetime.fromtimestamp(total + self.time_ofs)
+        logger.info(f'ts: {dt.strftime("%Y-%m-%d %H:%M:%S")}')
+
+        self.__forward_msg()
+        self.__send_ack_rsp(0x1810, ftype)
+
+    def msg_response(self):
+        data = self._recv_buffer[self.header_len:]
+        result = struct.unpack_from('<BBLL', data, 0)
+        ftype = result[0]  # always 2
+        valid = result[1] == 1  # status
+        ts = result[2]
+        set_hb = result[3]  # always 60 or 120
+        logger.debug(f'ftype:{ftype} accepted:{valid}'
+                     f' ts:{ts:08x}  nextHeartbeat: {set_hb}s')
+
+        dt = datetime.fromtimestamp(ts)
+        logger.debug(f'ts: {dt.strftime("%Y-%m-%d %H:%M:%S")}')

app/src/infos.py

@@ -1,356 +1,645 @@
-import struct, json, logging, os
+import logging
+import json
+import os
+from enum import Enum
+from typing import Generator
 
 
+class ProxyMode(Enum):
+    SERVER = 1
+    CLIENT = 2
 
-class Infos:    
+
+class Register(Enum):
+    COLLECTOR_FW_VERSION = 1
+    CHIP_TYPE = 2
+    CHIP_MODEL = 3
+    TRACE_URL = 4
+    LOGGER_URL = 5
+    PRODUCT_NAME = 20
+    MANUFACTURER = 21
+    VERSION = 22
+    SERIAL_NUMBER = 23
+    EQUIPMENT_MODEL = 24
+    NO_INPUTS = 25
+    MAX_DESIGNED_POWER = 26
+    OUTPUT_COEFFICIENT = 27
+    INVERTER_CNT = 50
+    UNKNOWN_SNR = 51
+    UNKNOWN_MSG = 52
+    INVALID_DATA_TYPE = 53
+    INTERNAL_ERROR = 54
+    UNKNOWN_CTRL = 55
+    OTA_START_MSG = 56
+    SW_EXCEPTION = 57
+    INVALID_MSG_FMT = 58
+    AT_COMMAND = 59
+    MODBUS_COMMAND = 60
+    AT_COMMAND_BLOCKED = 61
+    OUTPUT_POWER = 83
+    RATED_POWER = 84
+    INVERTER_TEMP = 85
+    INVERTER_STATUS = 86
+    PV1_VOLTAGE = 100
+    PV1_CURRENT = 101
+    PV1_POWER = 102
+    PV1_MANUFACTURER = 103
+    PV1_MODEL = 104
+    PV2_VOLTAGE = 110
+    PV2_CURRENT = 111
+    PV2_POWER = 112
+    PV2_MANUFACTURER = 113
+    PV2_MODEL = 114
+    PV3_VOLTAGE = 120
+    PV3_CURRENT = 121
+    PV3_POWER = 122
+    PV3_MANUFACTURER = 123
+    PV3_MODEL = 124
+    PV4_VOLTAGE = 130
+    PV4_CURRENT = 131
+    PV4_POWER = 132
+    PV4_MANUFACTURER = 133
+    PV4_MODEL = 134
+    PV5_VOLTAGE = 140
+    PV5_CURRENT = 141
+    PV5_POWER = 142
+    PV5_MANUFACTURER = 143
+    PV5_MODEL = 144
+    PV6_VOLTAGE = 150
+    PV6_CURRENT = 151
+    PV6_POWER = 152
+    PV6_MANUFACTURER = 153
+    PV6_MODEL = 154
+    PV1_DAILY_GENERATION = 200
+    PV1_TOTAL_GENERATION = 201
+    PV2_DAILY_GENERATION = 210
+    PV2_TOTAL_GENERATION = 211
+    PV3_DAILY_GENERATION = 220
+    PV3_TOTAL_GENERATION = 221
+    PV4_DAILY_GENERATION = 230
+    PV4_TOTAL_GENERATION = 231
+    PV5_DAILY_GENERATION = 240
+    PV5_TOTAL_GENERATION = 241
+    PV6_DAILY_GENERATION = 250
+    PV6_TOTAL_GENERATION = 251
+    GRID_VOLTAGE = 300
+    GRID_CURRENT = 301
+    GRID_FREQUENCY = 302
+    DAILY_GENERATION = 303
+    TOTAL_GENERATION = 304
+    COMMUNICATION_TYPE = 400
+    SIGNAL_STRENGTH = 401
+    POWER_ON_TIME = 402
+    COLLECT_INTERVAL = 403
+    DATA_UP_INTERVAL = 404
+    CONNECT_COUNT = 405
+    HEARTBEAT_INTERVAL = 406
+    IP_ADDRESS = 407
+    POLLING_INTERVAL = 408
+    EVENT_401 = 500
+    EVENT_402 = 501
+    EVENT_403 = 502
+    EVENT_404 = 503
+    EVENT_405 = 504
+    EVENT_406 = 505
+    EVENT_407 = 506
+    EVENT_408 = 507
+    EVENT_409 = 508
+    EVENT_410 = 509
+    EVENT_411 = 510
+    EVENT_412 = 511
+    EVENT_413 = 512
+    EVENT_414 = 513
+    EVENT_415 = 514
+    EVENT_416 = 515
+    TS_INPUT = 600
+    TS_GRID = 601
+    TS_TOTAL = 602
+    VALUE_1 = 9000
+    TEST_REG1 = 10000
+    TEST_REG2 = 10001
+
+
+class ClrAtMidnight:
+    __clr_at_midnight = [Register.PV1_DAILY_GENERATION, Register.PV2_DAILY_GENERATION, Register.PV3_DAILY_GENERATION, Register.PV4_DAILY_GENERATION, Register.PV5_DAILY_GENERATION, Register.PV6_DAILY_GENERATION, Register.DAILY_GENERATION]   # noqa: E501
+    db = {}
+
+    @classmethod
+    def add(cls, keys: list, prfx: str, reg: Register) -> None:
+        if reg not in cls.__clr_at_midnight:
+            return
+
+        prfx += f'{keys[0]}'
+        dict = cls.db
+        if prfx not in dict:
+            dict[prfx] = {}
+        dict = dict[prfx]
+
+        for key in keys[1:-1]:
+            if key not in dict:
+                dict[key] = {}
+            dict = dict[key]
+        dict[keys[-1]] = 0
+
+    @classmethod
+    def elm(cls) -> Generator[tuple[str, dict], None, None]:
+        for reg, name in cls.db.items():
+            yield reg, name
+        cls.db = {}
+
+
+class Infos:
     stat = {}
     app_name = os.getenv('SERVICE_NAME', 'proxy')
-    version  = os.getenv('VERSION', 'unknown')
+    version = os.getenv('VERSION', 'unknown')
+
+    new_stat_data = {}
 
     @classmethod
     def static_init(cls):
-        logging.info('Initialize proxy statistics')
+        logging.debug('Initialize proxy statistics')
         # init proxy counter in the class.stat dictionary
         cls.stat['proxy'] = {}
         for key in cls.__info_defs:
             name = cls.__info_defs[key]['name']
-            if name[0]=='proxy':
+            if name[0] == 'proxy':
                 cls.stat['proxy'][name[1]] = 0
 
         # add values from the environment to the device definition table
         prxy = cls.__info_devs['proxy']
-        prxy['sw']  = cls.version
+        prxy['sw'] = cls.version
         prxy['mdl'] = cls.app_name
 
- 
     def __init__(self):
         self.db = {}
         self.tracer = logging.getLogger('data')
 
-    __info_devs={
-        'proxy':     {'singleton': True,  'name':'Proxy', 'mf':'Stefan Allius'},
-        'controller':{'via':'proxy',      'name':'Controller',     'mdl':0x00092f90, 'mf':0x000927c0, 'sw':0x00092ba8},
-        'inverter':  {'via':'controller', 'name':'Micro Inverter', 'mdl':0x00000032, 'mf':0x00000014, 'sw':0x0000001e},
-        'input_pv1': {'via':'inverter',   'name':'Module PV1'},
-        'input_pv2': {'via':'inverter',   'name':'Module PV2', 'dep':{'reg':0x00095b50, 'gte': 2}},
-        'input_pv3': {'via':'inverter',   'name':'Module PV3', 'dep':{'reg':0x00095b50, 'gte': 3}},
-        'input_pv4': {'via':'inverter',   'name':'Module PV4', 'dep':{'reg':0x00095b50, 'gte': 4}},
+    __info_devs = {
+        'proxy':      {'singleton': True,   'name': 'Proxy', 'mf': 'Stefan Allius'},  # noqa: E501
+        'controller': {'via': 'proxy',      'name': 'Controller',     'mdl': Register.CHIP_MODEL, 'mf': Register.CHIP_TYPE, 'sw': Register.COLLECTOR_FW_VERSION},  # noqa: E501
+        'inverter':   {'via': 'controller', 'name': 'Micro Inverter', 'mdl': Register.EQUIPMENT_MODEL, 'mf': Register.MANUFACTURER, 'sw': Register.VERSION},  # noqa: E501
+        'input_pv1':  {'via': 'inverter',   'name': 'Module PV1', 'mdl': Register.PV1_MODEL, 'mf': Register.PV1_MANUFACTURER},  # noqa: E501
+        'input_pv2':  {'via': 'inverter',   'name': 'Module PV2', 'mdl': Register.PV2_MODEL, 'mf': Register.PV2_MANUFACTURER, 'dep': {'reg': Register.NO_INPUTS, 'gte': 2}},  # noqa: E501
+        'input_pv3':  {'via': 'inverter',   'name': 'Module PV3', 'mdl': Register.PV3_MODEL, 'mf': Register.PV3_MANUFACTURER, 'dep': {'reg': Register.NO_INPUTS, 'gte': 3}},  # noqa: E501
+        'input_pv4':  {'via': 'inverter',   'name': 'Module PV4', 'mdl': Register.PV4_MODEL, 'mf': Register.PV4_MANUFACTURER, 'dep': {'reg': Register.NO_INPUTS, 'gte': 4}},  # noqa: E501
+        'input_pv5':  {'via': 'inverter',   'name': 'Module PV5', 'mdl': Register.PV5_MODEL, 'mf': Register.PV5_MANUFACTURER, 'dep': {'reg': Register.NO_INPUTS, 'gte': 5}},  # noqa: E501
+        'input_pv6':  {'via': 'inverter',   'name': 'Module PV6', 'mdl': Register.PV6_MODEL, 'mf': Register.PV6_MANUFACTURER, 'dep': {'reg': Register.NO_INPUTS, 'gte': 6}},  # noqa: E501
     }
 
-    __info_defs={
+    __comm_type_val_tpl = "{%set com_types = ['n/a','Wi-Fi', 'G4', 'G5', 'GPRS'] %}{{com_types[value_json['Communication_Type']|int(0)]|default(value_json['Communication_Type'])}}"    # noqa: E501
+    __status_type_val_tpl = "{%set inv_status = ['Off-line', 'On-grid', 'Off-grid'] %}{{inv_status[value_json['Inverter_Status']|int(0)]|default(value_json['Inverter_Status'])}}"    # noqa: E501
+    __rated_power_val_tpl = "{% if 'Rated_Power' in value_json and value_json['Rated_Power'] != None %}{{value_json['Rated_Power']|string() +' W'}}{% else %}{{ this.state }}{% endif %}"  # noqa: E501
+    __designed_power_val_tpl = '''
+{% if 'Max_Designed_Power' in value_json and
+      value_json['Max_Designed_Power'] != None %}
+  {% if value_json['Max_Designed_Power'] | int(0xffff) < 0x8000 %}
+    {{value_json['Max_Designed_Power']|string() +' W'}}
+  {% else %}
+    n/a
+  {% endif %}
+{% else %}
+  {{ this.state }}
+{% endif %}
+'''
+    __output_coef_val_tpl = "{% if 'Output_Coefficient' in value_json and value_json['Output_Coefficient'] != None %}{{value_json['Output_Coefficient']|string() +' %'}}{% else %}{{ this.state }}{% endif %}"  # noqa: E501
+
+    __info_defs = {
         # collector values used for device registration:
-            0x00092ba8:  {'name':['collector', 'Collector_Fw_Version'],       'level': logging.INFO,  'unit': ''},
-            0x000927c0:  {'name':['collector', 'Chip_Type'],                  'level': logging.DEBUG, 'unit': ''},
-            0x00092f90:  {'name':['collector', 'Chip_Model'],                 'level': logging.DEBUG, 'unit': ''},
-            0x00095a88:  {'name':['collector', 'Trace_URL'],                  'level': logging.DEBUG, 'unit': ''},
-            0x00095aec:  {'name':['collector', 'Logger_URL'],                 'level': logging.DEBUG, 'unit': ''},
-            0x00095b50:  {'name':['collector', 'No_Inputs'],                  'level': logging.DEBUG, 'unit': ''},
-       
+        Register.COLLECTOR_FW_VERSION:  {'name': ['collector', 'Collector_Fw_Version'],       'level': logging.INFO,  'unit': ''},  # noqa: E501
+        Register.CHIP_TYPE:  {'name': ['collector', 'Chip_Type'],        'singleton': False,  'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.CHIP_MODEL: {'name': ['collector', 'Chip_Model'],       'singleton': False,  'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.TRACE_URL:  {'name': ['collector', 'Trace_URL'],        'singleton': False,  'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.LOGGER_URL: {'name': ['collector', 'Logger_URL'],       'singleton': False,  'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+
         # inverter values used for device registration:
-            0x0000000a:  {'name':['inverter', 'Product_Name'],                'level': logging.DEBUG, 'unit': ''},
-            0x00000014:  {'name':['inverter', 'Manufacturer'],                'level': logging.DEBUG, 'unit': ''},
-            0x0000001e:  {'name':['inverter', 'Version'],                     'level': logging.INFO,  'unit': ''},
-            0x00000028:  {'name':['inverter', 'Serial_Number'],               'level': logging.DEBUG, 'unit': ''},
-            0x00000032:  {'name':['inverter', 'Equipment_Model'],             'level': logging.DEBUG, 'unit': ''},
-        
+        Register.PRODUCT_NAME:    {'name': ['inverter', 'Product_Name'],           'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.MANUFACTURER:    {'name': ['inverter', 'Manufacturer'],           'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.VERSION:         {'name': ['inverter', 'Version'],                'level': logging.INFO,  'unit': ''},  # noqa: E501
+        Register.SERIAL_NUMBER:   {'name': ['inverter', 'Serial_Number'],          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EQUIPMENT_MODEL: {'name': ['inverter', 'Equipment_Model'],        'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.NO_INPUTS:       {'name': ['inverter', 'No_Inputs'],              'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.MAX_DESIGNED_POWER: {'name': ['inverter',  'Max_Designed_Power'], 'level': logging.INFO,  'unit': 'W',    'ha': {'dev': 'inverter', 'dev_cla': None, 'stat_cla': None, 'id': 'designed_power_', 'val_tpl': __designed_power_val_tpl, 'name': 'Max Designed Power', 'icon': 'mdi:lightning-bolt', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.RATED_POWER:        {'name': ['inverter',  'Rated_Power'],        'level': logging.DEBUG, 'unit': 'W',    'ha': {'dev': 'inverter', 'dev_cla': None, 'stat_cla': None, 'id': 'rated_power_',    'val_tpl': __rated_power_val_tpl,    'name': 'Rated Power',        'icon': 'mdi:lightning-bolt', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.OUTPUT_COEFFICIENT: {'name': ['inverter',  'Output_Coefficient'], 'level': logging.INFO,  'unit': '%',    'ha': {'dev': 'inverter', 'dev_cla': None, 'stat_cla': None, 'id': 'output_coef_',    'val_tpl': __output_coef_val_tpl,    'name': 'Output Coefficient', 'icon': 'mdi:lightning-bolt', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.PV1_MANUFACTURER: {'name': ['inverter', 'PV1_Manufacturer'],      'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.PV1_MODEL:        {'name': ['inverter', 'PV1_Model'],             'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.PV2_MANUFACTURER: {'name': ['inverter', 'PV2_Manufacturer'],      'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.PV2_MODEL:        {'name': ['inverter', 'PV2_Model'],             'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.PV3_MANUFACTURER: {'name': ['inverter', 'PV3_Manufacturer'],      'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.PV3_MODEL:        {'name': ['inverter', 'PV3_Model'],             'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.PV4_MANUFACTURER: {'name': ['inverter', 'PV4_Manufacturer'],      'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.PV4_MODEL:        {'name': ['inverter', 'PV4_Model'],             'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.PV5_MANUFACTURER: {'name': ['inverter', 'PV5_Manufacturer'],      'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.PV5_MODEL:        {'name': ['inverter', 'PV5_Model'],             'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.PV6_MANUFACTURER: {'name': ['inverter', 'PV6_Manufacturer'],      'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.PV6_MODEL:        {'name': ['inverter', 'PV6_Model'],             'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+
         # proxy:
-            0xffffff00:  {'name':['proxy', 'Inverter_Cnt'], 'singleton': True,                                        'ha':{'dev':'proxy', 'comp' : 'sensor', 'dev_cla': None,       'stat_cla': None, 'id':'inv_count_',   'fmt':'| int', 'name': 'Active Inverter Connections',    'icon':'mdi:counter'}}, 
-            0xffffff01:  {'name':['proxy', 'Unknown_SNR'],  'singleton': True,                                        'ha':{'dev':'proxy', 'comp' : 'sensor', 'dev_cla': None,       'stat_cla': None, 'id':'unknown_snr_', 'fmt':'| int', 'name': 'Unknown Serial No', 'icon':'mdi:counter', 'ent_cat':'diagnostic'}}, 
-            0xffffff02:  {'name':['proxy', 'Unknown_Msg'],  'singleton': True,                                        'ha':{'dev':'proxy', 'comp' : 'sensor', 'dev_cla': None,       'stat_cla': None, 'id':'unknown_msg_', 'fmt':'| int', 'name': 'Unknown Msg Type',  'icon':'mdi:counter', 'ent_cat':'diagnostic'}}, 
-            0xffffff03:  {'name':['proxy', 'Invalid_Data_Type'],  'singleton': True,                                  'ha':{'dev':'proxy', 'comp' : 'sensor', 'dev_cla': None,       'stat_cla': None, 'id':'inv_data_type_', 'fmt':'| int', 'name': 'Invalid Data Type','icon':'mdi:counter', 'ent_cat':'diagnostic'}}, 
-            0xffffff04:  {'name':['proxy', 'Internal_Error'],  'singleton': True,                                     'ha':{'dev':'proxy', 'comp' : 'sensor', 'dev_cla': None,       'stat_cla': None, 'id':'intern_err_',  'fmt':'| int', 'name': 'Internal Error',    'icon':'mdi:counter', 'ent_cat':'diagnostic', 'en':False}}, 
-#            0xffffff03:  {'name':['proxy', 'Voltage'],                        'level': logging.DEBUG, 'unit': 'V',    'ha':{'dev':'proxy', 'dev_cla': 'voltage',     'stat_cla': 'measurement', 'id':'proxy_volt_',  'fmt':'| float','name': 'Grid Voltage'}},
+        Register.INVERTER_CNT:       {'name': ['proxy', 'Inverter_Cnt'],       'singleton': True,   'ha': {'dev': 'proxy', 'comp': 'sensor', 'dev_cla': None, 'stat_cla': None, 'id': 'inv_count_',     'fmt': '| int', 'name': 'Active Inverter Connections',    'icon': 'mdi:counter'}},  # noqa: E501
+        Register.UNKNOWN_SNR:        {'name': ['proxy', 'Unknown_SNR'],        'singleton': True,   'ha': {'dev': 'proxy', 'comp': 'sensor', 'dev_cla': None, 'stat_cla': None, 'id': 'unknown_snr_',   'fmt': '| int', 'name': 'Unknown Serial No',    'icon': 'mdi:counter', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.UNKNOWN_MSG:        {'name': ['proxy', 'Unknown_Msg'],        'singleton': True,   'ha': {'dev': 'proxy', 'comp': 'sensor', 'dev_cla': None, 'stat_cla': None, 'id': 'unknown_msg_',   'fmt': '| int', 'name': 'Unknown Msg Type',     'icon': 'mdi:counter', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.INVALID_DATA_TYPE:  {'name': ['proxy', 'Invalid_Data_Type'],  'singleton': True,   'ha': {'dev': 'proxy', 'comp': 'sensor', 'dev_cla': None, 'stat_cla': None, 'id': 'inv_data_type_', 'fmt': '| int', 'name': 'Invalid Data Type',    'icon': 'mdi:counter', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.INTERNAL_ERROR:     {'name': ['proxy', 'Internal_Error'],     'singleton': True,   'ha': {'dev': 'proxy', 'comp': 'sensor', 'dev_cla': None, 'stat_cla': None, 'id': 'intern_err_',    'fmt': '| int', 'name': 'Internal Error',       'icon': 'mdi:counter', 'ent_cat': 'diagnostic', 'en': False}},  # noqa: E501
+        Register.UNKNOWN_CTRL:       {'name': ['proxy', 'Unknown_Ctrl'],       'singleton': True,   'ha': {'dev': 'proxy', 'comp': 'sensor', 'dev_cla': None, 'stat_cla': None, 'id': 'unknown_ctrl_',  'fmt': '| int', 'name': 'Unknown Control Type', 'icon': 'mdi:counter', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.OTA_START_MSG:      {'name': ['proxy', 'OTA_Start_Msg'],      'singleton': True,   'ha': {'dev': 'proxy', 'comp': 'sensor', 'dev_cla': None, 'stat_cla': None, 'id': 'ota_start_cmd_', 'fmt': '| int', 'name': 'OTA Start Cmd',        'icon': 'mdi:counter', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.SW_EXCEPTION:       {'name': ['proxy', 'SW_Exception'],       'singleton': True,   'ha': {'dev': 'proxy', 'comp': 'sensor', 'dev_cla': None, 'stat_cla': None, 'id': 'sw_exception_',  'fmt': '| int', 'name': 'Internal SW Exception', 'icon': 'mdi:counter', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.INVALID_MSG_FMT:    {'name': ['proxy', 'Invalid_Msg_Format'], 'singleton': True,   'ha': {'dev': 'proxy', 'comp': 'sensor', 'dev_cla': None, 'stat_cla': None, 'id': 'inv_msg_fmt_',   'fmt': '| int', 'name': 'Invalid Message Format', 'icon': 'mdi:counter', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.AT_COMMAND:         {'name': ['proxy', 'AT_Command'],         'singleton': True,   'ha': {'dev': 'proxy', 'comp': 'sensor', 'dev_cla': None, 'stat_cla': None, 'id': 'at_cmd_',        'fmt': '| int', 'name': 'AT Command',           'icon': 'mdi:counter', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.AT_COMMAND_BLOCKED: {'name': ['proxy', 'AT_Command_Blocked'], 'singleton': True,   'ha': {'dev': 'proxy', 'comp': 'sensor', 'dev_cla': None, 'stat_cla': None, 'id': 'at_cmd_blocked_', 'fmt': '| int', 'name': 'AT Command Blocked',   'icon': 'mdi:counter', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.MODBUS_COMMAND:     {'name': ['proxy', 'Modbus_Command'],     'singleton': True,   'ha': {'dev': 'proxy', 'comp': 'sensor', 'dev_cla': None, 'stat_cla': None, 'id': 'modbus_cmd_',    'fmt': '| int', 'name': 'Modbus Command',       'icon': 'mdi:counter', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        # 0xffffff03:  {'name':['proxy', 'Voltage'],                        'level': logging.DEBUG, 'unit': 'V',    'ha':{'dev':'proxy', 'dev_cla': 'voltage',     'stat_cla': 'measurement', 'id':'proxy_volt_',  'fmt':'| float','name': 'Grid Voltage'}},  # noqa: E501
 
         # events
-            0x00000191:  {'name':['events', '401_'],                          'level': logging.DEBUG, 'unit': ''},
-            0x00000192:  {'name':['events', '402_'],                          'level': logging.DEBUG, 'unit': ''},
-            0x00000193:  {'name':['events', '403_'],                          'level': logging.DEBUG, 'unit': ''},
-            0x00000194:  {'name':['events', '404_'],                          'level': logging.DEBUG, 'unit': ''},
-            0x00000195:  {'name':['events', '405_'],                          'level': logging.DEBUG, 'unit': ''},
-            0x00000196:  {'name':['events', '406_'],                          'level': logging.DEBUG, 'unit': ''},
-            0x00000197:  {'name':['events', '407_'],                          'level': logging.DEBUG, 'unit': ''},
-            0x00000198:  {'name':['events', '408_'],                          'level': logging.DEBUG, 'unit': ''},
-            0x00000199:  {'name':['events', '409_'],                          'level': logging.DEBUG, 'unit': ''},
-            0x0000019a:  {'name':['events', '410_'],                          'level': logging.DEBUG, 'unit': ''},
-            0x0000019b:  {'name':['events', '411_'],                          'level': logging.DEBUG, 'unit': ''},
-            0x0000019c:  {'name':['events', '412_'],                          'level': logging.DEBUG, 'unit': ''},
-            0x0000019d:  {'name':['events', '413_'],                          'level': logging.DEBUG, 'unit': ''},
-            0x0000019e:  {'name':['events', '414_'],                          'level': logging.DEBUG, 'unit': ''},
-            0x0000019f:  {'name':['events', '415_GridFreqOverRating'],        'level': logging.DEBUG, 'unit': ''},
-            0x000001a0:  {'name':['events', '416_'],                          'level': logging.DEBUG, 'unit': ''},
-        
-        # grid measures:
-            0x000003e8:  {'name':['grid', 'Voltage'],                         'level': logging.DEBUG, 'unit': 'V',    'ha':{'dev':'inverter', 'dev_cla': 'voltage',     'stat_cla': 'measurement', 'id':'out_volt_',  'fmt':'| float','name': 'Grid Voltage','ent_cat':'diagnostic'}},
-            0x0000044c:  {'name':['grid', 'Current'],                         'level': logging.DEBUG, 'unit': 'A',    'ha':{'dev':'inverter', 'dev_cla': 'current',     'stat_cla': 'measurement', 'id':'out_cur_',   'fmt':'| float','name': 'Grid Current','ent_cat':'diagnostic'}},
-            0x000004b0:  {'name':['grid', 'Frequency'],                       'level': logging.DEBUG, 'unit': 'Hz',   'ha':{'dev':'inverter', 'dev_cla': 'frequency',   'stat_cla': 'measurement', 'id':'out_freq_',  'fmt':'| float','name': 'Grid Frequency','ent_cat':'diagnostic'}},
-            0x00000640:  {'name':['grid', 'Output_Power'],                    'level': logging.INFO,  'unit': 'W',    'ha':{'dev':'inverter', 'dev_cla': 'power',       'stat_cla': 'measurement', 'id':'out_power_', 'fmt':'| float','name': 'Power'}},
-            0x000005dc:  {'name':['env',  'Rated_Power'],                     'level': logging.DEBUG, 'unit': 'W',    'ha':{'dev':'inverter', 'dev_cla': None,          'stat_cla': None,          'id':'rated_power_', 'fmt':'| int', 'name': 'Rated Power','ent_cat':'diagnostic'}},
-            0x00000514:  {'name':['env',  'Inverter_Temp'],                   'level': logging.DEBUG, 'unit': '°C',   'ha':{'dev':'inverter', 'dev_cla': 'temperature', 'stat_cla': 'measurement', 'id':'temp_',   'fmt':'| int','name': 'Temperature'}},
+        Register.EVENT_401:  {'name': ['events', '401_'],                          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_402:  {'name': ['events', '402_'],                          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_403:  {'name': ['events', '403_'],                          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_404:  {'name': ['events', '404_'],                          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_405:  {'name': ['events', '405_'],                          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_406:  {'name': ['events', '406_'],                          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_407:  {'name': ['events', '407_'],                          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_408:  {'name': ['events', '408_'],                          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_409:  {'name': ['events', '409_No_Utility'],                'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_410:  {'name': ['events', '410_'],                          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_411:  {'name': ['events', '411_'],                          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_412:  {'name': ['events', '412_'],                          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_413:  {'name': ['events', '413_'],                          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_414:  {'name': ['events', '414_'],                          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_415:  {'name': ['events', '415_GridFreqOverRating'],        'level': logging.DEBUG, 'unit': ''},  # noqa: E501
+        Register.EVENT_416:  {'name': ['events', '416_'],                          'level': logging.DEBUG, 'unit': ''},  # noqa: E501
 
-        # input measures:           
-            0x000006a4:  {'name':['input', 'pv1', 'Voltage'],                 'level': logging.DEBUG, 'unit': 'V',    'ha':{'dev':'input_pv1', 'dev_cla': 'voltage', 'stat_cla': 'measurement', 'id':'volt_pv1_', 'val_tpl' :"{{ (value_json['pv1']['Voltage'] | float)}}", 'icon':'mdi:gauge','ent_cat':'diagnostic'}},
-            0x00000708:  {'name':['input', 'pv1', 'Current'],                 'level': logging.DEBUG, 'unit': 'A',    'ha':{'dev':'input_pv1', 'dev_cla': 'current', 'stat_cla': 'measurement', 'id':'cur_pv1_',  'val_tpl' :"{{ (value_json['pv1']['Current'] | float)}}", 'icon':'mdi:gauge','ent_cat':'diagnostic'}},
-            0x0000076c:  {'name':['input', 'pv1', 'Power'],                   'level': logging.INFO,  'unit': 'W',    'ha':{'dev':'input_pv1', 'dev_cla': 'power',   'stat_cla': 'measurement', 'id':'power_pv1_','val_tpl' :"{{ (value_json['pv1']['Power'] | float)}}"}},
-            0x000007d0:  {'name':['input', 'pv2', 'Voltage'],                 'level': logging.DEBUG, 'unit': 'V',    'ha':{'dev':'input_pv2', 'dev_cla': 'voltage', 'stat_cla': 'measurement', 'id':'volt_pv2_', 'val_tpl' :"{{ (value_json['pv2']['Voltage'] | float)}}", 'icon':'mdi:gauge','ent_cat':'diagnostic'}},
-            0x00000834:  {'name':['input', 'pv2', 'Current'],                 'level': logging.DEBUG, 'unit': 'A',    'ha':{'dev':'input_pv2', 'dev_cla': 'current', 'stat_cla': 'measurement', 'id':'cur_pv2_',  'val_tpl' :"{{ (value_json['pv2']['Current'] | float)}}", 'icon':'mdi:gauge','ent_cat':'diagnostic'}},
-            0x00000898:  {'name':['input', 'pv2', 'Power'],                   'level': logging.INFO,  'unit': 'W',    'ha':{'dev':'input_pv2', 'dev_cla': 'power',   'stat_cla': 'measurement', 'id':'power_pv2_','val_tpl' :"{{ (value_json['pv2']['Power'] | float)}}"}},
-            0x000008fc:  {'name':['input', 'pv3', 'Voltage'],                 'level': logging.DEBUG, 'unit': 'V',    'ha':{'dev':'input_pv3', 'dev_cla': 'voltage', 'stat_cla': 'measurement', 'id':'volt_pv3_', 'val_tpl' :"{{ (value_json['pv3']['Voltage'] | float)}}", 'icon':'mdi:gauge','ent_cat':'diagnostic'}},
-            0x00000960:  {'name':['input', 'pv3', 'Current'],                 'level': logging.DEBUG, 'unit': 'A',    'ha':{'dev':'input_pv3', 'dev_cla': 'current', 'stat_cla': 'measurement', 'id':'cur_pv3_',  'val_tpl' :"{{ (value_json['pv3']['Current'] | float)}}", 'icon':'mdi:gauge','ent_cat':'diagnostic'}},
-            0x000009c4:  {'name':['input', 'pv3', 'Power'],                   'level': logging.DEBUG, 'unit': 'W',    'ha':{'dev':'input_pv3', 'dev_cla': 'power',   'stat_cla': 'measurement', 'id':'power_pv3_','val_tpl' :"{{ (value_json['pv3']['Power'] | float)}}"}},
-            0x00000a28:  {'name':['input', 'pv4', 'Voltage'],                 'level': logging.DEBUG, 'unit': 'V',    'ha':{'dev':'input_pv4', 'dev_cla': 'voltage', 'stat_cla': 'measurement', 'id':'volt_pv4_', 'val_tpl' :"{{ (value_json['pv4']['Voltage'] | float)}}", 'icon':'mdi:gauge','ent_cat':'diagnostic'}},
-            0x00000a8c:  {'name':['input', 'pv4', 'Current'],                 'level': logging.DEBUG, 'unit': 'A',    'ha':{'dev':'input_pv4', 'dev_cla': 'current', 'stat_cla': 'measurement', 'id':'cur_pv4_',  'val_tpl' :"{{ (value_json['pv4']['Current'] | float)}}", 'icon':'mdi:gauge','ent_cat':'diagnostic'}},
-            0x00000af0:  {'name':['input', 'pv4', 'Power'],                   'level': logging.DEBUG, 'unit': 'W',    'ha':{'dev':'input_pv4', 'dev_cla': 'power',   'stat_cla': 'measurement', 'id':'power_pv4_','val_tpl' :"{{ (value_json['pv4']['Power'] | float)}}"}},
-            0x00000c1c:  {'name':['input', 'pv1', 'Daily_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha':{'dev':'input_pv1', 'dev_cla': 'energy', 'stat_cla': 'total_increasing', 'id':'daily_gen_pv1_','name': 'Daily Generation', 'val_tpl' :"{{ (value_json['pv1']['Daily_Generation'] | float)}}", 'icon':'mdi:solar-power-variant', 'must_incr':True}},          
-            0x00000c80:  {'name':['input', 'pv1', 'Total_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha':{'dev':'input_pv1', 'dev_cla': 'energy', 'stat_cla': 'total',            'id':'total_gen_pv1_','name': 'Total Generation', 'val_tpl' :"{{ (value_json['pv1']['Total_Generation'] | float)}}", 'icon':'mdi:solar-power', 'must_incr':True}},                    
-            0x00000ce4:  {'name':['input', 'pv2', 'Daily_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha':{'dev':'input_pv2', 'dev_cla': 'energy', 'stat_cla': 'total_increasing', 'id':'daily_gen_pv2_','name': 'Daily Generation', 'val_tpl' :"{{ (value_json['pv2']['Daily_Generation'] | float)}}", 'icon':'mdi:solar-power-variant', 'must_incr':True}},          
-            0x00000d48:  {'name':['input', 'pv2', 'Total_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha':{'dev':'input_pv2', 'dev_cla': 'energy', 'stat_cla': 'total',            'id':'total_gen_pv2_','name': 'Total Generation', 'val_tpl' :"{{ (value_json['pv2']['Total_Generation'] | float)}}", 'icon':'mdi:solar-power', 'must_incr':True}},          
-            0x00000dac:  {'name':['input', 'pv3', 'Daily_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha':{'dev':'input_pv3', 'dev_cla': 'energy', 'stat_cla': 'total_increasing', 'id':'daily_gen_pv3_','name': 'Daily Generation', 'val_tpl' :"{{ (value_json['pv3']['Daily_Generation'] | float)}}", 'icon':'mdi:solar-power-variant', 'must_incr':True}},          
-            0x00000e10:  {'name':['input', 'pv3', 'Total_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha':{'dev':'input_pv3', 'dev_cla': 'energy', 'stat_cla': 'total',            'id':'total_gen_pv3_','name': 'Total Generation', 'val_tpl' :"{{ (value_json['pv3']['Total_Generation'] | float)}}", 'icon':'mdi:solar-power', 'must_incr':True}},                    
-            0x00000e74:  {'name':['input', 'pv4', 'Daily_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha':{'dev':'input_pv4', 'dev_cla': 'energy', 'stat_cla': 'total_increasing', 'id':'daily_gen_pv4_','name': 'Daily Generation', 'val_tpl' :"{{ (value_json['pv4']['Daily_Generation'] | float)}}", 'icon':'mdi:solar-power-variant', 'must_incr':True}},          
-            0x00000ed8:  {'name':['input', 'pv4', 'Total_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha':{'dev':'input_pv4', 'dev_cla': 'energy', 'stat_cla': 'total',            'id':'total_gen_pv4_','name': 'Total Generation', 'val_tpl' :"{{ (value_json['pv4']['Total_Generation'] | float)}}", 'icon':'mdi:solar-power', 'must_incr':True}},          
-        # total:    
-            0x00000b54:  {'name':['total', 'Daily_Generation'],               'level': logging.INFO,  'unit': 'kWh',  'ha':{'dev':'inverter', 'dev_cla': 'energy', 'stat_cla': 'total_increasing', 'id':'daily_gen_', 'fmt':'| float','name': 'Daily Generation', 'icon':'mdi:solar-power-variant', 'must_incr':True}},
-            0x00000bb8:  {'name':['total', 'Total_Generation'],               'level': logging.INFO,  'unit': 'kWh',  'ha':{'dev':'inverter', 'dev_cla': 'energy', 'stat_cla': 'total',            'id':'total_gen_', 'fmt':'| float','name': 'Total Generation', 'icon':'mdi:solar-power', 'must_incr':True}},
+        # grid measures:
+        Register.TS_GRID:         {'name': ['grid', 'Timestamp'],                  'level': logging.INFO,  'unit': ''},  # noqa: E501
+        Register.GRID_VOLTAGE:    {'name': ['grid', 'Voltage'],                    'level': logging.DEBUG, 'unit': 'V',    'ha': {'dev': 'inverter', 'dev_cla': 'voltage',     'stat_cla': 'measurement', 'id': 'out_volt_',  'fmt': '| float', 'name': 'Grid Voltage', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.GRID_CURRENT:    {'name': ['grid', 'Current'],                    'level': logging.DEBUG, 'unit': 'A',    'ha': {'dev': 'inverter', 'dev_cla': 'current',     'stat_cla': 'measurement', 'id': 'out_cur_',   'fmt': '| float', 'name': 'Grid Current', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.GRID_FREQUENCY:  {'name': ['grid', 'Frequency'],                  'level': logging.DEBUG, 'unit': 'Hz',   'ha': {'dev': 'inverter', 'dev_cla': 'frequency',   'stat_cla': 'measurement', 'id': 'out_freq_',  'fmt': '| float', 'name': 'Grid Frequency', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.OUTPUT_POWER:    {'name': ['grid', 'Output_Power'],               'level': logging.INFO,  'unit': 'W',    'ha': {'dev': 'inverter', 'dev_cla': 'power',       'stat_cla': 'measurement', 'id': 'out_power_', 'fmt': '| float', 'name': 'Power'}},  # noqa: E501
+        Register.INVERTER_TEMP:   {'name': ['env',  'Inverter_Temp'],              'level': logging.DEBUG, 'unit': '°C',   'ha': {'dev': 'inverter', 'dev_cla': 'temperature', 'stat_cla': 'measurement', 'id': 'temp_',       'fmt': '| int', 'name': 'Temperature'}},  # noqa: E501
+        Register.INVERTER_STATUS: {'name': ['env',  'Inverter_Status'],            'level': logging.INFO,  'unit': '',     'ha': {'dev': 'inverter', 'comp': 'sensor', 'dev_cla': None, 'stat_cla': None, 'id': 'inv_status_', 'name': 'Inverter Status', 'val_tpl': __status_type_val_tpl,          'icon': 'mdi:power'}},  # noqa: E501
+
+        # input measures:
+        Register.TS_INPUT:     {'name': ['input', 'Timestamp'],                    'level': logging.INFO,  'unit': ''},  # noqa: E501
+        Register.PV1_VOLTAGE:  {'name': ['input', 'pv1', 'Voltage'],               'level': logging.DEBUG, 'unit': 'V',    'ha': {'dev': 'input_pv1', 'dev_cla': 'voltage', 'stat_cla': 'measurement', 'id': 'volt_pv1_',  'val_tpl': "{{ (value_json['pv1']['Voltage'] | float)}}", 'icon': 'mdi:gauge', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.PV1_CURRENT:  {'name': ['input', 'pv1', 'Current'],               'level': logging.DEBUG, 'unit': 'A',    'ha': {'dev': 'input_pv1', 'dev_cla': 'current', 'stat_cla': 'measurement', 'id': 'cur_pv1_',   'val_tpl': "{{ (value_json['pv1']['Current'] | float)}}", 'icon': 'mdi:gauge', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.PV1_POWER:    {'name': ['input', 'pv1', 'Power'],                 'level': logging.DEBUG, 'unit': 'W',    'ha': {'dev': 'input_pv1', 'dev_cla': 'power',   'stat_cla': 'measurement', 'id': 'power_pv1_', 'val_tpl': "{{ (value_json['pv1']['Power'] | float)}}"}},  # noqa: E501
+        Register.PV2_VOLTAGE:  {'name': ['input', 'pv2', 'Voltage'],               'level': logging.DEBUG, 'unit': 'V',    'ha': {'dev': 'input_pv2', 'dev_cla': 'voltage', 'stat_cla': 'measurement', 'id': 'volt_pv2_',  'val_tpl': "{{ (value_json['pv2']['Voltage'] | float)}}", 'icon': 'mdi:gauge', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.PV2_CURRENT:  {'name': ['input', 'pv2', 'Current'],               'level': logging.DEBUG, 'unit': 'A',    'ha': {'dev': 'input_pv2', 'dev_cla': 'current', 'stat_cla': 'measurement', 'id': 'cur_pv2_',   'val_tpl': "{{ (value_json['pv2']['Current'] | float)}}", 'icon': 'mdi:gauge', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.PV2_POWER:    {'name': ['input', 'pv2', 'Power'],                 'level': logging.DEBUG, 'unit': 'W',    'ha': {'dev': 'input_pv2', 'dev_cla': 'power',   'stat_cla': 'measurement', 'id': 'power_pv2_', 'val_tpl': "{{ (value_json['pv2']['Power'] | float)}}"}},  # noqa: E501
+        Register.PV3_VOLTAGE:  {'name': ['input', 'pv3', 'Voltage'],               'level': logging.DEBUG, 'unit': 'V',    'ha': {'dev': 'input_pv3', 'dev_cla': 'voltage', 'stat_cla': 'measurement', 'id': 'volt_pv3_',  'val_tpl': "{{ (value_json['pv3']['Voltage'] | float)}}", 'icon': 'mdi:gauge', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.PV3_CURRENT:  {'name': ['input', 'pv3', 'Current'],               'level': logging.DEBUG, 'unit': 'A',    'ha': {'dev': 'input_pv3', 'dev_cla': 'current', 'stat_cla': 'measurement', 'id': 'cur_pv3_',   'val_tpl': "{{ (value_json['pv3']['Current'] | float)}}", 'icon': 'mdi:gauge', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.PV3_POWER:    {'name': ['input', 'pv3', 'Power'],                 'level': logging.DEBUG, 'unit': 'W',    'ha': {'dev': 'input_pv3', 'dev_cla': 'power',   'stat_cla': 'measurement', 'id': 'power_pv3_', 'val_tpl': "{{ (value_json['pv3']['Power'] | float)}}"}},  # noqa: E501
+        Register.PV4_VOLTAGE:  {'name': ['input', 'pv4', 'Voltage'],               'level': logging.DEBUG, 'unit': 'V',    'ha': {'dev': 'input_pv4', 'dev_cla': 'voltage', 'stat_cla': 'measurement', 'id': 'volt_pv4_',  'val_tpl': "{{ (value_json['pv4']['Voltage'] | float)}}", 'icon': 'mdi:gauge', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.PV4_CURRENT:  {'name': ['input', 'pv4', 'Current'],               'level': logging.DEBUG, 'unit': 'A',    'ha': {'dev': 'input_pv4', 'dev_cla': 'current', 'stat_cla': 'measurement', 'id': 'cur_pv4_',   'val_tpl': "{{ (value_json['pv4']['Current'] | float)}}", 'icon': 'mdi:gauge', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.PV4_POWER:    {'name': ['input', 'pv4', 'Power'],                 'level': logging.DEBUG, 'unit': 'W',    'ha': {'dev': 'input_pv4', 'dev_cla': 'power',   'stat_cla': 'measurement', 'id': 'power_pv4_', 'val_tpl': "{{ (value_json['pv4']['Power'] | float)}}"}},  # noqa: E501
+        Register.PV5_VOLTAGE:  {'name': ['input', 'pv5', 'Voltage'],               'level': logging.DEBUG, 'unit': 'V',    'ha': {'dev': 'input_pv5', 'dev_cla': 'voltage', 'stat_cla': 'measurement', 'id': 'volt_pv5_',  'val_tpl': "{{ (value_json['pv5']['Voltage'] | float)}}", 'icon': 'mdi:gauge', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.PV5_CURRENT:  {'name': ['input', 'pv5', 'Current'],               'level': logging.DEBUG, 'unit': 'A',    'ha': {'dev': 'input_pv5', 'dev_cla': 'current', 'stat_cla': 'measurement', 'id': 'cur_pv5_',   'val_tpl': "{{ (value_json['pv5']['Current'] | float)}}", 'icon': 'mdi:gauge', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.PV5_POWER:    {'name': ['input', 'pv5', 'Power'],                 'level': logging.DEBUG, 'unit': 'W',    'ha': {'dev': 'input_pv5', 'dev_cla': 'power',   'stat_cla': 'measurement', 'id': 'power_pv5_', 'val_tpl': "{{ (value_json['pv5']['Power'] | float)}}"}},  # noqa: E501
+        Register.PV6_VOLTAGE:  {'name': ['input', 'pv6', 'Voltage'],               'level': logging.DEBUG, 'unit': 'V',    'ha': {'dev': 'input_pv6', 'dev_cla': 'voltage', 'stat_cla': 'measurement', 'id': 'volt_pv6_',  'val_tpl': "{{ (value_json['pv6']['Voltage'] | float)}}", 'icon': 'mdi:gauge', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.PV6_CURRENT:  {'name': ['input', 'pv6', 'Current'],               'level': logging.DEBUG, 'unit': 'A',    'ha': {'dev': 'input_pv6', 'dev_cla': 'current', 'stat_cla': 'measurement', 'id': 'cur_pv6_',   'val_tpl': "{{ (value_json['pv6']['Current'] | float)}}", 'icon': 'mdi:gauge', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.PV6_POWER:    {'name': ['input', 'pv6', 'Power'],                 'level': logging.DEBUG, 'unit': 'W',    'ha': {'dev': 'input_pv6', 'dev_cla': 'power',   'stat_cla': 'measurement', 'id': 'power_pv6_', 'val_tpl': "{{ (value_json['pv6']['Power'] | float)}}"}},  # noqa: E501
+        Register.PV1_DAILY_GENERATION:  {'name': ['input', 'pv1', 'Daily_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha': {'dev': 'input_pv1', 'dev_cla': 'energy', 'stat_cla': 'total_increasing', 'id': 'daily_gen_pv1_', 'name': 'Daily Generation', 'val_tpl': "{{ (value_json['pv1']['Daily_Generation'] | float)}}", 'icon': 'mdi:solar-power-variant', 'must_incr': True}},  # noqa: E501
+        Register.PV1_TOTAL_GENERATION:  {'name': ['input', 'pv1', 'Total_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha': {'dev': 'input_pv1', 'dev_cla': 'energy', 'stat_cla': 'total',            'id': 'total_gen_pv1_', 'name': 'Total Generation', 'val_tpl': "{{ (value_json['pv1']['Total_Generation'] | float)}}", 'icon': 'mdi:solar-power', 'must_incr': True}},  # noqa: E501
+        Register.PV2_DAILY_GENERATION:  {'name': ['input', 'pv2', 'Daily_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha': {'dev': 'input_pv2', 'dev_cla': 'energy', 'stat_cla': 'total_increasing', 'id': 'daily_gen_pv2_', 'name': 'Daily Generation', 'val_tpl': "{{ (value_json['pv2']['Daily_Generation'] | float)}}", 'icon': 'mdi:solar-power-variant', 'must_incr': True}},  # noqa: E501
+        Register.PV2_TOTAL_GENERATION:  {'name': ['input', 'pv2', 'Total_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha': {'dev': 'input_pv2', 'dev_cla': 'energy', 'stat_cla': 'total',            'id': 'total_gen_pv2_', 'name': 'Total Generation', 'val_tpl': "{{ (value_json['pv2']['Total_Generation'] | float)}}", 'icon': 'mdi:solar-power', 'must_incr': True}},  # noqa: E501
+        Register.PV3_DAILY_GENERATION:  {'name': ['input', 'pv3', 'Daily_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha': {'dev': 'input_pv3', 'dev_cla': 'energy', 'stat_cla': 'total_increasing', 'id': 'daily_gen_pv3_', 'name': 'Daily Generation', 'val_tpl': "{{ (value_json['pv3']['Daily_Generation'] | float)}}", 'icon': 'mdi:solar-power-variant', 'must_incr': True}},  # noqa: E501
+        Register.PV3_TOTAL_GENERATION:  {'name': ['input', 'pv3', 'Total_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha': {'dev': 'input_pv3', 'dev_cla': 'energy', 'stat_cla': 'total',            'id': 'total_gen_pv3_', 'name': 'Total Generation', 'val_tpl': "{{ (value_json['pv3']['Total_Generation'] | float)}}", 'icon': 'mdi:solar-power', 'must_incr': True}},  # noqa: E501
+        Register.PV4_DAILY_GENERATION:  {'name': ['input', 'pv4', 'Daily_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha': {'dev': 'input_pv4', 'dev_cla': 'energy', 'stat_cla': 'total_increasing', 'id': 'daily_gen_pv4_', 'name': 'Daily Generation', 'val_tpl': "{{ (value_json['pv4']['Daily_Generation'] | float)}}", 'icon': 'mdi:solar-power-variant', 'must_incr': True}},  # noqa: E501
+        Register.PV4_TOTAL_GENERATION:  {'name': ['input', 'pv4', 'Total_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha': {'dev': 'input_pv4', 'dev_cla': 'energy', 'stat_cla': 'total',            'id': 'total_gen_pv4_', 'name': 'Total Generation', 'val_tpl': "{{ (value_json['pv4']['Total_Generation'] | float)}}", 'icon': 'mdi:solar-power', 'must_incr': True}},  # noqa: E501
+        Register.PV5_DAILY_GENERATION:  {'name': ['input', 'pv5', 'Daily_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha': {'dev': 'input_pv5', 'dev_cla': 'energy', 'stat_cla': 'total_increasing', 'id': 'daily_gen_pv5_', 'name': 'Daily Generation', 'val_tpl': "{{ (value_json['pv5']['Daily_Generation'] | float)}}", 'icon': 'mdi:solar-power-variant', 'must_incr': True}},  # noqa: E501
+        Register.PV5_TOTAL_GENERATION:  {'name': ['input', 'pv5', 'Total_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha': {'dev': 'input_pv5', 'dev_cla': 'energy', 'stat_cla': 'total',            'id': 'total_gen_pv5_', 'name': 'Total Generation', 'val_tpl': "{{ (value_json['pv5']['Total_Generation'] | float)}}", 'icon': 'mdi:solar-power', 'must_incr': True}},  # noqa: E501
+        Register.PV6_DAILY_GENERATION:  {'name': ['input', 'pv6', 'Daily_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha': {'dev': 'input_pv6', 'dev_cla': 'energy', 'stat_cla': 'total_increasing', 'id': 'daily_gen_pv6_', 'name': 'Daily Generation', 'val_tpl': "{{ (value_json['pv6']['Daily_Generation'] | float)}}", 'icon': 'mdi:solar-power-variant', 'must_incr': True}},  # noqa: E501
+        Register.PV6_TOTAL_GENERATION:  {'name': ['input', 'pv6', 'Total_Generation'],        'level': logging.DEBUG, 'unit': 'kWh',  'ha': {'dev': 'input_pv6', 'dev_cla': 'energy', 'stat_cla': 'total',            'id': 'total_gen_pv6_', 'name': 'Total Generation', 'val_tpl': "{{ (value_json['pv6']['Total_Generation'] | float)}}", 'icon': 'mdi:solar-power', 'must_incr': True}},  # noqa: E501
+        # total:
+        Register.TS_TOTAL:          {'name': ['total', 'Timestamp'],               'level': logging.INFO,  'unit': ''},  # noqa: E501
+        Register.DAILY_GENERATION:  {'name': ['total', 'Daily_Generation'],        'level': logging.INFO,  'unit': 'kWh',  'ha': {'dev': 'inverter', 'dev_cla': 'energy', 'stat_cla': 'total_increasing', 'id': 'daily_gen_', 'fmt': '| float', 'name': 'Daily Generation', 'icon': 'mdi:solar-power-variant', 'must_incr': True}},  # noqa: E501
+        Register.TOTAL_GENERATION:  {'name': ['total', 'Total_Generation'],        'level': logging.INFO,  'unit': 'kWh',  'ha': {'dev': 'inverter', 'dev_cla': 'energy', 'stat_cla': 'total',            'id': 'total_gen_', 'fmt': '| float', 'name': 'Total Generation', 'icon': 'mdi:solar-power', 'must_incr': True}},  # noqa: E501
 
         # controller:
-            0x000c3500:  {'name':['controller', 'Signal_Strength'],           'level': logging.DEBUG, 'unit': '%' ,  'ha':{'dev':'controller', 'dev_cla': None,       'stat_cla': 'measurement', 'id':'signal_',         'fmt':'| int', 'name': 'Signal Strength', 'icon':'mdi:wifi'}},
-            0x000c96a8:  {'name':['controller', 'Power_On_Time'],             'level': logging.DEBUG, 'unit': 's',   'ha':{'dev':'controller', 'dev_cla': 'duration', 'stat_cla': 'measurement', 'id':'power_on_time_',                 'name': 'Power on Time',   'val_tpl':"{{ (value_json['Power_On_Time'] | float)}}", 'nat_prc':'3','ent_cat':'diagnostic'}},
-            0x000cf850:  {'name':['controller', 'Data_Up_Interval'],          'level': logging.DEBUG, 'unit': 's',   'ha':{'dev':'controller', 'dev_cla': None,       'stat_cla': 'measurement', 'id':'data_up_intval_', 'fmt':'| int', 'name': 'Data Up Interval', 'icon':'mdi:update','ent_cat':'diagnostic'}},
+        Register.SIGNAL_STRENGTH:    {'name': ['controller', 'Signal_Strength'],    'level': logging.DEBUG, 'unit': '%',    'ha': {'dev': 'controller', 'dev_cla': None,       'stat_cla': 'measurement', 'id': 'signal_',              'fmt': '| int',           'name': 'Signal Strength', 'icon': 'mdi:wifi'}},  # noqa: E501
+        Register.POWER_ON_TIME:      {'name': ['controller', 'Power_On_Time'],      'level': logging.DEBUG, 'unit': 's',    'ha': {'dev': 'controller', 'dev_cla': 'duration', 'stat_cla': 'measurement', 'id': 'power_on_time_',       'fmt': '| int',           'name': 'Power on Time', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.COLLECT_INTERVAL:   {'name': ['controller', 'Collect_Interval'],   'level': logging.DEBUG, 'unit': 'min',  'ha': {'dev': 'controller', 'dev_cla': None,       'stat_cla': None,          'id': 'data_collect_intval_', 'fmt': '| string + " min"', 'name': 'Data Collect Interval', 'icon': 'mdi:update', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.CONNECT_COUNT:      {'name': ['controller', 'Connect_Count'],      'level': logging.DEBUG, 'unit': '',     'ha': {'dev': 'controller', 'dev_cla': None,       'stat_cla': None,          'id': 'connect_count_',       'fmt': '| int',           'name': 'Connect Count',    'icon': 'mdi:counter', 'comp': 'sensor', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.COMMUNICATION_TYPE: {'name': ['controller', 'Communication_Type'], 'level': logging.DEBUG, 'unit': '',     'ha': {'dev': 'controller', 'dev_cla': None,       'stat_cla': None,          'id': 'comm_type_',           'name': 'Communication Type', 'val_tpl': __comm_type_val_tpl, 'comp': 'sensor', 'icon': 'mdi:wifi'}},  # noqa: E501
+        Register.DATA_UP_INTERVAL:   {'name': ['controller', 'Data_Up_Interval'],   'level': logging.DEBUG, 'unit': 's',    'ha': {'dev': 'controller', 'dev_cla': None,       'stat_cla': None,          'id': 'data_up_intval_', 'fmt': '| string + " s"', 'name': 'Data Up Interval', 'icon': 'mdi:update', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.HEARTBEAT_INTERVAL: {'name': ['controller', 'Heartbeat_Interval'], 'level': logging.DEBUG, 'unit': 's',    'ha': {'dev': 'controller', 'dev_cla': None,       'stat_cla': None,          'id': 'heartbeat_intval_',    'fmt': '| string + " s"', 'name': 'Heartbeat Interval', 'icon': 'mdi:update', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.IP_ADDRESS:         {'name': ['controller', 'IP_Address'],         'level': logging.DEBUG, 'unit': '',     'ha': {'dev': 'controller', 'dev_cla': None,       'stat_cla': None,          'id': 'ip_address_',           'fmt': '| string',        'name': 'IP Address', 'icon': 'mdi:wifi', 'ent_cat': 'diagnostic'}},  # noqa: E501
+        Register.POLLING_INTERVAL:   {'name': ['controller', 'Polling_Interval'],   'level': logging.DEBUG, 'unit': 's',    'ha': {'dev': 'controller', 'dev_cla': None,       'stat_cla': None,          'id': 'polling_intval_', 'fmt': '| string + " s"', 'name': 'Polling Interval', 'icon': 'mdi:update', 'ent_cat': 'diagnostic'}},  # noqa: E501
+    }
 
-    }          
-                                    
-    def dev_value(self, idx:str|int) -> str|int|float|None:
+    @property
+    def info_devs(self) -> dict:
+        return self.__info_devs
+
+    @property
+    def info_defs(self) -> dict:
+        return self.__info_defs
+
+    def dev_value(self, idx: str | int) -> str | int | float | None:
         '''returns the stored device value from our database
 
-        idx:int ==> lookup the value in the database and return it as str, int or flout. If the value is not available return 'None'
-        idx:str ==> returns the string as a fixed value without a database loopup  
+        idx:int ==> lookup the value in the database and return it as str,
+                    int or float. If the value is not available return 'None'
+        idx:str ==> returns the string as a fixed value without a
+                    database lookup
         '''
-        if type (idx) is str:
-            return idx               # return idx as a fixed value 
-        elif idx in self.__info_defs:
-            row = self.__info_defs[idx]
+        if type(idx) is str:
+            return idx               # return idx as a fixed value
+        elif idx in self.info_defs:
+            row = self.info_defs[idx]
             if 'singleton' in row and row['singleton']:
                 dict = self.stat
             else:
                 dict = self.db
-            
+
             keys = row['name']
 
             for key in keys:
                 if key not in dict:
                     return None      # value not found in the database
                 dict = dict[key]
-            return dict              # value of the reqeusted entry   
-            
-        return None                  # unknwon idx, not in __info_defs
+            return dict              # value of the reqeusted entry
 
-    def ignore_this_device(self, dep:dict) -> bool:
-        '''Checks the equation in the dep dict
-        
-            returns 'False' only if the equation is valid; 'True'  in any other case'''
-        if 'reg' in dep:
-            value = self.dev_value(dep['reg'])
-            if not value: return True
-            
-            if 'gte' in dep:
-                return not value >= dep['gte']
-            elif 'less_eq' in dep:
-                return not value <= dep['less_eq']
-        return True
-    
-    def ha_confs(self, ha_prfx, inv_node_id, inv_snr, proxy_node_id, proxy_unique_id, sug_area =''):
-        '''Generator function yields a json register struct for home-assistant auto configuration and a unique entity string
+        return None                  # unknwon idx, not in info_defs
 
-        arguments:
-        prfx:str     ==> MQTT prefix for the home assistant 'stat_t string
-        snr:str      ==> serial number of the inverter, used to build unique entity strings
-        sug_area:str ==> suggested area string from the config file'''
-        tab = self.__info_defs
-        for key in tab:
-            row = tab[key]
-            if 'singleton' in row and row['singleton']:
-                node_id = proxy_node_id
-                snr = proxy_unique_id
-            else:
-                node_id = inv_node_id
-                snr = inv_snr
-            prfx = ha_prfx + node_id
-
-            #check if we have details for home assistant
-            if 'ha' in row:
-                ha = row['ha']
-                if 'comp' in ha:
-                    component = ha['comp']
-                else:
-                    component = 'sensor'
-                attr = {}                                     # dict to collect all the sensor entity details 
-                if 'name' in ha:
-                    attr['name']   = ha['name']               # take the entity name from the ha dict
-                else:                 
-                    attr['name']   = row['name'][-1]          # otherwise take a name from the name array 
-
-                attr['stat_t'] = prfx +row['name'][0]         # eg.  'stat_t': "tsun/garagendach/grid"
-                attr['dev_cla']  = ha['dev_cla']              # eg.  'dev_cla': 'power'
-                attr['stat_cla'] = ha['stat_cla']             # eg.  'stat_cla': "measurement"
-                attr['uniq_id']  = ha['id']+snr               # build the 'uniq_id' from the id str + the serial no of the inverter
-                if 'val_tpl' in ha:
-                    attr['val_tpl']  = ha['val_tpl']          # get value template for complexe data structures
-                elif 'fmt' in ha:
-                    attr['val_tpl']  = '{{value_json' + f"['{row['name'][-1]}'] {ha['fmt']}" + '}}'       # eg.   'val_tpl': "{{ value_json['Output_Power']|float }}"
-                else:
-                    self.inc_counter('Internal_Error')
-                    logging.error(f"Infos.__info_defs: the row for {key} do not have a 'val_tpl' nor a 'fmt' value")
-
-                # add unit_of_meas only, if status_class isn't none. If status_cla is None we want a number format and not line graph in home assistant. 
-                # A unit will change the number format to a line graph
-                if 'unit' in row and attr['stat_cla'] != None:
-                    attr['unit_of_meas'] = row['unit']        # optional add a 'unit_of_meas' e.g. 'W'
-                if 'icon' in ha:
-                    attr['ic'] = ha['icon']                   # optional add an icon for the entity
-                if 'nat_prc' in ha:
-                    attr['sug_dsp_prc'] = ha['nat_prc']       # optional add the precison of floats
-                if 'ent_cat' in ha:
-                    attr['ent_cat'] = ha['ent_cat']           # diagnostic, config
-               
-                # enabled_by_default is deactivated, since it avoid the via setup of the devices
-                # it seems, that there is a bug in home assistant. tested with 'Home Assistant 2023.10.4'
-                #if 'en' in ha:                                # enabled_by_default
-                #    attr['en'] = ha['en']
-
-                # eg. 'dev':{'name':'Microinverter','mdl':'MS-600','ids':["inverter_123"],'mf':'TSUN','sa': 'auf Garagendach'}
-                # attr['dev'] = {'name':'Microinverter','mdl':'MS-600','ids':[f'inverter_{snr}'],'mf':'TSUN','sa': 'auf Garagendach'}   
-                if 'dev' in ha:
-                    device = self.__info_devs[ha['dev']]
-
-                    if 'dep' in device and self.ignore_this_device(device['dep']):
-                        continue
-
-                    dev = {}
-
-                    # the same name for 'name' and 'suggested area', so we get dedicated devices in home assistant with short value name and headline 
-                    if sug_area == '' or ('singleton' in device and device['singleton']):
-                        dev['name'] = device['name']
-                        dev['sa']   = device['name']
-                    else:
-                        dev['name'] = device['name']+' - '+sug_area
-                        dev['sa']   = device['name']+' - '+sug_area
-                        
-                    if 'via' in device:                       # add the link to the parent device
-                        via = device['via']
-                        if via in self.__info_devs:
-                            via_dev = self.__info_devs[via]
-                            if 'singleton' in via_dev and via_dev['singleton']:
-                                dev['via_device'] = via
-                            else:
-                                dev['via_device'] = f"{via}_{snr}"
-                        else:
-                            self.inc_counter('Internal_Error')
-                            logging.error(f"Infos.__info_defs: the row for {key} has an invalid via value: {via}")
-
-                   
-
-                    for key in ('mdl','mf', 'sw', 'hw'):      # add optional values fpr 'modell', 'manufaturer', 'sw version' and 'hw version'
-                        if key in device:
-                            data = self.dev_value(device[key])
-                            if data is not None: dev[key]  = data
-                    
-                    if 'singleton' in device and device['singleton']:
-                        dev['ids']  = [f"{ha['dev']}"]
-                    else:
-                        dev['ids']  = [f"{ha['dev']}_{snr}"]
-
-                    attr['dev'] = dev
-
-                    origin = {}
-                    origin['name'] = self.app_name
-                    origin['sw'] = self.version
-                    attr['o'] = origin
-                else:
-                    self.inc_counter('Internal_Error')
-                    logging.error(f"Infos.__info_defs: the row for {key} missing 'dev' value for ha register")
-
-
-                yield json.dumps (attr), component, node_id, attr['uniq_id']
-
-    def inc_counter (self, counter:str) -> None:
+    def inc_counter(self, counter: str) -> None:
         '''inc proxy statistic counter'''
         dict = self.stat['proxy']
-        dict[counter] += 1                                
- 
-    def dec_counter (self, counter:str) -> None:
+        dict[counter] += 1
+
+    def dec_counter(self, counter: str) -> None:
         '''dec proxy statistic counter'''
         dict = self.stat['proxy']
-        dict[counter] -= 1                                
-                                  
-    
-    def __key_obj(self, id) -> list:
-        d = self.__info_defs.get(id, {'name': None, 'level': logging.DEBUG, 'unit': ''})
+        dict[counter] -= 1
+
+    def ha_proxy_confs(self, ha_prfx: str, node_id: str, snr: str) \
+            -> Generator[tuple[str, str, str, str], None, None]:
+        '''Generator function yields json register struct for home-assistant
+        auto configuration and the unique entity string, for all proxy
+        registers
+
+        arguments:
+        ha_prfx:str  ==> MQTT prefix for the home assistant 'stat_t string
+        node_id:str  ==> node id of the inverter, used to build unique entity
+        snr:str      ==> serial number of the inverter, used to build unique
+                         entity strings
+        '''
+        # iterate over RegisterMap.map and get the register values for entries
+        # with Singleton=True, which means that this is a proxy register
+        for reg in self.info_defs.keys():
+            res = self.ha_conf(reg, ha_prfx, node_id, snr, True)  # noqa: E501
+            if res:
+                yield res
+
+    def ha_conf(self, key, ha_prfx, node_id, snr,  singleton: bool,
+                sug_area: str = '') -> tuple[str, str, str, str] | None:
+        '''Method to build json register struct for home-assistant
+        auto configuration and the unique entity string, for all proxy
+        registers
+
+        arguments:
+        key          ==> index of info_defs dict which reference the topic
+        ha_prfx:str  ==> MQTT prefix for the home assistant 'stat_t string
+        node_id:str  ==> node id of the inverter, used to build unique entity
+        snr:str      ==> serial number of the inverter, used to build unique
+                         entity strings
+        singleton    ==> bool to allow/disaalow proxy topics which are common
+                         for all invters
+        sug_area     ==> area name for home assistant
+        '''
+        if key not in self.info_defs:
+            return None
+        row = self.info_defs[key]
+
+        if 'singleton' in row:
+            if singleton != row['singleton']:
+                return None
+        elif singleton:
+            return None
+        prfx = ha_prfx + node_id
+
+        # check if we have details for home assistant
+        if 'ha' in row:
+            ha = row['ha']
+            if 'comp' in ha:
+                component = ha['comp']
+            else:
+                component = 'sensor'
+            attr = {}
+            if 'name' in ha:
+                attr['name'] = ha['name']
+            else:
+                attr['name'] = row['name'][-1]
+            attr['stat_t'] = prfx + row['name'][0]
+            attr['dev_cla'] = ha['dev_cla']
+            attr['stat_cla'] = ha['stat_cla']
+            attr['uniq_id'] = ha['id']+snr
+            if 'val_tpl' in ha:
+                attr['val_tpl'] = ha['val_tpl']
+            elif 'fmt' in ha:
+                attr['val_tpl'] = '{{value_json' + f"['{row['name'][-1]}'] {ha['fmt']}" + '}}'       # eg.   'val_tpl': "{{ value_json['Output_Power']|float }} # noqa: E501
+            else:
+                self.inc_counter('Internal_Error')
+                logging.error(f"Infos.info_defs: the row for {key} do"
+                              " not have a 'val_tpl' nor a 'fmt' value")
+            # add unit_of_meas only, if status_class isn't none. If
+            # status_cla is None we want a number format and not line
+            # graph in home assistant. A unit will change the number
+            # format to a line graph
+            if 'unit' in row and attr['stat_cla'] is not None:
+                attr['unit_of_meas'] = row['unit']  # 'unit_of_meas'
+            if 'icon' in ha:
+                attr['ic'] = ha['icon']             # icon for the entity
+            if 'nat_prc' in ha:  # pragma: no cover
+                attr['sug_dsp_prc'] = ha['nat_prc']  # precison of floats
+            if 'ent_cat' in ha:
+                attr['ent_cat'] = ha['ent_cat']     # diagnostic, config
+            # enabled_by_default is deactivated, since it avoid the via
+            # setup of the devices. It seems, that there is a bug in home
+            # assistant. tested with 'Home Assistant 2023.10.4'
+            # if 'en' in ha:                       # enabled_by_default
+            #    attr['en'] = ha['en']
+            if 'dev' in ha:
+                device = self.info_devs[ha['dev']]
+                if 'dep' in device and self.ignore_this_device(device['dep']):  # noqa: E501
+                    return None
+                dev = {}
+                # the same name for 'name' and 'suggested area', so we get
+                # dedicated devices in home assistant with short value
+                # name and headline
+                if (sug_area == '' or
+                        ('singleton' in device and device['singleton'])):
+                    dev['name'] = device['name']
+                    dev['sa'] = device['name']
+                else:
+                    dev['name'] = device['name']+' - '+sug_area
+                    dev['sa'] = device['name']+' - '+sug_area
+                if 'via' in device:  # add the link to the parent device
+                    via = device['via']
+                    if via in self.info_devs:
+                        via_dev = self.info_devs[via]
+                        if 'singleton' in via_dev and via_dev['singleton']:
+                            dev['via_device'] = via
+                        else:
+                            dev['via_device'] = f"{via}_{snr}"
+                    else:
+                        self.inc_counter('Internal_Error')
+                        logging.error(f"Infos.info_defs: the row for "
+                                      f"{key} has an invalid via value: "
+                                      f"{via}")
+                for key in ('mdl', 'mf', 'sw', 'hw'):      # add optional
+                    # values fpr 'modell', 'manufacturer', 'sw version' and
+                    # 'hw version'
+                    if key in device:
+                        data = self.dev_value(device[key])
+                        if data is not None:
+                            dev[key] = data
+                if 'singleton' in device and device['singleton']:
+                    dev['ids'] = [f"{ha['dev']}"]
+                else:
+                    dev['ids'] = [f"{ha['dev']}_{snr}"]
+                attr['dev'] = dev
+                origin = {}
+                origin['name'] = self.app_name
+                origin['sw'] = self.version
+                attr['o'] = origin
+            else:
+                self.inc_counter('Internal_Error')
+                logging.error(f"Infos.info_defs: the row for {key} "
+                              "missing 'dev' value for ha register")
+            return json.dumps(attr), component, node_id, attr['uniq_id']
+        return None
+
+    def ha_remove(self, key, node_id, snr) -> tuple[str, str, str, str] | None:
+        '''Method to build json unregister struct for home-assistant
+        to remove topics per auto configuration. Only for inverer topics.
+
+        arguments:
+        key          ==> index of info_defs dict which reference the topic
+        node_id:str  ==> node id of the inverter, used to build unique entity
+        snr:str      ==> serial number of the inverter, used to build unique
+                         entity strings
+
+        hint:
+        the returned tuple must have the same format as self.ha_conf()
+        '''
+        if key not in self.info_defs:
+            return None
+        row = self.info_defs[key]
+
+        if 'singleton' in row:
+            if row['singleton']:
+                return None
+
+        # check if we have details for home assistant
+        if 'ha' in row:
+            ha = row['ha']
+            if 'comp' in ha:
+                component = ha['comp']
+            else:
+                component = 'sensor'
+            attr = {}
+            uniq_id = ha['id']+snr
+
+            return json.dumps(attr), component, node_id, uniq_id
+        return None
+
+    def _key_obj(self, id: Register) -> list:
+        d = self.info_defs.get(id, {'name': None, 'level': logging.DEBUG,
+                                    'unit': ''})
         if 'ha' in d and 'must_incr' in d['ha']:
             must_incr = d['ha']['must_incr']
         else:
             must_incr = False
 
         return d['name'], d['level'], d['unit'], must_incr
-    
-    
-    def parse(self, buf) -> None:
-        '''parse a data sequence received from the inverter and stores the values in Infos.db
-        
-        buf: buffer of the sequence to parse'''
-        result = struct.unpack_from('!l', buf, 0)
-        elms = result[0]
-        i = 0
-        ind = 4
-        while i < elms:
-            result = struct.unpack_from('!lB', buf, ind)
-            info_id   = result[0]             
-            data_type = result[1]
-            ind += 5
-            keys, level, unit, must_incr = self.__key_obj(info_id)
-            
-            if data_type==0x54:   # 'T' -> Pascal-String
-                str_len = buf[ind]
-                result = struct.unpack_from(f'!{str_len+1}p', buf, ind)[0].decode(encoding='ascii', errors='replace')
-                ind += str_len+1
-            
-            elif data_type==0x49: # 'I' -> int32
-                result = struct.unpack_from(f'!l', buf, ind)[0]
-                ind += 4
 
-            elif data_type==0x53: # 'S' -> short
-                result = struct.unpack_from(f'!h', buf, ind)[0]
-                ind += 2
-
-            elif data_type==0x46: # 'F' -> float32
-                result = round(struct.unpack_from(f'!f', buf, ind)[0],2)
-                ind += 4
+    def update_db(self, keys: list, must_incr: bool, result):
+        name = ''
+        dict = self.db
+        for key in keys[:-1]:
+            if key not in dict:
+                dict[key] = {}
+            dict = dict[key]
+            name += key + '.'
+        if keys[-1] not in dict:
+            update = (not must_incr or result > 0)
+        else:
+            if must_incr:
+                update = dict[keys[-1]] < result
             else:
-                self.inc_counter('Invalid_Data_Type')
-                logging.error(f"Infos.parse: data_type: {data_type} not supported")
-                return
-    
+                update = dict[keys[-1]] != result
+        if update:
+            dict[keys[-1]] = result
+        name += keys[-1]
+        return name, update
 
+    def set_db_def_value(self, id: Register, value) -> None:
+        '''set default value'''
+        row = self.info_defs[id]
+        if isinstance(row, dict):
+            keys = row['name']
+            self.update_db(keys, False, value)
 
-            if keys:
-                dict = self.db
-                name = ''
-            
-                for key in keys[:-1]:
-                    if key not in dict:
-                        dict[key] = {}
-                    dict = dict[key]
-                    name += key + '.'
+    def reg_clr_at_midnight(self, prfx: str,
+                            check_dependencies: bool = True) -> None:
+        '''register all registers for the 'ClrAtMidnight' class and
+        check if device of every register is available otherwise ignore
+        the register.
 
-                update = keys[-1] not in dict or (not must_incr and dict[keys[-1]] != result) or (must_incr and dict[keys[-1]] < result)
-                if update: dict[keys[-1]] = result
-                name += keys[-1]
-                yield keys[0], update    
-            else:
-                update = False    
-                name = str(f'info-id.0x{info_id:x}')
-            
-            self.tracer.log(level, f'{name} : {result}{unit}')
- 
-            i +=1
+        prfx:str ==> prefix for the home assistant 'stat_t string''
+        '''
+        for id, row in self.info_defs.items():
+            if check_dependencies and 'ha' in row:
+                ha = row['ha']
+                if 'dev' in ha:
+                    device = self.info_devs[ha['dev']]
+                    if 'dep' in device and self.ignore_this_device(device['dep']):  # noqa: E501
+                        continue
 
-   
+            keys = row['name']
+            ClrAtMidnight.add(keys, prfx, id)
+
+    def get_db_value(self, id: Register, not_found_result: any = None):
+        '''get database value'''
+        row = self.info_defs[id]
+        if isinstance(row, dict):
+            keys = row['name']
+            elm = self.db
+            for key in keys:
+                if key not in elm:
+                    return not_found_result
+                elm = elm[key]
+            return elm
+        return not_found_result
+
+    def ignore_this_device(self, dep: dict) -> bool:
+        '''Checks the equation in the dep(endency) dict
+
+            returns 'False' only if the equation is valid;
+                    'True'  in any other case'''
+        if 'reg' in dep:
+            value = self.dev_value(dep['reg'])
+            if not value:
+                return True
+
+            if 'gte' in dep:
+                return not value >= dep['gte']
+            elif 'less_eq' in dep:
+                return not value <= dep['less_eq']
+        return True
+
+    def set_pv_module_details(self, inv: dict) -> None:
+        map = {'pv1': {'manufacturer': Register.PV1_MANUFACTURER, 'model': Register.PV1_MODEL},  # noqa: E501
+               'pv2': {'manufacturer': Register.PV2_MANUFACTURER, 'model': Register.PV2_MODEL},  # noqa: E501
+               'pv3': {'manufacturer': Register.PV3_MANUFACTURER, 'model': Register.PV3_MODEL},  # noqa: E501
+               'pv4': {'manufacturer': Register.PV4_MANUFACTURER, 'model': Register.PV4_MODEL},  # noqa: E501
+               'pv5': {'manufacturer': Register.PV5_MANUFACTURER, 'model': Register.PV5_MODEL},  # noqa: E501
+               'pv6': {'manufacturer': Register.PV6_MANUFACTURER, 'model': Register.PV6_MODEL}  # noqa: E501
+               }
+
+        for key, reg in map.items():
+            if key in inv:
+                if 'manufacturer' in inv[key]:
+                    self.set_db_def_value(reg['manufacturer'],
+                                          inv[key]['manufacturer'])
+                if 'type' in inv[key]:
+                    self.set_db_def_value(reg['model'], inv[key]['type'])

app/src/inverter.py

@@ -1,121 +1,79 @@
-import asyncio, logging, traceback, json
+import asyncio
+import logging
+import json
 from config import Config
-from async_stream import AsyncStream
 from mqtt import Mqtt
-#import gc
+from infos import Infos
 
-#logger = logging.getLogger('conn')
+# logger = logging.getLogger('conn')
 logger_mqtt = logging.getLogger('mqtt')
 
 
+class Inverter():
+    @classmethod
+    def class_init(cls) -> None:
+        logging.debug('Inverter.class_init')
+        # initialize the proxy statistics
+        Infos.static_init()
+        cls.db_stat = Infos()
 
-class Inverter(AsyncStream):
-
-    def __init__ (self, reader, writer, addr):
-        super().__init__(reader, writer, addr, None, True)
-        self.mqtt = Mqtt()
-        self.ha_restarts = -1
         ha = Config.get('ha')
-        self.entity_prfx = ha['entity_prefix'] + '/'
-        self.discovery_prfx = ha['discovery_prefix'] + '/'
-        self.proxy_node_id  = ha['proxy_node_id'] + '/'
-        self.proxy_unique_id = ha['proxy_unique_id']
+        cls.entity_prfx = ha['entity_prefix'] + '/'
+        cls.discovery_prfx = ha['discovery_prefix'] + '/'
+        cls.proxy_node_id = ha['proxy_node_id'] + '/'
+        cls.proxy_unique_id = ha['proxy_unique_id']
 
+        # call Mqtt singleton to establisch the connection to the mqtt broker
+        cls.mqtt = Mqtt(cls._cb_mqtt_is_up)
 
-    async def server_loop(self, addr):
-        '''Loop for receiving messages from the inverter (server-side)'''
-        logging.info(f'Accept connection from  {addr}')        
-        self.inc_counter ('Inverter_Cnt')
-        await self.loop()
-        self.dec_counter ('Inverter_Cnt')
-        logging.info(f'Server loop stopped for {addr}')
-        
-        # if the server connection closes, we also have to disconnect the connection to te TSUN cloud
-        if self.remoteStream:
-            logging.debug ("disconnect client connection")
-            self.remoteStream.disc()
+        # register all counters which should be reset at midnight.
+        # This is needed if the proxy is restated before midnight
+        # and the inverters are offline, cause the normal refgistering
+        # needs an update on the counters.
+        # Without this registration here the counters would not be
+        # reset at midnight when you restart the proxy just before
+        # midnight!
+        inverters = Config.get('inverters')
+        # logger.debug(f'Inverters: {inverters}')
+        for inv in inverters.values():
+            if (type(inv) is dict):
+                node_id = inv['node_id']
+                cls.db_stat.reg_clr_at_midnight(f'{cls.entity_prfx}{node_id}',
+                                                check_dependencies=False)
 
-        await self.__async_publ_mqtt_packet('proxy')
-        
-    async def client_loop(self, addr):
-        '''Loop for receiving messages from the TSUN cloud (client-side)'''
-        await self.remoteStream.loop()    
-        logging.info(f'Client loop stopped for {addr}')
+    @classmethod
+    async def _cb_mqtt_is_up(cls) -> None:
+        logging.info('Initialize proxy device on home assistant')
+        # register proxy status counters at home assistant
+        await cls._register_proxy_stat_home_assistant()
 
-        # if the client connection closes, we don't touch the server connection. Instead we erase the client
-        # connection stream, thus on the next received packet from the inverter, we can establish a new connection 
-        # to the TSUN cloud
-        self.remoteStream.remoteStream = None    # erase backlink to inverter instance
-        self.remoteStream = None                 # than erase client connection 
-        
-    async def async_create_remote(self) -> None:
-        '''Establish a client connection to the TSUN cloud'''
-        tsun = Config.get('tsun')
-        host = tsun['host']
-        port = tsun['port']     
-        addr = (host, port)
-            
-        try:
-            logging.info(f'Connected to {addr}')
-            connect = asyncio.open_connection(host, port)
-            reader, writer = await connect    
-            self.remoteStream = AsyncStream(reader, writer, addr, self, False)
-            asyncio.create_task(self.client_loop(addr))
-            
-        except ConnectionRefusedError as error:
-            logging.info(f'{error}')
-        except Exception:
-            logging.error(
-                f"Inverter: Exception for {addr}:\n"
-                f"{traceback.format_exc()}")
-        
-    
+        # send values of the proxy status counters
+        await asyncio.sleep(0.5)            # wait a bit, before sending data
+        Infos.new_stat_data['proxy'] = True   # force sending data to sync ha
+        await cls._async_publ_mqtt_proxy_stat('proxy')
 
-    async def async_publ_mqtt(self) -> None:
-        '''puplish data to MQTT broker'''
-        # check if new inverter or collector infos are available or when the home assistant has changed the status back to online
-        if (('inverter' in self.new_data and self.new_data['inverter']) or 
-             ('collector' in self.new_data and self.new_data['collector']) or
-               self.mqtt.ha_restarts != self.ha_restarts):
-            await self.__register_home_assistant()
-            self.ha_restarts = self.mqtt.ha_restarts
-
-        for key in self.new_data:
-            await self.__async_publ_mqtt_packet(key)
-
-    async def __async_publ_mqtt_packet(self, key):
-        db = self.db.db
-        stat = self.db.stat
-        if self.new_data[key]:
-            if key in db:
-                data_json = json.dumps(db[key])
-                node_id = self.node_id
-            elif key in stat:
-                data_json = json.dumps(stat[key])
-                node_id = self.proxy_node_id
-            else:
-                return     
-            logger_mqtt.debug(f'{key}: {data_json}')
-            await self.mqtt.publish(f"{self.entity_prfx}{node_id}{key}", data_json)
-            self.new_data[key] = False
-
-    async def __register_home_assistant(self) -> None:
+    @classmethod
+    async def _register_proxy_stat_home_assistant(cls) -> None:
         '''register all our topics at home assistant'''
-        try:
-            for data_json, component, node_id, id in self.db.ha_confs(self.entity_prfx, self.node_id, self.unique_id, self.proxy_node_id, self.proxy_unique_id, self.sug_area):
-                    logger_mqtt.debug(f"MQTT Register: cmp:'{component}' node_id:'{node_id}' {data_json}")                                
-                    await self.mqtt.publish(f"{self.discovery_prfx}{component}/{node_id}{id}/config", data_json)
-        except Exception:
-            logging.error(
-                f"Inverter: Exception:\n"
-                f"{traceback.format_exc()}")
-            
-    def close(self) -> None:
-        logging.debug(f'Inverter.close() {self.addr}')
-        super().close()         # call close handler in the parent class
-#        logger.debug (f'Inverter refs: {gc.get_referrers(self)}')
+        for data_json, component, node_id, id in cls.db_stat.ha_proxy_confs(
+                 cls.entity_prfx, cls.proxy_node_id, cls.proxy_unique_id):
+            logger_mqtt.debug(f"MQTT Register: cmp:'{component}' node_id:'{node_id}' {data_json}")      # noqa: E501
+            await cls.mqtt.publish(f'{cls.discovery_prfx}{component}/{node_id}{id}/config', data_json)  # noqa: E501
 
+    @classmethod
+    async def _async_publ_mqtt_proxy_stat(cls, key) -> None:
+        stat = Infos.stat
+        if key in stat and Infos.new_stat_data[key]:
+            data_json = json.dumps(stat[key])
+            node_id = cls.proxy_node_id
+            logger_mqtt.debug(f'{key}: {data_json}')
+            await cls.mqtt.publish(f"{cls.entity_prfx}{node_id}{key}",
+                                   data_json)
+            Infos.new_stat_data[key] = False
 
-    def __del__ (self):
-        logging.debug ("Inverter.__del__")
-        super().__del__()  
+    @classmethod
+    def class_close(cls, loop) -> None:
+        logging.debug('Inverter.class_close')
+        logging.info('Close MQTT Task')
+        loop.run_until_complete(cls.mqtt.close())
+        cls.mqtt = None

app/src/logging.ini

@@ -1,5 +1,5 @@
 [loggers]
-keys=root,tracer,mesg,conn,data,mqtt
+keys=root,tracer,mesg,conn,data,mqtt,asyncio
 
 [handlers]
 keys=console_handler,file_handler_name1,file_handler_name2
@@ -24,6 +24,12 @@ handlers=console_handler,file_handler_name1
 propagate=0
 qualname=mqtt
 
+[logger_asyncio]
+level=INFO
+handlers=console_handler,file_handler_name1
+propagate=0
+qualname=asyncio
+
 [logger_data]
 level=DEBUG
 handlers=file_handler_name1

app/src/messages.py

@@ -1,346 +1,139 @@
-import struct, logging, time, datetime
+import logging
 import weakref
-from datetime import datetime
+from typing import Callable, Generator
+from enum import Enum
+
 
 if __name__ == "app.src.messages":
-    from app.src.infos import Infos
-    from app.src.config import Config
-else:
-    from infos import Infos
-    from config import Config
+    from app.src.infos import Infos, Register
+    from app.src.modbus import Modbus
+else:  # pragma: no cover
+    from infos import Infos, Register
+    from modbus import Modbus
 
 logger = logging.getLogger('msg')
-    
-        
-def hex_dump_memory(level, info, data, num):
-    s = ''
+
+
+def hex_dump_memory(level, info, data, data_len):
     n = 0
     lines = []
     lines.append(info)
     tracer = logging.getLogger('tracer')
-    if not tracer.isEnabledFor(level): return
+    if not tracer.isEnabledFor(level):
+        return
 
-
-    #data = list((num * ctypes.c_byte).from_address(ptr))
-
-    if len(data) == 0:
-        return '<empty>'
-
-    for i in range(0, num, 16):
+    for i in range(0, data_len, 16):
         line = '  '
         line += '%04x | ' % (i)
         n += 16
 
         for j in range(n-16, n):
-            if j >= len(data): break
+            if j >= data_len:
+                break
             line += '%02x ' % abs(data[j])
 
         line += ' ' * (3 * 16 + 9 - len(line)) + ' | '
 
         for j in range(n-16, n):
-            if j >= len(data): break
+            if j >= data_len:
+                break
             c = data[j] if not (data[j] < 0x20 or data[j] > 0x7e) else '.'
             line += '%c' % c
 
         lines.append(line)
 
     tracer.log(level, '\n'.join(lines))
-        
-    #return '\n'.join(lines)
 
 
-class Control:
-    def __init__(self, ctrl:int):
-        self.ctrl = ctrl
-    
-    def __int__(self) -> int:
-        return self.ctrl
-
-    def is_ind(self) -> bool:
-        return not (self.ctrl & 0x08)
-    
-    #def is_req(self) -> bool:
-    #    return not (self.ctrl & 0x08)
-    
-    def is_resp(self) -> bool:
-        return self.ctrl & 0x08
-
 class IterRegistry(type):
-    def __iter__(cls):
+    def __iter__(cls) -> Generator['Message', None, None]:
         for ref in cls._registry:
             obj = ref()
-            if obj is not None: yield obj
+            if obj is not None:
+                yield obj
+
+
+class State(Enum):
+    '''state of the logical connection'''
+    init = 0
+    '''just created'''
+    received = 1
+    '''at least one packet received'''
+    up = 2
+    '''at least one cmd-rsp transaction'''
+    pend = 3
+    '''inverter transaction pending, don't send MODBUS cmds'''
+    closed = 4
+    '''connection closed'''
 
 
 class Message(metaclass=IterRegistry):
     _registry = []
 
-    def __init__(self):
+    def __init__(self, server_side: bool, send_modbus_cb:
+                 Callable[[bytes, int, str], None], mb_timeout: int):
         self._registry.append(weakref.ref(self))
+
+        self.server_side = server_side
+        if server_side:
+            self.mb = Modbus(send_modbus_cb, mb_timeout)
+        else:
+            self.mb = None
+
         self.header_valid = False
         self.header_len = 0
         self.data_len = 0
         self.unique_id = 0
-        self.node_id = ''
+        self.node_id = ''  # will be overwritten in the child class's __init__
         self.sug_area = ''
-        self._recv_buffer = b''
+        self._recv_buffer = bytearray(0)
         self._send_buffer = bytearray(0)
         self._forward_buffer = bytearray(0)
-        self.db = Infos() 
         self.new_data = {}
-        self.switch={
-            0x00: self.msg_contact_info,
-            0x22: self.msg_get_time,
-            0x71: self.msg_collector_data,
-            0x04: self.msg_inverter_data,
-        }
-    
+        self.state = State.init
+        self.shutdown_started = False
+
     '''
-    Empty methods, that have to be implemented in any child class which don't use asyncio
+    Empty methods, that have to be implemented in any child class which
+    don't use asyncio
     '''
-    def _read(self) -> None:     # read data bytes from socket and copy them to our _recv_buffer
-        return  
+    def _read(self) -> None:     # read data bytes from socket and copy them
+        # to our _recv_buffer
+        return  # pragma: no cover
+
+    def _update_header(self, _forward_buffer):
+        '''callback for updating the header of the forward buffer'''
+        return  # pragma: no cover
+
+    def _set_mqtt_timestamp(self, key, ts: float | None):
+        if type(ts) is not None and \
+           key not in self.new_data or \
+           not self.new_data[key]:
+            if key == 'grid':
+                info_id = Register.TS_GRID
+            elif key == 'input':
+                info_id = Register.TS_INPUT
+            elif key == 'total':
+                info_id = Register.TS_TOTAL
+            else:
+                return
+            # tstr = time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime(ts))
+            # logger.info(f'update: key: {key} ts:{tstr}'
+            self.db.set_db_def_value(info_id, round(ts))
 
     '''
     Our puplic methods
     '''
     def close(self) -> None:
-        # we have refernces to methods of this class in self.switch
-        # so we have to erase self.switch, otherwise this instance can't be 
-        # deallocated by the garbage collector ==> we get a memory leak
-        del self.switch 
+        if self.mb:
+            self.mb.close()
+            self.mb = None
+        pass  # pragma: no cover
 
-    def inc_counter(self, counter:str) -> None:
+    def inc_counter(self, counter: str) -> None:
         self.db.inc_counter(counter)
-        self.new_data['proxy'] = True
+        Infos.new_stat_data['proxy'] = True
 
-    def dec_counter(self, counter:str) -> None:
+    def dec_counter(self, counter: str) -> None:
         self.db.dec_counter(counter)
-        self.new_data['proxy'] = True
-       
-    def set_serial_no(self, serial_no : str):
-        
-        if self.unique_id == serial_no: 
-            logger.debug(f'SerialNo: {serial_no}')
-        else:
-            inverters = Config.get('inverters')
-            #logger.debug(f'Inverters: {inverters}')
-                
-            if serial_no in inverters:
-                inv = inverters[serial_no]
-                self.node_id = inv['node_id']
-                self.sug_area    = inv['suggested_area']
-                logger.debug(f'SerialNo {serial_no} allowed!  area:{self.sug_area}')
-            else:    
-                self.node_id = ''
-                self.sug_area    = ''
-                if 'allow_all' not in inverters or not inverters['allow_all']:
-                    self.inc_counter('Unknown_SNR')
-                    self.unique_id = None
-                    logger.warning(f'ignore message from unknow inverter! (SerialNo: {serial_no})')
-                    return
-                logger.debug(f'SerialNo {serial_no} not known but accepted!')
-                    
-            self.unique_id = serial_no
-    
-
-    def read(self) -> None:
-        self._read()
-        
-        if not self.header_valid:
-            self.__parse_header(self._recv_buffer, len(self._recv_buffer))
-        
-        if self.header_valid and len(self._recv_buffer) >= (self.header_len+self.data_len):
-            hex_dump_memory(logging.INFO, f'Received from {self.addr}:', self._recv_buffer, self.header_len+self.data_len)
-
-            if self.id_str:
-                self.set_serial_no(self.id_str.decode("utf-8"))
-            
-            self.__dispatch_msg()
-            self.__flush_recv_msg()
-        return
-        
-    def forward(self, buffer, buflen) -> None:
-        tsun = Config.get('tsun')
-        if tsun['enabled']:
-            self._forward_buffer = buffer[:buflen]
-            hex_dump_memory(logging.DEBUG, 'Store for forwarding:', buffer, buflen)
-
-            self.__parse_header(self._forward_buffer, len(self._forward_buffer))
-            fnc = self.switch.get(self.msg_id, self.msg_unknown)
-            logger.info(self.__flow_str(self.server_side, 'forwrd') + f' Ctl: {int(self.ctrl):#02x} Msg: {fnc.__name__!r}' )
-        return
-    
-    '''
-    Our private methods
-    '''
-    def __flow_str(self, server_side:bool, type:('rx','tx','forwrd', 'drop')):
-        switch={
-            'rx':     '  <',
-            'tx':     '  >',
-            'forwrd': '<< ',
-            'drop':   ' xx',
-            'rxS':    '>  ',
-            'txS':    '<  ',
-            'forwrdS':' >>',
-            'dropS':  'xx ',
-        }
-        if server_side: type +='S'
-        return switch.get(type, '???')
-
-    def __timestamp(self):
-        if False:
-            # utc as epoche 
-            ts = time.time()
-        else:
-            # convert localtime in epoche 
-            ts = (datetime.now() - datetime(1970,1,1)).total_seconds()
-        return round(ts*1000)    
-
-    # check if there is a complete header in the buffer, parse it 
-    # and set
-    #   self.header_len
-    #   self.data_len
-    #   self.id_str
-    #   self.ctrl
-    #   self.msg_id
-    #
-    # if the header is incomplete, than self.header_len is still 0
-    #   
-    def __parse_header(self, buf:bytes, buf_len:int) -> None:
-    
-        if (buf_len <5):      # enough bytes to read len and id_len?
-            return
-        result = struct.unpack_from('!lB', buf, 0)
-        len    = result[0]    # len of complete message
-        id_len = result[1]    # len of variable id string
-
-        hdr_len = 5+id_len+2
-    
-        if (buf_len < hdr_len):  # enough bytes for complete header?
-            return
-        
-        result = struct.unpack_from(f'!{id_len+1}pBB', buf, 4)
-        
-        # store parsed header values in the class 
-        self.id_str = result[0]
-        self.ctrl   = Control(result[1])
-        self.msg_id = result[2] 
-        self.data_len = len-id_len-3
-        self.header_len = hdr_len
-        self.header_valid = True
-        return
-    
-    def __build_header(self, ctrl) -> None:
-        self.send_msg_ofs = len (self._send_buffer)
-        self._send_buffer += struct.pack(f'!l{len(self.id_str)+1}pBB', 0, self.id_str, ctrl, self.msg_id)
-        fnc = self.switch.get(self.msg_id, self.msg_unknown)
-        logger.info(self.__flow_str(self.server_side, 'tx') + f' Ctl: {int(ctrl):#02x} Msg: {fnc.__name__!r}' )
-        
-    def __finish_send_msg(self) -> None:
-        _len = len(self._send_buffer) - self.send_msg_ofs
-        struct.pack_into('!l',self._send_buffer, self.send_msg_ofs, _len-4)
-    
-    
-
-    def __dispatch_msg(self) -> None:
-        fnc = self.switch.get(self.msg_id, self.msg_unknown)
-        if self.unique_id: 
-            logger.info(self.__flow_str(self.server_side, 'rx') + f' Ctl: {int(self.ctrl):#02x} Msg: {fnc.__name__!r}' )
-            fnc()
-        else:
-            logger.info(self.__flow_str(self.server_side, 'drop') + f' Ctl: {int(self.ctrl):#02x} Msg: {fnc.__name__!r}' )
-        
-    
-    def __flush_recv_msg(self) -> None:
-        self._recv_buffer = self._recv_buffer[(self.header_len+self.data_len):]
-        self.header_valid = False
-
-    '''
-    Message handler methods
-    '''
-    def msg_contact_info(self):
-        if self.ctrl.is_ind():
-            self.__build_header(0x99)
-            self._send_buffer += b'\x01'
-            self.__finish_send_msg()
-        elif self.ctrl.is_resp():
-            return # ignore received response from tsun
-        self.forward(self._recv_buffer, self.header_len+self.data_len)
-
-    def msg_get_time(self):
-        if self.ctrl.is_ind():
-            ts = self.__timestamp()
-            logger.debug(f'time: {ts:08x}')
-            
-            self.__build_header(0x99)
-            self._send_buffer += struct.pack('!q', ts)
-            self.__finish_send_msg()
-    
-        elif self.ctrl.is_resp():
-            result = struct.unpack_from(f'!q', self._recv_buffer, self.header_len)
-            logger.debug(f'tsun-time: {result[0]:08x}')
-            return # ignore received response from tsun
-        
-        self.forward(self._recv_buffer, self.header_len+self.data_len)
-            
-
-
-    def parse_msg_header(self):
-        result = struct.unpack_from('!lB', self._recv_buffer, self.header_len)
-
-        data_id = result[0]    # len of complete message
-        id_len  = result[1]    # len of variable id string
-        logger.debug(f'Data_ID: {data_id}   id_len:  {id_len}')
- 
-        msg_hdr_len= 5+id_len+9
-    
-        result = struct.unpack_from(f'!{id_len+1}pBq', self._recv_buffer, self.header_len+4)
- 
-        logger.debug(f'ID: {result[0]}  B: {result[1]}')
-        logger.debug(f'time: {result[2]:08x}')
-        #logger.info(f'time: {datetime.utcfromtimestamp(result[2]).strftime("%Y-%m-%d %H:%M:%S")}')
-        return msg_hdr_len
- 
-
-    def msg_collector_data(self):
-        if self.ctrl.is_ind():
-            self.__build_header(0x99)
-            self._send_buffer += b'\x01'
-            self.__finish_send_msg()
-        
-        elif self.ctrl.is_resp():
-            return # ignore received response
-        
-        self.forward(self._recv_buffer, self.header_len+self.data_len)
-        self.__process_data()
-
-
-    def msg_inverter_data(self):
-        if self.ctrl.is_ind():
-            self.__build_header(0x99)
-            self._send_buffer += b'\x01'
-            self.__finish_send_msg()
-        
-        elif self.ctrl.is_resp():
-            return # ignore received response
-        
-        self.forward(self._recv_buffer, self.header_len+self.data_len)
-        self.__process_data()
-        
-    def __process_data(self):
-        msg_hdr_len = self.parse_msg_header()
-
-        for key, update in self.db.parse(self._recv_buffer[self.header_len + msg_hdr_len:]):
-            if update: self.new_data[key] = True
-
-
-
-    def msg_unknown(self):
-        logger.warning (f"Unknow Msg: ID:{self.msg_id}")       
-        self.inc_counter('Unknown_Msg')
-        self.forward(self._recv_buffer, self.header_len+self.data_len)
-
-        
-
-
+        Infos.new_stat_data['proxy'] = True

app/src/modbus.py

@@ -0,0 +1,322 @@
+'''MODBUS  module for TSUN inverter support
+
+TSUN uses the MODBUS in the RTU transmission mode over serial line.
+see: https://modbus.org/docs/Modbus_Application_Protocol_V1_1b3.pdf
+see: https://modbus.org/docs/Modbus_over_serial_line_V1_02.pdf
+
+A Modbus PDU consists of: 'Function-Code' + 'Data'
+A Modbus RTU message consists of: 'Addr' + 'Modbus-PDU' + 'CRC-16'
+The inverter is a MODBUS server and the proxy the MODBUS client.
+
+The 16-bit CRC is known as CRC-16-ANSI(reverse)
+see: https://en.wikipedia.org/wiki/Computation_of_cyclic_redundancy_checks
+'''
+import struct
+import logging
+import asyncio
+from typing import Generator, Callable
+
+if __name__ == "app.src.modbus":
+    from app.src.infos import Register
+else:  # pragma: no cover
+    from infos import Register
+
+logger = logging.getLogger('data')
+
+CRC_POLY = 0xA001  # (LSBF/reverse)
+CRC_INIT = 0xFFFF
+
+
+class Modbus():
+    '''Simple MODBUS implementation with TX queue and retransmit timer'''
+    INV_ADDR = 1
+    '''MODBUS server address of the TSUN inverter'''
+    READ_REGS = 3
+    '''MODBUS function code: Read Holding Register'''
+    READ_INPUTS = 4
+    '''MODBUS function code: Read Input Register'''
+    WRITE_SINGLE_REG = 6
+    '''Modbus function code: Write Single Register'''
+
+    __crc_tab = []
+    map = {
+        0x2007: {'reg': Register.MAX_DESIGNED_POWER,   'fmt': '!H', 'ratio':  1},  # noqa: E501
+        0x202c: {'reg': Register.OUTPUT_COEFFICIENT,   'fmt': '!H', 'ratio':  100/1024},  # noqa: E501
+
+        0x3000: {'reg': Register.INVERTER_STATUS,      'fmt': '!H'},                 # noqa: E501
+        0x3008: {'reg': Register.VERSION,              'fmt': '!H', 'eval': "f'V{(result>>12)}.{(result>>8)&0xf}.{(result>>4)&0xf}{result&0xf:1X}'"},  # noqa: E501
+        0x3009: {'reg': Register.GRID_VOLTAGE,         'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x300a: {'reg': Register.GRID_CURRENT,         'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x300b: {'reg': Register.GRID_FREQUENCY,       'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x300c: {'reg': Register.INVERTER_TEMP,        'fmt': '!H', 'eval': 'result-40'},  # noqa: E501
+        # 0x300d
+        0x300e: {'reg': Register.RATED_POWER,          'fmt': '!H', 'ratio':    1},  # noqa: E501
+        0x300f: {'reg': Register.OUTPUT_POWER,         'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x3010: {'reg': Register.PV1_VOLTAGE,          'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x3011: {'reg': Register.PV1_CURRENT,          'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x3012: {'reg': Register.PV1_POWER,            'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x3013: {'reg': Register.PV2_VOLTAGE,          'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x3014: {'reg': Register.PV2_CURRENT,          'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x3015: {'reg': Register.PV2_POWER,            'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x3016: {'reg': Register.PV3_VOLTAGE,          'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x3017: {'reg': Register.PV3_CURRENT,          'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x3018: {'reg': Register.PV3_POWER,            'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x3019: {'reg': Register.PV4_VOLTAGE,          'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x301a: {'reg': Register.PV4_CURRENT,          'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x301b: {'reg': Register.PV4_POWER,            'fmt': '!H', 'ratio':  0.1},  # noqa: E501
+        0x301c: {'reg': Register.DAILY_GENERATION,     'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x301d: {'reg': Register.TOTAL_GENERATION,     'fmt': '!L', 'ratio': 0.01},  # noqa: E501
+        0x301f: {'reg': Register.PV1_DAILY_GENERATION, 'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x3020: {'reg': Register.PV1_TOTAL_GENERATION, 'fmt': '!L', 'ratio': 0.01},  # noqa: E501
+        0x3022: {'reg': Register.PV2_DAILY_GENERATION, 'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x3023: {'reg': Register.PV2_TOTAL_GENERATION, 'fmt': '!L', 'ratio': 0.01},  # noqa: E501
+        0x3025: {'reg': Register.PV3_DAILY_GENERATION, 'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x3026: {'reg': Register.PV3_TOTAL_GENERATION, 'fmt': '!L', 'ratio': 0.01},  # noqa: E501
+        0x3028: {'reg': Register.PV4_DAILY_GENERATION, 'fmt': '!H', 'ratio': 0.01},  # noqa: E501
+        0x3029: {'reg': Register.PV4_TOTAL_GENERATION, 'fmt': '!L', 'ratio': 0.01},  # noqa: E501
+    }
+
+    def __init__(self, snd_handler: Callable[[bytes, int, str], None],
+                 timeout: int = 1):
+        if not len(self.__crc_tab):
+            self.__build_crc_tab(CRC_POLY)
+        self.que = asyncio.Queue(100)
+        self.snd_handler = snd_handler
+        '''Send handler to transmit a MODBUS RTU request'''
+        self.rsp_handler = None
+        '''Response handler to forward the response'''
+        self.timeout = timeout
+        '''MODBUS response timeout in seconds'''
+        self.max_retries = 1
+        '''Max retransmit for MODBUS requests'''
+        self.retry_cnt = 0
+        self.last_req = b''
+        self.counter = {}
+        '''Dictenary with statistic counter'''
+        self.counter['timeouts'] = 0
+        self.counter['retries'] = {}
+        for i in range(0, self.max_retries+1):
+            self.counter['retries'][f'{i}'] = 0
+        self.last_log_lvl = logging.DEBUG
+        self.last_addr = 0
+        self.last_fcode = 0
+        self.last_len = 0
+        self.last_reg = 0
+        self.err = 0
+        self.loop = asyncio.get_event_loop()
+        self.req_pend = False
+        self.tim = None
+
+    def close(self):
+        """free the queue and erase the callback handlers"""
+        logging.debug('Modbus close:')
+        self.__stop_timer()
+        self.rsp_handler = None
+        self.snd_handler = None
+        while not self.que.empty():
+            self.que.get_nowait()
+
+    def __del__(self):
+        """log statistics on the deleting of a MODBUS instance"""
+        logging.debug(f'Modbus __del__:\n {self.counter}')
+
+    def build_msg(self, addr: int, func: int, reg: int, val: int,
+                  log_lvl=logging.DEBUG) -> None:
+        """Build MODBUS RTU request frame and add it to the tx queue
+
+        Keyword arguments:
+            addr: RTU server address (inverter)
+            func: MODBUS function code
+            reg:  16-bit register number
+            val:  16 bit value
+        """
+        msg = struct.pack('>BBHH', addr, func, reg, val)
+        msg += struct.pack('<H', self.__calc_crc(msg))
+        self.que.put_nowait({'req': msg,
+                             'rsp_hdl': None,
+                             'log_lvl': log_lvl})
+        if self.que.qsize() == 1:
+            self.__send_next_from_que()
+
+    def recv_req(self, buf: bytearray,
+                 rsp_handler: Callable[[None], None] = None) -> bool:
+        """Add the received Modbus RTU request to the tx queue
+
+        Keyword arguments:
+            buf: Modbus RTU pdu incl ADDR byte and trailing CRC
+            rsp_handler: Callback, if the received pdu is valid
+
+        Returns:
+            True:   PDU was added to the queue
+            False:  PDU was ignored, due to an error
+        """
+        # logging.info(f'recv_req: first byte modbus:{buf[0]} len:{len(buf)}')
+        if not self.__check_crc(buf):
+            self.err = 1
+            logger.error('Modbus recv: CRC error')
+            return False
+        self.que.put_nowait({'req': buf,
+                             'rsp_hdl': rsp_handler,
+                             'log_lvl': logging.INFO})
+        if self.que.qsize() == 1:
+            self.__send_next_from_que()
+
+        return True
+
+    def recv_resp(self, info_db, buf: bytearray, node_id: str) -> \
+            Generator[tuple[str, bool, int | float | str], None, None]:
+        """Generator which check and parse a received MODBUS response.
+
+        Keyword arguments:
+            info_db: database for info lockups
+            buf: received Modbus RTU response frame
+            node_id: string for logging which identifies the slave
+
+        Returns on error and set Self.err to:
+            1: CRC error
+            2: Wrong server address
+            3: Unexpected function code
+            4: Unexpected data length
+            5: No MODBUS request pending
+        """
+        # logging.info(f'recv_resp: first byte modbus:{buf[0]} len:{len(buf)}')
+        if not self.req_pend:
+            self.err = 5
+            return
+        if not self.__check_crc(buf):
+            logger.error(f'[{node_id}] Modbus resp: CRC error')
+            self.err = 1
+            return
+        if buf[0] != self.last_addr:
+            logger.info(f'[{node_id}] Modbus resp: Wrong addr {buf[0]}')
+            self.err = 2
+            return
+        fcode = buf[1]
+        if fcode != self.last_fcode:
+            logger.info(f'[{node_id}] Modbus: Wrong fcode {fcode}'
+                        f' != {self.last_fcode}')
+            self.err = 3
+            return
+        if self.last_addr == self.INV_ADDR and \
+                (fcode == 3 or fcode == 4):
+            elmlen = buf[2] >> 1
+            if elmlen != self.last_len:
+                logger.info(f'[{node_id}] Modbus: len error {elmlen}'
+                            f' != {self.last_len}')
+                self.err = 4
+                return
+            first_reg = self.last_reg  # save last_reg before sending next pdu
+            self.__stop_timer()          # stop timer and send next pdu
+
+            for i in range(0, elmlen):
+                addr = first_reg+i
+                if addr in self.map:
+                    row = self.map[addr]
+                    info_id = row['reg']
+                    fmt = row['fmt']
+                    val = struct.unpack_from(fmt, buf, 3+2*i)
+                    result = val[0]
+
+                    if 'eval' in row:
+                        result = eval(row['eval'])
+                    if 'ratio' in row:
+                        result = round(result * row['ratio'], 2)
+
+                    keys, level, unit, must_incr = info_db._key_obj(info_id)
+
+                    if keys:
+                        name, update = info_db.update_db(keys, must_incr,
+                                                         result)
+                        yield keys[0], update, result
+                        if update:
+                            info_db.tracer.log(level,
+                                               f'[{node_id}] MODBUS: {name}'
+                                               f' : {result}{unit}')
+        else:
+            self.__stop_timer()
+
+        self.counter['retries'][f'{self.retry_cnt}'] += 1
+        if self.rsp_handler:
+            self.rsp_handler()
+        self.__send_next_from_que()
+
+    '''
+    MODBUS response timer
+    '''
+    def __start_timer(self) -> None:
+        '''Start response timer and set `req_pend` to True'''
+        self.req_pend = True
+        self.tim = self.loop.call_later(self.timeout, self.__timeout_cb)
+        # logging.debug(f'Modbus start timer {self}')
+
+    def __stop_timer(self) -> None:
+        '''Stop response timer and set `req_pend` to False'''
+        self.req_pend = False
+        # logging.debug(f'Modbus stop timer {self}')
+        if self.tim:
+            self.tim.cancel()
+            self.tim = None
+
+    def __timeout_cb(self) -> None:
+        '''Rsponse timeout handler retransmit pdu or send next pdu'''
+        self.req_pend = False
+
+        if self.retry_cnt < self.max_retries:
+            logger.debug(f'Modbus retrans {self}')
+            self.retry_cnt += 1
+            self.__start_timer()
+            self.snd_handler(self.last_req, self.last_log_lvl, state='Retrans')
+        else:
+            logger.info(f'Modbus timeout {self}')
+            self.counter['timeouts'] += 1
+            self.__send_next_from_que()
+
+    def __send_next_from_que(self) -> None:
+        '''Get next MODBUS pdu from queue and transmit it'''
+        if self.req_pend:
+            return
+        try:
+            item = self.que.get_nowait()
+            req = item['req']
+            self.last_req = req
+            self.rsp_handler = item['rsp_hdl']
+            self.last_log_lvl = item['log_lvl']
+            self.last_addr = req[0]
+            self.last_fcode = req[1]
+
+            res = struct.unpack_from('>HH', req, 2)
+            self.last_reg = res[0]
+            self.last_len = res[1]
+            self.retry_cnt = 0
+            self.__start_timer()
+            self.snd_handler(self.last_req, self.last_log_lvl, state='Command')
+        except asyncio.QueueEmpty:
+            pass
+
+    '''
+    Helper function for CRC-16 handling
+    '''
+    def __check_crc(self, msg: bytearray) -> bool:
+        '''Check CRC-16 and returns True if valid'''
+        return 0 == self.__calc_crc(msg)
+
+    def __calc_crc(self, buffer: bytearray) -> int:
+        '''Build CRC-16 for buffer and returns it'''
+        crc = CRC_INIT
+
+        for cur in buffer:
+            crc = (crc >> 8) ^ self.__crc_tab[(crc ^ cur) & 0xFF]
+        return crc
+
+    def __build_crc_tab(self, poly: int) -> None:
+        '''Build CRC-16 helper table, must be called exactly one time'''
+        for index in range(256):
+            data = index << 1
+            crc = 0
+            for _ in range(8, 0, -1):
+                data >>= 1
+                if (data ^ crc) & 1:
+                    crc = (crc >> 1) ^ poly
+                else:
+                    crc >>= 1
+            self.__crc_tab.append(crc)

app/src/modbus_tcp.py

@@ -0,0 +1,76 @@
+import logging
+import traceback
+import asyncio
+from config import Config
+
+# import gc
+from gen3plus.inverter_g3p import InverterG3P
+
+logger = logging.getLogger('conn')
+
+
+class ModbusConn():
+    def __init__(self, host, port):
+        self.host = host
+        self.port = port
+        self.addr = (host, port)
+        self.stream = None
+
+    async def __aenter__(self) -> 'InverterG3P':
+        '''Establish a client connection to the TSUN cloud'''
+        connection = asyncio.open_connection(self.host, self.port)
+        reader, writer = await connection
+        self.stream = InverterG3P(reader, writer, self.addr,
+                                  client_mode=True)
+        logging.info(f'[{self.stream.node_id}:{self.stream.conn_no}] '
+                     f'Connected to {self.addr}')
+        self.stream.inc_counter('Inverter_Cnt')
+        await self.stream.publish_outstanding_mqtt()
+        return self.stream
+
+    async def __aexit__(self, exc_type, exc, tb):
+        self.stream.dec_counter('Inverter_Cnt')
+        await self.stream.publish_outstanding_mqtt()
+
+
+class ModbusTcp():
+
+    def __init__(self, loop) -> None:
+        inverters = Config.get('inverters')
+        # logging.info(f'Inverters: {inverters}')
+
+        for inv in inverters.values():
+            if (type(inv) is dict
+               and 'monitor_sn' in inv
+               and 'client_mode' in inv):
+                client = inv['client_mode']
+                # logging.info(f"SerialNo:{inv['monitor_sn']} host:{client['host']} port:{client['port']}")  # noqa: E501
+                loop.create_task(self.modbus_loop(client['host'],
+                                                  client['port'],
+                                                  inv['monitor_sn']))
+
+    async def modbus_loop(self, host, port, snr: int) -> None:
+        '''Loop for receiving messages from the TSUN cloud (client-side)'''
+        while True:
+            try:
+                async with ModbusConn(host, port) as stream:
+                    await stream.send_start_cmd(snr, host)
+                    await stream.loop()
+                    logger.info(f'[{stream.node_id}:{stream.conn_no}] '
+                                f'Connection closed - Shutdown: '
+                                f'{stream.shutdown_started}')
+                    if stream.shutdown_started:
+                        return
+
+            except (ConnectionRefusedError, TimeoutError) as error:
+                logging.debug(f'Inv-conn:{error}')
+
+            except OSError as error:
+                logging.info(f'os-error: {error}')
+
+            except Exception:
+                logging.error(
+                    f"ModbusTcpCreate: Exception for {(host,port)}:\n"
+                    f"{traceback.format_exc()}")
+
+            await asyncio.sleep(10)

app/src/mqtt.py

@@ -1,78 +1,189 @@
-import asyncio, logging
+import asyncio
+import logging
 import aiomqtt
+import traceback
+from modbus import Modbus
+from messages import Message
 from config import Config
+from singleton import Singleton
 
 logger_mqtt = logging.getLogger('mqtt')
 
 
-class Singleton(type):
-    _instances = {}
-    def __call__(cls, *args, **kwargs):
-        logger_mqtt.debug(f'singleton: __call__')                                
-        if cls not in cls._instances:
-            cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
-        return cls._instances[cls]
-
-
 class Mqtt(metaclass=Singleton):
-    client = None
-    
-    def __init__(self):
-        logger_mqtt.debug(f'MQTT: __init__') 
+    __client = None
+    __cb_MqttIsUp = None
+
+    def __init__(self, cb_MqttIsUp):
+        logger_mqtt.debug('MQTT: __init__')
+        if cb_MqttIsUp:
+            self.__cb_MqttIsUp = cb_MqttIsUp
         loop = asyncio.get_event_loop()
         self.task = loop.create_task(self.__loop())
-        self.ha_restarts = 0        
+        self.ha_restarts = 0
 
-    
     @property
     def ha_restarts(self):
         return self._ha_restarts
-        
+
     @ha_restarts.setter
     def ha_restarts(self, value):
         self._ha_restarts = value
-            
+
     def __del__(self):
-        logger_mqtt.debug(f'MQTT: __del__')
-    
+        logger_mqtt.debug('MQTT: __del__')
 
     async def close(self) -> None:
-        logger_mqtt.debug(f'MQTT: close') 
+        logger_mqtt.debug('MQTT: close')
         self.task.cancel()
         try:
             await self.task
-        except Exception as e:
+
+        except (asyncio.CancelledError, Exception) as e:
             logging.debug(f"Mqtt.close: exception: {e} ...")
-        
-  
-
-    async def publish(self, topic: str, payload: str | bytes | bytearray | int | float | None = None) -> None:
-        if self.client:
-            await self.client.publish(topic, payload)
 
+    async def publish(self, topic: str, payload: str | bytes | bytearray
+                      | int | float | None = None) -> None:
+        if self.__client:
+            await self.__client.publish(topic, payload)
 
     async def __loop(self) -> None:
         mqtt = Config.get('mqtt')
         ha = Config.get('ha')
-        logger_mqtt.info(f'start MQTT: host:{mqtt["host"]}  port:{mqtt["port"]}  user:{mqtt["user"]}')                                
-        self.client = aiomqtt.Client(hostname=mqtt['host'], port=mqtt['port'], username=mqtt['user'], password=mqtt['passwd'])
-     
+        logger_mqtt.info(f'start MQTT: host:{mqtt["host"]}  port:'
+                         f'{mqtt["port"]}  '
+                         f'user:{mqtt["user"]}')
+        self.__client = aiomqtt.Client(hostname=mqtt['host'],
+                                       port=mqtt['port'],
+                                       username=mqtt['user'],
+                                       password=mqtt['passwd'])
+
         interval = 5  # Seconds
+        ha_status_topic = f"{ha['auto_conf_prefix']}/status"
+        mb_rated_topic = "tsun/+/rated_load"  # fixme
+        mb_out_coeff_topic = "tsun/+/out_coeff"  # fixme
+        mb_reads_topic = "tsun/+/modbus_read_regs"  # fixme
+        mb_inputs_topic = "tsun/+/modbus_read_inputs"  # fixme
+        mb_at_cmd_topic = "tsun/+/at_cmd"  # fixme
+
         while True:
             try:
-                async with self.client:
-                    async with self.client.messages() as messages:
-                        await self.client.subscribe(f"{ha['auto_conf_prefix']}/status")
-                        async for message in messages:
+                async with self.__client:
+                    logger_mqtt.info('MQTT broker connection established')
+
+                    if self.__cb_MqttIsUp:
+                        await self.__cb_MqttIsUp()
+
+                    # async with self.__client.messages() as messages:
+                    await self.__client.subscribe(ha_status_topic)
+                    await self.__client.subscribe(mb_rated_topic)
+                    await self.__client.subscribe(mb_out_coeff_topic)
+                    await self.__client.subscribe(mb_reads_topic)
+                    await self.__client.subscribe(mb_inputs_topic)
+                    await self.__client.subscribe(mb_at_cmd_topic)
+
+                    async for message in self.__client.messages:
+                        if message.topic.matches(ha_status_topic):
                             status = message.payload.decode("UTF-8")
-                            logger_mqtt.info(f'Home-Assistant Status: {status}')
+                            logger_mqtt.info('Home-Assistant Status:'
+                                             f' {status}')
                             if status == 'online':
                                 self.ha_restarts += 1
+                                await self.__cb_MqttIsUp()
+
+                        if message.topic.matches(mb_rated_topic):
+                            await self.modbus_cmd(message,
+                                                  Modbus.WRITE_SINGLE_REG,
+                                                  1, 0x2008)
+
+                        if message.topic.matches(mb_out_coeff_topic):
+                            payload = message.payload.decode("UTF-8")
+                            val = round(float(payload) * 1024/100)
+
+                            if val < 0 or val > 1024:
+                                logger_mqtt.error('out_coeff: value must be in'
+                                                  'the range 0..100,'
+                                                  f' got: {payload}')
+                            else:
+                                await self.modbus_cmd(message,
+                                                      Modbus.WRITE_SINGLE_REG,
+                                                      0, 0x202c, val)
+
+                        if message.topic.matches(mb_reads_topic):
+                            await self.modbus_cmd(message,
+                                                  Modbus.READ_REGS, 2)
+
+                        if message.topic.matches(mb_inputs_topic):
+                            await self.modbus_cmd(message,
+                                                  Modbus.READ_INPUTS, 2)
+
+                        if message.topic.matches(mb_at_cmd_topic):
+                            await self.at_cmd(message)
 
             except aiomqtt.MqttError:
-                logger_mqtt.info(f"Connection lost; Reconnecting in {interval} seconds ...")
+                if Config.is_default('mqtt'):
+                    logger_mqtt.info(
+                        "MQTT is unconfigured; Check your config.toml!")
+                    interval = 30
+                else:
+                    interval = 5  # Seconds
+                    logger_mqtt.info(
+                        f"Connection lost; Reconnecting in {interval}"
+                        " seconds ...")
+
                 await asyncio.sleep(interval)
             except asyncio.CancelledError:
-                logger_mqtt.debug(f"MQTT task cancelled")
-                self.client = None
+                logger_mqtt.debug("MQTT task cancelled")
+                self.__client = None
                 return
+            except Exception:
+                # self.inc_counter('SW_Exception')   # fixme
+                logger_mqtt.error(
+                    f"Exception:\n"
+                    f"{traceback.format_exc()}")
+
+    def each_inverter(self, message, func_name: str):
+        topic = str(message.topic)
+        node_id = topic.split('/')[1] + '/'
+        found = False
+        for m in Message:
+            if m.server_side and (m.node_id == node_id):
+                found = True
+                logger_mqtt.debug(f'Found: {node_id}')
+                fnc = getattr(m, func_name, None)
+                if callable(fnc):
+                    yield fnc
+                else:
+                    logger_mqtt.warning(f'Cmd not supported by: {node_id}')
+
+        if not found:
+            logger_mqtt.warning(f'Node_id: {node_id} not found')
+
+    async def modbus_cmd(self, message, func, params=0, addr=0, val=0):
+        topic = str(message.topic)
+        node_id = topic.split('/')[1] + '/'
+        # refactor into a loop over a table
+        payload = message.payload.decode("UTF-8")
+        logger_mqtt.info(f'MODBUS via MQTT: {topic} = {payload}')
+        for m in Message:
+            if m.server_side and (m.node_id == node_id):
+                logger_mqtt.debug(f'Found: {node_id}')
+                fnc = getattr(m, "send_modbus_cmd", None)
+                res = payload.split(',')
+                if params > 0 and params != len(res):
+                    logger_mqtt.error(f'Parameter expected: {params}, '
+                                      f'got: {len(res)}')
+                    return
+
+                if callable(fnc):
+                    if params == 1:
+                        val = int(payload)
+                    elif params == 2:
+                        addr = int(res[0], base=16)
+                        val = int(res[1])  # lenght
+                    await fnc(func, addr, val, logging.INFO)
+
+    async def at_cmd(self, message):
+        payload = message.payload.decode("UTF-8")
+        for fnc in self.each_inverter(message, "send_at_cmd"):
+            await fnc(payload)

app/src/my_timer.py

@@ -0,0 +1,35 @@
+import asyncio
+import logging
+from itertools import count
+
+
+class Timer:
+    def __init__(self, cb, id_str: str = ''):
+        self.__timeout_cb = cb
+        self.loop = asyncio.get_event_loop()
+        self.tim = None
+        self.id_str = id_str
+        self.exp_count = count(0)
+
+    def start(self, timeout: float) -> None:
+        '''Start timer with timeout seconds'''
+        if self.tim:
+            self.tim.cancel()
+        self.tim = self.loop.call_later(timeout, self.__timeout)
+        logging.debug(f'[{self.id_str}]Start timer')
+
+    def stop(self) -> None:
+        '''Stop timer'''
+        logging.debug(f'[{self.id_str}]Stop timer')
+        if self.tim:
+            self.tim.cancel()
+            self.tim = None
+
+    def __timeout(self) -> None:
+        '''timer expired handler'''
+        logging.debug(f'[{self.id_str}]Timer expired')
+        self.__timeout_cb(next(self.exp_count))
+
+    def close(self) -> None:
+        self.stop()
+        self.__timeout_cb = None

app/src/scheduler.py

@@ -0,0 +1,30 @@
+import logging
+import json
+from mqtt import Mqtt
+from aiocron import crontab
+from infos import ClrAtMidnight
+
+logger_mqtt = logging.getLogger('mqtt')
+
+
+class Schedule:
+    mqtt = None
+    count = 0
+
+    @classmethod
+    def start(cls) -> None:
+        '''Start the scheduler and schedule the tasks (cron jobs)'''
+        logging.debug("Scheduler init")
+        cls.mqtt = Mqtt(None)
+
+        crontab('0 0 * * *', func=cls.atmidnight, start=True)
+
+    @classmethod
+    async def atmidnight(cls) -> None:
+        '''Clear daily counters at midnight'''
+        logging.info("Clear daily counters at midnight")
+
+        for key, data in ClrAtMidnight.elm():
+            logger_mqtt.debug(f'{key}: {data}')
+            data_json = json.dumps(data)
+            await cls.mqtt.publish(f"{key}", data_json)

app/src/server.py

@@ -1,44 +1,143 @@
-import logging, asyncio, signal, functools, os
-from logging import config
-from async_stream import AsyncStream
+import logging
+import asyncio
+import signal
+import os
+from asyncio import StreamReader, StreamWriter
+from aiohttp import web
+from logging import config  # noqa F401
+from messages import Message
 from inverter import Inverter
+from gen3.inverter_g3 import InverterG3
+from gen3plus.inverter_g3p import InverterG3P
+from scheduler import Schedule
 from config import Config
-from mqtt import Mqtt
-from infos import Infos
+from modbus_tcp import ModbusTcp
 
-    
-async def handle_client(reader, writer):
+routes = web.RouteTableDef()
+proxy_is_up = False
+
+
+@routes.get('/')
+async def hello(request):
+    return web.Response(text="Hello, world")
+
+
+@routes.get('/-/ready')
+async def ready(request):
+    if proxy_is_up:
+        status = 200
+        text = 'Is ready'
+    else:
+        status = 503
+        text = 'Not ready'
+    return web.Response(status=status, text=text)
+
+
+@routes.get('/-/healthy')
+async def healthy(request):
+
+    if proxy_is_up:
+        # logging.info('web reqeust healthy()')
+        for stream in Message:
+            try:
+                res = stream.healthy()
+                if not res:
+                    return web.Response(status=503, text="I have a problem")
+            except Exception as err:
+                logging.info(f'Exception:{err}')
+
+    return web.Response(status=200, text="I'm fine")
+
+
+async def webserver(addr, port):
+    '''coro running our webserver'''
+    app = web.Application()
+    app.add_routes(routes)
+    runner = web.AppRunner(app)
+
+    await runner.setup()
+    site = web.TCPSite(runner, addr, port)
+    await site.start()
+    logging.info(f'HTTP server listen on port: {port}')
+
+    try:
+        # Normal interaction with aiohttp
+        while True:
+            await asyncio.sleep(3600)  # sleep forever
+    except asyncio.CancelledError:
+        logging.info('HTTP server cancelled')
+        await runner.cleanup()
+        logging.debug('HTTP cleanup done')
+
+
+async def handle_client(reader: StreamReader, writer: StreamWriter):
     '''Handles a new incoming connection and starts an async loop'''
 
     addr = writer.get_extra_info('peername')
-    await Inverter(reader, writer, addr).server_loop(addr) 
+    await InverterG3(reader, writer, addr).server_loop(addr)
 
 
-def handle_SIGTERM(loop):
+async def handle_client_v2(reader: StreamReader, writer: StreamWriter):
+    '''Handles a new incoming connection and starts an async loop'''
+
+    addr = writer.get_extra_info('peername')
+    await InverterG3P(reader, writer, addr).server_loop(addr)
+
+
+async def handle_shutdown(web_task):
     '''Close all TCP connections and stop the event loop'''
 
     logging.info('Shutdown due to SIGTERM')
+    global proxy_is_up
+    proxy_is_up = False
 
     #
-    # first, close all open TCP connections
+    # first, disc all open TCP connections gracefully
     #
-    for stream in AsyncStream:
+    for stream in Message:
+        stream.shutdown_started = True
+        try:
+            await asyncio.wait_for(stream.disc(), 2)
+        except Exception:
+            pass
+    logging.info('Proxy disconnecting done')
+
+    #
+    # second, close all open TCP connections
+    #
+    for stream in Message:
         stream.close()
 
+    await asyncio.sleep(0.1)  # give time for closing
+    logging.info('Proxy closing done')
+
     #
-    # at last, we stop the loop
+    # third, cancel the web server
     #
+    web_task.cancel()
+    await web_task
+
+    #
+    # now cancel all remaining (pending) tasks
+    #
+    pending = asyncio.all_tasks()
+    for task in pending:
+        task.cancel()
+
+    #
+    # at last, start a coro for stopping the loop
+    #
+    logging.debug("Stop event loop")
     loop.stop()
 
-    logging.info('Shutdown complete')
-    
 
 def get_log_level() -> int:
-    '''checks if LOG_LVL is set in the environment and returns the corresponding logging.LOG_LEVEL'''
+    '''checks if LOG_LVL is set in the environment and returns the
+    corresponding logging.LOG_LEVEL'''
     log_level = os.getenv('LOG_LVL', 'INFO')
-    if log_level== 'DEBUG':
+    if log_level == 'DEBUG':
         log_level = logging.DEBUG
-    elif log_level== 'WARN':
+    elif log_level == 'WARN':
         log_level = logging.WARNING
     else:
         log_level = logging.INFO
@@ -50,50 +149,60 @@ if __name__ == "__main__":
     # Setup our daily, rotating logger
     #
     serv_name = os.getenv('SERVICE_NAME', 'proxy')
-    version   = os.getenv('VERSION', 'unknown')
-    
+    version = os.getenv('VERSION', 'unknown')
+
     logging.config.fileConfig('logging.ini')
     logging.info(f'Server "{serv_name} - {version}" will be started')
-    
+
     # set lowest-severity for 'root', 'msg', 'conn' and 'data' logger
     log_level = get_log_level()
     logging.getLogger().setLevel(log_level)
     logging.getLogger('msg').setLevel(log_level)
     logging.getLogger('conn').setLevel(log_level)
     logging.getLogger('data').setLevel(log_level)
-    
-    # read config file
-    Config.read()    
+    logging.getLogger('tracer').setLevel(log_level)
+    logging.getLogger('asyncio').setLevel(log_level)
+    # logging.getLogger('mqtt').setLevel(log_level)
 
     loop = asyncio.new_event_loop()
     asyncio.set_event_loop(loop)
 
-    # call Mqtt singleton to establisch the connection to the mqtt broker
-    mqtt = Mqtt()
-    # initialize the proxy statistics
-    Infos.static_init()
-    
-    #
-    # Register some UNIX Signal handler for a gracefully server shutdown on Docker restart and stop
-    #  
-    for signame in ('SIGINT','SIGTERM'):
-        loop.add_signal_handler(getattr(signal, signame), functools.partial(handle_SIGTERM, loop))
+    # read config file
+    ConfigErr = Config.class_init()
+    if ConfigErr is not None:
+        logging.info(f'ConfigErr: {ConfigErr}')
+    Inverter.class_init()
+    Schedule.start()
+    mb_tcp = ModbusTcp(loop)
 
     #
-    # Create a task for our listening server. This must be a task! If we call start_server directly out
-    # of our main task, the eventloop will be blocked and we can't receive and handle the UNIX signals!
-    #   
+    # Create tasks for our listening servers. These must be tasks! If we call
+    # start_server directly out of our main task, the eventloop will be blocked
+    # and we can't receive and handle the UNIX signals!
+    #
     loop.create_task(asyncio.start_server(handle_client, '0.0.0.0', 5005))
-       
+    loop.create_task(asyncio.start_server(handle_client_v2, '0.0.0.0', 10000))
+    web_task = loop.create_task(webserver('0.0.0.0', 8127))
+
+    #
+    # Register some UNIX Signal handler for a gracefully server shutdown
+    # on Docker restart and stop
+    #
+    for signame in ('SIGINT', 'SIGTERM'):
+        loop.add_signal_handler(getattr(signal, signame),
+                                lambda loop=loop: asyncio.create_task(
+                                    handle_shutdown(web_task)))
+
+    loop.set_debug(log_level == logging.DEBUG)
     try:
+        if ConfigErr is None:
+            proxy_is_up = True
         loop.run_forever()
     except KeyboardInterrupt:
         pass
     finally:
-        logging.info ('Close MQTT Task')        
-        loop.run_until_complete(mqtt.close())
-        mqtt = None   # release the last reference to the singleton
-        logging.info ('Close event loop')        
+        logging.info("Event loop is stopped")
+        Inverter.class_close(loop)
+        logging.debug('Close event loop')
         loop.close()
-        logging.info (f'Finally, exit Server "{serv_name}"')
-    
+        logging.info(f'Finally, exit Server "{serv_name}"')

app/src/singleton.py

@@ -0,0 +1,9 @@
+class Singleton(type):
+    _instances = {}
+
+    def __call__(cls, *args, **kwargs):
+        # logger_mqtt.debug('singleton: __call__')
+        if cls not in cls._instances:
+            cls._instances[cls] = super(Singleton,
+                                        cls).__call__(*args, **kwargs)
+        return cls._instances[cls]

app/tests/test_config.py

@@ -0,0 +1,142 @@
+# test_with_pytest.py
+import tomllib
+from schema import SchemaMissingKeyError
+from app.src.config import Config
+
+class TstConfig(Config):
+
+    @classmethod
+    def set(cls, cnf):
+        cls.config = cnf
+
+    @classmethod
+    def _read_config_file(cls) -> dict:
+        return cls.config
+
+
+def test_empty_config():
+    cnf = {}
+    try:
+        Config.conf_schema.validate(cnf)
+        assert False
+    except SchemaMissingKeyError:
+        assert True
+
+def test_default_config():
+    with open("app/config/default_config.toml", "rb") as f:
+        cnf = tomllib.load(f)
+
+    try:
+        validated = Config.conf_schema.validate(cnf)
+        assert True
+    except:
+        assert False
+    assert validated == {'gen3plus': {'at_acl': {'mqtt': {'allow': ['AT+'], 'block': []}, 'tsun': {'allow': ['AT+Z', 'AT+UPURL', 'AT+SUPDATE'], 'block': []}}}, 'tsun': {'enabled': True, 'host': 'logger.talent-monitoring.com', 'port': 5005}, 'solarman': {'enabled': True, 'host': 'iot.talent-monitoring.com', 'port': 10000}, 'mqtt': {'host': 'mqtt', 'port': 1883, 'user': None, 'passwd': None}, 'ha': {'auto_conf_prefix': 'homeassistant', 'discovery_prefix': 'homeassistant', 'entity_prefix': 'tsun', 'proxy_node_id': 'proxy', 'proxy_unique_id': 'P170000000000001'}, 'inverters': {'allow_all': True, 'R170000000000001': {'node_id': '', 'modbus_polling': True, 'monitor_sn': 0, 'suggested_area': ''}, 'Y170000000000001': {'modbus_polling': True, 'monitor_sn': 2000000000, 'node_id': '', 'suggested_area': ''}}}
+
+def test_full_config():
+    cnf = {'tsun': {'enabled': True, 'host': 'logger.talent-monitoring.com', 'port': 5005}, 
+           'gen3plus': {'at_acl': {'mqtt': {'allow': ['AT+'], 'block': []},
+                                   'tsun': {'allow': ['AT+Z', 'AT+UPURL', 'AT+SUPDATE'], 'block': []}}},
+           'solarman': {'enabled': True, 'host': 'iot.talent-monitoring.com', 'port': 10000}, 
+           'mqtt': {'host': 'mqtt', 'port': 1883, 'user': '', 'passwd': ''}, 
+           'ha': {'auto_conf_prefix': 'homeassistant', 'discovery_prefix': 'homeassistant', 'entity_prefix': 'tsun', 'proxy_node_id': 'proxy', 'proxy_unique_id': 'P170000000000001'}, 
+           'inverters': {'allow_all': True, 
+                         'R170000000000001': {'modbus_polling': True, 'node_id': '', 'suggested_area': '', 'pv1': {'type': 'type1', 'manufacturer': 'man1'}, 'pv2': {'type': 'type2', 'manufacturer': 'man2'}, 'pv3': {'type': 'type3', 'manufacturer': 'man3'}}, 
+                         'Y170000000000001': {'modbus_polling': True, 'monitor_sn': 2000000000, 'node_id': '', 'suggested_area': ''}}}
+    try:
+        validated = Config.conf_schema.validate(cnf)
+        assert True
+    except:
+        assert False
+    assert validated == {'gen3plus': {'at_acl': {'mqtt': {'allow': ['AT+'], 'block': []}, 'tsun': {'allow': ['AT+Z', 'AT+UPURL', 'AT+SUPDATE'], 'block': []}}}, 'tsun': {'enabled': True, 'host': 'logger.talent-monitoring.com', 'port': 5005}, 'solarman': {'enabled': True, 'host': 'iot.talent-monitoring.com', 'port': 10000}, 'mqtt': {'host': 'mqtt', 'port': 1883, 'user': None, 'passwd': None}, 'ha': {'auto_conf_prefix': 'homeassistant', 'discovery_prefix': 'homeassistant', 'entity_prefix': 'tsun', 'proxy_node_id': 'proxy', 'proxy_unique_id': 'P170000000000001'}, 'inverters': {'allow_all': True, 'R170000000000001': {'node_id': '', 'modbus_polling': True, 'monitor_sn': 0, 'pv1': {'manufacturer': 'man1','type': 'type1'},'pv2': {'manufacturer': 'man2','type': 'type2'},'pv3': {'manufacturer': 'man3','type': 'type3'}, 'suggested_area': ''}, 'Y170000000000001': {'modbus_polling': True, 'monitor_sn': 2000000000, 'node_id': '', 'suggested_area': ''}}}
+
+def test_mininum_config():
+    cnf = {'tsun': {'enabled': True, 'host': 'logger.talent-monitoring.com', 'port': 5005}, 
+           'gen3plus': {'at_acl': {'mqtt': {'allow': ['AT+']},
+                                   'tsun': {'allow': ['AT+Z', 'AT+UPURL', 'AT+SUPDATE']}}},
+           'solarman': {'enabled': True, 'host': 'iot.talent-monitoring.com', 'port': 10000}, 
+           'mqtt': {'host': 'mqtt', 'port': 1883, 'user': '', 'passwd': ''}, 
+           'ha': {'auto_conf_prefix': 'homeassistant', 'discovery_prefix': 'homeassistant', 'entity_prefix': 'tsun', 'proxy_node_id': 'proxy', 'proxy_unique_id': 'P170000000000001'}, 
+           'inverters': {'allow_all': True,
+                         'R170000000000001': {}}
+    } 
+
+    try:
+        validated = Config.conf_schema.validate(cnf)
+        assert True
+    except:
+        assert False
+    assert validated == {'gen3plus': {'at_acl': {'mqtt': {'allow': ['AT+'], 'block': []}, 'tsun': {'allow': ['AT+Z', 'AT+UPURL', 'AT+SUPDATE'], 'block': []}}}, 'tsun': {'enabled': True, 'host': 'logger.talent-monitoring.com', 'port': 5005}, 'solarman': {'enabled': True, 'host': 'iot.talent-monitoring.com', 'port': 10000}, 'mqtt': {'host': 'mqtt', 'port': 1883, 'user': None, 'passwd': None}, 'ha': {'auto_conf_prefix': 'homeassistant', 'discovery_prefix': 'homeassistant', 'entity_prefix': 'tsun', 'proxy_node_id': 'proxy', 'proxy_unique_id': 'P170000000000001'}, 'inverters': {'allow_all': True, 'R170000000000001': {'node_id': '', 'modbus_polling': True, 'monitor_sn': 0, 'suggested_area': ''}}}
+
+def test_read_empty():
+    cnf = {}
+    TstConfig.set(cnf)
+    err = TstConfig.read('app/config/')
+    assert err == None
+    cnf = TstConfig.get()
+    assert cnf == {'gen3plus': {'at_acl': {'mqtt': {'allow': ['AT+'], 'block': []}, 'tsun': {'allow': ['AT+Z', 'AT+UPURL', 'AT+SUPDATE'], 'block': []}}}, 'tsun': {'enabled': True, 'host': 'logger.talent-monitoring.com', 'port': 5005}, 'solarman': {'enabled': True, 'host': 'iot.talent-monitoring.com', 'port': 10000}, 'mqtt': {'host': 'mqtt', 'port': 1883, 'user': None, 'passwd': None}, 'ha': {'auto_conf_prefix': 'homeassistant', 'discovery_prefix': 'homeassistant', 'entity_prefix': 'tsun', 'proxy_node_id': 'proxy', 'proxy_unique_id': 'P170000000000001'}, 'inverters': {'allow_all': True, 'R170000000000001': {'suggested_area': '', 'modbus_polling': True, 'monitor_sn': 0, 'node_id': ''}, 'Y170000000000001': {'modbus_polling': True, 'monitor_sn': 2000000000, 'suggested_area': '', 'node_id': ''}}}
+    
+    defcnf = TstConfig.def_config.get('solarman') 
+    assert defcnf == {'enabled': True, 'host': 'iot.talent-monitoring.com', 'port': 10000}
+    assert True == TstConfig.is_default('solarman')
+
+def test_no_file():
+    cnf = {}
+    TstConfig.set(cnf)
+    err = TstConfig.read('')
+    assert err == "Config.read: [Errno 2] No such file or directory: 'default_config.toml'"
+    cnf = TstConfig.get()
+    assert cnf == {}    
+    defcnf = TstConfig.def_config.get('solarman') 
+    assert defcnf == None
+
+def test_read_cnf1():
+    cnf = {'solarman' : {'enabled': False}}
+    TstConfig.set(cnf)
+    err = TstConfig.read('app/config/')
+    assert err == None
+    cnf = TstConfig.get()
+    assert cnf == {'gen3plus': {'at_acl': {'mqtt': {'allow': ['AT+'], 'block': []}, 'tsun': {'allow': ['AT+Z', 'AT+UPURL', 'AT+SUPDATE'], 'block': []}}}, 'tsun': {'enabled': True, 'host': 'logger.talent-monitoring.com', 'port': 5005}, 'solarman': {'enabled': False, 'host': 'iot.talent-monitoring.com', 'port': 10000}, 'mqtt': {'host': 'mqtt', 'port': 1883, 'user': None, 'passwd': None}, 'ha': {'auto_conf_prefix': 'homeassistant', 'discovery_prefix': 'homeassistant', 'entity_prefix': 'tsun', 'proxy_node_id': 'proxy', 'proxy_unique_id': 'P170000000000001'}, 'inverters': {'allow_all': True, 'R170000000000001': {'suggested_area': '', 'modbus_polling': True, 'monitor_sn': 0, 'node_id': ''}, 'Y170000000000001': {'modbus_polling': True, 'monitor_sn': 2000000000, 'suggested_area': '', 'node_id': ''}}}
+    cnf = TstConfig.get('solarman')
+    assert cnf == {'enabled': False, 'host': 'iot.talent-monitoring.com', 'port': 10000}    
+    defcnf = TstConfig.def_config.get('solarman') 
+    assert defcnf == {'enabled': True, 'host': 'iot.talent-monitoring.com', 'port': 10000}
+    assert False == TstConfig.is_default('solarman')
+                   
+def test_read_cnf2():
+    cnf = {'solarman' : {'enabled': 'FALSE'}}
+    TstConfig.set(cnf)
+    err = TstConfig.read('app/config/')
+    assert err == None
+    cnf = TstConfig.get()
+    assert cnf == {'gen3plus': {'at_acl': {'mqtt': {'allow': ['AT+'], 'block': []}, 'tsun': {'allow': ['AT+Z', 'AT+UPURL', 'AT+SUPDATE'], 'block': []}}}, 'tsun': {'enabled': True, 'host': 'logger.talent-monitoring.com', 'port': 5005}, 'solarman': {'enabled': True, 'host': 'iot.talent-monitoring.com', 'port': 10000}, 'mqtt': {'host': 'mqtt', 'port': 1883, 'user': None, 'passwd': None}, 'ha': {'auto_conf_prefix': 'homeassistant', 'discovery_prefix': 'homeassistant', 'entity_prefix': 'tsun', 'proxy_node_id': 'proxy', 'proxy_unique_id': 'P170000000000001'}, 'inverters': {'allow_all': True, 'R170000000000001': {'suggested_area': '', 'modbus_polling': True, 'monitor_sn': 0, 'node_id': ''}, 'Y170000000000001': {'modbus_polling': True, 'monitor_sn': 2000000000, 'suggested_area': '', 'node_id': ''}}}
+    assert True == TstConfig.is_default('solarman')
+
+def test_read_cnf3():
+    cnf = {'solarman' : {'port': 'FALSE'}}
+    TstConfig.set(cnf)
+    err = TstConfig.read('app/config/')
+    assert err == 'Config.read: Key \'solarman\' error:\nKey \'port\' error:\nint(\'FALSE\') raised ValueError("invalid literal for int() with base 10: \'FALSE\'")'
+    cnf = TstConfig.get()
+    assert cnf == {'solarman': {'port': 'FALSE'}}
+
+def test_read_cnf4():
+    cnf = {'solarman' : {'port': 5000}}
+    TstConfig.set(cnf)
+    err = TstConfig.read('app/config/')
+    assert err == None
+    cnf = TstConfig.get()
+    assert cnf == {'gen3plus': {'at_acl': {'mqtt': {'allow': ['AT+'], 'block': []}, 'tsun': {'allow': ['AT+Z', 'AT+UPURL', 'AT+SUPDATE'], 'block': []}}}, 'tsun': {'enabled': True, 'host': 'logger.talent-monitoring.com', 'port': 5005}, 'solarman': {'enabled': True, 'host': 'iot.talent-monitoring.com', 'port': 5000}, 'mqtt': {'host': 'mqtt', 'port': 1883, 'user': None, 'passwd': None}, 'ha': {'auto_conf_prefix': 'homeassistant', 'discovery_prefix': 'homeassistant', 'entity_prefix': 'tsun', 'proxy_node_id': 'proxy', 'proxy_unique_id': 'P170000000000001'}, 'inverters': {'allow_all': True, 'R170000000000001': {'suggested_area': '', 'modbus_polling': True, 'monitor_sn': 0, 'node_id': ''}, 'Y170000000000001': {'modbus_polling': True, 'monitor_sn': 2000000000, 'suggested_area': '', 'node_id': ''}}}
+    assert False == TstConfig.is_default('solarman')
+
+def test_read_cnf5():
+    cnf = {'solarman' : {'port': 1023}}
+    TstConfig.set(cnf)
+    err = TstConfig.read('app/config/')
+    assert err != None
+
+def test_read_cnf6():
+    cnf = {'solarman' : {'port': 65536}}
+    TstConfig.set(cnf)
+    err = TstConfig.read('app/config/')
+    assert err != None

app/tests/test_infos.py

@@ -1,233 +1,10 @@
 # test_with_pytest.py
-import pytest, json
+import pytest
+import json
+import logging
+from app.src.infos import Register, ClrAtMidnight
 from app.src.infos import Infos
 
-@pytest.fixture
-def ContrDataSeq(): # Get Time Request message
-    msg =   b'\x00\x00\x00\x15\x00\x09\x2b\xa8\x54\x10\x52\x53\x57\x5f\x34\x30\x30\x5f\x56\x31\x2e\x30\x30\x2e\x30\x36\x00\x09\x27\xc0\x54\x06\x52\x61\x79\x6d\x6f'
-    msg +=  b'\x6e\x00\x09\x2f\x90\x54\x0b\x52\x53\x57\x2d\x31\x2d\x31\x30\x30\x30\x31\x00\x09\x5a\x88\x54\x0f\x74\x2e\x72\x61\x79\x6d\x6f\x6e\x69\x6f\x74\x2e\x63\x6f\x6d\x00\x09\x5a\xec\x54'
-    msg +=  b'\x1c\x6c\x6f\x67\x67\x65\x72\x2e\x74\x61\x6c\x65\x6e\x74\x2d\x6d\x6f\x6e\x69\x74\x6f\x72\x69\x6e\x67\x2e\x63\x6f\x6d\x00\x0d\x00\x20\x49\x00\x00\x00\x01\x00\x0c\x35\x00\x49\x00'
-    msg +=  b'\x00\x00\x64\x00\x0c\x96\xa8\x49\x00\x00\x00\x1d\x00\x0c\x7f\x38\x49\x00\x00\x00\x01\x00\x0c\xfc\x38\x49\x00\x00\x00\x01\x00\x0c\xf8\x50\x49\x00\x00\x01\x2c\x00\x0c\x63\xe0\x49'
-    msg +=  b'\x00\x00\x00\x00\x00\x0c\x67\xc8\x49\x00\x00\x00\x00\x00\x0c\x50\x58\x49\x00\x00\x00\x01\x00\x09\x5e\x70\x49\x00\x00\x13\x8d\x00\x09\x5e\xd4\x49\x00\x00\x13\x8d\x00\x09\x5b\x50'
-    msg +=  b'\x49\x00\x00\x00\x02\x00\x0d\x04\x08\x49\x00\x00\x00\x00\x00\x07\xa1\x84\x49\x00\x00\x00\x01\x00\x0c\x50\x59\x49\x00\x00\x00\x4c\x00\x0d\x1f\x60\x49\x00\x00\x00\x00'
-    return msg
-
-@pytest.fixture
-def InvDataSeq(): # Data indication from the controller
-    msg =   b'\x00\x00\x00\x06\x00\x00\x00\x0a\x54\x08\x4d\x69\x63\x72\x6f\x69\x6e\x76\x00\x00\x00\x14\x54\x04\x54\x53\x55\x4e\x00\x00\x00\x1E\x54\x07\x56\x35\x2e\x30\x2e\x31\x31\x00\x00\x00\x28'
-    msg +=  b'\x54\x10\x54\x31\x37\x45\x37\x33\x30\x37\x30\x32\x31\x44\x30\x30\x36\x41\x00\x00\x00\x32\x54\x0a\x54\x53\x4f\x4c\x2d\x4d\x53\x36\x30\x30\x00\x00\x00\x3c\x54\x05\x41\x2c\x42\x2c\x43'
-    return msg
-
-@pytest.fixture
-def InvalidDataSeq(): # Data indication from the controller
-    msg =   b'\x00\x00\x00\x06\x00\x00\x00\x0a\x54\x08\x4d\x69\x63\x72\x6f\x69\x6e\x76\x00\x00\x00\x14\x64\x04\x54\x53\x55\x4e\x00\x00\x00\x1E\x54\x07\x56\x35\x2e\x30\x2e\x31\x31\x00\x00\x00\x28'
-    msg +=  b'\x54\x10\x54\x31\x37\x45\x37\x33\x30\x37\x30\x32\x31\x44\x30\x30\x36\x41\x00\x00\x00\x32\x54\x0a\x54\x53\x4f\x4c\x2d\x4d\x53\x36\x30\x30\x00\x00\x00\x3c\x54\x05\x41\x2c\x42\x2c\x43'
-    return msg
-
-@pytest.fixture
-def InvDataSeq2(): # Data indication from the controller
-    msg =   b'\x00\x00\x00\xa3\x00\x00\x00\x64\x53\x00\x01\x00\x00\x00\xc8\x53\x00\x02\x00\x00\x01\x2c\x53\x00\x00\x00\x00\x01\x90\x49\x00\x00\x00\x00\x00\x00\x01\x91\x53\x00\x00'
-    msg +=  b'\x00\x00\x01\x92\x53\x00\x00\x00\x00\x01\x93\x53\x00\x00\x00\x00\x01\x94\x53\x00\x00\x00\x00\x01\x95\x53\x00\x00\x00\x00\x01\x96\x53\x00\x00\x00\x00\x01\x97\x53\x00'
-    msg +=  b'\x00\x00\x00\x01\x98\x53\x00\x00\x00\x00\x01\x99\x53\x00\x00\x00\x00\x01\x9a\x53\x00\x00\x00\x00\x01\x9b\x53\x00\x00\x00\x00\x01\x9c\x53\x00\x00\x00\x00\x01\x9d\x53'
-    msg +=  b'\x00\x00\x00\x00\x01\x9e\x53\x00\x00\x00\x00\x01\x9f\x53\x00\x00\x00\x00\x01\xa0\x53\x00\x00\x00\x00\x01\xf4\x49\x00\x00\x00\x00\x00\x00\x01\xf5\x53\x00\x00\x00\x00'
-    msg +=  b'\x01\xf6\x53\x00\x00\x00\x00\x01\xf7\x53\x00\x00\x00\x00\x01\xf8\x53\x00\x00\x00\x00\x01\xf9\x53\x00\x00\x00\x00\x01\xfa\x53\x00\x00\x00\x00\x01\xfb\x53\x00\x00\x00'
-    msg +=  b'\x00\x01\xfc\x53\x00\x00\x00\x00\x01\xfd\x53\x00\x00\x00\x00\x01\xfe\x53\x00\x00\x00\x00\x01\xff\x53\x00\x00\x00\x00\x02\x00\x53\x00\x00\x00\x00\x02\x01\x53\x00\x00'
-    msg +=  b'\x00\x00\x02\x02\x53\x00\x00\x00\x00\x02\x03\x53\x00\x00\x00\x00\x02\x04\x53\x00\x00\x00\x00\x02\x58\x49\x00\x00\x00\x00\x00\x00\x02\x59\x53\x00\x00\x00\x00\x02\x5a'
-    msg +=  b'\x53\x00\x00\x00\x00\x02\x5b\x53\x00\x00\x00\x00\x02\x5c\x53\x00\x00\x00\x00\x02\x5d\x53\x00\x00\x00\x00\x02\x5e\x53\x00\x00\x00\x00\x02\x5f\x53\x00\x00\x00\x00\x02'
-    msg +=  b'\x60\x53\x00\x00\x00\x00\x02\x61\x53\x00\x00\x00\x00\x02\x62\x53\x00\x00\x00\x00\x02\x63\x53\x00\x00\x00\x00\x02\x64\x53\x00\x00\x00\x00\x02\x65\x53\x00\x00\x00\x00'
-    msg +=  b'\x02\x66\x53\x00\x00\x00\x00\x02\x67\x53\x00\x00\x00\x00\x02\x68\x53\x00\x00\x00\x00\x02\xbc\x49\x00\x00\x00\x00\x00\x00\x02\xbd\x53\x00\x00\x00\x00\x02\xbe\x53\x00'
-    msg +=  b'\x00\x00\x00\x02\xbf\x53\x00\x00\x00\x00\x02\xc0\x53\x00\x00\x00\x00\x02\xc1\x53\x00\x00\x00\x00\x02\xc2\x53\x00\x00\x00\x00\x02\xc3\x53\x00\x00\x00\x00\x02\xc4\x53'
-    msg +=  b'\x00\x00\x00\x00\x02\xc5\x53\x00\x00\x00\x00\x02\xc6\x53\x00\x00\x00\x00\x02\xc7\x53\x00\x00\x00\x00\x02\xc8\x53\x00\x00\x00\x00\x02\xc9\x53\x00\x00\x00\x00\x02\xca'
-    msg +=  b'\x53\x00\x00\x00\x00\x02\xcb\x53\x00\x00\x00\x00\x02\xcc\x53\x00\x00\x00\x00\x03\x20\x53\x00\x00\x00\x00\x03\x84\x53\x50\x11\x00\x00\x03\xe8\x46\x43\x61\x66\x66\x00'
-    msg +=  b'\x00\x04\x4c\x46\x3e\xeb\x85\x1f\x00\x00\x04\xb0\x46\x42\x48\x14\x7b\x00\x00\x05\x14\x53\x00\x17\x00\x00\x05\x78\x53\x00\x00\x00\x00\x05\xdc\x53\x02\x58\x00\x00\x06'
-    msg +=  b'\x40\x46\x42\xd3\x66\x66\x00\x00\x06\xa4\x46\x42\x06\x66\x66\x00\x00\x07\x08\x46\x3f\xf4\x7a\xe1\x00\x00\x07\x6c\x46\x42\x81\x00\x00\x00\x00\x07\xd0\x46\x42\x06\x00'
-    msg +=  b'\x00\x00\x00\x08\x34\x46\x3f\xae\x14\x7b\x00\x00\x08\x98\x46\x42\x36\xcc\xcd\x00\x00\x08\xfc\x46\x00\x00\x00\x00\x00\x00\x09\x60\x46\x00\x00\x00\x00\x00\x00\x09\xc4'
-    msg +=  b'\x46\x00\x00\x00\x00\x00\x00\x0a\x28\x46\x00\x00\x00\x00\x00\x00\x0a\x8c\x46\x00\x00\x00\x00\x00\x00\x0a\xf0\x46\x00\x00\x00\x00\x00\x00\x0b\x54\x46\x3f\xd9\x99\x9a'
-    msg +=  b'\x00\x00\x0b\xb8\x46\x41\x8a\xe1\x48\x00\x00\x0c\x1c\x46\x3f\x8a\x3d\x71\x00\x00\x0c\x80\x46\x41\x1b\xd7\x0a\x00\x00\x0c\xe4\x46\x3f\x1e\xb8\x52\x00\x00\x0d\x48\x46'
-    msg +=  b'\x40\xf3\xd7\x0a\x00\x00\x0d\xac\x46\x00\x00\x00\x00\x00\x00\x0e\x10\x46\x00\x00\x00\x00\x00\x00\x0e\x74\x46\x00\x00\x00\x00\x00\x00\x0e\xd8\x46\x00\x00\x00\x00\x00'
-    msg +=  b'\x00\x0f\x3c\x53\x00\x00\x00\x00\x0f\xa0\x53\x00\x00\x00\x00\x10\x04\x53\x55\xaa\x00\x00\x10\x68\x53\x00\x00\x00\x00\x10\xcc\x53\x00\x00\x00\x00\x11\x30\x53\x00\x00'
-    msg +=  b'\x00\x00\x11\x94\x53\x00\x00\x00\x00\x11\xf8\x53\xff\xff\x00\x00\x12\x5c\x53\xff\xff\x00\x00\x12\xc0\x53\x00\x02\x00\x00\x13\x24\x53\xff\xff\x00\x00\x13\x88\x53\xff'
-    msg +=  b'\xff\x00\x00\x13\xec\x53\xff\xff\x00\x00\x14\x50\x53\xff\xff\x00\x00\x14\xb4\x53\xff\xff\x00\x00\x15\x18\x53\xff\xff\x00\x00\x15\x7c\x53\x00\x00\x00\x00\x27\x10\x53'
-    msg +=  b'\x00\x02\x00\x00\x27\x74\x53\x00\x3c\x00\x00\x27\xd8\x53\x00\x68\x00\x00\x28\x3c\x53\x05\x00\x00\x00\x28\xa0\x46\x43\x79\x00\x00\x00\x00\x29\x04\x46\x43\x48\x00\x00'
-    msg +=  b'\x00\x00\x29\x68\x46\x42\x48\x33\x33\x00\x00\x29\xcc\x46\x42\x3e\x3d\x71\x00\x00\x2a\x30\x53\x00\x01\x00\x00\x2a\x94\x46\x43\x37\x00\x00\x00\x00\x2a\xf8\x46\x42\xce'
-    msg +=  b'\x00\x00\x00\x00\x2b\x5c\x53\x00\x96\x00\x00\x2b\xc0\x53\x00\x10\x00\x00\x2c\x24\x46\x43\x90\x00\x00\x00\x00\x2c\x88\x46\x43\x95\x00\x00\x00\x00\x2c\xec\x53\x00\x06'
-    msg +=  b'\x00\x00\x2d\x50\x53\x00\x06\x00\x00\x2d\xb4\x46\x43\x7d\x00\x00\x00\x00\x2e\x18\x46\x42\x3d\xeb\x85\x00\x00\x2e\x7c\x46\x42\x3d\xeb\x85\x00\x00\x2e\xe0\x53\x00\x03'
-    msg +=  b'\x00\x00\x2f\x44\x53\x00\x03\x00\x00\x2f\xa8\x46\x42\x4d\xeb\x85\x00\x00\x30\x0c\x46\x42\x4d\xeb\x85\x00\x00\x30\x70\x53\x00\x03\x00\x00\x30\xd4\x53\x00\x03\x00\x00'
-    msg +=  b'\x31\x38\x46\x42\x08\x00\x00\x00\x00\x31\x9c\x53\x00\x05\x00\x00\x32\x00\x53\x04\x00\x00\x00\x32\x64\x53\x00\x01\x00\x00\x32\xc8\x53\x13\x9c\x00\x00\x33\x2c\x53\x0f'
-    msg +=  b'\xa0\x00\x00\x33\x90\x53\x00\x4f\x00\x00\x33\xf4\x53\x00\x66\x00\x00\x34\x58\x53\x03\xe8\x00\x00\x34\xbc\x53\x04\x00\x00\x00\x35\x20\x53\x00\x00\x00\x00\x35\x84\x53'
-    msg +=  b'\x00\x00\x00\x00\x35\xe8\x53\x00\x00\x00\x00\x36\x4c\x53\x00\x00\x00\x01\x38\x80\x53\x00\x02\x00\x01\x38\x81\x53\x00\x01\x00\x01\x38\x82\x53\x00\x01\x00\x01\x38\x83'
-    msg +=  b'\x53\x00\x00'  
-    return msg
-
-@pytest.fixture
-def InvDataSeq2_Zero(): # Data indication from the controller
-    msg =   b'\x00\x00\x00\xa3\x00\x00\x00\x64\x53\x00\x01\x00\x00\x00\xc8\x53\x00\x02\x00\x00\x01\x2c\x53\x00\x00\x00\x00\x01\x90\x49\x00\x00\x00\x00\x00\x00\x01\x91\x53\x00\x00'
-    msg +=  b'\x00\x00\x01\x92\x53\x00\x00\x00\x00\x01\x93\x53\x00\x00\x00\x00\x01\x94\x53\x00\x00\x00\x00\x01\x95\x53\x00\x00\x00\x00\x01\x96\x53\x00\x00\x00\x00\x01\x97\x53\x00'
-    msg +=  b'\x00\x00\x00\x01\x98\x53\x00\x00\x00\x00\x01\x99\x53\x00\x00\x00\x00\x01\x9a\x53\x00\x00\x00\x00\x01\x9b\x53\x00\x00\x00\x00\x01\x9c\x53\x00\x00\x00\x00\x01\x9d\x53'
-    msg +=  b'\x00\x00\x00\x00\x01\x9e\x53\x00\x00\x00\x00\x01\x9f\x53\x00\x00\x00\x00\x01\xa0\x53\x00\x00\x00\x00\x01\xf4\x49\x00\x00\x00\x00\x00\x00\x01\xf5\x53\x00\x00\x00\x00'
-    msg +=  b'\x01\xf6\x53\x00\x00\x00\x00\x01\xf7\x53\x00\x00\x00\x00\x01\xf8\x53\x00\x00\x00\x00\x01\xf9\x53\x00\x00\x00\x00\x01\xfa\x53\x00\x00\x00\x00\x01\xfb\x53\x00\x00\x00'
-    msg +=  b'\x00\x01\xfc\x53\x00\x00\x00\x00\x01\xfd\x53\x00\x00\x00\x00\x01\xfe\x53\x00\x00\x00\x00\x01\xff\x53\x00\x00\x00\x00\x02\x00\x53\x00\x00\x00\x00\x02\x01\x53\x00\x00'
-    msg +=  b'\x00\x00\x02\x02\x53\x00\x00\x00\x00\x02\x03\x53\x00\x00\x00\x00\x02\x04\x53\x00\x00\x00\x00\x02\x58\x49\x00\x00\x00\x00\x00\x00\x02\x59\x53\x00\x00\x00\x00\x02\x5a'
-    msg +=  b'\x53\x00\x00\x00\x00\x02\x5b\x53\x00\x00\x00\x00\x02\x5c\x53\x00\x00\x00\x00\x02\x5d\x53\x00\x00\x00\x00\x02\x5e\x53\x00\x00\x00\x00\x02\x5f\x53\x00\x00\x00\x00\x02'
-    msg +=  b'\x60\x53\x00\x00\x00\x00\x02\x61\x53\x00\x00\x00\x00\x02\x62\x53\x00\x00\x00\x00\x02\x63\x53\x00\x00\x00\x00\x02\x64\x53\x00\x00\x00\x00\x02\x65\x53\x00\x00\x00\x00'
-    msg +=  b'\x02\x66\x53\x00\x00\x00\x00\x02\x67\x53\x00\x00\x00\x00\x02\x68\x53\x00\x00\x00\x00\x02\xbc\x49\x00\x00\x00\x00\x00\x00\x02\xbd\x53\x00\x00\x00\x00\x02\xbe\x53\x00'
-    msg +=  b'\x00\x00\x00\x02\xbf\x53\x00\x00\x00\x00\x02\xc0\x53\x00\x00\x00\x00\x02\xc1\x53\x00\x00\x00\x00\x02\xc2\x53\x00\x00\x00\x00\x02\xc3\x53\x00\x00\x00\x00\x02\xc4\x53'
-    msg +=  b'\x00\x00\x00\x00\x02\xc5\x53\x00\x00\x00\x00\x02\xc6\x53\x00\x00\x00\x00\x02\xc7\x53\x00\x00\x00\x00\x02\xc8\x53\x00\x00\x00\x00\x02\xc9\x53\x00\x00\x00\x00\x02\xca'
-    msg +=  b'\x53\x00\x00\x00\x00\x02\xcb\x53\x00\x00\x00\x00\x02\xcc\x53\x00\x00\x00\x00\x03\x20\x53\x00\x00\x00\x00\x03\x84\x53\x50\x11\x00\x00\x03\xe8\x46\x43\x61\x66\x66\x00'
-    msg +=  b'\x00\x04\x4c\x46\x3e\xeb\x85\x1f\x00\x00\x04\xb0\x46\x42\x48\x14\x7b\x00\x00\x05\x14\x53\x00\x17\x00\x00\x05\x78\x53\x00\x00\x00\x00\x05\xdc\x53\x02\x58\x00\x00\x06'
-    msg +=  b'\x40\x46\x42\xd3\x66\x66\x00\x00\x06\xa4\x46\x42\x06\x66\x66\x00\x00\x07\x08\x46\x3f\xf4\x7a\xe1\x00\x00\x07\x6c\x46\x00\x00\x00\x00\x00\x00\x07\xd0\x46\x42\x06\x00'
-    msg +=  b'\x00\x00\x00\x08\x34\x46\x3f\xae\x14\x7b\x00\x00\x08\x98\x46\x00\x00\x00\x00\x00\x00\x08\xfc\x46\x00\x00\x00\x00\x00\x00\x09\x60\x46\x00\x00\x00\x00\x00\x00\x09\xc4'
-    msg +=  b'\x46\x00\x00\x00\x00\x00\x00\x0a\x28\x46\x00\x00\x00\x00\x00\x00\x0a\x8c\x46\x00\x00\x00\x00\x00\x00\x0a\xf0\x46\x00\x00\x00\x00\x00\x00\x0b\x54\x46\x00\x00\x00\x00'
-    msg +=  b'\x00\x00\x0b\xb8\x46\x00\x00\x00\x00\x00\x00\x0c\x1c\x46\x00\x00\x00\x00\x00\x00\x0c\x80\x46\x00\x00\x00\x00\x00\x00\x0c\xe4\x46\x00\x00\x00\x00\x00\x00\x0d\x48\x46'
-    msg +=  b'\x00\x00\x00\x00\x00\x00\x0d\xac\x46\x00\x00\x00\x00\x00\x00\x0e\x10\x46\x00\x00\x00\x00\x00\x00\x0e\x74\x46\x00\x00\x00\x00\x00\x00\x0e\xd8\x46\x00\x00\x00\x00\x00'
-    msg +=  b'\x00\x0f\x3c\x53\x00\x00\x00\x00\x0f\xa0\x53\x00\x00\x00\x00\x10\x04\x53\x55\xaa\x00\x00\x10\x68\x53\x00\x00\x00\x00\x10\xcc\x53\x00\x00\x00\x00\x11\x30\x53\x00\x00'
-    msg +=  b'\x00\x00\x11\x94\x53\x00\x00\x00\x00\x11\xf8\x53\xff\xff\x00\x00\x12\x5c\x53\xff\xff\x00\x00\x12\xc0\x53\x00\x02\x00\x00\x13\x24\x53\xff\xff\x00\x00\x13\x88\x53\xff'
-    msg +=  b'\xff\x00\x00\x13\xec\x53\xff\xff\x00\x00\x14\x50\x53\xff\xff\x00\x00\x14\xb4\x53\xff\xff\x00\x00\x15\x18\x53\xff\xff\x00\x00\x15\x7c\x53\x00\x00\x00\x00\x27\x10\x53'
-    msg +=  b'\x00\x02\x00\x00\x27\x74\x53\x00\x3c\x00\x00\x27\xd8\x53\x00\x68\x00\x00\x28\x3c\x53\x05\x00\x00\x00\x28\xa0\x46\x43\x79\x00\x00\x00\x00\x29\x04\x46\x43\x48\x00\x00'
-    msg +=  b'\x00\x00\x29\x68\x46\x42\x48\x33\x33\x00\x00\x29\xcc\x46\x42\x3e\x3d\x71\x00\x00\x2a\x30\x53\x00\x01\x00\x00\x2a\x94\x46\x43\x37\x00\x00\x00\x00\x2a\xf8\x46\x42\xce'
-    msg +=  b'\x00\x00\x00\x00\x2b\x5c\x53\x00\x96\x00\x00\x2b\xc0\x53\x00\x10\x00\x00\x2c\x24\x46\x43\x90\x00\x00\x00\x00\x2c\x88\x46\x43\x95\x00\x00\x00\x00\x2c\xec\x53\x00\x06'
-    msg +=  b'\x00\x00\x2d\x50\x53\x00\x06\x00\x00\x2d\xb4\x46\x43\x7d\x00\x00\x00\x00\x2e\x18\x46\x42\x3d\xeb\x85\x00\x00\x2e\x7c\x46\x42\x3d\xeb\x85\x00\x00\x2e\xe0\x53\x00\x03'
-    msg +=  b'\x00\x00\x2f\x44\x53\x00\x03\x00\x00\x2f\xa8\x46\x42\x4d\xeb\x85\x00\x00\x30\x0c\x46\x42\x4d\xeb\x85\x00\x00\x30\x70\x53\x00\x03\x00\x00\x30\xd4\x53\x00\x03\x00\x00'
-    msg +=  b'\x31\x38\x46\x42\x08\x00\x00\x00\x00\x31\x9c\x53\x00\x05\x00\x00\x32\x00\x53\x04\x00\x00\x00\x32\x64\x53\x00\x01\x00\x00\x32\xc8\x53\x13\x9c\x00\x00\x33\x2c\x53\x0f'
-    msg +=  b'\xa0\x00\x00\x33\x90\x53\x00\x4f\x00\x00\x33\xf4\x53\x00\x66\x00\x00\x34\x58\x53\x03\xe8\x00\x00\x34\xbc\x53\x04\x00\x00\x00\x35\x20\x53\x00\x00\x00\x00\x35\x84\x53'
-    msg +=  b'\x00\x00\x00\x00\x35\xe8\x53\x00\x00\x00\x00\x36\x4c\x53\x00\x00\x00\x01\x38\x80\x53\x00\x02\x00\x01\x38\x81\x53\x00\x01\x00\x01\x38\x82\x53\x00\x01\x00\x01\x38\x83'
-    msg +=  b'\x53\x00\x00'  
-    return msg
-
-
-def test_parse_control(ContrDataSeq):
-    i = Infos()
-    for key, result in i.parse (ContrDataSeq):
-        pass
-
-    assert json.dumps(i.db) == json.dumps(
-{"collector": {"Collector_Fw_Version": "RSW_400_V1.00.06", "Chip_Type": "Raymon", "Chip_Model": "RSW-1-10001", "Trace_URL": "t.raymoniot.com", "Logger_URL": "logger.talent-monitoring.com", "No_Inputs": 2}, "controller": {"Signal_Strength": 100, "Power_On_Time": 29, "Data_Up_Interval": 300}})        
-
-def test_parse_inverter(InvDataSeq):
-    i = Infos()
-    for key, result in i.parse (InvDataSeq):
-        pass
-
-    assert json.dumps(i.db) == json.dumps(
-{"inverter": {"Product_Name": "Microinv", "Manufacturer": "TSUN", "Version": "V5.0.11", "Serial_Number": "T17E7307021D006A", "Equipment_Model": "TSOL-MS600"}})
-
-def test_parse_cont_and_invert(ContrDataSeq, InvDataSeq):
-    i = Infos()
-    for key, result in i.parse (ContrDataSeq):
-        pass
-
-    for key, result in i.parse (InvDataSeq):
-        pass
-
-    assert json.dumps(i.db) == json.dumps(
-    {
-"collector": {"Collector_Fw_Version": "RSW_400_V1.00.06", "Chip_Type": "Raymon", "Chip_Model": "RSW-1-10001", "Trace_URL": "t.raymoniot.com", "Logger_URL": "logger.talent-monitoring.com", "No_Inputs": 2}, "controller": {"Signal_Strength": 100, "Power_On_Time": 29, "Data_Up_Interval": 300},
-"inverter": {"Product_Name": "Microinv", "Manufacturer": "TSUN", "Version": "V5.0.11", "Serial_Number": "T17E7307021D006A", "Equipment_Model": "TSOL-MS600"}})
-
-
-def test_build_ha_conf1(ContrDataSeq):
-    i = Infos()
-    i.static_init()                # initialize counter
-
-    tests = 0
-    for d_json, comp, node_id, id in i.ha_confs(ha_prfx="tsun/", inv_snr='123', inv_node_id="garagendach/",proxy_node_id = 'proxy/', proxy_unique_id = '456'):
-
-        if id == 'out_power_123':
-            assert comp == 'sensor'
-            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/grid", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "out_power_123", "val_tpl": "{{value_json['Output_Power'] | float}}", "unit_of_meas": "W", "dev": {"name": "Micro Inverter", "sa": "Micro Inverter", "via_device": "controller_123", "ids": ["inverter_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
-            tests +=1
-
-        elif id == 'daily_gen_123':
-            assert comp == 'sensor'
-            assert  d_json == json.dumps({"name": "Daily Generation", "stat_t": "tsun/garagendach/total", "dev_cla": "energy", "stat_cla": "total_increasing", "uniq_id": "daily_gen_123", "val_tpl": "{{value_json['Daily_Generation'] | float}}", "unit_of_meas": "kWh", "ic": "mdi:solar-power-variant", "dev": {"name": "Micro Inverter", "sa": "Micro Inverter", "via_device": "controller_123", "ids": ["inverter_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
-            tests +=1
-
-        elif id == 'power_pv1_123':
-            assert comp == 'sensor'
-            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/input", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "power_pv1_123", "val_tpl": "{{ (value_json['pv1']['Power'] | float)}}", "unit_of_meas": "W", "dev": {"name": "Module PV1", "sa": "Module PV1", "via_device": "inverter_123", "ids": ["input_pv1_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
-            tests +=1
-
-        elif id == 'power_pv2_123':
-            assert False    # if we haven't received and parsed a control data msg, we don't know the number of inputs. In this case we only register the first one!!
-
-
-        elif id == 'signal_123':
-            assert comp == 'sensor'
-            assert  d_json == json.dumps({"name": "Signal Strength", "stat_t": "tsun/garagendach/controller", "dev_cla": None, "stat_cla": "measurement", "uniq_id": "signal_123", "val_tpl": "{{value_json[\'Signal_Strength\'] | int}}", "unit_of_meas": "%", "ic": "mdi:wifi", "dev": {"name": "Controller", "sa": "Controller", "via_device": "proxy", "ids": ["controller_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
-            tests +=1
-        elif id == 'inv_count_456':
-            assert comp == 'sensor'
-            assert  d_json == json.dumps({"name": "Active Inverter Connections", "stat_t": "tsun/proxy/proxy", "dev_cla": None, "stat_cla": None, "uniq_id": "inv_count_456", "val_tpl": "{{value_json['Inverter_Cnt'] | int}}", "ic": "mdi:counter", "dev": {"name": "Proxy", "sa": "Proxy", "mdl": "proxy", "mf": "Stefan Allius", "sw": "unknown", "ids": ["proxy"]}, "o": {"name": "proxy", "sw": "unknown"}})
-            tests +=1
-
-    assert tests==5
-
-def test_build_ha_conf2(ContrDataSeq, InvDataSeq):
-    i = Infos()
-    for key, result in i.parse (ContrDataSeq):
-        pass
-
-    for key, result in i.parse (InvDataSeq):
-        pass
-
-    tests = 0
-    for d_json, comp, node_id, id in i.ha_confs(ha_prfx="tsun/", inv_snr='123', inv_node_id="garagendach/",proxy_node_id = 'proxy/', proxy_unique_id = '456', sug_area = 'roof'):
-
-        if id == 'out_power_123':
-            assert comp == 'sensor'
-            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/grid", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "out_power_123", "val_tpl": "{{value_json['Output_Power'] | float}}", "unit_of_meas": "W", "dev": {"name": "Micro Inverter - roof", "sa": "Micro Inverter - roof", "via_device": "controller_123", "mdl": "TSOL-MS600", "mf": "TSUN", "sw": "V5.0.11", "ids": ["inverter_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
-            tests +=1
-
-        if id == 'daily_gen_123':
-            assert comp == 'sensor'
-            assert  d_json == json.dumps({"name": "Daily Generation", "stat_t": "tsun/garagendach/total", "dev_cla": "energy", "stat_cla": "total_increasing", "uniq_id": "daily_gen_123", "val_tpl": "{{value_json['Daily_Generation'] | float}}", "unit_of_meas": "kWh", "ic": "mdi:solar-power-variant", "dev": {"name": "Micro Inverter - roof", "sa": "Micro Inverter - roof", "via_device": "controller_123", "mdl": "TSOL-MS600", "mf": "TSUN", "sw": "V5.0.11", "ids": ["inverter_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
-            tests +=1
-
-        elif id == 'power_pv1_123':
-            assert comp == 'sensor'
-            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/input", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "power_pv1_123", "val_tpl": "{{ (value_json['pv1']['Power'] | float)}}", "unit_of_meas": "W", "dev": {"name": "Module PV1 - roof", "sa": "Module PV1 - roof", "via_device": "inverter_123", "ids": ["input_pv1_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
-            tests +=1
-
-        elif id == 'power_pv2_123':
-            assert comp == 'sensor'
-            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/input", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "power_pv2_123", "val_tpl": "{{ (value_json['pv2']['Power'] | float)}}", "unit_of_meas": "W", "dev": {"name": "Module PV2 - roof", "sa": "Module PV2 - roof", "via_device": "inverter_123", "ids": ["input_pv2_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
-            tests +=1
-
-        elif id == 'signal_123':
-            assert comp == 'sensor'
-            assert  d_json == json.dumps({"name": "Signal Strength", "stat_t": "tsun/garagendach/controller", "dev_cla": None, "stat_cla": "measurement", "uniq_id": "signal_123", "val_tpl": "{{value_json[\'Signal_Strength\'] | int}}", "unit_of_meas": "%", "ic": "mdi:wifi", "dev": {"name": "Controller - roof", "sa": "Controller - roof", "via_device": "proxy", "mdl": "RSW-1-10001", "mf": "Raymon", "sw": "RSW_400_V1.00.06", "ids": ["controller_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
-            tests +=1
-    assert tests==5
-
-def test_must_incr_total(InvDataSeq2, InvDataSeq2_Zero):
-    i = Infos()
-    tests = 0
-    for key, update in i.parse (InvDataSeq2):
-        if key == 'total':
-            assert update == True
-            tests +=1   
-            pass
-        elif key == 'input':
-            assert update == True
-            tests +=1
-
-    assert tests==22
-    assert json.dumps(i.db['total']) == json.dumps({'Daily_Generation': 1.7, 'Total_Generation': 17.36})
-    assert json.dumps(i.db['input']) == json.dumps({"pv1": {"Voltage": 33.6, "Current": 1.91, "Power": 64.5, "Daily_Generation": 1.08, "Total_Generation": 9.74}, "pv2": {"Voltage": 33.5, "Current": 1.36, "Power": 45.7, "Daily_Generation": 0.62, "Total_Generation": 7.62}, "pv3": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0, "Daily_Generation": 0.0, "Total_Generation": 0.0}, "pv4": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0, "Daily_Generation": 0.0, "Total_Generation": 0.0}})
-
-    tests = 0
-    for key, update in i.parse (InvDataSeq2_Zero):
-        if key == 'total':
-            tests +=1   
-            pass
-        elif key == 'input':
-            tests +=1
-
-    assert tests==22
-    assert json.dumps(i.db['total']) == json.dumps({'Daily_Generation': 1.7, 'Total_Generation': 17.36})
-    assert json.dumps(i.db['input']) == json.dumps({"pv1": {"Voltage": 33.6, "Current": 1.91, "Power": 0.0, "Daily_Generation": 1.08, "Total_Generation": 9.74}, "pv2": {"Voltage": 33.5, "Current": 1.36, "Power": 0.0, "Daily_Generation": 0.62, "Total_Generation": 7.62}, "pv3": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0, "Daily_Generation": 0.0, "Total_Generation": 0.0}, "pv4": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0, "Daily_Generation": 0.0, "Total_Generation": 0.0}})
-
-
 def test_statistic_counter():
     i = Infos()
     val = i.dev_value("Test-String")
@@ -236,22 +13,22 @@ def test_statistic_counter():
     val = i.dev_value(0xffffffff)  # invalid addr
     assert val == None
     
-    val = i.dev_value(0xffffff00)  # valid addr but not initiliazed
+    val = i.dev_value(Register.INVERTER_CNT)  # valid addr but not initiliazed
     assert val == None or val == 0
 
     i.static_init()                # initialize counter
-    assert json.dumps(i.stat) == json.dumps({"proxy": {"Inverter_Cnt": 0, "Unknown_SNR": 0, "Unknown_Msg": 0, "Invalid_Data_Type": 0, "Internal_Error": 0}})
+    assert json.dumps(i.stat) == json.dumps({"proxy": {"Inverter_Cnt": 0, "Unknown_SNR": 0, "Unknown_Msg": 0, "Invalid_Data_Type": 0, "Internal_Error": 0,"Unknown_Ctrl": 0, "OTA_Start_Msg": 0, "SW_Exception": 0, "Invalid_Msg_Format": 0, "AT_Command": 0, "AT_Command_Blocked": 0, "Modbus_Command": 0}})
                                             
-    val = i.dev_value(0xffffff00)  # valid and initiliazed addr
+    val = i.dev_value(Register.INVERTER_CNT)  # valid and initiliazed addr
     assert val == 0
 
     i.inc_counter('Inverter_Cnt')
-    assert json.dumps(i.stat) == json.dumps({"proxy": {"Inverter_Cnt": 1, "Unknown_SNR": 0, "Unknown_Msg": 0, "Invalid_Data_Type": 0, "Internal_Error": 0}})
-    val = i.dev_value(0xffffff00)
+    assert json.dumps(i.stat) == json.dumps({"proxy": {"Inverter_Cnt": 1, "Unknown_SNR": 0, "Unknown_Msg": 0, "Invalid_Data_Type": 0, "Internal_Error": 0,"Unknown_Ctrl": 0, "OTA_Start_Msg": 0, "SW_Exception": 0, "Invalid_Msg_Format": 0, "AT_Command": 0, "AT_Command_Blocked": 0, "Modbus_Command": 0}})
+    val = i.dev_value(Register.INVERTER_CNT)
     assert val == 1
 
     i.dec_counter('Inverter_Cnt')
-    val = i.dev_value(0xffffff00)
+    val = i.dev_value(Register.INVERTER_CNT)
     assert val == 0
 
 def test_dep_rules():
@@ -265,94 +42,217 @@ def test_dep_rules():
     assert res == True
 
     i.inc_counter('Inverter_Cnt')    # is 1
-    val = i.dev_value(0xffffff00)
+    val = i.dev_value(Register.INVERTER_CNT)
     assert val == 1
-    res = i.ignore_this_device({'reg':0xffffff00})
+    res = i.ignore_this_device({'reg': Register.INVERTER_CNT})
     assert res == True
-    res = i.ignore_this_device({'reg':0xffffff00, 'less_eq': 2})
+    res = i.ignore_this_device({'reg': Register.INVERTER_CNT, 'less_eq': 2})
     assert res == False
-    res = i.ignore_this_device({'reg':0xffffff00, 'gte': 2})
+    res = i.ignore_this_device({'reg': Register.INVERTER_CNT, 'gte': 2})
     assert res == True
 
     i.inc_counter('Inverter_Cnt')   # is 2
-    res = i.ignore_this_device({'reg':0xffffff00, 'less_eq': 2})
+    res = i.ignore_this_device({'reg': Register.INVERTER_CNT, 'less_eq': 2})
     assert res == False
-    res = i.ignore_this_device({'reg':0xffffff00, 'gte': 2})
+    res = i.ignore_this_device({'reg': Register.INVERTER_CNT, 'gte': 2})
     assert res == False
 
-    i.inc_counter('Inverter_Cnt')   is 3
-    res = i.ignore_this_device({'reg':0xffffff00, 'less_eq': 2})
+    i.inc_counter('Inverter_Cnt')   # is 3
+    res = i.ignore_this_device({'reg': Register.INVERTER_CNT, 'less_eq': 2})
     assert res == True
-    res = i.ignore_this_device({'reg':0xffffff00, 'gte': 2})
+    res = i.ignore_this_device({'reg': Register.INVERTER_CNT, 'gte': 2})
     assert res == False
 
 def test_table_definition():
     i = Infos()
     i.static_init()                # initialize counter
 
-    val = i.dev_value(0xffffff04)  # check internal error counter
+    val = i.dev_value(Register.INTERNAL_ERROR)  # check internal error counter
     assert val == 0
 
-    for d_json, comp, node_id, id in i.ha_confs(ha_prfx="tsun/", inv_snr='123', inv_node_id="garagendach/",proxy_node_id = 'proxy/', proxy_unique_id = '456', sug_area = 'roof'):
+    # for d_json, comp, node_id, id in i.ha_confs(ha_prfx="tsun/", node_id="garagendach/", snr='123', sug_area = 'roof'):
+    #    pass
+    for reg in Register:
+        i.ha_conf(reg, ha_prfx="tsun/", node_id="garagendach/", snr='123', singleton=False, sug_area = 'roof')  # noqa: E501
+
+
+    for d_json, comp, node_id, id in i.ha_proxy_confs(ha_prfx="tsun/", node_id = 'proxy/', snr = '456'):
         pass
 
-    val = i.dev_value(0xffffff04)  # check internal error counter
+    val = i.dev_value(Register.INTERNAL_ERROR)  # check internal error counter
     assert val == 0
 
     # test missing 'fmt' value
-    Infos._Infos__info_defs[0xfffffffe] =  {'name':['proxy', 'Internal_Test1'],  'singleton': True, 'ha':{'dev':'proxy', 'dev_cla': None,       'stat_cla': None, 'id':'intern_test1_'}}
+    i.info_defs[Register.TEST_REG1] =  {'name':['proxy', 'Internal_Test1'],  'singleton': True, 'ha':{'dev':'proxy', 'dev_cla': None,       'stat_cla': None, 'id':'intern_test1_'}}
 
     tests = 0
-    for d_json, comp, node_id, id in i.ha_confs(ha_prfx="tsun/", inv_snr='123', inv_node_id="garagendach/",proxy_node_id = 'proxy/', proxy_unique_id = '456', sug_area = 'roof'):
+    for d_json, comp, node_id, id in i.ha_proxy_confs(ha_prfx="tsun/", node_id = 'proxy/', snr = '456'):
         if id == 'intern_test1_456':
             tests +=1
 
     assert tests == 1
 
-    val = i.dev_value(0xffffff04)  # check internal error counter
+    val = i.dev_value(Register.INTERNAL_ERROR)  # check internal error counter
     assert val == 1
 
     # test missing 'dev' value
-    Infos._Infos__info_defs[0xfffffffe] =  {'name':['proxy', 'Internal_Test2'],  'singleton': True, 'ha':{'dev_cla': None,       'stat_cla': None, 'id':'intern_test2_',  'fmt':'| int'}}
+    i.info_defs[Register.TEST_REG1] =  {'name':['proxy', 'Internal_Test2'],  'singleton': True, 'ha':{'dev_cla': None,       'stat_cla': None, 'id':'intern_test2_',  'fmt':'| int'}}
     tests = 0
-    for d_json, comp, node_id, id in i.ha_confs(ha_prfx="tsun/", inv_snr='123', inv_node_id="garagendach/",proxy_node_id = 'proxy/', proxy_unique_id = '456', sug_area = 'roof'):
+    for d_json, comp, node_id, id in i.ha_proxy_confs(ha_prfx="tsun/", node_id = 'proxy/', snr = '456'):
         if id == 'intern_test2_456':
             tests +=1
 
     assert tests == 1
 
-    val = i.dev_value(0xffffff04)  # check internal error counter
+    val = i.dev_value(Register.INTERNAL_ERROR)  # check internal error counter
     assert val == 2
 
 
 
     # test invalid 'via' value
-    Infos._Infos__info_devs['test_dev'] = {'via':'xyz',   'name':'Module PV1'}
+    i.info_devs['test_dev'] = {'via':'xyz',   'name':'Module PV1'}
 
-    Infos._Infos__info_defs[0xfffffffe] =  {'name':['proxy', 'Internal_Test2'],  'singleton': True, 'ha':{'dev':'test_dev', 'dev_cla': None,       'stat_cla': None, 'id':'intern_test2_',  'fmt':'| int'}}
+    i.info_defs[Register.TEST_REG1] =  {'name':['proxy', 'Internal_Test2'],  'singleton': True, 'ha':{'dev':'test_dev', 'dev_cla': None,       'stat_cla': None, 'id':'intern_test2_',  'fmt':'| int'}}
     tests = 0
-    for d_json, comp, node_id, id in i.ha_confs(ha_prfx="tsun/", inv_snr='123', inv_node_id="garagendach/",proxy_node_id = 'proxy/', proxy_unique_id = '456', sug_area = 'roof'):
+    for d_json, comp, node_id, id in i.ha_proxy_confs(ha_prfx="tsun/", node_id = 'proxy/', snr = '456'):
         if id == 'intern_test2_456':
             tests +=1
 
     assert tests == 1
 
-    val = i.dev_value(0xffffff04)  # check internal error counter
+    val = i.dev_value(Register.INTERNAL_ERROR)  # check internal error counter
     assert val == 3
 
-
-def test_invalid_data_type(InvalidDataSeq):
+def test_table_remove():
     i = Infos()
     i.static_init()                # initialize counter
 
-    val = i.dev_value(0xffffff03)  # check invalid data type counter
+    val = i.dev_value(Register.INTERNAL_ERROR)  # check internal error counter
+    assert val == 0
+
+    # for d_json, comp, node_id, id in i.ha_confs(ha_prfx="tsun/", node_id="garagendach/", snr='123', sug_area = 'roof'):
+    #    pass
+    test = 0
+    for reg in Register:
+        res = i.ha_remove(reg, node_id="garagendach/", snr='123')  # noqa: E501
+        if reg == Register.INVERTER_STATUS:
+            test += 1
+            assert res == ('{}', 'sensor', 'garagendach/', 'inv_status_123')
+        elif reg == Register.COLLECT_INTERVAL:
+            test += 1
+            assert res == ('{}', 'sensor', 'garagendach/', 'data_collect_intval_123')
+
+    assert test == 2
+    val = i.dev_value(Register.INTERNAL_ERROR)  # check internal error counter
     assert val == 0
 
 
-    for key, result in i.parse (InvalidDataSeq):
-        pass
-    assert json.dumps(i.db) == json.dumps({"inverter": {"Product_Name": "Microinv"}})
+def test_clr_at_midnight():
+    i = Infos()
+    i.static_init()                # initialize counter
+    i.set_db_def_value(Register.NO_INPUTS, 2)
+    val = i.dev_value(Register.NO_INPUTS)  # valid addr but not initiliazed     
+    assert val == 2
+    i.info_defs[Register.TEST_REG1] = {  # add a entry with incomplete ha definition 
+        'name': ['test', 'grp', 'REG_1'], 'ha': {'dev_cla': None }
+        }
+    i.reg_clr_at_midnight('tsun/inv_1/')
+    # tsun/inv_2/input
+    assert json.dumps(ClrAtMidnight.db['tsun/inv_1/total']) == json.dumps({'Daily_Generation': 0})
+    assert json.dumps(ClrAtMidnight.db['tsun/inv_1/input']) == json.dumps({"pv1": {"Daily_Generation": 0}, "pv2": {"Daily_Generation": 0}})
 
-    val = i.dev_value(0xffffff03)  # check invalid data type counter
-    assert val == 1
+    i.reg_clr_at_midnight('tsun/inv_1/')
+    assert json.dumps(ClrAtMidnight.db['tsun/inv_1/total']) == json.dumps({'Daily_Generation': 0})
+    assert json.dumps(ClrAtMidnight.db['tsun/inv_1/input']) == json.dumps({"pv1": {"Daily_Generation": 0}, "pv2": {"Daily_Generation": 0}})
 
+    test = 0
+    for key, data in ClrAtMidnight.elm():
+        if key == 'tsun/inv_1/total':
+            assert json.dumps(data) == json.dumps({'Daily_Generation': 0})
+            test += 1
+        elif key == 'tsun/inv_1/input':
+            assert json.dumps(data) == json.dumps({"pv1": {"Daily_Generation": 0}, "pv2": {"Daily_Generation": 0}})
+            test += 1
+    assert test == 2
+    assert json.dumps(ClrAtMidnight.db) == json.dumps({})
+
+    i.reg_clr_at_midnight('tsun/inv_1/')
+
+def test_pv_module_config():
+    i = Infos()
+    # i.set_db_def_value(Register.NO_INPUTS, 2)
+
+    dt = {
+        'pv1':{'manufacturer':'TSUN1','type': 'Module 100W'},
+        'pv2':{'manufacturer':'TSUN2'},
+        'pv3':{'manufacturer':'TSUN3','type': 'Module 300W'},
+        'pv4':{'type': 'Module 400W'},
+        'pv5':{},
+                 }
+    i.set_pv_module_details(dt)
+    assert 'TSUN1' == i.dev_value(Register.PV1_MANUFACTURER)
+    assert 'TSUN2' == i.dev_value(Register.PV2_MANUFACTURER)
+    assert 'TSUN3' == i.dev_value(Register.PV3_MANUFACTURER)
+    assert None == i.dev_value(Register.PV4_MANUFACTURER)
+    assert None == i.dev_value(Register.PV5_MANUFACTURER)
+    assert 'Module 100W' == i.dev_value(Register.PV1_MODEL)
+    assert  None == i.dev_value(Register.PV2_MODEL)
+    assert 'Module 300W' == i.dev_value(Register.PV3_MODEL)
+    assert 'Module 400W' == i.dev_value(Register.PV4_MODEL)
+    assert  None == i.dev_value(Register.PV5_MODEL)
+
+def test_broken_info_defs():
+    i = Infos()
+    val = i.get_db_value(Register.NO_INPUTS, 666)
+    assert val == 666
+    i.info_defs[Register.TEST_REG1] = 'test'  # add a string instead of a dict 
+    val = i.get_db_value(Register.TEST_REG1, 666)
+    assert val == 666
+    i.set_db_def_value(Register.TEST_REG1, 2)
+    del i.info_defs[Register.TEST_REG1]       # delete the broken entry 
+
+def test_get_value():
+    i = Infos()
+    assert None == i.get_db_value(Register.PV1_VOLTAGE, None)
+    assert None == i.get_db_value(Register.PV2_VOLTAGE, None)
+
+    i.set_db_def_value(Register.PV1_VOLTAGE, 30) 
+    assert 30 == i.get_db_value(Register.PV1_VOLTAGE, None)
+    assert None == i.get_db_value(Register.PV2_VOLTAGE, None)
+
+    i.set_db_def_value(Register.PV2_VOLTAGE, 30.3) 
+    assert 30 == i.get_db_value(Register.PV1_VOLTAGE, None)
+    assert 30.3 == i.get_db_value(Register.PV2_VOLTAGE, None)
+
+def test_update_value():
+    i = Infos()
+    assert None == i.get_db_value(Register.PV1_VOLTAGE, None)
+
+    keys = i.info_defs[Register.PV1_VOLTAGE]['name']
+    name, update = i.update_db(keys, True, 30) 
+    assert update == True
+    assert 30 == i.get_db_value(Register.PV1_VOLTAGE, None)
+
+    keys = i.info_defs[Register.PV1_VOLTAGE]['name']
+    name, update = i.update_db(keys, True, 30) 
+    assert update == False
+    assert 30 == i.get_db_value(Register.PV1_VOLTAGE, None)
+
+    keys = i.info_defs[Register.PV1_VOLTAGE]['name']
+    name, update = i.update_db(keys, False, 29) 
+    assert update == True
+    assert 29 == i.get_db_value(Register.PV1_VOLTAGE, None)
+
+def test_key_obj():
+    i = Infos()
+    keys, level, unit, must_incr = i._key_obj(Register.PV1_VOLTAGE)
+    assert keys == ['input', 'pv1', 'Voltage']
+    assert level == logging.DEBUG
+    assert unit == 'V'
+    assert must_incr == False
+
+    keys, level, unit, must_incr = i._key_obj(Register.PV1_DAILY_GENERATION)
+    assert keys == ['input', 'pv1', 'Daily_Generation'] 
+    assert level == logging.DEBUG
+    assert unit == 'kWh'
+    assert must_incr == True

app/tests/test_infos_g3.py

@@ -0,0 +1,503 @@
+# test_with_pytest.py
+import pytest, json
+from app.src.infos import Register, ClrAtMidnight
+from app.src.gen3.infos_g3 import InfosG3
+
+@pytest.fixture
+def ContrDataSeq(): # Get Time Request message
+    msg =   b'\x00\x00\x00\x15\x00\x09\x2b\xa8\x54\x10\x52\x53\x57\x5f\x34\x30\x30\x5f\x56\x31\x2e\x30\x30\x2e\x30\x36\x00\x09\x27\xc0\x54\x06\x52\x61\x79\x6d\x6f'
+    msg +=  b'\x6e\x00\x09\x2f\x90\x54\x0b\x52\x53\x57\x2d\x31\x2d\x31\x30\x30\x30\x31\x00\x09\x5a\x88\x54\x0f\x74\x2e\x72\x61\x79\x6d\x6f\x6e\x69\x6f\x74\x2e\x63\x6f\x6d\x00\x09\x5a\xec\x54'
+    msg +=  b'\x1c\x6c\x6f\x67\x67\x65\x72\x2e\x74\x61\x6c\x65\x6e\x74\x2d\x6d\x6f\x6e\x69\x74\x6f\x72\x69\x6e\x67\x2e\x63\x6f\x6d\x00\x0d\x00\x20\x49\x00\x00\x00\x01\x00\x0c\x35\x00\x49\x00'
+    msg +=  b'\x00\x00\x64\x00\x0c\x96\xa8\x49\x00\x00\x00\x1d\x00\x0c\x7f\x38\x49\x00\x00\x00\x01\x00\x0c\xfc\x38\x49\x00\x00\x00\x01\x00\x0c\xf8\x50\x49\x00\x00\x01\x2c\x00\x0c\x63\xe0\x49'
+    msg +=  b'\x00\x00\x00\x00\x00\x0c\x67\xc8\x49\x00\x00\x00\x00\x00\x0c\x50\x58\x49\x00\x00\x00\x01\x00\x09\x5e\x70\x49\x00\x00\x13\x8d\x00\x09\x5e\xd4\x49\x00\x00\x13\x8d\x00\x09\x5b\x50'
+    msg +=  b'\x49\x00\x00\x00\x02\x00\x0d\x04\x08\x49\x00\x00\x00\x00\x00\x07\xa1\x84\x49\x00\x00\x00\x01\x00\x0c\x50\x59\x49\x00\x00\x00\x4c\x00\x0d\x1f\x60\x49\x00\x00\x00\x00'
+    return msg
+
+@pytest.fixture
+def Contr2DataSeq(): # Get Time Request message
+    msg  =  b'\x00\x00\x00\x39\x00\x09\x2b\xa8\x54\x10\x52'
+    msg +=  b'\x53\x57\x5f\x34\x30\x30\x5f\x56\x31\x2e\x30\x30\x2e\x32\x30\x00'
+    msg +=  b'\x09\x27\xc0\x54\x06\x52\x61\x79\x6d\x6f\x6e\x00\x09\x2f\x90\x54'
+    msg +=  b'\x0b\x52\x53\x57\x2d\x31\x2d\x31\x30\x30\x30\x31\x00\x09\x5a\x88'
+    msg +=  b'\x54\x0f\x74\x2e\x72\x61\x79\x6d\x6f\x6e\x69\x6f\x74\x2e\x63\x6f'
+    msg +=  b'\x6d\x00\x09\x5a\xec\x54\x1c\x6c\x6f\x67\x67\x65\x72\x2e\x74\x61'
+    msg +=  b'\x6c\x65\x6e\x74\x2d\x6d\x6f\x6e\x69\x74\x6f\x72\x69\x6e\x67\x2e'
+    msg +=  b'\x63\x6f\x6d\x00\x0d\x2f\x00\x54\x10\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x0d\x32\xe8\x54\x10\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00'
+    msg +=  b'\x0d\x36\xd0\x54\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\x00\x0d\x3a\xb8\x54\x10\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x0d\x3e\xa0\x54'
+    msg +=  b'\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\x00\x0d\x42\x88\x54\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\x00\x0d\x46\x70\x54\x10\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x0d\x4a'
+    msg +=  b'\x58\x54\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\x00\x0d\x4e\x40\x54\x10\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x0d\x52\x28\x54\x10\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00'
+    msg +=  b'\x0d\x56\x10\x54\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\x00\x0d\x59\xf8\x54\x10\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x0d\x5d\xe0\x54'
+    msg +=  b'\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\x00\x0d\x61\xc8\x54\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\x00\x0d\x65\xb0\x54\x10\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x0d\x69'
+    msg +=  b'\x98\x54\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\x00\x0d\x6d\x80\x54\x10\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x0d\x71\x68\x54\x10\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00'
+    msg +=  b'\x0d\x75\x50\x54\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\x00\x0d\x79\x38\x54\x10\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x0d\x7d\x20\x54'
+    msg +=  b'\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\x00\x0d\x81\x08\x54\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\x00\x0d\x84\xf0\x54\x10\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x0d\x88'
+    msg +=  b'\xd8\x54\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\x00\x0d\x8c\xc0\x54\x10\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x0d\x90\xa8\x54\x10\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00'
+    msg +=  b'\x0d\x94\x90\x54\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\x00\x0d\x98\x78\x54\x10\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x0d\x9c\x60\x54'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'
+    msg +=  b'\x00\x0d\x00\x20\x49\x00\x00\x00\x01\x00\x0c\x35\x00\x49\x00\x00'
+    msg +=  b'\x00\x10\x00\x0c\x96\xa8\x49\x00\x00\x01\x4e\x00\x0c\x7f\x38\x49'
+    msg +=  b'\x00\x00\x00\x01\x00\x0c\xfc\x38\x49\x00\x00\x00\x01\x00\x0c\xf8'
+    msg +=  b'\x50\x49\x00\x00\x01\x2c\x00\x0c\x63\xe0\x49\x00\x00\x00\x00\x00'
+    msg +=  b'\x0c\x67\xc8\x49\x00\x00\x00\x00\x00\x0c\x50\x58\x49\x00\x00\x00'
+    msg +=  b'\x01\x00\x09\x5e\x70\x49\x00\x00\x13\x8d\x00\x09\x5e\xd4\x49\x00'
+    msg +=  b'\x00\x13\x8d\x00\x09\x5b\x50\x49\x00\x00\x00\x02\x00\x0d\x04\x08'
+    msg +=  b'\x49\x00\x00\x00\x00\x00\x07\xa1\x84\x49\x00\x00\x00\x01\x00\x0c'
+    msg +=  b'\x50\x59\x49\x00\x00\x00\x33\x00\x0d\x1f\x60\x49\x00\x00\x00\x00'
+    msg +=  b'\x00\x0d\x23\x48\x49\xff\xff\xff\xff\x00\x0d\x27\x30\x49\xff\xff'
+    msg +=  b'\xff\xff\x00\x0d\x2b\x18\x4c\x00\x00\x00\x00\x00\x00\xff\xff\x00'
+    msg +=  b'\x0c\xa2\x60\x49\x00\x00\x00\x00\x00\x0d\xa0\x48\x49\x00\x00\x00'
+    msg +=  b'\x00\x00\x0d\xa4\x30\x49\x00\x00\x00\x00\x00\x0d\xa8\x18\x49\x00'
+    msg +=  b'\x00\x00\x00'
+    return msg
+
+@pytest.fixture
+def InvDataSeq(): # Data indication from the controller
+    msg =   b'\x00\x00\x00\x06\x00\x00\x00\x0a\x54\x08\x4d\x69\x63\x72\x6f\x69\x6e\x76\x00\x00\x00\x14\x54\x04\x54\x53\x55\x4e\x00\x00\x00\x1E\x54\x07\x56\x35\x2e\x30\x2e\x31\x31\x00\x00\x00\x28'
+    msg +=  b'\x54\x10T170000000000001\x00\x00\x00\x32\x54\x0a\x54\x53\x4f\x4c\x2d\x4d\x53\x36\x30\x30\x00\x00\x00\x3c\x54\x05\x41\x2c\x42\x2c\x43'
+    return msg
+
+@pytest.fixture
+def InvalidDataSeq(): # Data indication from the controller
+    msg =   b'\x00\x00\x00\x06\x00\x00\x00\x0a\x54\x08\x4d\x69\x63\x72\x6f\x69\x6e\x76\x00\x00\x00\x14\x64\x04\x54\x53\x55\x4e\x00\x00\x00\x1E\x54\x07\x56\x35\x2e\x30\x2e\x31\x31\x00\x00\x00\x28'
+    msg +=  b'\x54\x10T170000000000001\x00\x00\x00\x32\x54\x0a\x54\x53\x4f\x4c\x2d\x4d\x53\x36\x30\x30\x00\x00\x00\x3c\x54\x05\x41\x2c\x42\x2c\x43'
+    return msg
+
+@pytest.fixture
+def InvDataSeq2(): # Data indication from the controller
+    msg =   b'\x00\x00\x00\xa3\x00\x00\x00\x64\x53\x00\x01\x00\x00\x00\xc8\x53\x00\x02\x00\x00\x01\x2c\x53\x00\x00\x00\x00\x01\x90\x49\x00\x00\x00\x00\x00\x00\x01\x91\x53\x00\x00'
+    msg +=  b'\x00\x00\x01\x92\x53\x00\x00\x00\x00\x01\x93\x53\x00\x00\x00\x00\x01\x94\x53\x00\x00\x00\x00\x01\x95\x53\x00\x00\x00\x00\x01\x96\x53\x00\x00\x00\x00\x01\x97\x53\x00'
+    msg +=  b'\x00\x00\x00\x01\x98\x53\x00\x00\x00\x00\x01\x99\x53\x00\x00\x00\x00\x01\x9a\x53\x00\x00\x00\x00\x01\x9b\x53\x00\x00\x00\x00\x01\x9c\x53\x00\x00\x00\x00\x01\x9d\x53'
+    msg +=  b'\x00\x00\x00\x00\x01\x9e\x53\x00\x00\x00\x00\x01\x9f\x53\x00\x00\x00\x00\x01\xa0\x53\x00\x00\x00\x00\x01\xf4\x49\x00\x00\x00\x00\x00\x00\x01\xf5\x53\x00\x00\x00\x00'
+    msg +=  b'\x01\xf6\x53\x00\x00\x00\x00\x01\xf7\x53\x00\x00\x00\x00\x01\xf8\x53\x00\x00\x00\x00\x01\xf9\x53\x00\x00\x00\x00\x01\xfa\x53\x00\x00\x00\x00\x01\xfb\x53\x00\x00\x00'
+    msg +=  b'\x00\x01\xfc\x53\x00\x00\x00\x00\x01\xfd\x53\x00\x00\x00\x00\x01\xfe\x53\x00\x00\x00\x00\x01\xff\x53\x00\x00\x00\x00\x02\x00\x53\x00\x00\x00\x00\x02\x01\x53\x00\x00'
+    msg +=  b'\x00\x00\x02\x02\x53\x00\x00\x00\x00\x02\x03\x53\x00\x00\x00\x00\x02\x04\x53\x00\x00\x00\x00\x02\x58\x49\x00\x00\x00\x00\x00\x00\x02\x59\x53\x00\x00\x00\x00\x02\x5a'
+    msg +=  b'\x53\x00\x00\x00\x00\x02\x5b\x53\x00\x00\x00\x00\x02\x5c\x53\x00\x00\x00\x00\x02\x5d\x53\x00\x00\x00\x00\x02\x5e\x53\x00\x00\x00\x00\x02\x5f\x53\x00\x00\x00\x00\x02'
+    msg +=  b'\x60\x53\x00\x00\x00\x00\x02\x61\x53\x00\x00\x00\x00\x02\x62\x53\x00\x00\x00\x00\x02\x63\x53\x00\x00\x00\x00\x02\x64\x53\x00\x00\x00\x00\x02\x65\x53\x00\x00\x00\x00'
+    msg +=  b'\x02\x66\x53\x00\x00\x00\x00\x02\x67\x53\x00\x00\x00\x00\x02\x68\x53\x00\x00\x00\x00\x02\xbc\x49\x00\x00\x00\x00\x00\x00\x02\xbd\x53\x00\x00\x00\x00\x02\xbe\x53\x00'
+    msg +=  b'\x00\x00\x00\x02\xbf\x53\x00\x00\x00\x00\x02\xc0\x53\x00\x00\x00\x00\x02\xc1\x53\x00\x00\x00\x00\x02\xc2\x53\x00\x00\x00\x00\x02\xc3\x53\x00\x00\x00\x00\x02\xc4\x53'
+    msg +=  b'\x00\x00\x00\x00\x02\xc5\x53\x00\x00\x00\x00\x02\xc6\x53\x00\x00\x00\x00\x02\xc7\x53\x00\x00\x00\x00\x02\xc8\x53\x00\x00\x00\x00\x02\xc9\x53\x00\x00\x00\x00\x02\xca'
+    msg +=  b'\x53\x00\x00\x00\x00\x02\xcb\x53\x00\x00\x00\x00\x02\xcc\x53\x00\x00\x00\x00\x03\x20\x53\x00\x00\x00\x00\x03\x84\x53\x50\x11\x00\x00\x03\xe8\x46\x43\x61\x66\x66\x00'
+    msg +=  b'\x00\x04\x4c\x46\x3e\xeb\x85\x1f\x00\x00\x04\xb0\x46\x42\x48\x14\x7b\x00\x00\x05\x14\x53\x00\x17\x00\x00\x05\x78\x53\x00\x00\x00\x00\x05\xdc\x53\x02\x58\x00\x00\x06'
+    msg +=  b'\x40\x46\x42\xd3\x66\x66\x00\x00\x06\xa4\x46\x42\x06\x66\x66\x00\x00\x07\x08\x46\x3f\xf4\x7a\xe1\x00\x00\x07\x6c\x46\x42\x81\x00\x00\x00\x00\x07\xd0\x46\x42\x06\x00'
+    msg +=  b'\x00\x00\x00\x08\x34\x46\x3f\xae\x14\x7b\x00\x00\x08\x98\x46\x42\x36\xcc\xcd\x00\x00\x08\xfc\x46\x00\x00\x00\x00\x00\x00\x09\x60\x46\x00\x00\x00\x00\x00\x00\x09\xc4'
+    msg +=  b'\x46\x00\x00\x00\x00\x00\x00\x0a\x28\x46\x00\x00\x00\x00\x00\x00\x0a\x8c\x46\x00\x00\x00\x00\x00\x00\x0a\xf0\x46\x00\x00\x00\x00\x00\x00\x0b\x54\x46\x3f\xd9\x99\x9a'
+    msg +=  b'\x00\x00\x0b\xb8\x46\x41\x8a\xe1\x48\x00\x00\x0c\x1c\x46\x3f\x8a\x3d\x71\x00\x00\x0c\x80\x46\x41\x1b\xd7\x0a\x00\x00\x0c\xe4\x46\x3f\x1e\xb8\x52\x00\x00\x0d\x48\x46'
+    msg +=  b'\x40\xf3\xd7\x0a\x00\x00\x0d\xac\x46\x00\x00\x00\x00\x00\x00\x0e\x10\x46\x00\x00\x00\x00\x00\x00\x0e\x74\x46\x00\x00\x00\x00\x00\x00\x0e\xd8\x46\x00\x00\x00\x00\x00'
+    msg +=  b'\x00\x0f\x3c\x53\x00\x00\x00\x00\x0f\xa0\x53\x00\x00\x00\x00\x10\x04\x53\x55\xaa\x00\x00\x10\x68\x53\x00\x00\x00\x00\x10\xcc\x53\x00\x00\x00\x00\x11\x30\x53\x00\x00'
+    msg +=  b'\x00\x00\x11\x94\x53\x00\x00\x00\x00\x11\xf8\x53\xff\xff\x00\x00\x12\x5c\x53\xff\xff\x00\x00\x12\xc0\x53\x00\x02\x00\x00\x13\x24\x53\xff\xff\x00\x00\x13\x88\x53\xff'
+    msg +=  b'\xff\x00\x00\x13\xec\x53\xff\xff\x00\x00\x14\x50\x53\xff\xff\x00\x00\x14\xb4\x53\xff\xff\x00\x00\x15\x18\x53\xff\xff\x00\x00\x15\x7c\x53\x00\x00\x00\x00\x27\x10\x53'
+    msg +=  b'\x00\x02\x00\x00\x27\x74\x53\x00\x3c\x00\x00\x27\xd8\x53\x00\x68\x00\x00\x28\x3c\x53\x05\x00\x00\x00\x28\xa0\x46\x43\x79\x00\x00\x00\x00\x29\x04\x46\x43\x48\x00\x00'
+    msg +=  b'\x00\x00\x29\x68\x46\x42\x48\x33\x33\x00\x00\x29\xcc\x46\x42\x3e\x3d\x71\x00\x00\x2a\x30\x53\x00\x01\x00\x00\x2a\x94\x46\x43\x37\x00\x00\x00\x00\x2a\xf8\x46\x42\xce'
+    msg +=  b'\x00\x00\x00\x00\x2b\x5c\x53\x00\x96\x00\x00\x2b\xc0\x53\x00\x10\x00\x00\x2c\x24\x46\x43\x90\x00\x00\x00\x00\x2c\x88\x46\x43\x95\x00\x00\x00\x00\x2c\xec\x53\x00\x06'
+    msg +=  b'\x00\x00\x2d\x50\x53\x00\x06\x00\x00\x2d\xb4\x46\x43\x7d\x00\x00\x00\x00\x2e\x18\x46\x42\x3d\xeb\x85\x00\x00\x2e\x7c\x46\x42\x3d\xeb\x85\x00\x00\x2e\xe0\x53\x00\x03'
+    msg +=  b'\x00\x00\x2f\x44\x53\x00\x03\x00\x00\x2f\xa8\x46\x42\x4d\xeb\x85\x00\x00\x30\x0c\x46\x42\x4d\xeb\x85\x00\x00\x30\x70\x53\x00\x03\x00\x00\x30\xd4\x53\x00\x03\x00\x00'
+    msg +=  b'\x31\x38\x46\x42\x08\x00\x00\x00\x00\x31\x9c\x53\x00\x05\x00\x00\x32\x00\x53\x04\x00\x00\x00\x32\x64\x53\x00\x01\x00\x00\x32\xc8\x53\x13\x9c\x00\x00\x33\x2c\x53\x0f'
+    msg +=  b'\xa0\x00\x00\x33\x90\x53\x00\x4f\x00\x00\x33\xf4\x53\x00\x66\x00\x00\x34\x58\x53\x03\xe8\x00\x00\x34\xbc\x53\x04\x00\x00\x00\x35\x20\x53\x00\x00\x00\x00\x35\x84\x53'
+    msg +=  b'\x00\x00\x00\x00\x35\xe8\x53\x00\x00\x00\x00\x36\x4c\x53\x00\x00\x00\x01\x38\x80\x53\x00\x02\x00\x01\x38\x81\x53\x00\x01\x00\x01\x38\x82\x53\x00\x01\x00\x01\x38\x83'
+    msg +=  b'\x53\x00\x00'  
+    return msg
+
+@pytest.fixture
+def InvDataNew(): # Data indication from DSP V5.0.17
+    msg  = b'\x00\x00\x00\xa3\x00\x00\x00\x00\x53\x00\x00'
+    msg += b'\x00\x00\x00\x80\x53\x00\x00\x00\x00\x01\x04\x53\x00\x00\x00\x00'
+    msg += b'\x01\x90\x41\x00\x00\x01\x91\x53\x00\x00\x00\x00\x01\x90\x53\x00'
+    msg += b'\x00\x00\x00\x01\x91\x53\x00\x00\x00\x00\x01\x90\x53\x00\x00\x00'
+    msg += b'\x00\x01\x91\x53\x00\x00\x00\x00\x01\x94\x53\x00\x00\x00\x00\x01'
+    msg += b'\x95\x53\x00\x00\x00\x00\x01\x98\x53\x00\x00\x00\x00\x01\x99\x53'
+    msg += b'\x00\x00\x00\x00\x01\x80\x53\x00\x00\x00\x00\x01\x90\x41\x00\x00'
+    msg += b'\x01\x94\x53\x00\x00\x00\x00\x01\x94\x53\x00\x00\x00\x00\x01\x96'
+    msg += b'\x53\x00\x00\x00\x00\x01\x98\x53\x00\x00\x00\x00\x01\xa0\x53\x00'
+    msg += b'\x00\x00\x00\x01\xf0\x41\x00\x00\x01\xf1\x53\x00\x00\x00\x00\x01'
+    msg += b'\xf4\x53\x00\x00\x00\x00\x01\xf5\x53\x00\x00\x00\x00\x01\xf8\x53'
+    msg += b'\x00\x00\x00\x00\x01\xf9\x53\x00\x00\x00\x00\x00\x00\x53\x00\x00'
+    msg += b'\x00\x00\x00\x01\x53\x00\x00\x00\x00\x00\x00\x53\x00\x00\x00\x00'
+    msg += b'\x00\x01\x53\x00\x00\x00\x00\x00\x04\x53\x00\x00\x00\x00\x00\x58'
+    msg += b'\x41\x00\x00\x02\x00\x53\x00\x00\x00\x00\x02\x00\x53\x00\x00\x00'
+    msg += b'\x00\x02\x02\x53\x00\x00\x00\x00\x02\x00\x53\x00\x00\x00\x00\x02'
+    msg += b'\x04\x53\x00\x00\x00\x00\x02\x58\x41\x00\x00\x02\x59\x53\x00\x00'
+    msg += b'\x00\x00\x02\x40\x53\x00\x00\x00\x00\x02\x41\x53\x00\x00\x00\x00'
+    msg += b'\x02\x40\x53\x00\x00\x00\x00\x02\x41\x53\x00\x00\x00\x00\x02\x44'
+    msg += b'\x53\x00\x00\x00\x00\x02\x45\x53\x00\x00\x00\x00\x02\x60\x53\x00'
+    msg += b'\x00\x00\x00\x02\x61\x53\x00\x00\x00\x00\x02\x60\x53\x00\x00\x00'
+    msg += b'\x00\x02\x20\x41\x00\x00\x02\x24\x53\x00\x00\x00\x00\x02\x24\x53'
+    msg += b'\x00\x00\x00\x00\x02\x26\x53\x00\x00\x00\x00\x02\x40\x53\x00\x00'
+    msg += b'\x00\x00\x02\x40\x53\x00\x00\x00\x00\x02\x80\x41\x00\x00\x02\x81'
+    msg += b'\x53\x00\x00\x00\x00\x02\x84\x53\x00\x00\x00\x00\x02\x85\x53\x00'
+    msg += b'\x00\x00\x00\x02\xc0\x53\x00\x00\x00\x00\x02\xc1\x53\x00\x00\x00'
+    msg += b'\x00\x02\xc0\x53\x00\x00\x00\x00\x02\xc1\x53\x00\x00\x00\x00\x02'
+    msg += b'\xc0\x53\x00\x00\x00\x00\x02\xc1\x53\x00\x00\x00\x00\x02\xc4\x53'
+    msg += b'\x00\x00\x00\x00\x02\x00\x53\x00\x00\x00\x00\x02\x80\x53\x00\x00'
+    msg += b'\x00\x00\x02\xc8\x42\x00\x00\x00\x00\x48\x42\x00\x00\x00\x00\x80'
+    msg += b'\x42\x00\x00\x00\x00\x04\x53\x00\x00\x00\x00\x01\x20\x53\x00\x00'
+    msg += b'\x00\x00\x01\x84\x53\x00\x10\x00\x00\x02\x40\x46\x00\x00\x00\x00'
+    msg += b'\x00\x00\x04\x04\x46\x02\x00\x46\x02\x00\x00\x04\x00\x46\x00\x00'
+    msg += b'\x00\x00\x00\x00\x05\x04\x42\x00\x00\x00\x05\x50\x42\x00\x00\x00'
+    msg += b'\x00\x14\x42\x00\x00\x00\x00\x00\x46\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\xa4\x46\x00\x00\x00\x00\x00\x00\x01\x00\x46\x00\x00\x00\x00\x00'
+    msg += b'\x00\x01\x44\x46\x00\x00\x00\x00\x00\x00\x02\x00\x46\x00\x00\x00'
+    msg += b'\x00\x00\x00\x08\x04\x46\x00\x00\x00\x00\x00\x00\x08\x90\x46\x00'
+    msg += b'\x00\x00\x00\x00\x00\x08\x54\x46\x00\x00\x00\x00\x00\x00\x09\x20'
+    msg += b'\x46\x00\x00\x00\x00\x00\x00\x08\x04\x46\x00\x00\x00\x00\x00\x00'
+    msg += b'\x08\x00\x46\x00\x00\x00\x00\x00\x00\x08\x84\x46\x00\x00\x00\x00'
+    msg += b'\x00\x00\x08\x40\x46\x00\x00\x00\x00\x00\x00\x09\x04\x46\x00\x00'
+    msg += b'\x00\x00\x00\x00\x0a\x10\x46\x00\x00\x00\x00\x00\x00\x0c\x14\x46'
+    msg += b'\x00\x00\x00\x00\x00\x00\x0c\x80\x46\x00\x00\x00\x00\x00\x00\x0c'
+    msg += b'\x24\x42\x00\x00\x00\x0d\x00\x42\x00\x00\x00\x00\x04\x42\x00\x00'
+    msg += b'\x00\x00\x00\x42\x00\x00\x00\x00\x44\x42\x00\x00\x00\x00\x10\x42'
+    msg += b'\x00\x00\x00\x01\x14\x53\x00\x00\x00\x00\x01\xa0\x53\x00\x00\x00'
+    msg += b'\x00\x10\x04\x53\x55\xaa\x00\x00\x10\x40\x53\x00\x00\x00\x00\x10'
+    msg += b'\x04\x53\x00\x00\x00\x00\x11\x00\x53\x00\x00\x00\x00\x11\x84\x53'
+    msg += b'\x00\x00\x00\x00\x10\x50\x53\xff\xff\x00\x00\x10\x14\x53\x03\x20'
+    msg += b'\x00\x00\x10\x00\x53\x00\x00\x00\x00\x11\x24\x53\x00\x00\x00\x00'
+    msg += b'\x03\x00\x53\x00\x00\x00\x00\x03\x64\x53\x00\x00\x00\x00\x04\x50'
+    msg += b'\x53\x00\x00\x00\x00\x00\x34\x53\x00\x00\x00\x00\x00\x00\x42\x02'
+    msg += b'\x00\x00\x01\x04\x42\x00\x00\x00\x21\x00\x42\x00\x00\x00\x21\x44'
+    msg += b'\x42\x00\x00\x00\x22\x10\x53\x00\x00\x00\x00\x28\x14\x42\x01\x00'
+    msg += b'\x00\x28\xa0\x46\x42\x48\x00\x00\x00\x00\x29\x04\x42\x00\x00\x00'
+    msg += b'\x29\x40\x42\x00\x00\x00\x28\x04\x46\x42\x10\x00\x00\x00\x00\x28'
+    msg += b'\x00\x42\x00\x00\x00\x28\x84\x42\x00\x00\x00\x28\x50\x42\x00\x00'
+    msg += b'\x00\x29\x14\x42\x00\x00\x00\x2a\x00\x42\x00\x00\x00\x2c\x24\x46'
+    msg += b'\x42\x10\x00\x00\x00\x00\x2c\x80\x42\x00\x00\x00\x2c\x44\x53\x00'
+    msg += b'\x02\x00\x00\x2d\x00\x42\x00\x00\x00\x20\x04\x46\x42\x4d\x00\x00'
+    msg += b'\x00\x00\x20\x10\x42\x00\x00\x00\x20\x54\x42\x00\x00\x00\x20\x20'
+    msg += b'\x42\x00\x00\x00\x21\x04\x53\x00\x01\x00\x00\x22\x00\x42\x00\x00'
+    msg += b'\x00\x30\x04\x42\x00\x00\x00\x30\x40\x53\x00\x00\x00\x00\x30\x04'
+    msg += b'\x53\x00\x00\x00\x00\x31\x10\x42\x00\x00\x00\x31\x94\x53\x00\x04'
+    msg += b'\x00\x00\x30\x00\x53\x00\x00\x00\x00\x30\x24\x53\x00\x00\x00\x00'
+    msg += b'\x30\x00\x53\x00\x00\x00\x00\x31\x04\x53\x00\x00\x00\x00\x31\x80'
+    msg += b'\x53\x00\x00\x00\x00\x32\x44\x53\x00\x00\x00\x00\x30\x00\x53\x00'
+    msg += b'\x00\x00\x00\x30\x80\x53\x00\x00\x00\x00\x30\x00\x53\x00\x00\x00'
+    msg += b'\x00\x30\x80\x53\x00\x00\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x03\x00'
+    msg += b'\x00\x00\x00\x00'
+    return msg
+
+@pytest.fixture
+def InvDataSeq2_Zero(): # Data indication from the controller
+    msg =   b'\x00\x00\x00\xa3\x00\x00\x00\x64\x53\x00\x01\x00\x00\x00\xc8\x53\x00\x02\x00\x00\x01\x2c\x53\x00\x00\x00\x00\x01\x90\x49\x00\x00\x00\x00\x00\x00\x01\x91\x53\x00\x00'
+    msg +=  b'\x00\x00\x01\x92\x53\x00\x00\x00\x00\x01\x93\x53\x00\x00\x00\x00\x01\x94\x53\x00\x00\x00\x00\x01\x95\x53\x00\x00\x00\x00\x01\x96\x53\x00\x00\x00\x00\x01\x97\x53\x00'
+    msg +=  b'\x00\x00\x00\x01\x98\x53\x00\x00\x00\x00\x01\x99\x53\x00\x00\x00\x00\x01\x9a\x53\x00\x00\x00\x00\x01\x9b\x53\x00\x00\x00\x00\x01\x9c\x53\x00\x00\x00\x00\x01\x9d\x53'
+    msg +=  b'\x00\x00\x00\x00\x01\x9e\x53\x00\x00\x00\x00\x01\x9f\x53\x00\x00\x00\x00\x01\xa0\x53\x00\x00\x00\x00\x01\xf4\x49\x00\x00\x00\x00\x00\x00\x01\xf5\x53\x00\x00\x00\x00'
+    msg +=  b'\x01\xf6\x53\x00\x00\x00\x00\x01\xf7\x53\x00\x00\x00\x00\x01\xf8\x53\x00\x00\x00\x00\x01\xf9\x53\x00\x00\x00\x00\x01\xfa\x53\x00\x00\x00\x00\x01\xfb\x53\x00\x00\x00'
+    msg +=  b'\x00\x01\xfc\x53\x00\x00\x00\x00\x01\xfd\x53\x00\x00\x00\x00\x01\xfe\x53\x00\x00\x00\x00\x01\xff\x53\x00\x00\x00\x00\x02\x00\x53\x00\x00\x00\x00\x02\x01\x53\x00\x00'
+    msg +=  b'\x00\x00\x02\x02\x53\x00\x00\x00\x00\x02\x03\x53\x00\x00\x00\x00\x02\x04\x53\x00\x00\x00\x00\x02\x58\x49\x00\x00\x00\x00\x00\x00\x02\x59\x53\x00\x00\x00\x00\x02\x5a'
+    msg +=  b'\x53\x00\x00\x00\x00\x02\x5b\x53\x00\x00\x00\x00\x02\x5c\x53\x00\x00\x00\x00\x02\x5d\x53\x00\x00\x00\x00\x02\x5e\x53\x00\x00\x00\x00\x02\x5f\x53\x00\x00\x00\x00\x02'
+    msg +=  b'\x60\x53\x00\x00\x00\x00\x02\x61\x53\x00\x00\x00\x00\x02\x62\x53\x00\x00\x00\x00\x02\x63\x53\x00\x00\x00\x00\x02\x64\x53\x00\x00\x00\x00\x02\x65\x53\x00\x00\x00\x00'
+    msg +=  b'\x02\x66\x53\x00\x00\x00\x00\x02\x67\x53\x00\x00\x00\x00\x02\x68\x53\x00\x00\x00\x00\x02\xbc\x49\x00\x00\x00\x00\x00\x00\x02\xbd\x53\x00\x00\x00\x00\x02\xbe\x53\x00'
+    msg +=  b'\x00\x00\x00\x02\xbf\x53\x00\x00\x00\x00\x02\xc0\x53\x00\x00\x00\x00\x02\xc1\x53\x00\x00\x00\x00\x02\xc2\x53\x00\x00\x00\x00\x02\xc3\x53\x00\x00\x00\x00\x02\xc4\x53'
+    msg +=  b'\x00\x00\x00\x00\x02\xc5\x53\x00\x00\x00\x00\x02\xc6\x53\x00\x00\x00\x00\x02\xc7\x53\x00\x00\x00\x00\x02\xc8\x53\x00\x00\x00\x00\x02\xc9\x53\x00\x00\x00\x00\x02\xca'
+    msg +=  b'\x53\x00\x00\x00\x00\x02\xcb\x53\x00\x00\x00\x00\x02\xcc\x53\x00\x00\x00\x00\x03\x20\x53\x00\x00\x00\x00\x03\x84\x53\x50\x11\x00\x00\x03\xe8\x46\x43\x61\x66\x66\x00'
+    msg +=  b'\x00\x04\x4c\x46\x3e\xeb\x85\x1f\x00\x00\x04\xb0\x46\x42\x48\x14\x7b\x00\x00\x05\x14\x53\x00\x00\x00\x00\x05\x78\x53\x00\x00\x00\x00\x05\xdc\x53\x00\x00\x00\x00\x06'
+    msg +=  b'\x40\x46\x42\xd3\x66\x66\x00\x00\x06\xa4\x46\x42\x06\x66\x66\x00\x00\x07\x08\x46\x3f\xf4\x7a\xe1\x00\x00\x07\x6c\x46\x00\x00\x00\x00\x00\x00\x07\xd0\x46\x42\x06\x00'
+    msg +=  b'\x00\x00\x00\x08\x34\x46\x3f\xae\x14\x7b\x00\x00\x08\x98\x46\x00\x00\x00\x00\x00\x00\x08\xfc\x46\x00\x00\x00\x00\x00\x00\x09\x60\x46\x00\x00\x00\x00\x00\x00\x09\xc4'
+    msg +=  b'\x46\x00\x00\x00\x00\x00\x00\x0a\x28\x46\x00\x00\x00\x00\x00\x00\x0a\x8c\x46\x00\x00\x00\x00\x00\x00\x0a\xf0\x46\x00\x00\x00\x00\x00\x00\x0b\x54\x46\x00\x00\x00\x00'
+    msg +=  b'\x00\x00\x0b\xb8\x46\x00\x00\x00\x00\x00\x00\x0c\x1c\x46\x00\x00\x00\x00\x00\x00\x0c\x80\x46\x00\x00\x00\x00\x00\x00\x0c\xe4\x46\x00\x00\x00\x00\x00\x00\x0d\x48\x46'
+    msg +=  b'\x00\x00\x00\x00\x00\x00\x0d\xac\x46\x00\x00\x00\x00\x00\x00\x0e\x10\x46\x00\x00\x00\x00\x00\x00\x0e\x74\x46\x00\x00\x00\x00\x00\x00\x0e\xd8\x46\x00\x00\x00\x00\x00'
+    msg +=  b'\x00\x0f\x3c\x53\x00\x00\x00\x00\x0f\xa0\x53\x00\x00\x00\x00\x10\x04\x53\x55\xaa\x00\x00\x10\x68\x53\x00\x00\x00\x00\x10\xcc\x53\x00\x00\x00\x00\x11\x30\x53\x00\x00'
+    msg +=  b'\x00\x00\x11\x94\x53\x00\x00\x00\x00\x11\xf8\x53\xff\xff\x00\x00\x12\x5c\x53\xff\xff\x00\x00\x12\xc0\x53\x00\x02\x00\x00\x13\x24\x53\xff\xff\x00\x00\x13\x88\x53\xff'
+    msg +=  b'\xff\x00\x00\x13\xec\x53\xff\xff\x00\x00\x14\x50\x53\xff\xff\x00\x00\x14\xb4\x53\xff\xff\x00\x00\x15\x18\x53\xff\xff\x00\x00\x15\x7c\x53\x00\x00\x00\x00\x27\x10\x53'
+    msg +=  b'\x00\x02\x00\x00\x27\x74\x53\x00\x3c\x00\x00\x27\xd8\x53\x00\x68\x00\x00\x28\x3c\x53\x05\x00\x00\x00\x28\xa0\x46\x43\x79\x00\x00\x00\x00\x29\x04\x46\x43\x48\x00\x00'
+    msg +=  b'\x00\x00\x29\x68\x46\x42\x48\x33\x33\x00\x00\x29\xcc\x46\x42\x3e\x3d\x71\x00\x00\x2a\x30\x53\x00\x01\x00\x00\x2a\x94\x46\x43\x37\x00\x00\x00\x00\x2a\xf8\x46\x42\xce'
+    msg +=  b'\x00\x00\x00\x00\x2b\x5c\x53\x00\x96\x00\x00\x2b\xc0\x53\x00\x10\x00\x00\x2c\x24\x46\x43\x90\x00\x00\x00\x00\x2c\x88\x46\x43\x95\x00\x00\x00\x00\x2c\xec\x53\x00\x06'
+    msg +=  b'\x00\x00\x2d\x50\x53\x00\x06\x00\x00\x2d\xb4\x46\x43\x7d\x00\x00\x00\x00\x2e\x18\x46\x42\x3d\xeb\x85\x00\x00\x2e\x7c\x46\x42\x3d\xeb\x85\x00\x00\x2e\xe0\x53\x00\x03'
+    msg +=  b'\x00\x00\x2f\x44\x53\x00\x03\x00\x00\x2f\xa8\x46\x42\x4d\xeb\x85\x00\x00\x30\x0c\x46\x42\x4d\xeb\x85\x00\x00\x30\x70\x53\x00\x03\x00\x00\x30\xd4\x53\x00\x03\x00\x00'
+    msg +=  b'\x31\x38\x46\x42\x08\x00\x00\x00\x00\x31\x9c\x53\x00\x05\x00\x00\x32\x00\x53\x04\x00\x00\x00\x32\x64\x53\x00\x01\x00\x00\x32\xc8\x53\x13\x9c\x00\x00\x33\x2c\x53\x0f'
+    msg +=  b'\xa0\x00\x00\x33\x90\x53\x00\x4f\x00\x00\x33\xf4\x53\x00\x66\x00\x00\x34\x58\x53\x03\xe8\x00\x00\x34\xbc\x53\x04\x00\x00\x00\x35\x20\x53\x00\x00\x00\x00\x35\x84\x53'
+    msg +=  b'\x00\x00\x00\x00\x35\xe8\x53\x00\x00\x00\x00\x36\x4c\x53\x00\x00\x00\x01\x38\x80\x53\x00\x02\x00\x01\x38\x81\x53\x00\x01\x00\x01\x38\x82\x53\x00\x01\x00\x01\x38\x83'
+    msg +=  b'\x53\x00\x00'  
+    return msg
+
+
+def test_parse_control(ContrDataSeq):
+    i = InfosG3()
+    for key, result in i.parse (ContrDataSeq):
+        pass
+
+    assert json.dumps(i.db) == json.dumps(
+{"collector": {"Collector_Fw_Version": "RSW_400_V1.00.06", "Chip_Type": "Raymon", "Chip_Model": "RSW-1-10001", "Trace_URL": "t.raymoniot.com", "Logger_URL": "logger.talent-monitoring.com"}, "controller": {"Collect_Interval": 1, "Signal_Strength": 100, "Power_On_Time": 29, "Communication_Type": 1, "Connect_Count": 1, "Data_Up_Interval": 300}})        
+
+def test_parse_control2(Contr2DataSeq):
+    i = InfosG3()
+    for key, result in i.parse (Contr2DataSeq):
+        pass
+
+    assert json.dumps(i.db) == json.dumps(
+{"collector": {"Collector_Fw_Version": "RSW_400_V1.00.20", "Chip_Type": "Raymon", "Chip_Model": "RSW-1-10001", "Trace_URL": "t.raymoniot.com", "Logger_URL": "logger.talent-monitoring.com"}, "controller": {"Collect_Interval": 1, "Signal_Strength": 16, "Power_On_Time": 334, "Communication_Type": 1, "Connect_Count": 1, "Data_Up_Interval": 300}})
+
+def test_parse_inverter(InvDataSeq):
+    i = InfosG3()
+    for key, result in i.parse (InvDataSeq):
+        pass
+
+    assert json.dumps(i.db) == json.dumps(
+{"inverter": {"Product_Name": "Microinv", "Manufacturer": "TSUN", "Version": "V5.0.11", "Serial_Number": "T170000000000001", "Equipment_Model": "TSOL-MS600"}})
+
+def test_parse_cont_and_invert(ContrDataSeq, InvDataSeq):
+    i = InfosG3()
+    for key, result in i.parse (ContrDataSeq):
+        pass
+
+    for key, result in i.parse (InvDataSeq):
+        pass
+
+    assert json.dumps(i.db) == json.dumps(
+    {
+"collector": {"Collector_Fw_Version": "RSW_400_V1.00.06", "Chip_Type": "Raymon", "Chip_Model": "RSW-1-10001", "Trace_URL": "t.raymoniot.com", "Logger_URL": "logger.talent-monitoring.com"}, "controller": {"Collect_Interval": 1, "Signal_Strength": 100, "Power_On_Time": 29, "Communication_Type": 1, "Connect_Count": 1, "Data_Up_Interval": 300},
+"inverter": {"Product_Name": "Microinv", "Manufacturer": "TSUN", "Version": "V5.0.11", "Serial_Number": "T170000000000001", "Equipment_Model": "TSOL-MS600"}})
+
+
+def test_build_ha_conf1(ContrDataSeq):
+    i = InfosG3()
+    i.static_init()                # initialize counter
+
+    tests = 0
+    for d_json, comp, node_id, id in i.ha_confs(ha_prfx="tsun/", node_id="garagendach/", snr='123'):
+
+        if id == 'out_power_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/grid", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "out_power_123", "val_tpl": "{{value_json['Output_Power'] | float}}", "unit_of_meas": "W", "dev": {"name": "Micro Inverter", "sa": "Micro Inverter", "via_device": "controller_123", "ids": ["inverter_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'daily_gen_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Daily Generation", "stat_t": "tsun/garagendach/total", "dev_cla": "energy", "stat_cla": "total_increasing", "uniq_id": "daily_gen_123", "val_tpl": "{{value_json['Daily_Generation'] | float}}", "unit_of_meas": "kWh", "ic": "mdi:solar-power-variant", "dev": {"name": "Micro Inverter", "sa": "Micro Inverter", "via_device": "controller_123", "ids": ["inverter_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'power_pv1_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/input", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "power_pv1_123", "val_tpl": "{{ (value_json['pv1']['Power'] | float)}}", "unit_of_meas": "W", "dev": {"name": "Module PV1", "sa": "Module PV1", "via_device": "inverter_123", "ids": ["input_pv1_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'power_pv2_123':
+            assert False    # if we haven't received and parsed a control data msg, we don't know the number of inputs. In this case we only register the first one!!
+
+
+        elif id == 'signal_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Signal Strength", "stat_t": "tsun/garagendach/controller", "dev_cla": None, "stat_cla": "measurement", "uniq_id": "signal_123", "val_tpl": "{{value_json[\'Signal_Strength\'] | int}}", "unit_of_meas": "%", "ic": "mdi:wifi", "dev": {"name": "Controller", "sa": "Controller", "via_device": "proxy", "ids": ["controller_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+        elif id == 'inv_count_456':
+            assert False
+
+    assert tests==4
+
+
+    for d_json, comp, node_id, id in i.ha_proxy_confs(ha_prfx="tsun/", node_id = 'proxy/', snr = '456'):
+
+        if id == 'out_power_123':
+            assert False
+        elif id == 'daily_gen_123':
+            assert False
+        elif id == 'power_pv1_123':
+            assert False
+        elif id == 'power_pv2_123':
+            assert False    # if we haven't received and parsed a control data msg, we don't know the number of inputs. In this case we only register the first one!!
+
+        elif id == 'signal_123':
+            assert False
+        elif id == 'inv_count_456':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Active Inverter Connections", "stat_t": "tsun/proxy/proxy", "dev_cla": None, "stat_cla": None, "uniq_id": "inv_count_456", "val_tpl": "{{value_json['Inverter_Cnt'] | int}}", "ic": "mdi:counter", "dev": {"name": "Proxy", "sa": "Proxy", "mdl": "proxy", "mf": "Stefan Allius", "sw": "unknown", "ids": ["proxy"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+    assert tests==5
+
+def test_build_ha_conf2(ContrDataSeq, InvDataSeq, InvDataSeq2):
+    i = InfosG3()
+    for key, result in i.parse (ContrDataSeq):
+        pass
+    for key, result in i.parse (InvDataSeq):
+        pass
+    for key, result in i.parse (InvDataSeq2):
+        pass
+
+    tests = 0
+    for d_json, comp, node_id, id in i.ha_confs(ha_prfx="tsun/", node_id="garagendach/", snr='123', sug_area = 'roof'):
+
+        if id == 'out_power_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/grid", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "out_power_123", "val_tpl": "{{value_json['Output_Power'] | float}}", "unit_of_meas": "W", "dev": {"name": "Micro Inverter - roof", "sa": "Micro Inverter - roof", "via_device": "controller_123", "mdl": "TSOL-MS600", "mf": "TSUN", "sw": "V5.0.11", "ids": ["inverter_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        if id == 'daily_gen_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Daily Generation", "stat_t": "tsun/garagendach/total", "dev_cla": "energy", "stat_cla": "total_increasing", "uniq_id": "daily_gen_123", "val_tpl": "{{value_json['Daily_Generation'] | float}}", "unit_of_meas": "kWh", "ic": "mdi:solar-power-variant", "dev": {"name": "Micro Inverter - roof", "sa": "Micro Inverter - roof", "via_device": "controller_123", "mdl": "TSOL-MS600", "mf": "TSUN", "sw": "V5.0.11", "ids": ["inverter_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'power_pv1_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/input", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "power_pv1_123", "val_tpl": "{{ (value_json['pv1']['Power'] | float)}}", "unit_of_meas": "W", "dev": {"name": "Module PV1 - roof", "sa": "Module PV1 - roof", "via_device": "inverter_123", "ids": ["input_pv1_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'power_pv2_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/input", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "power_pv2_123", "val_tpl": "{{ (value_json['pv2']['Power'] | float)}}", "unit_of_meas": "W", "dev": {"name": "Module PV2 - roof", "sa": "Module PV2 - roof", "via_device": "inverter_123", "ids": ["input_pv2_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'signal_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Signal Strength", "stat_t": "tsun/garagendach/controller", "dev_cla": None, "stat_cla": "measurement", "uniq_id": "signal_123", "val_tpl": "{{value_json[\'Signal_Strength\'] | int}}", "unit_of_meas": "%", "ic": "mdi:wifi", "dev": {"name": "Controller - roof", "sa": "Controller - roof", "via_device": "proxy", "mdl": "RSW-1-10001", "mf": "Raymon", "sw": "RSW_400_V1.00.06", "ids": ["controller_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+    assert tests==5
+
+def test_must_incr_total(InvDataSeq2, InvDataSeq2_Zero):
+    i = InfosG3()
+    tests = 0
+    for key, update in i.parse (InvDataSeq2):
+        if key == 'total' or key == 'inverter' or key == 'env':
+            assert update == True
+            tests +=1   
+    assert tests==5
+    assert json.dumps(i.db['total']) == json.dumps({'Daily_Generation': 1.7, 'Total_Generation': 17.36})
+    assert json.dumps(i.db['input']) == json.dumps({"pv1": {"Voltage": 33.6, "Current": 1.91, "Power": 64.5, "Daily_Generation": 1.08, "Total_Generation": 9.74}, "pv2": {"Voltage": 33.5, "Current": 1.36, "Power": 45.7, "Daily_Generation": 0.62, "Total_Generation": 7.62}, "pv3": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0}, "pv4": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0}})
+    assert json.dumps(i.db['env']) == json.dumps({"Inverter_Temp": 23})
+    tests = 0
+    for key, update in i.parse (InvDataSeq2):
+        if key == 'total':
+            assert update == False
+            tests +=1   
+        elif key == 'env':
+            assert update == False
+            tests +=1
+
+    assert tests==3
+    assert json.dumps(i.db['total']) == json.dumps({'Daily_Generation': 1.7, 'Total_Generation': 17.36})
+    assert json.dumps(i.db['input']) == json.dumps({"pv1": {"Voltage": 33.6, "Current": 1.91, "Power": 64.5, "Daily_Generation": 1.08, "Total_Generation": 9.74}, "pv2": {"Voltage": 33.5, "Current": 1.36, "Power": 45.7, "Daily_Generation": 0.62, "Total_Generation": 7.62}, "pv3": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0}, "pv4": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0}})
+    assert json.dumps(i.db['env']) == json.dumps({"Inverter_Temp": 23})
+    assert json.dumps(i.db['inverter']) == json.dumps({"Rated_Power": 600, "No_Inputs": 2})
+        
+    tests = 0
+    for key, update in i.parse (InvDataSeq2_Zero):
+        if key == 'total':
+            assert update == False
+            tests +=1   
+        elif key == 'env':
+            assert update == True
+            tests +=1   
+
+    assert tests==3
+    assert json.dumps(i.db['total']) == json.dumps({'Daily_Generation': 1.7, 'Total_Generation': 17.36})
+    assert json.dumps(i.db['input']) == json.dumps({"pv1": {"Voltage": 33.6, "Current": 1.91, "Power": 0.0, "Daily_Generation": 1.08, "Total_Generation": 9.74}, "pv2": {"Voltage": 33.5, "Current": 1.36, "Power": 0.0, "Daily_Generation": 0.62, "Total_Generation": 7.62}, "pv3": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0}, "pv4": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0}})
+    assert json.dumps(i.db['env']) == json.dumps({"Inverter_Temp": 0})
+
+def test_must_incr_total2(InvDataSeq2, InvDataSeq2_Zero):
+    i = InfosG3()
+    tests = 0
+    for key, update in i.parse (InvDataSeq2_Zero):
+        if key == 'total':
+            assert update == False
+            tests +=1   
+        elif key == 'env':
+            assert update == True
+            tests +=1
+
+    assert tests==3
+    assert json.dumps(i.db['total']) == json.dumps({})
+    assert json.dumps(i.db['input']) == json.dumps({"pv1": {"Voltage": 33.6, "Current": 1.91, "Power": 0.0}, "pv2": {"Voltage": 33.5, "Current": 1.36, "Power": 0.0}, "pv3": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0}, "pv4": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0}})
+    assert json.dumps(i.db['env']) == json.dumps({"Inverter_Temp": 0})
+    
+    tests = 0
+    for key, update in i.parse (InvDataSeq2_Zero):
+        if key == 'total':
+            assert update == False
+            tests +=1   
+        elif key == 'env':
+            assert update == False
+            tests +=1
+
+    assert tests==3
+    assert json.dumps(i.db['total']) == json.dumps({})
+    assert json.dumps(i.db['input']) == json.dumps({"pv1": {"Voltage": 33.6, "Current": 1.91, "Power": 0.0}, "pv2": {"Voltage": 33.5, "Current": 1.36, "Power": 0.0}, "pv3": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0}, "pv4": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0}})
+    assert json.dumps(i.db['env']) == json.dumps({"Inverter_Temp": 0})
+
+    tests = 0
+    for key, update in i.parse (InvDataSeq2):
+        if key == 'total':
+            assert update == True
+            tests +=1   
+        elif key == 'env':
+            assert update == True
+            tests +=1
+
+    assert tests==3
+    assert json.dumps(i.db['total']) == json.dumps({'Daily_Generation': 1.7, 'Total_Generation': 17.36})
+    assert json.dumps(i.db['input']) == json.dumps({"pv1": {"Voltage": 33.6, "Current": 1.91, "Power": 64.5, "Daily_Generation": 1.08, "Total_Generation": 9.74}, "pv2": {"Voltage": 33.5, "Current": 1.36, "Power": 45.7, "Daily_Generation": 0.62, "Total_Generation": 7.62}, "pv3": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0}, "pv4": {"Voltage": 0.0, "Current": 0.0, "Power": 0.0}})
+   
+def test_new_data_types(InvDataNew):
+    i = InfosG3()
+    tests = 0
+    for key, update in i.parse (InvDataNew):
+        if key == 'events':
+            tests +=1
+        elif key == 'inverter':
+            assert update == True
+            tests +=1
+        elif key == 'input':
+            assert update == False
+            tests +=1
+        else:
+            assert False
+
+    assert tests==15
+    assert json.dumps(i.db['inverter']) == json.dumps({"Manufacturer": 0})
+    assert json.dumps(i.db['input']) == json.dumps({"pv1": {}})
+    assert json.dumps(i.db['events']) == json.dumps({"401_": 0, "404_": 0, "405_": 0, "408_": 0, "409_No_Utility": 0, "406_": 0, "416_": 0})
+
+def test_invalid_data_type(InvalidDataSeq):
+    i = InfosG3()
+    i.static_init()                # initialize counter
+
+    val = i.dev_value(Register.INVALID_DATA_TYPE)  # check invalid data type counter
+    assert val == 0
+
+
+    for key, result in i.parse (InvalidDataSeq):
+        pass
+    assert json.dumps(i.db) == json.dumps({"inverter": {"Product_Name": "Microinv"}})
+
+    val = i.dev_value(Register.INVALID_DATA_TYPE)  # check invalid data type counter
+    assert val == 1

app/tests/test_infos_g3p.py

@@ -0,0 +1,260 @@
+
+# test_with_pytest.py
+import pytest, json
+from app.src.infos import Register
+from app.src.gen3plus.infos_g3p import InfosG3P
+from app.src.gen3plus.infos_g3p import RegisterMap
+
+@pytest.fixture
+def DeviceData(): # 0x4110 ftype: 0x02
+    msg  = b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\xba\xd2\x00\x00'
+    msg += b'\x19\x00\x00\x00\x00\x00\x00\x00\x05\x3c\x78\x01\x64\x01\x4c\x53'
+    msg += b'\x57\x35\x42\x4c\x45\x5f\x31\x37\x5f\x30\x32\x42\x30\x5f\x31\x2e'
+    msg += b'\x30\x35\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x40\x2a\x8f\x4f\x51\x54\x31\x39\x32\x2e'
+    msg += b'\x31\x36\x38\x2e\x38\x30\x2e\x34\x39\x00\x00\x00\x0f\x00\x01\xb0'
+    msg += b'\x02\x0f\x00\xff\x56\x31\x2e\x31\x2e\x30\x30\x2e\x30\x42\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xfe\xfe\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x41\x6c\x6c\x69\x75\x73\x2d\x48\x6f'
+    msg += b'\x6d\x65\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' 
+    return msg
+
+@pytest.fixture
+def InverterData():  # 0x4210 ftype: 0x01
+    msg  = b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\xb0\x02\xbc\xc8'
+    msg += b'\x24\x32\x6c\x1f\x00\x00\xa0\x47\xe4\x33\x01\x00\x03\x08\x00\x00'
+    msg += b'\x59\x31\x37\x45\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30\x45'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x01\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x40\x10\x08\xc8\x00\x49\x13\x8d\x00\x36\x00\x00\x02\x58\x06\x7a'
+    msg += b'\x01\x61\x00\xa8\x02\x54\x01\x5a\x00\x8a\x01\xe4\x01\x5a\x00\xbd'
+    msg += b'\x02\x8f\x00\x11\x00\x01\x00\x00\x00\x0b\x00\x00\x27\x98\x00\x04'
+    msg += b'\x00\x00\x0c\x04\x00\x03\x00\x00\x0a\xe7\x00\x05\x00\x00\x0c\x75'
+    msg += b'\x00\x00\x00\x00\x06\x16\x02\x00\x00\x00\x55\xaa\x00\x01\x00\x00'
+    msg += b'\x00\x00\x00\x00\xff\xff\x07\xd0\x00\x03\x04\x00\x04\x00\x04\x00'
+    msg += b'\x04\x00\x00\x01\xff\xff\x00\x01\x00\x06\x00\x68\x00\x68\x05\x00'
+    msg += b'\x09\xcd\x07\xb6\x13\x9c\x13\x24\x00\x01\x07\xae\x04\x0f\x00\x41'
+    msg += b'\x00\x0f\x0a\x64\x0a\x64\x00\x06\x00\x06\x09\xf6\x12\x8c\x12\x8c'
+    msg += b'\x00\x10\x00\x10\x14\x52\x14\x52\x00\x10\x00\x10\x01\x51\x00\x05'
+    msg += b'\x04\x00\x00\x01\x13\x9c\x0f\xa0\x00\x4e\x00\x66\x03\xe8\x04\x00'
+    msg += b'\x09\xce\x07\xa8\x13\x9c\x13\x26\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x04\x00\x04\x00\x00\x00\x00\x00\xff\xff\x00\x00'
+    msg += b'\x00\x00\x00\x00'
+    return msg
+
+
+def test_default_db():
+    i = InfosG3P(client_mode=False)
+    
+    assert json.dumps(i.db) == json.dumps({
+        "inverter": {"Manufacturer": "TSUN", "Equipment_Model": "TSOL-MSxx00", "No_Inputs": 4}, 
+        "collector": {"Chip_Type": "IGEN TECH"},
+        })
+
+def test_parse_4110(DeviceData: bytes):
+    i = InfosG3P(client_mode=False)
+    i.db.clear()
+    for key, update in i.parse (DeviceData, 0x41, 2):
+        pass
+
+    assert json.dumps(i.db) == json.dumps({
+        'controller': {"Data_Up_Interval": 300, "Collect_Interval": 1, "Heartbeat_Interval": 120, "Signal_Strength": 100, "IP_Address": "192.168.80.49"},
+        'collector': {"Chip_Model": "LSW5BLE_17_02B0_1.05", "Collector_Fw_Version": "V1.1.00.0B"},
+        })
+
+def test_parse_4210(InverterData: bytes):
+    i = InfosG3P(client_mode=False)
+    i.db.clear()
+    
+    for key, update in i.parse (InverterData, 0x42, 1):
+        pass
+
+    assert json.dumps(i.db) == json.dumps({
+         "controller": {"Power_On_Time": 2051}, 
+         "inverter": {"Serial_Number": "Y17E00000000000E", "Version": "V4.0.10", "Rated_Power": 600, "Max_Designed_Power": 2000}, 
+         "env": {"Inverter_Status": 1, "Inverter_Temp": 14}, 
+         "grid": {"Voltage": 224.8, "Current": 0.73, "Frequency": 50.05, "Output_Power": 165.8}, 
+         "input": {"pv1": {"Voltage": 35.3, "Current": 1.68, "Power": 59.6, "Daily_Generation": 0.04, "Total_Generation": 30.76}, 
+                   "pv2": {"Voltage": 34.6, "Current": 1.38, "Power": 48.4, "Daily_Generation": 0.03, "Total_Generation": 27.91}, 
+                   "pv3": {"Voltage": 34.6, "Current": 1.89, "Power": 65.5, "Daily_Generation": 0.05, "Total_Generation": 31.89}, 
+                   "pv4": {"Voltage": 1.7, "Current": 0.01, "Power": 0.0, "Total_Generation": 15.58}}, 
+         "total": {"Daily_Generation": 0.11, "Total_Generation": 101.36}
+        })
+    
+def test_build_ha_conf1():
+    i = InfosG3P(client_mode=False)
+    i.static_init()                # initialize counter
+
+    tests = 0
+    for d_json, comp, node_id, id in i.ha_confs(ha_prfx="tsun/", node_id="garagendach/", snr='123'):
+
+        if id == 'out_power_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/grid", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "out_power_123", "val_tpl": "{{value_json['Output_Power'] | float}}", "unit_of_meas": "W", "dev": {"name": "Micro Inverter", "sa": "Micro Inverter", "via_device": "controller_123", "mdl": "TSOL-MSxx00", "mf": "TSUN", "ids": ["inverter_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'daily_gen_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Daily Generation", "stat_t": "tsun/garagendach/total", "dev_cla": "energy", "stat_cla": "total_increasing", "uniq_id": "daily_gen_123", "val_tpl": "{{value_json['Daily_Generation'] | float}}", "unit_of_meas": "kWh", "ic": "mdi:solar-power-variant", "dev": {"name": "Micro Inverter", "sa": "Micro Inverter", "via_device": "controller_123", "mdl": "TSOL-MSxx00", "mf": "TSUN", "ids": ["inverter_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'power_pv1_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/input", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "power_pv1_123", "val_tpl": "{{ (value_json['pv1']['Power'] | float)}}", "unit_of_meas": "W", "dev": {"name": "Module PV1", "sa": "Module PV1", "via_device": "inverter_123", "ids": ["input_pv1_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'power_pv2_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/input", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "power_pv2_123", "val_tpl": "{{ (value_json['pv2']['Power'] | float)}}", "unit_of_meas": "W", "dev": {"name": "Module PV2", "sa": "Module PV2", "via_device": "inverter_123", "ids": ["input_pv2_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'power_pv3_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/input", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "power_pv3_123", "val_tpl": "{{ (value_json['pv3']['Power'] | float)}}", "unit_of_meas": "W", "dev": {"name": "Module PV3", "sa": "Module PV3", "via_device": "inverter_123", "ids": ["input_pv3_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'power_pv4_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/input", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "power_pv4_123", "val_tpl": "{{ (value_json['pv4']['Power'] | float)}}", "unit_of_meas": "W", "dev": {"name": "Module PV4", "sa": "Module PV4", "via_device": "inverter_123", "ids": ["input_pv4_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'signal_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Signal Strength", "stat_t": "tsun/garagendach/controller", "dev_cla": None, "stat_cla": "measurement", "uniq_id": "signal_123", "val_tpl": "{{value_json[\'Signal_Strength\'] | int}}", "unit_of_meas": "%", "ic": "mdi:wifi", "dev": {"name": "Controller", "sa": "Controller", "via_device": "proxy", "mf": "IGEN TECH", "ids": ["controller_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+        elif id == 'inv_count_456':
+            assert False
+
+    assert tests==7
+
+
+    for d_json, comp, node_id, id in i.ha_proxy_confs(ha_prfx="tsun/", node_id = 'proxy/', snr = '456'):
+
+        if id == 'out_power_123':
+            assert False
+        elif id == 'daily_gen_123':
+            assert False
+        elif id == 'power_pv1_123':
+            assert False
+        elif id == 'power_pv2_123':
+            assert False
+        elif id == 'power_pv3_123':
+            assert False
+        elif id == 'power_pv4_123':
+            assert False
+        elif id == 'signal_123':
+            assert False
+        elif id == 'inv_count_456':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Active Inverter Connections", "stat_t": "tsun/proxy/proxy", "dev_cla": None, "stat_cla": None, "uniq_id": "inv_count_456", "val_tpl": "{{value_json['Inverter_Cnt'] | int}}", "ic": "mdi:counter", "dev": {"name": "Proxy", "sa": "Proxy", "mdl": "proxy", "mf": "Stefan Allius", "sw": "unknown", "ids": ["proxy"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+    assert tests==8
+
+def test_build_ha_conf2():
+    i = InfosG3P(client_mode=True)
+    i.static_init()                # initialize counter
+
+    tests = 0
+    for d_json, comp, node_id, id in i.ha_confs(ha_prfx="tsun/", node_id="garagendach/", snr='123'):
+
+        if id == 'out_power_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/grid", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "out_power_123", "val_tpl": "{{value_json['Output_Power'] | float}}", "unit_of_meas": "W", "dev": {"name": "Micro Inverter", "sa": "Micro Inverter", "via_device": "controller_123", "mdl": "TSOL-MSxx00", "mf": "TSUN", "ids": ["inverter_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'daily_gen_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Daily Generation", "stat_t": "tsun/garagendach/total", "dev_cla": "energy", "stat_cla": "total_increasing", "uniq_id": "daily_gen_123", "val_tpl": "{{value_json['Daily_Generation'] | float}}", "unit_of_meas": "kWh", "ic": "mdi:solar-power-variant", "dev": {"name": "Micro Inverter", "sa": "Micro Inverter", "via_device": "controller_123", "mdl": "TSOL-MSxx00", "mf": "TSUN", "ids": ["inverter_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'power_pv1_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/input", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "power_pv1_123", "val_tpl": "{{ (value_json['pv1']['Power'] | float)}}", "unit_of_meas": "W", "dev": {"name": "Module PV1", "sa": "Module PV1", "via_device": "inverter_123", "ids": ["input_pv1_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'power_pv2_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/input", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "power_pv2_123", "val_tpl": "{{ (value_json['pv2']['Power'] | float)}}", "unit_of_meas": "W", "dev": {"name": "Module PV2", "sa": "Module PV2", "via_device": "inverter_123", "ids": ["input_pv2_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'power_pv3_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/input", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "power_pv3_123", "val_tpl": "{{ (value_json['pv3']['Power'] | float)}}", "unit_of_meas": "W", "dev": {"name": "Module PV3", "sa": "Module PV3", "via_device": "inverter_123", "ids": ["input_pv3_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'power_pv4_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Power", "stat_t": "tsun/garagendach/input", "dev_cla": "power", "stat_cla": "measurement", "uniq_id": "power_pv4_123", "val_tpl": "{{ (value_json['pv4']['Power'] | float)}}", "unit_of_meas": "W", "dev": {"name": "Module PV4", "sa": "Module PV4", "via_device": "inverter_123", "ids": ["input_pv4_123"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+        elif id == 'signal_123':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({})
+            tests +=1
+        elif id == 'inv_count_456':
+            assert False
+
+    assert tests==7
+
+
+    for d_json, comp, node_id, id in i.ha_proxy_confs(ha_prfx="tsun/", node_id = 'proxy/', snr = '456'):
+
+        if id == 'out_power_123':
+            assert False
+        elif id == 'daily_gen_123':
+            assert False
+        elif id == 'power_pv1_123':
+            assert False
+        elif id == 'power_pv2_123':
+            assert False
+        elif id == 'power_pv3_123':
+            assert False
+        elif id == 'power_pv4_123':
+            assert False
+        elif id == 'signal_123':
+            assert False
+        elif id == 'inv_count_456':
+            assert comp == 'sensor'
+            assert  d_json == json.dumps({"name": "Active Inverter Connections", "stat_t": "tsun/proxy/proxy", "dev_cla": None, "stat_cla": None, "uniq_id": "inv_count_456", "val_tpl": "{{value_json['Inverter_Cnt'] | int}}", "ic": "mdi:counter", "dev": {"name": "Proxy", "sa": "Proxy", "mdl": "proxy", "mf": "Stefan Allius", "sw": "unknown", "ids": ["proxy"]}, "o": {"name": "proxy", "sw": "unknown"}})
+            tests +=1
+
+    assert tests==8
+
+def test_exception_and_eval(InverterData: bytes):
+
+    # add eval to convert temperature from °F to °C
+    RegisterMap.map[0x420100d8]['eval'] =  '(result-32)/1.8'
+    # map PV1_VOLTAGE to invalid register  
+    RegisterMap.map[0x420100e0]['reg'] = Register.TEST_REG2
+    # set invalid maping entry for OUTPUT_POWER (string instead of dict type) 
+    Backup = RegisterMap.map[0x420100de]
+    RegisterMap.map[0x420100de] = 'invalid_entry'
+
+    i = InfosG3P(client_mode=False)
+    # i.db.clear()
+    
+    for key, update in i.parse (InverterData, 0x42, 1):
+        pass
+    assert 12.2222 == round (i.get_db_value(Register.INVERTER_TEMP, 0),4)
+    
+    del RegisterMap.map[0x420100d8]['eval'] # remove eval
+    RegisterMap.map[0x420100e0]['reg'] = Register.PV1_VOLTAGE # reset mapping
+    RegisterMap.map[0x420100de] = Backup # reset mapping
+    
+    for key, update in i.parse (InverterData, 0x42, 1):
+        pass
+    assert 54 == i.get_db_value(Register.INVERTER_TEMP, 0)
+ 

app/tests/test_messages.py

@@ -1,201 +0,0 @@
-# test_with_pytest.py
-import pytest
-from app.src.messages import Message, Control
-from app.src.config import Config
-from app.src.infos import Infos
-
-# initialize the proxy statistics
-Infos.static_init()
-
-class MemoryStream(Message):
-    def __init__(self, msg, chunks = (0,)):
-        super().__init__()
-        self.__msg = msg
-        self.__msg_len = len(msg)
-        self.__chunks = chunks
-        self.__offs = 0
-        self.__chunk_idx = 0
-        self.msg_count = 0
-        self.server_side = False
-        self.addr = 'Test: SrvSide'
-
-    def _read(self) -> int:
-        copied_bytes = 0
-        try:    
-            if (self.__offs < self.__msg_len):
-                len = self.__chunks[self.__chunk_idx]
-                self.__chunk_idx += 1
-                if len!=0:
-                    self._recv_buffer += self.__msg[self.__offs:len]
-                    copied_bytes = len - self.__offs
-                    self.__offs = len
-                else:
-                    self._recv_buffer += self.__msg[self.__offs:]
-                    copied_bytes = self.__msg_len - self.__offs
-                    self.__offs = self.__msg_len
-        except:
-            pass     
-        return copied_bytes
-    
-    
-    def _Message__flush_recv_msg(self) -> None:
-        super()._Message__flush_recv_msg()
-        self.msg_count += 1
-        return
-    
-    def __del__ (self):
-        super().__del__()
-
-
-@pytest.fixture
-def MsgContactInfo(): # Contact Info message
-    Config.config = {'tsun':{'enabled': True}}
-    return b'\x00\x00\x00\x2c\x10R170000000000001\x91\x00\x08solarhub\x0fsolarhub\x40123456'
-
-@pytest.fixture
-def MsgContactInfo_LongId():  # Contact Info message with longer ID
-    Config.config = {'tsun':{'enabled': True}}
-    return b'\x00\x00\x00\x2d\x11R1700000000000011\x91\x00\x08solarhub\x0fsolarhub\x40123456'
-
-@pytest.fixture
-def Msg2ContactInfo(): # two Contact Info messages
-    Config.config = {'tsun':{'enabled': True}}
-    return b'\x00\x00\x00\x2c\x10R170000000000001\x91\x00\x08solarhub\x0fsolarhub\x40123456\x00\x00\x00\x2c\x10R170000000000002\x91\x00\x08solarhub\x0fsolarhub\x40123456'
-
-
-
-
-def test_read_message(MsgContactInfo):
-    m = MemoryStream(MsgContactInfo, (0,))
-    m.read()         # read complete msg, and dispatch msg
-    assert not m.header_valid  # must be invalid, since msg was handled and buffer flushed
-    assert m.msg_count == 1
-    assert m.id_str == b"R170000000000001" 
-    assert int(m.ctrl)==145
-    assert m.msg_id==0 
-    assert m.header_len==23
-    assert m.data_len==25
-    m.close()
-
- 
-def test_read_message_long_id(MsgContactInfo_LongId):
-    m = MemoryStream(MsgContactInfo_LongId, (23,24))
-    m.read()        # read 23 bytes, one is missing
-    assert not m.header_valid  # must be invalid, since header not complete
-    assert m.msg_count == 0
-    m.read()        # read the missing byte
-    assert m.header_valid      # must be valid, since header is complete but not the msg
-    assert m.msg_count == 0
-    assert m.id_str == b"R1700000000000011" 
-    assert int(m.ctrl)==145
-    assert m.msg_id==0 
-    assert m.header_len==24
-    assert m.data_len==25
-    m.read()        # try to read rest of message, but there is no chunk available
-    assert m.header_valid      # must be valid, since header is complete but not the msg
-    assert m.msg_count == 0
-    m.close()
-    
-
-def test_read_message_in_chunks(MsgContactInfo):
-    m = MemoryStream(MsgContactInfo, (4,23,0))
-    m.read()        # read 4 bytes, header incomplere
-    assert not m.header_valid  # must be invalid, since header not complete
-    assert m.msg_count == 0
-    m.read()        # read missing bytes for complete header
-    assert m.header_valid      # must be valid, since header is complete but not the msg
-    assert m.msg_count == 0
-    assert m.id_str == b"R170000000000001" 
-    assert int(m.ctrl)==145
-    assert m.msg_id==0 
-    assert m.header_len==23
-    assert m.data_len==25
-    m.read()    # read rest of message
-    assert not m.header_valid  # must be invalid, since msg was handled and buffer flushed
-    assert m.msg_count == 1
-    m.close()
-    
-def test_read_message_in_chunks2(MsgContactInfo):
-    m = MemoryStream(MsgContactInfo, (4,10,0))
-    m.read()        # read 4 bytes, header incomplere
-    assert not m.header_valid
-    assert m.msg_count == 0
-    m.read()        # read 6 more bytes, header incomplere
-    assert not m.header_valid
-    assert m.msg_count == 0
-    m.read()        # read rest of message
-    assert not m.header_valid  # must be invalid, since msg was handled and buffer flushed
-    assert m.header_len==23
-    assert m.data_len==25
-    assert m.id_str == b"R170000000000001" 
-    assert int(m.ctrl)==145
-    assert m.msg_id==0 
-    assert m.msg_count == 1
-    while m.read(): # read rest of message
-        pass
-    assert m.msg_count == 1
-    assert not m.header_valid  # must be invalid, since msg was handled and buffer flushed
-    m.close()
-
-def test_read_two_messages(Msg2ContactInfo):
-    m = MemoryStream(Msg2ContactInfo, (0,))
-    m.read()         # read complete msg, and dispatch msg
-    assert not m.header_valid  # must be invalid, since msg was handled and buffer flushed
-    assert m.msg_count == 1
-    assert m.id_str == b"R170000000000001" 
-    assert int(m.ctrl)==145
-    assert m.msg_id==0 
-    assert m.header_len==23
-    assert m.data_len==25
-    m.read()         # read complete msg, and dispatch msg
-    assert not m.header_valid  # must be invalid, since msg was handled and buffer flushed
-    assert m.msg_count == 2
-    assert m.id_str == b"R170000000000002" 
-    assert int(m.ctrl)==145
-    assert m.msg_id==0 
-    assert m.header_len==23
-    assert m.data_len==25
-    m.close()
-
-def test_ctrl_byte():
-    c = Control(0x91)
-    assert c.is_ind()
-    assert not c.is_resp()    
-    c = Control(0x99)
-    assert not c.is_ind()
-    assert c.is_resp()    
-
-    
-def test_msg_iterator():
-    m1 = Message()
-    m2 = Message()
-    m3 = Message()
-    m3.close()
-    del m3
-    test1 = 0
-    test2 = 0
-    for key in Message:
-        if key == m1:
-            test1+=1
-        elif key == m2:
-            test2+=1
-        else:
-            assert False
-    assert test1 == 1
-    assert test2 == 1
-
-def test_proxy_counter():
-    m = Message()
-    assert m.new_data == {}
-    assert 'proxy' in m.db.stat
-    assert 0 == m.db.stat['proxy']['Unknown_Msg']
-
-    m.inc_counter('Unknown_Msg')
-    assert m.new_data == {'proxy': True}
-    assert 1 == m.db.stat['proxy']['Unknown_Msg']
-
-    m.new_data['proxy'] =  False
-    m.dec_counter('Unknown_Msg')
-    assert m.new_data == {'proxy': True}
-    assert 0 == m.db.stat['proxy']['Unknown_Msg']
-    m.close()

app/tests/test_modbus.py

@@ -0,0 +1,398 @@
+# test_with_pytest.py
+import pytest
+import asyncio
+from app.src.modbus import Modbus
+from app.src.infos import Infos, Register
+
+pytest_plugins = ('pytest_asyncio',)
+# pytestmark = pytest.mark.asyncio(scope="module")
+
+class ModbusTestHelper(Modbus):
+    def __init__(self):
+        super().__init__(self.send_cb)
+        self.db = Infos()
+        self.pdu = None
+        self.send_calls = 0
+        self.recv_responses = 0
+    def send_cb(self, pdu: bytearray, log_lvl: int, state: str):
+        self.pdu = pdu
+        self.send_calls += 1
+    def resp_handler(self):
+        self.recv_responses += 1
+
+def test_modbus_crc():
+    '''Check CRC-16 calculation'''
+    mb = Modbus(None)
+    assert 0x0b02 == mb._Modbus__calc_crc(b'\x01\x06\x20\x08\x00\x04')
+    assert 0 == mb._Modbus__calc_crc(b'\x01\x06\x20\x08\x00\x04\x02\x0b')
+    assert mb._Modbus__check_crc(b'\x01\x06\x20\x08\x00\x04\x02\x0b')
+
+    assert 0xc803 == mb._Modbus__calc_crc(b'\x01\x06\x20\x08\x00\x00')
+    assert 0 == mb._Modbus__calc_crc(b'\x01\x06\x20\x08\x00\x00\x03\xc8')
+    assert mb._Modbus__check_crc(b'\x01\x06\x20\x08\x00\x00\x03\xc8')
+
+    assert 0x5c75 == mb._Modbus__calc_crc(b'\x01\x03\x08\x01\x2c\x00\x2c\x02\x2c\x2c\x46')
+    msg = b'\x01\x03\x28\x51'
+    msg += b'\x0e\x08\xd3\x00\x29\x13\x87\x00\x3e\x00\x00\x01\x2c\x03\xb4\x00'
+    msg += b'\x08\x00\x00\x00\x00\x01\x59\x01\x21\x03\xe6\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\xe6\xef'
+    assert 0 == mb._Modbus__calc_crc(msg)
+
+def test_build_modbus_pdu():
+    '''Check building and sending a MODBUS RTU'''
+    mb = ModbusTestHelper()
+    mb.build_msg(1,6,0x2000,0x12)
+    assert mb.pdu == b'\x01\x06\x20\x00\x00\x12\x02\x07'
+    assert mb._Modbus__check_crc(mb.pdu)
+    assert mb.last_addr == 1
+    assert mb.last_fcode == 6   
+    assert mb.last_reg == 0x2000
+    assert mb.last_len == 18
+    assert mb.err == 0
+
+def test_recv_req():
+    '''Receive a valid request, which must transmitted'''
+    mb = ModbusTestHelper()
+    assert mb.recv_req(b'\x01\x06\x20\x00\x00\x12\x02\x07')
+    assert mb.last_fcode == 6
+    assert mb.last_reg == 0x2000
+    assert mb.last_len == 0x12
+    assert mb.err == 0
+
+def test_recv_req_crc_err():
+    '''Receive a request with invalid CRC, which must be dropped'''
+    mb = ModbusTestHelper()
+    assert not mb.recv_req(b'\x01\x06\x20\x00\x00\x12\x02\x08')
+    assert mb.pdu == None
+    assert mb.last_fcode == 0   
+    assert mb.last_reg == 0
+    assert mb.last_len == 0
+    assert mb.err == 1
+
+def test_recv_resp_crc_err():
+    '''Receive a response with invalid CRC, which must be dropped'''
+    mb = ModbusTestHelper()
+    # simulate a transmitted request
+    mb.req_pend = True
+    mb.last_addr = 1
+    mb.last_fcode = 3   
+    mb.last_reg == 0x300e
+    mb.last_len == 2
+    # check matching response, but with CRC error
+    call = 0
+    for key, update, val in mb.recv_resp(mb.db, b'\x01\x03\x04\x01\x2c\x00\x46\xbb\xf3', 'test'):
+        call += 1
+    assert mb.err == 1
+    assert 0 == call
+    assert mb.req_pend == True
+    # cleanup queue
+    mb._Modbus__stop_timer()
+    assert not mb.req_pend
+
+def test_recv_resp_invalid_addr():
+    '''Receive a response with wrong server addr, which must be dropped'''
+    mb = ModbusTestHelper()
+    mb.req_pend = True
+    # simulate a transmitted request
+    mb.last_addr = 1
+    mb.last_fcode = 3   
+    mb.last_reg == 0x300e
+    mb.last_len == 2
+
+    # check not matching response, with wrong server addr
+    call = 0
+    for key, update in mb.recv_resp(mb.db, b'\x02\x03\x04\x01\x2c\x00\x46\x88\xf4', 'test'):
+        call += 1
+    assert mb.err == 2
+    assert 0 == call
+    assert mb.req_pend == True
+    assert mb.que.qsize() == 0
+    
+    # cleanup queue
+    mb._Modbus__stop_timer()
+    assert not mb.req_pend
+
+def test_recv_recv_fcode():
+    '''Receive a response with wrong function code, which must be dropped'''
+    mb = ModbusTestHelper()
+    mb.build_msg(1,4,0x300e,2)
+    assert mb.que.qsize() == 0
+    assert mb.req_pend
+   
+    # check not matching response, with wrong function code
+    call = 0
+    for key, update, val in mb.recv_resp(mb.db, b'\x01\x03\x04\x01\x2c\x00\x46\xbb\xf4', 'test'):
+        call += 1
+
+    assert mb.err == 3
+    assert 0 == call
+    assert mb.req_pend == True
+    assert mb.que.qsize() == 0
+
+    # cleanup queue
+    mb._Modbus__stop_timer()
+    assert not mb.req_pend
+
+def test_recv_resp_len():
+    '''Receive a response with wrong data length, which must be dropped'''
+    mb = ModbusTestHelper()
+    mb.build_msg(1,3,0x300e,3)
+    assert mb.que.qsize() == 0
+    assert mb.req_pend
+    assert mb.last_len == 3
+
+    # check not matching response, with wrong data length
+    call = 0
+    for key, update, _ in mb.recv_resp(mb.db, b'\x01\x03\x04\x01\x2c\x00\x46\xbb\xf4', 'test'):
+        call += 1
+
+    assert mb.err == 4
+    assert 0 == call
+    assert mb.req_pend == True
+    assert mb.que.qsize() == 0
+
+    # cleanup queue
+    mb._Modbus__stop_timer()
+    assert not mb.req_pend
+
+def test_recv_unexpect_resp():
+    '''Receive a response when we havb't sent a request'''
+    mb = ModbusTestHelper()
+    assert not mb.req_pend
+   
+    # check unexpected response, which must be dropped
+    call = 0
+    for key, update, val in mb.recv_resp(mb.db, b'\x01\x03\x04\x01\x2c\x00\x46\xbb\xf4', 'test'):
+        call += 1
+
+    assert mb.err == 5
+    assert 0 == call
+    assert mb.req_pend == False
+    assert mb.que.qsize() == 0
+
+def test_parse_resp():
+    '''Receive matching response and parse the values'''
+    mb = ModbusTestHelper()
+    mb.build_msg(1,3,0x3007,6)
+    assert mb.que.qsize() == 0
+    assert mb.req_pend
+
+    call = 0
+    exp_result = ['V0.0.2C', 4.4, 0.7, 0.7, 30]
+    for key, update, val in mb.recv_resp(mb.db, b'\x01\x03\x0c\x01\x2c\x00\x2c\x00\x2c\x00\x46\x00\x46\x00\x46\x32\xc8', 'test'):
+        if key == 'grid':
+            assert update == True
+        elif key == 'inverter':
+            assert update == True
+        elif key == 'env':
+            assert update == True
+        else:
+            assert False
+        assert exp_result[call] == val
+        call += 1
+    assert 0 == mb.err
+    assert 5 == call
+    assert mb.que.qsize() == 0
+    assert not mb.req_pend
+
+def test_queue():
+    mb = ModbusTestHelper()
+    mb.build_msg(1,3,0x3022,4)
+    assert mb.que.qsize() == 0
+    assert mb.req_pend
+
+    assert mb.send_calls == 1
+    assert mb.pdu == b'\x01\x030"\x00\x04\xeb\x03'
+    mb.pdu = None
+    assert mb.send_calls == 1
+    assert mb.pdu == None
+
+    assert mb.que.qsize() == 0
+
+    # cleanup queue
+    mb._Modbus__stop_timer()
+    assert not mb.req_pend
+
+def test_queue2():
+    '''Check queue handling for build_msg() calls'''
+    mb = ModbusTestHelper()
+    mb.build_msg(1,3,0x3007,6)
+    mb.build_msg(1,6,0x2008,4)
+    assert mb.que.qsize() == 1
+    assert mb.req_pend
+    mb.build_msg(1,3,0x3007,6)
+    assert mb.que.qsize() == 2
+    assert mb.req_pend
+
+    assert mb.send_calls == 1
+    assert mb.pdu == b'\x01\x030\x07\x00\x06{\t'
+    call = 0
+    exp_result = ['V0.0.2C', 4.4, 0.7, 0.7, 30]
+    for key, update, val in mb.recv_resp(mb.db, b'\x01\x03\x0c\x01\x2c\x00\x2c\x00\x2c\x00\x46\x00\x46\x00\x46\x32\xc8', 'test'):
+        if key == 'grid':
+            assert update == True
+        elif key == 'inverter':
+            assert update == True
+        elif key == 'env':
+            assert update == True
+        else:
+            assert False
+        assert exp_result[call] == val
+        call += 1
+    assert 0 == mb.err
+    assert 5 == call
+
+    assert mb.que.qsize() == 1
+    assert mb.send_calls == 2
+    assert mb.pdu == b'\x01\x06\x20\x08\x00\x04\x02\x0b'
+
+    for key, update, val in mb.recv_resp(mb.db, b'\x01\x06\x20\x08\x00\x04\x02\x0b', 'test'):
+        pass
+
+    assert mb.que.qsize() == 0
+    assert mb.send_calls == 3
+    assert mb.pdu == b'\x01\x030\x07\x00\x06{\t'
+    call = 0
+    for key, update, val in mb.recv_resp(mb.db, b'\x01\x03\x0c\x01\x2c\x00\x2c\x00\x2c\x00\x46\x00\x46\x00\x46\x32\xc8', 'test'):
+        call += 1
+    assert 0 == mb.err
+    assert 5 == call
+
+    assert mb.que.qsize() == 0
+    assert not mb.req_pend
+
+def test_queue3():
+    '''Check queue handling for recv_req() calls'''
+    mb = ModbusTestHelper()
+    assert mb.recv_req(b'\x01\x03\x30\x07\x00\x06{\t', mb.resp_handler)
+    assert mb.recv_req(b'\x01\x06\x20\x08\x00\x04\x02\x0b', mb.resp_handler)
+    assert mb.que.qsize() == 1
+    assert mb.req_pend
+    assert mb.recv_req(b'\x01\x03\x30\x07\x00\x06{\t')
+    assert mb.que.qsize() == 2
+    assert mb.req_pend
+
+    assert mb.send_calls == 1
+    assert mb.pdu == b'\x01\x030\x07\x00\x06{\t'
+    assert mb.recv_responses == 0
+
+    call = 0
+    exp_result = ['V0.0.2C', 4.4, 0.7, 0.7, 30]
+    for key, update, val in mb.recv_resp(mb.db, b'\x01\x03\x0c\x01\x2c\x00\x2c\x00\x2c\x00\x46\x00\x46\x00\x46\x32\xc8', 'test'):
+        if key == 'grid':
+            assert update == True
+        elif key == 'inverter':
+            assert update == True
+        elif key == 'env':
+            assert update == True
+        else:
+            assert False
+        assert exp_result[call] == val
+        call += 1
+    assert 0 == mb.err
+    assert 5 == call
+    assert mb.recv_responses == 1
+
+    assert mb.que.qsize() == 1
+    assert mb.send_calls == 2
+    assert mb.pdu == b'\x01\x06\x20\x08\x00\x04\x02\x0b'
+
+    for key, update, val in mb.recv_resp(mb.db, b'\x01\x06\x20\x08\x00\x04\x02\x0b', 'test'):
+        pass
+    assert 0 == mb.err
+    assert mb.recv_responses == 2
+
+    assert mb.que.qsize() == 0
+    assert mb.send_calls == 3
+    assert mb.pdu == b'\x01\x030\x07\x00\x06{\t'
+    call = 0
+    for key, update, val in mb.recv_resp(mb.db, b'\x01\x03\x0c\x01\x2c\x00\x2c\x00\x2c\x00\x46\x00\x46\x00\x46\x32\xc8', 'test'):
+        call += 1
+    assert 0 == mb.err
+    assert mb.recv_responses == 2
+    assert 5 == call
+
+
+    assert mb.que.qsize() == 0
+    assert not mb.req_pend
+
+@pytest.mark.asyncio
+async def test_timeout():
+    '''Test MODBUS response timeout and RTU retransmitting'''
+    assert asyncio.get_running_loop()
+    mb = ModbusTestHelper()
+    mb.max_retries = 2
+    mb.timeout = 0.1  # 100ms timeout for fast testing, expect a time resolution of at least 10ms
+    assert asyncio.get_running_loop() == mb.loop
+    mb.build_msg(1,3,0x3007,6)
+    mb.build_msg(1,6,0x2008,4)
+
+    assert mb.que.qsize() == 1
+    assert mb.req_pend
+    assert mb.retry_cnt == 0
+    assert mb.send_calls == 1
+    assert mb.pdu == b'\x01\x030\x07\x00\x06{\t'
+
+    mb.pdu = None
+    await asyncio.sleep(0.11)    # wait for first timeout and retransmittion
+    assert mb.que.qsize() == 1
+    assert mb.req_pend
+    assert mb.retry_cnt == 1
+    assert mb.send_calls == 2
+    assert mb.pdu == b'\x01\x030\x07\x00\x06{\t'
+
+    mb.pdu = None
+    await asyncio.sleep(0.11)    # wait for second timeout and retransmittion
+    assert mb.que.qsize() == 1
+    assert mb.req_pend
+    assert mb.retry_cnt == 2
+    assert mb.send_calls == 3
+    assert mb.pdu == b'\x01\x030\x07\x00\x06{\t'
+
+    mb.pdu = None
+    await asyncio.sleep(0.11)    # wait for third timeout and next pdu
+    assert mb.que.qsize() == 0
+    assert mb.req_pend
+    assert mb.retry_cnt == 0
+    assert mb.send_calls == 4
+    assert mb.pdu == b'\x01\x06\x20\x08\x00\x04\x02\x0b'
+
+    mb.max_retries = 0          # next pdu without retranmsission
+    await asyncio.sleep(0.11)    # wait for fourth timout
+    assert mb.que.qsize() == 0
+    assert not mb.req_pend
+    assert mb.retry_cnt == 0
+    assert mb.send_calls == 4
+
+    # assert mb.counter == {}
+
+def test_recv_unknown_data():
+    '''Receive a response with an unknwon register'''
+    mb = ModbusTestHelper()
+    assert 0x9000 not in mb.map
+    mb.map[0x9000] = {'reg': Register.TEST_REG1,   'fmt': '!H', 'ratio':  1}
+
+    mb.build_msg(1,3,0x9000,2)
+
+    # check matching response, but with CRC error
+    call = 0
+    for key, update, val in mb.recv_resp(mb.db, b'\x01\x03\x04\x01\x2c\x00\x46\xbb\xf4', 'test'):
+        call += 1
+    assert mb.err == 0
+    assert 0 == call
+    assert not mb.req_pend
+
+    del mb.map[0x9000]
+
+def test_close():
+    '''Check queue handling for build_msg() calls'''
+    mb = ModbusTestHelper()
+    mb.build_msg(1,3,0x3007,6)
+    mb.build_msg(1,6,0x2008,4)
+    assert mb.que.qsize() == 1
+    mb.build_msg(1,3,0x3007,6)
+    assert mb.que.qsize() == 2
+    assert mb.que.empty() == False 
+    mb.close()
+    assert mb.que.qsize() == 0
+    assert mb.que.empty() == True 

docker-compose.yaml

@@ -1,6 +1,3 @@
-
-version: '3.0'
-
 services:
 #######  H O M E  -  A S S I S T A N T  ##### 
   home-assistant:
@@ -34,7 +31,7 @@ services:
     ports:
       - 8123:8123
     volumes:
-      - ${PROJECT_DIR}./homeassistant/config:/config
+      - ${PROJECT_DIR:-./}homeassistant/config:/config
       - /etc/localtime:/etc/localtime:ro
     healthcheck:
       test: curl --fail http://0.0.0.0:8123/auth/providers || exit 1
@@ -52,21 +49,22 @@ services:
   mqtt:
     container_name: mqtt-broker
     image: eclipse-mosquitto:2
+    restart: unless-stopped
     expose:
       - 1883
     volumes:
-      - ${PROJECT_DIR}./mosquitto/config:/mosquitto/config
-      - ${PROJECT_DIR}./mosquitto/data:/mosquitto/data
+      - ${PROJECT_DIR:-./}mosquitto/config:/mosquitto/config
+      - ${PROJECT_DIR:-./}mosquitto/data:/mosquitto/data
     networks:
-      outside:
-         ipv4_address: 172.28.1.5 # static IP required to receive mDNS traffic     
-    
+      - outside
 
+      
 
 #######  T S U N  -  P R O X Y  ######      
   tsun-proxy:
     container_name: tsun-proxy
     image: ghcr.io/s-allius/tsun-gen3-proxy:latest
+    # image: ghcr.io/s-allius/tsun-gen3-proxy:rc
     restart: unless-stopped
     depends_on:
       - mqtt
@@ -76,12 +74,18 @@ services:
       - GID=${GID:-1000}
     dns:
       - ${DNS1:-8.8.8.8}
-      - $(DNS2:-4.4.4.4}
+      - ${DNS2:-4.4.4.4}
     ports: 
       - 5005:5005
+      - 8127:8127
+      - 10000:10000
     volumes:
-      - ${PROJECT_DIR}./tsun-proxy/log:/home/tsun-proxy/log 
-      - ${PROJECT_DIR}./tsun-proxy/config:/home/tsun-proxy/config 
+      - ${PROJECT_DIR:-./}tsun-proxy/log:/home/tsun-proxy/log 
+      - ${PROJECT_DIR:-./}tsun-proxy/config:/home/tsun-proxy/config 
+    healthcheck:
+      test: wget --no-verbose --tries=1 --spider  http://localhost:8127/-/healthy || exit 1
+      interval: 10s
+      timeout: 3s    
     networks:
       - outside
 
@@ -91,11 +95,4 @@ services:
 networks:
   outside:
     name: home-assistant
-    external: true
-    ipam:
-      driver: default
-      config:
-        - subnet: 172.28.1.0/26
-          ip_range: 172.28.1.32/27
-          gateway: 172.28.1.62          
           

system_tests/test_tcp_socket.py

@@ -20,7 +20,7 @@ def MsgContactInfo(): # Contact Info message
 
 @pytest.fixture
 def MsgContactResp(): # Contact Response message
-    return b'\x00\x00\x00\x14\x10'+get_sn()+b'\x99\x00\x01'
+    return b'\x00\x00\x00\x14\x10'+get_sn()+b'\x91\x00\x01'
 
 @pytest.fixture
 def MsgContactInfo2(): # Contact Info message
@@ -28,7 +28,7 @@ def MsgContactInfo2(): # Contact Info message
 
 @pytest.fixture
 def MsgContactResp2(): # Contact Response message
-    return b'\x00\x00\x00\x14\x10'+get_invalid_sn()+b'\x99\x00\x01'
+    return b'\x00\x00\x00\x14\x10'+get_invalid_sn()+b'\x91\x00\x01'
 
 @pytest.fixture
 def MsgTimeStampReq(): # Get Time Request message
@@ -92,8 +92,29 @@ def MsgInverterInd(): # Data indication from the inverter
     msg +=  b'\x31\x38\x46\x42\x08\x00\x00\x00\x00\x31\x9c\x53\x00\x05\x00\x00\x32\x00\x53\x04\x00\x00\x00\x32\x64\x53\x00\x01\x00\x00\x32\xc8\x53\x13\x9c\x00\x00\x33\x2c\x53\x0f'
     msg +=  b'\xa0\x00\x00\x33\x90\x53\x00\x4f\x00\x00\x33\xf4\x53\x00\x66\x00\x00\x34\x58\x53\x03\xe8\x00\x00\x34\xbc\x53\x04\x00\x00\x00\x35\x20\x53\x00\x00\x00\x00\x35\x84\x53'
     msg +=  b'\x00\x00\x00\x00\x35\xe8\x53\x00\x00\x00\x00\x36\x4c\x53\x00\x00\x00\x01\x38\x80\x53\x00\x02\x00\x01\x38\x81\x53\x00\x01\x00\x01\x38\x82\x53\x00\x01\x00\x01\x38\x83'
-    msg +=  b'\x53\x00\x00' 
-    
+    msg +=  b'\x53\x00\x00'     
+    return msg
+
+@pytest.fixture
+def MsgOtaUpdateReq(): # Over the air update request from talent cloud
+    msg  = b'\x00\x00\x01\x16\x10'+ get_sn() + b'\x70\x13\x01\x02\x76\x35'
+    msg += b'\x70\x68\x74\x74\x70'
+    msg += b'\x3a\x2f\x2f\x77\x77\x77\x2e\x74\x61\x6c\x65\x6e\x74\x2d\x6d\x6f'
+    msg += b'\x6e\x69\x74\x6f\x72\x69\x6e\x67\x2e\x63\x6f\x6d\x3a\x39\x30\x30'
+    msg += b'\x32\x2f\x70\x72\x6f\x64\x2d\x61\x70\x69\x2f\x72\x6f\x6d\x2f\x75'
+    msg += b'\x70\x64\x61\x74\x65\x2f\x64\x6f\x77\x6e\x6c\x6f\x61\x64\x3f\x76'
+    msg += b'\x65\x72\x3d\x56\x31\x2e\x30\x30\x2e\x31\x37\x26\x6e\x61\x6d\x65'
+    msg += b'\x3d\x47\x33\x2d\x57\x69\x46\x69\x2b\x2d\x56\x31\x2e\x30\x30\x2e'
+    msg += b'\x31\x37\x2d\x4f\x54\x41\x26\x65\x78\x74\x3d\x30\x60\x68\x74\x74'
+    msg += b'\x70\x3a\x2f\x2f\x77\x77\x77\x2e\x74\x61\x6c\x65\x6e\x74\x2d\x6d'
+    msg += b'\x6f\x6e\x69\x74\x6f\x72\x69\x6e\x67\x2e\x63\x6f\x6d\x3a\x39\x30'
+    msg += b'\x30\x32\x2f\x70\x72\x6f\x64\x2d\x61\x70\x69\x2f\x72\x6f\x6d\x2f'
+    msg += b'\x75\x70\x64\x61\x74\x65\x2f\x63\x61\x6c\x6c\x62\x61\x63\x6b\x3f'
+    msg += b'\x71\x69\x64\x3d\x31\x35\x30\x33\x36\x32\x26\x72\x69\x64\x3d\x32'
+    msg += b'\x32\x39\x26\x64\x69\x64\x3d\x31\x33\x34\x32\x32\x35\x20\x36\x35'
+    msg += b'\x66\x30\x64\x37\x34\x34\x62\x66\x33\x39\x61\x62\x38\x32\x34\x64'
+    msg += b'\x32\x38\x62\x38\x34\x64\x31\x39\x65\x64\x33\x31\x31\x63\x06\x34'
+    msg += b'\x36\x38\x36\x33\x33\x01\x31\x01\x30\x00'
     return msg
 
 
@@ -107,6 +128,7 @@ def ClientConnection():
         s.connect((host, port))
         s.settimeout(1)
         yield s
+        time.sleep(2.5)
         s.close()
 
 def tempClientConnection():
@@ -137,6 +159,7 @@ def test_send_contact_info1(ClientConnection, MsgContactInfo, MsgContactResp):
         pass
     assert data == MsgContactResp
 
+
 def test_send_contact_info2(ClientConnection, MsgContactInfo2, MsgContactInfo, MsgContactResp):
     s = ClientConnection  
     try:
@@ -154,7 +177,20 @@ def test_send_contact_info2(ClientConnection, MsgContactInfo2, MsgContactInfo, M
         pass
     assert data == MsgContactResp
 
-
+def test_send_contact_info3(ClientConnection, MsgContactInfo, MsgContactResp, MsgTimeStampReq):
+    s = ClientConnection
+    try:
+        s.sendall(MsgContactInfo)
+        data = s.recv(1024)
+    except TimeoutError:
+        pass
+    assert data == MsgContactResp
+    try:
+        s.sendall(MsgTimeStampReq)
+        data = s.recv(1024)
+    except TimeoutError:
+        pass
+ 
 
 def test_send_contact_resp(ClientConnection, MsgContactResp):
     s = ClientConnection  
@@ -164,7 +200,7 @@ def test_send_contact_resp(ClientConnection, MsgContactResp):
     except TimeoutError:
         assert True
     else:
-        assert data ==''
+        assert data == b''
 
 def test_send_ctrl_data(ClientConnection, MsgTimeStampReq, MsgTimeStampResp, MsgContollerInd):
     s = ClientConnection  
@@ -188,7 +224,7 @@ def test_send_inv_data(ClientConnection, MsgTimeStampReq, MsgTimeStampResp, MsgI
         data = s.recv(1024)
     except TimeoutError:
         pass
-   # time.sleep(32.5)
+  #  time.sleep(32.5)
   #  assert data == MsgTimeStampResp
     try:
         s.sendall(MsgInvData)
@@ -197,3 +233,11 @@ def test_send_inv_data(ClientConnection, MsgTimeStampReq, MsgTimeStampResp, MsgI
         data = s.recv(1024)
     except TimeoutError:
         pass
+
+def test_ota_req(ClientConnection, MsgOtaUpdateReq):
+    s = ClientConnection  
+    try:
+        s.sendall(MsgOtaUpdateReq)
+        data = s.recv(1024)
+    except TimeoutError:
+        pass

system_tests/test_tcp_socket_v2.py

@@ -0,0 +1,194 @@
+# test_with_pytest.py and scapy
+#
+import pytest, socket, time, os
+from dotenv import load_dotenv
+
+#from scapy.all import *
+#from scapy.layers.inet import IP, TCP, TCP_client
+
+load_dotenv()
+
+SOLARMAN_SNR = os.getenv('SOLARMAN_SNR', '00000080')
+
+def get_sn() -> bytes:
+    return bytes.fromhex(SOLARMAN_SNR)
+
+def get_inv_no() -> bytes:
+    return b'T170000000000001'
+
+def get_invalid_sn():
+    return b'R170000000000002'
+
+def correct_checksum(buf):
+    checksum = sum(buf[1:]) & 0xff
+    return checksum.to_bytes(length=1)
+
+@pytest.fixture
+def MsgContactInfo(): # Contact Info message
+    msg  = b'\xa5\xd4\x00\x10\x41\x00\x01' +get_sn()  +b'\x02\xba\xd2\x00\x00'
+    msg += b'\x19\x00\x00\x00\x00\x00\x00\x00\x05\x3c\x78\x01\x64\x01\x4c\x53'
+    msg += b'\x57\x35\x42\x4c\x45\x5f\x31\x37\x5f\x30\x32\x42\x30\x5f\x31\x2e'
+    msg += b'\x30\x35\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x40\x2a\x8f\x4f\x51\x54\x31\x39\x32\x2e'
+    msg += b'\x31\x36\x38\x2e\x38\x30\x2e\x34\x39\x00\x00\x00\x0f\x00\x01\xb0'
+    msg += b'\x02\x0f\x00\xff\x56\x31\x2e\x31\x2e\x30\x30\x2e\x30\x42\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xfe\xfe\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x41\x6c\x6c\x69\x75\x73\x2d\x48\x6f'
+    msg += b'\x6d\x65\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x3c'
+    msg += b'\x15'
+    return msg
+
+@pytest.fixture
+def MsgContactResp(): # Contact Response message
+    msg  = b'\xa5\x0a\x00\x10\x11\x01\x01' +get_sn()  +b'\x02\x01\x6a\xfd\x8f'
+    msg += b'\x65\x3c\x00\x00\x00\x75\x15'
+    return msg
+
+@pytest.fixture
+def MsgDataInd(): 
+    msg  = b'\xa5\x99\x01\x10\x42\x59\x84' +get_sn()  +b'\x01\xb0\x02\x2c\x87'
+    msg += b'\x22\x32\xb7\x29\x00\x00\xd6\xcf\xe1\x33\x01\x00\x0c\x05\x00\x00'
+    msg += b'\x59\x31\x37\x45\x37\x41\x30\x46\x30\x31\x30\x42\x30\x31\x33\x45'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x01\x12\x02\x12\x12\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x40\x10\x08\xd8\x00\x09\x13\x84\x00\x35\x00\x00\x02\x58\x00\xd8'
+    msg += b'\x01\x3f\x00\x17\x00\x4d\x01\x44\x00\x14\x00\x43\x01\x45\x00\x18'
+    msg += b'\x00\x52\x00\x12\x00\x01\x00\x00\x00\x7c\x00\x00\x24\xed\x00\x2c'
+    msg += b'\x00\x00\x0b\x10\x00\x26\x00\x00\x0a\x0f\x00\x30\x00\x00\x0b\x76'
+
+    msg += b'\x00\x00\x00\x00\x06\x16\x00\x00\x00\x01\x55\xaa\x00\x01\x00\x00'
+    msg += b'\x00\x00\x00\x00\xff\xff\x07\xd0\x00\x03\x04\x00\x04\x00\x04\x00'
+    msg += b'\x04\x00\x00\x01\xff\xff\x00\x01\x00\x06\x00\x68\x00\x68\x05\x00'
+    msg += b'\x09\xcd\x07\xb6\x13\x9c\x13\x24\x00\x01\x07\xae\x04\x0f\x00\x41'
+    msg += b'\x00\x0f\x0a\x64\x0a\x64\x00\x06\x00\x06\x09\xf6\x12\x8c\x12\x8c'
+    msg += b'\x00\x10\x00\x10\x14\x52\x14\x52\x00\x10\x00\x10\x01\x51\x00\x05'
+    msg += b'\x04\x00\x00\x01\x13\x9c\x0f\xa0\x00\x4e\x00\x66\x03\xe8\x04\x00'
+    msg += b'\x09\xce\x07\xa8\x13\x9c\x13\x26\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x04\x00\x04\x00\x00\x00\x00\x00\xff\xff\x00\x00'
+    msg += b'\x00\x00\x00\x00'
+    msg += correct_checksum(msg)
+    msg += b'\x15'
+    return msg
+
+@pytest.fixture
+def MsgDataResp(): # Contact Response message
+    msg  = b'\xa5\x0a\x00\x10\x12\x80\x84' +get_sn()  +b'\x01\x01\xd1\x96\x04'
+    msg += b'\x66\x3c\x00\x00\x00\xed\x15'
+    return msg
+
+
+@pytest.fixture
+def MsgInvalidInfo(): # Contact Info message wrong start byte
+    msg  = b'\x47\xd4\x00\x10\x41\x00\x01' +get_sn()  +b'\x02\xba\xd2\x00\x00'
+    msg += b'\x19\x00\x00\x00\x00\x00\x00\x00\x05\x3c\x78\x01\x64\x01\x4c\x53'
+    msg += b'\x57\x35\x42\x4c\x45\x5f\x31\x37\x5f\x30\x32\x42\x30\x5f\x31\x2e'
+    msg += b'\x30\x35\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x40\x2a\x8f\x4f\x51\x54\x31\x39\x32\x2e'
+    msg += b'\x31\x36\x38\x2e\x38\x30\x2e\x34\x39\x00\x00\x00\x0f\x00\x01\xb0'
+    msg += b'\x02\x0f\x00\xff\x56\x31\x2e\x31\x2e\x30\x30\x2e\x30\x42\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xfe\xfe\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x41\x6c\x6c\x69\x75\x73\x2d\x48\x6f'
+    msg += b'\x6d\x65\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+    msg += b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x3c'
+    msg += b'\x15'
+    return msg
+
+
+@pytest.fixture(scope="session")
+def ClientConnection():
+    #host = '172.16.30.7'
+    host = 'logger.talent-monitoring.com'
+    #host = 'iot.talent-monitoring.com'
+    #host = '127.0.0.1'
+    port = 10000
+    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+        s.connect((host, port))
+        s.settimeout(1)
+        yield s
+        s.close()
+
+def checkResponse(data, Msg):
+    check = bytearray(data)
+    check[5]= Msg[5]            # ignore seq
+    check[13:18]= Msg[13:18]    # ignore timestamp + first byte of repeat time
+    check[21]= Msg[21]          # ignore crc
+    assert check == Msg
+
+
+def tempClientConnection():
+    #host = '172.16.30.7'
+    host = 'logger.talent-monitoring.com'
+    #host = 'iot.talent-monitoring.com'
+    #host = '127.0.0.1'
+    port = 10000
+    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+        s.connect((host, port))
+        s.settimeout(1)
+        yield s
+        time.sleep(2.5)
+        s.close()
+
+def test_open_close():
+    try:
+        for s in tempClientConnection():
+            pass
+    except:
+        assert False
+    assert True
+
+def test_conn_msg(ClientConnection,MsgContactInfo, MsgContactResp):
+    s = ClientConnection
+    try:
+        s.sendall(MsgContactInfo)
+        # time.sleep(2.5)
+        data = s.recv(1024)
+    except TimeoutError:
+        pass
+    # time.sleep(2.5)
+    checkResponse(data, MsgContactResp)
+
+def test_data_ind(ClientConnection,MsgDataInd, MsgDataResp):
+    s = ClientConnection
+    try:
+        s.sendall(MsgDataInd)
+        # time.sleep(2.5)
+        data = s.recv(1024)
+    except TimeoutError:
+        pass
+    # time.sleep(2.5)
+    checkResponse(data, MsgDataResp)
+
+def test_inavlid_msg(ClientConnection,MsgInvalidInfo,MsgContactInfo, MsgContactResp):
+    s = ClientConnection
+    try:
+        s.sendall(MsgInvalidInfo)
+        # time.sleep(2.5)
+        data = s.recv(1024)
+    except TimeoutError:
+        pass
+    # time.sleep(2.5)
+    try:
+        s.sendall(MsgContactInfo)
+        # time.sleep(2.5)
+        data = s.recv(1024)
+    except TimeoutError:
+        pass
+    # time.sleep(2.5)
+    checkResponse(data, MsgContactResp)
+
+