From b364fb3f8eec533afd4bae2656b002c91d8c349e Mon Sep 17 00:00:00 2001 From: Stefan Allius <122395479+s-allius@users.noreply.github.com> Date: Fri, 9 Aug 2024 23:16:47 +0200 Subject: [PATCH] Dev 0.10 (#151) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * S allius/issue117 (#118) * add shutdown flag * add more register definitions * add start commando for client side connections * add first support for port 8899 * fix shutdown * add client_mode configuration * read client_mode config to setup inverter connections * add client_mode connections over port 8899 * add preview build * Update README.md describe the new client-mode over port 8899 for GEN3PLUS * MODBUS: the last digit of the inverter version is a hexadecimal number (#121) * S allius/issue117 (#122) * add shutdown flag * add more register definitions * add start commando for client side connections * add first support for port 8899 * fix shutdown * add client_mode configuration * read client_mode config to setup inverter connections * add client_mode connections over port 8899 * add preview build * add documentation for client_mode * catch os error and log thme with DEBUG level * update changelog * make the maximum output coefficient configurable (#124) * S allius/issue120 (#126) * add config option to disable the modbus polling * read more modbus regs in polling mode * extend connection timeouts if polling mode is disabled * update changelog * S allius/issue125 (#127) * fix linter warning * move sequence diagramm to wiki * catch asyncio.CancelledError * S allius/issue128 (#130) * set Register.NO_INPUTS fix to 4 for GEN3PLUS * don't set Register.NO_INPUTS per MODBUS * fix unit tests * register OUTPUT_COEFFICIENT at HA * update changelog * - Home Assistant: improve inverter status value texts * - GEN3: add inverter status * on closing send outstanding MQTT data to the broker * force MQTT publish on every conn open and close * reset inverter state on close - workaround which reset the inverter status to offline when the inverter has a very low output power on connection close * improve client modified - reduce the polling cadence to 30s - set controller statistics for HA * client mode set controller IP for HA * S allius/issue131 (#132) * Make __publish_outstanding_mqtt public * update proxy counter - on client mode connection establishment or disconnecting update tje counection counter * Update README.md (#133) * reset inverter state on close - workaround which reset the inverter status to offline when the inverter has a very low output power on connection close * S allius/issue134 (#135) * add polling invertval and method ha_remove() * add client_mode arg to constructors - add PollingInvervall * hide some topics in client mode - we hide topics in HA by sending an empty register MQTT topic during HA auto configuration * add client_mode value * update class diagram * fix modbus close handler - fix empty call and cleanup que - add unit test * don't sent an initial 1710 msg in client mode * change HA icon for inverter status * increase test coverage * accelerate timer tests * bump aiomqtt and schema to latest release (#137) * MQTT timestamps and protocol improvements (#140) * add TS_INPUT, TS_GRID and TS_TOTAL * prepare MQTT timestamps - add _set_mqtt_timestamp method - fix hexdump printing * push dev and debug images to docker.io * add unix epoche timestamp for MQTT pakets * set timezone for unit tests * set name für setting timezone step * trigger new action * GEN3 and GEN3PLUS: handle multiple message - read: iterate over the receive buffer - forward: append messages to the forward buffer - _update_header: iterate over the forward buffer * GEN3: optimize timeout handling - longer timeout in state init and reveived - got to state pending only from state up * update changelog * cleanup * print coloured logs * Create sonarcloud.yml (#143) * Update sonarcloud.yml * Update sonarcloud.yml * Update sonarcloud.yml * Update sonarcloud.yml * Update sonarcloud.yml * build multi arch images with sboms (#146) * don't send MODBUS request when state is not up (#147) * adapt timings * don't send MODBUS request when state is note up * adapt unit test * make test code more clean (#148) * Make test code more clean (#149) * cleanup * Code coverage for SonarCloud (#150) * cleanup code and unit tests * add test coverage for SonarCloud * configure SonarCloud * update changelog * Do no build on *.yml changes * prepare release 0.10.0 * disable MODBUS_POLLING for GEN§PLUS in example config * bump aiohttp to version 3.10.2 * code cleanup * Fetch all history for all tags and branches --- .coveragerc | 3 +- .github/workflows/python-app.yml | 28 +- .sonarlint/connectedMode.json | 4 + .vscode/settings.json | 6 +- CHANGELOG.md | 23 + README.md | 27 +- app/build.sh | 67 ++- app/config/default_config.toml | 8 + app/proxy.svg | 473 ++++++++-------- app/proxy.yuml | 14 +- app/requirements.txt | 6 +- app/src/async_stream.py | 87 +-- app/src/config.py | 7 +- app/src/gen3/connection_g3.py | 7 +- app/src/gen3/infos_g3.py | 3 + app/src/gen3/inverter_g3.py | 10 +- app/src/gen3/talent.py | 181 +++--- app/src/gen3plus/connection_g3p.py | 12 +- app/src/gen3plus/infos_g3p.py | 33 +- app/src/gen3plus/inverter_g3p.py | 17 +- app/src/gen3plus/solarman_v5.py | 221 +++++--- app/src/infos.py | 155 +++-- app/src/messages.py | 32 +- app/src/modbus.py | 16 +- app/src/modbus_tcp.py | 76 +++ app/src/mqtt.py | 20 +- app/src/server.py | 10 + app/tests/test_config.py | 29 +- app/tests/test_infos.py | 36 +- app/tests/test_infos_g3p.py | 143 ++++- app/tests/test_modbus.py | 41 +- app/tests/test_solarman.py | 727 ++++++++++++------------ app/tests/test_talent.py | 876 ++++++++++++++++++----------- app/tests/timestamp_old.svg | 2 - app/tests/timestamp_old.yuml | 26 - system_tests/test_tcp_socket.py | 11 +- system_tests/test_tcp_socket_v2.py | 14 +- 37 files changed, 2096 insertions(+), 1355 deletions(-) create mode 100644 .sonarlint/connectedMode.json create mode 100644 app/src/modbus_tcp.py delete mode 100644 app/tests/timestamp_old.svg delete mode 100644 app/tests/timestamp_old.yuml diff --git a/.coveragerc b/.coveragerc index 6b08179..890dd1b 100644 --- a/.coveragerc +++ b/.coveragerc @@ -1,2 +1,3 @@ [run] -branch = True \ No newline at end of file +branch = True +relative_files = True \ No newline at end of file diff --git a/.github/workflows/python-app.yml b/.github/workflows/python-app.yml index f51ae3d..9061986 100644 --- a/.github/workflows/python-app.yml +++ b/.github/workflows/python-app.yml @@ -18,10 +18,11 @@ on: - '**.dockerfile' # Do no build on *.dockerfile changes - '**.sh' # Do no build on *.sh changes pull_request: - branches: [ "main" ] + branches: [ "main", "dev-*" ] permissions: contents: read + pull-requests: read # allows SonarCloud to decorate PRs with analysis results jobs: build: @@ -29,7 +30,15 @@ jobs: runs-on: ubuntu-latest steps: + - name: Set timezone + uses: szenius/set-timezone@v2.0 + with: + timezoneLinux: "Europe/Berlin" + timezoneMacos: "Europe/Berlin" + timezoneWindows: "Europe/Berlin" - uses: actions/checkout@v4 + with: + fetch-depth: 0 # Fetch all history for all tags and branches - name: Set up Python 3.12 uses: actions/setup-python@v5 with: @@ -47,4 +56,19 @@ jobs: flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics - name: Test with pytest run: | - python -m pytest app + pip install pytest pytest-cov + #pytest app --doctest-modules --junitxml=junit/test-results.xml --cov=com --cov-report=xml --cov-report=html + python -m pytest app --cov=app/src --cov-report=xml + - name: Analyze with SonarCloud + uses: SonarSource/sonarcloud-github-action@v2.2.0 + env: + SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} + with: + projectBaseDir: . + args: + -Dsonar.projectKey=s-allius_tsun-gen3-proxy + -Dsonar.organization=s-allius + -Dsonar.python.version=3.12 + -Dsonar.python.coverage.reportPaths=coverage.xml + -Dsonar.tests=system_tests,app/tests + -Dsonar.source=app/src diff --git a/.sonarlint/connectedMode.json b/.sonarlint/connectedMode.json new file mode 100644 index 0000000..4a9cc82 --- /dev/null +++ b/.sonarlint/connectedMode.json @@ -0,0 +1,4 @@ +{ + "sonarCloudOrganization": "s-allius", + "projectKey": "s-allius_tsun-gen3-proxy" +} \ No newline at end of file diff --git a/.vscode/settings.json b/.vscode/settings.json index 42fb2c2..dd2d0cf 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -11,5 +11,9 @@ "python.testing.pytestEnabled": true, "flake8.args": [ "--extend-exclude=app/tests/*.py system_tests/*.py" - ] + ], + "sonarlint.connectedMode.project": { + "connectionId": "s-allius", + "projectKey": "s-allius_tsun-gen3-proxy" + } } \ No newline at end of file diff --git a/CHANGELOG.md b/CHANGELOG.md index 934a71e..ffe1250 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [unreleased] +## [0.10.0] - 2024-08-09 + +- bump aiohttp to version 3.10.2 +- add SonarQube and code coverage support +- don't send MODBUS request when state is note up; adapt timeouts [#141](https://github.com/s-allius/tsun-gen3-proxy/issues/141) +- build multi arch images with sboms [#144](https://github.com/s-allius/tsun-gen3-proxy/issues/144) +- add timestamp to MQTT topics [#138](https://github.com/s-allius/tsun-gen3-proxy/issues/138) +- improve the message handling, to avoid hangs +- GEN3: allow long timeouts until we received first inverter data (not only device data) +- bump aiomqtt to version 2.2.0 +- bump schema to version 0.7.7 +- Home Assistant: improve inverter status value texts +- GEN3: add inverter status +- fix flapping registers [#128](https://github.com/s-allius/tsun-gen3-proxy/issues/128) +- register OUTPUT_COEFFICIENT at HA +- GEN3: INVERTER_STATUS, +- add config option to disable the MODBUS polling [#120](https://github.com/s-allius/tsun-gen3-proxy/issues/120) +- make the maximum output coefficient configurable [#123](https://github.com/s-allius/tsun-gen3-proxy/issues/123) +- cleanup shutdown +- add preview build +- MODBUS: the last digit of the inverter version is a hexadecimal number [#119](https://github.com/s-allius/tsun-gen3-proxy/issues/119) +- GEN3PLUS: add client_mode connection on port 8899 [#117](https://github.com/s-allius/tsun-gen3-proxy/issues/117) + ## [0.9.0] - 2024-07-01 - fix exception in MODBUS timeout callback diff --git a/README.md b/README.md index 67a6ead..1ec2294 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@

License: BSD-3-Clause Supported Python versions - Supported aiomqtt versions + Supported aiomqtt versions Supported aiocron versions Supported toml versions

@@ -165,6 +165,9 @@ pv2 = {type = 'RSM40-8-405M', manufacturer = 'Risen'} # Optional, PV module de monitor_sn = 2000000000 # The "Monitoring SN:" can be found on a sticker enclosed with the inverter node_id = 'inv_3' # MQTT replacement for inverters serial number suggested_area = 'garage' # suggested installation place for home-assistant +# if your inverter supports SSL connections you must use the client_mode. Pls, uncomment +# the next line and configure the fixed IP of your inverter +#client_mode = {host = '192.168.0.1', port = 8899} pv1 = {type = 'RSM40-8-410M', manufacturer = 'Risen'} # Optional, PV module descr pv2 = {type = 'RSM40-8-410M', manufacturer = 'Risen'} # Optional, PV module descr pv3 = {type = 'RSM40-8-410M', manufacturer = 'Risen'} # Optional, PV module descr @@ -188,8 +191,20 @@ The standard web interface of the inverter can be accessed at `http://