sv/tsun-gen3-proxy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Dev 0.11 (#200)

Browse Source 
* Code Cleanup (#158)


* print coverage report

* create sonar-project property file

* install all py dependencies in one step

* code cleanup

* reduce cognitive complexity

* do not build on *.yml changes

* optimise versionstring handling (#159)

- Reading the version string from the image updates
  it even if the image is re-pulled without re-deployment

* fix linter warning

* exclude *.pyi filese

* ignore some rules for tests

* cleanup (#160)

* Sonar qube 3 (#163)

fix SonarQube warnings in modbus.py

* Sonar qube 3 (#164)


* fix SonarQube warnings

* Sonar qube 3 (#165)

* cleanup

* Add support for TSUN Titan inverter
Fixes #161


* fix SonarQube warnings

* fix error

* rename field "config"

* SonarQube reads flake8 output

* don't stop on flake8 errors

* flake8 scan only app/src for SonarQube

* update flake8 run

* ignore flake8 C901

* cleanup

* fix linter warnings

* ignore changed *.yml files

* read sensor list solarman data packets

* catch 'No route to' error and log only in debug mode

* fix unit tests

* add sensor_list configuration

* adapt unit tests

* fix SonarQube warnings

* Sonar qube 3 (#166)

* add unittests for mqtt.py

* add mock

* move test requirements into a file

* fix unit tests

* fix formating

* initial version

* fix SonarQube warning

* Sonar qube 4 (#169)

* add unit test for inverter.py

* fix SonarQube warning

* Sonar qube 5 (#170)

* fix SonarLints warnings

* use random IP adresses for unit tests

* Docker: The description ist missing (#171)

Fixes #167

* S allius/issue167 (#172)

* cleanup

* Sonar qube 6 (#174)

* test class ModbusConn

* Sonar qube 3 (#178)

* add more unit tests

* GEN3: don't crash on overwritten msg in the receive buffer

* improve test coverage und reduce test delays

* reduce cognitive complexity

* fix merge

* fix merge conflikt

* fix merge conflict

* S allius/issue182 (#183)

* GEN3: After inverter firmware update the 'Unknown Msg Type' increases continuously
Fixes #182

* add support for Controller serial no and MAC

* test hardening

* GEN3: add support for new messages of version 3 firmwares

* bump libraries to latest versions

- bump aiomqtt to version 2.3.0
- bump aiohttp to version 3.10.5

* improve test coverage

* reduce cognective complexity

* fix target preview

* remove dubbled fixtures

* increase test coverage

* Update README.md (#185)

update badges

* S allius/issue186 (#187)

* Parse more values in Server Mode
Fixes #186

* read OUTPUT_COEFFICIENT and MAC_ADDR in SrvMode

* fix unit test

* increase test coverage

* S allius/issue186 (#188)

* increase test coverage

* update changelog

* add dokumentation

* change default config

* Update README.md (#189)

Config file is now foldable

* GEN3: Invalid Contact Info Msg (#192)

Fixes #191

* Refactoring async stream (#194)

* GEN3: Invalid Contact Info Msg
Fixes #191

* introduce ifc with FIFOs

* add object factory

* use AsyncIfc class with FIFO

* declare more methods as classmethods

* - refactoring

- remove _forward_buffer
- make async_write private

* remove _forward_buffer

* refactoring

* avoid mqtt handling for invalid serial numbers

* add two more callbacks

* FIX update_header_cb handling

* split AsyncStream in two classes

* split ConnectionG3(P) in server and client class

* update class diagramm

* refactor server creation

* remove duplicated imports

* reduce code duplication

* move StremPtr instances into Inverter class

* resolution of connection classes

- remove ConnectionG3Client
- remove ConnectionG3Server
- remove ConnectionG3PClient
- remove ConnectionG3PServer

* fix server connections

* fix client loop closing

* don't overwrite self.remote in constructor

* update class diagramm

* fixes

- fixes null pointer accesses
- initalize AsyncStreamClient with proper
  StreamPtr instance

* add close callback

* refactor close handling

* remove connection classes

* move more code into InverterBase class

* remove test_inverter_base.py

* add abstract inverter interface class

* initial commit

* fix sonar qube warnings

* rename class Inverter into Proxy

* fix typo

* move class InverterIfc into a separate file

* add more testcases

* use ProtocolIfc class

* add unit tests for AsyncStream class

* icrease test coverage

* reduce cognitive complexity

* increase test coverage

* increase tes coverage

* simplify heartbeat handler

* remove obsolete tx_get method

* add more unittests

* update changelog

* remove __del__ method for proper gc runs

* check releasing of ModbusConn instances

* call garbage collector to release unreachable objs

* decrease ref counter after the with block

* S allius/issue196 (#198)

* fix healthcheck

- on infrastructure with IPv6 support localhost
  might be resolved to an IPv6 adress. Since the
  proxy only support IPv4 for now, we replace
  localhost by 127.0.0.1, to fix this

* merge from main
This commit is contained in:
Stefan Allius
2024-10-13 18:12:10 +02:00
committed by GitHub
parent bfea38d9da
commit c956c13d13
39 changed files with 3299 additions and 1888 deletions
Download Patch File Download Diff File Expand all files Collapse all files

104
app/src/async_ifc.py Normal file
View File

@@ -0,0 +1,104 @@
from abc import ABC, abstractmethod
class AsyncIfc(ABC):
@abstractmethod
def get_conn_no(self):
pass # pragma: no cover
@abstractmethod
def set_node_id(self, value: str):
pass # pragma: no cover
#
# TX - QUEUE
#
@abstractmethod
def tx_add(self, data: bytearray):
''' add data to transmit queue'''
pass # pragma: no cover
@abstractmethod
def tx_flush(self):
''' send transmit queue and clears it'''
pass # pragma: no cover
@abstractmethod
def tx_peek(self, size: int = None) -> bytearray:
'''returns size numbers of byte without removing them'''
pass # pragma: no cover
@abstractmethod
def tx_log(self, level, info):
''' log the transmit queue'''
pass # pragma: no cover
@abstractmethod
def tx_clear(self):
''' clear transmit queue'''
pass # pragma: no cover
@abstractmethod
def tx_len(self):
''' get numner of bytes in the transmit queue'''
pass # pragma: no cover
#
# FORWARD - QUEUE
#
@abstractmethod
def fwd_add(self, data: bytearray):
''' add data to forward queue'''
pass # pragma: no cover
@abstractmethod
def fwd_log(self, level, info):
''' log the forward queue'''
pass # pragma: no cover
#
# RX - QUEUE
#
@abstractmethod
def rx_get(self, size: int = None) -> bytearray:
'''removes size numbers of bytes and return them'''
pass # pragma: no cover
@abstractmethod
def rx_peek(self, size: int = None) -> bytearray:
'''returns size numbers of byte without removing them'''
pass # pragma: no cover
@abstractmethod
def rx_log(self, level, info):
''' logs the receive queue'''
pass # pragma: no cover
@abstractmethod
def rx_clear(self):
''' clear receive queue'''
pass # pragma: no cover
@abstractmethod
def rx_len(self):
''' get numner of bytes in the receive queue'''
pass # pragma: no cover
@abstractmethod
def rx_set_cb(self, callback):
pass # pragma: no cover
#
# Protocol Callbacks
#
@abstractmethod
def prot_set_timeout_cb(self, callback):
pass # pragma: no cover
@abstractmethod
def prot_set_init_new_client_conn_cb(self, callback):
pass # pragma: no cover
@abstractmethod
def prot_set_update_header_cb(self, callback):
pass # pragma: no cover

415
app/src/async_stream.py
View File

@@ -7,17 +7,136 @@ from typing import Self
from itertools import count
if __name__ == "app.src.async_stream":
from app.src.messages import hex_dump_memory, State
from app.src.proxy import Proxy
from app.src.byte_fifo import ByteFifo
from app.src.async_ifc import AsyncIfc
from app.src.infos import Infos
else: # pragma: no cover
from messages import hex_dump_memory, State
from proxy import Proxy
from byte_fifo import ByteFifo
from async_ifc import AsyncIfc
from infos import Infos
import gc
logger = logging.getLogger('conn')
class AsyncStream():
class AsyncIfcImpl(AsyncIfc):
_ids = count(0)
def __init__(self) -> None:
logger.debug('AsyncIfcImpl.__init__')
self.fwd_fifo = ByteFifo()
self.tx_fifo = ByteFifo()
self.rx_fifo = ByteFifo()
self.conn_no = next(self._ids)
self.node_id = ''
self.timeout_cb = None
self.init_new_client_conn_cb = None
self.update_header_cb = None
def close(self):
self.timeout_cb = None
self.fwd_fifo.reg_trigger(None)
self.tx_fifo.reg_trigger(None)
self.rx_fifo.reg_trigger(None)
def set_node_id(self, value: str):
self.node_id = value
def get_conn_no(self):
return self.conn_no
def tx_add(self, data: bytearray):
''' add data to transmit queue'''
self.tx_fifo += data
def tx_flush(self):
''' send transmit queue and clears it'''
self.tx_fifo()
def tx_peek(self, size: int = None) -> bytearray:
'''returns size numbers of byte without removing them'''
return self.tx_fifo.peek(size)
def tx_log(self, level, info):
''' log the transmit queue'''
self.tx_fifo.logging(level, info)
def tx_clear(self):
''' clear transmit queue'''
self.tx_fifo.clear()
def tx_len(self):
''' get numner of bytes in the transmit queue'''
return len(self.tx_fifo)
def fwd_add(self, data: bytearray):
''' add data to forward queue'''
self.fwd_fifo += data
def fwd_log(self, level, info):
''' log the forward queue'''
self.fwd_fifo.logging(level, info)
def rx_get(self, size: int = None) -> bytearray:
'''removes size numbers of bytes and return them'''
return self.rx_fifo.get(size)
def rx_peek(self, size: int = None) -> bytearray:
'''returns size numbers of byte without removing them'''
return self.rx_fifo.peek(size)
def rx_log(self, level, info):
''' logs the receive queue'''
self.rx_fifo.logging(level, info)
def rx_clear(self):
''' clear receive queue'''
self.rx_fifo.clear()
def rx_len(self):
''' get numner of bytes in the receive queue'''
return len(self.rx_fifo)
def rx_set_cb(self, callback):
self.rx_fifo.reg_trigger(callback)
def prot_set_timeout_cb(self, callback):
self.timeout_cb = callback
def prot_set_init_new_client_conn_cb(self, callback):
self.init_new_client_conn_cb = callback
def prot_set_update_header_cb(self, callback):
self.update_header_cb = callback
class StreamPtr():
'''Descr StreamPtr'''
def __init__(self, _stream, _ifc=None):
self.stream = _stream
self.ifc = _ifc
@property
def ifc(self):
return self._ifc
@ifc.setter
def ifc(self, value):
self._ifc = value
@property
def stream(self):
return self._stream
@stream.setter
def stream(self, value):
self._stream = value
class AsyncStream(AsyncIfcImpl):
MAX_PROC_TIME = 2
'''maximum processing time for a received msg in sec'''
MAX_START_TIME = 400
@@ -28,95 +147,42 @@ class AsyncStream():
'''maximum default time without a received msg in sec'''
def __init__(self, reader: StreamReader, writer: StreamWriter,
addr) -> None:
rstream: "StreamPtr") -> None:
AsyncIfcImpl.__init__(self)
logger.debug('AsyncStream.__init__')
self.reader = reader
self.writer = writer
self.addr = addr
self.r_addr = ''
self.l_addr = ''
self.conn_no = next(self._ids)
self.remote = rstream
self.tx_fifo.reg_trigger(self.__write_cb)
self._reader = reader
self._writer = writer
self.r_addr = writer.get_extra_info('peername')
self.l_addr = writer.get_extra_info('sockname')
self.proc_start = None # start processing start timestamp
self.proc_max = 0
self.async_publ_mqtt = None # will be set AsyncStreamServer only
def __write_cb(self):
self._writer.write(self.tx_fifo.get())
def __timeout(self) -> int:
if self.state == State.init or self.state == State.received:
to = self.MAX_START_TIME
elif self.state == State.up and \
self.server_side and self.modbus_polling:
to = self.MAX_INV_IDLE_TIME
else:
to = self.MAX_DEF_IDLE_TIME
return to
async def publish_outstanding_mqtt(self):
'''Publish all outstanding MQTT topics'''
try:
if self.unique_id:
await self.async_publ_mqtt()
await self._async_publ_mqtt_proxy_stat('proxy')
except Exception:
pass
async def server_loop(self, addr: str) -> None:
'''Loop for receiving messages from the inverter (server-side)'''
logger.info(f'[{self.node_id}:{self.conn_no}] '
f'Accept connection from {addr}')
self.inc_counter('Inverter_Cnt')
await self.publish_outstanding_mqtt()
await self.loop()
self.dec_counter('Inverter_Cnt')
await self.publish_outstanding_mqtt()
logger.info(f'[{self.node_id}:{self.conn_no}] Server loop stopped for'
f' r{self.r_addr}')
# if the server connection closes, we also have to disconnect
# the connection to te TSUN cloud
if self.remote_stream:
logger.info(f'[{self.node_id}:{self.conn_no}] disc client '
f'connection: [{self.remote_stream.node_id}:'
f'{self.remote_stream.conn_no}]')
await self.remote_stream.disc()
async def client_loop(self, _: str) -> None:
'''Loop for receiving messages from the TSUN cloud (client-side)'''
client_stream = await self.remote_stream.loop()
logger.info(f'[{client_stream.node_id}:{client_stream.conn_no}] '
'Client loop stopped for'
f' l{client_stream.l_addr}')
# if the client connection closes, we don't touch the server
# connection. Instead we erase the client connection stream,
# thus on the next received packet from the inverter, we can
# establish a new connection to the TSUN cloud
# erase backlink to inverter
client_stream.remote_stream = None
if self.remote_stream == client_stream:
# logging.debug(f'Client l{client_stream.l_addr} refs:'
# f' {gc.get_referrers(client_stream)}')
# than erase client connection
self.remote_stream = None
if self.timeout_cb:
return self.timeout_cb()
return 360
async def loop(self) -> Self:
"""Async loop handler for precessing all received messages"""
self.r_addr = self.writer.get_extra_info('peername')
self.l_addr = self.writer.get_extra_info('sockname')
self.proc_start = time.time()
while True:
try:
proc = time.time() - self.proc_start
if proc > self.proc_max:
self.proc_max = proc
self.proc_start = None
self.__calc_proc_time()
dead_conn_to = self.__timeout()
await asyncio.wait_for(self.__async_read(),
dead_conn_to)
if self.unique_id:
await self.async_write()
await self.__async_forward()
await self.__async_write()
await self.__async_forward()
if self.async_publ_mqtt:
await self.async_publ_mqtt()
except asyncio.TimeoutError:
@@ -124,7 +190,6 @@ class AsyncStream():
f'connection timeout ({dead_conn_to}s) '
f'for {self.l_addr}')
await self.disc()
self.close()
return self
except OSError as error:
@@ -132,56 +197,54 @@ class AsyncStream():
f'{error} for l{self.l_addr} | '
f'r{self.r_addr}')
await self.disc()
self.close()
return self
except RuntimeError as error:
logger.info(f'[{self.node_id}:{self.conn_no}] '
f'{error} for {self.l_addr}')
await self.disc()
self.close()
return self
except Exception:
self.inc_counter('SW_Exception')
Infos.inc_counter('SW_Exception')
logger.error(
f"Exception for {self.addr}:\n"
f"Exception for {self.r_addr}:\n"
f"{traceback.format_exc()}")
await asyncio.sleep(0) # be cooperative to other task
async def async_write(self, headline: str = 'Transmit to ') -> None:
"""Async write handler to transmit the send_buffer"""
if self._send_buffer:
hex_dump_memory(logging.INFO, f'{headline}{self.addr}:',
self._send_buffer, len(self._send_buffer))
self.writer.write(self._send_buffer)
await self.writer.drain()
self._send_buffer = bytearray(0) # self._send_buffer[sent:]
def __calc_proc_time(self):
if self.proc_start:
proc = time.time() - self.proc_start
if proc > self.proc_max:
self.proc_max = proc
self.proc_start = None
async def disc(self) -> None:
"""Async disc handler for graceful disconnect"""
if self.writer.is_closing():
self.remote = None
if self._writer.is_closing():
return
logger.debug(f'AsyncStream.disc() l{self.l_addr} | r{self.r_addr}')
self.writer.close()
await self.writer.wait_closed()
self._writer.close()
await self._writer.wait_closed()
def close(self) -> None:
logging.debug(f'AsyncStream.close() l{self.l_addr} | r{self.r_addr}')
"""close handler for a no waiting disconnect
hint: must be called before releasing the connection instance
"""
self.reader.feed_eof() # abort awaited read
if self.writer.is_closing():
super().close()
self._reader.feed_eof() # abort awaited read
if self._writer.is_closing():
return
logger.debug(f'AsyncStream.close() l{self.l_addr} | r{self.r_addr}')
self.writer.close()
self._writer.close()
def healthy(self) -> bool:
elapsed = 0
if self.proc_start is not None:
elapsed = time.time() - self.proc_start
if self.state == State.closed or elapsed > self.MAX_PROC_TIME:
if elapsed > self.MAX_PROC_TIME:
logging.debug(f'[{self.node_id}:{self.conn_no}:'
f'{type(self).__name__}]'
f' act:{round(1000*elapsed)}ms'
@@ -194,61 +257,139 @@ class AsyncStream():
'''
async def __async_read(self) -> None:
"""Async read handler to read received data from TCP stream"""
data = await self.reader.read(4096)
data = await self._reader.read(4096)
if data:
self.proc_start = time.time()
self._recv_buffer += data
wait = self.read() # call read in parent class
if wait > 0:
self.rx_fifo += data
wait = self.rx_fifo() # call read in parent class
if wait and wait > 0:
await asyncio.sleep(wait)
else:
raise RuntimeError("Peer closed.")
async def __async_write(self, headline: str = 'Transmit to ') -> None:
"""Async write handler to transmit the send_buffer"""
if len(self.tx_fifo) > 0:
self.tx_fifo.logging(logging.INFO, f'{headline}{self.r_addr}:')
self._writer.write(self.tx_fifo.get())
await self._writer.drain()
async def __async_forward(self) -> None:
"""forward handler transmits data over the remote connection"""
if not self._forward_buffer:
if len(self.fwd_fifo) == 0:
return
try:
if not self.remote_stream:
await self.async_create_remote()
if self.remote_stream:
if self.remote_stream._init_new_client_conn():
await self.remote_stream.async_write()
if self.remote_stream:
self.remote_stream._update_header(self._forward_buffer)
hex_dump_memory(logging.INFO,
f'Forward to {self.remote_stream.addr}:',
self._forward_buffer,
len(self._forward_buffer))
self.remote_stream.writer.write(self._forward_buffer)
await self.remote_stream.writer.drain()
self._forward_buffer = bytearray(0)
await self._async_forward()
except OSError as error:
if self.remote_stream:
rmt = self.remote_stream
self.remote_stream = None
logger.error(f'[{rmt.node_id}:{rmt.conn_no}] Fwd: {error} for '
f'l{rmt.l_addr} | r{rmt.r_addr}')
await rmt.disc()
rmt.close()
if self.remote.stream:
rmt = self.remote
logger.error(f'[{rmt.stream.node_id}:{rmt.stream.conn_no}] '
f'Fwd: {error} for '
f'l{rmt.ifc.l_addr} | r{rmt.ifc.r_addr}')
await rmt.ifc.disc()
if rmt.ifc.close_cb:
rmt.ifc.close_cb()
except RuntimeError as error:
if self.remote_stream:
rmt = self.remote_stream
self.remote_stream = None
logger.info(f'[{rmt.node_id}:{rmt.conn_no}] '
f'Fwd: {error} for {rmt.l_addr}')
await rmt.disc()
rmt.close()
if self.remote.stream:
rmt = self.remote
logger.info(f'[{rmt.stream.node_id}:{rmt.stream.conn_no}] '
f'Fwd: {error} for {rmt.ifc.l_addr}')
await rmt.ifc.disc()
if rmt.ifc.close_cb:
rmt.ifc.close_cb()
except Exception:
self.inc_counter('SW_Exception')
Infos.inc_counter('SW_Exception')
logger.error(
f"Fwd Exception for {self.addr}:\n"
f"Fwd Exception for {self.r_addr}:\n"
f"{traceback.format_exc()}")
def __del__(self):
logger.debug(
f"AsyncStream.__del__ l{self.l_addr} | r{self.r_addr}")
class AsyncStreamServer(AsyncStream):
def __init__(self, reader: StreamReader, writer: StreamWriter,
async_publ_mqtt, create_remote,
rstream: "StreamPtr") -> None:
AsyncStream.__init__(self, reader, writer, rstream)
self.create_remote = create_remote
self.async_publ_mqtt = async_publ_mqtt
def close(self) -> None:
logging.debug('AsyncStreamServer.close()')
self.create_remote = None
self.async_publ_mqtt = None
super().close()
async def server_loop(self) -> None:
'''Loop for receiving messages from the inverter (server-side)'''
logger.info(f'[{self.node_id}:{self.conn_no}] '
f'Accept connection from {self.r_addr}')
Infos.inc_counter('Inverter_Cnt')
await self.publish_outstanding_mqtt()
await self.loop()
Infos.dec_counter('Inverter_Cnt')
await self.publish_outstanding_mqtt()
logger.info(f'[{self.node_id}:{self.conn_no}] Server loop stopped for'
f' r{self.r_addr}')
# if the server connection closes, we also have to disconnect
# the connection to te TSUN cloud
if self.remote and self.remote.stream:
logger.info(f'[{self.node_id}:{self.conn_no}] disc client '
f'connection: [{self.remote.ifc.node_id}:'
f'{self.remote.ifc.conn_no}]')
await self.remote.ifc.disc()
async def _async_forward(self) -> None:
"""forward handler transmits data over the remote connection"""
if not self.remote.stream:
await self.create_remote()
if self.remote.stream and \
self.remote.ifc.init_new_client_conn_cb():
await self.remote.ifc._AsyncStream__async_write()
if self.remote.stream:
self.remote.ifc.update_header_cb(self.fwd_fifo.peek())
self.fwd_fifo.logging(logging.INFO, 'Forward to '
f'{self.remote.ifc.r_addr}:')
self.remote.ifc._writer.write(self.fwd_fifo.get())
await self.remote.ifc._writer.drain()
async def publish_outstanding_mqtt(self):
'''Publish all outstanding MQTT topics'''
try:
await self.async_publ_mqtt()
await Proxy._async_publ_mqtt_proxy_stat('proxy')
except Exception:
pass
class AsyncStreamClient(AsyncStream):
def __init__(self, reader: StreamReader, writer: StreamWriter,
rstream: "StreamPtr", close_cb) -> None:
AsyncStream.__init__(self, reader, writer, rstream)
self.close_cb = close_cb
def close(self) -> None:
logging.debug('AsyncStreamClient.close()')
self.close_cb = None
super().close()
async def client_loop(self, _: str) -> None:
'''Loop for receiving messages from the TSUN cloud (client-side)'''
await self.loop()
logger.info(f'[{self.node_id}:{self.conn_no}] '
'Client loop stopped for'
f' l{self.l_addr}')
if self.close_cb:
self.close_cb()
async def _async_forward(self) -> None:
"""forward handler transmits data over the remote connection"""
if self.remote.stream:
self.remote.ifc.update_header_cb(self.fwd_fifo.peek())
self.fwd_fifo.logging(logging.INFO, 'Forward to '
f'{self.remote.ifc.r_addr}:')
self.remote.ifc._writer.write(self.fwd_fifo.get())
await self.remote.ifc._writer.drain()

54
app/src/byte_fifo.py Normal file
View File

@@ -0,0 +1,54 @@
if __name__ == "app.src.byte_fifo":
from app.src.messages import hex_dump_str, hex_dump_memory
else: # pragma: no cover
from messages import hex_dump_str, hex_dump_memory
class ByteFifo:
""" a byte FIFO buffer with trigger callback """
def __init__(self):
self.__buf = bytearray()
self.__trigger_cb = None
def reg_trigger(self, cb) -> None:
self.__trigger_cb = cb
def __iadd__(self, data):
self.__buf.extend(data)
return self
def __call__(self):
'''triggers the observer'''
if callable(self.__trigger_cb):
return self.__trigger_cb()
return None
def get(self, size: int = None) -> bytearray:
'''removes size numbers of byte and return them'''
if not size:
data = self.__buf
self.clear()
else:
data = self.__buf[:size]
# The fast delete syntax
self.__buf[:size] = b''
return data
def peek(self, size: int = None) -> bytearray:
'''returns size numbers of byte without removing them'''
if not size:
return self.__buf
return self.__buf[:size]
def clear(self):
self.__buf = bytearray()
def __len__(self) -> int:
return len(self.__buf)
def __str__(self) -> str:
return hex_dump_str(self.__buf, self.__len__())
def logging(self, level, info):
hex_dump_memory(level, info, self.__buf, self.__len__())

46
app/src/gen3/connection_g3.py
View File

@@ -1,46 +0,0 @@
import logging
from asyncio import StreamReader, StreamWriter
if __name__ == "app.src.gen3.connection_g3":
from app.src.async_stream import AsyncStream
from app.src.gen3.talent import Talent
else: # pragma: no cover
from async_stream import AsyncStream
from gen3.talent import Talent
logger = logging.getLogger('conn')
class ConnectionG3(AsyncStream, Talent):
def __init__(self, reader: StreamReader, writer: StreamWriter,
addr, remote_stream: 'ConnectionG3', server_side: bool,
id_str=b'') -> None:
AsyncStream.__init__(self, reader, writer, addr)
Talent.__init__(self, server_side, id_str)
self.remote_stream: 'ConnectionG3' = remote_stream
'''
Our puplic methods
'''
def close(self):
AsyncStream.close(self)
Talent.close(self)
# logger.info(f'AsyncStream refs: {gc.get_referrers(self)}')
async def async_create_remote(self) -> None:
pass # virtual interface # pragma: no cover
async def async_publ_mqtt(self) -> None:
pass # virtual interface # pragma: no cover
def healthy(self) -> bool:
logger.debug('ConnectionG3 healthy()')
return AsyncStream.healthy(self)
'''
Our private methods
'''
def __del__(self):
super().__del__()

139
app/src/gen3/inverter_g3.py
View File

@@ -1,137 +1,12 @@
import logging
import traceback
import json
import asyncio
from asyncio import StreamReader, StreamWriter
from aiomqtt import MqttCodeError
if __name__ == "app.src.gen3.inverter_g3":
from app.src.config import Config
from app.src.inverter import Inverter
from app.src.gen3.connection_g3 import ConnectionG3
from app.src.infos import Infos
from app.src.inverter_base import InverterBase
from app.src.gen3.talent import Talent
else: # pragma: no cover
from config import Config
from inverter import Inverter
from gen3.connection_g3 import ConnectionG3
from infos import Infos
from inverter_base import InverterBase
from gen3.talent import Talent
logger_mqtt = logging.getLogger('mqtt')
class InverterG3(Inverter, ConnectionG3):
'''class Inverter is a derivation of an Async_Stream
The class has some class method for managing common resources like a
connection to the MQTT broker or proxy error counter which are common
for all inverter connection
Instances of the class are connections to an inverter and can have an
optional link to an remote connection to the TSUN cloud. A remote
connection dies with the inverter connection.
class methods:
class_init(): initialize the common resources of the proxy (MQTT
broker, Proxy DB, etc). Must be called before the
first inverter instance can be created
class_close(): release the common resources of the proxy. Should not
be called before any instances of the class are
destroyed
methods:
server_loop(addr): Async loop method for receiving messages from the
inverter (server-side)
client_loop(addr): Async loop method for receiving messages from the
TSUN cloud (client-side)
async_create_remote(): Establish a client connection to the TSUN cloud
async_publ_mqtt(): Publish data to MQTT broker
close(): Release method which must be called before a instance can be
destroyed
'''
def __init__(self, reader: StreamReader, writer: StreamWriter, addr):
super().__init__(reader, writer, addr, None, True)
self.__ha_restarts = -1
async def async_create_remote(self) -> None:
'''Establish a client connection to the TSUN cloud'''
tsun = Config.get('tsun')
host = tsun['host']
port = tsun['port']
addr = (host, port)
try:
logging.info(f'[{self.node_id}] Connect to {addr}')
connect = asyncio.open_connection(host, port)
reader, writer = await connect
self.remote_stream = ConnectionG3(reader, writer, addr, self,
False, self.id_str)
logging.info(f'[{self.remote_stream.node_id}:'
f'{self.remote_stream.conn_no}] '
f'Connected to {addr}')
asyncio.create_task(self.client_loop(addr))
except (ConnectionRefusedError, TimeoutError) as error:
logging.info(f'{error}')
except Exception:
self.inc_counter('SW_Exception')
logging.error(
f"Inverter: Exception for {addr}:\n"
f"{traceback.format_exc()}")
async def async_publ_mqtt(self) -> None:
'''publish data to MQTT broker'''
# check if new inverter or collector infos are available or when the
# home assistant has changed the status back to online
try:
if (('inverter' in self.new_data and self.new_data['inverter'])
or ('collector' in self.new_data and
self.new_data['collector'])
or self.mqtt.ha_restarts != self.__ha_restarts):
await self._register_proxy_stat_home_assistant()
await self.__register_home_assistant()
self.__ha_restarts = self.mqtt.ha_restarts
for key in self.new_data:
await self.__async_publ_mqtt_packet(key)
for key in Infos.new_stat_data:
await self._async_publ_mqtt_proxy_stat(key)
except MqttCodeError as error:
logging.error(f'Mqtt except: {error}')
except Exception:
self.inc_counter('SW_Exception')
logging.error(
f"Inverter: Exception:\n"
f"{traceback.format_exc()}")
async def __async_publ_mqtt_packet(self, key):
db = self.db.db
if key in db and self.new_data[key]:
data_json = json.dumps(db[key])
node_id = self.node_id
logger_mqtt.debug(f'{key}: {data_json}')
await self.mqtt.publish(f'{self.entity_prfx}{node_id}{key}', data_json) # noqa: E501
self.new_data[key] = False
async def __register_home_assistant(self) -> None:
'''register all our topics at home assistant'''
for data_json, component, node_id, id in self.db.ha_confs(
self.entity_prfx, self.node_id, self.unique_id,
self.sug_area):
logger_mqtt.debug(f"MQTT Register: cmp:'{component}'"
f" node_id:'{node_id}' {data_json}")
await self.mqtt.publish(f"{self.discovery_prfx}{component}"
f"/{node_id}{id}/config", data_json)
self.db.reg_clr_at_midnight(f'{self.entity_prfx}{self.node_id}')
def close(self) -> None:
logging.debug(f'InverterG3.close() l{self.l_addr} | r{self.r_addr}')
super().close() # call close handler in the parent class
# logging.info(f'Inverter refs: {gc.get_referrers(self)}')
def __del__(self):
logging.debug("InverterG3.__del__")
super().__del__()
class InverterG3(InverterBase):
def __init__(self, reader: StreamReader, writer: StreamWriter):
super().__init__(reader, writer, 'tsun', Talent)

165
app/src/gen3/talent.py
View File

@@ -5,14 +5,16 @@ from datetime import datetime
from tzlocal import get_localzone
if __name__ == "app.src.gen3.talent":
from app.src.messages import hex_dump_memory, Message, State
from app.src.async_ifc import AsyncIfc
from app.src.messages import Message, State
from app.src.modbus import Modbus
from app.src.my_timer import Timer
from app.src.config import Config
from app.src.gen3.infos_g3 import InfosG3
from app.src.infos import Register
else: # pragma: no cover
from messages import hex_dump_memory, Message, State
from async_ifc import AsyncIfc
from messages import Message, State
from modbus import Modbus
from my_timer import Timer
from config import Config
@@ -44,8 +46,16 @@ class Talent(Message):
MB_REGULAR_TIMEOUT = 60
TXT_UNKNOWN_CTRL = 'Unknown Ctrl'
def __init__(self, server_side: bool, id_str=b''):
def __init__(self, addr, ifc: "AsyncIfc", server_side: bool,
client_mode: bool = False, id_str=b''):
super().__init__(server_side, self.send_modbus_cb, mb_timeout=15)
ifc.rx_set_cb(self.read)
ifc.prot_set_timeout_cb(self._timeout)
ifc.prot_set_init_new_client_conn_cb(self._init_new_client_conn)
ifc.prot_set_update_header_cb(self._update_header)
self.addr = addr
self.ifc = ifc
self.conn_no = ifc.get_conn_no()
self.await_conn_resp_cnt = 0
self.id_str = id_str
self.contact_name = b''
@@ -56,7 +66,7 @@ class Talent(Message):
0x00: self.msg_contact_info,
0x13: self.msg_ota_update,
0x22: self.msg_get_time,
0x99: self.msg_act_time,
0x99: self.msg_heartbeat,
0x71: self.msg_collector_data,
# 0x76:
0x77: self.msg_modbus,
@@ -103,6 +113,11 @@ class Talent(Message):
self.log_lvl.clear()
self.state = State.closed
self.mb_timer.close()
self.ifc.rx_set_cb(None)
self.ifc.prot_set_timeout_cb(None)
self.ifc.prot_set_init_new_client_conn_cb(None)
self.ifc.prot_set_update_header_cb(None)
self.ifc = None
super().close()
def __set_serial_no(self, serial_no: str):
@@ -138,10 +153,10 @@ class Talent(Message):
self._read()
while True:
if not self.header_valid:
self.__parse_header(self._recv_buffer, len(self._recv_buffer))
self.__parse_header(self.ifc.rx_peek(), self.ifc.rx_len())
if self.header_valid and \
len(self._recv_buffer) >= (self.header_len + self.data_len):
self.ifc.rx_len() >= (self.header_len + self.data_len):
if self.state == State.init:
self.state = State.received # received 1st package
@@ -149,11 +164,10 @@ class Talent(Message):
if callable(log_lvl):
log_lvl = log_lvl()
hex_dump_memory(log_lvl, f'Received from {self.addr}:'
f' BufLen: {len(self._recv_buffer)}'
self.ifc.rx_log(log_lvl, f'Received from {self.addr}:'
f' BufLen: {self.ifc.rx_len()}'
f' HdrLen: {self.header_len}'
f' DtaLen: {self.data_len}',
self._recv_buffer, len(self._recv_buffer))
f' DtaLen: {self.data_len}')
self.__set_serial_no(self.id_str.decode("utf-8"))
self.__dispatch_msg()
@@ -165,35 +179,15 @@ class Talent(Message):
'''add the actual receive msg to the forwarding queue'''
tsun = Config.get('tsun')
if tsun['enabled']:
buffer = self._recv_buffer
buflen = self.header_len+self.data_len
self._forward_buffer += buffer[:buflen]
hex_dump_memory(logging.DEBUG, 'Store for forwarding:',
buffer, buflen)
buffer = self.ifc.rx_peek(buflen)
self.ifc.fwd_add(buffer)
self.ifc.fwd_log(logging.DEBUG, 'Store for forwarding:')
fnc = self.switch.get(self.msg_id, self.msg_unknown)
logger.info(self.__flow_str(self.server_side, 'forwrd') +
f' Ctl: {int(self.ctrl):#02x} Msg: {fnc.__name__!r}')
def forward_snd(self) -> None:
'''add the actual receive msg to the forwarding queue'''
tsun = Config.get('tsun')
if tsun['enabled']:
_len = len(self._send_buffer) - self.send_msg_ofs
struct.pack_into('!l', self._send_buffer, self.send_msg_ofs,
_len-4)
buffer = self._send_buffer[self.send_msg_ofs:]
buflen = _len
self._forward_buffer += buffer[:buflen]
hex_dump_memory(logging.INFO, 'Store for forwarding:',
buffer, buflen)
fnc = self.switch.get(self.msg_id, self.msg_unknown)
logger.info(self.__flow_str(self.server_side, 'forwrd') +
f' Ctl: {int(self.ctrl):#02x} Msg: {fnc.__name__!r}')
self._send_buffer = self._send_buffer[:self.send_msg_ofs]
def send_modbus_cb(self, modbus_pdu: bytearray, log_lvl: int, state: str):
if self.state != State.up:
logger.warning(f'[{self.node_id}] ignore MODBUS cmd,'
@@ -201,15 +195,13 @@ class Talent(Message):
return
self.__build_header(0x70, 0x77)
self._send_buffer += b'\x00\x01\xa3\x28' # magic ?
self._send_buffer += struct.pack('!B', len(modbus_pdu))
self._send_buffer += modbus_pdu
self.ifc.tx_add(b'\x00\x01\xa3\x28') # magic ?
self.ifc.tx_add(struct.pack('!B', len(modbus_pdu)))
self.ifc.tx_add(modbus_pdu)
self.__finish_send_msg()
hex_dump_memory(log_lvl, f'Send Modbus {state}:{self.addr}:',
self._send_buffer, len(self._send_buffer))
self.writer.write(self._send_buffer)
self._send_buffer = bytearray(0) # self._send_buffer[sent:]
self.ifc.tx_log(log_lvl, f'Send Modbus {state}:{self.addr}:')
self.ifc.tx_flush()
def _send_modbus_cmd(self, func, addr, val, log_lvl) -> None:
if self.state != State.up:
@@ -237,9 +229,9 @@ class Talent(Message):
self.msg_id = 0
self.await_conn_resp_cnt += 1
self.__build_header(0x91)
self._send_buffer += struct.pack(f'!{len(contact_name)+1}p'
f'{len(contact_mail)+1}p',
contact_name, contact_mail)
self.ifc.tx_add(struct.pack(f'!{len(contact_name)+1}p'
f'{len(contact_mail)+1}p',
contact_name, contact_mail))
self.__finish_send_msg()
return True
@@ -323,7 +315,7 @@ class Talent(Message):
self.inc_counter('Invalid_Msg_Format')
# erase broken recv buffer
self._recv_buffer = bytearray()
self.ifc.rx_clear()
return
hdr_len = 5+id_len+2
@@ -344,16 +336,17 @@ class Talent(Message):
def __build_header(self, ctrl, msg_id=None) -> None:
if not msg_id:
msg_id = self.msg_id
self.send_msg_ofs = len(self._send_buffer)
self._send_buffer += struct.pack(f'!l{len(self.id_str)+1}pBB',
0, self.id_str, ctrl, msg_id)
self.send_msg_ofs = self.ifc.tx_len()
self.ifc.tx_add(struct.pack(f'!l{len(self.id_str)+1}pBB',
0, self.id_str, ctrl, msg_id))
fnc = self.switch.get(msg_id, self.msg_unknown)
logger.info(self.__flow_str(self.server_side, 'tx') +
f' Ctl: {int(ctrl):#02x} Msg: {fnc.__name__!r}')
def __finish_send_msg(self) -> None:
_len = len(self._send_buffer) - self.send_msg_ofs
struct.pack_into('!l', self._send_buffer, self.send_msg_ofs, _len-4)
_len = self.ifc.tx_len() - self.send_msg_ofs
struct.pack_into('!l', self.ifc.tx_peek(), self.send_msg_ofs,
_len-4)
def __dispatch_msg(self) -> None:
fnc = self.switch.get(self.msg_id, self.msg_unknown)
@@ -367,7 +360,7 @@ class Talent(Message):
f' Ctl: {int(self.ctrl):#02x} Msg: {fnc.__name__!r}')
def __flush_recv_msg(self) -> None:
self._recv_buffer = self._recv_buffer[(self.header_len+self.data_len):]
self.ifc.rx_get(self.header_len+self.data_len)
self.header_valid = False
'''
@@ -377,7 +370,7 @@ class Talent(Message):
if self.ctrl.is_ind():
if self.server_side and self.__process_contact_info():
self.__build_header(0x91)
self._send_buffer += b'\x01'
self.ifc.tx_add(b'\x01')
self.__finish_send_msg()
# don't forward this contact info here, we will build one
# when the remote connection is established
@@ -391,19 +384,21 @@ class Talent(Message):
self.forward()
def __process_contact_info(self) -> bool:
result = struct.unpack_from('!B', self._recv_buffer, self.header_len)
buf = self.ifc.rx_peek()
result = struct.unpack_from('!B', buf, self.header_len)
name_len = result[0]
if self.data_len < name_len+2:
if self.data_len == 1: # this is a response withone status byte
return False
result = struct.unpack_from(f'!{name_len+1}pB', self._recv_buffer,
self.header_len)
self.contact_name = result[0]
mail_len = result[1]
logger.info(f'name: {self.contact_name}')
if self.data_len >= name_len+2:
result = struct.unpack_from(f'!{name_len+1}pB', buf,
self.header_len)
self.contact_name = result[0]
mail_len = result[1]
logger.info(f'name: {self.contact_name}')
result = struct.unpack_from(f'!{mail_len+1}p', self._recv_buffer,
self.header_len+name_len+1)
self.contact_mail = result[0]
result = struct.unpack_from(f'!{mail_len+1}p', buf,
self.header_len+name_len+1)
self.contact_mail = result[0]
logger.info(f'mail: {self.contact_mail}')
return True
@@ -416,16 +411,16 @@ class Talent(Message):
ts = self._timestamp()
logger.debug(f'time: {ts:08x}')
self.__build_header(0x91)
self._send_buffer += struct.pack('!q', ts)
self.ifc.tx_add(struct.pack('!q', ts))
self.__finish_send_msg()
elif self.data_len >= 8:
ts = self._timestamp()
result = struct.unpack_from('!q', self._recv_buffer,
result = struct.unpack_from('!q', self.ifc.rx_peek(),
self.header_len)
self.ts_offset = result[0]-ts
if self.remote_stream:
self.remote_stream.ts_offset = self.ts_offset
if self.ifc.remote.stream:
self.ifc.remote.stream.ts_offset = self.ts_offset
logger.debug(f'tsun-time: {int(result[0]):08x}'
f' proxy-time: {ts:08x}'
f' offset: {self.ts_offset}')
@@ -436,7 +431,7 @@ class Talent(Message):
self.forward()
def msg_act_time(self):
def msg_heartbeat(self):
if self.ctrl.is_ind():
if self.data_len == 9:
self.state = State.up # allow MODBUS cmds
@@ -445,25 +440,23 @@ class Talent(Message):
self.db.set_db_def_value(Register.POLLING_INTERVAL,
self.mb_timeout)
self.__build_header(0x99)
self._send_buffer += b'\x02'
self.ifc.tx_add(b'\x02')
self.__finish_send_msg()
result = struct.unpack_from('!Bq', self._recv_buffer,
result = struct.unpack_from('!Bq', self.ifc.rx_peek(),
self.header_len)
resp_code = result[0]
ts = result[1]+self.ts_offset
logger.debug(f'inv-time: {int(result[1]):08x}'
f' tsun-time: {ts:08x}'
f' offset: {self.ts_offset}')
self.__build_header(0x91)
self._send_buffer += struct.pack('!Bq', resp_code, ts)
self.forward_snd()
return
struct.pack_into('!Bq', self.ifc.rx_peek(),
self.header_len, resp_code, ts)
elif self.ctrl.is_resp():
result = struct.unpack_from('!B', self._recv_buffer,
result = struct.unpack_from('!B', self.ifc.rx_peek(),
self.header_len)
resp_code = result[0]
logging.debug(f'TimeActRespCode: {resp_code}')
logging.debug(f'Heartbeat-RespCode: {resp_code}')
return
else:
logger.warning(self.TXT_UNKNOWN_CTRL)
@@ -472,7 +465,8 @@ class Talent(Message):
self.forward()
def parse_msg_header(self):
result = struct.unpack_from('!lB', self._recv_buffer, self.header_len)
result = struct.unpack_from('!lB', self.ifc.rx_peek(),
self.header_len)
data_id = result[0] # len of complete message
id_len = result[1] # len of variable id string
@@ -480,7 +474,7 @@ class Talent(Message):
msg_hdr_len = 5+id_len+9
result = struct.unpack_from(f'!{id_len+1}pBq', self._recv_buffer,
result = struct.unpack_from(f'!{id_len+1}pBq', self.ifc.rx_peek(),
self.header_len + 4)
timestamp = result[2]
@@ -493,7 +487,7 @@ class Talent(Message):
def msg_collector_data(self):
if self.ctrl.is_ind():
self.__build_header(0x99)
self._send_buffer += b'\x01'
self.ifc.tx_add(b'\x01')
self.__finish_send_msg()
self.__process_data()
@@ -508,7 +502,7 @@ class Talent(Message):
def msg_inverter_data(self):
if self.ctrl.is_ind():
self.__build_header(0x99)
self._send_buffer += b'\x01'
self.ifc.tx_add(b'\x01')
self.__finish_send_msg()
self.__process_data()
self.state = State.up # allow MODBUS cmds
@@ -528,7 +522,7 @@ class Talent(Message):
def __process_data(self):
msg_hdr_len, ts = self.parse_msg_header()
for key, update in self.db.parse(self._recv_buffer, self.header_len
for key, update in self.db.parse(self.ifc.rx_peek(), self.header_len
+ msg_hdr_len, self.node_id):
if update:
self._set_mqtt_timestamp(key, self._utcfromts(ts))
@@ -548,7 +542,7 @@ class Talent(Message):
msg_hdr_len = 5
result = struct.unpack_from('!lBB', self._recv_buffer,
result = struct.unpack_from('!lBB', self.ifc.rx_peek(),
self.header_len)
modbus_len = result[1]
return msg_hdr_len, modbus_len
@@ -557,7 +551,7 @@ class Talent(Message):
msg_hdr_len = 6
result = struct.unpack_from('!lBBB', self._recv_buffer,
result = struct.unpack_from('!lBBB', self.ifc.rx_peek(),
self.header_len)
modbus_len = result[2]
return msg_hdr_len, modbus_len
@@ -578,13 +572,12 @@ class Talent(Message):
self.__msg_modbus(hdr_len)
def __msg_modbus(self, hdr_len):
data = self._recv_buffer[self.header_len:
self.header_len+self.data_len]
data = self.ifc.rx_peek()[self.header_len:
self.header_len+self.data_len]
if self.ctrl.is_req():
if self.remote_stream.mb.recv_req(data[hdr_len:],
self.remote_stream.
msg_forward):
rstream = self.ifc.remote.stream
if rstream.mb.recv_req(data[hdr_len:], rstream.msg_forward):
self.inc_counter('Modbus_Command')
else:
self.inc_counter('Invalid_Msg_Format')

47
app/src/gen3plus/connection_g3p.py
View File

@@ -1,47 +0,0 @@
import logging
from asyncio import StreamReader, StreamWriter
if __name__ == "app.src.gen3plus.connection_g3p":
from app.src.async_stream import AsyncStream
from app.src.gen3plus.solarman_v5 import SolarmanV5
else: # pragma: no cover
from async_stream import AsyncStream
from gen3plus.solarman_v5 import SolarmanV5
logger = logging.getLogger('conn')
class ConnectionG3P(AsyncStream, SolarmanV5):
def __init__(self, reader: StreamReader, writer: StreamWriter,
addr, remote_stream: 'ConnectionG3P',
server_side: bool,
client_mode: bool) -> None:
AsyncStream.__init__(self, reader, writer, addr)
SolarmanV5.__init__(self, server_side, client_mode)
self.remote_stream: 'ConnectionG3P' = remote_stream
'''
Our puplic methods
'''
def close(self):
AsyncStream.close(self)
SolarmanV5.close(self)
# logger.info(f'AsyncStream refs: {gc.get_referrers(self)}')
async def async_create_remote(self) -> None:
pass # virtual interface # pragma: no cover
async def async_publ_mqtt(self) -> None:
pass # virtual interface # pragma: no cover
def healthy(self) -> bool:
logger.debug('ConnectionG3P healthy()')
return AsyncStream.healthy(self)
'''
Our private methods
'''
def __del__(self):
super().__del__()

141
app/src/gen3plus/inverter_g3p.py
View File

@@ -1,140 +1,15 @@
import logging
import traceback
import json
import asyncio
from asyncio import StreamReader, StreamWriter
from aiomqtt import MqttCodeError
if __name__ == "app.src.gen3plus.inverter_g3p":
from app.src.config import Config
from app.src.inverter import Inverter
from app.src.gen3plus.connection_g3p import ConnectionG3P
from app.src.infos import Infos
from app.src.inverter_base import InverterBase
from app.src.gen3plus.solarman_v5 import SolarmanV5
else: # pragma: no cover
from config import Config
from inverter import Inverter
from gen3plus.connection_g3p import ConnectionG3P
from infos import Infos
from inverter_base import InverterBase
from gen3plus.solarman_v5 import SolarmanV5
logger_mqtt = logging.getLogger('mqtt')
class InverterG3P(Inverter, ConnectionG3P):
'''class Inverter is a derivation of an Async_Stream
The class has some class method for managing common resources like a
connection to the MQTT broker or proxy error counter which are common
for all inverter connection
Instances of the class are connections to an inverter and can have an
optional link to an remote connection to the TSUN cloud. A remote
connection dies with the inverter connection.
class methods:
class_init(): initialize the common resources of the proxy (MQTT
broker, Proxy DB, etc). Must be called before the
first inverter instance can be created
class_close(): release the common resources of the proxy. Should not
be called before any instances of the class are
destroyed
methods:
server_loop(addr): Async loop method for receiving messages from the
inverter (server-side)
client_loop(addr): Async loop method for receiving messages from the
TSUN cloud (client-side)
async_create_remote(): Establish a client connection to the TSUN cloud
async_publ_mqtt(): Publish data to MQTT broker
close(): Release method which must be called before a instance can be
destroyed
'''
def __init__(self, reader: StreamReader, writer: StreamWriter, addr,
class InverterG3P(InverterBase):
def __init__(self, reader: StreamReader, writer: StreamWriter,
client_mode: bool = False):
super().__init__(reader, writer, addr, None,
server_side=True, client_mode=client_mode)
self.__ha_restarts = -1
async def async_create_remote(self) -> None:
'''Establish a client connection to the TSUN cloud'''
tsun = Config.get('solarman')
host = tsun['host']
port = tsun['port']
addr = (host, port)
try:
logging.info(f'[{self.node_id}] Connect to {addr}')
connect = asyncio.open_connection(host, port)
reader, writer = await connect
self.remote_stream = ConnectionG3P(reader, writer, addr, self,
server_side=False,
client_mode=False)
logging.info(f'[{self.remote_stream.node_id}:'
f'{self.remote_stream.conn_no}] '
f'Connected to {addr}')
asyncio.create_task(self.client_loop(addr))
except (ConnectionRefusedError, TimeoutError) as error:
logging.info(f'{error}')
except Exception:
self.inc_counter('SW_Exception')
logging.error(
f"Inverter: Exception for {addr}:\n"
f"{traceback.format_exc()}")
async def async_publ_mqtt(self) -> None:
'''publish data to MQTT broker'''
# check if new inverter or collector infos are available or when the
# home assistant has changed the status back to online
try:
if (('inverter' in self.new_data and self.new_data['inverter'])
or ('collector' in self.new_data and
self.new_data['collector'])
or self.mqtt.ha_restarts != self.__ha_restarts):
await self._register_proxy_stat_home_assistant()
await self.__register_home_assistant()
self.__ha_restarts = self.mqtt.ha_restarts
for key in self.new_data:
await self.__async_publ_mqtt_packet(key)
for key in Infos.new_stat_data:
await self._async_publ_mqtt_proxy_stat(key)
except MqttCodeError as error:
logging.error(f'Mqtt except: {error}')
except Exception:
self.inc_counter('SW_Exception')
logging.error(
f"Inverter: Exception:\n"
f"{traceback.format_exc()}")
async def __async_publ_mqtt_packet(self, key):
db = self.db.db
if key in db and self.new_data[key]:
data_json = json.dumps(db[key])
node_id = self.node_id
logger_mqtt.debug(f'{key}: {data_json}')
await self.mqtt.publish(f'{self.entity_prfx}{node_id}{key}', data_json) # noqa: E501
self.new_data[key] = False
async def __register_home_assistant(self) -> None:
'''register all our topics at home assistant'''
for data_json, component, node_id, id in self.db.ha_confs(
self.entity_prfx, self.node_id, self.unique_id,
self.sug_area):
logger_mqtt.debug(f"MQTT Register: cmp:'{component}'"
f" node_id:'{node_id}' {data_json}")
await self.mqtt.publish(f"{self.discovery_prfx}{component}"
f"/{node_id}{id}/config", data_json)
self.db.reg_clr_at_midnight(f'{self.entity_prfx}{self.node_id}')
def close(self) -> None:
logging.debug(f'InverterG3P.close() l{self.l_addr} | r{self.r_addr}')
super().close() # call close handler in the parent class
# logger.debug (f'Inverter refs: {gc.get_referrers(self)}')
def __del__(self):
logging.debug("InverterG3P.__del__")
super().__del__()
super().__init__(reader, writer, 'solarman',
SolarmanV5, client_mode)

122
app/src/gen3plus/solarman_v5.py
View File

@@ -5,6 +5,7 @@ import asyncio
from datetime import datetime
if __name__ == "app.src.gen3plus.solarman_v5":
from app.src.async_ifc import AsyncIfc
from app.src.messages import hex_dump_memory, Message, State
from app.src.modbus import Modbus
from app.src.my_timer import Timer
@@ -12,6 +13,7 @@ if __name__ == "app.src.gen3plus.solarman_v5":
from app.src.gen3plus.infos_g3p import InfosG3P
from app.src.infos import Register
else: # pragma: no cover
from async_ifc import AsyncIfc
from messages import hex_dump_memory, Message, State
from config import Config
from modbus import Modbus
@@ -60,9 +62,17 @@ class SolarmanV5(Message):
HDR_FMT = '<BLLL'
'''format string for packing of the header'''
def __init__(self, server_side: bool, client_mode: bool):
def __init__(self, addr, ifc: "AsyncIfc",
server_side: bool, client_mode: bool):
super().__init__(server_side, self.send_modbus_cb, mb_timeout=8)
ifc.rx_set_cb(self.read)
ifc.prot_set_timeout_cb(self._timeout)
ifc.prot_set_init_new_client_conn_cb(self._init_new_client_conn)
ifc.prot_set_update_header_cb(self._update_header)
self.addr = addr
self.ifc = ifc
self.conn_no = ifc.get_conn_no()
self.header_len = 11 # overwrite construcor in class Message
self.control = 0
self.seq = Sequence(server_side)
@@ -160,6 +170,11 @@ class SolarmanV5(Message):
self.log_lvl.clear()
self.state = State.closed
self.mb_timer.close()
self.ifc.rx_set_cb(None)
self.ifc.prot_set_timeout_cb(None)
self.ifc.prot_set_init_new_client_conn_cb(None)
self.ifc.prot_set_update_header_cb(None)
self.ifc = None
super().close()
async def send_start_cmd(self, snr: int, host: str,
@@ -230,9 +245,10 @@ class SolarmanV5(Message):
self._read()
while True:
if not self.header_valid:
self.__parse_header(self._recv_buffer, len(self._recv_buffer))
self.__parse_header(self.ifc.rx_peek(),
self.ifc.rx_len())
if self.header_valid and len(self._recv_buffer) >= \
if self.header_valid and self.ifc.rx_len() >= \
(self.header_len + self.data_len+2):
self.__process_complete_received_msg()
self.__flush_recv_msg()
@@ -243,10 +259,10 @@ class SolarmanV5(Message):
log_lvl = self.log_lvl.get(self.control, logging.WARNING)
if callable(log_lvl):
log_lvl = log_lvl()
hex_dump_memory(log_lvl, f'Received from {self.addr}:',
self._recv_buffer, self.header_len +
self.data_len+2)
if self.__trailer_is_ok(self._recv_buffer, self.header_len
self.ifc.rx_log(log_lvl, f'Received from {self.addr}:')
# self._recv_buffer, self.header_len +
# self.data_len+2)
if self.__trailer_is_ok(self.ifc.rx_peek(), self.header_len
+ self.data_len + 2):
if self.state == State.init:
self.state = State.received
@@ -259,9 +275,8 @@ class SolarmanV5(Message):
return
tsun = Config.get('solarman')
if tsun['enabled']:
self._forward_buffer += buffer[:buflen]
hex_dump_memory(logging.DEBUG, 'Store for forwarding:',
buffer, buflen)
self.ifc.fwd_add(buffer[:buflen])
self.ifc.fwd_log(logging.DEBUG, 'Store for forwarding:')
fnc = self.switch.get(self.control, self.msg_unknown)
logger.info(self.__flow_str(self.server_side, 'forwrd') +
@@ -317,7 +332,7 @@ class SolarmanV5(Message):
self.inc_counter('Invalid_Msg_Format')
# erase broken recv buffer
self._recv_buffer = bytearray()
self.ifc.rx_clear()
return
self.header_valid = True
@@ -329,11 +344,11 @@ class SolarmanV5(Message):
'Drop packet w invalid stop byte from '
f'{self.addr}:', buf, buf_len)
self.inc_counter('Invalid_Msg_Format')
if len(self._recv_buffer) > (self.data_len+13):
if self.ifc.rx_len() > (self.data_len+13):
next_start = buf[self.data_len+13]
if next_start != 0xa5:
# erase broken recv buffer
self._recv_buffer = bytearray()
self.ifc.rx_clear()
return False
@@ -349,21 +364,22 @@ class SolarmanV5(Message):
def __build_header(self, ctrl) -> None:
'''build header for new transmit message'''
self.send_msg_ofs = len(self._send_buffer)
self.send_msg_ofs = self.ifc.tx_len()
self._send_buffer += struct.pack(
'<BHHHL', 0xA5, 0, ctrl, self.seq.get_send(), self.snr)
self.ifc.tx_add(struct.pack(
'<BHHHL', 0xA5, 0, ctrl, self.seq.get_send(), self.snr))
fnc = self.switch.get(ctrl, self.msg_unknown)
logger.info(self.__flow_str(self.server_side, 'tx') +
f' Ctl: {int(ctrl):#04x} Msg: {fnc.__name__!r}')
def __finish_send_msg(self) -> None:
'''finish the transmit message, set lenght and checksum'''
_len = len(self._send_buffer) - self.send_msg_ofs
struct.pack_into('<H', self._send_buffer, self.send_msg_ofs+1, _len-11)
check = sum(self._send_buffer[self.send_msg_ofs+1:self.send_msg_ofs +
_len]) & 0xff
self._send_buffer += struct.pack('<BB', check, 0x15) # crc & stop
_len = self.ifc.tx_len() - self.send_msg_ofs
struct.pack_into('<H', self.ifc.tx_peek(), self.send_msg_ofs+1,
_len-11)
check = sum(self.ifc.tx_peek()[
self.send_msg_ofs+1:self.send_msg_ofs + _len]) & 0xff
self.ifc.tx_add(struct.pack('<BB', check, 0x15)) # crc & stop
def _update_header(self, _forward_buffer):
'''update header for message before forwarding,
@@ -394,15 +410,14 @@ class SolarmanV5(Message):
f' Msg: {fnc.__name__!r}')
def __flush_recv_msg(self) -> None:
self._recv_buffer = self._recv_buffer[(self.header_len +
self.data_len+2):]
self.ifc.rx_get(self.header_len + self.data_len+2)
self.header_valid = False
def __send_ack_rsp(self, msgtype, ftype, ack=1):
self.__build_header(msgtype)
self._send_buffer += struct.pack('<BBLL', ftype, ack,
self._timestamp(),
self._heartbeat())
self.ifc.tx_add(struct.pack('<BBLL', ftype, ack,
self._timestamp(),
self._heartbeat()))
self.__finish_send_msg()
def send_modbus_cb(self, pdu: bytearray, log_lvl: int, state: str):
@@ -411,14 +426,12 @@ class SolarmanV5(Message):
' cause the state is not UP anymore')
return
self.__build_header(0x4510)
self._send_buffer += struct.pack('<BHLLL', self.MB_RTU_CMD,
self.sensor_list, 0, 0, 0)
self._send_buffer += pdu
self.ifc.tx_add(struct.pack('<BHLLL', self.MB_RTU_CMD,
self.sensor_list, 0, 0, 0))
self.ifc.tx_add(pdu)
self.__finish_send_msg()
hex_dump_memory(log_lvl, f'Send Modbus {state}:{self.addr}:',
self._send_buffer, len(self._send_buffer))
self.writer.write(self._send_buffer)
self._send_buffer = bytearray(0) # self._send_buffer[sent:]
self.ifc.tx_log(log_lvl, f'Send Modbus {state}:{self.addr}:')
self.ifc.tx_flush()
def _send_modbus_cmd(self, func, addr, val, log_lvl) -> None:
if self.state != State.up:
@@ -460,17 +473,18 @@ class SolarmanV5(Message):
self.forward_at_cmd_resp = False
self.__build_header(0x4510)
self._send_buffer += struct.pack(f'<BHLLL{len(at_cmd)}sc', self.AT_CMD,
0x0002, 0, 0, 0,
at_cmd.encode('utf-8'), b'\r')
self.ifc.tx_add(struct.pack(f'<BHLLL{len(at_cmd)}sc', self.AT_CMD,
0x0002, 0, 0, 0,
at_cmd.encode('utf-8'), b'\r'))
self.__finish_send_msg()
self.ifc.tx_log(logging.INFO, 'Send AT Command:')
try:
await self.async_write('Send AT Command:')
self.ifc.tx_flush()
except Exception:
self._send_buffer = bytearray(0)
self.ifc.tx_clear()
def __forward_msg(self):
self.forward(self._recv_buffer, self.header_len+self.data_len+2)
self.forward(self.ifc.rx_peek(), self.header_len+self.data_len+2)
def __build_model_name(self):
db = self.db
@@ -491,7 +505,7 @@ class SolarmanV5(Message):
def __process_data(self, ftype, ts):
inv_update = False
msg_type = self.control >> 8
for key, update in self.db.parse(self._recv_buffer, msg_type, ftype,
for key, update in self.db.parse(self.ifc.rx_peek(), msg_type, ftype,
self.node_id):
if update:
if key == 'inverter':
@@ -510,7 +524,7 @@ class SolarmanV5(Message):
self.__forward_msg()
def msg_dev_ind(self):
data = self._recv_buffer[self.header_len:]
data = self.ifc.rx_peek()[self.header_len:]
result = struct.unpack_from(self.HDR_FMT, data, 0)
ftype = result[0] # always 2
total = result[1]
@@ -531,7 +545,7 @@ class SolarmanV5(Message):
self.__send_ack_rsp(0x1110, ftype)
def msg_data_ind(self):
data = self._recv_buffer
data = self.ifc.rx_peek()
result = struct.unpack_from('<BHLLLHL', data, self.header_len)
ftype = result[0] # 1 or 0x81
sensor = result[1]
@@ -559,7 +573,7 @@ class SolarmanV5(Message):
self.new_state_up()
def msg_sync_start(self):
data = self._recv_buffer[self.header_len:]
data = self.ifc.rx_peek()[self.header_len:]
result = struct.unpack_from(self.HDR_FMT, data, 0)
ftype = result[0]
total = result[1]
@@ -572,8 +586,8 @@ class SolarmanV5(Message):
self.__send_ack_rsp(0x1310, ftype)
def msg_command_req(self):
data = self._recv_buffer[self.header_len:
self.header_len+self.data_len]
data = self.ifc.rx_peek()[self.header_len:
self.header_len+self.data_len]
result = struct.unpack_from('<B', data, 0)
ftype = result[0]
if ftype == self.AT_CMD:
@@ -585,9 +599,9 @@ class SolarmanV5(Message):
self.forward_at_cmd_resp = True
elif ftype == self.MB_RTU_CMD:
if self.remote_stream.mb.recv_req(data[15:],
self.remote_stream.
__forward_msg):
rstream = self.ifc.remote.stream
if rstream.mb.recv_req(data[15:],
rstream.__forward_msg):
self.inc_counter('Modbus_Command')
else:
logger.error('Invalid Modbus Msg')
@@ -601,7 +615,7 @@ class SolarmanV5(Message):
self.mqtt.publish(key, data))
def get_cmd_rsp_log_lvl(self) -> int:
ftype = self._recv_buffer[self.header_len]
ftype = self.ifc.rx_peek()[self.header_len]
if ftype == self.AT_CMD:
if self.forward_at_cmd_resp:
return logging.INFO
@@ -613,8 +627,8 @@ class SolarmanV5(Message):
return logging.WARNING
def msg_command_rsp(self):
data = self._recv_buffer[self.header_len:
self.header_len+self.data_len]
data = self.ifc.rx_peek()[self.header_len:
self.header_len+self.data_len]
ftype = data[0]
if ftype == self.AT_CMD:
if not self.forward_at_cmd_resp:
@@ -650,7 +664,7 @@ class SolarmanV5(Message):
self.__build_model_name()
def msg_hbeat_ind(self):
data = self._recv_buffer[self.header_len:]
data = self.ifc.rx_peek()[self.header_len:]
result = struct.unpack_from('<B', data, 0)
ftype = result[0]
@@ -659,7 +673,7 @@ class SolarmanV5(Message):
self.new_state_up()
def msg_sync_end(self):
data = self._recv_buffer[self.header_len:]
data = self.ifc.rx_peek()[self.header_len:]
result = struct.unpack_from(self.HDR_FMT, data, 0)
ftype = result[0]
total = result[1]
@@ -672,7 +686,7 @@ class SolarmanV5(Message):
self.__send_ack_rsp(0x1810, ftype)
def msg_response(self):
data = self._recv_buffer[self.header_len:]
data = self.ifc.rx_peek()[self.header_len:]
result = struct.unpack_from('<BBLL', data, 0)
ftype = result[0] # always 2
valid = result[1] == 1 # status

12
app/src/infos.py
View File

@@ -378,15 +378,19 @@ class Infos:
return None # unknwon idx, not in info_defs
def inc_counter(self, counter: str) -> None:
@classmethod
def inc_counter(cls, counter: str) -> None:
'''inc proxy statistic counter'''
db_dict = self.stat['proxy']
db_dict = cls.stat['proxy']
db_dict[counter] += 1
cls.new_stat_data['proxy'] = True
def dec_counter(self, counter: str) -> None:
@classmethod
def dec_counter(cls, counter: str) -> None:
'''dec proxy statistic counter'''
db_dict = self.stat['proxy']
db_dict = cls.stat['proxy']
db_dict[counter] -= 1
cls.new_stat_data['proxy'] = True
def ha_proxy_confs(self, ha_prfx: str, node_id: str, snr: str) \
-> Generator[tuple[str, str, str, str], None, None]:

183
app/src/inverter_base.py Normal file
View File

@@ -0,0 +1,183 @@
import weakref
import asyncio
import logging
import traceback
import json
import gc
from aiomqtt import MqttCodeError
from asyncio import StreamReader, StreamWriter
if __name__ == "app.src.inverter_base":
from app.src.inverter_ifc import InverterIfc
from app.src.proxy import Proxy
from app.src.async_stream import StreamPtr
from app.src.async_stream import AsyncStreamClient
from app.src.async_stream import AsyncStreamServer
from app.src.config import Config
from app.src.infos import Infos
else: # pragma: no cover
from inverter_ifc import InverterIfc
from proxy import Proxy
from async_stream import StreamPtr
from async_stream import AsyncStreamClient
from async_stream import AsyncStreamServer
from config import Config
from infos import Infos
logger_mqtt = logging.getLogger('mqtt')
class InverterBase(InverterIfc, Proxy):
def __init__(self, reader: StreamReader, writer: StreamWriter,
config_id: str, prot_class,
client_mode: bool = False):
Proxy.__init__(self)
self._registry.append(weakref.ref(self))
self.addr = writer.get_extra_info('peername')
self.config_id = config_id
self.prot_class = prot_class
self.__ha_restarts = -1
self.remote = StreamPtr(None)
ifc = AsyncStreamServer(reader, writer,
self.async_publ_mqtt,
self.create_remote,
self.remote)
self.local = StreamPtr(
self.prot_class(self.addr, ifc, True, client_mode), ifc
)
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb) -> None:
logging.debug(f'InverterBase.__exit__() {self.addr}')
self.__del_remote()
self.local.stream.close()
self.local.stream = None
self.local.ifc.close()
self.local.ifc = None
# now explicitly call garbage collector to release unreachable objects
unreachable_obj = gc.collect()
logging.debug(
f'InverterBase.__exit: freed unreachable obj: {unreachable_obj}')
def __del_remote(self):
if self.remote.stream:
self.remote.stream.close()
self.remote.stream = None
if self.remote.ifc:
self.remote.ifc.close()
self.remote.ifc = None
async def disc(self, shutdown_started=False) -> None:
if self.remote.stream:
self.remote.stream.shutdown_started = shutdown_started
if self.remote.ifc:
await self.remote.ifc.disc()
if self.local.stream:
self.local.stream.shutdown_started = shutdown_started
if self.local.ifc:
await self.local.ifc.disc()
def healthy(self) -> bool:
logging.debug('InverterBase healthy()')
if self.local.ifc and not self.local.ifc.healthy():
return False
if self.remote.ifc and not self.remote.ifc.healthy():
return False
return True
async def create_remote(self) -> None:
'''Establish a client connection to the TSUN cloud'''
tsun = Config.get(self.config_id)
host = tsun['host']
port = tsun['port']
addr = (host, port)
stream = self.local.stream
try:
logging.info(f'[{stream.node_id}] Connect to {addr}')
connect = asyncio.open_connection(host, port)
reader, writer = await connect
ifc = AsyncStreamClient(
reader, writer, self.local, self.__del_remote)
self.remote.ifc = ifc
if hasattr(stream, 'id_str'):
self.remote.stream = self.prot_class(
addr, ifc, server_side=False,
client_mode=False, id_str=stream.id_str)
else:
self.remote.stream = self.prot_class(
addr, ifc, server_side=False,
client_mode=False)
logging.info(f'[{self.remote.stream.node_id}:'
f'{self.remote.stream.conn_no}] '
f'Connected to {addr}')
asyncio.create_task(self.remote.ifc.client_loop(addr))
except (ConnectionRefusedError, TimeoutError) as error:
logging.info(f'{error}')
except Exception:
Infos.inc_counter('SW_Exception')
logging.error(
f"Inverter: Exception for {addr}:\n"
f"{traceback.format_exc()}")
async def async_publ_mqtt(self) -> None:
'''publish data to MQTT broker'''
stream = self.local.stream
if not stream or not stream.unique_id:
return
# check if new inverter or collector infos are available or when the
# home assistant has changed the status back to online
try:
if (('inverter' in stream.new_data and stream.new_data['inverter'])
or ('collector' in stream.new_data and
stream.new_data['collector'])
or self.mqtt.ha_restarts != self.__ha_restarts):
await self._register_proxy_stat_home_assistant()
await self.__register_home_assistant(stream)
self.__ha_restarts = self.mqtt.ha_restarts
for key in stream.new_data:
await self.__async_publ_mqtt_packet(stream, key)
for key in Infos.new_stat_data:
await Proxy._async_publ_mqtt_proxy_stat(key)
except MqttCodeError as error:
logging.error(f'Mqtt except: {error}')
except Exception:
Infos.inc_counter('SW_Exception')
logging.error(
f"Inverter: Exception:\n"
f"{traceback.format_exc()}")
async def __async_publ_mqtt_packet(self, stream, key):
db = stream.db.db
if key in db and stream.new_data[key]:
data_json = json.dumps(db[key])
node_id = stream.node_id
logger_mqtt.debug(f'{key}: {data_json}')
await self.mqtt.publish(f'{self.entity_prfx}{node_id}{key}', data_json) # noqa: E501
stream.new_data[key] = False
async def __register_home_assistant(self, stream) -> None:
'''register all our topics at home assistant'''
for data_json, component, node_id, id in stream.db.ha_confs(
self.entity_prfx, stream.node_id, stream.unique_id,
stream.sug_area):
logger_mqtt.debug(f"MQTT Register: cmp:'{component}'"
f" node_id:'{node_id}' {data_json}")
await self.mqtt.publish(f"{self.discovery_prfx}{component}"
f"/{node_id}{id}/config", data_json)
stream.db.reg_clr_at_midnight(f'{self.entity_prfx}{stream.node_id}')

40
app/src/inverter_ifc.py Normal file
View File

@@ -0,0 +1,40 @@
from abc import abstractmethod
import logging
from asyncio import StreamReader, StreamWriter
if __name__ == "app.src.inverter_ifc":
from app.src.iter_registry import AbstractIterMeta
else: # pragma: no cover
from iter_registry import AbstractIterMeta
logger_mqtt = logging.getLogger('mqtt')
class InverterIfc(metaclass=AbstractIterMeta):
_registry = []
@abstractmethod
def __init__(self, reader: StreamReader, writer: StreamWriter,
config_id: str, prot_class,
client_mode: bool):
pass # pragma: no cover
@abstractmethod
def __enter__(self):
pass # pragma: no cover
@abstractmethod
def __exit__(self, exc_type, exc, tb):
pass # pragma: no cover
@abstractmethod
def healthy(self) -> bool:
pass # pragma: no cover
@abstractmethod
async def disc(self, shutdown_started=False) -> None:
pass # pragma: no cover
@abstractmethod
async def create_remote(self) -> None:
pass # pragma: no cover

9
app/src/iter_registry.py Normal file
View File

@@ -0,0 +1,9 @@
from abc import ABCMeta
class AbstractIterMeta(ABCMeta):
def __iter__(cls):
for ref in cls._registry:
obj = ref()
if obj is not None:
yield obj

72
app/src/messages.py
View File

@@ -1,13 +1,15 @@
import logging
import weakref
from typing import Callable, Generator
from typing import Callable
from enum import Enum
if __name__ == "app.src.messages":
from app.src.protocol_ifc import ProtocolIfc
from app.src.infos import Infos, Register
from app.src.modbus import Modbus
else: # pragma: no cover
from protocol_ifc import ProtocolIfc
from infos import Infos, Register
from modbus import Modbus
@@ -33,13 +35,9 @@ def __asc_val(n, data, data_len):
return line
def hex_dump_memory(level, info, data, data_len):
def hex_dump(data, data_len) -> list:
n = 0
lines = []
lines.append(info)
tracer = logging.getLogger('tracer')
if not tracer.isEnabledFor(level):
return
for i in range(0, data_len, 16):
line = ' '
@@ -50,17 +48,26 @@ def hex_dump_memory(level, info, data, data_len):
line += __asc_val(n, data, data_len)
lines.append(line)
return lines
def hex_dump_str(data, data_len):
lines = hex_dump(data, data_len)
return '\n'.join(lines)
def hex_dump_memory(level, info, data, data_len):
lines = []
lines.append(info)
tracer = logging.getLogger('tracer')
if not tracer.isEnabledFor(level):
return
lines += hex_dump(data, data_len)
tracer.log(level, '\n'.join(lines))
class IterRegistry(type):
def __iter__(cls) -> Generator['Message', None, None]:
for ref in cls._registry:
obj = ref()
if obj is not None:
yield obj
class State(Enum):
'''state of the logical connection'''
init = 0
@@ -75,8 +82,13 @@ class State(Enum):
'''connection closed'''
class Message(metaclass=IterRegistry):
_registry = []
class Message(ProtocolIfc):
MAX_START_TIME = 400
'''maximum time without a received msg in sec'''
MAX_INV_IDLE_TIME = 120
'''maximum time without a received msg from the inverter in sec'''
MAX_DEF_IDLE_TIME = 360
'''maximum default time without a received msg in sec'''
def __init__(self, server_side: bool, send_modbus_cb:
Callable[[bytes, int, str], None], mb_timeout: int):
@@ -92,15 +104,21 @@ class Message(metaclass=IterRegistry):
self.header_len = 0
self.data_len = 0
self.unique_id = 0
self.node_id = '' # will be overwritten in the child class's __init__
self._node_id = ''
self.sug_area = ''
self._recv_buffer = bytearray(0)
self._send_buffer = bytearray(0)
self._forward_buffer = bytearray(0)
self.new_data = {}
self.state = State.init
self.shutdown_started = False
@property
def node_id(self):
return self._node_id
@node_id.setter
def node_id(self, value):
self._node_id = value
self.ifc.set_node_id(value)
'''
Empty methods, that have to be implemented in any child class which
don't use asyncio
@@ -109,10 +127,6 @@ class Message(metaclass=IterRegistry):
# to our _recv_buffer
return # pragma: no cover
def _update_header(self, _forward_buffer):
'''callback for updating the header of the forward buffer'''
pass # pragma: no cover
def _set_mqtt_timestamp(self, key, ts: float | None):
if key not in self.new_data or \
not self.new_data[key]:
@@ -128,6 +142,16 @@ class Message(metaclass=IterRegistry):
# logger.info(f'update: key: {key} ts:{tstr}'
self.db.set_db_def_value(info_id, round(ts))
def _timeout(self) -> int:
if self.state == State.init or self.state == State.received:
to = self.MAX_START_TIME
elif self.state == State.up and \
self.server_side and self.modbus_polling:
to = self.MAX_INV_IDLE_TIME
else:
to = self.MAX_DEF_IDLE_TIME
return to
'''
Our puplic methods
'''

4
app/src/modbus.py
View File

@@ -117,10 +117,6 @@ class Modbus():
while not self.que.empty():
self.que.get_nowait()
def __del__(self):
"""log statistics on the deleting of a MODBUS instance"""
logging.debug(f'Modbus __del__:\n {self.counter}')
def build_msg(self, addr: int, func: int, reg: int, val: int,
log_lvl=logging.DEBUG) -> None:
"""Build MODBUS RTU request frame and add it to the tx queue

31
app/src/modbus_tcp.py
View File

@@ -5,9 +5,11 @@ import asyncio
if __name__ == "app.src.modbus_tcp":
from app.src.config import Config
from app.src.gen3plus.inverter_g3p import InverterG3P
from app.src.infos import Infos
else: # pragma: no cover
from config import Config
from gen3plus.inverter_g3p import InverterG3P
from infos import Infos
logger = logging.getLogger('conn')
@@ -17,23 +19,26 @@ class ModbusConn():
self.host = host
self.port = port
self.addr = (host, port)
self.stream = None
self.inverter = None
async def __aenter__(self) -> 'InverterG3P':
'''Establish a client connection to the TSUN cloud'''
connection = asyncio.open_connection(self.host, self.port)
reader, writer = await connection
self.stream = InverterG3P(reader, writer, self.addr,
client_mode=True)
logging.info(f'[{self.stream.node_id}:{self.stream.conn_no}] '
self.inverter = InverterG3P(reader, writer,
client_mode=True)
self.inverter.__enter__()
stream = self.inverter.local.stream
logging.info(f'[{stream.node_id}:{stream.conn_no}] '
f'Connected to {self.addr}')
self.stream.inc_counter('Inverter_Cnt')
await self.stream.publish_outstanding_mqtt()
return self.stream
Infos.inc_counter('Inverter_Cnt')
await self.inverter.local.ifc.publish_outstanding_mqtt()
return self.inverter
async def __aexit__(self, exc_type, exc, tb):
self.stream.dec_counter('Inverter_Cnt')
await self.stream.publish_outstanding_mqtt()
Infos.dec_counter('Inverter_Cnt')
await self.inverter.local.ifc.publish_outstanding_mqtt()
self.inverter.__exit__(exc_type, exc, tb)
class ModbusTcp():
@@ -58,20 +63,22 @@ class ModbusTcp():
'''Loop for receiving messages from the TSUN cloud (client-side)'''
while True:
try:
async with ModbusConn(host, port) as stream:
async with ModbusConn(host, port) as inverter:
stream = inverter.local.stream
await stream.send_start_cmd(snr, host)
await stream.loop()
await stream.ifc.loop()
logger.info(f'[{stream.node_id}:{stream.conn_no}] '
f'Connection closed - Shutdown: '
f'{stream.shutdown_started}')
if stream.shutdown_started:
return
del inverter # decrease ref counter after the with block
except (ConnectionRefusedError, TimeoutError) as error:
logging.debug(f'Inv-conn:{error}')
except OSError as error:
if error.errno == 113:
if error.errno == 113: # pragma: no cover
logging.debug(f'os-error:{error}')
else:
logging.info(f'os-error: {error}')

3
app/src/mqtt.py
View File

@@ -44,9 +44,6 @@ class Mqtt(metaclass=Singleton):
def ha_restarts(self, value):
self._ha_restarts = value
def __del__(self):
logger_mqtt.debug('MQTT: __del__')
async def close(self) -> None:
logger_mqtt.debug('MQTT: close')
self.task.cancel()

21
app/src/protocol_ifc.py Normal file
View File

@@ -0,0 +1,21 @@
from abc import abstractmethod
if __name__ == "app.src.protocol_ifc":
from app.src.iter_registry import AbstractIterMeta
from app.src.async_ifc import AsyncIfc
else: # pragma: no cover
from iter_registry import AbstractIterMeta
from async_ifc import AsyncIfc
class ProtocolIfc(metaclass=AbstractIterMeta):
_registry = []
@abstractmethod
def __init__(self, addr, ifc: "AsyncIfc", server_side: bool,
client_mode: bool = False, id_str=b''):
pass # pragma: no cover
@abstractmethod
def close(self):
pass # pragma: no cover

33
app/src/inverter.py → app/src/proxy.py
View File

@@ -1,7 +1,8 @@
import asyncio
import logging
import json
if __name__ == "app.src.inverter":
if __name__ == "app.src.proxy":
from app.src.config import Config
from app.src.mqtt import Mqtt
from app.src.infos import Infos
@@ -13,10 +14,32 @@ else: # pragma: no cover
logger_mqtt = logging.getLogger('mqtt')
class Inverter():
class Proxy():
'''class Proxy is a baseclass
The class has some class method for managing common resources like a
connection to the MQTT broker or proxy error counter which are common
for all inverter connection
Instances of the class are connections to an inverter and can have an
optional link to an remote connection to the TSUN cloud. A remote
connection dies with the inverter connection.
class methods:
class_init(): initialize the common resources of the proxy (MQTT
broker, Proxy DB, etc). Must be called before the
first inverter instance can be created
class_close(): release the common resources of the proxy. Should not
be called before any instances of the class are
destroyed
methods:
create_remote(): Establish a client connection to the TSUN cloud
async_publ_mqtt(): Publish data to MQTT broker
'''
@classmethod
def class_init(cls) -> None:
logging.debug('Inverter.class_init')
logging.debug('Proxy.class_init')
# initialize the proxy statistics
Infos.static_init()
cls.db_stat = Infos()
@@ -38,7 +61,7 @@ class Inverter():
# reset at midnight when you restart the proxy just before
# midnight!
inverters = Config.get('inverters')
# logger.debug(f'Inverters: {inverters}')
# logger.debug(f'Proxys: {inverters}')
for inv in inverters.values():
if (type(inv) is dict):
node_id = inv['node_id']
@@ -77,7 +100,7 @@ class Inverter():
@classmethod
def class_close(cls, loop) -> None: # pragma: no cover
logging.debug('Inverter.class_close')
logging.debug('Proxy.class_close')
logging.info('Close MQTT Task')
loop.run_until_complete(cls.mqtt.close())
cls.mqtt = None

53
app/src/server.py
View File

@@ -5,8 +5,8 @@ import os
from asyncio import StreamReader, StreamWriter
from aiohttp import web
from logging import config # noqa F401
from messages import Message
from inverter import Inverter
from proxy import Proxy
from inverter_ifc import InverterIfc
from gen3.inverter_g3 import InverterG3
from gen3plus.inverter_g3p import InverterG3P
from scheduler import Schedule
@@ -38,9 +38,9 @@ async def healthy(request):
if proxy_is_up:
# logging.info('web reqeust healthy()')
for stream in Message:
for inverter in InverterIfc:
try:
res = stream.healthy()
res = inverter.healthy()
if not res:
return web.Response(status=503, text="I have a problem")
except Exception as err:
@@ -70,18 +70,11 @@ async def webserver(addr, port):
logging.debug('HTTP cleanup done')
async def handle_client(reader: StreamReader, writer: StreamWriter):
async def handle_client(reader: StreamReader, writer: StreamWriter, inv_class):
'''Handles a new incoming connection and starts an async loop'''
addr = writer.get_extra_info('peername')
await InverterG3(reader, writer, addr).server_loop(addr)
async def handle_client_v2(reader: StreamReader, writer: StreamWriter):
'''Handles a new incoming connection and starts an async loop'''
addr = writer.get_extra_info('peername')
await InverterG3P(reader, writer, addr).server_loop(addr)
with inv_class(reader, writer) as inv:
await inv.local.ifc.server_loop()
async def handle_shutdown(web_task):
@@ -94,25 +87,13 @@ async def handle_shutdown(web_task):
#
# first, disc all open TCP connections gracefully
#
for stream in Message:
stream.shutdown_started = True
try:
await asyncio.wait_for(stream.disc(), 2)
except Exception:
pass
for inverter in InverterIfc:
await inverter.disc(True)
logging.info('Proxy disconnecting done')
#
# second, close all open TCP connections
#
for stream in Message:
stream.close()
await asyncio.sleep(0.1) # give time for closing
logging.info('Proxy closing done')
#
# third, cancel the web server
# second, cancel the web server
#
web_task.cancel()
await web_task
@@ -171,17 +152,19 @@ if __name__ == "__main__":
ConfigErr = Config.class_init()
if ConfigErr is not None:
logging.info(f'ConfigErr: {ConfigErr}')
Inverter.class_init()
Proxy.class_init()
Schedule.start()
mb_tcp = ModbusTcp(loop)
ModbusTcp(loop)
#
# Create tasks for our listening servers. These must be tasks! If we call
# start_server directly out of our main task, the eventloop will be blocked
# and we can't receive and handle the UNIX signals!
#
loop.create_task(asyncio.start_server(handle_client, '0.0.0.0', 5005))
loop.create_task(asyncio.start_server(handle_client_v2, '0.0.0.0', 10000))
for inv_class, port in [(InverterG3, 5005), (InverterG3P, 10000)]:
loop.create_task(asyncio.start_server(lambda r, w, i=inv_class:
handle_client(r, w, i),
'0.0.0.0', port))
web_task = loop.create_task(webserver('0.0.0.0', 8127))
#
@@ -202,7 +185,7 @@ if __name__ == "__main__":
pass
finally:
logging.info("Event loop is stopped")
Inverter.class_close(loop)
Proxy.class_close(loop)
logging.debug('Close event loop')
loop.close()
logging.info(f'Finally, exit Server "{serv_name}"')