DSMR via 3rd party device

[ihulsbus]

Hi Jan,

I currently have a HomeWizard Wi-Fi P1 meter connected to the central meter to export some power data. Now, this device sends it both to an remote system and exposes the data via a local, unprotected endpoint. I would like to see if it would be possible to integrate this data with the data from our solar panel inverters. Would it be possible to support such endpoints?

I realize of course that it is impossible to support such devices all from the datalogger but maybe if users supply their own field mapping for the endpoint, assuming the data is exposed in JSON, it could work?

I'm currently looking at the following exported data:

{
    "smr_version": 42,
    "meter_model": "Kaifa KAIFA-METER",
    "wifi_ssid": "<REDACTED>",
    "wifi_strength": 54,
    "total_power_import_t1_kwh": 19076.256,
    "total_power_import_t2_kwh": 16463.774,
    "total_power_export_t1_kwh": 3110.004,
    "total_power_export_t2_kwh": 7837.318,
    "active_power_w": 826,
    "active_power_l1_w": 219,
    "active_power_l2_w": 4,
    "active_power_l3_w": 603,
    "total_gas_m3": null,
    "gas_timestamp": null
}

Unfortunately I do not have this information for the gas part as it's not a "smart" gasmeter.

If this cannot be supported, could I do the same data combining afterwards via my InfluxDB collections? Or does the logger do some magic internally to match the two sources together?

Thanks,
Ian

[jbouwh]

The logger combines DSMR data with the closets timestamp to the solar data. But if the solar data is polled directly from the inverter, this is not an issue. The current datalogger reads the P1 port. My meter pushes data every second, and updates occur every 10s. It should be possible though to poll a JSON based API instead. Perhaps some data speciation exists that could be used.

[ihulsbus]

Thanks. Looking at the output of the device, it seems to be a proprietary format, and not something publicly available, making integration difficult.

I think I'll try to do the math in InfluxDB afterwards.

I got two additional problems I'm running into, one is the logger not able to authorize against influxDB with a token or basicAuth. The second one has to do with the scraping of an inverter. one can be scraped over 8899 (Inverter connected over WiFi) the other not (connected over Ethernet). Both are the same age inverter with a serial number that should be high enough (16XXXXXXXX and 19XXXXXXXX)

I'll create a separate issue for the first one, but I'm not sure about the inverter issue?

[jbouwh]

If connecting over 8899 is possible for all inverters than that is ok. for Influx you need to enable v1 auth if you are using Influx v2

[ihulsbus]

It is for one, but strangely enough not for the other. The main difference is connection medium and the serial number is of a higher number. Firmware on both is exactly the same. I'm still not sure what's going on there.

Saw the Influx v2 PR. I'll try it as soon as I have the chance. Thanks!

[jbouwh]

It is for one, but strangely enough not for the other. The main difference is connection medium and the serial number is of a higher number. Firmware on both is exactly the same. I'm still not sure what's going on there.

Saw the Influx v2 PR. I'll try it as soon as I have the chance. Thanks!

To reverse engineer the data it is, to begin with a good thing to have data samples.
If you set the loglevel to debugging tcpclient will also log a base64 datadump if the data block is >= 99 bytes.

[ihulsbus]

InfluxDB V2 implementation works excellent! Thanks!

For the second inverter that's not responding on 8899 I only see the following:

INFO:omnik.datalogger:I 2022-02-26T17:05:53.341080 Initializing: Trying to reach the inverter for plant 2 over port 8899.
DEBUG:omnik.datalogger:@ 2022-02-26T17:05:53.341146 Connecting to <IP> port 8899
DEBUG:omnik.datalogger:@ 2022-02-26T17:05:53.341698 Sending request message with logger_sn: <redacted>
INFO:omnik.datalogger:I 2022-02-26T17:06:33.353863 Inverter is not reachable over port 8899, this is normal when it is dark. timed out
DEBUG:omnik.datalogger:@ 2022-02-26T17:06:33.354145 Try to fetch information from url http://<IP>/js/status.js
DEBUG:omnik.datalogger:@ 2022-02-26T17:06:33.359267 Timed data logging processed
DEBUG:omnik:@ 2022-02-26T17:06:33.359391 new poll in 15 seconds at 2022-02-26T16:06:39.359376+00:00.

it does not seem to do a datadump unfortunately, suggesting there's no response at all, but it also seems the JS endpoint is not working as expected. I'll keep digging for now.

[jbouwh]

My inverter does not support client connections either. It sends an update over port 10004 about every 5 minutes.It is possible to intercept this traffic by using a NAT rule with IP tables and a static forward on my router.

[ihulsbus]

That's what I'm currently trying as well. I spun up a second instance of the datalogger with the tcp_client and the remote server function in the inverter, but I've yet got to figure out how it works exactly. The basis does seem to work:

INFO:omnik.datalogger:I 2022-02-26T18:01:04.213989 Starting tcp_proxy server. listening at <IP-A>:10004.
----------------------------------------
Exception occurred during processing of request from ('<IP-B>', 51838)
Traceback (most recent call last):
File "/usr/local/lib/python3.9/socketserver.py", line 316, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/local/lib/python3.9/socketserver.py", line 347, in process_request
self.finish_request(request, client_address)
File "/usr/local/lib/python3.9/socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/local/lib/python3.9/socketserver.py", line 720, in __init__
self.handle()
File "/usr/local/lib/python3.9/site-packages/omnikdatalogger-1.11.1-py3.9.egg/omnik/plugin_localproxy/tcp_proxy.py", line 13, in handle
data = self.request.recv(1024)
ConnectionResetError: [Errno 104] Connection reset by peer
----------------------------------------

I do have various protocol options on the inverter side:

Ongoing process.

With the first inverter in Influx, I'm going to try and see if I can combine the P1 data with the solar data and do the relevant calculations in Influx or Grafana.

[jbouwh]

I suppose that should work, you can use omnikdataloggerproxy to capture the data on an other host and then forward the data to omnikdatalogger. See https://jbsoft.nl/site/omnik-datalogger/logger-interception-using-omnik-datalogger/