Nonsense values in PV_Generation_Today metric

[kubaceg]

I use this logger reader sending data through MQTT to my Home Assistant. Sometimes my reader sens nonsense PV_Generation_Today metric value e.g. 345178783 which means that my solar generates 3.45 GWh power today :D quite good value but unfortunately it's not real value. After such peak values are corrected to the proper one.
I don't know if these bad values came from the inverter or maybe from errors in deserialization, it's hard to debug.

The problem is in Home Assistant because the energy dashboard stores every diff value from the sensor and displays graphs based on the sum of these diff values, so I have peaks like this on screenshot and fixing it manually is a pain in the ass.

Here is PV_Generation_Today and PV_Generation_Total graph from grafana:

My idea to fix this issue is to introduce filter mechanism which can check for failed metrics and don't send them. This filter will be activated for each exporter individually (I want to see those errors in my grafana). I have an idea for 2 filters now:

• don't send if PV_Generation_Today is greater than PV_Generation_Total.
• dont' send if there is if any of Fault* metrics are greater than 0.

Has anyone else encountered this problem?

[neilbacon]

Yes, my latest commit in my PR has a crude filter as a 1st attempt. I also had annoying spikes down to zero in PV_Generation_Total and also in other fields. But if, as miikaforma suggests, the errors can be detected at a lower level in the inverter data that would of course be a much better solution.

[kubaceg]

@neilbacon I'll make an additional logger for those bytes and if it confirms that a faulty response comes with a value other than 0x03 in byte 26 I'll write some code for skipping such a response. It's quite easy :)

Filter based on Faults was not 100% accurate. There were responses without Fault and with wrong reading, so I hope that's a better way.

[miikaforma]

Hi, I've got some good results with that function code check myself (haven't had any invalid values from what I've seen). When I started debugging this more, I made my own C# version of the logger because my Python and GO knowledge isn't very strong. I hadn't even checked the total generation value until now (as I now got pinged by @kubaceg 's reply in the other repo). And it does look like I had been getting those drops to zero as well when there was no function code check. That said, I think you can see at which point I switched to using my logger with the function code check :D

Total generation data from Grafana

---

There is one other error in the responses I've seen though. Sometimes the data can also be partial. Like the function code is correct but it's not the whole response. It doesn't affect my logger because of the way my logger works (it throws an exception if there's not enough data to read all values that I expect to be there). However from what I've seen, the data it has isn't faulty, it's just not all of it. I think this is something to keep in mind depending on how your logger works (to not log for example zeros in place of missing data). Maybe best to ensure the whole data is there by reading the length after the function code and then seeking that there's those bytes.

Modbus frame structure (afaik)

Data Property	Value Type	Example
Slave address	Byte	0x01
Function code	Byte	0x03
Data length	Byte	0x50
Data	Byte[Data length]	0x50 bytes
Crc16	Ushort	0x0000

---

During testing, at first I just threw away the responses that had the wrong function code at the first modbus frame and the modbus frame I requested after it. But since the invalid responses that I've checked all had the valid modbus frame in there somewhere, I made my logger to just throw away the garbage frames and search for the valid frame (https://github.com/miikaforma/SofarKTLXLogger/blob/main/SofarKTLXLogger/SolarmanV5/Protocol/ProtocolResponse.cs#L17). The results so far seem good and haven't seen any issues (the above total generation image speaks in favour of this as well). That said, it's probably easier to just ignore the whole response if the first modbus frame isn't incorrect.

[kubaceg]

Good job! I have to implement such algorithm in my go logger :) thanks for resolving this problem