Where is all of the ESS/inverter/battery/BMS information? about hass-sunpower HOT 34 CLOSED

A few notes:

• customer_state_of_charge is always less than system_state_of_charge - the system "holds back" some charge to keep the system running. Probably partly to help preserve the life of the battery, but I think it might also be necessary to "jump start" the system with an external power source if it drains completely.
• operational_mode has the following possible values:

# from sunpower dl_cgi swagger - valid modes, my assumed descriptions:
#                  "UNKNOWN",
#                  "STANDBY",               Probably the pre-Permission To Operate state
#                  "MANUAL_CHARGE",         Unclear - if you know what this does, let me know!
#                  "MANUAL_DCM",            DCM = "Demand Charge Management?"
#                  "DCM",                   DCM = "Demand Charge Management?"
#                  "TARIFF_OPTIMIZER",      Unclear - if you know what this does, let me know!
#                  "ENERGY_ARBITRAGE",      ("Cost Savings" in SunPower app)
#                  "SELF_CONSUMPTION",      ("Self-Supply" in SunPower app)
#                  "BACKUP_ONLY",           Only discharge when off-grid.
#                  "HECO_ZERO_EXPORT"       Hawaiian Electric Company gives no credit for exports 9AM-4PM.

(source)
I've only ever seen ENERGY_ARBITRAGE (in the app as "Cost Savings") and SELF_CONSUMPTION (in the app as "Self-Supply") in practice.

Enjoy!

from hass-sunpower.

I have spent the last few days working on this specific issue. Thus far it seems to work but I don't know how accurate the information is. I can see the battery charging, discharging. The system doesn't report total input / output power from the battery so I had to calculate instantaneous power and then use the integrator integration (readme has details) to integrate that into a total input and output value. This can then be utilized by the Energy panel. I need a couple of days to compare the sunpower app's daily report to what home assistant reports but it should be pretty good. I am looking into adding the state of the battery (grid available or not) so if I lose grid power I can use automation to turn off certain devices. Additionally I am looking at addressing the constants that are deprecated.

https://github.com/CanisUrsa/hass-sunpower/tree/refactor_with_ess

If you do end up using this know that I changed the unique ids of the fields to closer match to the example shown here https://developers.home-assistant.io/docs/entity_registry_index/

from hass-sunpower.

I think ttymxc5 is RS-485 based on some strings in tools in the firmware image called rs485test (which does the obvious) and psoc4_fwup (which can update the firmware of the PSoC in the MIDC - side note, PSoC chips are all over the system).

Which - it makes sense that the PVS gets this data over RS-485. It doesn't really explain the function of the Ethernet connection from the Conext Gateway to the PVS. But, hopefully this is enough to address the original issue of getting data from the PVS into Home Assistant.

from hass-sunpower.

I have a Conext 6848 inverter that I've integrated into home assistant using modbus maps published by Schneider. Those are supposed to be available via RS-485 or modbus TCP if using their gateway. The Sunspec protocols offer a limited set of data. Schneider's own modbus maps offer a lot more info. Here's the spec for the Conext XW inverter. There's a different set for the gateway.
Conext XW Modbus 503 spec 990-6268B.pdf

Not sure if this applies to what you all are discussing since I don't have batteries integrated into my Sunpower setup. Happy to share more details if you see the ability to directly use modbus via TCP or RS-485.

from hass-sunpower.

I'm in the same boat, and I even tried thinking of ways to infer the battery charge - but came up empty.

My guess is there's another undocumented local endpoint that offers this information. Or perhaps cloud only?

On the https://sds.mysunpower.com/ web app, there's a GraphQL call with Storage information. There's also a CSV download, so you could in theory clean up your old data, but wouldn't be real-time without heavy automation work.

I've also thought about getting this information with a web scraper. Huginn , Crawlbase, Multi Scrape, etc. If I get anything running, I'll post something here.

from hass-sunpower.

Of course reporting of the battery state/fullness must be available through the interface already utilized. As long as we don't know how to ascertain that, it may be possible to get the data another way. One possibility I believe I saw someplace was to use a Shelley EM device - it uses a CT clamp-on transformer and shows Kilowatts, etc. flowing both in and out of an appliance. I had a very good experience with one -- getting the current and power flows to my electric air source heat pump. It connects over wifi and then the values are collected in HASS. It works well in the History panel. So I wonder if the Shelley EM could do as much for the power flows into and out of the Sunvault battery system.

from hass-sunpower.

If the power flows in and out are AC, then it's possible; if they are DC, then it won't be usable. I have Shelly 3EMs monitoring my air-source heat pumps and other appliances now, and they work well, but as you noted they'll only be able to report on flow, not status (like 'how full is the battery').

from hass-sunpower.

@kpfleming Thanks for that information!
It's clear that because Sunpower's app and customer portal show the battery fullness and other data, then it must be available already in the data communicated over the internet to their servers.

from hass-sunpower.

Just some notes on this. I don't have a Sunpower ESS so I'm a little blind here so I have just not attempted to code for it because I didn't know where to get the data. Reading the debug dump you sent it realy doesn't look like the battery data is in that "API" (there are entries, but not any useful charge state details).

The below look interesting and possible to add (the pattern to add these is fairly easy to follow in the code and I may try to do this in a future version but welcome a pull request too)

}, {
"ISDETAIL": true,
"SERIAL": "SY2111536749A0867",
"TYPE": "PV-DISCONNECT",
"STATE": "working",
"STATEDESCR": "Working",
"MODEL": "SunPower PV Disconnect Relay",
"DESCR": "PV Disconnect SY2111536749A0867",
"DEVICE_TYPE": "PV Disconnect",
"hw_version": "0.2.0",
"interface": "ttymxc5",
"slave": 230,
"SWVER": "0.2.10",
"PORT": "P0, Modbus, Slave 230",
"DATATIME": "2022,12,10,21,59,44",
"event_history": "32",
"fw_error": "0",
"relay_mode": "0",
"relay1_state": "1",
"relay2_state": "1",
"relay1_error": "0",
"relay2_error": "0",
"v1n_grid_v": "121",
"v2n_grid_v": "121.3",
"v1n_pv_v": "121.3",
"v2n_pv_v": "121.3",
"origin": "data_logger",
"OPERATION": "noop",
"CURTIME": "2022,12,10,21,59,46"
}, {

}, {
"ISDETAIL": true,
"SERIAL": "SY2121534552D1805",
"TYPE": "EQUINOX-MIO",
"STATE": "working",
"STATEDESCR": "Working",
"MODEL": "SunPower MIO",
"DESCR": "ESS Hub SY2121534552D1805",
"DEVICE_TYPE": "ESS Hub",
"hw_version": "0.4.0",
"interface": "ttymxc5",
"parent": 11,
"slave": 221,
"SWVER": "0.7.9",
"PORT": "P0, Modbus, Slave 221",
"DATATIME": "2022,12,10,21,59,41",
"t_degc": "27",
"humidity": "26",
"v_dcdc_spply_v": "11.397",
"v_spply_v": "11.331",
"v_gateway_v": "11.32",
"fan_actv_fl": "0",
"fw_error": "0",
"event_history": "2112",
"origin": "data_logger",
"OPERATION": "noop",
"PARENT": "00001C39A01C_M0012205101B5B_M0012205101B63",
"CURTIME": "2022,12,10,21,59,46"
}, {

from hass-sunpower.

If anyone has an ESS and can play around with the "API" I'd be interested (here's the detail I built this plugin from:
https://github.com/ginoledesma/sunpower-pvs-exporter/blob/master/sunpower_pvs_notes.md

I'd try all of those endpoints and see if you can get any more data, I'd also just point a web browser at the system with the browser network monitor running and see if there is a way to get the data.

Immediate idea is if the "DeviceDetails" API gets more info on the battery?
http://10.10.10.123//cgi-bin/dl_cgi?Command=DeviceDetails&SerialNumber=M0012205101B5B
http://10.10.10.123//cgi-bin/dl_cgi?Command=DeviceDetails&SerialNumber=M0012205101B63
http://10.10.10.123//cgi-bin/dl_cgi?Command=DeviceDetails&SerialNumber=00001C39A01C_M0012205101B5B_M0012205101B63

Also surprising your batteries are reporting an error state

from hass-sunpower.

I tried those commands on my system, and I just get the following:
{ "result": "unknown command" }

I'm assuming that the normal DeviceList command is defaulted to return those details, seeing as the first line of my response has the PV Supervisor returning:
"DETAIL": "detail",

and my devices all show
"ISDETAIL": true,

from hass-sunpower.

I tried those commands on my system, and I just get the following: { "result": "unknown command" }

I'm assuming that the normal DeviceList command is defaulted to return those details, seeing as the first line of my response has the PV Supervisor returning: "DETAIL": "detail",

and my devices all show "ISDETAIL": true,

I got the same responses.

Also surprising your batteries are reporting an error state

My Batteries are reporting the same.

from hass-sunpower.

I have been thinking about this. I have done some network packet captures and I couldn't find any IP based communication between the PVS and BMS.

I have noticed that the"Port" for the batteries show SunSpec which is an RS485 serial communication. Because we have interrupted the serial communication with the BMS and this is why the batteries is showing "error".

I will be attempting to fix this "error" and find an alternate method to query the server for the data.

from hass-sunpower.

Me too - I got my sunvault battery box open - my first time to see what's inside. I could see both the ethernet and the RS485 cables (black and yellow, respectively) go between the Sunvault and the PVS6. They both terminate in the Sunvault at its Schneider Conext Gateway #865-0329 (https://solar.se.com/us/en/product/conext-gateway/). In briefly scanning its documentation, I can see that this box seems to have Xanbus and Modbus capabilities in addition to the RS485 and ethernet.

So how are the SunPower servers getting the values of the battery activity - production, consumption, % full, time to full charge or time to empty, etc.? They must be passing over the internet from my PVS6. So if the battery data isn't available on the tapped internal ethernet, maybe it's only on the R485. If that is the case, then plausibly the place for us to access the battery data is in network packets on our normal LAN (whether connecting the PVS to the internet via ethernet or wifi).

I'm thinking about sniffing packets on my LAN between the PVS and the internet. Unless the connection is encrypted, we ought to be able to find the data we want in those. My guess is that we may find the communication between the PVS and Sunpower is using HTTP/S for the reason that it tends to "just work" with default configurations of home firewall routers.
-- or MAYBE: the Conext gateway does have a USB port on it and some other things which might permit us to do some sort of allowed "debugging" or monitoring of its activity. That is, provided that in its configuration we're not disallowed by Sunpower's configuring it to lock us out.

Does this line of thinking make sense? To get a look what setup and tools to assemble? Something like a managed ethernet switch with a listening replication port? (practically no hubs are used anymore), Wire Shark or some such? and what else?

from hass-sunpower.

From the packet sniffing I have done troubleshooting SunPower App Related Issues I found the App Data sent to the AWS EC2 (Amazon Cloud) instances for SunPower is TLS v1.2 encrypted when it is transmitted to from the PVS to the cloud. Cracking this would be beyond my abilities.

My current thinking is that if I remove my networking equipment (Some use a Raspberry Pi) and returning the Ethernet cable from the PVS to the SunVault will correct the "error" message seen.

I know that I am able to connect my cell phone and laptop to the PVS's installer WiFi. Once there I am able to query the normal 'api' for the device details. This is referenced in the documentation for this integration.
https://us.sunpower.com/sites/default/files/tech-note-commissioning-equinoxtm-systems-using-wifi-534241-004_0.pdf

I will be using another router or a cheap gli.Net router with the WiFi as the WAN and the Ethernet as the Lan and bridge them so that I can reconnect it to my networking gear and maintain pulling data that way. <-> the RaspberryPi equivalent would be to use the Pi's WiFi for the PVS instead of your local network and the Ethernet port for the local network instead of the PVS.

My planned connection:

My home assistant server <-ethernet-> my router
My router <-ethernet-> gli.net router <-wifi-> PVS

from hass-sunpower.

Certainly we'll not try to crack the usual SSL/HTTPS encryption which is reported to be applied to the data between our Sunpower PVS6 and the SunPower servers on AWS or elsewhere on the internet.
On the other hand the battery information must be communicated between the Conext Gateway and the PVS6 on either the internal ethernet or the RS-485 interfaces. So which is it? Why does the PVS6 have both interfaces connecting it to the SunVault (battery box)? - which interface is used for which purpose?

As a very last resort I suppose one could create a logged-in web session which extracts the information from the MySunpower monitoring web pages...

from hass-sunpower.

I found the ESS status is available on the local PVS API via the following URL:

http://172.x.x.1/cgi-bin/dl_cgi/energy-storage-system/status

The output looks like the following:

{
	"errors":	[],
	"ess_report":	{
		"battery_status":	[{
				"battery_amperage":	{
					"unit":	"A",
					"value":	-24.2
				},
				"battery_voltage":	{
					"unit":	"V",
					"value":	52.1
				},
				"customer_state_of_charge":	{
					"unit":	"%",
					"value":	45
				},
				"last_updated":	"2023-05-06 04:07:32",
				"serial_number":	"BCXXXXXXXXXXXXXXXXX",
				"system_state_of_charge":	{
					"unit":	"%",
					"value":	53
				},
				"temperature":	{
					"unit":	"C",
					"value":	26.400000000000002
				}
			}],
		"ess_state":	[{
				"operational_mode":	"SELF_CONSUMPTION",
				"permission_to_operate":	true,
				"storage_controller_status":	"RUNNING"
			}],
		"ess_status":	[{
				"enclosure_humidity":	{
					"unit":	"%",
					"value":	20
				},
				"enclosure_temperature":	{
					"unit":	"C",
					"value":	35
				},
				"ess_meter_reading":	{
					"agg_power":	{
						"unit":	"kW",
						"value":	1.223
					},
					"last_updated":	"2023-05-06 04:07:35",
					"meter_a":	{
						"reading":	{
							"current":	{
								"unit":	"A",
								"value":	4.48
							},
							"last_updated":	"2023-05-06 04:07:35",
							"power":	{
								"unit":	"W",
								"value":	552.07040000000006
							},
							"voltage":	{
								"unit":	"V",
								"value":	123.23
							}
						}
					},
					"meter_b":	{
						"reading":	{
							"current":	{
								"unit":	"A",
								"value":	4.5200000000000005
							},
							"last_updated":	"2023-05-06 04:07:35",
							"power":	{
								"unit":	"W",
								"value":	556.9996000000001
							},
							"voltage":	{
								"unit":	"V",
								"value":	123.23
							}
						}
					}
				},
				"last_updated":	"2023-05-06 04:07:32",
				"serial_number":	"XXXXXXXXXXX_M00XXXXXXXXX_M001XXXXXXXXXX"
			}],
		"hub_plus_status":	{
			"aux_port_voltage":	{
				"unit":	"V",
				"value":	11.221
			},
			"contactor_error":	"NONE",
			"contactor_position":	"CLOSED",
			"grid_frequency_state":	"METER_FREQ_IN_RANGE",
			"grid_phase1_voltage":	{
				"unit":	"V",
				"value":	123.9
			},
			"grid_phase2_voltage":	{
				"unit":	"V",
				"value":	123.80000000000001
			},
			"grid_voltage_state":	"METER_VOLTAGE_IN_RANGE",
			"hub_humidity":	{
				"unit":	"%",
				"value":	34
			},
			"hub_temperature":	{
				"unit":	"C",
				"value":	26
			},
			"inverter_connection_voltage":	{
				"unit":	"V",
				"value":	0.277
			},
			"jump_start_voltage":	{
				"unit":	"V",
				"value":	1.397
			},
			"last_updated":	"2023-05-06 04:07:35",
			"load_frequency_state":	"METER_FREQ_IN_RANGE",
			"load_phase1_voltage":	{
				"unit":	"V",
				"value":	123.9
			},
			"load_phase2_voltage":	{
				"unit":	"V",
				"value":	123.7
			},
			"load_voltage_state":	"METER_VOLTAGE_IN_RANGE",
			"main_voltage":	{
				"unit":	"V",
				"value":	11.375
			},
			"serial_number":	"SYXXXXXXX-XXXXXX.XXXXX"
		},
		"inverter_status":	[{
				"a_n_voltage":	{
					"unit":	"V",
					"value":	123.23
				},
				"ac_current":	{
					"unit":	"A",
					"value":	4.5
				},
				"ac_power":	{
					"unit":	"kW",
					"value":	1.223
				},
				"b_n_voltage":	{
					"unit":	"V",
					"value":	123.23
				},
				"last_updated":	"2023-05-06 04:07:35",
				"phase_a_current":	{
					"unit":	"A",
					"value":	4.48
				},
				"phase_b_current":	{
					"unit":	"A",
					"value":	4.5200000000000005
				},
				"serial_number":	"00001XXXXXXX",
				"temperature":	{
					"unit":	"C",
					"value":	35.13
				}
			}],
		"last_updated":	"2023-05-06 04:07:36"
	}
}

The PVS's local API is documented by a Swagger JSON embedded in the firmware image.

from hass-sunpower.

One last thing - I still haven't figured out how the PVS obtains this information. It might be over RS-485, it might be Modbus over TCP on the Ethernet cable (probably SunSpec), it might be another API on the Conext Gateway, it might be something else.

from hass-sunpower.

If you look at each device, in the '172.x.x.1/cgi-bin/dl_cgi?Command=DeviceList', there is an 'Interface' that lists the communication method that I suspect each device communicates with the PVS by. The Batteries are 'none', the Meters & Micros are 'mime' and the 'SchneiderElectric-ConextGateway' & 'Storage Inverter' & BMS are SunSpec. The 'Hub+' and others that are ttymxc5.
If the Interface is SunSpec or ttymxc5 there is additional info under 'Port' that will identify the comms and unique identifier for each device.
I believe this will give a clue as to where the communication method is.

from hass-sunpower.

@koleson I believe it does. I believe I have an idea how to perform that integration.

from hass-sunpower.

Great job finding that endpoint, I'm getting the same response as well.

The app doesn't report the actual timestamp of its data, so who knows how delayed it actually is. I'm showing a difference of 8% from system_state_of_charge to customer_state_of_charge, and a ~5% difference between customer_state_of_charge and the SunPower app. I hope this is helpful in normalizing the numbers for the HA dashboard.

Happy to test a pre-release if you need testing help.

from hass-sunpower.

Please add me too for any needed help on pre-release testing!

from hass-sunpower.

I have developed a simple python script that is able to pull the json live and parse it to get the values. I am now attempting to integrate it.
@krbaker Are you able to assist?

from hass-sunpower.

@heyhewmike I can certainly try, I'm happy to take a patch too, but I will look (its a little harder not having ESS myself). Do you have a link to the script you wrote?

I'm thinking I might split the ESS integration into its own integration? Thoughts?

from hass-sunpower.

@heyhewmike I can certainly try, I'm happy to take a patch too, but I will look (its a little harder not having ESS myself). Do you have a link to the script you wrote?

@krbaker here is a link.
https://github.com/heyhewmike/hass-sunpower/tree/beta-ESS/custom_components%2Fsunpower

There are 2 files that are of importance. sunvault.py and sunvault.output.txt (output of what the script returns).

I also started trying to edit the const.py to incorporate the values in.

The comment here shows the actual PVS response when queried.

I haven't been able to do much since the pvdr tripped in my hub+ and now my ess gateway is under an RMA.

I'm thinking I might split the ESS integration into its own integration? Thoughts?

I am just sticking my toe into coding here. My first thought with a separate integration would mean having 2 to update each integration with each PR or update made to the current integration.
If possible during the configuration, or reconfiguration, of the integration to be able to select if you have an ESS or not and provide a logic to either ignore (No ESS) or use (Yes ESS) the sunvault code might be easiest long term as you will have a single code base to maintain. However setting it up in the first place might be more difficult.

Easier path now might not be the best long term.

from hass-sunpower.

Just wanted to put a note here: I found out a few months ago that the PVS-ESS communication no longer uses RS-485 - it's now just Modbus over TCP on the (black?) Ethernet cable. I haven't fussed with that yet - I have an ongoing issue where every 5-6 months my PVS6 has to be replaced, and I don't want to give anyone any reason to try to blame me for the problem.

from hass-sunpower.

I have an ESS system and would be happy to help contribute to this endeavor. Let me know how I can help gather data or test out commands on my system.

from hass-sunpower.

I recently had a short in a comm wire between the PVDR and the ESS that caused a major headache and loss of production over a month period.

I am not going to be able to assist anymore.

Sadly I lost my Spouse Approval Factor, SAF, so I am no longer able to integrate the PVS monitoring into my HA for automation and data analytics. There was fear from my spouse that what I did for the integration would be seen by the MFG as modifying the system and voiding a warranty.

from hass-sunpower.

@krbaker Is there anything I can do to help get the battery information imported into HA?

from hass-sunpower.

I've been toying with https://github.com/CanisUrsa/hass-sunpower/tree/refactor_with_ess for the last couple of days, and aside from some fairly minor foibles, it's doing the thing as best I can tell. (As in it's populating all the data I'm keen to get my grubby hands on.)

Admittedly, I've not taken the time to debug the status quo extensively, but I did add an else to return {} for https://github.com/CanisUrsa/hass-sunpower/blob/af69f9c7e212c6e3ef4dcb7e8aab42aa2dce85cd/custom_components/sunpower/__init__.py#L47, as I was seeing polling results flap everything to "unavailable" every few minutes. If it fell through into https://github.com/CanisUrsa/hass-sunpower/blob/af69f9c7e212c6e3ef4dcb7e8aab42aa2dce85cd/custom_components/sunpower/__init__.py#L60 and beyond, there was nothing in the data dict for it to operate on; so errors ensued.

Aside from that, the SunVault, HUB+, BMS, and ESS all report a state of "error", which I've not yet put any effort into figuring out. (My http://foo.bar.baz/cgi-bin/dl_cgi/energy-storage-system/status endpoint does show an error code, so that may well be reasonable.)

If there's some level of interest in getting the ESS bits into a more consumable place, and addressing the deprecated constants etc, I'd be happy to lend a hand. (No experience with the HA side of things, but somewhat Python literate.)

from hass-sunpower.

I'm working on pulling in an ESS branch, you can follow here: #61

from hass-sunpower.

moving this to be a full release now that at least one user has gotten data.

from hass-sunpower.

I've also moved over to the updated release as of this evening... so far, it seems all is well! 🤞

Thanks for making it happen folks! 🤘

from hass-sunpower.

marking this as closed, feel free to open bugs (I don't have an ESS myself) but I hope any end up just being improvements etc.

from hass-sunpower.