You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Very fruitful - finally understood how the mySunPower app's SunVault mode changes are communicated to the PVS6, which performs the system control described by the modes, and found a potential method of changing the command and data acquisition server from one in the cloud to one that is locally controlled.
Future Experiments
Emulated EDP MQTT Server
Tools
Rooted PVS6
Probably Swift and an MQTT library
Protobuf schemas extracted from communicator
Risks
Minimal - don't have to try anything risky to see if it fundamentally "works" with a spare
Rewards: Very High
Reuse nearly all PVS6 code, avoiding the need to re-implement C++ binaries with lots of logic
Completely obviate the remaining reason to need to connect to SunPower at all (battery control)
Protobuf shows there's a wide swath of control and telemetry available over the EDP channel
Questions to be answered
Can communicator connect to an MQTT server with no mTLS?
would make setup and debugging much simpler
What data does communicator send to its EDP server? On what interval?
would probably be much better for self-monitoring than dl_cgi API
push vs pull
async vs sync
per-device intervals vs fixed interval for all devices
many device characteristics in DeviceList are nearly static
some device characteristics in DeviceList take an extremely long time to ingest and should be queried rarely
and certainly shouldn't be queried alongside all the other data that can be gained in a short time
What EDPCommands can we document as working?
there are a ton of them, and probably most of them should be left alone.
Automatic U-Boot from USB
Tools
The U-Boot console password
U-Boot docs
manual commands issued to make PVS6 boot from USB
Risks: Minimal
can usefully test this on a spare
Rewards: High
can create backups of SunPower firmware for use when eMMC dies of write exhaustion
can avert write exhaustion more or less completely by booting from USB normally
easy to make modifications on USB stick and test them in a system without modifying "pristine" eMMC firmware
USB sticks are cheap
USB sticks don't require SMD hot air desoldering / reballing / soldering when they fail
Questions to be answered
Is there some way to completely disable writing to / reading from the eMMC from operating once the system is booted from USB?
On systems with eMMC write exhaustion, it seems like attempting to access the eMMC can hang for long enough to make the watchdog reboot the PVS.
We also want to be sure that we aren't modifying the eMMC inadvertently when booted from USB
What is the text of the script to boot from USB automatically? What are the relevant variables?
Does this count as text I can release? Is it source code?
If so, could I describe it as modifications to the existing scripts?
Can we fall back to eMMC if the USB boot device is not present?
Is it worth investigating network boot?
RAM isn't terribly constrained on the PVS6
Since we don't need on-device redundancy, we need much less space than we would on the eMMC
Could probably disable a lot of services and delete a lot of files once we know SunPower won't be updating them!
and/or we don't want their updates
Persistent storage locations
Questions to be answered
Where is persistent state stored for various components between boots?
Summary of results to date + possible future experiments
It turns out that it is possible to control many system aspects via MQTT.
Summary: This approach allows controlling the battery discharge mode (i.e. Cost Savings/ENERGY_ARBITRAGE or Self-Supply/SELF_CONSUMPTION or reserve) and reserved SoC (e.g. save 40% for backup, save 4% for backup) - among many other system parameters - using a simple protobuf-emitting MQTT server. (To my knowledge, there was no known way to do this without SunPower's cloud services until now.) It also allows a more efficient capture of system monitoring data, as the PVS emits data asynchronously and on its own timetable rather than in response to a synchronous long-running HTTP request that has numerous asynchronous internal datasources as with dl_cgi.
This approach also allows us to use nearly all of the code in the PVS6 without having to re-implement it, which is an absolutely ENORMOUS win given that much of it is in difficult-to-modify C++ binaries, we don't have much of a spec, there's a ton of logic in the binaries and libraries, and testing any re-implementation on a live system is certainly unwise and indeed probably dangerous and potentially costly.
In production, the PVS6 connects to AWS IoT Core, a managed MQTT service. The MQTT connection is authenticated using mTLS: mutual TLS, where both the client and the server demonstrate holding their private key and the other side's certificate. These certificates are held in the app0 mount. The PVS6's serial number is used as the MQTT client ID. MQTT 3.1.1 protocol is used.
The communicator process that initiates the connection to MQTT is configurable through command line flags, including a flag to set the MQTT host, flags to set the various input and output topics, and flags to set the mTLS certificate and private key files.
Over the MQTT channel, protobuf-encoded messages are exchanged. The protobuf schema files are embedded in communicator and can be extracted with protodump.
It seems that the WAN port must be up for communicator to attempt communicating with the MQTT server even if the MQTT server is reachable via the LAN interface.
Doing something like this Meshtastic Home Assistant component, decoding and encoding protobufs and using the internal HA MQTT server, might make this method accessible to a broader swath of people.
It also might be possible, if the SunPower servers stay up for a while yet, to build something like Juicepassproxy or IOXY that is a dedicated transparent MQTT proxy with poisoned ARP and DNS: the PVS's DNS servers are hard-coded in various places to Google DNS (8.8.8.8 / 8.8.4.4), so it's a bit more of a brutalist approach. But that approach won't work once the SunPower AWS IoT servers stop accepting connections / renewing certificates for clients based on server-side rules (i.e. only renewing for Lease/PPA customers), which could be any day now.
The biggest constraint for both of those approaches will continue to be that the PVS needs to be "rooted" in order to change communicator's connection parameters or obtain the certificate/key material from app0.
Original experimental plan/results
Tools
AWS IoT Embedded C SDK ended up just using various MQTT clients + this Amazon doc
client/[serial]/live - send live data that probably gets piped right back out to the mySunPower app
probably just the useful subset described in the protobuf EDPLiveData below
communicator can be configured!
/home/communicator/bin/communicator--capath [root.crt]--cpath [device_cert.crt]--pkpath [private_key.pem]-u [hostname, sometimes called endpoint]--tpclcmd=[command_topic--tpcmdrsp=[command_response_topic]--tpdata=[data_topic]--tplive=[live data?]--tpevent=[event_topic]--tptoi=[technical_topic]--tpsynctime=[time_sync_topic]
changing the configuration with command line params seems to persist the updated configuration to /app0/app_data/communicator/communicator.cfg in protobuf format (PBCommunicator.proto schema)
it looks like you can't change the port from the command line
8883 is the default port
not much reason to change this because mTLS auth is required
it looks like mTLS auth is required
with no certs in the expected app0 folder, communicator systemd service exits quickly with message
it looks like the certs have to be valid according to OpenSSL in the formatting sense - not checked against system trust store
following generic instructions to set up a CA with OpenSSL, make a client cert signing request and fulfill the request seems to work
can use SEP output to just write output to a file (? untested)
contains numerous interesting protocol buffers (not all of them are listed here)
DcmEnableState
bool ctrl_enabl_fl - Control enable flag, one assumes
bool edp_log_enabl_fl - EDP logging enable flag, one assumes
float dchrg_min_th_soc_val - minimum discharge SoC as a percent
float fcst_tgt_soc_val - ? (forecast target SoC value not used elsewhere in my experience)
uint32 fcst_ntvl_qty - ?
.EDPMessaging.DcmEssChargeConstraint chrg_cnstr_enum - charge constraint as described below
bool sgip_compl_enabl_fl - SGIP (California Public Utilities Commission Self-Generation Incentive Program, rebate payments for DER) flag of some kind
SGIP has stipulations on how an ESS can be charged to be eligible for rebates.
In short, you can only charge from solar.
uint32 ess_yrly_dchrg_cyc_qty - ESS yearly discharge cycle quantity; perhaps a target for systems that can charge from sources other than solar?
string tz_str - time zone string, probably in tz database / Olson database format
.EDPMessaging.DcmCtrlType ctrl_type_enum - exposed as Self-Consumption/Cost Savings/Reserve in the mySunPower app - detailed description below
ess_exp_enabl_fl - ESS (grid) Export Enable flag
Some installations (HECO, NEM 1/2 Zero-Export retrofit) are disallowed from and/or don't economically benefit from exporting energy to the grid
uint64 vld_fld_msk - "valid fields mask"?
BackupParameters - allows setting a DcmEssOpMode to replace the current mode until an end time
EDPHeader - msg_crt_eps timestamp (?), serial number, model number, command token (string)
VppEvent - describes a Virtual Power Plant Event
string vpp_evt_uid - event ID for scheduling, cancelling, etc
uint64 vpp_evt_start - epoch timestamp of when the VPP event starts
uint32 vpp_evt_duration - length of the VPP event in seconds
float vpp_evt_reservesoc - percentage of battery that should be reserved for customer use
bool vpp_evt_export_constrained - maybe in the overall program setup there's a max discharge power dsecription; if flag is true, only export that much power rather than running inverter full-bore
.EDPMessaging.DcmEssOpMode vpp_evt_exitmode - when the VPP event ends, change to this discharge mode
EDPCommand - a command from EDP (the cloud)
SendBufferedDataFromTimeInterval - probably what it says on the tin; max keys, start/end timestamps
GetDataLoggerParameters/SetDataLoggerParameters - get/set several parameters at once
UpgradeFirmware - from included upd_spec_url
RebootSystem
ResetEnergyStorageSystem
ConnectToSSHServer with string srvr_id and uint32 fwd_port_no, then CloseSSHConnection
TriggerDeviceDiscovery and then asynchronously GetDeviceDiscovery
ChangeDeviceSerialNumber
SetEphemeralAccessCode - not sure what this does at the moment!
WiFiSetCredentials - unclear if this is for broadcasting or connecting to an AP, but ssid/bssid/pwd are there
WiFiGetStatus/WiFiScan/WiFiForgetNetwork
ZigBeeSEPGet/ZigBeeSEPSet
METStationCalibrationGet/METStationCalibrationSet
SiteOperationalGet/SiteOperationalSet
NetworkInterfaceReportGet
CTScalingFactorGet/CTScalingFactorSet
AdhocCommand - array of AdhocData
DeviceDisassociate
CellPrimarySet
DcmControllerParametersGet/DCMControllerParametersSet (member types described elsewhere)
The PVS6 (aka PV Supervisor 6) is a monitoring and control system for SunPower solar systems installed from 2019.
On August 6, 2024, SunPower filed for Chapter 11 bankruptcy.
As of September 28, 2024, the mySunPower app that allows consumers to view monitoring data and control SunVault systems is still online. Much of the monitoring data provided by the app is available on the local dl_cgi API, although problems with flash wear and partitions filling up have been reported with long-term use of that API.
The SunVault (aka Equinox storage) is a battery time-shifting / backup / virtual-power-plant system announced in September 2019 and discontinued in early 2024.
Unfortunately, as of September 28, 2024, I'm unaware of any local API on the PVS6 to control the SunVault's settings. The SunVault system seems to receive discharge mode, discharge minimum SoC, consumption power, production power, and grid power information from the PVS6, conveying that information over RS-485 and a protocol that looks like Modbus. Numerous parts of the system are SunPower proprietary, so spares and repairs are expected to be a challenge.
These notes document the hardware and software of these systems as found through reverse engineering of spares.
Motorola/ON Semiconductor MC74ACT541 Octal Buffer/Line Driver with 3-State Outputs
Skyworks Si88243EC quad digital isolator + DC-DC converter (up to 100Mbps)
Up to 6 current transformer measurement channels (3 populated - Production, Consumption 1, Consumption 2; additional channels require population of twin STM32)
very old Hub+ PVS6 6.02 had U-Boot 2017.03-staging-prod-adama+g277c30979b (Dec 02 2019 - 08:32:49 +0000)
upgrading PVS6 firmware to latest did not (immediately, at least) upgrade bootloader
newer PVS6 6.02 without ethernet ports had U-Boot 2017.03-imx_v2017.03_4.9.88_2.0.0_ga+geaeb74722e (Sep 09 2021 - 07:42:26 +0000)
possible to boot directly via USB with custom boot script on newer build
Older build (Dec 02 2019 - 08:32:49 +0000) shows USB hubs but does not see attached USB Mass Storage devices
Also reported by various folks using U-Boot for NXP devices going back to 2014, 2015
Would be nice to try updating U-Boot but I don't have a recovery plan if it doesn't work
U-Boot is GPL v2+, so in theory someone who has a device with the newer U-Boot build could ask for the as-built-and-distributed source code for the newer build.
to set the LED to green, start USB subsystem, load a Linux zImage and flat device tree from a sole connected USB Mass Storage:
set_pvs_led g
usb start
ext4load usb 0:0 ${loadaddr} ${image}
ext4load usb 0:0 ${fdt_addr} ${fdt_file}
setenv bootargs console=ttymxc0,115200 root=/dev/sda rootdelay=2
bootz ${loadaddr} - ${fdt_addr}
rootdelay=2 is necessary to allow the USB drive to "start up" even if it's a flash drive
we're mostly reusing built-in env vars here, but you can use ext4ls to snoop around / make sure the USB drive is working
possible to boot via TFTP
but not super useful without a local rootfs!
can boot kernel via TFTP and use USB rootfs
to set the LED to green, get an IP via DHCP, and load a Linux zImage and flat device tree via TFTP (that you populated with the relevant files from a firmware rootfs)
rootdelay=5 is a bit longer than the delay needed if booting directly from USB
this only loads the kernel and device tree via TFTP, which can get you bootstrapped if your PVS is on the old (Dec 02 2019) version of U-Boot
Linux Kernel 4.9.88 - built with Yocto
Firmware updates are verified by the firmware update script against a public key stored on the device.
The update script allows verification to be skipped by passing a flag.
Automatic updates will NOT BE PERFORMED if flashwear levels are too high
Uses jfrog for a custom apt repo for several tools / holdbacks / custom versions (SSL Cert auth)
Uses D-Bus for a lot of IPC / messaging / etc
org.sunpower.ESMM seems to be the ESS management D-Bus.
used for
dcm
DCM (Discharge Management? Discharge Control Manager?)
VPP (Virtual Power Plant - utility-commanded discharges during peak hours)
PCS (Power Control System - see elsewhere in document)
esmm
reset ESS
superman
site operational get/set
grid profile get/set
Uses custom binaries and libraries, apparently written in C++
Also uses some Python 2.7, including site-packages
/app0 appears to be related to an AWS IoT layer using MQTT - perhaps the main EDP reporting/control complex?
probably uses the AWS IoT Device Embedded C, maybe as open-sourced here
uses per-device cert-based auth
certs need to be rotated regularly for continued access
3-month validity
Can be controlled by crafted SMS messages to the cellular modem
check system uptime
toggle logging
reboot/reset/hibernate
restore
switch boot partition
detailed status
cellular up/down/reset/set primary / always-on
get/set WAN/LAN/wi-fi mode/SSID
establish arbitrary SSH tunnel
execute firmware update from URL
set configuration
force cert client to run to refresh against given endpoint
SSH is open, root has a password set but SSH will only accept public key auth
data_logger user has most binaries for controlling and updating other hardware subsystems
mime user has binaries for PLC communication with microinverters
envoy_manager - relatively new, has configurations for envoy, powerwall, gateway, inverter, livedata, meter, gateway, so it seems to be ESS-related
varserver / Variable Server - somehow related to envoy_manager
ieee2030.5
superman
installcheckd
seems to be part of the ESS commissioning process
communicates over websocket using WebSocket++
port 9002?
telemetry-ws
communicates over websocket using WebSocket++
port 9002?
Uses two partitions to handle updates - mmcblk1p5 and mmcblk1p6 (~256MB)
update image is installed to non-active partition, persistent data is copied, migration scripts are run, and system reboots to formerly non-active partition.
see partition map secton below for more info
Uses systemd for numerous periodic tasks.
Curiously, this includes rebooting the PVS every day at 07:00 UTC
unless the PVS6 is attached to a SunVault that is offline or fewer than 24 hours have passed since commissioning.
ttymxc2 is connected to a PSoC 4 "watchdog"
COBS communication
9600 baud
PSoC 4 can also be addressed over GPIO with bitbang comms
hilariously, capable of accepting morse code commands via a button (maybe not populated on factory PVS6, maybe a jumper?)
short press: enable WiFi AP
long press: soft reboot
"eternal" long press: detected but not used
F: factory reset
N: network restart
P: toggle PTO mode (Permission to Operate?)
S: WPS enable
LEDs should reflect accepted inputs
ttymxc4 seems to be the UART connected over the gray 10-pin ribbon cable to the "sidecar" power monitoring board
I think the other end is the Cypress CY8C5668AXI PSoC5 chip, but haven't verified.
ttymxc5 seems to be RS-485 2-pin
Can do firmware updates from a USB stick. Looks for a lua script at sunpower/PVS6/fwup.lua upon USB storage device insertion.
It appears the upgrade script signature is validated against a public key by fwup. Passing -k NONE disables this check.
A swagger spec for the dl_cgi API is available at /cgi-bin/swagger.json on the PVS.
An example fwup.lua URL is present in that swagger.json.
Note that the rootfs archives are sometimes in xz compression format in spite of the .tgz extension which usually means gzip compression.
Use a hex editor to validate magic number at start of file
xz: FD 37 7A 58 5A 00
gzip: 1F 8B
SSH tunneling seems to go through pvsfleet.us.sunpower.com
Live data seems to go through edp-api.edp.sunpower.com
Certs for MQTT/AWS IoT seem to be retrieved via https://register.edp.sunpower.com:3001/ (unsure of how auth'd - SSL client cert?)
Google DNS (8.8.8.8 / 8.8.4.4) seems to be hard-coded as DNS for the device, seemingly ignoring DHCP-provided DNS, making DNS redirection more difficult.
ESS Control
based on GraphQL API (as used here) we know a few strings used for control
ENERGY_ARBITRAGE is one; matching binaries might involved in ESS control
or maybe they just need status info and there's another representation for control
8 / mmcblk1p8 - 2.4GB: app0 mount - app_data and secrets
Unresponsive PVS6 with rotating "rainbow" LED
yellow/green/red, random order/timing, often several seconds between changes
is a symptom of a write-exhausted eMMC.
background: unlike magnetic storage, eMMC/flash memory in general has a finite lifetime of writes.
writes happen as "block writes", so even just changing one character results in a block write if flushed to eMMC.
a common mistake is to flush writes too frequently and exhaust - run out of - writes.
after this exhaustion, the eMMC will not write reliably or will prevent writes permanently.
the PVS has a flash wear check that occurs regularly
if the flash wear exceeds some value, it will report high flash wear to SunPower servers
it will also stop automatically updating the firmware
the UART will be alive and printing, and ethernet lights might be on, but the system won't boot.
the eMMC is still able to be read to pull U-Boot and whatever environment variables were set before it ran out of writes
but those environment variables cannot be saved to the eMMC
it seems the eMMC spec includes extra resiliency for the boot areas
attempting to read the eMMC outside the boot areas of a write-exhausted eMMC has resulted in timeouts for me, oddly.
this implies that if one wishes to be able to use the PVS6 past the useful life of the eMMC,
steps must be taken to change the bootscripts/env vars to point to other media before the writes are exhausted,
for example, copying the eMMC to a USB flash drive and booting from that with something like:
usb start
ext4load usb 0:0 ${loadaddr} ${image}
ext4load usb 0:0 ${fdt_addr} ${fdt_file}
setenv bootargs console=ttymxc0,115200 root=/dev/sda rootdelay=2
bootz ${loadaddr} - ${fdt_addr}
note also that there are hard-coded references in the firmware to mmcblk1 that would need to be updated
for a new root/boot device.
Enphase Retrofit
As of September 2024, Enphase offered customers with Enphase microinverters the opportunity to purchase an Enphase monitoring system.
The system is based on the IQ Combiner 4 which works with IQ6/7 grid-tied systems and IQ8 grid-forming systems.
The IQ Combiner includes an integrated IQ Gateway.
Unclear how/if this will work with the SunVault
As of September 28, 2024, Enphase's response is "it doesn't" which makes sense
Less clear is "why it doesn't work"
Does SunPower PCS use MI data directly for SunVault discharge regulation?
Does Enphase's drop-in change the keys on the MIs such that a PVS would be "out of the loop" on MI data?
it seems like production/consumption CTs on the PVS are necessaryandsufficient to run PCS, but maybe I've missed something
designed in Canada, so was there ever really any other choice?
an SD card is installed
the slot is covered with a sticker
the PVS6 checks the SD card's contents on occasion
can run an SSH server but it is disabled most of the time
the PVS6 can enable SSH through some mechanism I haven't yet uncovered
Interfaces
CAN: PVS6 to MIDC (obsoleted in later versions)
RS-485: PVS6 to MIDC (blue cable with RJ-45/8P8C ends), MIDC to PVDR, PVDR to MIO - SunSpec bus?
Multi-battery systems: additional XW Pro inverter/chargers for added AC output, additional batteries for added energy storage. Only one Gateway is needed for any number of inverters and batteries; each inverter needs an MIO.
MIO: CAN to Schneider Gateway, CAN to battery, Aux temperature input, XW-RPO input, "Comm from Hub+", comm to next MIO, ESS enclosure fan AC power in/out, LED board output, 48VDC to XW
A lot of this is applicable to reverse engineering in general.
Spares
A lot of stuff that would be terrifying to do to / with production hardware is much less scary when you're operating on a spare.
eBay can randomly be your best friend.
If they're inexpensive enough, even broken spares can be a worthwhile investment. In some cases, they can even be more useful than working spares: when something fails, other parts can exhibit useful new behavior.
I spent an incredibly long time trying to get root access to my spare PVS6, but without the U-Boot console password, it seemed hopeless. A used (RMA'd, as it turned out) PVS6 that I acquired cheaply turned out to be a godsend: the eMMC was write-exhausted and therefore U-Boot could not read the firmware image, so the boot script failed and dropped me to the U-Boot console where I could poke around and figure out how to boot from USB as I had been trying to do for years.
After I was able to boot from USB, plans that had been coming together for ages moved forward quickly.
Hardware
Logic Analyzer
For a long time I misunderstood the purpose and operation of a logic analyzer.
An oscilloscope captures and displays analog waveform data, maybe over one channel, maybe over a few.
A logic analyzer captures and displays (often on a computer) digital waveform data, usually over several channels. But hey, you can also analyze a single pin. It samples and captures digital pins at a given sample rate.
Since inter-IC communication is digital data (really, voltage high and low), we can use a logic analyzer to capture the data for later analysis even if we don't know the underlying protocol or data rate.
Analysis is part brain power and context - like what protocols might be in use and what data we might find in the capture - and part computation - like applying a UART analyzer to some captured timing data to see ASCII characters being communicated.
Even an inexpensive ($12, for example) logic analyzer makes quick work of understanding what data is being sent over a known communications pin or trace. Most of the communications going on in these systems is very low-speed and is often serial, but of unknown baud and parity and so on. A logic analyzer and some timing data makes discovery of the correct settings fast and easy.
In fact, I would recommend using an inexpensive LA for these parts: though it hasn't happened to me so far, it would be a darn shame to be poking around with something expensive and have it go "zap."
Get started with something you control fully - an Arduino project with an i2c sensor and firmware you control and understand - before poking around PCBs you aren't familiar with.
UART breakout
Once you've logic analyzed your way into the correct settings, a UART will let you capture data more easily and with less fuss. Soldering one to the relevant pins will save you a lot of sanity - buying small CP2102/CP2014/similar breakouts from Aliexpress or similar in bulk makes committing one to a circuit that much less of a hassle.
JST-XH connectors can be re-pinned and used with 2.54mm spaced pin headers like the main UART on the PVS6 if you don't want to solder.
Often, you can be happy with just connecting GND and breakout RX / target TX. Don't connect breakout TX / target RX unless you plan to use it. VCC can also usually be skipped in my experience and it's one fewer thing to go wrong.
Evaluation Kits
Getting a sense for how things work when they aren't integrated with the system is much easier than when they are programmed with some source you don't have compiled into firmware binaries you probably also don't have. Spending money here can add up quickly, but when you're working with systems that are difficult to repair and replace, they're a much safer place to test hypotheses.
Sometimes you can also get binary firmware blobs running on such eval kits. The eval kits usually have PCBs and headers that are easier to navigate than shipping products.
Software
Your favorite Linux in a CLI virtual machine
I use Debian inside UTM. Reading and dumping ext4 and so on is that much easier. I ssh in from my regular OS terminal.
Executable binary in in, embedded protobuf files out. You can usually spot files containing them in a hex editor or in a grep for .google.protobuf.
Documentation
Datasheets
Any time you see an IC, look for its datasheet. Bullet points about what interfaces and peripherals are available can give you a sense of what ICs might be talking to each other before tracing them on a PCB. In 2024, microcontrollers are way overprovisioned in terms of peripherals, so don't read too far into them being utilized simply because they are present.
Most fun pinouts to look for, because they're easily transparently eavesdropped in-situ and in-use:
UARTs - RS-232 and RS-485
i2c
SPI
ethernet
Others that might be interesting:
JTAG
User Manuals
Not always the richest, but can give you a more complete understanding of what the thing can do.
Installation Manuals
Usually a step up from User Manuals, often includes more detail on what talks to what.
FCC certification documentation
If it has a radio in it, you can probably find a PDF of photos of the internals of the system and what frequencies the radio operates on by searching for "FCC (model number)".
Less fun than tearing the thing down yourself, but if you don't have a spare, it's a great place to start.
Hi, thank you for the work you're doing here. Very helpful in a community who desperately needs it right now. Do you have any idea where the admin password might be stored for the Schneider InsightFacility? I have been unable to get SunPower support to help me with my battery for obvious reasons, so I've taken to poking around myself. The user and guest accounts are default passwords, but the admin account is not.
@CraftyCanine I haven't been able to dig as much on the Conext Gateway / Insight Facility because I don't have a spare and they're pretty pricey even in the used market.
If it's any consolation, I don't think there's much of help in there: the PVS6 runs the PCS (Power Control System) that tells the XW inverter how much to charge/discharge the battery and if the PVS6 isn't commissioned for the SunVault I wouldn't expect much to happen.
SunPower won't be much help themselves, but a SunPower-authorized third-party installer should still be able to commission the system for you.
I have several MI, the Parthenon solarbridge 250w and 320w. I haven't found any PCB/firmware info for these units. I also have several PSV5x units that I have taken apart and started looking at the firmware for.
I don't know of any installer that can or would work on any of the older systems without the newer enphase MI. Atleast I haven't found anyone in the SF bay area that is capable of updating or activating a PSV5 with Solarbridge inverters.
@benhadad I don't think anyone can commission a PVS5/PVS5x anymore. There's a step in the commissioning process where the PVS5 checks for a firmware update, but it always fails. I think SunPower simply took down the endpoint when they transitioned all installers to ProConnect.
The PVS6 originally used the same mechanism but now uses a different one where ProConnect uploads the firmware to the PVS6 directly rather than the PVS6 downloading it from the web.
I think the ProConnect app still calls various local APIs to control the PVS, SunPower just removed the UI for those calls and created an app instead, so they can control who is able to log in to the app. For example, there is the POST /cgi-bin/dl_cgi/commission/config command. Unfortunately, I only have my active PVS so I can't test it, but all of the non-destructive commands I found in the swagger spec work fine, so I would guess that one does too.
I don't think all the relevant endpoints are present on the PVS5x. I could be wrong, but the firmware upload endpoint returns the same error as endpoints that for sure aren't there. (PVS5x Firmware 2018.1)
@CraftyCanine I'm starting to think you're right actually. The Pro Connect release notes talk about PVS5 support. I might be looking in the wrong place.
Does anyone know which component actually runs the BMS? I got into the user mode on the InsightLocal, and from what I could see, it only shows 4 devices in my case, the two XWs and two SunPower v1.2 BMS. However, as best as I can tell, the CAN is daisy chained through all four battery packs.
Side note, the fault codes on the system showed that it temporarily lost comms with its “SunSpec Controller” when I was interposing the ethernet switch. Seems like perhaps the PVS6 is controlling the XW with modbus over ethernet.
From that I found out the PVS6 basically sets the max charge and max discharge power of the XW in accordance with current consumption/production and discharge mode (cost savings/arbitrage vs self-consumption).
There are some register maps in that repo that might be helpful. If I remember right, the BMS is exposed over Modbus TCP.
@benhadad I don't think anyone can commission a PVS5/PVS5x anymore. There's a step in the commissioning process where the PVS5 checks for a firmware update, but it always fails. I think SunPower simply took down the endpoint when they transitioned all installers to ProConnect.
The PVS6 originally used the same mechanism but now uses a different one where ProConnect uploads the firmware to the PVS6 directly rather than the PVS6 downloading it from the web.
It fairly easy to get past the firmware check, however when you get to the very end to do the final commissioning it requires a pro account login for the final approval.
I have disassembled a 320W MI solarbridge. It was a painful process as it potted with what appears to be a silicon foam, which I so far haven’t found a better way than to remove than mechanical removal. I have tired all the legal removal chemistry, but have been told that real MEK or Tolune are really the best tools to use (which are both not available in CA). If someone needs an MI to work with I have a couple of damaged ones I could part with.
Does anyone know which component actually runs the BMS? I got into the user mode on the InsightLocal, and from what I could see, it only shows 4 devices in my case, the two XWs and two SunPower v1.2 BMS. However, as best as I can tell, the CAN is daisy chained through all four battery packs.
Side note, the fault codes on the system showed that it temporarily lost comms with its “SunSpec Controller” when I was interposing the ethernet switch. Seems like perhaps the PVS6 is controlling the XW with modbus over ethernet.
From what I have seen with my digging it appears the BMS is built into the LiFePo4 cells themself and is communicated to through the "HOST" port. Then the Main BMS Controls all others based on which Comm Port the data comes in on, either HOST or COM IN. If you look at the Cells you will see that which ever one has the "HOST" cable will have COM OUT running to the second Cell COM IN. The HOST cable goes to the SunPower Multi i/O device.
I have reached out to Schneider Electric Support/Sales who has told me that we can reset the InsightLocal Facility UI login credentials without changing any of the configurations. I have also reached out asking their support about the BMS that I see in the InsightLocal Facility UI. The InsightLocal Facility is connected to the XW Pro via XanBus
There appears to be 2 forms of communication between the SunVault and the PVS. The Canbus and Ethernet. Both are connected to the InsightLocal Facility and the XW Pro is connected to the InsightLocal Facility and PVS via the SunPower Modbus Multi i/O.
I have begun searching into the FCC ID on the Multi i/O
Indeed, I would be very interested in resetting the admin password.
My thought is to perhaps swap out the Hub+/PVS6 guts with those from a Schneider BCS and an Enphase Gateway (for panel monitoring). Keeping the BMS configuration would be ideal for now, though.
I do wonder how autonomous the functions of the MIO boards are. Do they need the PVS6 to work at all, or will things like the status display, CAN interfacing, and fan control work headless?
Indeed, I would be very interested in resetting the admin password.
My thought is to perhaps swap out the Hub+/PVS6 guts with those from a Schneider BCS and an Enphase Gateway (for panel monitoring). Keeping the BMS configuration would be ideal for now, though.
I do wonder how autonomous the functions of the MIO boards are. Do they need the PVS6 to work at all, or will things like the status display, CAN interfacing, and fan control work headless?
I have 2 updates from Schneider Electric and Enphase.
Schneider Electric reports that they are aware that there are core components of theirs inside the SunVault but that SunPower has made some custom modifications. They are working with SP to become more familiar with it so that they can better support.
Enphase has told me that if I rewire my house so that the Schneider Electric XW Pro is between my House/Solar and the Grid they can install the IQ Gateway but their warranty will only be limited to Product Warranty of the IQ Gateway and nothing else.
Yeah, I pretty much assume that if I take any action to gain more control over my system that it would limit the warranty. That said, SunVault is conspicuously absent from their warranty resources and FAQ pages.
Yeah, I pretty much assume that if I take any action to gain more control over my system that it would limit the warranty. That said, SunVault is conspicuously absent from their warranty resources and FAQ pages.
Yes absent from the FAQ and warranty resources page but I am quoting the SE Solar Tech Support:
Schneider Electric can work with you with limited support and unit replacement that are under warranty with proof of purchase.
FWIW, the Sunvault PCS is definitely completely independent from module communications. I have some additional QCells panels with IQ8 MIs monitored with an IQ Gateway that are connected into my generation PAN. The PVS shows total system production and consumption from the CTs and manages charge/discharge of the XWs accordingly.
Related, I have not had any problems with PLC noise or crosstalk between the IQ8’s and the IQ7’s built into the SPWR panels. The two generations use different frequencies and protocols for their PLC. Unfortunately this does mean I would still need a second gateway to migrate panel level monitoring from the PVS over to my Enphase system.
That said, I don’t think it would be a good idea to try to rehome the panels from the PVS to an IQ gateway with both active on the same PLC domain. Without separating them with a PLC filter, I could imaging chaos ensuing with the commissioned PVS searching for and trying to set the grid profile of them at the same time as the gateway.
@dmbryson yeah, that's what i can't square about what Enphase reps have said (the IQ Gateway can run alongside the PVS6 and monitor the same panels without removing the PVS6, but it currently cannot be installed alongside a SunVault) and how PCS seems to work (it doesn't seem to rely on MI data directly - just uses the CTs).
@benhadad, over the past months I’ve noticed that several of my panels with SolarBridge MI320 microinverters would get into an oscillating run/shutdown mode especially when there was high production from the panel DC outputs. For what it’s worth I found it to be super easy to remove the SolarBridge inverters and purchase Enphase IQ7XS microinverters from EBay and glue them onto the backside of the Sunpower panels. My PVS5 doesn’t seem to know or want to talk to the IQ7XS but they work well and the panels operate with no apparent issues. I can still get rudimentary monitoring of panels from Tesla Powerwall gateway which has its own CTs for monitoring current from the panels. Thank you to everyone who’s digging into the Sunpower issues. Very much appreciated.
Do you have info on the connection settings for RS-485 (Modbus? Baud rate).