Skip to content

Instantly share code, notes, and snippets.

@koleson
Last active October 31, 2024 02:40
Show Gist options
  • Save koleson/5c719620039e0282976a8263c068e85c to your computer and use it in GitHub Desktop.
Save koleson/5c719620039e0282976a8263c068e85c to your computer and use it in GitHub Desktop.
PVS6 Notes

Originally this was just PVS6_NOTES.

Lately, a lot more information - and need for it - has come along, so I split the files up.

Past and Future Experiments

Ideas for profitable investigations and an index of experimental results. Free to a good home; will post results as they come in.

Past Experiments

MQTT reroute - MQTT_Reroute.md

Very fruitful - finally understood how the mySunPower app's SunVault mode changes are communicated to the PVS6, which performs the system control described by the modes, and found a potential method of changing the command and data acquisition server from one in the cloud to one that is locally controlled.

Future Experiments

Emulated EDP MQTT Server

  • Tools
    • Rooted PVS6
    • Probably Swift and an MQTT library
    • Protobuf schemas extracted from communicator
  • Risks
    • Minimal - don't have to try anything risky to see if it fundamentally "works" with a spare
  • Rewards: Very High
    • Reuse nearly all PVS6 code, avoiding the need to re-implement C++ binaries with lots of logic
    • Completely obviate the remaining reason to need to connect to SunPower at all (battery control)
    • Protobuf shows there's a wide swath of control and telemetry available over the EDP channel
  • Questions to be answered
    • Can communicator connect to an MQTT server with no mTLS?
      • would make setup and debugging much simpler
    • What data does communicator send to its EDP server? On what interval?
      • would probably be much better for self-monitoring than dl_cgi API
        • push vs pull
        • async vs sync
        • per-device intervals vs fixed interval for all devices
          • many device characteristics in DeviceList are nearly static
          • some device characteristics in DeviceList take an extremely long time to ingest and should be queried rarely
            • and certainly shouldn't be queried alongside all the other data that can be gained in a short time
    • What EDPCommands can we document as working?
      • there are a ton of them, and probably most of them should be left alone.

Automatic U-Boot from USB

  • Tools
    • The U-Boot console password
    • U-Boot docs
    • manual commands issued to make PVS6 boot from USB
  • Risks: Minimal
    • can usefully test this on a spare
  • Rewards: High
    • can create backups of SunPower firmware for use when eMMC dies of write exhaustion
    • can avert write exhaustion more or less completely by booting from USB normally
    • easy to make modifications on USB stick and test them in a system without modifying "pristine" eMMC firmware
    • USB sticks are cheap
    • USB sticks don't require SMD hot air desoldering / reballing / soldering when they fail
  • Questions to be answered
    • Is there some way to completely disable writing to / reading from the eMMC from operating once the system is booted from USB?
      • On systems with eMMC write exhaustion, it seems like attempting to access the eMMC can hang for long enough to make the watchdog reboot the PVS.
      • We also want to be sure that we aren't modifying the eMMC inadvertently when booted from USB
    • What is the text of the script to boot from USB automatically? What are the relevant variables?
    • Does this count as text I can release? Is it source code?
      • If so, could I describe it as modifications to the existing scripts?
    • Can we fall back to eMMC if the USB boot device is not present?
    • Is it worth investigating network boot?
      • RAM isn't terribly constrained on the PVS6
      • Since we don't need on-device redundancy, we need much less space than we would on the eMMC
      • Could probably disable a lot of services and delete a lot of files once we know SunPower won't be updating them!
        • and/or we don't want their updates

Persistent storage locations

  • Questions to be answered
    • Where is persistent state stored for various components between boots?
      • ESS Energy Arbitrage mode tariff data
      • ESS mode
      • MI serial numbers (used by mime for polling)
      • system commissioning state

MIDC/MID/MIO RS-485 packet capture tool

  • Tools
    • RS-485 to USB converter (e.g. Waveshare)
    • two-jack RJ-45/8P8C breakout board with screw terminals (example)
      • allows using the RS-485 to USB converter while system operates
    • Modbus Sniffer
      • outputs pcap files for analysis by Wireshark
      • when I have tried, reports a whole lot of checksum errors
      • see below for notes on serial parameters
    • Wireshark
  • Risks
    • Minimal - passive capture can be done in an operating system or on spares
  • Rewards: High
    • If it's standard Modbus or a close variant, libraries exist to ease proxying this information to capture it and to modify it
    • Key functioning components if PVS6 ceases to function
    • Key functioning components if mySunPower app ceases to function
    • Replacing Hub+ with other hardware would run into the thousands of dollars
    • Control and monitoring replicable without rooted PVS6
    • Inexpensive, easily obtained and assembled parts
  • Questions to be answered
    • Does the system use Modbus or does the protocol just look similar?
      • Why do some messages have CRC checksums that are correct but others look incorrect?
        • Maybe the "incorrect" ones just use a different polynomial or starting value.
      • It has to be awfully similar or actually Modbus
        • 0x03 (read registers) and 0x10 (write registers) commands are in the right place and format other than incorrect CRCs
        • 0x03 (read registers) responses correctly list PVDR L1-N and L2-N line voltages
    • What are the unknown modbus device IDs?
      • Known: 10, 220, 221, 230
      • Unknown but observed: (several, generally sending commands/requests)
    • What are the RS-485 serial parameters?
      • Looks like 9600 baud 8N1 based on logic analyzer

Understand the MIO hardware

  • Tools
    • Spare MIO
  • Risks
    • Minimal on a spare
    • Medium on a shutdown production part
  • Rewards: Medium
    • I have a feeling there's another PSoC5 in there
  • Questions to be answered
    • Is there a PSoC5?
      • If so, probably not much we can do if the MIO fails in the future

Boardshots

  • Tools
    • Camera
    • Dropcloth
    • Softbox
  • Risks
    • Basically none with spares
  • Rewards: Low
    • More eyeballs on PCBs to find interesting things I don't know about
    • Not having to tear things down from production systems more than once
    • I've already documented as many of the ICs as possible in text / with links
    • Could maybe do an iFixIt style teardown labelling
      • seems a bit of a time sink since there is a paucity of spares
      • and even if spares flooded the market, unless you're rooting them, good luck commissioning them

JTAG All The Things

  • Tools
    • ARM Cortex debugger
  • Risks
    • minimal if performed on spare PVS6
    • low if performed on production PVS6
      • existing connectors not all keyed
      • soldering involved if connector not present
  • Rewards: Low
    • Can already obtain most component firmwares in some form or another
    • Difficult to modify binary firmware in any conceivably useful way
    • Might be fused/locked out anyways
  • Questions to be answered
    • Which devices have a populated JTAG header?
    • Which devices have JTAG available via PCB?
    • Which of those devices offers the best potential ROI from tinkering?

PVS6 control simulator for MIDC/MID/MIO RS-485

  • Tools
    • RS-485
    • Some ARM dev board (Raspberry Pi, etc)
    • Modbus (or similar) protocol as determined from earlier experiment (not yet performed)
  • Risks
    • HIGH
      • safety-critical systems
      • no specification
      • probably should never be done by anyone anywhere
  • Rewards: High
    • PVS6 could fail (and probably will due to eMMC lifetime) and this would still be viable
      • eMMC write exhaustion mitigation: boot from USB stick rather than eMMC; ensure eMMC U-Boot script is set to boot to USB by default
      • There are perhaps other parts that could fail in the PVS6, and even with spares, recommissioning wouldn't be fun.
        • Maybe just copy the eMMC raw data directly from one PVS6 to another? Depending on hardware changes, it might "just work"?
    • MIDC/MID/MIO could probably run on the same firmware indefinitely - no logging/writes
    • Hub+ expensive to replace as noted above
  • Questions to be answered
    • Are there hardware failsafes that are sufficient to even attempt this?
      • List them
    • What bad things could happen?
      • Probably could only find out by testing spares outside of production. Good news - I have some.
        • Could also emulate spares of less common parts if Modbus (or similar) is understood
    • Is there any way to share such software?
      • Given the above bullet points, personal liability could be too high.
      • I also personally cannot share open-source software at the moment.
    • Technical questions
      • Picking a language
        • Would such control software be performance-critical?
        • Is it likely to be built and maintained by more than one person?
        • Distribution and future compatibility concerns
          • Python will outlast Homo sapiens but that doesn't mean source code compatibility with newer version
          • C from 50 years ago probably still compiles but also has a lot of sharp edges
      • Picking hardware
        • Finding something sufficiently powerful should not be a problem
          • More power might just be a liability, honestly
        • Finding something that will still be around in a decade is harder
        • Finding something that documentation would still apply to in that timeframe is harder still

MQTT reroute

Summary of results to date + possible future experiments

It turns out that it is possible to control many system aspects via MQTT.

Summary: This approach allows controlling the battery discharge mode (i.e. Cost Savings/ENERGY_ARBITRAGE or Self-Supply/SELF_CONSUMPTION or reserve) and reserved SoC (e.g. save 40% for backup, save 4% for backup) - among many other system parameters - using a simple protobuf-emitting MQTT server. (To my knowledge, there was no known way to do this without SunPower's cloud services until now.) It also allows a more efficient capture of system monitoring data, as the PVS emits data asynchronously and on its own timetable rather than in response to a synchronous long-running HTTP request that has numerous asynchronous internal datasources as with dl_cgi.

This approach also allows us to use nearly all of the code in the PVS6 without having to re-implement it, which is an absolutely ENORMOUS win given that much of it is in difficult-to-modify C++ binaries, we don't have much of a spec, there's a ton of logic in the binaries and libraries, and testing any re-implementation on a live system is certainly unwise and indeed probably dangerous and potentially costly.

In production, the PVS6 connects to AWS IoT Core, a managed MQTT service. The MQTT connection is authenticated using mTLS: mutual TLS, where both the client and the server demonstrate holding their private key and the other side's certificate. These certificates are held in the app0 mount. The PVS6's serial number is used as the MQTT client ID. MQTT 3.1.1 protocol is used.

The communicator process that initiates the connection to MQTT is configurable through command line flags, including a flag to set the MQTT host, flags to set the various input and output topics, and flags to set the mTLS certificate and private key files.

Over the MQTT channel, protobuf-encoded messages are exchanged. The protobuf schema files are embedded in communicator and can be extracted with protodump.

It seems that the WAN port must be up for communicator to attempt communicating with the MQTT server even if the MQTT server is reachable via the LAN interface.

Doing something like this Meshtastic Home Assistant component, decoding and encoding protobufs and using the internal HA MQTT server, might make this method accessible to a broader swath of people.

It also might be possible, if the SunPower servers stay up for a while yet, to build something like Juicepassproxy or IOXY that is a dedicated transparent MQTT proxy with poisoned ARP and DNS: the PVS's DNS servers are hard-coded in various places to Google DNS (8.8.8.8 / 8.8.4.4), so it's a bit more of a brutalist approach. But that approach won't work once the SunPower AWS IoT servers stop accepting connections / renewing certificates for clients based on server-side rules (i.e. only renewing for Lease/PPA customers), which could be any day now.

The biggest constraint for both of those approaches will continue to be that the PVS needs to be "rooted" in order to change communicator's connection parameters or obtain the certificate/key material from app0.

Original experimental plan/results

  • Tools
    • AWS IoT Embedded C SDK ended up just using various MQTT clients + this Amazon doc
    • Rooted PVS6 (to extract certs/keys from app0)
    • communicator binary
    • protodump
  • Risks
    • minimal if tested on spare PVS6
    • medium if tested on production PVS6
  • Rewards: Medium
    • Reward increases if mySunPower app stops working
    • It's not clear what data is being sent and received over this mechanism because it is over TLS
      • It's possible this is the missing link in how to get the PVS6 to command the SunVault easily
      • It's equally possible this is a big nothingburger and it's just sending telemetry already available from the dl_cgi APIs
      • MQTT would at least probably give us data at a good pace across devices with minimal CPU load / wait times / blocking I/O /etc
  • Questions to be answered
    • Is it possible to change the existing AWS IoT MQTT SDK client to connect to a non-TLS MQTT server?
      • It looks like it is! It looks like mTLS is required.
    • Is it possible to change the existing AWS IoT MQTT SDK client to connect to a TLS MQTT server of our choosing with our own provided credentials?
      • We can definitely set the hostname easily
      • We can set the port with some difficulty
      • mTLS certs/key material appears to be required (at least for the systemd service to work)
    • Is it possible to learn what data traverses this link without changing the MQTT target server?
      • Maybe not, but since we can change the target, who cares!
  • Investigation results
    • MQTT client ID should be PVS6 serial number
    • client/[serial]/command: the Server-to-PVS topic.
      • Change of battery mode and reserved SoC (among other things) is definitely communicated over this topic
      • data is protobuf-encoded EDPCommand as described below
    • client/[serial]/time communicates the current epoch time in milliseconds
      • unclear why this is needed when NTP exists but here we are
    • other topics appear to be PVS-to-Server communication topics
      • client/[serial]/command/update - probably the reply channel for commands received over MQTT (see --tpcmdrsp below)
      • client/[serial]/data - send more complete data to EDP?
      • client/[serial]/event - send anomalies to EDP?
      • client/[serial]/toi - "Technical Operational Intelligence" messages
      • client/[serial]/live - send live data that probably gets piped right back out to the mySunPower app
        • probably just the useful subset described in the protobuf EDPLiveData below
    • communicator can be configured!
      • /home/communicator/bin/communicator --capath [root.crt] --cpath [device_cert.crt] --pkpath [private_key.pem] -u [hostname, sometimes called endpoint] --tpclcmd=[command_topic --tpcmdrsp=[command_response_topic] --tpdata=[data_topic] --tplive=[live data?] --tpevent=[event_topic] --tptoi=[technical_topic] --tpsynctime=[time_sync_topic]

        • changing the configuration with command line params seems to persist the updated configuration to /app0/app_data/communicator/communicator.cfg in protobuf format (PBCommunicator.proto schema)
        • it looks like you can't change the port from the command line
          • 8883 is the default port
          • not much reason to change this because mTLS auth is required
        • it looks like mTLS auth is required
          • with no certs in the expected app0 folder, communicator systemd service exits quickly with message
          • it looks like the certs have to be valid according to OpenSSL in the formatting sense - not checked against system trust store
          • following generic instructions to set up a CA with OpenSSL, make a client cert signing request and fulfill the request seems to work
      • can use SEP output to just write output to a file (? untested)

      • contains numerous interesting protocol buffers (not all of them are listed here)

        • DcmEnableState
          • bool ctrl_enabl_fl - Control enable flag, one assumes
          • bool edp_log_enabl_fl - EDP logging enable flag, one assumes
        • DcmEssParameters - type, energy, power, round-trip efficiency
        • DcmCtrlParameters
          • float dchrg_min_th_soc_val - minimum discharge SoC as a percent
          • float fcst_tgt_soc_val - ? (forecast target SoC value not used elsewhere in my experience)
          • uint32 fcst_ntvl_qty - ?
          • .EDPMessaging.DcmEssChargeConstraint chrg_cnstr_enum - charge constraint as described below
          • bool sgip_compl_enabl_fl - SGIP (California Public Utilities Commission Self-Generation Incentive Program, rebate payments for DER) flag of some kind
            • SGIP has stipulations on how an ESS can be charged to be eligible for rebates.
            • In short, you can only charge from solar.
          • uint32 ess_yrly_dchrg_cyc_qty - ESS yearly discharge cycle quantity; perhaps a target for systems that can charge from sources other than solar?
          • string tz_str - time zone string, probably in tz database / Olson database format
          • .EDPMessaging.DcmCtrlType ctrl_type_enum - exposed as Self-Consumption/Cost Savings/Reserve in the mySunPower app - detailed description below
          • ess_exp_enabl_fl - ESS (grid) Export Enable flag
            • Some installations (HECO, NEM 1/2 Zero-Export retrofit) are disallowed from and/or don't economically benefit from exporting energy to the grid
          • uint64 vld_fld_msk - "valid fields mask"?
        • BackupParameters - allows setting a DcmEssOpMode to replace the current mode until an end time
        • EDPHeader - msg_crt_eps timestamp (?), serial number, model number, command token (string)
        • VppEvent - describes a Virtual Power Plant Event
          • string vpp_evt_uid - event ID for scheduling, cancelling, etc
          • uint64 vpp_evt_start - epoch timestamp of when the VPP event starts
          • uint32 vpp_evt_duration - length of the VPP event in seconds
          • float vpp_evt_reservesoc - percentage of battery that should be reserved for customer use
          • bool vpp_evt_export_constrained - maybe in the overall program setup there's a max discharge power dsecription; if flag is true, only export that much power rather than running inverter full-bore
          • .EDPMessaging.DcmEssOpMode vpp_evt_exitmode - when the VPP event ends, change to this discharge mode
        • EDPCommand - a command from EDP (the cloud)
          • SendBufferedDataFromTimeInterval - probably what it says on the tin; max keys, start/end timestamps
          • GetDataLoggerParameters/SetDataLoggerParameters - get/set several parameters at once
          • UpgradeFirmware - from included upd_spec_url
          • RebootSystem
          • ResetEnergyStorageSystem
          • ConnectToSSHServer with string srvr_id and uint32 fwd_port_no, then CloseSSHConnection
          • TriggerDeviceDiscovery and then asynchronously GetDeviceDiscovery
          • ChangeDeviceSerialNumber
          • SetEphemeralAccessCode - not sure what this does at the moment!
          • WiFiSetCredentials - unclear if this is for broadcasting or connecting to an AP, but ssid/bssid/pwd are there
          • WiFiGetStatus/WiFiScan/WiFiForgetNetwork
          • ZigBeeSEPGet/ZigBeeSEPSet
          • METStationCalibrationGet/METStationCalibrationSet
          • SiteOperationalGet/SiteOperationalSet
          • NetworkInterfaceReportGet
          • CTScalingFactorGet/CTScalingFactorSet
          • AdhocCommand - array of AdhocData
          • DeviceDisassociate
          • CellPrimarySet
          • DcmControllerParametersGet/DCMControllerParametersSet (member types described elsewhere)
            • .EDPMessaging.DcmEnableState enable_state
            • .EDPMessaging.DcmEssParameters ess_params
            • .EDPMessaging.DcmMetering metering
            • .EDPMessaging.DcmCtrlParameters ctrl_params\
          • PcsParametersGet/PcsParametersSet - import enabled / export enabled
          • VppEventCreate/VppEventDelete/VppEventsGet
          • DcmBillingPeriodGet/DcmBillingPeriodSet
          • DcmControlModeGet/DcmControlModeSet
            • .EDPMessaging.DcmEssOpMode ess_op_mode_enum - control mode to set (as one can do in mySunPower app)
            • float dmnd_tgt_kw - demand (?) target kilowatts, maybe only applies to some modes?
            • float p_rto - usually p is power, rto might be "ready to operate", but unsure
            • float tgt_soc_val - lowest SoC to discharge to (as one can set in mySunPower app)
            • .EDPMessaging.BackupParameters bkup_params
          • DcmTariffDataSet - sets up utility tariff information on a calendar for Cost Savings mode to interpret
          • LiveDataControl
          • GridProfileGet/GridProfileSet
          • MeterSubtypeSet
          • AccessCodeScope - WAN, cellular, inverter settings, LEDs, MIB (?) settings, unrestricted, watchdog, zigbee
        • EDPLiveData
          • LiveData - matches up nicely to what we see in mySunPower!
            • float production_pwr_kw - shown as power from panels on home illustration / tab
            • float net_consumption_pwr_kw - shown as power to/from grid on home illustration / tab
            • float battery_pwr_kw - shown as power to/from battery where applicable on home illustration / tab
            • float battery_soc_val - shown as remaining battery % on SunVault tab
            • .EDPMessaging.MIDState mid_state_enum - used to pick descriptive text (?) on home illustration / tab
            • float site_load_pwr_kw - shown as "Home Usage" power on home illustration / tab
            • uint32 backup_time_remaining_min - shown as hours/minutes remaining on SunVault tab during grid-down

SunPower PVS6

The PVS6 (aka PV Supervisor 6) is a monitoring and control system for SunPower solar systems installed from 2019.

On August 6, 2024, SunPower filed for Chapter 11 bankruptcy.

As of September 28, 2024, the mySunPower app that allows consumers to view monitoring data and control SunVault systems is still online. Much of the monitoring data provided by the app is available on the local dl_cgi API, although problems with flash wear and partitions filling up have been reported with long-term use of that API.

The SunVault (aka Equinox storage) is a battery time-shifting / backup / virtual-power-plant system announced in September 2019 and discontinued in early 2024.

Unfortunately, as of September 28, 2024, I'm unaware of any local API on the PVS6 to control the SunVault's settings. The SunVault system seems to receive discharge mode, discharge minimum SoC, consumption power, production power, and grid power information from the PVS6, conveying that information over RS-485 and a protocol that looks like Modbus. Numerous parts of the system are SunPower proprietary, so spares and repairs are expected to be a challenge.

These notes document the hardware and software of these systems as found through reverse engineering of spares.

Want to contribute? Ideas here.

Control Board

aka Part 45-403-000690

  • CPU: Freescale/NXP i.MX6 UltraLite
    • Single Core ARMv7 (32-bit)
    • armhf instruction set
    • Automotive Grade - -40°C to 120°C operating temperature
    • 528MHz part underclocked to 396MHz
  • eMMC: Kingston EMMC04G-W627-M06U (32Gbit/4GB)
    • NAND MLC, BGA153 footprint, eMMC5.1 (HS400), i-Temp industrial operating temperature range - -40°C to +85°C
    • later models use Micron eMMC, perhaps to avoid write-exhaustion mentioned below
  • RAM: 2x Nanya1917 (???) - 1GiB total
  • "PSOC4": Cypress/Infineon PSoC4 Microcontroller
  • CAN Controller: NXP TJA1042T - SO8 footprint (Datasheet)
  • CAN Ports (originally 2, later 1): Molex Nano-Fit 4-pin (2x) connectors (counterpart: Molex Part 1053081204)
  • PMIC: NXP MC32PF3000A7 - 48QFN footprint (Datasheet)
  • DC Power: Accepts 12V from ESS, which means a reboot requires removing Hub+ cover.
  • Onboard Ethernet/Switch: Micrel KSZ8863RLI
    • 3-port 10/100 Managed Ethernet Switch (2 10/100BASETX, one RMII)
    • LQFP48 footprint
  • Ethernet Transformers for circuit isolation are built into the back of the RJ-45 ports ("magjacks")
  • LTE Modem: Quectel BG96 (Sales Sheet)
    • earlier revisions use BG95
  • RF (Bluetooth): Silicon Labs EFR32MG1 "Mighty Gecko" Multi-Protocol SOC
    • supports Zigbee (!), Thread (!), and Bluetooth LE
  • Wi-Fi: AP6398S module with Broadcom chipset
    • MIMO Wi-Fi 2.4GHz + 5GHz - 802.11a/b/g/n/ac(!), has Bluetooth 5.0 capability.
    • Connects via SDIO
  • I2C/SMBus I/O expander: Texas Instruments TCA6424 (Datasheet)
  • Unpopulated LCD 50-pin ribbon connector - at boot, video for 480x272 LCD display claims to be enabled.
    • The NXP i.MX6 UltraLite EVK suggests a 4.3" TianMa model.
    • Unfortunately, adding a connector does not seem to drive the LCD in the same way as the i.MX6 EVK.
    • a "VLCD_5V" block is present and populated on the PVS6 board.
  • Unpopulated MicroSD card slot lands
    • seems to be mutually exclusive to Wi-Fi, as I think there are only two SDIO ports available on the CPU
  • Live main UART near jumper block
    • lone pin is ground, pinout from left: Ground, empty, Rx, Tx, 3v3.
    • 115,200 baud console
  • JTAG: A full-size 20-pin JTAG port, as found on the i.MX6 EVK, has a footprint but is not populated with a connector.
    • Two small 10-pin JTAG ports are located on the PCB front near the eMMC and PSoC4 and the PCB back near the Wi-Fi module

Power Monitoring Board aka LIB (line input board) aka Sidecar (Part 820-00210-A01-00)

  • Communicates to main Control Board via 50-mil 10-pin ribbon (similar to / commonly used for small JTAG)
  • AC/DC Power Supply: Mean Well IRM-30-12 (Board-mount, 240VAC in max, 12V/2.5A out max)
  • Unknown Enphase IC - Maybe Powerline Communication controller?
    • LQFP128 footprint - back of PCB - Enphase 480-00036 - PM9B21.00A-1 - 151
    • looks like an unpopulated JTAG header land on front of PCB (J12)
  • Unknown Enphase IC - Maybe Powerline Communication helper IC? -TSSOP14 footprint - back of PCB - Enphase 480-00035-01 - KAW741.1R-1 - 1803
  • Adjacent to Enphase ICs: Winbond W25Q128JVSQ 128Mbit/16MByte - SOIC-8 footprint
  • Microcontroller: CY8C5668AXI - Cypress/Infineon PSoC5LP
    • ARM Cortex-M3 (32-bit), 80MHz
    • unpopulated JTAG header land (J8)
  • Motorola/ON Semiconductor MC74ACT541 Octal Buffer/Line Driver with 3-State Outputs
  • Skyworks Si88243EC quad digital isolator + DC-DC converter (up to 100Mbps)
  • Up to 6 current transformer measurement channels (3 populated - Production, Consumption 1, Consumption 2; additional channels require population of twin STM32)
  • Current Transformers: Rowgowski coils, 12V DC (https://en.wikipedia.org/wiki/Rogowski_coil)
  • Input power fused (glass type)
  • STM32 F-RAM: Cypress/Infineon FM25L16B
  • STM32: STM32F373C8T6 - 32-bit ARM Coretex-M4, 64KB Flash, LQFP48 footprint
  • 5 OpAmps (+ 1 twin unpopulated footprint) - On Semiconductor LMV358, Micro8 footprint (Datasheet: https://www.onsemi.com/download/data-sheet/pdf/lmv321-d.pdf)
  • Example UART comms to PVS6 (hex)
    • aa 01 00 00 00 00 08 a4 04 02 00 f8 c9 31 61 00 00 00 00 5a 54 32 32 32 39 38 35 30 30 30 XX XX XX 43 XX XX XX XX 00 00 00 00 00 00 00 00 00 10 00 00 00 90 01 00 00 b8 0b 00 00 00 00 40 9c 00 00
      • Masked portion is PVS6 serial number
    • 4e 00 00 00 00 00 08 a4
    • aa 01 00 00 0d 26 6a 00 00 00 00 00 00 00 00 00 00 00 00
      • Seen most frequently, often repeated several times

SunVault-specific hardware

see SunVault Notes

Bonus: PVS5x Notes

  • CPU: Atheros AR9350
    • 600MHz MIPS 74Kc
  • Powerline Communication: Atheros AR7420 (HomePlug AV) - PHY 10/100Mbps; peak 600Mbps
  • Multi-UART: FTDI FT4232HL
  • RAM: 128MB DDR2 (2x Winbond W9751G6KB-25 512Mbit)
  • eMMC: Spansion S34ML02G200TFI00 256MB (2Gbit) TSOP48
  • WiFi: Atheros AR9350 802.11n
  • Zigbee: space for two Zigbee daughtercards (?) - JTAGs labelled for both
  • Zigbee daughtercards: Silicon Labs EM357
  • Cellular: Mini-PCI (3G)
  • UART: same form factor as that of the PVS6, but doesn't seem to be live
  • RS-485: has 2-wire and 4-wire RS-485 ports
  • Firmware: probably based on OpenWrt as open-sourced by SunPower here

Software (point-in-time)

  • Very Abridged Version History

    • Version 2020.9, Build 8127
      • loaded on a PVS6 integrated in a Hub+, so must have had some kind of SunVault support by this version
    • Version 2023.3, Build 61410
      • last version available via the dl_cgi CheckFW command
      • any later updates were "pushed" over EDP MQTT / communicator
    • Version 2024.6, Build 61707
      • latest version I have seen
  • U-Boot 2017.03

    • does not respond to "any key" to interrupt boot as you might be used to with other U-Boots
    • requires passphrase during timeout to go to U-Boot console (input is not visible)
    • U-boot passphrase is stored in cleartext in the first boot sector of the eMMC
      • near the string "Autoboot in %d seconds" you'll find bootdelaykey and its value
    • multiple builds seen in the wild
      • very old Hub+ PVS6 6.02 had U-Boot 2017.03-staging-prod-adama+g277c30979b (Dec 02 2019 - 08:32:49 +0000)
        • upgrading PVS6 firmware to latest did not (immediately, at least) upgrade bootloader
      • newer PVS6 6.02 without ethernet ports had U-Boot 2017.03-imx_v2017.03_4.9.88_2.0.0_ga+geaeb74722e (Sep 09 2021 - 07:42:26 +0000)
    • possible to boot directly via USB with custom boot script on newer build
      • Older build (Dec 02 2019 - 08:32:49 +0000) shows USB hubs but does not see attached USB Mass Storage devices
        • Also reported by various folks using U-Boot for NXP devices going back to 2014, 2015
        • Would be nice to try updating U-Boot but I don't have a recovery plan if it doesn't work
      • U-Boot is GPL v2+, so in theory someone who has a device with the newer U-Boot build could ask for the as-built-and-distributed source code for the newer build.
      • to set the LED to green, start USB subsystem, load a Linux zImage and flat device tree from a sole connected USB Mass Storage:
        set_pvs_led g
        usb start
        ext4load usb 0:0 ${loadaddr} ${image}
        ext4load usb 0:0 ${fdt_addr} ${fdt_file}
        setenv bootargs console=ttymxc0,115200 root=/dev/sda rootdelay=2
        bootz ${loadaddr} - ${fdt_addr}
        
        • rootdelay=2 is necessary to allow the USB drive to "start up" even if it's a flash drive
        • we're mostly reusing built-in env vars here, but you can use ext4ls to snoop around / make sure the USB drive is working
    • possible to boot via TFTP
      • but not super useful without a local rootfs!
      • can boot kernel via TFTP and use USB rootfs
      • to set the LED to green, get an IP via DHCP, and load a Linux zImage and flat device tree via TFTP (that you populated with the relevant files from a firmware rootfs)
        set_pvs_led g
        dhcp [tftp_server_ip]:${image}
        dhcp ${fdt_addr} [tftp_server_ip]:${fdt_file}
        setenv bootargs console=ttymxc0,115200 root=/dev/sda rootdelay=5
        bootz ${loadaddr} - ${fdt_addr}
        
        • rootdelay=5 is a bit longer than the delay needed if booting directly from USB
        • this only loads the kernel and device tree via TFTP, which can get you bootstrapped if your PVS is on the old (Dec 02 2019) version of U-Boot
  • Linux Kernel 4.9.88 - built with Yocto

  • Firmware updates are verified by the firmware update script against a public key stored on the device.

    • The update script allows verification to be skipped by passing a flag.
    • Automatic updates will NOT BE PERFORMED if flashwear levels are too high
  • Uses jfrog for a custom apt repo for several tools / holdbacks / custom versions (SSL Cert auth)

  • Uses D-Bus for a lot of IPC / messaging / etc

    • org.sunpower.ESMM seems to be the ESS management D-Bus.
    • used for
      • dcm
        • DCM (Discharge Management? Discharge Control Manager?)
        • VPP (Virtual Power Plant - utility-commanded discharges during peak hours)
        • PCS (Power Control System - see elsewhere in document)
      • esmm
        • reset ESS
      • superman
        • site operational get/set
        • grid profile get/set
  • Uses custom binaries and libraries, apparently written in C++

  • Also uses some Python 2.7, including site-packages

  • /app0 appears to be related to an AWS IoT layer using MQTT - perhaps the main EDP reporting/control complex?

    • probably uses the AWS IoT Device Embedded C, maybe as open-sourced here
    • uses per-device cert-based auth
    • certs need to be rotated regularly for continued access
      • 3-month validity
  • Can be controlled by crafted SMS messages to the cellular modem

    • check system uptime
    • toggle logging
    • reboot/reset/hibernate
    • restore
    • switch boot partition
    • detailed status
    • cellular up/down/reset/set primary / always-on
    • get/set WAN/LAN/wi-fi mode/SSID
    • establish arbitrary SSH tunnel
    • execute firmware update from URL
    • set configuration
    • force cert client to run to refresh against given endpoint
  • SSH is open, root has a password set but SSH will only accept public key auth

  • data_logger user has most binaries for controlling and updating other hardware subsystems

  • mime user has binaries for PLC communication with microinverters

  • envoy_manager - relatively new, has configurations for envoy, powerwall, gateway, inverter, livedata, meter, gateway, so it seems to be ESS-related

  • varserver / Variable Server - somehow related to envoy_manager

  • ieee2030.5

  • superman

  • installcheckd

    • seems to be part of the ESS commissioning process
    • communicates over websocket using WebSocket++
    • port 9002?
  • telemetry-ws

    • communicates over websocket using WebSocket++
    • port 9002?
  • Uses two partitions to handle updates - mmcblk1p5 and mmcblk1p6 (~256MB)

    • update image is installed to non-active partition, persistent data is copied, migration scripts are run, and system reboots to formerly non-active partition.
    • see partition map secton below for more info
  • Uses systemd for numerous periodic tasks.

    • Curiously, this includes rebooting the PVS every day at 07:00 UTC
      • unless the PVS6 is attached to a SunVault that is offline or fewer than 24 hours have passed since commissioning.
  • ttymxc2 is connected to a PSoC 4 "watchdog"

    • COBS communication
    • 9600 baud
    • PSoC 4 can also be addressed over GPIO with bitbang comms
    • hilariously, capable of accepting morse code commands via a button (maybe not populated on factory PVS6, maybe a jumper?)
      • short press: enable WiFi AP
      • long press: soft reboot
      • "eternal" long press: detected but not used
      • F: factory reset
      • N: network restart
      • P: toggle PTO mode (Permission to Operate?)
      • S: WPS enable
      • LEDs should reflect accepted inputs
  • ttymxc4 seems to be the UART connected over the gray 10-pin ribbon cable to the "sidecar" power monitoring board

    • I think the other end is the Cypress CY8C5668AXI PSoC5 chip, but haven't verified.
  • ttymxc5 seems to be RS-485 2-pin

  • Can do firmware updates from a USB stick. Looks for a lua script at sunpower/PVS6/fwup.lua upon USB storage device insertion.

    • It appears the upgrade script signature is validated against a public key by fwup. Passing -k NONE disables this check.
  • A swagger spec for the dl_cgi API is available at /cgi-bin/swagger.json on the PVS.

    • An example fwup.lua URL is present in that swagger.json.
      • Note that the rootfs archives are sometimes in xz compression format in spite of the .tgz extension which usually means gzip compression.
        • Use a hex editor to validate magic number at start of file
          • xz: FD 37 7A 58 5A 00
          • gzip: 1F 8B
  • SSH tunneling seems to go through pvsfleet.us.sunpower.com

  • Live data seems to go through edp-api.edp.sunpower.com

  • Certs for MQTT/AWS IoT seem to be retrieved via https://register.edp.sunpower.com:3001/ (unsure of how auth'd - SSL client cert?)

  • Google DNS (8.8.8.8 / 8.8.4.4) seems to be hard-coded as DNS for the device, seemingly ignoring DHCP-provided DNS, making DNS redirection more difficult.

  • ESS Control

    • based on GraphQL API (as used here) we know a few strings used for control
    • ENERGY_ARBITRAGE is one; matching binaries might involved in ESS control
      • or maybe they just need status info and there's another representation for control
      • Binary file /home/data_logger/bin/data_logger matches
      • Binary file /home/data_logger/bin/vdm matches
      • Binary file /home/data_logger/lib/libesmm.so matches
      • Binary file /home/data_logger/lib/libdcm.so matches
      • Binary file /home/communicator/bin/communicator matches
      • dcm seems to be another string to investigate
        • dcmTariffDataSet suggests dcm uses tariff data to schedule discharge like ENERGY_ARBITRAGE does
      • pcs is not

Partition Map

partition map dump from U-Boot:

=> mmc part

Partition Map for MMC device 1  --   Partition Type: DOS

Part    Start Sector    Num Sectors     UUID            Type
  1     2048            20480           XXXXXXXX-01     83
  2     22528           131072          XXXXXXXX-02     83
  3     153600          131072          XXXXXXXX-03     83
  4     284672          7350272         XXXXXXXX-04     05 Extd
  5     286720          524288          XXXXXXXX-05     83
  6     813056          524288          XXXXXXXX-06     83
  7     1339392         1048576         XXXXXXXX-07     83
  8     2390016         5244928         XXXXXXXX-08     83

Descriptions:

  • 1 / mmcblk1p1 - 10MB: (tbd)
  • 2 / mmcblk1p2 - 64MB: (tbd)
  • 3 / mmcblk1p3 - 64MB: (tbd)
  • 4 / mmcblk1p4 - 1K: (tbd)
  • 5 / mmcblk1p5 - 256MB: Linux Root FS 1
  • 6 / mmcblk1p6 - 256MB: Linux Root FS 2
  • 7 / mmcblk1p7 - 512MB: (tbd)
  • 8 / mmcblk1p8 - 2.4GB: app0 mount - app_data and secrets

Unresponsive PVS6 with rotating "rainbow" LED

  • yellow/green/red, random order/timing, often several seconds between changes
  • is a symptom of a write-exhausted eMMC.
    • background: unlike magnetic storage, eMMC/flash memory in general has a finite lifetime of writes.
      • writes happen as "block writes", so even just changing one character results in a block write if flushed to eMMC.
      • a common mistake is to flush writes too frequently and exhaust - run out of - writes.
      • after this exhaustion, the eMMC will not write reliably or will prevent writes permanently.
  • the PVS has a flash wear check that occurs regularly
    • if the flash wear exceeds some value, it will report high flash wear to SunPower servers
    • it will also stop automatically updating the firmware
  • the UART will be alive and printing, and ethernet lights might be on, but the system won't boot.
  • the eMMC is still able to be read to pull U-Boot and whatever environment variables were set before it ran out of writes
    • but those environment variables cannot be saved to the eMMC
    • it seems the eMMC spec includes extra resiliency for the boot areas
    • attempting to read the eMMC outside the boot areas of a write-exhausted eMMC has resulted in timeouts for me, oddly.
    • this implies that if one wishes to be able to use the PVS6 past the useful life of the eMMC, steps must be taken to change the bootscripts/env vars to point to other media before the writes are exhausted, for example, copying the eMMC to a USB flash drive and booting from that with something like:
      usb start
      ext4load usb 0:0 ${loadaddr} ${image}
      ext4load usb 0:0 ${fdt_addr} ${fdt_file}
      setenv bootargs console=ttymxc0,115200 root=/dev/sda rootdelay=2
      bootz ${loadaddr} - ${fdt_addr}
      
      • note also that there are hard-coded references in the firmware to mmcblk1 that would need to be updated for a new root/boot device.

Enphase Retrofit

As of September 2024, Enphase offered customers with Enphase microinverters the opportunity to purchase an Enphase monitoring system.

  • The system is based on the IQ Combiner 4 which works with IQ6/7 grid-tied systems and IQ8 grid-forming systems.
  • The IQ Combiner includes an integrated IQ Gateway.
  • Unclear how/if this will work with the SunVault
    • As of September 28, 2024, Enphase's response is "it doesn't" which makes sense
    • Less clear is "why it doesn't work"
      • Does SunPower PCS use MI data directly for SunVault discharge regulation?
      • Does Enphase's drop-in change the keys on the MIs such that a PVS would be "out of the loop" on MI data?
      • it seems like production/consumption CTs on the PVS are necessary and sufficient to run PCS, but maybe I've missed something

SunPower SunVault

SunVault hardware overview

Virtual Devices

  • Created by PVS6 at commission-time
    • HS-DCM-ctrl
      • dcm controller
    • HS-DCM-meter
      • power meter

Hub+

  • MIDC (Microgrid Interconnect Device Controller)
  • PVDR (Photovoltaic Disconnect Relay)
  • grid isolation contactor
  • bus bars + breakers
    • backup + grid
    • grid-only

Battery Cabinet

  • Inverter/Charger: Schneider Electric XW Pro - connected to Gateway via Xanbus
    • Modbus device 10
  • Includes Schneider Conext Gateway - connected to PVS6 via Ethernet
    • SD card contains something; slot blocked with a sticker
    • Capable of running an SSH server, which is activated and deactivated by the PVS6 as needed
  • Other (later?) installs use Schneider InsightFacility in place of Conext Gateway
  • Batteries
    • 2020 version: up to 2x POWERAMP Komodo 48V Nominal battery modules (LiFePO4)
      • Includes 200A breaker to disconnect battery DC from inverter
    • 2022 version: up to 3x battery modules
      • DOES NOT include 200A breaker to disconnect battery DC from inverter
    • BMS has some kind of "commissioned" flag
  • SunPower MIO inside SunVault cabinet

SunVault internal cable colors

  • Blue in Hub+ (UTP, 8P8C/RJ-45): PVS - MIDC J10 Comm (probably Modbus over RS-485)
  • Gray (UTP, 8P8C/RJ-45): MIO J1 - MIDC J14 Comm (probably Modbus over RS-485)
  • Orange (UTP, 8P8C/RJ-45): MIO J2 - BMS Comm (CAN)
  • Green (UTP, 8P8C/RJ-45 to bare leads: MIO J4 - Conext Comm CAN1 breakout (CAN)
  • Blue in SunVault (UTP, 8P8C/RJ-45): MIO J7 - Sunvault LED display (unknown protocol)

RS-485 over UTP/STP info

  • PVS6 uses same color and label for RS-485 2-wire as PVS5, which connected to SMA inverters over RS-485 2-wire
  • in SMA-US22 systems, wiring for T-568B cables is solid blue (B) to pin 2 (D+), stripe blue (b) to pin 7 (D-), solid brown (BR) to ground (GND)
  • T-568B and T-568B cables change orange and green pairs terminations; blue and brown pairs terminate the same in both
  • Thus, in T-568A/T-568B alike, in RJ-45 / 8P8C termination:
    • RS-485 A / D- = pin 5 (blue stripe)
    • RS-485 B / D+ = pin 4 (blue solid)
    • RS-485 GND = pin 8
  • Appears to be 9600 baud, no parity, 1 stop bit (9600 8N1)
    • Some unit numbers and CRCs are as expected; others seem random.

MIDC (Microgrid Interconnect Device Controller - also known as ADS Controller, ATS Controller, Automatic Transfer Switch) (Part 820-00550)

  • Revision History
    • Revision 5 seems to be first shipped to customers
      • Unclear if this can be used without CAN connection
    • Revision 6.3 seems to be most recent
      • Can definitely be used without CAN connection
  • Modbus over RS-485 Device 220 (0xDC)
  • Early versions used CAN in addition to RS-485
  • controls main contactor, aux contactor (PVDR)
  • thus, senses Grid + Island voltages and controls contactors / PVDR accordingly
  • has jumpstart 12VDC power port
    • only accessible by removing the dead front (six screws)
      • seems like a nice thing to have access to!
    • connector is Molex Micro-Fit 3.0 connector
      • pigtail: part 2147511022 (300mm)
      • square pin = GND
      • keyed pin = +12V DC
  • can also control a secondary or remote PVDR.
  • Cypress PSoC 4 part (TQFP-64 package; part number obscured)
  • two Atmel MCUs for reading current transformers
  • one green CT Phoenix terminal 4-pin (J4) is hidden by plastic casing on my system
  • Analog Devices ADUM4401 quad-channel digital isolator (4kV RMS)
  • uses same Mean Well IRM-30-12 board-mount 12-volt AC/DC power supply as PVS6
  • CAN input (not used on my system)
  • RJ-45 jacks for RS-485 from PVS6 / to MIO in ESS
  • Contactor drive/connectors for PVDR and main grid contactor
  • J2: PCS (Power Control System) Required Consumption CTS
    • (UL 1741 CRD, noted here)
  • Dry contact relay for genset (J3)
  • DC-DC converter for running 12VDC systems directly from LiFePO4 battery voltage when needed
    • TPS23751PWP (IEEE 802.3at PoE interface with flyback DC-DC controller)
  • receives optional RPO (Remote Power Off) switch input; relays that input to XW Pro inverter
  • has a port for generator control (NC/NO/COM contacts available)
  • communicates with PVS6 via RS-485

MIO (Main I/O)

  • Modbus over RS-485 Device 221 (0xDD)
  • has some kind of firmware onboard

PVDR (Photovoltaic Disconnect Relay)

  • Modbus over RS-485 Device 230 (0xE6)
  • has some kind of firmware onboard

Schneider Conext Gateway

  • communicates to PVS6 via Ethernet - Modbus over TCP (cleartext)
  • communicates with XW Pro via Xanbus
  • runs QNX
    • designed in Canada, so was there ever really any other choice?
  • an SD card is installed
    • the slot is covered with a sticker
    • the PVS6 checks the SD card's contents on occasion
  • can run an SSH server but it is disabled most of the time
    • the PVS6 can enable SSH through some mechanism I haven't yet uncovered

Interfaces

  • CAN: PVS6 to MIDC (obsoleted in later versions)
  • RS-485: PVS6 to MIDC (blue cable with RJ-45/8P8C ends), MIDC to PVDR, PVDR to MIO - SunSpec bus?
  • Multi-battery systems: additional XW Pro inverter/chargers for added AC output, additional batteries for added energy storage. Only one Gateway is needed for any number of inverters and batteries; each inverter needs an MIO.
  • MIO: CAN to Schneider Gateway, CAN to battery, Aux temperature input, XW-RPO input, "Comm from Hub+", comm to next MIO, ESS enclosure fan AC power in/out, LED board output, 48VDC to XW

Useful Tools

A lot of this is applicable to reverse engineering in general.

Spares

A lot of stuff that would be terrifying to do to / with production hardware is much less scary when you're operating on a spare.

eBay can randomly be your best friend.

If they're inexpensive enough, even broken spares can be a worthwhile investment. In some cases, they can even be more useful than working spares: when something fails, other parts can exhibit useful new behavior.

I spent an incredibly long time trying to get root access to my spare PVS6, but without the U-Boot console password, it seemed hopeless. A used (RMA'd, as it turned out) PVS6 that I acquired cheaply turned out to be a godsend: the eMMC was write-exhausted and therefore U-Boot could not read the firmware image, so the boot script failed and dropped me to the U-Boot console where I could poke around and figure out how to boot from USB as I had been trying to do for years.

After I was able to boot from USB, plans that had been coming together for ages moved forward quickly.

Hardware

Logic Analyzer

For a long time I misunderstood the purpose and operation of a logic analyzer.

An oscilloscope captures and displays analog waveform data, maybe over one channel, maybe over a few.

A logic analyzer captures and displays (often on a computer) digital waveform data, usually over several channels. But hey, you can also analyze a single pin. It samples and captures digital pins at a given sample rate.

Since inter-IC communication is digital data (really, voltage high and low), we can use a logic analyzer to capture the data for later analysis even if we don't know the underlying protocol or data rate.

Analysis is part brain power and context - like what protocols might be in use and what data we might find in the capture - and part computation - like applying a UART analyzer to some captured timing data to see ASCII characters being communicated.

Even an inexpensive ($12, for example) logic analyzer makes quick work of understanding what data is being sent over a known communications pin or trace. Most of the communications going on in these systems is very low-speed and is often serial, but of unknown baud and parity and so on. A logic analyzer and some timing data makes discovery of the correct settings fast and easy.

In fact, I would recommend using an inexpensive LA for these parts: though it hasn't happened to me so far, it would be a darn shame to be poking around with something expensive and have it go "zap."

Get started with something you control fully - an Arduino project with an i2c sensor and firmware you control and understand - before poking around PCBs you aren't familiar with.

UART breakout

Once you've logic analyzed your way into the correct settings, a UART will let you capture data more easily and with less fuss. Soldering one to the relevant pins will save you a lot of sanity - buying small CP2102/CP2014/similar breakouts from Aliexpress or similar in bulk makes committing one to a circuit that much less of a hassle.

JST-XH connectors can be re-pinned and used with 2.54mm spaced pin headers like the main UART on the PVS6 if you don't want to solder.

Often, you can be happy with just connecting GND and breakout RX / target TX. Don't connect breakout TX / target RX unless you plan to use it. VCC can also usually be skipped in my experience and it's one fewer thing to go wrong.

Evaluation Kits

Getting a sense for how things work when they aren't integrated with the system is much easier than when they are programmed with some source you don't have compiled into firmware binaries you probably also don't have. Spending money here can add up quickly, but when you're working with systems that are difficult to repair and replace, they're a much safer place to test hypotheses.

Sometimes you can also get binary firmware blobs running on such eval kits. The eval kits usually have PCBs and headers that are easier to navigate than shipping products.

Software

Your favorite Linux in a CLI virtual machine

I use Debian inside UTM. Reading and dumping ext4 and so on is that much easier. I ssh in from my regular OS terminal.

Salae Logic 2

Fantastic capture performance and analysis tools.

Executable binary in in, embedded protobuf files out. You can usually spot files containing them in a hex editor or in a grep for .google.protobuf.

Documentation

Datasheets

Any time you see an IC, look for its datasheet. Bullet points about what interfaces and peripherals are available can give you a sense of what ICs might be talking to each other before tracing them on a PCB. In 2024, microcontrollers are way overprovisioned in terms of peripherals, so don't read too far into them being utilized simply because they are present.

Most fun pinouts to look for, because they're easily transparently eavesdropped in-situ and in-use:

  • UARTs - RS-232 and RS-485
  • i2c
  • SPI
  • ethernet

Others that might be interesting:

  • JTAG

User Manuals

Not always the richest, but can give you a more complete understanding of what the thing can do.

Installation Manuals

Usually a step up from User Manuals, often includes more detail on what talks to what.

FCC certification documentation

If it has a radio in it, you can probably find a PDF of photos of the internals of the system and what frequencies the radio operates on by searching for "FCC (model number)".

Less fun than tearing the thing down yourself, but if you don't have a spare, it's a great place to start.

@benhadad
Copy link

benhadad commented Sep 8, 2024

Do you have info on the connection settings for RS-485 (Modbus? Baud rate).

@koleson
Copy link
Author

koleson commented Sep 8, 2024

My guess would be 9600. It could also be 19200 or 115200. It should be Modbus.

@CraftyCanine
Copy link

Hi, thank you for the work you're doing here. Very helpful in a community who desperately needs it right now. Do you have any idea where the admin password might be stored for the Schneider InsightFacility? I have been unable to get SunPower support to help me with my battery for obvious reasons, so I've taken to poking around myself. The user and guest accounts are default passwords, but the admin account is not.

@koleson
Copy link
Author

koleson commented Sep 23, 2024

@CraftyCanine I haven't been able to dig as much on the Conext Gateway / Insight Facility because I don't have a spare and they're pretty pricey even in the used market.

If it's any consolation, I don't think there's much of help in there: the PVS6 runs the PCS (Power Control System) that tells the XW inverter how much to charge/discharge the battery and if the PVS6 isn't commissioned for the SunVault I wouldn't expect much to happen.

SunPower won't be much help themselves, but a SunPower-authorized third-party installer should still be able to commission the system for you.

@benhadad
Copy link

I have several MI, the Parthenon solarbridge 250w and 320w. I haven't found any PCB/firmware info for these units. I also have several PSV5x units that I have taken apart and started looking at the firmware for.

@benhadad
Copy link

I don't know of any installer that can or would work on any of the older systems without the newer enphase MI. Atleast I haven't found anyone in the SF bay area that is capable of updating or activating a PSV5 with Solarbridge inverters.

@koleson
Copy link
Author

koleson commented Sep 26, 2024

@benhadad I don't think anyone can commission a PVS5/PVS5x anymore. There's a step in the commissioning process where the PVS5 checks for a firmware update, but it always fails. I think SunPower simply took down the endpoint when they transitioned all installers to ProConnect.

The PVS6 originally used the same mechanism but now uses a different one where ProConnect uploads the firmware to the PVS6 directly rather than the PVS6 downloading it from the web.

@CraftyCanine
Copy link

I think the ProConnect app still calls various local APIs to control the PVS, SunPower just removed the UI for those calls and created an app instead, so they can control who is able to log in to the app. For example, there is the POST /cgi-bin/dl_cgi/commission/config command. Unfortunately, I only have my active PVS so I can't test it, but all of the non-destructive commands I found in the swagger spec work fine, so I would guess that one does too.

@koleson
Copy link
Author

koleson commented Sep 26, 2024

I have a spare PVS5 so I can take a look tonight.

@koleson
Copy link
Author

koleson commented Sep 26, 2024

I don't think all the relevant endpoints are present on the PVS5x. I could be wrong, but the firmware upload endpoint returns the same error as endpoints that for sure aren't there. (PVS5x Firmware 2018.1)

@koleson
Copy link
Author

koleson commented Sep 29, 2024

@CraftyCanine I'm starting to think you're right actually. The Pro Connect release notes talk about PVS5 support. I might be looking in the wrong place.

@dmbryson
Copy link

Does anyone know which component actually runs the BMS? I got into the user mode on the InsightLocal, and from what I could see, it only shows 4 devices in my case, the two XWs and two SunPower v1.2 BMS. However, as best as I can tell, the CAN is daisy chained through all four battery packs.

Side note, the fault codes on the system showed that it temporarily lost comms with its “SunSpec Controller” when I was interposing the ethernet switch. Seems like perhaps the PVS6 is controlling the XW with modbus over ethernet.

@koleson
Copy link
Author

koleson commented Sep 29, 2024

As far as I could tell, the Conext/Insight pulls BMS data over CAN and exposes it to the PVS over - as you might have seen - Modbus TCP.

I worked on a tool to gather Modbus TCP data and throw it in InfluxDB once upon a time: https://github.com/koleson/spessmod

From that I found out the PVS6 basically sets the max charge and max discharge power of the XW in accordance with current consumption/production and discharge mode (cost savings/arbitrage vs self-consumption).

There are some register maps in that repo that might be helpful. If I remember right, the BMS is exposed over Modbus TCP.

@benhadad
Copy link

benhadad commented Oct 1, 2024

@benhadad I don't think anyone can commission a PVS5/PVS5x anymore. There's a step in the commissioning process where the PVS5 checks for a firmware update, but it always fails. I think SunPower simply took down the endpoint when they transitioned all installers to ProConnect.

The PVS6 originally used the same mechanism but now uses a different one where ProConnect uploads the firmware to the PVS6 directly rather than the PVS6 downloading it from the web.

It fairly easy to get past the firmware check, however when you get to the very end to do the final commissioning it requires a pro account login for the final approval.

@benhadad
Copy link

benhadad commented Oct 1, 2024

I have disassembled a 320W MI solarbridge. It was a painful process as it potted with what appears to be a silicon foam, which I so far haven’t found a better way than to remove than mechanical removal. I have tired all the legal removal chemistry, but have been told that real MEK or Tolune are really the best tools to use (which are both not available in CA). If someone needs an MI to work with I have a couple of damaged ones I could part with.

@heyhewmike
Copy link

Does anyone know which component actually runs the BMS? I got into the user mode on the InsightLocal, and from what I could see, it only shows 4 devices in my case, the two XWs and two SunPower v1.2 BMS. However, as best as I can tell, the CAN is daisy chained through all four battery packs.

Side note, the fault codes on the system showed that it temporarily lost comms with its “SunSpec Controller” when I was interposing the ethernet switch. Seems like perhaps the PVS6 is controlling the XW with modbus over ethernet.

From what I have seen with my digging it appears the BMS is built into the LiFePo4 cells themself and is communicated to through the "HOST" port. Then the Main BMS Controls all others based on which Comm Port the data comes in on, either HOST or COM IN. If you look at the Cells you will see that which ever one has the "HOST" cable will have COM OUT running to the second Cell COM IN. The HOST cable goes to the SunPower Multi i/O device.

I have reached out to Schneider Electric Support/Sales who has told me that we can reset the InsightLocal Facility UI login credentials without changing any of the configurations. I have also reached out asking their support about the BMS that I see in the InsightLocal Facility UI. The InsightLocal Facility is connected to the XW Pro via XanBus

There appears to be 2 forms of communication between the SunVault and the PVS. The Canbus and Ethernet. Both are connected to the InsightLocal Facility and the XW Pro is connected to the InsightLocal Facility and PVS via the SunPower Modbus Multi i/O.

I have begun searching into the FCC ID on the Multi i/O

@dmbryson
Copy link

dmbryson commented Oct 8, 2024

Indeed, I would be very interested in resetting the admin password.

My thought is to perhaps swap out the Hub+/PVS6 guts with those from a Schneider BCS and an Enphase Gateway (for panel monitoring). Keeping the BMS configuration would be ideal for now, though.

I do wonder how autonomous the functions of the MIO boards are. Do they need the PVS6 to work at all, or will things like the status display, CAN interfacing, and fan control work headless?

@heyhewmike
Copy link

heyhewmike commented Oct 8, 2024

Indeed, I would be very interested in resetting the admin password.

My thought is to perhaps swap out the Hub+/PVS6 guts with those from a Schneider BCS and an Enphase Gateway (for panel monitoring). Keeping the BMS configuration would be ideal for now, though.

I do wonder how autonomous the functions of the MIO boards are. Do they need the PVS6 to work at all, or will things like the status display, CAN interfacing, and fan control work headless?

I have 2 updates from Schneider Electric and Enphase.

  1. Schneider Electric reports that they are aware that there are core components of theirs inside the SunVault but that SunPower has made some custom modifications. They are working with SP to become more familiar with it so that they can better support.
  2. Enphase has told me that if I rewire my house so that the Schneider Electric XW Pro is between my House/Solar and the Grid they can install the IQ Gateway but their warranty will only be limited to Product Warranty of the IQ Gateway and nothing else.

@dmbryson
Copy link

dmbryson commented Oct 9, 2024

Yeah, I pretty much assume that if I take any action to gain more control over my system that it would limit the warranty. That said, SunVault is conspicuously absent from their warranty resources and FAQ pages.

@heyhewmike
Copy link

Yeah, I pretty much assume that if I take any action to gain more control over my system that it would limit the warranty. That said, SunVault is conspicuously absent from their warranty resources and FAQ pages.

Yes absent from the FAQ and warranty resources page but I am quoting the SE Solar Tech Support:
Schneider Electric can work with you with limited support and unit replacement that are under warranty with proof of purchase.

@benhadad
Copy link

Here are a few pictures of inside the solarbridge (sunpower labeled) MI-320
PXL_20240925_070005709
PXL_20240925_102536733

PXL_20240925_102540532

@benhadad
Copy link

Some SMD parts
Si823x, PSoC 5 CY8C56LP, 74ACT541

@dmbryson
Copy link

FWIW, the Sunvault PCS is definitely completely independent from module communications. I have some additional QCells panels with IQ8 MIs monitored with an IQ Gateway that are connected into my generation PAN. The PVS shows total system production and consumption from the CTs and manages charge/discharge of the XWs accordingly.

Related, I have not had any problems with PLC noise or crosstalk between the IQ8’s and the IQ7’s built into the SPWR panels. The two generations use different frequencies and protocols for their PLC. Unfortunately this does mean I would still need a second gateway to migrate panel level monitoring from the PVS over to my Enphase system.

@dmbryson
Copy link

That said, I don’t think it would be a good idea to try to rehome the panels from the PVS to an IQ gateway with both active on the same PLC domain. Without separating them with a PLC filter, I could imaging chaos ensuing with the commissioned PVS searching for and trying to set the grid profile of them at the same time as the gateway.

@koleson
Copy link
Author

koleson commented Oct 20, 2024

@dmbryson yeah, that's what i can't square about what Enphase reps have said (the IQ Gateway can run alongside the PVS6 and monitor the same panels without removing the PVS6, but it currently cannot be installed alongside a SunVault) and how PCS seems to work (it doesn't seem to rely on MI data directly - just uses the CTs).

@Joefant
Copy link

Joefant commented Oct 25, 2024

@benhadad, over the past months I’ve noticed that several of my panels with SolarBridge MI320 microinverters would get into an oscillating run/shutdown mode especially when there was high production from the panel DC outputs. For what it’s worth I found it to be super easy to remove the SolarBridge inverters and purchase Enphase IQ7XS microinverters from EBay and glue them onto the backside of the Sunpower panels. My PVS5 doesn’t seem to know or want to talk to the IQ7XS but they work well and the panels operate with no apparent issues. I can still get rudimentary monitoring of panels from Tesla Powerwall gateway which has its own CTs for monitoring current from the panels. Thank you to everyone who’s digging into the Sunpower issues. Very much appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment