You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The technical information provided in this document/repository was obtained through lawful reverse engineering of software running on a device legally owned by the author.
This documentation is published solely for the purpose of enabling interoperability with independently developed monitoring tools, systems, or components. No original source code or proprietary content is reproduced, and no attempt has been made to circumvent any access control mechanisms protected under the DMCA.
This work is intended to support compatibility and repair under the exemptions granted by:
• 17 U.S. Code § 1201(f) (reverse engineering for interoperability),
• The Library of Congress DMCA Rulemakings, and
• Applicable fair use principles under 17 U.S. Code § 107.
Redistribution or use of this information for other purposes (e.g., unauthorized duplication, bypassing protections, or commercial exploitation beyond interoperability) is not authorized by the author.
communicator is a C++ binary in the SunPower PVS firmware that:
queries various devices for state
via data_logger, over DBus
via varserver?
aggregates that state
in a local SQLite database
publishes that data to AWS IoT via MQTT
or any other MQTT server by custom configuration
receives and interprets incoming commands from AWS IoT via MQTT
looking closer
we can view the current running configuration and status with:
communicator -s
it's a good idea to copy and paste the output of this command to a safe place for future reference.
you can interact with a running instance of communicator via a named pipe, which communicator often refers to as FIFO.
each line echoed into the fifo executes a command.
to increase the verbosity of communicator, send a loglevel set to level 6 (maximum):
echo loglevel=6 > /var/run/communicator.fifo
you'll be able to see the details of the protobufs being sent and received by communicator live by following the unit journal:
journalctl -fu communicator
local push data setup with MQTT
avoid the myriad problems that come with polling dl_cgi by allowing the PVS to push data to you at its own pace.
lower CPU usage (~40-60% load, vs 90-120% load when polling)
exceeding 100% CPU time means processes sometimes hang and stutter, resulting in weird system behavior
exceeding 100% CPU time especially bad on systems with SunVault
more things to poll
more things to heartbeat
some weird errors can happen when polling
lower eMMC flashwear
no extra writes per-datapoint to eMMC vs normal mySunPower monitoring in pre-critical flashwear
(see also eMMC write exhaustion doc)
optional bonus: higher frequency data updates
with the EDPLiveData (see Local EDP Server doc) commands/publishes, get data every 2 seconds
grid (net) power
solar production power
house load/consumption power
battery charge/discharge power for SunVault systems
MID (backup/grid) state
current battery SoC
current battery backup time (if the grid went down now)
batteries not included - i cannot share source code for this
polls devices at sensible rates
some devices and their properties almost never change
some devices and their properties change constantly
dl_cgi's DeviceList command is stuck getting everything, every time.
polling in communicator is - to an approximation - set up to reflect this
one-to-many data distribution in real-time
caching proxies can allow one-to-many polling without additional CPU load, but data can get relatively stale
many polling clients = more CPU load
MQTT is designed for one-to-many real-time dissemination
different clients can subscribe to and publish to individual topics per their interests
optional bonus: can decide what comes from / goes to SunPower servers with an MQTT proxy tool
control over what firmware updates are allowed
batteries not included - i cannot share source code for this
optional bonus: can control SunVault locally with appropriate tool
no internet connection necessary to set SunVault discharge mode or reserve SoC
batteries not included - i cannot share source for this
outline
to date, it appears you need to set up mutual TLS
set up a CA (certificate authority)
set up a certificate for the MQTT server
set up a client certificate for communicator in the same CA
configure communicator
setting up the certificate authority
lots of good tools for this, pick your favorite:
openssl commands
Smallstep (step ca)
Vault
i don't think Let's Encrypt is well-suited for this type of setup, but others assure me it's possible
setting up the MQTT server for mTLS
(details to come)
optional: basic ACLs for shared MQTT servers
(details to come)
setting up certificates and key material
scp the client certificate, client private key, and CA root cert to the PVS
a good place to store them is near the existing AWS secrets
they're in /app0/secrets/mqtt
ideally, don't overwrite those
if you overwrite them by accident, they can be recovered, don't panic
/app0/secrets/mqtt_local is the example i'll use here
changing the MQTT target host and auth
gather the required info
hostname
CA root cert filename (absolute path)
client cert filename (absolute path)
client private key filename (absolute path)
pre-build the command to set the relevant parameters in a text editor
long strings and long flag names make it a bit of a mess to try and edit this in the shell
copy and paste ftw
optional: setting the topics
(more to come)
other tidbits
the configuration, in protobuf format, lives in /app0/app_data/communicator/communicator.cfg.
what good is a protobuf-formatted config without a protobuf schema? glad you asked:
you can find the .proto schema files inside the communicator binary using protodump.
it's easier to do this by scping /home/communicator/bin/communicator to another machine and dumping the schemas there.
NAND flash memory, used for persistent storage in the PVS6, stores data in "pages" of memory. Each page has a finite "lifetime" of writes. For the sake of discussion, let's say it's 3000 writes - each page can be written 3000 times before it becomes unreliable.
A common problem in embedded devices is that small changes to a file will result in the entire page containing that file to be written. A specific version of that common problem is when log files are updated frequently (every few seconds) and are flushed to flash just as frequently. In this case, it does not take very long at all to accumulate 3000 writes.
Flash devices have "wear-levelling" algorithms that move data to new physical pages on some basis to avoid wearing out a particular page entirely. This can greatly extend the life of the flash device, but fundamentally, writing small amounts of data very frequently will wear out the flash device in a matter of months rather than a matter of years or a few decades.
To avoid this problem, embedded systems can use a "RAMdisk" to store frequently-changed files. The benefit: you're not hitting a flash write every time the file changes. You can just "flush" the RAMdisk to flash on some infrequent basis. The drawback: data in RAM is lost if power is lost unexpectedly. That can be a pretty big drawback when trying to troubleshoot a device like the PVS where power loss is often part of the troubleshooting process - there's no way for a consumer or technician to initiate a safe shutdown on the PVS6.
It seems like SunPower had a problem with flashwear due to excessive writes on the PVS6 firmware until at least March 2024.
It also seems like the most recent version of the PVS6 firmware has implemented the RAMdisk method of coping with the problem: most logs for services running on the device are now stored on a tmpfs (volatile/log). It is not clear exactly when this was implemented.
To extend the life of PVS6 devices facing this problem, SunPower also implemented a check for flashwear and disallows automatic firmware updates for devices above 80% of lifetime writes estimated.
The eMMC is soldered to the PVS6 main board, so replacing it is difficult. The part is only $5-6 when ordered in quantity, but it is hard to acquire just one or just a few.
USB flash drives have similar write-lifetime problems, but have the benefit of being inexpensive, easily obtained, and easily replaced.
U-Boot from USB Flash Drive
Some PVS6 units ship with U-Boot capable of booting directly from a USB flash drive. I want to figure out how to make the flash drive the default boot device by modifying the U-Boot script of a PVS. This must be done before the writes are fully exhausted and eMMC "write-locks" itself: the U-Boot script cannot be modified after that happens. From there, USB flash drives can be copied to new flash drives regularly and indefinitely.
U-Boot from Network
The PVS6 units that ship with an older U-Boot that is not capable of booting directly from a USB flash drive can boot from the network via TFTP. A zImage of the kernel can be placed on a TFTP server accessible by the PVS6 and advertised via DHCP. A local USB drive can then be used as the rootfs. It would be nice to develop this ability as well since many PVS6 units don't have the new U-Boot and I haven't tried updating it yet.
System running but not uploading to mySunPower problem
I need to try and remember the symptoms here but basically the way I remember this is the system would produce power and operate the SunVault on the last known setting, but would no longer upload production data for the mySunPower app to display.
Unresponsive PVS6 with rotating "rainbow" LED problem
yellow/green/red, random order/timing, often several seconds between changes
is a symptom of a write-exhausted eMMC.
background: unlike magnetic storage, eMMC/flash memory in general has a finite lifetime of writes.
writes happen as "block writes", so even just changing one character results in a block write if flushed to eMMC.
a common mistake is to flush writes too frequently and exhaust - run out of - writes.
after this exhaustion, the eMMC will not write reliably or will prevent writes permanently.
the PVS has a flash wear check that occurs regularly
if the flash wear exceeds some value, it will report high flash wear to SunPower servers
it will also stop automatically updating the firmware
the UART will be alive and printing, and ethernet lights might be on, but the system won't boot.
the eMMC is still able to be read to pull U-Boot and whatever environment variables were set before it ran out of writes
but those environment variables cannot be saved to the eMMC
it seems the eMMC spec includes extra resiliency for the boot areas
attempting to read the eMMC outside the boot areas of a write-exhausted eMMC has resulted in timeouts for me, oddly.
this implies that if one wishes to be able to use the PVS6 past the useful life of the eMMC,
steps must be taken to change the bootscripts/env vars to point to other media before the writes are exhausted,
for example, copying the eMMC to a USB flash drive and booting from that with something like:
usb start
ext4load usb 0:0 ${loadaddr} ${image}
ext4load usb 0:0 ${fdt_addr} ${fdt_file}
setenv bootargs console=ttymxc0,115200 root=/dev/sda rootdelay=2
bootz ${loadaddr} - ${fdt_addr}
note also that there are hard-coded references in the firmware to mmcblk1 that would need to be updated
for a new root/boot device.
Checking Flash Wear
With pre-SunStrong firmware (or unknown firmware)
From the installer port, one can open a system journal viewer. For example, at:
http://172.27.153.1:19531
From that viewer, one can select which service to view logs for. Pick automatic_upgrade.
If you see something along the lines of
automatic_upgrade[#####] Already disabled
your PVS has probably entered CRITICAL flashwear state (80%+ lifetime writes expended) and will not automatically upgrade its firmware.
With rooted PVS6
mmc extcsd read /dev/mmcblk1 | grep "eMMC Life Time Estimation B"
The output will be in hex. 0x08, for example, means "about 80% of lifetime writes have been consumed."
Any hexadecimal value with letters is above 100% estimated lifetime writes and should be considered ready to fail at any moment.
Very fruitful - finally understood how the mySunPower app's SunVault mode changes are communicated to the PVS6, which performs the system control described by the modes, and found a potential method of changing the command and data acquisition server from one in the cloud to one that is locally controlled.
Many small obstacles following the MQTT reroute made this take a long time. The protobufs being compressed in some cases with LZ4 and not in others made it hard to make progress.
Once that bottle was uncorked, things moved pretty quickly. I was able to decode the once-per-five-minutes message sent by default, and was able to send commands to the PVS to trigger live data output.
Work here is ongoing: I need to try more commands and document what works, but in general things in EDPCommand have worked as expected.
Future Experiments
Automatic U-Boot from USB to avoid eMMC write exhaustion
Tools
The U-Boot console password
U-Boot docs
manual commands issued to make PVS6 boot from USB
Risks: Minimal
can usefully test this on a spare
Rewards: High
can create backups of SunPower firmware for use when eMMC dies of write exhaustion
can avert write exhaustion more or less completely by booting from USB normally
easy to make modifications on USB stick and test them in a system without modifying "pristine" eMMC firmware
USB sticks are cheap
USB sticks don't require SMD hot air desoldering / reballing / soldering when they fail
Questions to be answered
Is there some way to completely disable writing to / reading from the eMMC from operating once the system is booted from USB?
On systems with eMMC write exhaustion, it seems like attempting to access the eMMC can hang for long enough to make the watchdog reboot the PVS.
We also want to be sure that we aren't modifying the eMMC inadvertently when booted from USB
What is the text of the script to boot from USB automatically? What are the relevant variables?
Does this count as text I can release? Is it source code?
If so, could I describe it as modifications to the existing scripts?
Can we fall back to eMMC if the USB boot device is not present?
Is it worth investigating network boot?
RAM isn't terribly constrained on the PVS6
Since we don't need on-device redundancy, we need much less space than we would on the eMMC
Could probably disable a lot of services and delete a lot of files once we know SunPower won't be updating them!
and/or we don't want their updates
Persistent storage locations
Partial Results
NOT PERSISTED: root ssh public keys
distributed with the rootfs
so you need to modify firmware update scripts and repackage the rootfs, or touch the device to reinstall them after every firmware update.
/app0/app_data/edp/log/
An overall config, versioned by date, but the path name doesn't seem promising
dates seem to match up to installer commissioning attempts
/home/data_logger/
config.lua
c*.lua
d*.lua
/home/mime/a*.json
grid profile?
/home/mime/mime.cfg
site ID, lease state, active power percent, device groups
Summary of results to date + possible future experiments
(moved these updates from MQTT_Reroute.md)
UPDATE (23 Dec 2024): Decoding the publishes from the PVS to the data topic has been tricky. It looks like there may be more than one protobuf message per publish:
-i, --interval Set the MQTT send interval for aggregated protobuf messages
UPDATE 2 (23 Dec 2024): The publishes from the PVS to the data topic are in LZ4 frame format. This is indicated in a hex dump of the published message by the leading 04 22 4d 18 which is the Big Endian / network-byte-order form of the LZ4 Frame Format Magic Number (0x184D2204).
So, one assumes the stats from a time interval are aggregated into the less-frequently-sent messages. (They're sent once every 5 minutes.)
Sending the command to provide live data might change this.
So far, the publishes all contain EDPData protocol buffers, which is exactly what one would expect from the topic naming / protobuf naming similarities.
UPDATE 3 (1 Jan 2025): Sending the command to get live data does indeed result in EDPLiveData publishes on the live topic. I haven't experimented to find out how long that effect lasts - I'm guessing it's 15 minutes or something, and one has to send another command to keep it going.
(end of updates moved from MQTT_Reroute.md)
UPDATE 4 (1 Jan 2025):
Some notes about work over the past couple weeks:
A major component of getting this working was issuing TLS certs locally. I used Smallstep and its CLI tool to issue short-lived TLS certs for my Mosquitto server, my proxy client, and the spare PVS.
Then, since they're short-lived, they need to be renewed regularly. Usually, you just install the Smallstep CLI tool on the machine running the service that needs a TLS certificate, but I didn't want to spend the time to make that work on the PVS6. Instead, I use scp to pull the certs from the PVS, run step ca renew on them locally, and upload them if they have hit the time threshold for renewal.
I originally hard-coded the path to the certs within app0 in my maintenance script. I also set up a script to pull down the communicator.cfg file and commit it to a local git repo to track changes to it. I set up a script to use protoc to generate a human-readable version of communicator.cfg - which is a PBCommunicatorConfiguration object and add that to the git repo as well.
I was able to pull the command-line-configured paths for the auth data from that human-readable version and use it to configure my scp pull of that cert material for more flexibility.
If I really wanted to change the connection port of MQTT on the PVS, I could do that by creating a Configuration protobuf - that's all that communicator.cfg is.
I also tested what happens if communicator.cfg is removed and communicator is started. As expected, it creates a new mostly-empty communicator.cfg since most of the values are default.
Weirdly, I think the original communicator.cfg came with values populated for most of the fields - they just happened to match the defaults.
UPDATE 5 (5 Jan 2025):
MQTT Data Details
The EDPLiveData is constrained to what you see on the main mySunPower app Home tab and SunVault tab. It updates every two seconds on the live topic.
Production (aka PV) power (kW)
Net (Grid) power (kW)
Battery charge/discharge power (kW)
Load (home usage) power (kW)
MID state (MID_STATE_CLOSED, etc)
Battery SoC (percent)
Backup time if grid went down now (minutes)
EDPData is more complete, but not every data type is published on every message:
Summary of results to date + possible future experiments
It turns out that it is possible to control many system aspects via MQTT.
Summary: This approach allows controlling the battery discharge mode (i.e. Cost Savings/ENERGY_ARBITRAGE or Self-Supply/SELF_CONSUMPTION or reserve) and reserved SoC (e.g. save 40% for backup, save 4% for backup) - among many other system parameters - using a simple protobuf-emitting MQTT server. (To my knowledge, there was no known way to do this without SunPower's cloud services until now.) It also allows a more efficient capture of system monitoring data, as the PVS emits data asynchronously and on its own timetable rather than in response to a synchronous long-running HTTP request that has numerous asynchronous internal datasources as with dl_cgi.
This approach also allows us to use nearly all of the code in the PVS6 without having to re-implement it, which is an absolutely ENORMOUS win given that much of it is in difficult-to-modify C++ binaries, we don't have much of a spec, there's a ton of logic in the binaries and libraries, and testing any re-implementation on a live system is certainly unwise and indeed probably dangerous and potentially costly.
In production, the PVS6 connects to AWS IoT Core, a managed MQTT service. The MQTT connection is authenticated using mTLS: mutual TLS, where both the client and the server demonstrate holding their private key and the other side's certificate. These certificates are held in the app0 mount. The PVS6's serial number is used as the MQTT client ID. MQTT 3.1.1 protocol is used.
The communicator process that initiates the connection to MQTT is configurable through command line flags, including a flag to set the MQTT host, flags to set the various input and output topics, and flags to set the mTLS certificate and private key files.
Over the MQTT channel, protobuf-encoded messages are exchanged. The protobuf schema files are embedded in communicator and can be extracted with protodump.
It seems that the WAN port must be up for communicator to attempt communicating with the MQTT server even if the MQTT server is reachable via the LAN interface.
Doing something like this Meshtastic Home Assistant component, decoding and encoding protobufs and using the internal HA MQTT server, might make this method accessible to a broader swath of people.
It also might be possible, if the SunPower servers stay up for a while yet, to build something like Juicepassproxy or IOXY that is a dedicated transparent MQTT proxy with poisoned ARP and DNS: the PVS's DNS servers are hard-coded in various places to Google DNS (8.8.8.8 / 8.8.4.4), so it's a bit more of a brutalist approach. But that approach won't work once the SunPower AWS IoT servers stop accepting connections / renewing certificates for clients based on server-side rules (i.e. only renewing for Lease/PPA customers), which could be any day now.
The juicepassproxy approach won't do much good here because the MQTT communication is over TLS and I don't think we have a way to update the system trust store without rooting.
The biggest constraint for both of those approaches will continue to be that the PVS needs to be "rooted" in order to change communicator's connection parameters or obtain the certificate/key material from app0.
(Some updates that were here have been moved to Local_EDP_Server.md.)
Original experimental plan/results
Tools
AWS IoT Embedded C SDK ended up just using various MQTT clients + this Amazon doc
client/[serial]/live - send live data that probably gets piped right back out to the mySunPower app
probably just the useful subset described in the protobuf EDPLiveData below
communicator can be configured!
/home/communicator/bin/communicator--capath [root.crt]--cpath [device_cert.crt]--pkpath [private_key.pem]-u [hostname, sometimes called endpoint]--tpclcmd=[command_topic--tpcmdrsp=[command_response_topic]--tpdata=[data_topic]--tplive=[live data?]--tpevent=[event_topic]--tptoi=[technical_topic]--tpsynctime=[time_sync_topic]
changing the configuration with command line params seems to persist the updated configuration to /app0/app_data/communicator/communicator.cfg in protobuf format (PBCommunicator.proto schema)
it looks like you can't change the port from the command line
8883 is the default port
not much reason to change this because mTLS auth is required
it looks like mTLS auth is required
with no certs in the expected app0 folder, communicator systemd service exits quickly with message
it looks like the certs have to be valid according to OpenSSL in the formatting sense - not checked against system trust store
following generic instructions to set up a CA with OpenSSL, make a client cert signing request and fulfill the request seems to work
can use SEP output to just write output to a file (? untested)
contains numerous interesting protocol buffers (not all of them are listed here)
DcmEnableState
bool ctrl_enabl_fl - Control enable flag, one assumes
bool edp_log_enabl_fl - EDP logging enable flag, one assumes
float dchrg_min_th_soc_val - minimum discharge SoC as a percent
float fcst_tgt_soc_val - ? (forecast target SoC value not used elsewhere in my experience)
uint32 fcst_ntvl_qty - ?
.EDPMessaging.DcmEssChargeConstraint chrg_cnstr_enum - charge constraint as described below
bool sgip_compl_enabl_fl - SGIP (California Public Utilities Commission Self-Generation Incentive Program, rebate payments for DER) flag of some kind
SGIP has stipulations on how an ESS can be charged to be eligible for rebates.
In short, you can only charge from solar.
uint32 ess_yrly_dchrg_cyc_qty - ESS yearly discharge cycle quantity; perhaps a target for systems that can charge from sources other than solar?
string tz_str - time zone string, probably in tz database / Olson database format
.EDPMessaging.DcmCtrlType ctrl_type_enum - exposed as Self-Consumption/Cost Savings/Reserve in the mySunPower app - detailed description below
ess_exp_enabl_fl - ESS (grid) Export Enable flag
Some installations (HECO, NEM 1/2 Zero-Export retrofit) are disallowed from and/or don't economically benefit from exporting energy to the grid
uint64 vld_fld_msk - "valid fields mask"?
BackupParameters - allows setting a DcmEssOpMode to replace the current mode until an end time
EDPHeader - msg_crt_eps timestamp (?), serial number, model number, command token (string)
VppEvent - describes a Virtual Power Plant Event
string vpp_evt_uid - event ID for scheduling, cancelling, etc
uint64 vpp_evt_start - epoch timestamp of when the VPP event starts
uint32 vpp_evt_duration - length of the VPP event in seconds
float vpp_evt_reservesoc - percentage of battery that should be reserved for customer use
bool vpp_evt_export_constrained - maybe in the overall program setup there's a max discharge power dsecription; if flag is true, only export that much power rather than running inverter full-bore
.EDPMessaging.DcmEssOpMode vpp_evt_exitmode - when the VPP event ends, change to this discharge mode
EDPCommand - a command from EDP (the cloud)
SendBufferedDataFromTimeInterval - probably what it says on the tin; max keys, start/end timestamps
GetDataLoggerParameters/SetDataLoggerParameters - get/set several parameters at once
UpgradeFirmware - from included upd_spec_url
RebootSystem
ResetEnergyStorageSystem
ConnectToSSHServer with string srvr_id and uint32 fwd_port_no, then CloseSSHConnection
TriggerDeviceDiscovery and then asynchronously GetDeviceDiscovery
ChangeDeviceSerialNumber
SetEphemeralAccessCode - not sure what this does at the moment!
WiFiSetCredentials - unclear if this is for broadcasting or connecting to an AP, but ssid/bssid/pwd are there
WiFiGetStatus/WiFiScan/WiFiForgetNetwork
ZigBeeSEPGet/ZigBeeSEPSet
METStationCalibrationGet/METStationCalibrationSet
SiteOperationalGet/SiteOperationalSet
NetworkInterfaceReportGet
CTScalingFactorGet/CTScalingFactorSet
AdhocCommand - array of AdhocData
DeviceDisassociate
CellPrimarySet
DcmControllerParametersGet/DCMControllerParametersSet (member types described elsewhere)
The PVS6 (aka PV Supervisor 6) is a monitoring and control system for SunPower solar systems installed from 2019.
On August 6, 2024, SunPower filed for Chapter 11 bankruptcy.
As of January 1, 2025, the mySunPower app that allows consumers to view monitoring data and control SunVault systems is still online. Running that app and those servers has been taken over by SunStrong Management, which I believe bought out SunPower's share in SunStrong Capital and will be providing service to lease customers for the remainder of their leases. Much of the monitoring data provided by the app is available on the local dl_cgi API, although problems with flash wear and partitions filling up have been reported with long-term use of that API. (See eMMC_Write_Exhaustion.md)
The SunVault (aka Equinox storage) is a battery time-shifting / backup / virtual-power-plant system announced in September 2019 and discontinued in early 2024.
Unfortunately, as of September 28, 2024, I'm unaware of any local API on the PVS6 to control the SunVault's settings. While it's quite a challenge, local control of the SunVault is possible through rooting of the PVS6 and creation of a local MQTT server and protobuf-emitting control program. The SunVault system receives discharge mode, discharge minimum SoC, consumption power, production power, and grid power information from the PVS6, conveying that information over RS-485 and a protocol that looks like Modbus. Numerous parts of the system are SunPower proprietary, so spares and repairs are expected to be a challenge.
These notes document the hardware and software of these systems as found through reverse engineering of spares.
Motorola/ON Semiconductor MC74ACT541 Octal Buffer/Line Driver with 3-State Outputs
Skyworks Si88243EC quad digital isolator + DC-DC converter (up to 100Mbps)
Up to 6 current transformer measurement channels (3 populated - Production, Consumption 1, Consumption 2; additional channels require population of twin STM32)
very old Hub+ PVS6 6.02 had U-Boot 2017.03-staging-prod-adama+g277c30979b (Dec 02 2019 - 08:32:49 +0000)
upgrading PVS6 firmware to latest did not (immediately, at least) upgrade bootloader
newer PVS6 6.02 without ethernet ports had U-Boot 2017.03-imx_v2017.03_4.9.88_2.0.0_ga+geaeb74722e (Sep 09 2021 - 07:42:26 +0000)
possible to boot directly via USB with custom boot script on newer build
Older build (Dec 02 2019 - 08:32:49 +0000) shows USB hubs but does not see attached USB Mass Storage devices
Also reported by various folks using U-Boot for NXP devices going back to 2014, 2015
Would be nice to try updating U-Boot but I don't have a recovery plan if it doesn't work
U-Boot is GPL v2+, so in theory someone who has a device with the newer U-Boot build could ask for the as-built-and-distributed source code for the newer build.
to set the LED to green, start USB subsystem, load a Linux zImage and flat device tree from a sole connected USB Mass Storage:
set_pvs_led g
usb start
ext4load usb 0:0 ${loadaddr} ${image}
ext4load usb 0:0 ${fdt_addr} ${fdt_file}
setenv bootargs console=ttymxc0,115200 root=/dev/sda rootdelay=2
bootz ${loadaddr} - ${fdt_addr}
rootdelay=2 is necessary to allow the USB drive to "start up" even if it's a flash drive
we're mostly reusing built-in env vars here, but you can use ext4ls to snoop around / make sure the USB drive is working
possible to boot via TFTP
but not super useful without a local rootfs!
can boot kernel via TFTP and use USB rootfs
to set the LED to green, get an IP via DHCP, and load a Linux zImage and flat device tree via TFTP (that you populated with the relevant files from a firmware rootfs)
seems to use libvarserver, libcurl, libsqlite3, libsystemd, and jsonrpc-lean
Uses two partitions to handle updates - mmcblk1p5 and mmcblk1p6 (~256MB)
update image is installed to non-active partition, persistent data is copied, migration scripts are run, and system reboots to formerly non-active partition.
see partition map secton below for more info
Uses systemd for numerous periodic tasks.
Curiously, this includes rebooting the PVS every day at 07:00 UTC
unless the PVS6 is attached to a SunVault that is offline or fewer than 24 hours have passed since commissioning.
ttymxc2 is connected to a PSoC 4 "watchdog"
COBS communication
9600 baud
PSoC 4 can also be addressed over GPIO with bitbang comms
hilariously, capable of accepting morse code commands via a button (maybe not populated on factory PVS6, maybe a jumper?)
short press: enable WiFi AP
long press: soft reboot
"eternal" long press: detected but not used
F: factory reset
N: network restart
P: toggle PTO mode (Permission to Operate?)
S: WPS enable
LEDs should reflect accepted inputs
ttymxc4 seems to be the UART connected over the gray 10-pin ribbon cable to the "sidecar" power monitoring board
I think the other end is the Cypress CY8C5668AXI PSoC5 chip, but haven't verified.
ttymxc5 seems to be RS-485 2-pin
Can do firmware updates from a USB stick. Looks for a lua script at sunpower/PVS6/fwup.lua upon USB storage device insertion.
It appears the upgrade script signature is validated against a public key by fwup. Passing -k NONE disables this check.
A swagger spec for the dl_cgi API is available at /cgi-bin/swagger.json on the PVS.
An example fwup.lua URL is present in that swagger.json.
Note that the rootfs archives are sometimes in xz compression format in spite of the .tgz extension which usually means gzip compression.
Use a hex editor to validate magic number at start of file
xz: FD 37 7A 58 5A 00
gzip: 1F 8B
SSH tunneling seems to go through pvsfleet.us.sunpower.com
Live data seems to go through edp-api.edp.sunpower.com
Certs for MQTT/AWS IoT seem to be retrieved via https://register.edp.sunpower.com:3001/ (unsure of how auth'd - SSL client cert?)
Google DNS (8.8.8.8 / 8.8.4.4) seems to be hard-coded as DNS for the device, seemingly ignoring DHCP-provided DNS, making DNS redirection more difficult.
ESS Control
based on GraphQL API (as used here) we know a few strings used for control
ENERGY_ARBITRAGE is one; matching binaries might involved in ESS control
or maybe they just need status info and there's another representation for control
Usually, a PVS6 has a dedicated 15 amp breaker. Toggling that breaker reboots the PVS6.
If your system has a SunVault, there's an extra step to reboot the PVS6.
First, turn off the breaker in the Hub+ labeled PVS6 - it's on the left, between the solar breakers in the second section of breakers.
Then, remove the top plastic cover of the Hub+. It's held on with two Torx T25 screws, visible at the top of the area covered by the Hub+ door. Remove the screws, then lift the plastic cover up and then toward you.
Finally, there's a barrel jack that feeds DC power to the PVS6 from the batteries even when the PVS6 breakers are off. It's at the top right corner of the front circuit board. Pull it out, wait a few seconds, then plug it back in.
It's worth turning the PVS breakers on and seeing that everything works before putting the top cover back on.
Once everything is working, replace the top plastic cover and replace the two Torx T25 screws.
DRAFT: While I have obtained root access successfully on a spare PVS6 with this general method, I haven't tested this exact list of actions in this order, and I haven't written out all the details for the steps. That said, the key pieces of information here are "there's a U-Boot password", "It's in plaintext in the first boot sector of the eMMC,", "there's an exposed CPU UART and here's its pinout," and "public key auth can get you root without knowing the password."
The rest is just "here's how to boot from U-Boot" and "here's an inventory of things you can do now that your PVS6 is 'just another Linux box.'"
NOTE: While all actions listed here can be reversed, many of them can also be detected, and it doesn't mean reversal is fast or easy.
Carefully consider whether you're willing to invest the time needed to maintain this setup and understand the entire process before proceeding. If at any time you're in doubt, restore from a backup.
Procedure
In short, we need to boot Linux on the PVS from media we control. From there, we can mount the relevant PVS6 firmware partitions on the eMMC and modify authorized_keys to include our SSH public key. After that, we will no longer need to use the custom boot media and can remove it, booting the standard PVS6 firmware normally.
Obtain the PVS6 U-Boot password
To my knowledge, the U-Boot password is the same for all PVS6 Devices.
The password is case-sensitive
The password I have seen is 5 characters.
Perhaps to be revealed at a later date: 2fb1745b2fc8e02f40ce35ca4e97487a15f4a9eeedff47137e7f8f02de1197124697f7dac74efe3b4d66abd3830bed9dd9b8bcf1d10154ffe95eefbac2dc1c47
As noted in PVS6_NOTES.md:
U-Boot does not respond to "any key" to interrupt boot as you might be used to with other U-Boots.
It requires a passphrase during initial boot wait to go to U-Boot console.
Passphrase input is not visible during input.
The U-boot passphrase is stored in cleartext in the first boot sector of the eMMC.
Near the string "Autoboot in %d seconds" you'll find bootdelaykey and its value.
Plenty of stuff in the NXP image that isn't necessary for what we're trying to do here.
Could build an image without any proprietary inclusions and redistribute it
Could add useful tools to that image
Would be a great thing to share with the community!
Figure out if your PVS supports USB boot
PVS6es with serial numbers ending in F#### always seem able to boot from USB
PVS6es with serial numbers ending in A#### always seem unable to boot from USB
PVS6es with serial numbers ending in W#### I haven't tested yet
Others I also haven't tested yet
To see if your PVS supports USB boot, with any USB mass storage device attached, run usb start
You'll see output similar to
=> usb start
starting USB...
USB0: USB EHCI 1.00
USB1: USB EHCI 1.00
scanning bus 0 for devices... 3 USB Device(s) found
scanning bus 1 for devices... 2 USB Device(s) found
scanning usb for storage devices... 1 Storage Device(s) found
If you see "0 Storage Device(s) found" with a USB storage device attached, you must boot from TFTP.
this might be a problem for PVS6es with serial numbers ending in W#### - unclear if U-Boot will have drivers for USB ethernet dongles
If you see more than 0, you can boot from USB.
Prepare boot media
If you can boot from USB, it's probably easiest to use that
If you downloaded the Linux image from NXP, you can simply dd the image to USB media
You'll need a TFTP server. (Left as an exercise to the reader)
Details to come
Generate an SSH key for use with the PVS6
Details to come
Copy your SSH public key to the bootable TFTP image or USB stick
If you used the NXP image, there should be a FAT32 partition with .dtb (Device Tree) files
That partition is a convenient place to put your SSH public key
You could also put a bash script there to automate copying the SSH public key to both firmware partitions
the relevant partitions will be mounted automatically to a folder
Details to come: what's the folder path?
can just append to [mount]/home/root/.ssh/authorized_keys
Boot from TFTP or USB
Booting from USB
If you are using the NXP Linux image, the EVK Device Tree files are available on partition 1 of the USB stick
The PVS6 Device Tree files are available on the eMMC on the firmware partitions (mmcblk1p5 / mmcblk1p6) in boot
TBD which one is the best choice for this purpose
Details to come
This is the point of no return. Once you perform the following steps, even though the modifications are reversible, it is possible your modifications will be detected and your warranty voided.
Copy your SSH public key to the PVS6 Device eMMC
be sure to copy to authorized_keys in /home/root/.ssh on both mmcblk1p5 and mmcblk1p6
UPDATE 28 March 2025: The 4GB SLC flash drive is indeed product ID 0x0009.
Made in USA! 🇺🇸✨
It adds a new key to PVS.json to cover this
app0_usb now reports flashwear of the USB flash drive
It seems likely that the brand/model USB drive to be used is constrained in order to ensure the relevant drive reliability parameters are supported by smartctl
This would also make it a lot easier to reconfigure communicator, as it would no longer require root access. A user could simply (erm, yeah, Simply™) mount the ext4-formatted USB drive on a Linux machine and change the communicator.cfg (erm, yeah, simply change the protobuf and add the key/cert material).
UPDATE 10 April 2025: This was a false hope, unfortunately: the systemd script that launches communicator specifies MQTT host and so on as command-line flags from a file that isn't on app0. :(
UPDATE 28 March 2025 late: inserting the Delkin 4GB SLC flash drive does indeed run a migration script to create an ext4 partition on the USB stick. The PVS then uses it as an overlayfs with the existing /app0 partition as the lowerdir and the new /mnt/usb/app0_upperdir as the upperdir.
open question: what happens if you remove the USB stick after this happens? is there any procedure in place to sync the upper layer (USB stick) to the lower layer (eMMC partition 8)?
there may be some kind of script to sync these periodically to avoid losing data but also avoid writing too frequently? but if so, haven't found it yet.
root ssh public authorized key has been rotated
identified as rsa-key-20241210 (...I1OdvQ==) (was sunpower/...FkGC0I0x)\
communicator seems to use a different mechanism for configuring what host to use for MQTT connection
looks like it uses varserver for this now rather than just communicator.cfg?
set_dbpath.sh seems to take highest precedent?
2025.4 (Build 61829)
Rollout started April 17, 2025.
New Stuff
U-Boot autoboot timer duration changed to 0
This might prevent rooting a PVS6 even if you know the U-Boot password
found another way that is actually even easier than the old U-Boot dance way
SunVault Modbus: Schneider Conext Gateway / Insight Facility data in Home Assistant
The Schneider Conext Gateway and Insight Facility provide a Modbusover TCP socket. This allows for retrieving detailed SunVault internal operation parameters not available through dl_cgi, as well as retrieving commonly-used parameters at greater speed, frequency, and efficiency than getting them from the PVS6.
We can reuse hardware used for proxying dl_cgi to proxy Modbus with haproxy.
IMPORTANT: SECURITY AND SAFETY
By design and for compatibility purposes, Modbus over TCP has no built-in security mechanisms: no encryption and no authentication. Anyone and anything with access to the relevant TCP port can perform any commands available on the device, including those that configure safety-critical features and those that can shut down the device. For example, this was exploited in the FrostyGoop attack to disable heating to 600 apartment buildings in Ukraine during sub-zero temperatures in January 2024 for several days.
Locate your proxying device as topologically close to the Modbus-providing device as possible
Same switch, ideally
Constrain access to this port deeply
Firewall rules should allow only specific LAN IP addresses to access it
Proxy configuration should only allow specific LAN IP addresses to access it
Consider using a VPN like WireGuard between your consuming device and the proxying device if they are not the same device
Adds encryption and authentication in transit to proxy: even with firewall rules, eavesdropping and MITM and replay could be issues given these are unencrypted raw TCP connections.
Register Maps
There is a LOT of information available to us through this mechanism. I previously documented the registers I've observed in use in a SunVault system in this document in my spessmod project.
There are also detailed register maps available from Schneider for the Conext Gateway/InsightFacility and XW inverter on the InsightFacility product page, under documents, as InsightHome InsightFacility and Gateway Modbus Maps.
Finding your gateway
(More to come, but if you have read this far, you probably already know.)
Setting up haproxy to proxy Modbus over TCP
Each Modbus over TCP port we wish to proxy needs a frontend and a backend.
For example:
frontend ess_modbus
mode tcp
acl allowed_ip src [your Home Assistant server's IP]
bind :9502
tcp-request connection reject if !allowed_ip
default_backend ess_modbus_backend
backend ess_modbus_backend
mode tcp
balance leastconn
server conext [your Conext Gateway/InsightFacility's IP - 172.27.153.x]:502
There are servers on port 502 and port 503 with different characteristics.
Configuring Home Assistant
Because Modbus is a low-level protocol, we need to do a lot of configuration, and the Modbus component has to be configured via YAML.
For example:
modbus:
- name: "ESS Modbus Direct"
type: tcp
host: [your haproxy IP]
port: 9502
sensors:
- name: "External power control command"
slave: 10
address: 40213
data_type: uint16 # should be uint8 but HA not supporting that
- name: "External power control Maximum Discharge Power"
slave: 10
address: 40220
unit_of_measurement: "%"
scale: 0.01
data_type: uint16
- name: "External power control Maximum Charge Power"
slave: 10
address: 40221
unit_of_measurement: "%"
scale: 0.01
data_type: uint16
From there, there are plenty of other possibilities.
You'll probably also want helpers for some of the more opaque items. For example, look at register 40253 - Charger Status. There are some constant values that map to state names; a helper can make these readable for dashboards.
For example, a template helper for 40252 - Inverter Status` could use this template:
The default interval for updating items in the Modbus integration is 15 seconds. Many of these items update infrequently and could reasonably be polled much less often; some of these items update very frequently and could reasonably be polled every few seconds. Remember that this device is critical to the correct functioning of the SunVault: don't pummel it to get slightly more recent data.
How the PVS6 uses Modbus over TCP to control SunVault behavior
I created a tool to decode the Modbus over TCP packets sent from the PVS6 to the Schneider system. A mirroring switch was used to send all data between them out to a monitoring computer where the tool was run.
In short, the PVS6 writes the Maximum Charge Power and Maximum Discharge Power registers based on data from the home load and grid CTs. If in Cost Savings mode, it also uses the time of day to decide when to use battery power to cover home loads.
Maximum Charge Power and Maximum Discharge Power registers are specified as a percentage of maximum inverter output power, so the PVS6 needs to know that to perform its calculation as well.
The PVS6 also reads many other registers regularly, probably for use in dl_cgi and other system health checks.
Future Possibilities: Control?
The MIDC/MID and PVDR are crucial portions of the SunVault off-grid operation behavior. I still don't have enough understanding of how they interact to feel good proceeding with any plot to remove the PVS6 from the control loop.
However, an intermediate step towards that goal would be to develop algorithms that reflect our understanding of how the PVS6 commands the Conext Gateway/InsightFacility over Modbus and "shadow test" them: run them in parallel, compare their outputs, and understand any divergences in the data. This process requires a lot of data to provide confidence that the algorithms are correct, and is even more critical because 1. we don't have the source code for the original system; 2. the system operates in a typical manner 99.999% of the time and in a exceptional manner during the other 0.001% of the time; 3. a full test setup would not be economical; 4. we have no way of inducing the exceptional behavior safely, realistically, and reliably; 5. mistakes can cost substantial money; and probably most importantly, 6. the system to be understood may need to control safety-critical ancillary devices and systems that have their own internal logic that we do not control and is difficult to comprehensively observe.
Once you have access to the Conext / Insight Facility, you may notice faults or errors present or in history
Bold ones are fairly common
F.45: Capacitor Over Temperature
Check the operation of the fans and the cleanliness of the filters on the sides of the SunVault and on the bottom of the XW.
F.69: External Sync Failed
I have never seen this. I'm assuming it can only happen in systems with multiple XW Pros
F.71: Battery Discharge Over Current
This can happen from time to time. If it happens repeatedly, it's worth understanding why. Otherwise, the BMS will take care of the cells.
F.72: External AC Contactor Malfunction
This is fairly common. The MID gets out of sync with the XW's expectations.
Turning everything off then on often fixes this
F.73: Battery Charge Over Current
This can happen from time to time. If it happens repeatedly, it's worth understanding why. Otherwise, the BMS will take care of the cells.
F.74: Battery Under Voltage
This happens very frequently and can generally be ignored. The BMS will take care of the cells.
F.75: Battery Over Voltage
This happens very frequently and can generally be ignored. The BMS will take care of the cells.
F.76: External Battery Stop Command
I have never seen this actually happen
F.77: AC Breaker Trip
Usually caused by an overcurrent condition - something tried to pull more than 40 amps from the ESS battery
Turn ESS AC breakers off then on again
F.93 / W.93: SunSpec Controller Comms Lost
This happens every day when the PVS6 reboots itself automatically
Interfaces
CAN: PVS6 to MIDC (obsoleted in later versions)
RS-485: PVS6 to MIDC (blue cable with RJ-45/8P8C ends), MIDC to PVDR, PVDR to MIO - SunSpec bus?
Multi-battery systems: additional XW Pro inverter/chargers for added AC output, additional batteries for added energy storage. Only one Gateway is needed for any number of inverters and batteries; each inverter needs an MIO.
MIO: CAN to Schneider Gateway, CAN to battery, Aux temperature input, XW-RPO input, "Comm from Hub+", comm to next MIO, ESS enclosure fan AC power in/out, LED board output, 48VDC to XW
SunVault internal cable colors
Blue in Hub+ (UTP, 8P8C/RJ-45): PVS - MIDC J10 Comm (probably Modbus over RS-485)
Gray (UTP, 8P8C/RJ-45): MIO J1 - MIDC J14 Comm (probably Modbus over RS-485)
Orange (UTP, 8P8C/RJ-45): MIO J2 - BMS Comm (CAN)
Green (UTP, 8P8C/RJ-45 to bare leads: MIO J4 - Conext Comm CAN1 breakout (CAN)
Blue in SunVault (UTP, 8P8C/RJ-45): MIO J7 - Sunvault LED display (WS2812B or similar addressable RGB protocol)
RS-485 over UTP/STP info
PVS6 uses same color and label for RS-485 2-wire as PVS5, which connected to SMA inverters over RS-485 2-wire
in SMA-US22 systems, wiring for T-568B cables is solid blue (B) to pin 2 (D+), stripe blue (b) to pin 7 (D-), solid brown (BR) to ground (GND)
T-568B and T-568B cables change orange and green pairs terminations; blue and brown pairs terminate the same in both
Thus, in T-568A/T-568B alike, in RJ-45 / 8P8C termination:
RS-485 A / D- = pin 5 (blue stripe)
RS-485 B / D+ = pin 4 (blue solid)
RS-485 GND = pin 8
Appears to be 9600 baud, no parity, 1 stop bit (9600 8N1)
Some unit numbers and CRCs are as expected; others seem random.
A lot of this is applicable to reverse engineering in general.
Spares
A lot of stuff that would be terrifying to do to / with production hardware is much less scary when you're operating on a spare.
eBay can randomly be your best friend.
If they're inexpensive enough, even broken spares can be a worthwhile investment. In some cases, they can even be more useful than working spares: when something fails, other parts can exhibit useful new behavior.
I spent an incredibly long time trying to get root access to my spare PVS6, but without the U-Boot console password, it seemed hopeless. A used (RMA'd, as it turned out) PVS6 that I acquired cheaply turned out to be a godsend: the eMMC was write-exhausted and therefore U-Boot could not read the firmware image, so the boot script failed and dropped me to the U-Boot console where I could poke around and figure out how to boot from USB as I had been trying to do for years.
After I was able to boot from USB, plans that had been coming together for ages moved forward quickly.
Hardware
Logic Analyzer
For a long time I misunderstood the purpose and operation of a logic analyzer.
An oscilloscope captures and displays analog waveform data, maybe over one channel, maybe over a few.
A logic analyzer captures and displays (often on a computer) digital waveform data, usually over several channels. But hey, you can also analyze a single pin. It samples and captures digital pins at a given sample rate.
Since inter-IC communication is digital data (really, voltage high and low), we can use a logic analyzer to capture the data for later analysis even if we don't know the underlying protocol or data rate.
Analysis is part brain power and context - like what protocols might be in use and what data we might find in the capture - and part computation - like applying a UART analyzer to some captured timing data to see ASCII characters being communicated.
Even an inexpensive ($12, for example) logic analyzer makes quick work of understanding what data is being sent over a known communications pin or trace. Most of the communications going on in these systems is very low-speed and is often serial, but of unknown baud and parity and so on. A logic analyzer and some timing data makes discovery of the correct settings fast and easy.
In fact, I would recommend using an inexpensive LA for these parts: though it hasn't happened to me so far, it would be a darn shame to be poking around with something expensive and have it go "zap."
Get started with something you control fully - an Arduino project with an i2c sensor and firmware you control and understand - before poking around PCBs you aren't familiar with.
UART breakout
Once you've logic analyzed your way into the correct settings, a UART will let you capture data more easily and with less fuss. Soldering one to the relevant pins will save you a lot of sanity - buying small CP2102/CP2014/similar breakouts from Aliexpress or similar in bulk makes committing one to a circuit that much less of a hassle.
JST-XH connectors can be re-pinned and used with 2.54mm spaced pin headers like the main UART on the PVS6 if you don't want to solder.
Often, you can be happy with just connecting GND and breakout RX / target TX. Don't connect breakout TX / target RX unless you plan to use it. VCC can also usually be skipped in my experience and it's one fewer thing to go wrong.
Evaluation Kits
Getting a sense for how things work when they aren't integrated with the system is much easier than when they are programmed with some source you don't have compiled into firmware binaries you probably also don't have. Spending money here can add up quickly, but when you're working with systems that are difficult to repair and replace, they're a much safer place to test hypotheses.
Sometimes you can also get binary firmware blobs running on such eval kits. The eval kits usually have PCBs and headers that are easier to navigate than shipping products.
Software
Your favorite Linux in a CLI virtual machine
I use Debian inside UTM. Reading and dumping ext4 and so on is that much easier. I ssh in from my regular OS terminal.
Executable binary in in, embedded protobuf files out. You can usually spot files containing them in a hex editor or in a grep for .google.protobuf.
Documentation
Datasheets
Any time you see an IC, look for its datasheet. Bullet points about what interfaces and peripherals are available can give you a sense of what ICs might be talking to each other before tracing them on a PCB. In 2024, microcontrollers are way overprovisioned in terms of peripherals, so don't read too far into them being utilized simply because they are present.
Most fun pinouts to look for, because they're easily transparently eavesdropped in-situ and in-use:
UARTs - RS-232 and RS-485
i2c
SPI
ethernet
Others that might be interesting:
JTAG
User Manuals
Not always the richest, but can give you a more complete understanding of what the thing can do.
Installation Manuals
Usually a step up from User Manuals, often includes more detail on what talks to what.
FCC certification documentation
If it has a radio in it, you can probably find a PDF of photos of the internals of the system and what frequencies the radio operates on by searching for "FCC (model number)".
Less fun than tearing the thing down yourself, but if you don't have a spare, it's a great place to start.
The README.md over there tells you about everything you need to know to make use of it. Dump the vars with vars -v, stand in amazement, have fun.
udpt
There's a nifty little idea that doesn't seem to have made the light of day. varserver can use templates to generate strings. udpt seems to be a little mechanism to broadcast udp packets generated from varserver templates. This could be used for very low-overhead, unauthenticated data dissemination.
Don't see anything crazy there. I can't remember if the coils have a polarity where the colored wire has to be in one slot and the white wire has to be in another - that was sort of my original suspicion. The CTs can work backwards, they'd just show negative numbers when they're positive. The coils terminate in the lower "sidecar" board behind the main PCB - use caution, there's 240VAC there when the PVS breaker is energized.
@koleson The rotary switch on the left side of a Sunvault V3 disconnects the feeder for the XW/Pro's AC1. There's just wire between the XW/Pro and the batteries.
Here's a photo of the schematic and wiring instructions from one of my Sunvaults:
The app usually shows old data on the Home tab, but current data on the Analyze and Panels tabs. If I force-quit and relaunch the app, it will usually show live data on the Home tab.
As for the battery, it still charges and discharges, but starting around March 17 (which seems likely to coincide with the firmware update since the version says 2025.03), about every 5 minutes it stops discharging completely for about 15 seconds, then ramps back up. This cycle keeps repeating. Same thing happens when it is charging.
Before:
After:
Zoomed in view (from emporia vue monitoring one phase of the battery):
Yeah, I was hoping for an easy way to get shell access to see if it is a CPU load issue. The holy grail for me would be to divorce from the unreliable SunPower/SunStrong cloud and have everything monitored and controlled locally.
The above is all passive monitoring at the moment, and the panel data is only showing my iQ8 panels right now and not the Sunpower ones yet. Parts involved, 3 Raspberry Pis, an iQ Gateway, four CTs, a power monitor hat, a couple ethernet switches, a CAT5 splitter, a CAT5 breakout connector, and CAN bus hat, a 48v industrial equipment USB-C power supply, and other assorted accouterment.
Fun times, my Sunvault has started acting up with the new firmware now too. Seems like it will work fine for a few minutes, then loses sync with the XWs and then resets.
Is there a nondestructive test that can be done to check for USB booting? I have a W#### Serial Number PVS6
after hooking up the UART, at boot, the PVS will print the U-Boot version. in the U-Boot 2017.03 bullet of PVS6_NOTES.md, there are some notes on which build dates have which features. I think Dec 02 2019 is the only version that doesn't have working USB boot - newer dates should have working USB boot.
If they're inexpensive enough, even broken spares can be a worthwhile investment. In some cases, they can even be more useful than working spares: when something fails, other parts can exhibit useful new behavior.
They're being sold as part of the dregs of the sunpower training facility at that location. A few PVS6's on the wall with various demo installations of sunvaults. Also a bunch of sunvault/sunpower training and installation manuals.
I picked up a spare new-in-box PVS6 from eBay. Is there any way to read the contents of the eMMC without desoldering it?
Also, the swagger doc shows an example firmware download URL - do you know the download URL for the new SunStrong firmware?
Finally, have you tested the USB 4GB SLC flash drive migration? Do you have to power off the PVS and then insert the drive and power on? What happens if you remove the drive afterwards? What is in the /app0 directory tree?
@TheFaceCo heck yeah! i was lucky enough to get a Hub+ for $150 on eBay a few months back, but might have to hit up that closeout garage sale and see what's left over. Those installation manuals look tempting.
@webdeck if there's a way to read the eMMC in-situ without the U-Boot password, I haven't found it yet.
I don't have the download URL for the latest firmware.
SLC flash drive thing is pretty simple - insert it at any time, the PVS6 formats the drive. If you remove the drive, anything written to it goes away, but I don't think there's anything on that partition that can't be recreated from what's on the main firmware partitions. /app0 has a variety of config files, firmwares for other devices, and a grab bag of other things. It doesn't have the system commissioning info, so that's kind of a bummer.
SLC flash drive thing is pretty simple - insert it at any time, the PVS6 formats the drive. If you remove the drive, anything written to it goes away, but I don't think there's anything on that partition that can't be recreated from what's on the main firmware partitions. /app0 has a variety of config files, firmwares for other devices, and a grab bag of other things. It doesn't have the system commissioning info, so that's kind of a bummer.
Am I doing something wrong? I inserted the flash drive, left it in there for a couple of days, and when I removed it, I don't see much of anything on it. Here is what I see:
@webdeck because it's an overlay, files will only be populated on the drive as they change from what is currently on the eMMC.
Right, but it didn't write anything over a couple of days time, so I'm not sure how this reduces the eMMC wear problem. Does it mainly get written to when there are software updates? I guess I can just leave it plugged in and check again after the next software update.
I'm also confused by the two sqlite databases that appear to be character devices instead of regular files... what's up with those?
I'm also confused by the two sqlite databases that appear to be character devices instead of regular files... what's up with those?
@webdeck they're what overlayfs calls "whiteout" files (docs), effectively a delete marker for those files on the underlying ("lower") filesystem. Which makes sense, the -shm and -wal files associated with a SQLite database only exist while some process has the database file open (they're deleted when the last one closes it), see docs. The fact that you don't have the corresponding .sqlite file in upperdir suggests the database contents haven't been modified since the overlay was set up.
I'd suggest caution with unplugging the flash drive while the PVS is running: it will almost certainly crash anything that had those files open, and even if it doesn't crash them, reverting the filesystem to an older state while things are running could easily lead to state corruption of one form or another. Plugging the drive back in after the PVS has been running without it risks overlaying old changes on a modified eMMC lower directory, which could similarly cause all sorts of application misbehavior. Depending on the overlayfs mount options (I don't know which ones are in use), it could also cause overlayfs itself to misbehave (docs).
Don't see anything crazy there. I can't remember if the coils have a polarity where the colored wire has to be in one slot and the white wire has to be in another - that was sort of my original suspicion. The CTs can work backwards, they'd just show negative numbers when they're positive. The coils terminate in the lower "sidecar" board behind the main PCB - use caution, there's 240VAC there when the PVS breaker is energized.