Unbricking an APC SMT3000 UPS after a failed firmware update

Background

I recently replaced the batteries in my APC SMT3000RM2U UPS and decided it would be a good idea to update the firmware as I barely got 3 years out of the previous set. I thought maybe APC may have changed the charging algorithm or something and thought it was worth a shot...

After discovering my device was a SMT3000 with hardware version ID18, the APC documentation stated that the firmware file SMT18UPS_15-0.enc would be the most up to date one to use.

What Happened?

After initiating the firmware update from my AP9631 network management card, the NMC eventually showed an error message stating "Lost the local network management interface-to-UPS communication". At this time there was also a frozen message on the UPS front panel:

After waiting a couple hours I decided to power cycle the unit. After power cycling I was greeted by an empty display:

The UPS serial console and USB ports were non responsive as well. I did some Googling and ran across posts where people were able to use the serial console and xmodem to upload a firmware to a bricked device, however my serial console was completely non responsive and I couldn't get that to work.

There are comments on the EEVblog forum that indicate that if you update to 09.4 (SMT18UPS_09-4.enc) first, then update to 15.0, then this wouldn't have happened. Version 15 relies on a newer bootloader which is installed with previous firmware versions. If you install version 15 while running an old bootloader, the firmware gets written to the wrong position in the microcontroller memory and will fail to boot. It would have been nice if APC had a note of that in the release notes but it is what it is. If you haven't updated yet you can find a copy of the 09.4 firmware in the files section below.

Recovery

Warning: This device has dangerous voltage present when plugged in. Additionally, if the device has been plugged in to AC power, the capacitors on board can potentially hold a lethal voltage for some time. Under no circumstances should you ever operate this device without the case or attempt to flash it while plugged into AC power. If you are unsure of the risks, and how to mitigate them, you should probably not attempt this. Follow these instructions at your own risk.

These instructions should only be attempted on an APC SMT ID18 board. It should work on ID18 SMT750/SMT1000/SMT1500/SMT3000 models. The instructions are probably the same for other IDs but a different firmware dump would be required.

This wouldn't have been doable without the posts that user AstroAU at the EEVblog forum made. They dumped the UPS management board STM32F103RCT6 microcontroller from their working ID18 SMT1500 and provided programming pinouts. I made the logical leap that the firmware would work on the SMT3000 when another user who had a SMT750 had success flashing this.

The first task was removing the UPS management board from the UPS. The UPS top cover is held on by a bunch of Phillips head screws and comes right off.

The UPS management board is located underneath the SmartSlot. The slot needs to be removed from the chassis by pushing in the two little tabs holding it to the retaining clip. The SmartSlot and management board are located in the bottom right of this photo:

The UPS management board is held on by some plastic standoffs. Squeeze those with pliers and the board will pull right off the board.

To flash the board you will need a ST-LINK USB programmer and 5 female/female dupont cables. These can be found on Amazon or AliExpress if you can wait a little longer. I conveniently had a programmer already from when I was playing with a STM32 Blue Pill dev board.

Connect the pins of the ST-LINK to the UPS management board using dupont cables as per below:

You need to download and install the STM32 ST-LINK Utility to interface with the ST-LINK. Once it is installed, launch it and change the size input box to show 0x40000 (256K). Then plug the ST-LINK into a USB port and click Target > Connect. If everything is connected correctly you should see something like this:

The first thing I'd recommend is saving a copy of the current device memory. Click the floppy save button and save it to your computer as "brokenupsfirmware.bin" or something similar.

Next download the working firmware dump from below (APC_SMT18_15-0_STM32F103RCT6.zip) and extract it's contents to your computer. Then click Target > Program & Verify. Select the APC_SMT18_15-0_STM32F103RCT6.bin file and ensure "verify while programming" is selected. Click Start to upload.

If successful, you should see log messages similar to this:

118:52:23 : [APC_SMT18_15-0_STM32F103RCT6.bin] opened successfully.
218:52:23 : [APC_SMT18_15-0_STM32F103RCT6.bin] checksum : 0x01C727B8 
318:53:06 : Memory programmed in 15s and 734ms.
418:53:06 : Verification...OK
518:53:06 : Programmed memory Checksum: 0x01C727B8

At this point the board can be reinstalled into the UPS, SmartSlot put back in place and case closed. Plug the UPS in and with a little luck the front display should light up and be working.

The UPS serial number should still remain the same as that information is held on a separate 93LC66 EEPROM on the board.

After recovering using this method, the APC Firmware Upgrade Wizard reports these as my firmware versions:

Updating MCU/MBL

After the successful recovery, I wanted to make sure the MCU and MBL were updated to the latest version (20.0). I was able to do this by using the Smart-UPS Firmware Upgrade Wizard via USB to downgrade the UPS to release 09.6 (SMT18UPS_09-6.enc). This release contains the updates for that microcontroller, 15.0 does not for some reason. After that was applied, I updated to SMT18UPS_15-0.enc using the upgrade wizard to bring the UPS back to 15.0.

This is probably not needed but I just wanted to be fully up to date.

Files

STM32 ST-LINK Utility
STM32F103RCT6 firmware dump from a working ID18 SMT1500 (v15.0)
Stock APC SMT ID18 firmware (15.0, SMT18UPS_15-0.enc)
Stock APC SMT ID18 firmware (09.6, SMT18UPS_09-6.enc)
Stock APC SMT ID18 firmware (09.4, SMT18UPS_09-4.enc)
Smart-UPS Firmware Upgrade Wizard 4.3.8

References

https://www.eevblog.com/forum/repair/help-with-bricked-apc-ups-smt1500/
https://community.se.com/t5/APC-UPS-Data-Center-Enterprise/APC-UPS-SMX1500-firmware-update-failure-and-fixed/td-p/294993