Problem, background: My Sophos AP15 became âbrickedâ after an update.
Symptom: The LED is solid orange (yellow) and there is no response using ethernet. In addition, no maintenance port exists, at least not on my AP.
Initial actions:
- A chat with Sophos support did not result in anything useful.
- Googling the problem resulted in quite a few hits with people complaing about the same problem.
- Good news: The AP is probably not dead. Something just went awfully wrong at while updating.
- One solution was found; a german gentleman who removed a circuit and successfully reprogrammed it. However, this seems a bit complicated. Is there perhaps an easier way?
Investigation:
1. Open the AP. Four screws under the rubber pads.
Conclusion, notes:
- It looks promising â there are two terminal blocks which may provide a way to communicate with the AP.
- However, the blocks are unmarked. This needs to be investigated.
2. I look closer at the terminal blocks. Let’s say that the big one is A and the small one is B.
Conclusion, notes:
- Interface A is probably a 14 pin JTAG.
- Interface B is more interesting. Could it be a serial interface? Now,
let´s number pins from left to right, 1 through 4 and do some measurements.
3. Conduct some measurements with a DVM to find out what these pins actually do. The AP is probably sending something when booting so conduct all measurements at startup.
| Pin | Voltage DC | Voltage AC | Resistance (4k), ref GND |
| 1 | 2.57 | 0.03 | OL |
| 2 | 0.00 | 0.02 | 0.00 |
| 3 | 0.00 | 0.01 | 0.561 |
| 4 | 3.38 | 0.00 | OL |
Conclusion, notes:
- Pin 1 is most likely a TX pin
- Pin 2 is a GND pin
- Pin 3 is most likely an RX pin
- Pin 4 is a VCC pin (3.3 VDC)
- This could be a serial interface
4. Now, letâs use an oscilloscope and have a look at pin 1.
Conclusion, notes:
- The signal looks very much like standard serial communication.
- If it is a serial signal the first low part of the signal constitutes one bit.
5. Adjust the timebase to be able to measure the width.
The width=t ~ 8.7 us which means baudrate=1/t ~ 114943 baud. This number is very close to the standard baudrate = 115200.
Conclusion, notes:
- Terminal block B is a 3.3 V serial interface with the following pinout:
- Pin 1 is TX
- Pin 2 is GND
- Pin 3 is RX
- Pin 4 is VCC (no use)
- Baudrate is probably 115200
6. Now, I try the conclusion in 5. Use a TTL-USB converter cable. In my case, I already had a FTDI cable model TTL-232RG-VREG3V3-WE around. It has the following pinout:
7. I attach some test leads to the TTL-USB converter cable following the same colours as in the specification above. I attach them accordingly, i.e. TX-RX and RX-TX.
8. I then fire up good old Putty and use the following settings:
- Serial settings: 8N1 (most common), no Flow Control (flow control is rarely used these days)
- Speed: 115200 baud
- Com Port: COM10 (in my case. It may vary. Use Device manager to check)
9. I attach power to the AP and keep my fingers crossed.
Yipiieee!!! The research paid off. There is text on the screen!!!!
U-Boot 1.1.4-gb47de1b6 (Jan 24 2017 - 11:22:47)
ELX version: 1.0.0
7679WSC - Scorpion 1.0DRAM:
sri
Scorpion 1.0
ath_ddr_initial_config(178): (32bit) ddr2 init
tap = 0x00000003
Tap (low, high) = (0x4, 0x1f)
Tap values = (0x11, 0x11, 0x11, 0x11)
128 MB
Flash Manuf Id 0xc2, DeviceId0 0x20, DeviceId1 0x18
Flash [MX25L12845E] sectors: 256
Flash: 16 MB
*** Warning *** : PCIe WLAN Module not found !!!
In: serial
Out: serial
Err: serial
Net: ath_gmac_enet_initialize...
athrs_sgmii_res_cal: cal value = 0x1
ath_gmac_enet_initialize: reset mask:c02200
Scorpion ---->8035 PHY*
AR8035 PHY reg init
: cfg1 0x80000000 cfg2 0x7114
eth0: 00:00:aa:bb:cc:dd
AR8035 found!
[0:4]Phy ID 4d:d072
Port 0, Neg Success
eth0 up
eth0
Setting 0x18116290 to 0x458ba14f
Hit any key to stop autoboot: 0
## Booting image at 9f070000 ...
Image Name: MIPS OpenWrt Linux-3.18.11
Created: 2019-11-18 9:53:25 UTC
Image Type: MIPS Linux Kernel Image (gzip compressed)
Data Size: 6900777 Bytes = 6.6 MB
Load Address: 80060000
Entry Point: 80060000
Verifying Checksum at 0x9f070040 ...Bad Data CRCConclusion, notes:
- A standard boot loader, U-Boot is used in Sophos AP 15. U-Boot can easibly be googled.
- At the end it is obvious that it is indeed a firmware update that has gone wrong; âBad Data CRCâ
- The fault is in SW and thanks to the presence of U-Boot it might be fixable.
10. Now, letâs restart the AP and interrupt the autoboot by pressing any key at the right time (see above, âhit any key to stop autobootâ). This brings us to a prompt, âath>â. Looking at the U-Boot manual on the internet I now try the command âhelpâ to see what it does.
ath> help
? - alias for 'help'
autoscr - run script from memory
base - print or set address offset
bdinfo - print Board Info structure
boot - boot default, i.e., run 'bootcmd'
bootd - boot default, i.e., run 'bootcmd'
bootelf - Boot from an ELF image in memory
bootm - boot application image from memory
bootp - boot image via network using BootP/TFTP protocol
bootvx - Boot vxWorks from an ELF image
cmp - memory compare
coninfo - print console devices and information
cp - memory copy
crc32 - checksum calculation
dhcp - invoke DHCP client to obtain IP/boot params
echo - echo args to console
erase - erase FLASH memory
ethreg - S26 PHY Reg rd/wr utility
exit - exit script
flinfo - print FLASH memory information
go - start application at address 'addr'
help - print online help
iminfo - print header information for application image
itest - return true/false on integer compare
loop - infinite loop on address range
md - memory display
compute MD5 message digestmii - MII utility commands
mm - memory modify (auto-incrementing)
mtest - simple RAM test
mw - memory write (fill)
nfs - boot image via network using NFS protocol
nm - memory modify (constant address)
pci - list and access PCI Configuration Space
ping - send ICMP ECHO_REQUEST to network host
pll cpu-pll dither ddr-pll dither - Set to change CPU & DDR speed
pll erase
pll get
printenv- print environment variables
progmac - Set ethernet MAC addresses
protect - enable or disable FLASH write protection
rarpboot- boot image via network using RARP/TFTP protocol
reset - Perform RESET of the CPU
run - run commands in an environment variable
saveenv - save environment variables to persistent storage
sendmagic - (usage) send/broadcast MAGIC PACKET to network host
- <timeout> timeout for response
- <retry> number of times magic to be sent to network host
- <devid_base_addr> baseaddr of sector containing devid
- <devid_len> offset to base addr
- <offset_to_baseaddr> offset to base addr
sendsts - send status of firmware recovery process
- <stscode> 0 - send apstate, non-zero - send specified statuscode
setenv - set environment variables
sleep - delay execution for some time
test - minimal test like /bin/sh
tftpboot- boot image via network using TFTP protocol
version - print monitor version
ath>Conclusion, notes:
- The full U-Boot seems to be present and there are several useful commands available.
- A new firmware is most likely transferred by using TFTP (Trivial File Transfer Protocol), i.e. tftpboot. TFTP is a simple lockstep File Transfer Protocol which allows a client to get a file from or put a file onto a remote host.
11. If indeed tftpboot is used, I want to know how it is configured, i.e. I want to know the IP of the AP and the IP of the server where the AP expects the firmware to be. There should be no harm in trying that command when the AP is not connected to anything. So, letâs try âtftpbootâ at the prompt.
ath> tftpboot
eth0 link down
Using eth0 device
TFTP from server 192.168.99.8; our IP address is 192.168.99.9
Filename 'uImage_AP15'.
Load address: 0x81000000
Loading: Tx Timed out
Abort
ath>Conclusion, notes:
- It is highly likely that tftpboot is used to download a new firmware to the AP using the below parameters:
- The IP of the AP is: 192.168.99.9
- The IP of the server, i.e. usually the firewall itself, is: 192.168.99.8
- The filename of the image the AP tries to download is: uImage_AP15
12. The way forward must then be to create a local environment as above, i.e. to create a TFTP-server with the image file on it and then connect the AP to it. Now, the next step is to find the firmware. I spent quite a bit of time looking on Sophos homepage but I was unable to find any firmware.
Conclusion, notes:
- The firmware is probably distributed as a part of an update for the entire firewall. In that case, the firmware should be present as a file called âuImage_AP15â somewhere in the file system of the firewall.
13. I enable SSH access in the firewall (I usually keep SSH access disabled).
14. I connect to the firewall using Putty and use the Unix command âfindâ to look for âAP15â. I get a hit in /etc/wireless/firmware. Good, the path name makes sense.
loginuser@stargate:/etc/wireless/firmware > ls -al
total 74140
drwxr-xr-x 2 root root 4096 Feb 10 18:49 .
drwxr-xr-x 4 root root 4096 Nov 20 14:48 ..
-rw-r--r-- 1 root root 134 Nov 20 11:55 AP100C.devinfo
-rw-r--r-- 1 root root 7147656 Nov 20 11:55 AP100C.uimage
-rw-r--r-- 1 root root 133 Nov 20 11:55 AP100.devinfo
-rw-r--r-- 1 root root 7149112 Nov 20 11:55 AP100.uimage
-rw-r--r-- 1 root root 134 Nov 20 11:55 AP100X.devinfo
-rw-r--r-- 1 root root 7147507 Nov 20 11:55 AP100X.uimage
-rw-r--r-- 1 root root 124 Nov 20 11:54 AP10.devinfo
-rw-r--r-- 1 root root 2872508 Nov 20 11:54 AP10.uimage
-rw-r--r-- 1 root root 133 Nov 20 11:54 AP15C.devinfo
-rw-r--r-- 1 root root 6896868 Nov 20 11:54 AP15C.uimage
-rw-r--r-- 1 root root 132 Nov 20 11:54 AP15.devinfo
-rw-r--r-- 1 root root 6900841 Nov 20 11:54 AP15.uimage
-rw-r--r-- 1 root root 124 Nov 20 11:54 AP30.devinfo
-rw-r--r-- 1 root root 2872041 Nov 20 11:54 AP30.uimage
-rw-r--r-- 1 root root 124 Nov 20 11:55 AP50.devinfo
-rw-r--r-- 1 root root 3897845 Nov 20 11:55 AP50.uimage
-rw-r--r-- 1 root root 133 Nov 20 11:55 AP55C.devinfo
-rw-r--r-- 1 root root 7145137 Nov 20 11:55 AP55C.uimage
-rw-r--r-- 1 root root 132 Nov 20 11:55 AP55.devinfo
-rw-r--r-- 1 root root 7148578 Nov 20 11:55 AP55.uimage
-rw-r--r-- 1 root root 53 Nov 20 14:48 AP5.devinfo
-rw-r--r-- 1 root root 249 Nov 20 11:55 APX.devinfo
-rw-r--r-- 1 root root 16654365 Nov 20 11:55 APX.uimage
-rw-r--r-- 1 root root 56 Nov 20 15:18 RED15w.devinfo
loginuser@stargate:/etc/wireless/firmware >Conclusion, notes:
- The folder seems to contain firmware for all Sophos AP products, including the one I want, AP15. Two files are of interest for my AP:
- AP15.uimage, 6900841 bytes
- AP15.devinfo, 132 bytes
14. I download all files in the folder on my firewall using the Bitvise SSH Client.
15. Looking at AP.devinfo with Notepad++ results in the following:
DEVICE_TYPE=AP15
FIRMWARE_VERSION=9500-wifi-0c67c7fe-b19f3b0
QCA_VERSION=c3496dd
FIRMWARE_LENGTH=6900841
FIRMWARE_TAG=apfw_11.0.010Conclusion, notes:
- The length specified in the devinfo file matches the size on disk for the uimage file.
- No CRC info but since the firmware is distributed as a part of the update bundle for the entire firewall it is likely that this file is intact and that the CRC error is a result from a bad transfer from the firewall to the AP.
17. Now I have to use a local TFTP server. I guess any SW can be used but I download and install this tftp server from Sourceforge.
18. I configure the TFTP-server. The setup is quite straight forward. The IP from 11 is used, i.e. the address that the AP wants to have: 192.168.99.8.
19. I connect the AP to a switch which is also connected to the NIC of my computer where the TFTP Server has been installed.
20. I copy the firmware, âAP15.uimageâ, into the root directory of the TFTP Server and rename it like the AP wants, i.e. âuImage_AP15â (without filetype for some, unknown reason).
21. I now connect to the AP again using the serial interface and interrupt the autoboot as in 10 to get to the prompt.
22. At the prompt I now type âtftpbootâ and keep my fingers crossed.
ath> tftpboot
Speed is 100TX
Using eth0 device
TFTP from server 192.168.99.8; our IP address is 192.168.99.9
Filename 'uImage_AP15'.
Load address: 0x81000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
################################################
done
Bytes transferred = 6900841 (694c69 hex)
ath>Great! This looks promising!
Conclusion, notes:
- Now I have the firmware image at 0x81000000.
- However, according to 9 the AP wants to boot from 0x9f070000. Why doesnât this match?
- After a bit of googling I realize that the procedure is first to download a new image, then erase the previous version and finally copy the new image to the correct address, in this case 0x9f070000.
- Therefore, the next step is to erase the erroneous (bad CRC) section in the flash memory of the AP and finally write the image now located at 0x81000000 to the correct address from where AP is trying to boot.
23. Erasing is possible using the command found in 10. So, now I need to find the EXACT part to erase. The command âbdinfoâ seems to be a good choice.
ath> bdinfo
boot_params = 0x87F7BFB0
memstart = 0x80000000
memsize = 0x08000000
flashstart = 0x9F000000
flashsize = 0x01000000
flashoffset = 0x00029BE4
ethaddr = 00:00:AA:BB:CC:DD
ip_addr = 192.168.99.9
baudrate = 115200 bps
ath>24. From 9 I know that the AP wants to boot from 0x9f070000 so that is obviously the start address. After a bit of googling I realize that the key is the parameter flashsize in 23, i.e. the end address is simply 0x9f070000 plus 0x01000000. This can be done using the Windows Calculator and the result is 0xA0070000.
Conclusion, notes:
- The start address and the end address have been found and the command to use should be âera 0x9f070000 0xA0070000â at the prompt in the AP.
25. So, time to erase, a VERY SCARY operation, indeed. However, after having double checked everything a couple of times I think I am ready to proceed.
ath> era 0x9f070000 0xA0070000
Erasing flash...
First 0x7 last 0xff sector size 0x10000 255
Erased 249 sectors
ath>Conclusion, notes:
- No smoke or anything. It seems OK but I cannot tell if everything is fine until after the final step.
26. So, to the best of my knowledge this should be the final step, i.e. copying the firmware from 0x81000000 to 0x9f070000. Reading the U-Boot manual I conclude that the command is:
cp.b <from_address> <to_address> <size>
In my case this is:
cp.b 0x81000000 0x9f070000 0x694c69
ath> cp.b 0x81000000 0x9f070000 0x694c69
Copy to Flash...
Copy 6900841 [0x694c69] byte to Flash... write addr: 9f070000
done
ath>Conclusion, notes:
- OK, again no smoke. The copying is done and everything seems OK.
27. The final step is now to try to boot the AP. This is done using the command âbootâat the prompt.
ath> boot
## Booting image at 9f070000 ...
Image Name: MIPS OpenWrt Linux-3.18.11
Created: 2019-11-18 9:53:25 UTC
Image Type: MIPS Linux Kernel Image (gzip compressed)
Data Size: 6900777 Bytes = 6.6 MB
Load Address: 80060000
Entry Point: 80060000
Verifying Checksum at 0x9f070040 ...OK
Uncompressing Kernel Image ... OK
Starting kernel ...
[ 0.000000] Linux version 3.18.11 (bamboo@ip-10-104-116-153) (gcc version 4.8 .3 (OpenWrt/Linaro GCC 4.8-2014.04 unknown) ) #3 Mon Nov 18 09:53:13 UTC 2019
[ 0.000000] bootconsole [early0] enabled
[ 0.000000] CPU0 revision is: 00019750 (MIPS 74Kc)
[ 0.000000] SoC: Qualcomm Atheros QCA9558 ver 1 rev 0
âŚ<snip>Yay!!!!!!!!!!
Much better than a CRC failure….!!
Conclusion, notes:
- After having disconnected the serial interface and put the AP back together it was installed in my system again and it immediately worked perfectly đ.
- A few hours of investigation (and fun) paid off.
- Never give up if there is still hope!
- Things certainly do not always work out the way you want or expect. So, do not underestimate what some good whisky can do along the way to keep your spirits up!
I’m impressed!!