Cisco WLC 7.2 FUS code release

Cisco recently released version 7.2 of their Wireless LAN Controller code. Along with this update came something new for several administrators in the form of an ‘FUS’ update. This update is available for the 5500 , WiSM 2, and the Flex 7500 platforms and contains a variety of firmware specific updates for each platform including:

  • For the 5500 and WiSM2
  • Field Recovery image update
  • Bootloader updates to 1.0.16
  • Offline Field Diagnostics to version 0.9.28
  • USB Console to 2.2
  • MCU image update too 1.8 (5500 only)
  • FPGA update to 1.7 (5500 only)

For the Flex 7500 controllers, there is a RAID firmware update. There is no FUS update for the 2500 controller or any of the legacy platforms (they’re not supported in release 7.2 in general anyway). Buried in the release notes are a variety of nuggets, but it is imperative that this update be installed by itself with a reboot between it and the main 7.2 code release. The order is not important, just the fact that there is a reboot in-between. Additionally, in order for the FUS image to actually update the various components, you need to have a serial attachment to the WLC during the reboot and you must interact with the image upgrades in order for them to execute. This means that if you’re used to doing the ER updates that you just ‘apply and forget’, this is going to be a deviation from that process. To add to this, each update requires you to answer ‘yes’ in order happen but they’re not quick. You will end up burning somewhere south of about a half an hour to pull off a complete upgrade and if you happen to miss one, you’ll have to reupload the image and step through it again. Cisco is nice enough to tell us during the update approximately how long each will take and these numbers are fairly close to what I’ve experienced in the field. The tally on a 5500 is:

Upgrade Bootloader from 1.0.1 to 1.0.16

  • Erasing Flash (estimated 6 seconds)
  • Writing to Flash (estimated 41 seconds)
  • Checking Boot loader integrity (estimated 2 seconds)
  • Total: 49 seconds

Upgrading FPGA from rev 1.3 to rev 1.7

  • Upgrade takes about 75 seconds to complete

Upgrading Env from rev 1.6 to rev 1.8

  • Upgrade takes about 4 seconds to complete

Upgrading USB from rev 1.27 to rev 2.2

  • Upgrade takes about 11 seconds to complete

Upgrade OFD from version WLCNG OFD 0.8.1 to WLCNG OFD 0.9.28

  • Erasing Flash (estimated 24 seconds)
  • Writing to flash (estimated 111 seconds)
  • Total: 135 seconds

Upgrade Field Recovery Image from version 6.0.182.0 to 7.0.112.21

  • Erasing Flash (estimated 49 seconds)
  • Writing to flash (estimated 716 seconds)
  • Total: 765 seconds

Yes, you read that correctly – the Field Recovery Image takes a whopping 13 minutes to execute! Of interest to those of you that use the USB serial console built into the WLC is the fact that the USB update will flat out break your session. Once you kick off that particular update, you should suspend you session and wait for it to complete. The kicker of course is that you won’t know since you don’t have a console session. The lesson here is that while it is possible to perform these updates using the USB console, you’ll not regret preferring the good old fashioned RJ-45 console cable method.

If you happen to miss an update and have to reapply the image, you’ll notice that the FUS image will proactively check to see if the updates have been applied already:

====================

Checking for Bootloader upgrade

Bootloader upgrade …

Bootloader 1.0.16 is up to date.

====================

Checking for FPGA upgrade

FPGA upgrade …

FPGA image is up to date

It will perform this check for all components, but when it gets to the Field Recovery Image, it will actually ask you if you want to re-apply it:

Field Recovery Image upgrade …

        Field recovery image Current version 7.0.112.21 is up-to-date.

        Answer “y” below will force upgrade to run again.

        Are you sure you want to proceed (y/N) ? n

Again, note that if you re-apply this particular update, you’re in for a thrilling 13 minutes of ‘edge of your seat’ thrills while it completes. There is no way to cancel it and as you’re warned numerous times throughout the FUS process in bad english:

      * Lost POWER will completely kill this unit and not recoverable. *

      * There may be multiple reboot. Please let the program run.      *

Once you’ve completed your updates, and you’re observing the production image boot, it will verbosely tell you what the version of all of these components are so you can tell that they’ve been successfully applied or not:

Cisco AireOS Version 7.2.103.0

Firmware Version FPGA 1.7, Env 1.6, USB console 2.2

Initializing OS Services: ok

Applying these updates is important and does resolve a variety of issues so it is recommended to go through whatever outage window you’re going to require to apply them or you may want to consider pulling a spare (+1) controller out of service, upgrading it and moving all of your Access Points over to free up your primary for upgrade. Either way, you should do this – just make sure the updates actually apply!

Resurrecting a bricked NM-AIR-WLC6

Cisco recently posted this addendum to their Software Downloads section for the Cisco Wireless LAN Controller Module:

Warning: the Wireless LAN Controller Network Module (NM-AIR-WLC6-K9) is not supported in any software release after 4.2.209.0. Attempting to install 5.0 or later software can permanently damage the module.

This is a pretty recent addition and appears to have been an oversight the past year or so while they’ve been happily releasing version after version of NMWLC code without this disclaimer. If you’re like me, you’ve been keeping up on your latest and greatest software releases and you may find yourself in some murky waters if you happen to have this module. Where I landed was a module that would boot fine, but would not establish any network connectivity (management, AP join, etc). You should note that this article in it’s entirety does not apply to the NME module, just the NM. The NME module has more memory and a 1G internal interface to the ISR, the regular old NM has less memory and only a 10/100 interface. You can tell which module you have by looking at the silkscreen on the back of the module or by doing a ‘show sysinfo’ at the CLI of the controller.

This article is not supported by Cisco, TAC, or myself. You may further damage your NM if you proceed. This article is not for the faint of heart and will most certainly void any warranty you may have. If you have a bricked NM under SmartNET, you should contact TAC for a replacement unit, not follow the directions in this post. I do not guarantee any work here and you can severely damage your module, it’s flash, or your PC. Read and follow this article at your own risk!

Now thats out of the way, the specifics of my problem landed me in a situation where I could not roll back the version of code on my flash (not having network connectivity really limits you, I gotta admit). You may find yourself with a corrupt flash, unable to boot, or other general mayhem. Once Cisco released this notification, it dawned on me that the version of code on my flash was likely the culprit. Since the NMs are basically an Pentium III with some memory and a flash to boot off (similar to the CUE or Content Engine modules) running Linux, I figured I should be able to copy the flash from a good NM and I’d be back in business. Having located a donor NM (thanks to Robert B. for his support here), I assembled  the following items to move on:

  • Donor working NM to rob/copy the flash off of
  • A small screwdriver
  • Old laptop with Cardbus/PCMCIA slot
  • CF to PCMCIA adapter (like one of these)
  • USB Flash drive larger than 256M formatted something that Linux can write to
  • A Linux distribution that I could boot off of CD like DSL
  • A static bag to work off of
Getting a donor module
The first thing I did was to remove the CF module from the donor NM.
Step 1) Place the donor NM on a static safe work place. The bag it came in would be good.
Step 2) Confirm that the module you’re working on is an NM, not an NME.
Step 3) Locate the cover that hides the flash module.
Extracting the flash
You must then remove the protective cap around the flash module.
Step 1) Unscrew the CF housing.
Step 2) Lift up gently on the right edge of the cap and it should fall off the module.
Remove the flash from the NM
Gently grasp the Cisco flash module by both edges and pull it directly out of the NM.
Insert the flash module into your CF reader.
This should be pretty straightforward.
Once you have the flash module in a CF reader, we’re going to be focused on getting a good block image off of it. The rest of this article will discuss how to take an image of the flash module and store it on a USB flash drive. Once you have your favorite LiveCD of Linux (or BSD if you prefer) downloaded, boot your old laptop off of it. We chose a LiveCD release so that we can do this on a laptop without having to do a fully blown installation of Linux just for this one project. Feel free to use any sort of Linux box you happen to have laying around. 🙂
Once you’ve successfully booted Linux, you’ll need to open a terminal window. In DSL, there is a link to the Terminal app in the bottom left corner. Attach your USB drive and insert your PCMCIA flash reader once your system is booted and your terminal is up.
Type:
sudo su


  -This puts us into super-user mode so we don’t run into any permissions issues
Then:
dmesg


  -This gives you a dump of system messages. In particular we’re looking for two things. The USB drive and the CF adapter. In my system, this looked like:
<6>hub.c: new USB device 00:1d.1-1, assigned address 3
<6>scsi2 : SCSI emulation for USB Mass Storage devices
<4>  Vendor: USB 2.0   Model: FLASH DISK        Rev: 1.0
<4>  Type:   Direct-Access                      ANSI SCSI revision: 02
<4>Attached scsi removable disk sdb at scsi2, channel 0, id 0, lun 0
<4>SCSI device sdb: 2033664 512-byte hdwr sectors (1041 MB)
<4>sdb: Write Protect is off
<6> sdb: sdb1
This tells us that our flash drive is at /dev/sdb1 (the last line above) so let’s mount it using:
mount /dev/sdb1 /mnt
Next we look for our flash reader. In my system, this looked like:
<6>cs: memory probe 0xa0000000-0xa0ffffff: excluding 0xa0000000-0xa0ffffff
<6>cs: memory probe 0x60000000-0x60ffffff: clean.
<4>hde: STI Flash 8.0.0, ATA DISK drive
<4>ide2 at 0x100-0x107,0x10e on irq 11
<4>hde: attached ide-disk driver.
<6>hde: 501760 sectors (257 MB), CHS=980/16/32
<6> hde: hde1 hde2 hde3
<6>ide_cs: hde: Vcc = 3.3, Vpp = 0.0
For this one, we’re not interested in any partition information like we were on the USB device, we’re just interested in the device name. Here, we see that this device is hde (the beginning of the third line) . Once we have both the USB drive mounted and the flash drive identified, we’re going to use dd to take a block image of the device by typing:


dd if=/dev/hde of=/mnt/sdb1/nm.image
This deconstructs like this:
  -dd is the name of the application we’re going to use to take the image.
  -if=/dev/hde tells dd that the input file is the device of /dev/hde (our CF).
  -of=/dev/sdb1/nm.image tells dd that the output file is a file on our USB drive called nm.image.


This will take some time since we’re reading the CF block by block and writing it out to the USB flash drive. The resultant image will be the same size as the CF (257M in this case) since it’s copying everything – data, unused bits, partition info, etc.
Once the read of the flash is complete, you should be back at a command prompt. You can confirm that it’s there and the right size by typing:
ls -lh /mnt/sdb1


Shutdown your laptop by using:
halt


Eject the CF adapter and re-install it into your donor NM following the instructions in reverse. Once you’ve put away your good hardware, shutdown your ISR with the bad NM, remove it out of the ISR, and extract the flash out of it as described above. Insert it into your CF reader as described above, boot your laptop as described above, insert your devices (USB and CF) into the laptop as described above, and open the terminal application as described above.


Once you’re at your terminal prompt, we’re going to do the following:
Type:
sudo su


  -This puts us into super-user mode so we don’t run into any permissions issues
Then:
dmesg


  -This gives you a dump of system messages. Look for your USB device and CF device like you did before and confirm they’re there.
Now we’re going to take our image that we created above and write it out to our bad flash:
dd if=/mnt/sdb1/nm.image of=/dev/hde



This deconstructs like this:
  -dd is the name of the application we’re going to use to take the image.
  -if=/dev/sdb1/nm.image tells dd that the input file is a file on our USB drive called nm.image.
-of=/dev/hde tells dd that the output file is the device of /dev/hde (our CF).


This will take some time as well since we’re now reconstructing all of the data bits back onto our module. Once that completes, shutdown your laptop by using:
halt


Once it’s powered off, you should have a complete copy of the flash from your donor module in your hands. Reassemble your module and re-insert it back into your ISR. Power it all back on and you should be able to use:


service-module wlan-controller 1/0 session


to confirm that your module boots successfully. One of the more obvious side effects of this is that you’ll loose your NM configuration and you’ll have your donor NMs configuration now on yours. You’ll want to watch the card boot and do a clear config first off to ensure you have a good starting point. If you don’t have a donor NM to get this process done, you may want to look around to see if anyone else in your situation has the data bits from the dd process above. Once you have an extracted image, this should work on any of the like platforms regardless of where it came from.