Resurrecting a bricked NM-AIR-WLC6

Cisco recently posted this addendum to their Software Downloads section for the Cisco Wireless LAN Controller Module:

Warning: the Wireless LAN Controller Network Module (NM-AIR-WLC6-K9) is not supported in any software release after Attempting to install 5.0 or later software can permanently damage the module.

This is a pretty recent addition and appears to have been an oversight the past year or so while they’ve been happily releasing version after version of NMWLC code without this disclaimer. If you’re like me, you’ve been keeping up on your latest and greatest software releases and you may find yourself in some murky waters if you happen to have this module. Where I landed was a module that would boot fine, but would not establish any network connectivity (management, AP join, etc). You should note that this article in it’s entirety does not apply to the NME module, just the NM. The NME module has more memory and a 1G internal interface to the ISR, the regular old NM has less memory and only a 10/100 interface. You can tell which module you have by looking at the silkscreen on the back of the module or by doing a ‘show sysinfo’ at the CLI of the controller.

This article is not supported by Cisco, TAC, or myself. You may further damage your NM if you proceed. This article is not for the faint of heart and will most certainly void any warranty you may have. If you have a bricked NM under SmartNET, you should contact TAC for a replacement unit, not follow the directions in this post. I do not guarantee any work here and you can severely damage your module, it’s flash, or your PC. Read and follow this article at your own risk!

Now thats out of the way, the specifics of my problem landed me in a situation where I could not roll back the version of code on my flash (not having network connectivity really limits you, I gotta admit). You may find yourself with a corrupt flash, unable to boot, or other general mayhem. Once Cisco released this notification, it dawned on me that the version of code on my flash was likely the culprit. Since the NMs are basically an Pentium III with some memory and a flash to boot off (similar to the CUE or Content Engine modules) running Linux, I figured I should be able to copy the flash from a good NM and I’d be back in business. Having located a donor NM (thanks to Robert B. for his support here), I assembled  the following items to move on:

  • Donor working NM to rob/copy the flash off of
  • A small screwdriver
  • Old laptop with Cardbus/PCMCIA slot
  • CF to PCMCIA adapter (like one of these)
  • USB Flash drive larger than 256M formatted something that Linux can write to
  • A Linux distribution that I could boot off of CD like DSL
  • A static bag to work off of
Getting a donor module
The first thing I did was to remove the CF module from the donor NM.
Step 1) Place the donor NM on a static safe work place. The bag it came in would be good.
Step 2) Confirm that the module you’re working on is an NM, not an NME.
Step 3) Locate the cover that hides the flash module.
Extracting the flash
You must then remove the protective cap around the flash module.
Step 1) Unscrew the CF housing.
Step 2) Lift up gently on the right edge of the cap and it should fall off the module.
Remove the flash from the NM
Gently grasp the Cisco flash module by both edges and pull it directly out of the NM.
Insert the flash module into your CF reader.
This should be pretty straightforward.
Once you have the flash module in a CF reader, we’re going to be focused on getting a good block image off of it. The rest of this article will discuss how to take an image of the flash module and store it on a USB flash drive. Once you have your favorite LiveCD of Linux (or BSD if you prefer) downloaded, boot your old laptop off of it. We chose a LiveCD release so that we can do this on a laptop without having to do a fully blown installation of Linux just for this one project. Feel free to use any sort of Linux box you happen to have laying around. 🙂
Once you’ve successfully booted Linux, you’ll need to open a terminal window. In DSL, there is a link to the Terminal app in the bottom left corner. Attach your USB drive and insert your PCMCIA flash reader once your system is booted and your terminal is up.
sudo su

  -This puts us into super-user mode so we don’t run into any permissions issues

  -This gives you a dump of system messages. In particular we’re looking for two things. The USB drive and the CF adapter. In my system, this looked like:
<6>hub.c: new USB device 00:1d.1-1, assigned address 3
<6>scsi2 : SCSI emulation for USB Mass Storage devices
<4>  Vendor: USB 2.0   Model: FLASH DISK        Rev: 1.0
<4>  Type:   Direct-Access                      ANSI SCSI revision: 02
<4>Attached scsi removable disk sdb at scsi2, channel 0, id 0, lun 0
<4>SCSI device sdb: 2033664 512-byte hdwr sectors (1041 MB)
<4>sdb: Write Protect is off
<6> sdb: sdb1
This tells us that our flash drive is at /dev/sdb1 (the last line above) so let’s mount it using:
mount /dev/sdb1 /mnt
Next we look for our flash reader. In my system, this looked like:
<6>cs: memory probe 0xa0000000-0xa0ffffff: excluding 0xa0000000-0xa0ffffff
<6>cs: memory probe 0x60000000-0x60ffffff: clean.
<4>hde: STI Flash 8.0.0, ATA DISK drive
<4>ide2 at 0x100-0x107,0x10e on irq 11
<4>hde: attached ide-disk driver.
<6>hde: 501760 sectors (257 MB), CHS=980/16/32
<6> hde: hde1 hde2 hde3
<6>ide_cs: hde: Vcc = 3.3, Vpp = 0.0
For this one, we’re not interested in any partition information like we were on the USB device, we’re just interested in the device name. Here, we see that this device is hde (the beginning of the third line) . Once we have both the USB drive mounted and the flash drive identified, we’re going to use dd to take a block image of the device by typing:

dd if=/dev/hde of=/mnt/sdb1/nm.image
This deconstructs like this:
  -dd is the name of the application we’re going to use to take the image.
  -if=/dev/hde tells dd that the input file is the device of /dev/hde (our CF).
  -of=/dev/sdb1/nm.image tells dd that the output file is a file on our USB drive called nm.image.

This will take some time since we’re reading the CF block by block and writing it out to the USB flash drive. The resultant image will be the same size as the CF (257M in this case) since it’s copying everything – data, unused bits, partition info, etc.
Once the read of the flash is complete, you should be back at a command prompt. You can confirm that it’s there and the right size by typing:
ls -lh /mnt/sdb1

Shutdown your laptop by using:

Eject the CF adapter and re-install it into your donor NM following the instructions in reverse. Once you’ve put away your good hardware, shutdown your ISR with the bad NM, remove it out of the ISR, and extract the flash out of it as described above. Insert it into your CF reader as described above, boot your laptop as described above, insert your devices (USB and CF) into the laptop as described above, and open the terminal application as described above.

Once you’re at your terminal prompt, we’re going to do the following:
sudo su

  -This puts us into super-user mode so we don’t run into any permissions issues

  -This gives you a dump of system messages. Look for your USB device and CF device like you did before and confirm they’re there.
Now we’re going to take our image that we created above and write it out to our bad flash:
dd if=/mnt/sdb1/nm.image of=/dev/hde

This deconstructs like this:
  -dd is the name of the application we’re going to use to take the image.
  -if=/dev/sdb1/nm.image tells dd that the input file is a file on our USB drive called nm.image.
-of=/dev/hde tells dd that the output file is the device of /dev/hde (our CF).

This will take some time as well since we’re now reconstructing all of the data bits back onto our module. Once that completes, shutdown your laptop by using:

Once it’s powered off, you should have a complete copy of the flash from your donor module in your hands. Reassemble your module and re-insert it back into your ISR. Power it all back on and you should be able to use:

service-module wlan-controller 1/0 session

to confirm that your module boots successfully. One of the more obvious side effects of this is that you’ll loose your NM configuration and you’ll have your donor NMs configuration now on yours. You’ll want to watch the card boot and do a clear config first off to ensure you have a good starting point. If you don’t have a donor NM to get this process done, you may want to look around to see if anyone else in your situation has the data bits from the dd process above. Once you have an extracted image, this should work on any of the like platforms regardless of where it came from.

6 Responses to Resurrecting a bricked NM-AIR-WLC6

  1. Pete says:

    what files are on the flash card besides the .aes file?

    • scwifi says:

      All of the filesystem files – just like a regular old linux box. The kernel, configuration files, etc. Think of the .aes file as a .tar file that is encrypted.

  2. Ilya says:

    My cisco controller NME-AIR-WLC6-K9 unable to boot too.
    As I can see something happen with my cf card.
    I would appreciate if you can send me flash image.
    So I could try to boot my controller too.

    • scwifi says:

      Im sorry, but my image is a) not for the NME unit (although the process should be identical) and b) illegal for me to share.

  3. Kryil says:

    Thanks a lot! The procedure also saved a bricked WLC 2106. I borrowed a working flash from another 2106 and copied it to a new CF card. Worked like a charm!

  4. Pingback: Recovering bricked Cisco WLC 4400 « Network Experts Blog

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: