Drag racing the bus

Picture it. You’re a school district transportation engineer. You’re in charge of purchasing a fleet of new school busses for your district. The big ones. The expensive ones. The ones that will last you for the foreseeable future. So, you call up Bus Vendor A, B, and C and inform them that you’re in the process of selecting a fleet of new school busses. The following week each vendor dutifully delivers their ‘bus of choice’ to be evaluated. You then grab your intern, put him at the midway point of the bus from ‘Vendor A’ and take it for a spin! You see how fast it goes from 0 to 60. You see how it corners. People hear tires screeching from all over the city as you and your one other occupant sling this bad boy around town ‘evaluating’ the bus. You then repeat the same process for ‘Vendor B’ and ‘Vendor C’. You aggregate your data. You correlated your data. You make pie charts about your data. You do ROI calculations on your data. You do comfort analysis on your data. You do handling analysis on your data. You made your ‘educated’ recommendation and purchased a fleet.

Day 1 of school rolls around and the first thing your brand spanking new fleet of school busses does is immediately do the one thing you neglected to test: they loaded up with kids and trudged along at 20 MPH safely around town. You start getting complaints. They don’t stop well. They don’t handle well. They don’t get good gas mileage. They bounce all over the place and your district has to send 2,000 kids to chiropractic care because you didn’t evaluate the bus under the conditions it’s going to be used in. Instead, you took it for a joy ride. You drag raced it. The one bus that went the fastest with a single guy in it, you bought. When you deployed it, it broke because you didn’t test it using real world scenarios.

Please, don’t drag race your evaluation Access Points. Test them like you’re going to operate them. That way you get a realistic view of how they’re going to behave in the real world. Do your self a favor. Stop joy riding your vendors gear and put it in the real world to test it.

This blog inspiration courtesy of @florwj . Go follow him. He’s awesome.


The FCC.

Here in the states, we have a regulatory body called the Federal Communications Commission (FCC). As it pertains to the Wi-Fi world, they tell us what channels we can use how obnoxious we can be (strength) in those channels. We have what you would consider to be a ‘blanket rule’ that basically states ‘within a given number of frequencies, you can do anything you want as long as you limit yourself to a maximum power’. A very intentional byproduct of these rules is the relatively low cost of WiFi components. Since we don’t have to submit everything we operate to the FCC for validation, we have no ‘validation costs’ to pass onto our end users. In short, the FCC, as a regulatory body imposes rules and restrictions on our use of wireless frequencies in the name of the greater good. This generally works very well, creating the ecosystem of ‘small cell’ give and take that we live in today. You are given the choice to make your own determination if analog video cameras, microwave ovens, X-Box controllers, etc should take priority or if your Wi-Fi should. Political challenges aside, we’re masters of our own domain.

So what happens when someone does something outside the norm? What happens when someone violates the FCC specifications? What happens when someone fires up 10 watt outdoor analog video feed in 2.4GHz and points it at the broad side of a hospital?

As it turns out, someone recently did just that. I was asked to assist with locating what was being detected as a whole bunch of analog video cameras that hogging up all of Channel 1 in 2.4GHz along the broad side of a hospital. As you could imagine, with a good 100 or so Access Points all excluding channel 1 (due to interference) from their channel plan, this meant that a two channel plan was all that was left (6 and 11). After much sleuthing, we determined that the signal was traveling well in excess of 10 city blocks! In my book, that certainly fit the bill for ‘obnoxious’. With more than a little hesitancy, I went to the FCC web page for complaints and filed one.

We're concerned with Wi-Fi Jammers.

We’re concerned with Wi-Fi Jammers.

Once my complaint went off into oblivion, I’ll be honest with you, I didn’t expect to hear anything from them at all. Instead, a few weeks later, I got a letter in the mail with the usual FCC ‘devices must accept interference’ text that you’d expect from a Federal entity. I was heartened by the fact that I got a response however, and there was a ‘for more information call this number’. They offered, I did. The nice Federal employee took my call, listened to my acknowledgement of the letter, listened to my insistence the letter was insufficient, and listened to my complaint that there was something going on that the FCC clearly needed to get involved in. She thanked me for my time and stated that she would escalate my case. This was the last I personally heard from them.

I was fortunate enough to have some contacts near the building that we suspected was generating the noise. Sure enough, a couple of weeks later, they informed me that a FCC field agent showed up asking questions. Shortly thereafter, the video camera signatures stopped being detected, the channel cleared up, and things got back to normal at my customer.

The point of all of this is that you do have a friend in the FCC. They’re not the most communicative, timely, or ‘feel good’ organization I’ve ever worked with, but if you have no other choice, and you can prove reasonably that there is a strong need for them to get involved with a neighbor that being obnoxious, they will. Start to finish, once I engaged the FCC, it took roughly 2 months to get back to normal. Don’t expect them to be quick. Don’t expect them to believe you. Don’t expect them to understand what you’re saying. Do be patient. Do be persistent. Do be kind. You don’t want to make a Fed angry.

Here are some good times to engage the FCC:

  • You can prove beyond the shadow of a doubt that something beyond your sphere of influence is causing harmful interference to your Wi-Fi network.

Here are some good times to not engage the FCC:

  • Your ER department won’t buy new Microwave ovens
  • Your security team is installing analog video cameras
  • Your customers are bringing gaming consoles onto your property and using them
  • Your co-worker put a 20dBi antenna on a 200mW radio outside (although, this is a good way to get them called on you!)
  • You believe you have external interference, but don’t have a Spectrum Analyzer to prove it

If you need a Spectrum Analyzer, head on over to the MetaGeek folks and check out their Wi-Spy Mini or their Wi-Spy DBx for a good cost effective way to tackle interference issues. If they get too big, rest assured that Big Brother is out there – just a complaint form and a couple phone calls away…

Cisco releases new WLC UI, Changes default values (finally)

Cisco released WLC code version which brings with it (among other things) a new User Interface for the 2504 WLC. When you use the new simplified setup, it also changes many of the default values that haven’t yet been enabled by default in the base code. The new default values are:

Aironet IE: Disabled
DHCP Address Assignment (Guest SSID): Enabled
Client Band Select: Enabled
Local HTTP and DHCP Profiling: Enabled
Guest ACL: Applied
CleanAir: Enabled
Event Driven RRM: Enabled
Event Driven RRM Sensitivity, 2.4GHz: Low
Event Driven RRM Sensitivity, 5GHz: Medium
Channel Bonding, 5GHz: Enabled
DCA Channel Width: 40MHz
mDNS Global Snooping: Enabled
Default mDNS profile: Add better printer support, Add HTTP
AVC (no Control, only Visibility): Enabled*
Management via Wireless Clients: Enabled
HTTP/HTTPS Access: Enabled
WebAuth Secure Web: Enabled
Virtual IP Address:
Multicast Address: Not configured
Mobility Domain Name: Name of employee SSID
RF Group Name: Default

*AVC stands for Application Visibility and Control. Control means remarking or blocking – for the purposes of the default setup, you’re inspecting only, Control is disabled and must be enabled manually. This also requires a current boot loader which should only be important if you’re setting up an older unit that’s been cleared.

You should note that to get these default values auto-set, you must use the new setup wizard – if you do the regular CLI setup of your controller, or if you just upgrade an existing controller without clearing it’s config, these are not set. You should also note that, for now at least, this only applies to the 2504 controller, not the 5508, WiSM2, 7510, 8510, or Virtual WLC platforms.

WPA/TKIP only going away in Cisco WLC release 8.0

Cisco is readying the next major release of their WLC code, version 8.0. At the advocation of the WFA, this will bring with it a very significant change in security capabilities that you may find impacting if you’re caught unaware. In an attempt to raise awareness, Cisco has approved an discussion of this change first mentioned here. Cisco, in accordance with the new WFA guidelines, will no longer be allowing an SSID configuration with WPA/TKIP only security. If you are currently using an SSID that has WPA/TKIP only security, your configuration will automatically be updated to enable WPA2/AES connectivity as well as WPA/TKIP. You may want to start validation testing now if you are currently supporting legacy devices on a WPA/TKIP only SSID today. The easiest way to ensure you’re not caught by this change is to enable WPA2/AES along with WPA/TKIP and check to make sure your devices still behave as expected. I have confirmed in the lab that this change will be automatic:

WPA-TKIP only configuration pre-8.0

WPA-TKIP only SSID configuration, Pre-8.0

WPA2-AES added post-update

Same SSID with WPA2/AES enabled post-update.


To summarize of the variety of allowed and disallowed potential configuration options you have available and if they’ll be supported in WLC 8.0:

WPA1-TKIP (Disallowed due to eliminating TKIP)
WPA1-AES (Allowed by Extension Policy)
WPA1-TKIP/AES (Disallowed since not used in conjunction with WPA2-AES)
WPA2-TKIP (Disallowed due to eliminating TKIP)
WPA2-AES (Certified and allowed)
WPA2-TKIP/AES (Disallowed due to WPA2-TKIP)
WPA1-TKIP + WPA2-TKIP (Disallowed – no AES support)
WPA1-TKIP + WPA2-AES (Certified and allowed)
WPA1-TKIP + WPA2-TKIP/AES (Disallowed due to WPA2-TKIP)
WPA1-AES + WPA2-TKIP (Disallowed due to WPA2-TKIP)
WPA1-AES + WPA2-AES (Allowed by Extension Policy)
WPA1-AES + WPA2-TKIP/AES (Disallowed due to WPA2-TKIP)
WPA1-TKIP/AES + WPA2-TKIP (Disallowed due to WPA2-TKIP)
WPA1-TKIP/AES + WPA2-AES (Allowed by Extension Policy)
WPA1-TKIP/AES + WPA2-TKIP/AES (Disallowed due to WPA2-TKIP)

Other SSIDs and security configurations are not impacted, including Open SSIDs, any SSID that currently has AES enabled, and WEP SSIDs.

Aruba forces customers into Cloud: Offers no way out

Authors note: Aruba has addressed this satisfactorily by removing the offending release of code. Further details are available in the comments section of the post below. This header was added as a post-script to the original blog which remains below for posterity. Well done Aruba.

Software updates are a matter of life. Developers and coders aren’t perfect, bugs need to be fixed, new features need to be rolled out, new hardware needs to be supported. As networking professionals, we’ve come to terms with the sometimes continual churn that patch vehicles have enabled – the Internet as a distribution mechanism has made it commonplace for manufactures to ship incomplete features or functionality under the guise of ‘by the time you complain about it, we’ll have a download ready to fix it”. They all do it. We all consume it.

Sometimes a manufacturer does something so egregious and underhandedly reprehensible that it simply cannot be. Aruba release their ArubaOS on Feb 15th, 2014 and snuck in a ‘phone home’ feature. They state twice that they’re enabling this feature by default:

Once buried on their dashboard if you happen to navigate to their code by clicking on the words in their tree, not the ‘+’.

You'll miss the warning if you use the boxes!

You’ll miss the warning if you use the boxes!

The second is in their release notes. I applaud their efforts to make their customers aware that they will be changing the default values in their release notes, but the download page navigation needs some help. Their release notes clearly state:

The PhoneHome Automatic Reporting feature is enabled by default. PhoneHome feature can be disabled by selecting the Disable option under Maintenance > Aruba TAC Server section of the WebUI.

What they do not tell you is that even if you have explicitly disabled this option previously, when you update your code to the afflicted build, it automatically re-enables this feature without warning you! Aruba intentionally snuck in an information gathering option and mislead their customers by stating that it was only enabling the ‘feature’ by default. In common english, this means when you’ve accepted the default settings – for example on a new build or a controller default, this option will be enabled. What they did not tell you is that there is no ‘opt-out’ way of having this turned on during a code update! They certainly give you guidance on how to disable it post-installation, but by that point in time, it can safely be assumed that your controller has phoned home and divulged all of your enterprise wireless configurations – information that you may not want to send to them – such as your PSKs, your RADIUS shared secrets, your default local administrator credentials – all without warning their unsuspecting, and all-too trusting customers, that Aruba was going to be maintaining a list of their most secret enterprise data.

Aruba is secretly compiling data from all of their existing enterprise customers regardless of any opt-out that was previously set.

What if your company had a security policy in place that stated that no configurations were to be shared outside of your organization? What if you had no Cloud strategy? How does having Aruba keep a copy of your configuration expose your organization to compliance issues? How will that impact FIPS compliance? Seemingly no one at Aruba cares about these issues and as far as I’m aware, there are only two ways to prevent this from occurring – and both need to be planned for before you update your software:

  1. blackhole or ACL off your controllers from talking to the Aruba cloud until you can log into the controller.
  2. disconnect your controller from the network during your update and console into it to undo the command.

Either way is intrusive and difficult to manage and if you were not aware of this data-heist prior to your update, you have no way of confirming if your data has already been compromised. In my opinion Aruba needs to do damage control here real-quick like and:

  1. Pull the malicious and infected code off of their downloads section immediately.
  2. Apologize for negligently harvesting customer data.
  3. Clearly articulate what customer data was stolen and how it was stored.
  4. Clearly articulate how long once a controller has come online that it ‘rats your configs out’ to the manufacturer.
  5. Give customers the option to have their data removed from Arubas servers and have a third party audit this removal for confirmation.
  6. Modify the code to obey previous opt-out configurations and re-post it.

Until then, you may want to let your SOC, CTO, IT Director, or resident person in charge that if you did the code update from Aruba that you should consider your network keys, shared secrets, and local admin accounts compromised and start working on a remediation plan immediately.

MythBusters WiFi: Xirrus

I’ll be the first to admit that when I see something ‘out of the norm’ I shudder and have a knee-jerk reaction that is not always positive. There is so much success around the tried and true enterprise approach to wireless of using omni directional antennas that when you see someone intentionally deviate from it, it can be jarring to say the least! I’ve had the pleasure to be present at the Xirrus Wireless Field Day sessions for WFD5 and WFD6 and I can honestly say that they did a superb job of taking a contentious topic and addressing it head on. For those that are unfamiliar with the Xirrus product, their unique approach to wireless is to stack multiple Access Point radios into a single housing and use highly directional antennas to create an ‘Umbrella Corporation logo’ of coverage:

Image likely copyrighted by Capcom

(it should be noted that I do not believe that Xirrus is somehow evil, involved in genetic engineering, or otherwise a bad company – the logo is simply an easy way to represent a high degree of directionality from a centralized point)

Adding multiple radios into a single housing is not a unique approach. Most everyone on the market starts with two radios in their Access Points and you can even find a handful of three-radio solutions, but Xirrus takes this approach to the next level by stacking as many as 16 radios into a single Access Point! The challenge that other manufacturers do not need to address is overlap. In a standard two or three-radio approach, you typically operate one radio in 2.4GHz and the other in 5GHz (and for those rare 3rd radio guys, they usually stick that radio in monitor or listen-only mode). Tack some omni directional antennas on those bad boys and you’ve got yourself an AP! Xirrus however, intends to have multiple radios in a single AP operate on in the same frequency. This presents some challenges about the efficiency you can gain from having more than one radio in the same frequency in the same physical package (AP). At WFD5, there was much gnashing of teeth regarding how you accomplish this in one package. I’ll not go over the gory details, the video is posted here. Xirrus came back to WFD6 and brought with them their Director of RF Engineering, Avi Hartenstein and tacked the conversation head on.

My goal was simple with this post. I wanted to prove that directionality does exist in the Xirrus product once and for all. I was able to acquire a 4 radio array and decided that the best approach to visualize this purported phenomenon was to actually survey with an AP in an area with no obstructions (scientific, no?). I took the array, statically assigned one of the radios to a fixed channel, turned its power down to -15dBm (yes, they go *that* low!), and took it outside. The results speak for themselves:

 Umbrella logo likely copyrighted by Capcom

Ladies and gentlemen, you saw it here first (or maybe not) – the myth of the Xirrus wedge is real! You can see near the bottom of the image where the AP was placed (at random orientation). I peeled this wedge view out using Ekahau’s Site Survey application after quite a walk outside in the cold. This directionality is fairly easy to see even with my coarse sampling outside. It should be noted that it will be near impossible to visualize this dramatic of a wedge indoors however due to the prevalence of those pesky attenuators otherwise known as walls and furniture.

I’ve seen my fair share of wireless deployments, I know what I’m comfortable with, I know when I move outside of my comfort zone. An experience I reference often is an educational facility that was using directional patch antennas indoors on 10 ft ceilings pointed at the ground. While this is a startling design, when I dug deeper into their design methodology, I discovered that they surveyed using the exact model of antenna and AP that was deployed, correctly visualized the resultant zone of coverage, and validated that this met their applications need. While not a solution I would lead with, there was no fault with their design methodology or implementation, and the infrastructure operated as designed. When you tackle a Xirrus deployment, I would advocate the same approach: understand your needs (throughput, density, coverage, etc), and design to meet those needs using the gear you will be deploying. Survey what you deploy, deploy what you survey. In the Xirrus world, this presents a few design choices to consider:

1) Orientation of the AP.

The Xirrus array has a compass in it. Use this compass to determine the orientation of your array during the survey and ensure that when it gets deployed that this lines up correctly (use the logo on the housing if you need to).

2) Oversubscription.

You must pay close consideration to the number of uplinks to your array and balance this with your deployment expectations. Oversubscription is nothing new so don’t let this scare you – just be aware that you’re moving your uplinks (and potential bottlenecks) further up the line (closer to the AP). This is going to be particularly important as you consider updating your array to newer technologies such as 802.11ac.

3) Powering the Array

Ensure that you have made concessions for powering the array. This will likely require an external power injector but sourcing them along with the array should not be problematic.

4) Antennas change with the modules.

One of the most insightful things I learned from the WFD sessions are a reinforcement that the antenna is part of the radio module. When you replace that module, you replace the antennas that are a part of that module. This could potentially impact the RF of your deployment and will most assuredly change the visualization of your survey data.

Xirrus uses highly directional antennas in a unique way to extend the reach of a radio. This coupled with a low powered radio gives you a number of excellent design pieces for most any wifi environment or need. Pay close attention to the number of radios that you need, apply some logic and reason to your design (don’t expect 8 of your 16 radios to operate in 2.4GHz for example) and make sure your celling has sufficient mount points. The arrays can be weighty. :)

Meraki: The bolt on Cloud that wasn’t

When Cisco acquired Meraki last year, there was much confusion. Being ‘down in the trenches’ I struggled as much as the next guy trying to wrap my head around the acquisition and I believe I have a good handle on it. Others not so much. I regularly consult with customers that are just as confused today as they were last year. Cloud is such an over used buzz word and so many vendors are trying to jump on the buzzword bandwagon de jour that it’s easy to get lost admist the jargon and solutions, much less the technical merits or differences in the platforms. I’m here to offer some advice on the strategy and perhaps a perspective on the acquisition that you haven’t yet considered. First some advice:

Don’t purchase Meraki Access Points. You read that right. Don’t do it. Also, don’t purchase Meraki switches. For that matter, don’t buy the Meraki firewall either. If you purchase a Meraki Access Point, a Meraki switch, or a Meraki firewall, you’re not buying an Access Point, you’re not buying a switch, you’re not buying a firewall. You’re buying ‘The Cloud’. When you consider purchasing infrastructure equipment that is ‘Cloud Enabled’, this should be a purchase that lines up with your organizations Cloud Strategy first and foremost. Don’t have a Cloud Strategy? Don’t be so sure. There are a few questions to ask yourself before you jump to that conclusion. Does your organization use DropBox? Salesforce.com? Office 365? Webex or Goto Meeting? Google Mail? All of these are examples of Cloud Applications. If you use these, someone, somewhere in your organization has made the determination to embrace services from ‘The Cloud’. Understand this strategy. Understand what this enables. Understand what this means to your data and where your data lives. Then (and only then) should you consider purchasing ‘Cloud Managed Infrastructure devices’.

Let’s be frank about it, there’s nothing special about the hardware in a Meraki Access Point. There’s nothing special about the hardware in a Meraki Switch, nothing special about the hardware in a Meraki firewall. When you purchase Meraki equipment, this gear is purpose built to be Cloud Managed with features driven by that Cloud Management. When you make a Meraki purchase, purchase an end-to-end Cloud-enabled infrastructure. If it’s right for one component, it’s right for all of them. If it’s not right for all of them, it’s not right for any of them.

Now some perspective. Everyone is talking about Cloud. Everyone wants in on the Cloud action. Everyone is ‘bolting on’ Cloud to their existing products in some fashion or another. When Cisco purchased Meraki, they made a decision to not ‘bolt on’. They decided to pick the one organization that understood Cloud from bottom to top and embrace that strategy despite the fact that there was some hardware overlap. The Meraki acquisition wasn’t about Access Points, switches, or firewalls. It was about finding the one organization that was never built for ‘on premises’ management and this shines through in every aspect of their products. Others tout ‘free protocols’, ‘cloud provisioning’, or a variety of other nonsense but at the end of the day, these are bolt-on solutions that are all afterthoughts. I would encourage you to revisit the Meraki product portfolio but when you do, ask yourself the following questions:

  • What are my existing Cloud Applications?
  • How do I rely on ‘the Cloud’ today?
  • Do I want to leverage that existing strategy in my infrastructure?
  • Do I want a solution that is built from the ground up around ‘the Cloud’ with a no-compromises featureset or do I want to deal with someone bolting on features to their existing ‘heavy gear’?

Then go buy a Meraki AP.


Get every new post delivered to your Inbox.

Join 38 other followers