Aruba forces customers into Cloud: Offers no way out
February 18, 2014 6 Comments
Authors note: Aruba has addressed this satisfactorily by removing the offending release of code. Further details are available in the comments section of the post below. This header was added as a post-script to the original blog which remains below for posterity. Well done Aruba.
Software updates are a matter of life. Developers and coders aren’t perfect, bugs need to be fixed, new features need to be rolled out, new hardware needs to be supported. As networking professionals, we’ve come to terms with the sometimes continual churn that patch vehicles have enabled – the Internet as a distribution mechanism has made it commonplace for manufactures to ship incomplete features or functionality under the guise of ‘by the time you complain about it, we’ll have a download ready to fix it”. They all do it. We all consume it.
Sometimes a manufacturer does something so egregious and underhandedly reprehensible that it simply cannot be. Aruba release their 6.4.0.0 ArubaOS on Feb 15th, 2014 and snuck in a ‘phone home’ feature. They state twice that they’re enabling this feature by default:
Once buried on their dashboard if you happen to navigate to their code by clicking on the words in their tree, not the ‘+’.
You’ll miss the warning if you use the boxes!
The second is in their release notes. I applaud their efforts to make their customers aware that they will be changing the default values in their release notes, but the download page navigation needs some help. Their release notes clearly state:
The PhoneHome Automatic Reporting feature is enabled by default. PhoneHome feature can be disabled by selecting the Disable option under Maintenance > Aruba TAC Server section of the WebUI.
What they do not tell you is that even if you have explicitly disabled this option previously, when you update your code to the afflicted 6.4.0.0 build, it automatically re-enables this feature without warning you! Aruba intentionally snuck in an information gathering option and mislead their customers by stating that it was only enabling the ‘feature’ by default. In common english, this means when you’ve accepted the default settings – for example on a new build or a controller default, this option will be enabled. What they did not tell you is that there is no ‘opt-out’ way of having this turned on during a code update! They certainly give you guidance on how to disable it post-installation, but by that point in time, it can safely be assumed that your controller has phoned home and divulged all of your enterprise wireless configurations – information that you may not want to send to them – such as your PSKs, your RADIUS shared secrets, your default local administrator credentials – all without warning their unsuspecting, and all-too trusting customers, that Aruba was going to be maintaining a list of their most secret enterprise data.
Aruba is secretly compiling data from all of their existing enterprise customers regardless of any opt-out that was previously set.
What if your company had a security policy in place that stated that no configurations were to be shared outside of your organization? What if you had no Cloud strategy? How does having Aruba keep a copy of your configuration expose your organization to compliance issues? How will that impact FIPS compliance? Seemingly no one at Aruba cares about these issues and as far as I’m aware, there are only two ways to prevent this from occurring – and both need to be planned for before you update your software:
- blackhole or ACL off your controllers from talking to the Aruba cloud until you can log into the controller.
- disconnect your controller from the network during your update and console into it to undo the command.
Either way is intrusive and difficult to manage and if you were not aware of this data-heist prior to your update, you have no way of confirming if your data has already been compromised. In my opinion Aruba needs to do damage control here real-quick like and:
- Pull the malicious and infected code off of their downloads section immediately.
- Apologize for negligently harvesting customer data.
- Clearly articulate what customer data was stolen and how it was stored.
- Clearly articulate how long once a controller has come online that it ‘rats your configs out’ to the manufacturer.
- Give customers the option to have their data removed from Arubas servers and have a third party audit this removal for confirmation.
- Modify the code to obey previous opt-out configurations and re-post it.
Until then, you may want to let your SOC, CTO, IT Director, or resident person in charge that if you did the 6.4.0.0 code update from Aruba that you should consider your network keys, shared secrets, and local admin accounts compromised and start working on a remediation plan immediately.
SC-WiFi 
To make accusations, especially if you’re biased against the manufacturer in question, is in fact reprehensible.
I appreciate the effort you put into your 16 word ‘shot across the bow’. Unfortunately, I found it lacking in substance and would like to address it as best as I believe I can:
1) Everyone has biases – this does not mean they should not have an opinion nor does it mean they should not express their opinion. This is especially poignant when security and privacy is involved. If I was not a Target customer (or if I only shopped at Wal-Mart) does this mean that my opinion on their security policies is somehow invalidated?
2) I like Aruba. They do an excellent job of creating a reasonable product out of commodity hardware. Their new 802.11ac outdoor AP is ahead of the pack, their acquisitions in AirWave and Amigopod were well thought out and I would certainly classify as a strong competitor. Even if I did not like them, would this somehow invalidate my opinion of their security or privacy policies?
Please take the time to read the article, formulate an actual response, and engage in a well thought out conversation about security, full disclosure, and warnings about modifying their users existing opt-out configurations without their express consent. Until then, I suppose I can stomach being a pin cushion for the sake of your quips.
-Sam
Well done Aruba.
I’m pleased to announce that the described release of code (6.4.0.0) has been pulled from distribution and a .1 release is being worked on to turn this feature back off by default. After posting some questions on the Aruba AirHeads Community, Chief AirHead Sean Rynerson took the time to raise awareness of this internally and address several outstanding concerns I had (copied at the bottom of this post for your perusal). I would love the opportunity to further address the crux of this problem, which I believe to be poor verbiage in the Release Notes, and am working with Sean and team to bring clarity to this kind of situation moving forward.
-Sam
How do I prevent a controller from phoning home at first boot?
Aruba Networks >>> It takes 1 week for the first “phone home” to occur after AOS 6.4 upgrade. The phone home process kicks in on a weekly basis after that.
Customers can disable phone by
a) Through WebUI, select PhoneHome Disable under Maintenance > Aruba TAC Server.
b) To disable via the CLI, type “phonehome disable”
What specifically is stored on Aruba servers?
Aruba Networks >>> Primarily the Config information – the feature on/off, Config values, SW version running on the controller, critical controller performance parameters etc. The primary use for this data is to understand what versions of the code are most popular and are in production, which features are most popular. This will enable us to proactively assess the impact of features, product enhancements, product life cycle decisions.
Does that data include my PSK, shared secrets, or local admin accounts in any form (encrypted or otherwise)?
Aruba Networks >>> All of the sensitive fields are encrypted/hashed and we do not process any of the PSKs, shared secrets or any other information that is sensitive from a customer perspective.
How is the data transfered? (wrapped in SSL I presume)
Aruba Networks >>> It used to be in the form of files attached to public key based encrypted e-mail. We have changed to HTTPS mode of communication in AOS 6.4.
How is that data secured on Aruba servers?
Aruba Networks >>> The data is secured just like that of any cloud service provider, secure data centers with adequate physical security and appropriate password and access control protections.
Who has access to that data?
Aruba Networks >>> A small subset of Aruba engg, product management, TAC team members have access to processed “reports” based on raw phone home data. Our TAC team if need be can access the raw data on a per customer basis as part of their customer support, TAC ticket resolution process.
If I did not intend to send you my data, how do I get Aruba to remove it from their database?
Aruba Networks >>> Prior to AOS 6.4 data belonging to customers that have voluntarily enabled the phone home feature and/or voluntarily sent us logs for TAC ticket resolution are stored on our servers. Any customer can contact TAC and ask that their data be deleted.
support@arubaneworks.com – http://www.arubanetworks.com/support-services/support-program/contact-support/
How do I ensure that the removal from the database is complaint with my internal data security policies (vetted by a trusted 3rd party)?
Aruba Networks >>> We can work with customers that are concerned on a case by case basis to review their company specific policies if any and ensure compliance & satisfaction.
If the data becomes compromised, how are you going to alert me that my PSKs, SNMP strings, shared secrets, or local admin accounts have been compromised?
Aruba Networks >>> To this date there has been no compromise of this data. We will take any/all reasonable steps to inform customers that have opened TAC tickets and have voluntarily uploaded Config, log files for troubleshooting purposes. Prior to AOS 6.4 the only way for Aruba to gain access to customer phone home data is either the customer sharing the data for TAC ticket resolution or enabling phone home voluntarily. Considering we have pulled the AOS 6.4 build from our support site after hearing some concerns and are posting AOS 6.4.0.1 we will only be collecting information if customers voluntarily “opt in” per the model that existed prior to 6.4.
Sam, Well done.
Awesome level of detail. This blog post made me pull over to a rest stop along the highway home to read after getting the alert from Siri.
And on a side note to the Wifi guys comment. He obviously doesn’t know you well enough when it comes to being biased on a vendor.
WirelessStew
Cheers mate! Thanks for the kind words. 🙂
Reblogged this on @WirelessStew and commented:
Awesome work by my friend Sam. Its detail work like this that makes us Wifi folks make a difference in the community