To accomplish the mammoth task of informing about 50 different vendors and various ISPs we teamed up with CERT/CC (VU#566724). We would really like to report that our efforts were successful, but as it turns out the number of devices on the web using known private keys for HTTPS server certificates has gone up by 40% in the last nine months (3.2 million in November 2015 vs. 4.5 million now). There are many explanations for this development. The inability of vendors to provide patches for security vulnerabilities including but not limited to legacy/EoL products might be a significant factor, but even when patches are available, embedded systems are rarely patched. Insufficient firewalling of devices on the WAN side (by users, but also ISPs in case of ISP-supplied customer premises equipment, CPE) and the trend of IoT-enabled products are surely a factor as well.
In our initial study we analyzed SSH host key use as well. Unfortunately there is no recent scan data on SSH host keys available (however there is a ticket over at the awesome ZMap project).
In the spirit of open research we are releasing the raw data today on Github. The data we are publishing consists of 331 certificates including the matching private key as well as 553 individual private keys. We’ve also included the names of products that contain the certificates/keys. Cryptographic keys that were not found in Internet-wide scan data (Scans.io and Censys.io, HTTPS/SSH) are included as well. These might be used in other protocols such as EAP/802.1X, FTPS etc. The data we are publishing allows researchers to reproduce the results of our study, find more cases or cryptographic key reuse, attribute cryptographic keys to specific vendors/products, but also to develop tools for detecting and exploiting this vulnerability class in the course of penetration tests. Releasing the private keys is not something we take lightly as it allows global adversaries to exploit this vulnerability class on a large scale. However we think that any determined attacker can repeat our research and get the private keys from publicly available firmware with ease.
Some highlights from the data set were already mentioned in our initial blog post last year, including our beloved Broadcom SDK “Daniel” certificate, that is still used by 500.000 devices, or the Multitech/Texas Instruments certificate that is used by 280.000 devices on the web.
Our upcoming IoT/CPE firmware security analysis solution for device vendors and ISPs detects cryptographic key reuse vulnerabilities.
Ubiquiti Networks
Last year we published a blog post titled The Omnipresence of Ubiquiti Networks Devices on the Public Web in which we discussed why so many Ubiquiti Networks devices are on the public web. In this case we can report some improvement. The number of devices on the web has gone down by 62% (1.1 million in November 2015 vs. 410.000 now). The bad news is that the major drop is likely caused by various botnets that exploit weak credentials as well as critical vulnerabilities including an innocently titled “Arbritrary file Upload” vulnerability (straightforward remote code execution, Metasploit module available). These botnets might have forced Ubiquiti customers to firewall their devices. Symantec has released a report on one of the malware families.
The Ubiquiti support forums are filled with people who struggle to remove malware from customers’ devices (1, 2, 3, 4, …). Ubiquiti has even started to include functionality to remove malware in their firmware updates and released a standalone tool called “CureMalware” that promises removal of the botnets “Skynet, PimPamPum and ExploitIM”. The tool comes with the helpful instructions: “NOTE: If you cannot access the devices, please try the username: moth3r and password: fuck.3r or moth3r/fucker with CureMalware.”. What a mess… and Ubiquiti still has remote management on the WAN port enabled by default.
Aruba / Alcatel-Lucent
What we did not mention publicly back in 2015 is a curious certificate/private key pair we found in various Alcatel-Lucent OmniAccess firmware. Contrary to most other certificates we’ve found, the certificate is signed by a browser-trusted CA (GeoTrust), is issued to “securelogin.arubanetworks.com” and valid until August 2017. Turns out that Aruba Networks is the OEM for the Alcatel-Lucent OmniAccess product line. The certificate is part of ArubaOS and the certificate is used in various Aruba Networks products as well. 49.000 devices on the web are using this certificate.
The certificate is not only the default HTTPS server certificate (captive portal and web administration) but also for WPA2-Enterprise 802.1X authentication. This allows attackers to do all kinds of nasty MITM attacks (active/passive HTTPS decryption, rogue access points, etc.). We informed Aruba back in May 2015. They will not provide a new browser-trusted default certificate in the future but move to device specific self-signed certificates instead. However they have decided against revoking the (still valid) current certificate. Aruba’s “security guy” Jon Green has published particularly quotable post on the issue: “In the past we were persuaded by the “but certificates are too complicated – just leave the factory default cert as-is and customers who care about security can update it” argument, but I now think we’re doing a disservice to customers by giving them too much rope with which to hang themselves.”
Update 2016-09-08:
It seems the Aruba certificate with serial number 0x1da52 has been revoked, see GeoTrust CRL.
Further information can be found in our advisory.
What can be done?
Our recommendations have not changed:
Vendors should make sure that each device uses random, unique cryptographic keys. These can be computed in the factory or on first boot. In the case of CPE devices, both the ISP and the vendor have to work together to provide fixed firmware for affected devices.
Furthermore ISPs have to make sure remote access via the WAN port to CPEs is not possible. In case the ISP needs access for remote support purposes, setting up a dedicated management VLAN with strict ACLs (no CPE to CPE communication) is recommended.
End users should change the SSH host keys and X.509 certificates to device-specific ones. This is not always possible as some products do not allow this configuration to be changed or users do not have permissions to do it (frequent in CPE devices). The required technical steps (generating a certificate or RSA/DSA key pair etc.) are not something that can be expected of a regular home user.
This study was conducted by Stefan Viehböck, Senior Security Consultant at SEC Consult.
