House Of Keys: Industry-wide Https Certificate And SSH Key Reuse Endangers Millions Of Devices Worldwide

How Do Static Keys Come Into Existence?

They have been embedded, essentially “baked in” the firmware image (operating system) of devices and are mostly used for providing HTTPS and SSH access to the device. This is a problem because all devices that use the firmware use the exact same keys. The impact is described later in this post. The keys are not a result of bad randomness (e.g. Debian weak keys, PRNG boot gap, etc.).

The source of the keys is an interesting aspect. Some keys are only found in one product or several products in the same product line. In other cases we found the same keys in products from various different vendors. The reasons vary from shared/leaked/stolen code, white-label devices produced by different vendors (OEM, ODM products) to hardware/chipset/SoC vendor software development kits (SDKs) or board support packages firmware is based on. Here are some examples:

• A certificate issued to a “Daniel”, email (kiding@broadcom.com) is used in firmware from Actiontec, Aztech, Comtrend, Innatech, Linksys, Smart RG, Zhone and ZyXEL. This certificate is found in a Broadcom SDK. The affected vendors used it as a basis to develop their own firmware. More than 480.000 devices on the web are using this single certificate.
• A certificate issued to Multitech in Bangalore, India is used in firmware from Aztech, Bewan, Observa Telecom, NetComm Wireless, Zhone, ZTE and ZyXEL. This certificate can be attributed to a Texas Instruments SDK for ADSL2+ routers. Over 300.000 devices on the web are using this certificate. A SSH host key can be attributed to this SDK as well.
• A certificate issued to “MatrixSSL Sample Server Cert” is used in WiMAX gateways from Green Packet, Huawei, Seowon Intech, ZTE and ZyXEL. All affected devices use the same code base, which is likely developed by ZyXEL. At least 80.000 devices on the web are using this certificate.

Why Are So Many Devices Exposed To The Web?

Another aspect to the whole story is the large number of devices directly accessible on the web. Just by looking at the numbers one can deduce that it is highly unlikely that each device is intentionally exposed on the web (remote management via HTTPS/SSH from WAN IP). Enabling remote management exposes an additional attack surface and enables attackers to exploit vulnerabilities in the device firmware as well as weak credentials set by the user.

In some cases this behaviour can be attributed to a vendor’s insecure default configuration. An example is Ubiquiti Networks, who have remote management enabled by default in most products. This issue is documented in a separate blog post. We have also noticed that there is a surprisingly high number of Seagate devices on the web. Around 80.000 Seagate GoFlex home NAS devices are exposing HTTPS and SSH. The Seagate Share feature sets up port forwarding via UPnP. This feature is most likely responsible for a lot of the exposed devices and it is unclear if users are even aware of this problem.

In other cases ISPs expose their subscriber’s devices (CPE – customer premises equipment) to the Internet. By correlating the affected hosts with GeoIP information, we found large clusters of devices with the same keys located in the networks of different ISPs. We can deduce that devices are CPEs provided to subscribers. These devices are owned, distributed and managed by ISPs and use ISP-specific firmware. Some ISPs have a particularly bad track record when it comes to exposing remote management interfaces:

• US-based ISP CenturyLink exposes HTTPS remote administration on more than half a million devices, which is close to 10 percent of their total subscribers (6.1 million). Affected products include ZyXEL’s Q1000, C1000Z, Actiontec’s GT784WN, C2000A, C1000A, V1000H, and Technicolor’s C2000T and C2100T.
• TELMEX in Mexico exposes HTTPS remote administration on more than one million devices (22 million subscribers total). Affected products are various Huawei Internet Gateways including HG658d.
• Telefónica in Spain (Movistar) exposes SSH remote administration on more than 170.000 devices. Affected products are Comtrend’s VG-8050 and possibly some ZyXEL DSL gateways.
• China Telecom exposes SSH remote administration on more than 100.000 ZyXEL and ZTE devices.
• VTR Globalcom in Chile exposes SSH remote administration on more than 55.000 Ubee Interactive cable modems.
• Chunghwa Telecom in Taiwan exposes SSH remote administration on more than 45.000 ZyXEL devices.
• Telstra in Australia exposes SSH remote administration on more than 26.000 Cisco devices.

What Is The Impact Of The Vulnerability?

Impersonation, man-in-the-middle or passive decryption attacks are possible. These attacks allow an attacker to gain access to sensitive information like administrator credentials which can be used in further attacks. In order to exploit this vulnerability, an attacker has to be in the position to monitor/intercept communication. This is easily feasible when the attacker is located within the same network segment (local network). Exploiting this vulnerability via the Internet is significantly more difficult, as an attacker has to be able to get access to the data that is exchanged. Attack vectors can be BGP hijacking, an “evil ISP”, or a global adversary with the capability to monitor Internet traffic.

Searching for key fingerprints in data from Internet-wide scans is a low-cost way of finding the IP addresses of specific products/product groups. This enables researchers to measure the extent of the problem but attackers can use this approach as well to exploit vulnerabilities (e.g. weak passwords or vulnerabilities in firmware) at scale.

Which Vendors/products Are Affected?

We found more than 900 products from about 50 vendors to be vulnerable. Of course our data is limited to the firmware we had access to. Affected vendors are: ADB, AMX, Actiontec, Adtran, Alcatel-Lucent, Alpha Networks, Aruba Networks, Aztech, Bewan, Busch-Jaeger, CTC Union, Cisco, Clear, Comtrend, D-Link, Deutsche Telekom, DrayTek, Edimax, General Electric (GE), Green Packet, Huawei, Infomark, Innatech, Linksys, Motorola, Moxa, NETGEAR, NetComm Wireless, ONT, Observa Telecom, Opengear, Pace, Philips, Pirelli , Robustel, Sagemcom, Seagate, Seowon Intech, Sierra Wireless, Smart RG, TP-LINK, TRENDnet, Technicolor, Tenda, Totolink, unify, UPVEL, Ubee Interactive, Ubiquiti Networks, Vodafone, Western Digital, ZTE, Zhone and ZyXEL.

We have generated tree maps in order to interactively explore the HTTPS certificate and SSH host key exposure on the public web including distribution on a per-country and ISP basis. Affected products are included as well in the hover boxes.

Did You Inform Affected Parties?

We have worked together with CERT/CC to address this issue since early August 2015. CERT/CC made great efforts to inform affected device vendors, chipset manufacturers and affected ISPs. Some vendors have responded and have started to implement fixes. CERT/CC informed major browser vendors as well. This vulnerability is tracked as VU#566724. Several CVE-IDs have been assigned as well.

What Can Be Done?

As a first step, vendors should make sure that each device uses random, unique cryptographic keys. These can be computed in the factory or on first boot. In the case of CPE devices, both the ISP and the vendor have to work together to provide fixed firmware for affected devices.

Furthermore ISPs have to make sure remote access via the WAN port to CPEs is not possible. In case the ISP needs access for remote support purposes, setting up a dedicated management VLAN with strict ACLs (no CPE to CPE communication) is recommended.

End users should change the SSH host keys and X.509 certificates to device-specific ones. This is not always possible as some products do not allow this configuration to be changed or users do not have permissions to do it (frequent in CPE devices). The required technical steps (generating a certificate or RSA/DSA key pair etc.) are not something that can be expected of a regular home user.

Recommended selected elements of a mitigation strategy for vendors to avoid deploying insecure, ‘toxic’ devices are:

• Execution of application security tests with high assurance level for all product components
• Enforcement of Software Security Assurance in the whole software development process
• Ongoing reevaluation of the maturity of application security
• Maintenance of trust in the market through a long-term, fundamental approach to application security rather than a reactive and opportunistic one

All identified certificates and private keys will be released shortly.

We want to thank CERT/CC for coordinating this vulnerability and the Scans.io/ZMap/Censys team at the University of Michigan and Rapid7 for publishing their Internet-wide scan data.

This study was conducted by Stefan Viehböck, Senior Security Consultant at SEC Consult.