- On 24. May 2018
TR-069, the CPE WAN Management Protocol (CWMP), is the most widespread IoT management protocol.
It was first published in 2004 by the Broadband Forum. Most likely, you have a few devices in your home that use it on a daily basis. In case you’ve never heard of it before, here is the short version:
ISPs use TR-069 to manage their Customer Premises Equipment (CPE). The most common CPE devices are broadband gateways (e.g. a Wi-Fi router that provides internet access via DSL, Cable, LTE or WiMAX), set-top boxes and SIP phones. The counterpart on the ISP side is the Auto Configuration Server (ACS, no more telco-lingo from this point on, promise!). The ACS can set and query parameters on the CPE. In the TR-069 world a parameter is a setting on the device e.g. the password for the SIP server or a diagnostic value e.g. the signal-to-noise ratio on the radio interface.
A CPE either has the URL of the ACS hardcoded into the firmware or learns it via DHCP options. On first boot the CPE connects to the ACS in “bootstrap” mode. The ACS will then provision the device depending on the device type. What parameters are set during this part depend on the ISP. Some ISPs do a zero-touch provisioning via TR-069. This means that the customer must only plug in the CPE and the internet/phone connection is set up automatically. What happens in the background is that the parameters for the PPPoE or SIP credentials are set, firewall ACLs are set, device specific TR-069 credentials (connection request user/password) are set, accounts for remote assistance are added, a firmware update is scheduled etc.
TR-069 auto-provisioning is pretty interesting from a security point of view and probably worth a deep dive in a subsequent blog post. ISPs usually underestimate the risk involved. A common misconception is to assume that the CPE itself is somehow trusted and only trusted devices would talk to the ACS. In reality, an attacker can connect to the ACS and pretend to be another customer’s CPE in bootstrap mode (spoofing). In cases where zero-touch provisioning is implemented, and the ISP is not protected against spoofing attacks (e.g. by checking if the physical connection used for the connection is actually related to the customer), the attacker will get the SIP credentials or other interesting information like PPPoE credentials of random customers and can use them for fraudulent calls using premium numbers or more advanced attacks.
After bootstrap, all CPEs connect to the ACS periodically (e. g. every 24 hours) or on device boot. Here, the ACS has a chance to query and set parameters.TR-069 sessions are always initiated by the CPE.
For the sake of completeness: Usually, the ACS server is only accessible from within the ISP’s network (read: only subscribers can access it). If an ISP does not use TR-069 for managing CPEs right now, it probably uses SNMP (probably in connection with DOCSIS). It seems that TR-069 use is getting more common though.
While each CPE usually has a vendor specific TR-069 client implementation, the ACS solutions used by most ISPs are Nokia’s Motive Connected Device Platform (formerly called Motive Home Device Manager), Friendly Technologies, AVSystem and Axiros. If an attacker can find a serious vulnerability in the ACS server software, he/she can likely compromise all CPEs via deployment of malicious firmware.
Call me back? Please?!!
If the ACS does not want to wait until the CPE connects again, it can send a connection request to the CPE. This works via a “Connection Request Server” HTTP server that listens on TCP port 7547 on the CPE. The ACS connects to this server and authenticates using a connection request user/password set during bootstrap.
From a TR-069 protocol view, the Connection Request Server is not a security risk. It can only be used to tell the CPE to connect to the ISP’s auto configuration server (ACS). But in the past, there have been various vulnerabilities in the implementation of the Connection Request Server:
The most widely known vulnerability is “Misfortune Cookie”, found by CheckPoint in 2014. It’s a memory corruption vulnerability in some ancient versions of Allego Software’s commercial HTTP server called RomPager. ZyXEL embedded RomPager into ZyNOS, their proprietary operating system for embedded systems and also some Linux based products. The supply chains for IoT devices are quite hard to follow, but it is likely that ZyXEL (or its sister company MitraStar) work as an OEM/ODM for various other vendors like TP-Link, Huawei, D-Link, ZTE, Edimax, etc. That explains why products from those vendors are vulnerable to “Misfortune Cookie” as well.
Another vulnerability again affecting ZyXEL/MitraStar developed firmware is the “rom-0 vulnerability” also found in 2014. Again, Huawei, TP-Link, D-Link and ZTE products are affected as well. This is an information disclosure vulnerability. An attacker can download the device configuration including the admin password by requesting a file called “rom-0” via the webserver (e.g. http://IP:port/rom-0, exploit).
Both “Misfortune Cookie” and the “rom-0 vulnerability” are not specific to TR-069. ZyXEL just happens to use the same webserver code they used for the administration web interface of the device for the TR-069 Connection Request Server.
In November 2016, another vulnerability affecting TR-069 Connection Request servers was discovered. A guy called “Kenzo2017” published a blog post regarding a vulnerability in one of the CPEs from the Irish ISP “Eir”, called “NewNTPServer vulnerability”. As it turns out, this command injection vulnerability again affects various devices from ZyXEL/MitraStar (you see the pattern?). Botnet herders quickly picked up the vulnerability and started exploiting devices worldwide, causing a bit of attention. What’s interesting about this vulnerability is that it exploits a command injection via a TR-064 command.
Interlude: TR-064 is another standard by the Broadband Forum for “LAN-Side DSL CPE Configuration”. It’s not as popular as TR-069 and is intended to be used by customers to configure their CPE. It’s designed and implemented as an extension to UPnP and should only be available via the LAN. The most probable use-case is changing the WiFi SSID/channel/passphrase. You might already be using TR-064 via a smartphone application provided by your ISP. We have sufficient material for TR-064 security for another blogpost… so much for now: If TR-064 is implemented according to the specification, it allows the customer to change settings that are probably not available to customers via the web interface of CPE. This includes changing the TR-069 ACS URL. Researchers and attackers can set up their own ACS server and get a foothold in the CPE e.g. by adding highly privileged users, pushing firmware updates etc. If the vendor has not taken appropriate precautions, there probably is a default, hardcoded TR-064 config/reset password.
Let’s go back to November 2016: ZyXEL/Mitrastar made a real mess by exposing their vulnerable TR-064 server via the TR-069 Connection Request Server on the WAN port. Even without the command injection vulnerability this should never ever happen.
A few days later, TR-069 was in the news again. About 1.200.000 customers of Germany’s Deutsche Telekom were offline. The outage was a side-effect of botnet herders who tried to exploit the TR-064 “NewNTPServer vulnerability” in ZyXEL devices. The flood of requests caused the Deutsche Telekom CPEs from the Taiwanese OEM Arcadyan to crash. The exact root cause is not known, but analysis by Comsecuris shows that the Arcadyan devices do not even run Linux but do react allergic to TCP flooding. The police even caught the guy who was responsible and brought him to justice. Brian Krebs published great research on the background of the attacker.
“Devil’s Ivy” vulnerability
In July 2017 Senrio published a vulnerability they found in the third-party code library gSOAP, a C/C++ SDK for developing SOAP/XML web services. This component is not specific to IoT devices or TR-069, but Senrio was able to exploit the vulnerability in Axis IP cameras. The ONVIF SOAP/XML interface running on the devices was built using a vulnerable version of gSOAP.
As TR-069 is also a SOAP-based protocol, various vendors implement the TR-069 components using gSOAP. We estimate about 25% of all exposed TR-069 Connection Request Servers are based on gSOAP.
Exploitation of “Misfortune Cookie”, the “rom-0” and the “NewNTPServer” vulnerabilities by botnet herders continues. In addition to basic Telnet brute-force attacks Mirai botnet spinoffs are now attacking TR-069 Connection Request Servers as well. There are reports of “BrickerBot” variants that are attacking (and bricking) CPEs of US ISP Sierra Tel via TR-069. According to Kaspersky, the “Hijame” botnet is also capable of infecting CPEs via TR-069. According to Wordfence reports, attacks originating from CPEs affected by “Misfortune Cookie” are on the rise.
One might ask: Why should the TR-069 Connection Request Server be exposed to the web? The answer is: There is none. Only negligence on the ISPS’s side. Only the ISP’s ACS server has a legitimate reason to connect to the Connection Request Server on CPEs. ISPs can easily set ACLs on the CPE that prohibit any other IPs from accessing the port or set up a dedicated internal network segment for remote management.
Peak CPE: 75 Million
We did a bit of analysis on publicly available historical scan data for TCP port 7547 published by the always awesome Censys project and correlated it with GeoIP/Org data to determine which ISPs are affected. Data is available starting from September 2015.
On November 16th 2016, 12 days before the outage at Deutsche Telekom, 75 Million devices on the web listened on TCP port 7547. This is about one device per 1000 people living on this planet! That sounds like an actual internet of things!
In the aftermath of the attack against Deutsche Telekom end of November 2016, various ISPs (including Deutsche Telekom) have started to limit access to this port. Here is an excerpt:
- Germany: Deutsche Telekom AG (From 4.5 million exposed hosts to about 0 hosts)
- Brazil: Vivo (From 2.3 million to ~0)
- Argentina: Telefonica de Argentina (From 1 million to ~0)
- Colombia: Colombia Telecomunicaciones S.A. Esp (From 0.6 million ~0)
- Ireland: Eircom (From 0.4 million to ~0)
- Canada: Bell Aliant (From 0.3 million to ~0)
- Great Britain: TalkTalk (From 1.8 million to ~0)
- Turkey: Turk Telekom (From 1.3 million to ~0)
- Thailand: 3BB Broadband (From 0.5 million to 0.1 million)
- Czech Republic: O2 Czech Republic (From 0.4 million to ~0)
- Great Britain: Tiscali UK Limited (From 0.4 million to ~0)
- Great Britain: Post Office Broadband (From 0.1 million to ~0)
- Russia: OJSC Sibirtelecom (From 0.1 million to ~0)
- Russia: Rostelecom (From 0.2 million to ~0)
- USA: AT&T Internet Services / AT&T U-verse (From 6.3 million to ~0)
- Mexico: Telmex (From 3.2 million to 0.5 million)
- USA: Windstream Communications (From 1.0 million to ~0)
- Vietnam: FPT Telecom Company (From 0.2 million to ~0)
- Egypt: TE Data (From 1.8 million to ~0)
- Italy: WIND Telecomunicazioni S.p.A (From 0.4 million to ~0)
- Hungary: Magyar Telekom (From 0.2 million to 0.1 million)
- Poland: Neostrada Plus (From 0.2 million to ~0)
- Philippines: Philippine Long Distance Telephone (From 0.6 million to ~0)
- Latvia: Tele2 Latvia (From 0.2 million to ~0)
- USA: Comcast Cable (From 14.5 million to ~2.8 million)
- India: BSNL (From 1.0 million to ~0)
- Brazil: Oi Internet / BR_Oi Velox (From 0.6 million to ~0)
- Lithuania: Tele2 Mobile (From 0.5 million to ~0)
- Indonesia: PT Telkom Indonesia (From 0.4 million to ~0)
- Vietnam: Vietnam Posts and Telecommunications (VNPT) (From 0.3 million to ~0)
- India: Airtel Broadband (From 0.3 million to ~0)
- Croatia: Mobile Services (From 0.2 million to ~0)
- India: Mahanagar Telephone Nigam (From 0.2 million to ~0)
- Iran: Information Technology Company (From 0.2 million to ~0)
- Iran: Telecommunication Company of Tehran (From 0.2 million to ~0)
- Australia: FOXTEL MANAGEMENT PTY LTD (From 0.2 million to ~0)
As of 29th of November 2017, only about 30 million hosts are still affected. This is a reduction of about 60% percent compared to the 75 million hosts in November 2016.
The “Top 20 ISPs” that still allow access to the TR-069 Connection Request Server are:
- Korea: Korea Telecom (5 million)
- USA: Time Warner Cable (4.3 million)
- USA: Comcast Cable (2.8 million)
- Great Britain: British Telecom (BT) (1.9 million)
- Saudi Arabia: SaudiNet (1 million)
- USA: Charter Communications (0.9 million)
- Mexico: Telmex (0.5 million)
- USA: Sprint PCS (0.5 million)
- USA: Frontier Communications (0.4 million)
- Algeria: Telecom Algeria (0.4 million)
- Dominican Republic: Claro Dominican Republic (0.3 million)
- Great Britain: Sky Broadband (0.3 million)
- Spain: Telefonica de Espana (0.3 million)
- Russia: OJSC Rostelecom (0.3 million)
- Thailand: TOT (0.3 million)
- Canada: Shaw Communications (0.3 Million)
- Australia: Telstra Internet 0.3 million)
- Russia: Beeline Home 0.2 million)
- Argentina: Cablevision S.A (0.2 million)
- USA: Google Fiber (0.2 million)
ISPs should limit the attack surface on CPEs as much as possible by blocking all incoming connections from the internet. This is not just limited to the TR-069 Connection Request Server. Our research on HTTPS Certificate and SSH Key reuse has shown that millions of CPE devices also provide SSH access and HTTPS access via the internet.
Even if there are no known vulnerabilities in the TR-069 /HTTP/SNMP/[any component name here] server implementation of one CPE right now, attackers could find and exploit one tomorrow. Most CPEs are developed in pure C/C++ code that was probably never properly audited, scary vulnerabilities around every corner!
The firmware security analysis platform IoT Inspector detects the various vulnerabilities in IoT firmware, including the “Misfortune Cookie” vulnerabilities discussed above. IoT Inspector also detects various protocol implementations in uploaded firmware. This makes it easy to check which devices in your “IoT Zoo” could be affected by TR-064/TR-069 vulnerabilities.
Subscribers can check if the ISP exposes any services on the CPE by doing a scan of the WAN IP from another host on the web. This can be accomplished for example via a Nmap port scan. The German publication heise Online hosts a very helpful “Port Check” tool that checks if TR-069, HTTP, UPnP and a few other services are exposed (unfortunately in German only).
This research was done by Stefan Viehböck on behalf of SEC Consult Vulnerability Lab.
SEC Consult is always searching for talented security professionals to join the team. More information can be found here.