- On 9. Oct 2018
Xiong-who?! And Why We Care
Hangzhou Xiongmai Technology Co., Ltd. is one of the largest manufacturers of video surveillance equipment (surveillance cameras, digital video recorders (DVRs), and network video recorders (NVRs)) in the world. If it were not for the Mirai IoT botnet, their business strategy of acting only as an original equipment manufacturer (OEM) would have kept them a unheard-of corporation located in East-China. Starting in 2016, Mirai and variants of Mirai exploited critical vulnerabilities in Xiongmai devices, namely the fact that the devices offered high-privileged shell access over TCP ports 23 (Telnet) and 9527 (a Telnet-like console interface) using hard-coded credentials. Hundreds of thousands of Xiongmai-manufactured devices were infected and used as part of one of the largest distributed denial of service (DDoS) attack to date.
Remember the DDoS attacks against Dyn that caused outages for Netflix, Twitter, GitHub, Spotify, Airbnb, …. – basically for half of the internet? Or the attacks on the “Krebs on Security” Blog that caused Akamai to stop their Anti-DDoS protection service for the blog? Large parts of that DDoS firepower came from hacked Xiongmai devices.
What happened since?
The people behind Mirai were caught and sentenced and Xiongmai eventually fixed the vulnerabilities in new products. IoT botnet actors are still looking for vulnerable Xiongmai devices, large numbers of these devices have been “bricked” by an “anti-botnet botnet” dubbed Brickerbot.
Why is the SEC Consult Vulnerability Lab involved?
The automated firmware security analysis platform IoT Inspector reliably detects the kind of vulnerabilities Mirai and other IoT botnets are exploiting. The analysis works for all kinds of IoT devices including ones from Xiongmai. One of our motivations for doing this kind of research is to use the results to improve the automated analysis by IoT Inspector.
XMEye P2P Cloud
Common guidance for IT systems usually is: “Do not expose them to the web”. In the Mirai case, this worked well, fortunately only Xiongmai devices with exposed TCP ports 23 or 9527 could be infected, while all others were safe. Unfortunately, Xiongmai, like other video surveillance products (e.g. Gwelltimes/FREDI, MiSafes), has another attack vector, advertised as “P2P cloud”, “remote viewing capabilities”, or just “watch from anywhere in the world – there is an app available for iOS/Android”.
All Xiongmai devices come with a feature called “XMEye P2P Cloud” that is enabled by default. One part of it is a proprietary protocol that allows users to access their IP cameras or NVRs/DVRs via the Internet. Users can connect to their devices using various XMEye apps (Android, iOS), a desktop application called “VMS”, or an SDK for app developers. All connections are established via a cloud server infrastructure provided by Xiongmai.
From a usability perspective, this makes it easier for users to interact with the device, since the user does not have to be in the same network (e.g. the same Wi-Fi network) in order to connect to the device. Additionally, no firewall rules, port forwarding rules, or DDNS setup are required on the router, which makes this option convenient also for non-tech-savvy users.
However, this approach has several security implications:
- The cloud server provider gets all the data (e.g. video streams that are viewed). Open questions:
- Who runs these servers?
- Who controls these servers? Where are they located?
- Do they comply with local jurisdiction?
- Does the service comply with EU GDPR?
- If the data connection is not properly encrypted (spoiler alert: it’s not, we’ve checked!), anyone who can intercept the connection is able to monitor all data that is exchanged.
- The “P2P Cloud” feature bypasses firewalls and effectively allows remote connections into private networks. Now, attackers cannot only attack devices that have been intentionally/unintentionally exposed to the web (classic “Shodan hacking” or the Mirai approach), but a large number of devices that are exposed via the “P2P Cloud”.
Predictable XMEye Cloud IDs
So how does this “XMEye P2P Cloud” feature work in practice? Each device has a unique ID, called cloud ID or UID. Here is an example:
68ab8124db83c8db. Using this ID, the user can connect to the device through one of the supported apps. One would assume that the cloud ID is sufficiently random and complex to make guessing correct cloud IDs hard. Well, it’s not!
We reverse engineered parts of the Xiongmai firmware and found that the cloud ID is derived from the device’s MAC address. The MAC address is not a good source of randomness. It has a well-defined structure: a 3-byte OUI (organizationally unique identifier of the vendor) + 3-byte NIC ID (Interface ID). Xiongmai uses a few different OUIs and assigns interface IDs in ascending order. Further details can be found in our technical security advisory. This makes it easy for attackers to enumerate potential MACs/cloud IDs … and find valid ones. E.g. by trying the cloud ID for MAC
001210FF0000, 001210FF0001, 001210FF0002 and so on.
Cloud ID Scanning
After doing a bit of reverse engineering on the proprietary protocol used by apps to communicate with cloud servers, we developed a scanner that scans the Xiongmai cloud infrastructure for valid cloud IDs. The responses from the cloud indicate if there is a device online that uses the given cloud ID. Additionally, they contain the IP of a “cloud hop server”, which is geographically close to the device. One query requires just one UDP packet.
In 03/2018 we scanned 0.02% of the devices (random sample) in each OUI range (16 million devices per range). Based on the extrapolated results we estimate that there were at least 9 million devices online in the given OUI ranges at this time. The cloud IDs are not distributed randomly over the OUI ranges: in one OUI range there is a 15% chance to find a valid cloud ID, other OUI ranges have a significantly lower chance.
The information regarding the “cloud hop servers” allows us to estimate the geographic distribution of the devices:
- Hop server location China: 5,438,000 online devices
- Hop server location Germany: 1,319,000 online devices
- Hop server location USA: 742,000 online devices
- Hop server location Singapore: 697,000 online devices
- Hop server location Japan: 577,000 online devices
- Hop server location Turkey: 189,000 online devices
In most cases, we observed that hop servers on the same continent are used, e.g. the hop server for an IP camera located in the United Kingdom will be associated with the hop server in Germany.
Most hop servers are hosted on Amazon Web Services (Germany, USA, Singapore, Japan), Radore (Turkey) and Alibaba, Kingsoft or CloudVSP in China. During our scanning efforts, we encountered more than 30 different hop server IPs. All of this suggests that Xiongmai has put significant effort into building and operating their cloud infrastructure.
We did not attempt to mask the scanning activity. Although we sent more than 33,000 queries from one single IP address, we were not banned from the cloud infrastructure. This indicates that there is no brute force protection in place.
Interlude: How to Save Some Money on MAC Addresses
MAC OUIs are assigned by the IEEE. Registering an OUI costs money ($2,820). Interestingly Xiongmai does not own a single OUI, but instead just appropriates the OUIs of other companies. The following OUIs are used by Xiongmai devices (OUIs based on our internet research and scanning, the company names are from the IEEE registry):
Protechna Herbst GmbH & Co. KG
Koenig & Bauer AG
iStor Networks, Inc.
ICP Internet Communication Payment AG
We have established that it is possible to connect to millions of Xiongmai devices via the XMEye cloud. After connecting to a device, valid credentials are required. The default password of the admin user (username is “admin”) is blank. Users are not required to set a secure password in the initial setup phase, so it is likely that a large number of devices are accessible via these default credentials. The admin user can view video streams, change the device configuration and even issue firmware updates.
In addition to the admin user, by default there is an undocumented user with the name “default”. The password of this user is “tluafed” (default in reverse). We have verified that this user can be used to log in to a device via the XMEye cloud (checked via custom client using the Xiongmai NetSDK). This user seems to at least have permissions to access/view video streams.
Note: The password hash of the “default” user is “OxhlwSG8” (stored in /mtd/Config/Account1). The hash algorithm was reverse engineered before and is implemented on GitHub. Basically, it is a result of
MD5(password) and compressed even further. For complex passwords it should be more efficient to find a hash collision than to crack the password. Interestingly, the same hash algorithm is used in products from Dahua Technology. Possibly Xiongmai copied from Dahua or the hash algorithm is part of the Huawei HiSilicon SoC SDK both vendors use?
So even if the device has been secured by changing the admin password, it can be accessed via the XMEye cloud via the “default” user.
Code Execution via Firmware Update
We have established that it is possible to connect to millions of Xiongmai devices via the XMEye cloud and then login using default credentials. A next step from an attacker would be to find a way to execute arbitrary code on the device.
Firmware updates are not signed. It is possible to create a firmware update that contains malicious code. This is either possible by modifying the filesystems, contained in a firmware update, or modifying the “InstallDesc” file in a firmware update file. The “InstallDesc” is a text file that contains commands that are executed during the update.
Malicious firmware updates can be deployed via the XMEye cloud. For this, an attacker has to impersonate the cloud update server “upgrade.secu100.net” by changing the DNS configuration of the device (also part of XMEye API). Executing the
H264_DVR_Upgrade_Cloud() XMEye cloud API command causes the device to fetch and install the malicious firmware.
This technique allows an attacker to persist malware onto the devices’ flash memory. Unlike in the Mirai case, it cannot be removed any more by rebooting the device.
Different types of attackers can be differentiated as follows:
- “The Voyeur”: Using the vulnerabilities, an attacker can spy on users of Xiongmai surveillance products. Some even have a two-way audio intercom, so it is even possible to interact with victims as well.
- “The APT Lateral Mover”: Xiongmai devices are used in various “interesting” organizations/networks all over the world. An attacker can use the vulnerabilities to get an initial foothold in the local network and then use lateral movement techniques to gain access to other systems (lateral movement).
- “The Botnet Herder”: Using the vulnerabilities, millions of devices can be infected by a Mirai-like malware. The resulting botnet would likely be the largest IoT botnet in history.
Who is affected?
All Xiongmai devices are affected. Unfortunately, this information does not really help at all, because the devices will not mention “Xiongmai” anywhere (not in the manual, box art, web interface or DVR/NVR interface). This is because Xiongmai only acts as OEM and not as a brand. This strategy is called “white labeling”.
Various vendors sell branded devices with Xiongmai hardware/firmware inside. How many? We we have found more than one hundred vendors who sell branded Xiongmai devices. They are: 9Trading, Abowone, AHWVSE, ANRAN, ASECAM, Autoeye, AZISHN, A-ZONE, BESDER/BESDERSEC, BESSKY, Bestmo, BFMore, BOAVISION, BULWARK, CANAVIS, CWH, DAGRO, datocctv, DEFEWAY, digoo, DiySecurityCameraWorld, DONPHIA, ENKLOV, ESAMACT, ESCAM, EVTEVISION, Fayele, FLOUREON , Funi, GADINAN, GARUNK, HAMROL, HAMROLTE, Highfly, Hiseeu, HISVISION, HMQC, IHOMEGUARD, ISSEUSEE, iTooner, JENNOV, Jooan, Jshida, JUESENWDM, JUFENG, JZTEK, KERUI, KKMOON, KONLEN, Kopda, Lenyes, LESHP, LEVCOECAM, LINGSEE, LOOSAFE, MIEBUL, MISECU, Nextrend, OEM, OLOEY, OUERTECH, QNTSQ, SACAM, SANNCE, SANSCO, SecTec, Shell film, Sifvision / sifsecurityvision, smar, SMTSEC, SSICON, SUNBA, Sunivision, Susikum, TECBOX, Techage, Techege, TianAnXun, TMEZON, TVPSii, Unique Vision, unitoptek, USAFEQLO, VOLDRELI, Westmile, Westshine, Wistino, Witrue, WNK Security Technology, WOFEA, WOSHIJIA, WUSONLUSAN, XIAO MA, XinAnX, xloongx, YiiSPO, YUCHENG, YUNSYE, zclever, zilnk, ZJUXIN, zmodo, ZRHUNTER
How to identify a Xiongmai device
Identification via Web Interface
With network access to the device, a Xiongmai device can be identified. The login usually looks as follows, however it can be customized by OEM customers:
Identification via Error page
Another way of fingerprinting Xiongmai devices works via an error page called
err.htm (http://IP/err.htm). This is one of just a few places in the firmware where “Xiongmai” is mentioned. The response looks as follows:
Identification via Product Description
Without having access to a device, the easiest way is to check if the product documentation/specifications mention the “XMEye” feature. However, some vendors use branded apps, which are built using the Xiongmai SDK (e.g. Goodeye, iCSee Pro, JFeye). A search on Amazon.com for “XMEye” returns hundreds of product results. Here are two examples for Xiongmai products sold on Amazon.com.
The first one is a set of IP cameras and an NVR from “A-ZONE”. The product is quite popular, 140 ratings, #59 in the “Surveillance DVR Kits” category.
The second one is an outdoor dome camera from “SUNBA”. This product has 138 ratings and is #575 in “Dome cameras”.
Popular US retail stores also offer Xiongmai devices.
- HomeDepot’s “SecurityMan Wireless 720TVL Outdoor iSecurity Camera” is a Xiongmai device.
- Walmart offers various Xiongmai devices from “KKMoon” and “Evertech”
Identification via IoT Inspector
Another way of identifying the OEM of a device is the firmware security analysis platform IoT Inspector. It detects vulnerabilities in the firmware of IoT devices. This includes hard-coded passwords (“backdoors”) as well as P2P cloud functionality in the firmware of IoT devices, such as the XMEye P2P Platform, or various other Chinese P2P Platforms used in OEM devices: Dahua easy4ip, Dahua Lechange, Uniview EZCloud, Ozvision, Gwelltimes “Cloud-Links”, ThroughTek TUTK Kalay Platform, etc.
IoT Inspector can help you to avoid purchasing vulnerable devices from your supplier and introducing them in your organization.
The firmware of a Xiongmai device is available as part of our IoT Inspector Demo. Interested? Register online at www.iot-inspector.com to get a demo.
Vendor communication & Final Thoughts
We have worked together with ICS-CERT to address this issue since March 2018. ICS-CERT made great efforts to get in touch with Xiongmai and the Chinese CNCERT/CC and inform them about the issues. Although Xiongmai had seven months’ notice, they have not fixed any of the issues.
The conversation with them over the past months has shown that security is just not a priority to them at all.
Our recommendation is to stop using Xiongmai and Xiongmai OEM devices altogether. Please note, identifying a Xiongmai devices is difficult (see “Am I affected?” section above). The company has a bad security track record including its role in Mirai and various other IoT botnets. There are vulnerabilities that have been published in 2017, which are still not fixed in the most recent firmware version. This includes a directory traversal vulnerability and various buffer overflow vulnerabilities (CVE-2017-16725, CVE-2018-10088, complete exploit chain available).
The usual recommendations, like changing default passwords, strict firewalling and network segmentation, unfortunately do not mitigate the whole range of discovered issues.
What’s interesting, the recently passed “Defense Authorization Act for Fiscal Year 2019” has a section on “Prohibition On Use Or Procurement”, which bans all kinds of video surveillance equipment from Xiongmai’s competitors Hangzhou Hikvision Digital Technology Company and Dahua Technology. We think Xiongmai should receive the same treatment.
Addendum: “Indicators of Compromise”
Want to hunt for devices using the XMeye P2P Cloud in your infrastructure? Here are some domains/IPs used by the platform:
SEC Consult is always searching for talented security professionals to join the team. More information can be found here.