Protecting the Databricks platform and continuously raising the bar with security improvements is the mission of our Security team and the main reason why we invest in our bug bounty program. Through this program, we encourage (and reward) submissions from talented industry professionals who bring potential concerns to our attention. Working together with the larger security community, we can uncover and remediate newly discovered product issues and make the Databricks platform an even more secure and safe place.
When feasible and interesting to the security community, we also share success stories from collaborations that come out of our bug bounty program. Today we would like to showcase how a well-written report from SEC Consult helped accelerate the sunsetting of certain deprecated legacy features and the adoption of our new feature, workspace files.
This blog includes separate sections authored by Databricks and SEC Consult respectively. It goes into the technical details of the discovered security concerns, the affected configurations, the impact, and the solutions implemented to address the vulnerability. We would like to thank SEC Consult for their professionalism and collaboration on this disclosure.
At end of January 2023, Databricks received a report from SEC Consult about a potential privilege escalation issue that may allow an authenticated, low-privileged user of a cluster to elevate privileges and gain admin-level access to other clusters within the boundary of the same workspace and organization.
Our initial investigation aligned with the finder’s report and showed that exploitation of this issue required (a) a potential attacker to be in possession of a valid authenticated account, and (b) the applicable workspace to have either legacy global init script for clusters enabled, or alternatively, the presence of a preconfigured init script (cluster-named or cluster-scoped) stored on DBFS. In contrast to the case of cluster init scripts stored in DBFS, where the vulnerability can only be exploited where a script is present, enablement of legacy global init script (without a script file) is enough to be exposed to this issue.
In both cases (legacy global init script enabled or cluster init scripts stored in DBFS), an authenticated low-privileged user could add or take control of an init script and execute additional commands using the elevated privileges associated with running init scripts. Databricks has not found evidence of such privilege escalations occurring in practice.
The following table summarizes the most common scenarios for different types of init scripts (AWS | Azure | GCP):
In response to this report from SEC Consult, we took the opportunity to harden our platform and keep customers safe with a series of additional steps and new product features:
Guidance and recommendations
We've been encouraging customers to move away from legacy and deprecated init scripts for the past three years and this security finding recently reported by SEC Consult only emphasizes why customers should complete this migration journey as soon as possible. At the same time, the introduction of workspace files for init scripts (AWS | Azure | GCP) marks the initial milestone of our plan for offering a modern and more secure storage alternative to DBFS.
Customers can increase the security for their Databricks deployments and mitigate the security issue discussed in this blog by doing the following:
- Immediately disable legacy global init scripts (AWS | Azure) if not actively used: it’s a safe, easy, and immediate step to close this potential attack vector.
- Customers with legacy global init scripts deployed should first migrate legacy scripts to the new global init script type (this notebook can be used to automate the migration work) and, after this migration step, proceed to disable the legacy version as indicated in the previous step.
- Cluster-named init scripts are similarly affected by the issue and are also deprecated: customers still using this type of init scripts should disable cluster-named init scripts (AWS | Azure), migrate them to cluster-scoped scripts, and make sure that the scripts are stored in the new workspace files storage location (AWS | Azure | GCP). This notebook can be used to automate the migration work.
- Existing cluster-scoped init scripts stored on DBFS should be migrated to the alternative, safer workspace files location (AWS | Azure | GCP).
- Use Databricks Security Analysis Tool (SAT) to automate security health checks of your Databricks workspace configurations against Databricks security best practices.
The following section is a reproduction of the technical report authored by SEC Consult researcher Florian Roth and Marius Bartholdy. While the research described below was conducted and tested with Azure Databricks as an example, the findings related to the deprecated init scripts types affect other cloud providers as set forth in the table above.
Thank you again to SEC Consult, and all of the security researchers who are working with us to make Databricks more secure every day. If you are a security researcher, we will see you at hackerone.com/databricks.
2) Attack chain using legacy global init scripts
The same attack vector affected legacy global init scripts. These were deprecated in 2020, but left enabled by default in all workspaces and were also stored on the DBFS, specifically at
dbfs:/databricks/init/. Any cluster would execute their content on initialization. Therefore, simply creating a new script in that directory would eventually lead to code execution on all clusters.
With the Vulnerability Lab, SEC Consult operates its own internal security laboratory, in order to ensure an international know-how advantage over attackers in the areas of network and application security. It is nice to see vendors taking the security status of their products seriously. In this case, Databricks took the opportunity to not only fix a certain vulnerability but to harden their platform.
You can find more technical details in our Security Advisory "Bypassing cluster isolation in Databricks Platform" by Florian Roth (Atos) and Marius Bartholdy (SEC Office Berlin).
We also want to thank the whole Databricks team for their outstanding coordination and responsible disclosure process.