Cosmos DB users advised to regenerate their keys following serious vulnerability

The Azure vulnerability, which affects only those using the Jupyter Notebook feature, gives attackers access to data in databases.

Thousands of organizations using the Cosmos DB service on Microsoft Azure need to regenerate their primary read-write keys after researchers found a vulnerability that would have given external attackers access to copy, delete or modify data stored in databases. Microsoft fixed the vulnerability on the server-side and said they've seen no evidence of data being accessed by unauthorized parties, but the researchers who found the flaw and the US Cybersecurity and Infrastructure Security Agency (CISA) recommend that all Cosmos DB users who had the Jupyter Notebook feature enabled rotate their keys.

Jupyter Notebook privilege escalation

The flaw was located in Jupyter Notebook, a Microsoft Azure tool for data exploration, data cleaning, data transformations, numerical simulations, statistical modeling, data visualization and machine learning. The tool integrates with Cosmos DB, a NoSQL-type database service used to store data for Azure-hosted applications by thousands of organizations, including many Fortune 500 companies.

"A series of misconfigurations in the notebook feature opened up a new attack vector we were able to exploit," researchers from cloud security company Wiz said in a report. "In short, the notebook container allowed for a privilege escalation into other customer notebooks. As a result, an attacker could gain access to customers’ Cosmos DB primary keys and other highly sensitive secrets such as the notebook blob storage access token. "

The primary keys provide administrator level access to Cosmos DB instances with full read/write/delete permissions. The database service also requires users to have secondary keys as well to make the key rotation process easier. The secondary keys were not impacted.

Since it found no evidence that this flaw has been exploited before the researchers found and reported the vulnerability on August 12, Microsoft only notified the Cosmos DB users who could potentially be affected due to the researchers' activity. The researchers estimate that only around 30% of users potentially had their primary long-term keys exposed to this issue. That's because Jupyter Notebook has been enabled by default for every new Cosmos DB account created since late January 2021 and could be manually enabled by older accounts, too. Meanwhile, the research activity only lasted for around one week before the flaw was reported.

"Starting this February, every newly created Cosmos DB account had the notebook feature enabled by default and their primary key could have been exposed even if the customer was not aware of it and never used the feature," the researchers said. "If the customer didn’t use the feature in the first three days, it was automatically disabled. An attacker who exploited the vulnerability during that window could obtain the primary key and have ongoing access to the Cosmos DB account."

Mitigating the Cosmos DB vulnerability

In addition to replacing their primary read-write keys, Cosmos DB users should also review and restrict the network exposure of their databases. If databases are accessible directly from the internet, they are obviously at higher risk, but even those that are not could have been exposed to this vulnerability.

The Wiz researchers point out that it's common for Azure firewall configurations to have an exception enabled called “Accept connections from within public Azure datacenters.” This is required for some Azure services to function, but it also means that any other tenant within the Azure network has network access to the database, too, and it's not hard for an attacker to get an Azure account and attack databases from within the Azure IP space.

One way to restrict access is to define specific IP addresses that are allowed to connect to the database. Another is to configure access to Cosmos DB via a virtual network, and a third option is to configure an Azure Private Link for private endpoint connections.

Regenerating primary keys requires replacing the primary key in all applications with the secondary key and then regenerating the primary keys as described in this Microsoft guide. However, before that process is started, organizations should have an inventory of their Cosmos DBs with Jupyter Notebook enabled.

An easy way of doing that is to perform a nslookup for [cosmosdb-name].notebook.cosmos.azure.com where [cosmosdb-name] is the name of every Cosmos DB account they have. If the Jupyter feature is enabled, that subdomain should exist and nslookup should return information about it. If nslookup responds with domain not found, it means that Jupyter is not enabled for that account.

Since key regeneration can be a lengthy process that depends on how quickly the configuration of applications can be updated, security teams might want to monitor if and when the keys have been regenerated for every Cosmos DB account. Wiz created a short PowerShell script that can achieve that by analyzing the logs.

Another long-term mitigation to consider is migrating from primary and secondary access keys to role-based access control (RBAC). This allows a more fine-grained access control per user and security principal to Cosmos DB and the identities can be audited in the Azure Cosmos DB diagnostic logs. Enabling diagnostic logging has also help discover access from suspicious or unauthorized IP addresses.