VMware has released a new critical security advisory, VMSA-2021-0028, in response to the industry-wide issue regarding the open source Apache Software Foundation log4j Java logging component, which was discovered to have a critical vulnerability (CVE-2021-44228).

Because the log4j component is used by many vendors and software packages, this needs your immediate attention, not just at the VMware product level, but also for all other software in your environment.

Because of the ubiquity of the log4j component and the suddenness of this zero-day disclosure, the situation is still developing. Please subscribe to our Security Advisories mailing list (found on the right side of the VMSA page), and revisit the VMSA-2021-0028 advisory and this page periodically to get the latest information.

First, if you haven’t read the original advisory or are a returning visitor, here are some links to the different resources available:

• VMware Security Advisory VMSA-2021-0028 (descriptions of the issues and workarounds)
• VMSA-2021-0028 Questions & Answers (common questions & answers)
• VMSA-2021-0028 & Log4j: What You Need to Know (this blog post)
• VMware Communities Thread On CVE-2021-44228 (A great place to ask questions)
• Tips for Patching VMware vSphere & Cloud Infrastructure (practical advice for patching success)
• VMware vSphere Security Configuration Guide (baseline security best practices for vSphere)
• VMware Ransomware Resource Center (discussion around resilience tactics)
• VMware Ports & Protocols Firewalling Guidance (ports.vmware.com)

This post will be updated as new information develops.

Who is affected?

A VMware Security Advisory will always list the specific supported products and versions that are affected. This situation is developing, and the VMSA will be updated with more information.

Cloud-based VMware services are protected and operational. Customers of VMware Cloud on AWS are protected as well. Some customers with overly permissive management gateway firewall rules have had action taken to reduce their exposure from scanning and exploit activity occurring across the Internet. Those affected have seen direct communications from VMware.

When do I need to do something about this?

Immediately. The ramifications of this vulnerability are serious for any system, especially ones that accept traffic from the open Internet. Because of the suddenness of this “zero-day” disclosure, affected software is still being updated. This gives attackers the advantage.

Organizations that practice change management using the ITIL definitions of change types would consider this an “emergency change.” All environments are different, have different tolerance for risk, and have different security controls and defense-in-depth to mitigate risk, so the decision on how to proceed is up to you. However, given the severity, we strongly recommend that you act.

Why am I affected?

CVE-2021-44228 is in an Apache Software Foundation component called “log4j” that is used to log information from Java-based software. It has industry-wide impact. The vulnerability is critical, rated 10 out of 10 on the CVSS 3.1 scoring scale, because it is an unauthenticated remote code execution (RCE) vulnerability. It allows attackers to run commands on affected systems and workloads by simply getting them to log a specific string. The library that does the logging interprets that string as a command, instead of just writing it to the log. For example, an attacker might be able to use a login page, placing the attack string in the username field where they know it will be logged.

Every piece of software that has ever used log4j is potentially vulnerable. VMware uses log4j as well, which is why we have issued VMSA-2021-0028. However, this vulnerability also affects customer workloads. Customers need to assess their entire environment for use of log4j, in both infrastructure and workloads, and remediate it as soon as possible either through patches or workarounds.

The vulnerability was announced suddenly, as a “zero-day” vulnerability, taking the industry by surprise. Normally a vulnerability is reported privately to the software maintainers, who then have time to repair the issue and release an update, so attackers don’t gain a temporary advantage. With a zero-day disclosure like this one, attackers have an advantage while software maintainers scramble to develop the fix.

Regardless of the timing, the ubiquitous use of log4j guaranteed the vulnerability would have enormous impact, no matter when it surfaced.

A remote code execution (RCE) vulnerability is where an attacker who can reach the affected software over the network can execute commands on it and bypass the security controls in place. This leaves perimeter and microsegmented firewall controls as the last line of defense until the vulnerability is remediated. Beyond firewalling, most organizations employ good security strategies, such as defense-in-depth, zero trust and isolation, good password and account hygiene, etc. and usually these tactics help immensely when a security vulnerability is disclosed. Every vulnerability disclosure is different, though, and organizations should take steps to proactively & positively confirm that there are no open attack vectors that make them vulnerable.

Knowing that even a failed login to an impacted system or workload can be used as an attack vector, organizations who have placed affected systems and workloads on networks that are directly accessible from the Internet, or even from a wide audience inside their own organization, may not have these lines of defense and should audit their systems for compromise. They should also take steps to implement additional security controls to limit the scope of traffic until a workaround or patch is implemented. Ransomware groups have repeatedly demonstrated to the world that they are able to compromise corporate networks while remaining extremely patient, waiting for a new vulnerability in order to attack from inside a network. This is universally observed behavior throughout the industry and informs our suggestions here. Organizations may want to consider additional security controls and isolation between their IT infrastructure and other corporate networks as part of an effort to implement modern zero-trust security strategies.

What should I do to protect myself?

First, check VMSA-2021-0028 to ensure you are running an affected version of VMware software. If a workaround or patch is listed, apply it. If not, be sure to check back regularly as new information is being added regularly. Like other software vendors who use log4j in their products, VMware found out about this in a zero-day scenario and is now working nonstop to help protect customers and test updates.

Check your other workloads, too. Log4j is an incredibly popular logging library. It is especially important to protect anything that accepts traffic from the Internet.

You may have other security controls in your environment that can help protect you until you are able to patch. Use network perimeter access controls or NSX IDS/IPS and NDR technologies to detect and contain attacks against your workloads. For Cloud Infrastructure products like VMware vSphere, VMware Cloud Foundation, and VMware Cloud, as well as cloud add-on components like the HCX, Site Recovery Manager, NSX-T, and Cloud Gateway Appliances, we strongly suggest limiting access to management interfaces to only Virtualization Admins. Drive any direct workload management activity through the VM network connections instead of the VM console. This simplifies access control and makes the RDP or ssh management traffic subject to other security controls, such as IDS/IPS and monitoring.

Please ensure that network perimeter access controls are as restrictive as they can be. “Any/any” rules that make system access easy for you also make access easy for attackers and should be avoided.

If you are a VMware Carbon Black customer there is new intelligence posted to the Community with analytics to detect vulnerable libraries on systems.

Thank You

Critical security advisories are always difficult conversations, and unfortunately, part of the landscape in IT. They are no easier in a zero-day situation when all critical stakeholders are surprised at once. Please check back with the VMSA, subscribe to the VMware Security Advisory mailing list, and let us know what other questions we can answer. We appreciate you very much. Thank you for being our customers, and we wish the very best for you and others around you.