While you were watching the F1 title decider between Max Verstappen and Lewis Hamilton or excited for the Succession finale, companies running the internet were scared shitless.

You might not have noticed it because services like Twitter, Facebook, Gmail, and smaller ones all stayed up. But a bug in an open-source tech called Log4j was (and still is) causing panic amongst the infosec community across the world.

While the bug has affected billions of devices, and companies are scrambling to apply fixes, the open-source community has a raging debate going on about funding volunteers that maintain projects like Log4j.

Before we dive into all that, here’s a brief background about the technology and the issue.

What is Log4j?

Log4j 2 is a Java-based open-sourced logging framework that comes under the Apache Foundation services, so anyone can use it for free. The logging software is used by companies to track activity on their servers (or even client-side apps).

For instance, when you visit a website, your IP address, your browser, and the pages you visit are registered by the logger. This data related to activity can help companies solve any problem with their service.

Since the library is Java-based, there’s a chance that billions of devices supported by the framework might be at risk.

What is the Log4j bug?

The bug, listed under CVE-2021-44228 last week, allows attackers to remotely execute code through a specially crafted string. Since Log4j is so common, cybercriminals can easily manipulate log strings and control the server or the client.

One of the main reasons for this bug’s existence is that some versions of Log4j are capable of executing arbitrary text through a directory lookup protocol (LDAP protocol).

If you want to read a detailed explanation of the issue in simple language, read this Twitter thread.

Explaining #log4j for non technical people, because the internet is burning down and y'all might want to know what's happening and why there's all this "${jndi:ldap" stuff out there#Log4Shell#log4jRCE

— Emy | eq 🌈 (@entropyqueen_) December 11, 2021

Who’s affected?

To be honest, I should’ve named this section “Who’s not affected?” Log4j’s open-source nature and widespread compatibility means it’s a great choice and tons of companies use it — including Apple, Microsoft, Steam, Twitter, Baidu, and Cloudflare.

As soon as the details about the vulnerability were out, several people started scanning servers to check if they were open to exploitation.

Mass scanning activity detected from multiple hosts checking for servers using Apache Log4j (Java logging library) vulnerable to remote code execution (https://t.co/GgksMUlf94).

Query our API for "tags=CVE-2021-44228" for source IP addresses and other IOCs. #threatintel

— Bad Packets (@bad_packets) December 10, 2021

Some programmers tested various sites and services to check the potential reach of the attack; you can check out the full list here.

Importantly, as Ars Technica noted, many of these companies have several lines of defense against cyber attacks. So even if some of the servers are open to the Log4j attack, other security systems can thwart the attack.

The issue came to the fore through Minecraft sites, when researchers found out that a specific string passed to the server or even in-game chat could give attackers control of the system. Web infrastructure firm Cloudflare’s CEO, Matthew Prince, tweeted the company will roll out some protection to all customers — including the ones using the free tier.

We’ve made the determination that #Log4J is so bad we’re going to try and roll out at least some protection for all @Cloudflare customers by default, even free customers who do not have our WAF. Working on how to do that safely now.

— Matthew Prince 🌥 (@eastdakota) December 10, 2021

Notably, there are several appliances that run Java server components, and Log4j could be a part of that. So your toaster could be a hacking target too (?).

Data security platform LunaSec has a lengthy technical explanation about the impact of the bug on its blog.

What’s the mitigation?

Hours after the bug was discovered, the security team at the Apache Foundation released a new version (v15.0) of Log4j with patches that secure systems from attacks.

Minecraft’s owner, Mojang Studios, also published a guide for people running the official game client and game servers to fix the bug. It warned that people running third-party game clients might have to wait for the provider to patch the exploit. Plus, the Swiss government’s computer response team (CERT) has tweeted an easy-to-understand prevention guide in a picture format.

Heads up: We see a lot of scanning against the #log4j vulnerability and decided to publish a blog post with some guidance:https://t.co/dtEXfY1G16

Please patch now! pic.twitter.com/HSedlSed0V

— GovCERT.ch (@GovCERT_CH) December 12, 2021

Companies like Red Hat and VMWare released their own fixes to plug the issue. For server admins, severalsecurity blogs have detailed guides for mitigations strategies. A GitHub user has made a super useful list of patches issued by companies, including Netflix, Citrix, Docker, and Oracle.

It’s not over yet

While service providers and the open-source community are working round the clock to fix the issue and secure as many systems as possible. The danger is looming over our heads.

As Bleeping Computer noted in its report last night, attackers are installing cryptominers, malware, and deploying botnets for DDoS (Distributed Denial of Service) attacks. Microsoft’s threat intelligence team has noted that many attackers are using the Colbat Strike penetration detection tool for credential theft.

Sean Gallagher, a senior threat researcher at Sophos, also said that the security company has detected many attempts of remote code execution:

Since Dec.9, Sophos has detected hundreds of thousands of attempts to remotely execute code using the Log4Shell vulnerability. Initially, these were Proof-of-Concept(PoC) exploit tests by security researchers and potential attackers, among others, as well as many online scans for the vulnerability. This was quickly followed by attempts to install coin miners, including the Kinsing miner botnet.

The most recent intelligence suggests attackers are trying to exploit the vulnerability to expose the keys used by Amazon Web Service accounts.

Security researcher Greg Linares tweeted that threat actors might be building a worm to attack many servers through the Log4j bug and cause damage or demand money.

#Log4J based on what I've seen, there is evidence that a worm will be developed for this in the next 24 to 48 hours.

Self propagating with the ability to stand up a self hosted server on compromised endpoints.

In addition to spraying traffic, dropping files, it will have c2c

— Greg Linares (@Laughing_Mantis) December 12, 2021

The Cybersecurity and Infrastructure Security Agency (CISA) has also issued a warning stating that the bug “is being widely exploited by a growing set of threat actors, presents an urgent challenge to network defenders given its broad use.” The agency suggested organizations to disable external-facing devices running Log4j and install additional firewall security with rules related to the vulnerability. 

The open-source community enraged by low payment for contributors

This Log4j bug has once again established that large corporations around the globe depend on free open-source software heavily. That’s all fine and dandy, but the open-source community is now outraged by the fact about contributors not getting enough pay.

Filippo Valsorda, a cryptographer at Google, noted on his personal blog that the maintainer who fixed the Log4j bug contributed to the project part-time, and had just three GitHub Sponsors (a way for people to pay project volunteers).

He noted that contributors only get a small amount through GitHub or Patreon, while their projects might be responsible for millions of dollars of business. Plus, a bug in these important systems can break the internet, and it’s unfair to depend on people who work on these projects in their spare time without any assured rewards.

This is the maintainer who fixed the vulnerability that's causing millions(++?) of dollars of damage.

"I work on Log4j in my spare time"
"always dreamed of working on open source full time"
"3 sponsors are funding @rgoers's work: Michael, Glenn, Matt"

People, what are we doing. pic.twitter.com/2hAxUWCjuC

— Filippo ${jndi:ldap://filippo.io/x} Valsorda (@FiloSottile) December 10, 2021

Matthew Green, a professor of cryptography at Johns Hopkins University, argued that the industry needs a list of open-source libraries and technologies with widespread reach so more people can work on projects that are important to run the internet smoothly.

Ok, so what I’m saying is that if you want to solve this problem, to me the missing resource is not money. It’s *visibility*. We all know the landmines are out there — but we can’t see them. 8/

— Matthew Green (@matthew_d_green) December 11, 2021

Security firm Chainguard’s founder, Dan Lorenc, said that while corporations are willing to fund open-source projects, it’s hard for them to find out critical projects that need paid maintainers. He also noted that ideally, the industry should fund teams that maintain three to four projects.

This is going to sound blunt, but it's a distribution problem not a funding problem. $ is easy.

Corporations have budget and are willing to spend, but it takes too much time. Finding projects that need help and maintainers willing to help in exchange for money is hard. pic.twitter.com/mFkOoOVYXn

— Dan Lorenc (@lorenc_dan) December 12, 2021

Contributors cited that their pay through contributions is super low, it’s really hard to maintain expenses by just doing that. That’s a sad state of affairs.

Hopefully, this issue will finally wake up the industry to pay attention towards coming together and contribute towards open-source projects that keep the internet afloat. But perhaps they might go back to sleep if the bug doesn’t cause much damage.

I have to end this with an XKCD comic summing up the situation.