6

Akamai to Acquire Linode

 2 years ago
source link: https://news.ycombinator.com/item?id=30352772
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

Akamai to Acquire Linode

I was really close to becoming a Linode customer last week for several large boxes, but ended up buying hardware and self-hosting.

The cost to cut in an enterprise fiber link (1000/400) to our office was surprisingly low.

My experience trialling linode as a paying customer was really positive (VERY competitive price, great performance), only complaint being their NodeBalancer can’t automatically handle certificates like CF and AWS can.

Also, it needs to be mentioned because there’s a lot of negative in this thread: HUGE shout out to Linode for being massive community supporters and donating HEAPS of free compute to uni clubs and such, pretty much anyone with a charity certificate who asks for it. Fantastic company that’s easy to ‘just talk to people’ rather than trying to find the right support silo.

s.gif
I've used Linode for 10+ years with a handful of long running projects. Support in the rare occasion (until 1 slow ticket recently) has always been absolutely amazing - real humans. If Josh is reading this your name has come up on a few tickets and answered promptly. A grade! Services have been solid. And pricing has been very reasonable. (knock on wood...) never ever had a significant outage and just don't think about going anywhere else for small scale projects. Even when I've needed some stuff from AWS I just host the core bits in Linode and pick and choose the AWS items.

Hope Akamai doesn't do the dirty...

s.gif
Linode's great support is what is preventing us to migrate to AWS or GCP.

Funny story: 2 months ago GCP blacklisted a few of Linode's IP subnets and we were cut out from using Google APIs. I was unable to get in touch with GCP support (not event complaining on social media) as they charge for support.

I asked to Linode's support if there is something that I could do even though it was not an issue created by them and they were aware of the issue and offered me to migrate our VMs to an IP subnet that was not banned.

s.gif
AWS' support is pretty awesome in my experience. Never would use anything ran by Google tho.
s.gif
Yeah. I've never paid for any sort of support contract and have never once ran into an issue where (1) I couldn't reach someone and (2) that person wasn't empowered to fix my issues. And by "issues" in some cases I'm talking like "facilitate a $150k refund".

Hell, I've been testing some "beta" products in AWS on my personal account with a spend of like... a couple grand a year and been put in touch with the lead engineer on the team responsible for the product to dig into my problems.

Google, on the other hand, even paying over a million a year for AdWords I regularly ran into problems that crossed into other domains and while the AdWords support staff were relatively easy to get a hold of, even once you were over the wall they were pretty honest in saying they wouldn't be able to get support on issues crossing into other teams and just suggested trying to find workarounds.

s.gif
Yeah I'll continue to use Linode for the foreseeable future. Although they're kind of expensive, the support is top notch.

A year ago I was between jobs and broke, and asked for my upcoming $65 bill to be delayed until the month after. Since I had been a customer for quite a few years Support comped the then outstanding $65 and added another $65 to my Linode balance so I got a free month.

I hope the aquisition by Akamai makes their volume storage a bit cheaper though ;)

s.gif
> I hope the aquisition by Akamai makes their volume storage a bit cheaper though ;)

Damn it! Another shirt needs to go to a dry cleaner...

s.gif
Second that, fond memories of using their service ages ago.

First VMs I had was from them. They had very good documentation and everything was simple.

Then I was introduced to 'wall of text' as a service, AWS, and never looked back.

s.gif
Why move to AWS if you’re happy with Linode? Genuine question
s.gif
Not OP, but I can answer - also was very happy with Linode for boxes, but there's a lot more than just boxes these days - queues, streaming, metrics, monitoring, containers, etc. Once an application moves from 'runs on a fixed number of boxes' to 'autoscaling, automatic healing/recovery and distributed databases', the overhead of DIY gets high (and I'm paid to do that!).

My clients pay a fairly high premium for AWS because it allows them to move _faster_, not that it's _cheaper_.

s.gif
Depends on the day of the week - literally. For one client, I wear the devops hat, enabling developers, automating CI pipelines, etc. For another, my role is more "experience in the room", telling you 'no, you really don't want to do that, here's your risks, here's your tradeoffs, here's a better way'.

The jobs both derive out of a very similar set of tools and expertise, it's typically the client that decides what hat I'm wearing.

s.gif
Linode was very limited in what you could do, from what I remember. They had good service and documentation and I just like it now cause, nostalgia.

AWS is extremely flexible and very very feature rich compared to linode. At that point, AWS was considered about 10 years ahead of competition.

s.gif
What makes support absolutely amazing and A grade? Wanna compare how our support compares :)
s.gif
I can give you a counter example. Previously I was a Linode customer - for nearly a decade up until the point I was not. My account was frozen, and when I called, I was told I was not [any longer] the account owner. I provided a billing statement -9- years old to validate that I was indeed the account holder, and always had been. The support rep had to talk to their manager, and asked me to call back(not wait on hold, or wait for them to call me). When I called back, I was told the situation is "very sensitive legally". Then they asked if I knew a first and last name combination that was not mine, because that name was now listed as the account owner. I said that I did indeed know this person, they were the C-level employee at a company I previously worked for. The C-level employee had contacted Linode and cowed them into believing they were the rightful owner of an account that existed before I ever worked for them. They gave the account to them, refused to give it back, and suggested I retain an attorney for any further movement on the issue.

If Linode is reading this - the incident took place over summer/fall of 2018. Yes, I'm that guy. No, I haven't forgotten. I've told _everyone_ who will listen this exact story, and warned them of having their own hard work and assets misappropriated by anyone with a fancy job title calling Linode support and demanding it. A phone call from Linode after everything was settled would have completely changed this sentiment. There's my own story of your outstanding lack of customer support. I was your biggest fan(for 9 years!) up until that point, and referred a dozen or more customers to your service.

s.gif
Sounds like you were just trying to take over one of your employee's accounts...
s.gif
It sounded to me like an ex-employer successfully sold an ex-employees linode account.
s.gif
What was very visible for me personally is that support people are very highly qualified, and can actually solve the problems themselves instead of escalating. So think of exposing higher qualified people to the front lines of support. Moreover, the feeling of speaking to a real person, who actually cares is something hyperscale cloud guys will never give you.
s.gif
Well I'm sure there's always opposite stories - but mine is I just get tickets answered whenever which is important as I've often ticked at some ungodly hour in the US (being overseas). You get human responses and even when a couple of times it's been outside of Linodes control or responsibility (routing/peering issues) they've been helpful and informative.

Everyone will have their own counter of course but I've been happy with what I've been getting for the money!

s.gif
It's not that bad, but not amazing in my opinion, yes they reply, yes they will answer promptly and share info if you prompt them - tell me what happened etc.. But it's not very forthcoming, delayed acknowledgements of data center wide issues has been an issue for several years, every pledge to update the status page within minutes has never materialized - I think I saw an update within 15minutes once and nearly fell out my chair, but it's still poor when you have to hit Twitter to find dozens of other people with broken connectivity and find the momentary relief that it's probably not your VMs.
s.gif
Fast, friendly, informed, useful— and all done with clear, high quality communication.
s.gif
We've moved to AWS since, but we started our early cloud infra on Linode and was always so impressed by their customer support. Genuine good quality. Let's hope it keeps going well.
s.gif
Customer support really is top notch. Linode and DO are so close/competitive on most things, but the DO support is garbage and they will screw you over with no remorse. Linode on the other hand will answer you with a human who cares. Even if it's a stupid PEBKAC issue.

I may end up moving to AWS, but I'm dreading the pricing. Just estimating alone you damn near have to hire an expert. There's dozens of different ways to screw yourself and pay too much.

s.gif
I have only great experiences with DO support. Really fast and helpful. I even got some free credits ($50?) at some point for a small inconvenience .
s.gif
We needed to scale quickly, AWS allowed us to consolidate a lot of services, and our team made the collective decision to migrate and take advantage of all there is to offer. It's worked out very well. We spend more, yes, on services but less on hours spent working.
s.gif
Their support is the best of any cloud provider I’ve used. Hands down. They are excellent.
s.gif
I agree but one thing I noticed was like the first couple of tickets I opened responses were almost immediate. And then after that response times went down to 24 hours. I'm not sure if that's because I was on free trial credits and I hadn't actually made any purchases so it was deprioritized. Or if maybe I was using the wrong support email address? Has anyone had this experience? At first I uniformly thought their support was top notch as well, and I still think the quality is among the best I've seen, but the response times went down. I don't know why.
s.gif
Could it possibly be that their customers base is limited so they can have better support for each of them?
s.gif
Try vultr, they've worked well for me for small->medium projects.
s.gif
I've heard really good things about Vultr. I'll have to keep their bookmark handy in case Akamai brutally screws Linode up.
s.gif
Likewise with Hetzner. Great prices with decent support.
I think this is Akamai figuring out they need to address the self-serve market.

Akamai has 6x the edge network footprint of Cloudflare and has all the cool trendy stuff like edge workers, they just suck at selling to the developer.

s.gif
> Akamai has 6x the edge network footprint of Cloudflare and has all the cool trendy stuff like edge workers, they just suck at selling to the developer.

Akamai sucks at selling to everyone.

Here's a typical Akamai experience:

* An Account Manager (AM) - in a suit - he is going to be the one talking. Not an expert in anything, except maybe Brooks Brothers suits from a few seasons ago.

* Account Manager's Lackey (AML) - he is going to be the one with an iPad. His job is to type in the notes of the meeting, typically using a two finger peck method. This of course takes a very long time. So everyone speaks slowly. Sometimes AM would ask AML something about things that are coming or being rolled out using Akamai internal jargon.

* Account Manager's Attachment (AMA) - does not really talk, looks uncomfortable. Based on the way AMA is dressed it might be a sales engineer

* Every single response to your question would be something that you can either find on the website ( AM, AML and AMA are convinced you cannot read ) which does not actually answer the question or is met with "I will need to loop in the appropriate parties and get you an answer"

* Nearly every request that one would presume can be completed programatically is met with "We need to engage professional services for that." which is billed at some insane amount

That's experience of paying Akamai millions of dollars a year. Needless to say, spending three years untangling Akamai integration was worth it.

s.gif
I used to sit on an akamai customer panel. ~10 years ago at their conferences they were talking about to lower the sales burden. Pricing, Digital signup. Clearly they think it will devalue thjeir existing product.
s.gif
What no young pretty girl? These guys are rank amateurs.
s.gif
They explicitly choose not to sell to the developer. They want big fish customers. This is why I can’t find pricing on their homepage.
s.gif
Historically that's all there was. Random companies didn't need CDNs and associated services. They ran a couple servers in a data-center somewhere and called it good. Not like a website for a company that manufactures dairy processing equipment needs to handle a lot of load. The big corporate customers where who bought that stuff. Then AWS came along.

It's a classic case of not seeing the up and coming market because you were winning the existing market.

s.gif
One could reasonably conclude that they will start doing this soon under their newly acquired Linode subsidiary.
s.gif
Or they'll make Linode "call for pricing".
s.gif
Linode isn't used by Akamai customers and won't be.

It would be a much bigger hurdle for Linode to become enterprise-friendly than for Akamai to become developer-friendly.

Akamai is "call for pricing" because they never evolved from an enterprise product. And no "enterprise" just buys off the shelf. They negotiate with a long budget, vendor risk management, legal, etc. process. Its unlikely that many of their contracts are even the same verbiage.

s.gif
Do you have examples of companies “evolving from an enterprise product”?

As far as I know companies usually evolve into an enterprise product, moving up the ladder, rather than going down.

Yes, the sales cycles are longer, but there’s more money and less headache with customer support and customer success.

E.g. 1 customer paying $10,000 per month vs 100 paying $99 p/m.

s.gif
The closest I can think of is the IBM PC.

But we can all think of 100s of companies that did not do this and either died or are hollow shells of their former selves.

Cloudflare et al are commoditizing their gig. As more new blood goes into enterprises, they’ll want to use what they’re already familiar with vs the big enterprise thing they’ve never touched and find awkward.

I don’t know that a) I’m right or b) that they’ll be successful.

But Enterprising Linode would be folly. Even late entrants like Oracle Cloud are struggling due to lack of support from Enterprise cloud products. Changing them to support all the annoying acronym requirements of an Enterprise is a many year journey. AWS and Azure really only did that to support their huge government contracts.

I think the only other viable theory is for them to try and keep Linode as Linode as just a diversification play.

That would be odd, to me at least, because they’re buying an offering that seems to be slowly dying anyway. It really never left the hobbyist/tiny shop market, and even there, AWS/Azure is eating into them. I do have a soft spot for them as they’re a much more “human” company.

s.gif
maybe this will help Akamai identify Linode customers who are evolving into potential Enterprise customers and hook them.
s.gif
I find this unlikely. It would be a massive waste of capital to have acquired a developer-focused company such as as Linode only to make it, “contact us for pricing.” I think it’s more likely that they will sell the CDN product as another service within Linode.
s.gif
As someone who has used Akamai and talked to them, they seem to be so rooted in, talk to your account manager for it, who will just upsell you stuff you don't need.
s.gif
Worked with Akamai in a company I was working with. That company is so rooted into the 90s that the Spice Girls look fresh and modern. :P
s.gif
Not to mention that until they finally had a first-class Terraform provider, their CDN was absolute nightmare to manage at-scale. If you only have a handful of properties then sure, it's fine. You can just point-and-click your way around and make it work. Anything more than that and it was very painful.
s.gif
"they just suck at selling to the developer."

They have all the major compagnies / F500, though so no need to appeal to the HN crowd.

s.gif
Yep, that's exactly how large businesses like IBM and HP have failed.
s.gif
They're doing so terrible, only making $5.7B in profit in 2021.

I wish I could fail so hard.

s.gif
The best unicorns lose that much money in a year. Profits are for people who don’t “get it”
s.gif
Down 35% over the past ten years, which is terrible
s.gif
Sounds like the market is terrible at pricing
s.gif
Insane! I wasn't aware Akamai was so huge in scale as compared to Cloudflare.
s.gif
I mean Akamai was in the business since 1998 and has been a leading player in CDNs.
s.gif
Yeah I seem to remember they became well-known when they were the CDN for the Apple Trailers site around the (Sorenson encoded) Episode 1 Trailer times?
s.gif
That figure is misleading. Every one of Cloudflare’s pops run every service they offer. A large number of Akamai’s are running small CDN only deployments. They also have 20 worldwide ddos scrubbing centers. Cloudflare’s 250 pops all do it.

It’s not really apples to apples in terms of scale in the context of footprint

s.gif
Have they gotten faster at applying updates? it would take something like 45 minutes to an hour to make any changes back in 2014, when Fastly was doing sub-minute updates for any CDN changes.
s.gif
Yes, Akamai let’s you deploy changes within minutes if not quicker, that changes several years ago.
s.gif
Production network activation takes about 10 mins. Staging network activation around 3 mins.
s.gif
Their dev tools and docs need some love. Cloudflare has a smooth DX that Fastly nor Akamai emulate at the moment
s.gif
They should just expand their services to individual/small business
s.gif
They have had "edge workers" in some form or another for 20 years. Most of the common use cases for things you'd want to do at the edge could be done in their config programming language. Now they have VMs/containers in 15k+ locations.
s.gif
Yes

You can do most thing with configuration. EdgeWorkers are also available if you want to run your own JS at the Edge https://developer.akamai.com/akamai-edgeworkers-overview

As a long time Linode user, I hung in there through a lot of service issues and growing pains. Linode has always been just one of a handful of providers I have, and I figured at _some_ point, all of these issues will result in a solid platform. Well, over the last couple years, Linode HAS achieved this, and is now one of my most reliable platforms.

I also use Prolexic DDoS service from Akamai, which was a standalone company until 2014. Akamai bought them, and the service actually vastly improved, so let's hope that Linode + Akamai's size and buying power will result in something even better.

s.gif
Thanks for sharing regarding Prolexic. That does give me some hope.

If Akamai helps polish up some of the rough edges (there aren't very many for the record) that would be great. I've been eagerly anticipating managed databases for a little while, as well as bare metal and some other things. Would be neat if those things happen a little faster now that they have bigger capital behind them.

I think Linode is the only ad I’ve ever clicked on the internet.

Around 2003 they were advertising via static banner ads on sourceforge.net, and I remember the offer being great for the time. Root in a “VM” (it was user mode Linux in those days) for something like $15/mo. I hung out in their IRC channel with caker and the early crew. I remember them talking about unboxing and racking servers and what their DCs were like. Can you imagine just casually chatting with someone who has intimate knowledge of your EC2 machine and its network?! It was a great time. I scored a free Xen VM when they were beta testing their transition and promptly had my first (and last so far!) experience getting a server hacked. Left it running with root ssh logins and a weak password. I migrated to other hosting services before the scandals hit I think. I’m glad to only have great memories and wish them all the best!

I've been using Linode since the mid 2000s when Chris Aker, the owner and CEO, provided support on IRC ( OFTC: caker) and the company only had two or three staff.

Great to see how he built Linode organically to the scale it has reached. In the context of venture capital and Y-combinator its a good contrasting story on how successful a bootstrapped, patiently built, profitable sustainable business can become when the focus is on excellent (and personal) customer service.

Really happy for Linode. I cannot say enough good things about the quality of their products and support.

As a customer who's built significant value on their infrastructure, I'm a little bit worried about the impact this acquisition will have on their operations. I really hope it gives them the resources to improve on the same mission, and not the beginning of significant change!

s.gif
I've been a happy Linode customer for a full decade now. One thing I've always appreciated is that they grew sustainably and sensibly, keeping a focus on core competencies (VMs, disks, networking) rather than offering dozens of new services a year to keep pace with AWS, GCP, etc. Compared to the confusing, complex, and often buggy interfaces provided by Big Cloud, Linode has always been a breath of fresh air. I hope Akamai will realize that Linode customers really value this simplicity. Don't complect it!
s.gif
Same here. Linode has been my go to infrastructure ever since I decided I needed to leave MediaTemple and become serious about my server infrastructure. I feel like Linode made me a better developer by helping me learn the devOps side of the business. Back in the day, their guides were the best in the business, though I feel like DO is winning that game now, especially from an SEO standpoint. These days I look for DO guides on how to tune my Linode machines.

Congrats to the Linode team, please don't leave us hanging!

s.gif
Same. I'm definitely worried that they'll become more corporate as part of this, but at the same time a lot of those changes have already happened. When I first started using Linode, they had Rickrolls in their X-Mailer headers for all requests. It was honestly something that made me choose them in the first place - it felt like it was being made by people who would pay attention to those small things.

I've been super happy with their service over the decade I've been using them, but I've steadily been using Vultr more and more recently.

s.gif
As a long time Linode user, and I've given a try to others as well ... e.g. Digital Ocean, Linode's been my infrastructure choice for many many years.
s.gif
Linode is like a Digital Ocean that has customer support and doesn't hate their small customers. I really hope they don't change too much.
s.gif
I think you're probably safe assuming Akamai will ruin Linode. I went through this once before with Slicehost, and had to move to Linode once Rackspace made Slicehost unusable.
s.gif
This is exactly my current concern after hearing this news. Purely anecdotal, but I've had nothing but good experiences with both Linode and Digital Ocean over the years. I genuinely hope that Akamai does not destroy everything good about Linode, but at least if they do, thankfully there's more'n a few other excellent options to switch to.
s.gif
As another long time user of Linode, I have had nothing but good experiences with them. I hope their quality stays up.
I have a lot of infrastructure on Linode, and have been a customer for many years. I've been very happy with the price/performance, especially compared with large providers some of my customers use like Rackspace.

I hope that being part of a public company won't cause too much pressure to reduce services or raise prices.

s.gif
Same, I moved from rackspace to linode and never looked back.
s.gif
To be fair moving from rackspace to an OVH datacentre that's on fire is a step up from my experience. I occasionally have to deal with a Rackspace acquisition and it's hell.
s.gif
Me too. Used a different service more than 15 years ago that provides a ton of resources at an inexpensive price, then it was bought by rack space. Quickly migrated to Linode after that.

There aren't any clear alternatives to Linode this time.

I'm very disappointed at all the negativity here. I get it, I'm worried too. I hate the modern trend of the exit always being an acquisition. I hate that the big companies scoop up the good little companies and ruin them. I hate it all. I feel very jaded too, and I'm very worried about what this means too. I have significant investment running on linode and I don't want to have to move it, which also means paying more, and settling for less.

But that said, we are missing a lot of details. As worried as I am, I'm going to give a company who really deserves the benefit of the doubt, the benefit of the doubt.

I would really, really appreciate a blog post or announcement or something though from Linode that details what things are changing and when. I know you may not know yet, but when you know if you could share I would greatly appreciate it.

s.gif
>I'm very disappointed at all the negativity here.

Really? I'm not.

Also, it's not "negativity." It's realism. When human-scale firms like Linode get acquired by massive companies, they lose their way. The things that made them good -- excellent support, understandable product lines, etc -- vanish in the name of integration with the mothership. It's happened over and over in tech, including and especially in hosting/VPS providers.

The press release is borderline offensive in its "nothing will change" babble. Obviously nothing will change TOMORROW, but within 24 to 36 months Akamai will have ruined every good thing about Linode.

It's nice the founders got a payday, but everyone who relies on Linode's current responsiveness should be looking for an alternative provider immediately. It's irrational to expect this will pan out any differently than every other time a massive leviathan absorbed a beloved small player.

>But that said, we are missing a lot of details.

There are no details that will change any of the above.

s.gif
> When human-scale firms like Linode get acquired by massive companies, they lose their way.

Packet.net seems to be doing well under Equinix

s.gif
It takes a little time for the big-corp cancer to ruin the acquired firm, usually. The Equinix deal was only like 18 months ago.
s.gif
Equinix isn't Akamai, I'd consider them human scale.
s.gif
Equinix market cap is 62B and 10000 employees

Akamai market cap is 18B and 8300 employees

s.gif
Agreed. I think we are all a bit too wary of destructive or anticompetitive acquisitions... but this really doesn't look like one of them, from what I understand they genuinely complement each other.

Linode already use some of Cloudflare as part of their offering e.g nameservers, so I can already see how Akamai could begin to replace those under the hood and then augment Linode's existing services further to offer new capabilities to the small customers (that's how I envision it happening, Linode has a good customer experience and the Linode Manager is very polished now so it would make sense that Akamai would lean on it).

I'm discounting all the anecdotes and handful of past explosions, as any company as old as Linode will have them, and I find comparisons to AWS kind of absurd where you will likely be talking to a robot.

s.gif
I have had multi million dollar accounts with both Akamai and AWS. Akamai's support was woeful bordering on incompetent. AWS on the other had has been at a very high standard. And no robots.
s.gif
> Akamai's support was woeful bordering on incompetent.

Isn't that the point? Akamai's weakness and Linode's strength.

I've no experience with Akamai at all so i'm interested in your's. However support quality and service quality are not always correlated, especially when support has been highly siloed or outsourced. Obviously they affect each other, but are you able to distinguish them? Would you be able to offer us any insight into the quality of the service they provided separate from the support issues?

> I have had multi million dollar accounts [...] AWS on the other had has been at a very high standard. And no robots.

I mean, once you are paying millions you are guaranteed a real person. I was talking about the small scale support issues which from what I hear on AWS usually results in automated responses. This is where Linode and AWS definitely differ.

s.gif
Even small accounts (<10K) will get phone calls from reps asking about your use case / if they can help you, and if you pay the $30/mo for dev support the cases that need it will get a person.

Amazon are huge so mistakes will be made but AWS's support reputation is not just hype. (My experiences were APAC region, can't speak for rest of world).

s.gif
Interesting, my anecdotes come from other people IRL... Not sure why there is this inconsistency, perhaps it depends on the type of issue.
s.gif
Yeah now that I think about it, it's going to be hard to generalise because experience will depend on the type of issues. Ok, here goes for me:

Limit increases: looked automated?, person might get involved if automation can't approve.

Stuff outside the control plane (e.g. how do I install nginx on this?): - no idea, never asked those questions, don't expect they'd hold your hand much unless you're spending a lot. Maybe they'll send you a link.

Something broken with AWS in general: usually not timely responses but path here is to use PHD (personal health dashboard).

Something broken with AWS in my account: occasionally stuff inside AWS gets "stuck". I've had issues with CloudFormation, CloudFront, Custom Domain Names and ACM certs - when I was creating / destroying these things lots of times in integration tests. I generally got timely response for these and eventually the issue stopped happening.

s.gif
> Something broken with AWS in general: usually not timely responses but path here is to use PHD (personal health dashboard).

Ah, I think this is the type of issue i've heard of where you hit a wall on AWS.

In my experience with Linode when there has been infra issues they are very open if pushed, which is useful because sometimes there are ways you can mitigate it if you understand what's going on... they will also pre-emptively open a ticket with your account in case you want to discuss it, most of the time these resolve themselves, but you can dig if you have concerns. e.g one time we had a few VPSs go through a vague "physical host failure" migration unusually close in time, so I asked if there was anything going on with hypervisor bugs (turns out there was a regression) and whether we could pre-emptively migrate stuff to patched hosts on our own terms rather than wait for it to be randomly picked - and they just opened up, explained what was going on and helped us schedule our other VPSs.

> Stuff outside the control plane (e.g. how do I install nginx on this?): - no idea, never asked those questions

Me neither, it's usually Linode specific support I use, they also have extended paid support for that type of stuff and I've never used it but it's there for people who need it.

s.gif
Vs what we saw from Google cloud? AWS is top tier lol. Akamai I've no experience with at all and have heard mixed reviews.
s.gif
Quote from Chris Aker's Linode blog post [0]:

"For the immediate future, we will continue to operate as we always have. Akamai has no intention of changing what has made us successful. This acquisition will propel us both forward — not take anything away. Linode will soon be able to call on the power of Akamai to offer entirely new products, services, expertise, locations, and scale, while Akamai will be able to tap into Linode’s deep expertise in compute, storage, and on-demand infrastructure-as-a-service.

Today, we start a new chapter in Linode’s story — one that strengthens our position in a market increasingly looking for alternatives. The transaction is expected to close in the first quarter of 2022.

Thank you — our customers, partners, and community — for supporting and empowering us to make cloud computing simple, affordable, and accessible."

[0] https://www.linode.com/blog/linode/linode-and-akamai/

s.gif
That's what every Our Incredible Journey post says though... In my experience it has no effect on reality, it's just a form letter.
s.gif
That's just meaningless noise. Not even Aker probably believes this acquisition will go any differently than any other big-swallows-little deal.

1. In the short term, it's probably the same.

2. In the medium term, things start to get weird.

3. By the time Aker exits, Linode probably only exists as a brand. Everything special about it will be gone. Support will be offshore and terrible.

s.gif
3) -> I call it the bean-counter syndrome; penny wise, pound foolish; quarterly curse.
s.gif
"for the immediate future, we will continue to operate as we always have." == 18-24 mos of transition, then founders exit, and company starts its merge into larger org.

Happens every time

s.gif
I've been using Linode exclusively to host my (admittedly paltry) compliment of VPS's for several years now. They have provided me with outstanding service throughout my time with them, including truly being there for me a few times during the onset of the pandemic in 2020 where I really needed a helping hand to keep my business going. I don't have any experience with Akamai, but Linode has earned my trust and loyalty, and that will continue until I see some reason to reconsider that.
s.gif
What other kind of "exit" is there? There's IPO or sale. My problem is VC funded startups is that they aren't trying to build a business.

There needs to be another way for a business to take on investment without focusing on an exit.

The DAO model is really interesting (I current work for one) which allows investors to exit any time they want by just selling the tokens they bought.

s.gif
It may be okay, Redhat has been fine under IBM. I'm just pessimistic by nature, I'm not disappointed that often, and delightfully surprised sometimes, if infrequently. Just because you have a pessimistic outlook on life doesn't mean you can't enjoy it, I think it's more of a Buddhist (some forms anyway) way of expecting suffering and delighting in the joy of life nonetheless.
s.gif
The Red Hat acquisition was two years ago. On the timeframe or "corporate parent ruins acquired", it's still the short term, so I don't think we can call a verdict at this point.
s.gif
In that time, Red Hat has:

1. Killed the sole reason why people use CentOS. 2. Has so tightly integrated systemd and NetworkManager to Fedora that it's essentially unusable for server environments.

Sure, Red Hat has been fine.

s.gif
Didn't they end CentOS and ruin a bunch of peoples plans?
This worries me. I've been a happy Linode customer for a while. I've mostly only noticed Akamai when they were screwing something up for one of their customers. I try to be ready to migrate off of any service like this at the drop of a hat, but I suppose this is a good reminder to test my plans for getting off Linode, just in case.
s.gif
> I've mostly only noticed Akamai when they were screwing something up for one of their customers

Normally their service is pretty invisible to the end user so it would be weird if you noticed them when things were working fine.

s.gif
Where are you gonna go if you leave Linode? Do you know yet?

Digital Ocean is the most similar, but after getting burned hard by them I will never put myself in that vulnerable position again.

I've been pretty happy with OVH thus far, but at least last time I checked the US presence was minimal and that's important to my customers.

s.gif
The difference that sticks with me is this:

Linode: "We're really sorry, there's a kernel bug and we absolutely have to patch it. See here for the CVE. We're scheduling it for time X but you can do it in advance on the web panel at a better time for you."

Digital Ocean: "Isn't it great? We're rebooting your machine at time X and you'll get new better stuff! No, there's no way to change the time or delay it."

s.gif
I currently have most of my stuff split between Digital Ocean and Linode. So if I had to move quickly, it'd be Digital Ocean.

I'm going to be kicking the tires on Hetzner and Vultr soon, because DO is not a super satisfying backup plan for me.

I've used Azure enough to get spun up quickly there, but that would be an unhappy upward adjustment on my budget or an unhappy downward adjustment on the performance I expect.

s.gif
Hetzner is good but I’ve found that their connection from EU to US is weak, apparently in Level 3 somewhere.
s.gif
Hetzner Cloud have recently launched a US location so this shouldn't be a problem.
s.gif
Ashburn, its working fine. Extremely happy with their pricing. Their customer support can be a bit 'off'. Feels like "closing ticket" attitude. But hey, their pricing is 50% of competitors, so I can live with that.
s.gif
I've already moved most of my servers from Linode to UpCloud. I did this as the disk IO speed at UpCloud is insane and made some intensive apps measurably faster.

The downside is price as UpCloud bundle less and charge for everything.

s.gif
DigitalOcean gives out /124 subnets, why would anyone use DO?
s.gif
You might look to move your deployments to Kubernetes, once you get past the learning curve. Its by far the easiest way to remain "cloud agnostic" in todays age.
I liked using Linode, though I can't help but feel some apprehension about what will happen to it in the medium/long term
Happy linode user since 2004 when I was 13 years old. Has always been my go-to for just having a linux vps running somewhere in the cloud. Really can't complain hope the acquisition doesn't change anything about how they operate.
This is an interesting mix. In my experience Linode has a good UX and is well targeted at SMEs, whereas I'm not sure anyone has bought an Akamai contract anywhere but a golf course, and it has a UX to match.

Is this Akamai trying to buy access to a market who would previously not even consider them, or is it Akamai trying to buy access to the more general cloud infrastructure market? If they're trying to do both I can't see it going down well.

s.gif
We failed to build Linode like things internally. We innovate through acquisition.

[ source: I was a part of those failures ]

s.gif
If you don't mind sharing, how long ago was this? And do you know if the attitude is, "this is hard and we blew it, let's by an expert and learn from them" or was it, "we only failed because <excuse> let's buy a starter platform and use it as the foundation?"
After dealing with Akamai at a past role I can only offer my condolences to Linode customers.
s.gif
I'm trying to think of an Akamai acquisition that has worked out but coming up blank.
s.gif
Prolexic (security), Blaze.io ("front-end" web app performance), SOASTA (web performance monitoring) - all integrated into Akamai flagship products & continuously improved.

Disclaimer: Worked @ Akamai

s.gif
Thanks, I hadn't heard of any of those but good to know they can be decent custodians.
s.gif
Mind sharing your previous experience? Expensive? bad customer support? poor product?
I can't think of many cases where an acquisition served customers well. Often the distinctive things that make a company appealing are lost as costs are slashed, good staff flee and the parent tries to capture customers into their existing products and services.

My impression of Akamai was always if you have to ask the price then you can't afford it. It seems an odd match. I guess the thinking is that they both have to compete with AWS and provide a one stop shop which is probably true to some extent but certainly not for a customer like me.

I have been a small Linode customer off and on for over 15 years and in that time they have had ups and downs, made very public mistakes and learned from them. Overall they generally supplied a competitive product and decent customer service. So I hope this works out well for all involved. If not there is lots of competition still in this space.

s.gif
I think you make reasonable points, but I'm hoping that this means Akamai was realizing that they need to get more into self-serve and started looking around for someone to buy (since that's what bigger companies always do these days). I may be that they want to be more like Linode. At least, that's what I'm hoping.
I have been a Linode customer for most of their existence, and I mostly don't have to think about them. Which is awesome. I sure hope that doesn't change...
I love Linode. Strongly recommended.

For many years, I have run all my websites and services (back-ends for apps, etc.) on a few $5/month Nanodes.

For temporary/experimental stuff I sometimes use Vultr and Digital Ocean, but Linode's service and reliability is superior (especially versus Vultr). Linode is just top-notch.

s.gif
You've clearly never experienced their Fremont datacenter.
s.gif
For example, my https://NN-512.com is served by a Nanode in Linode's Fremont datacenter (45.56.92.68).

That same Nanode has been running the back-end of an Android app for almost 5 years, on another port.

s.gif
They moved data centers (to Hurricane Electric Fremont 2). It's not unusually unreliable.
How does DigitalOcean compare to Linode these days? One thing I noticed off the bat is that Linode still offers phone and email support, versus just a ticketing system interface.
s.gif
And they answer the phone.

I've called their support line many times to report business-impacting issues, and they always answer within seconds (literally), provide competent support, follow-up, and genuinely care. Linode support may be better than Amex Platinum Card support. :-)

s.gif
> Linode support may be better than Amex Platinum Card support.

If talking about today, almost assuredly given your description.

This is pretty offtopic to the OP but I felt compelled to chime in as it really irritates me, that Amex support has gone into the toilet over the past 2 years or so. I remember a time when you called, and someone would just answer "hello mr xxx" if it was the number on file. Now it's phone menus, typing in your card number, other details, hold times, etc...even on Platinum support.

s.gif
If you start the call via the app it adds a PIN code onto the phone number which will automatically bypass all the menus/identification stuff and take you directly to a person.
s.gif
Huh…I haven’t had a Platinum Card for a few years (some other cards provided better value), but that’s disappointing.
s.gif
They answer the phone at 11pm central (USA) time, with a human who speaks good English, even when the customer (me) pays them ~$15 per month.
s.gif
It should go without saying the word of mouth advertising they get from people like you, as well as how we in aggregate influence purchasing decisions makes your account significantly more valuable.

Compare to how word of mouth of Google horror stories is part of why their market share in cloud computing is running even with Alibaba's based on what I just found for 2021.

s.gif
I use DO and have no plans of moving away from there, because on a technical and cost level I am happy with their offering, but I can agree that their customer service is not at all good.
s.gif
Pricing and offering is near identical. The biggest difference is the customer support, and the fact that Linode doesn't immediately lock accounts when the algorithm thinks something is fishy. DO really screwed me a few years back with that.

There are some smaller differences. DO has more managed DB offerings, but Linode is catching up quickly. I really like Linode's Lish tool. I also like Linode's Green Light Program.

s.gif
I wonder if Cloudflare is thinking about acquiring DO to round up their next-gen cloud.

DO's UI has gone downhill since 2017. It is so bloated and large. Used to be very compact.

Folks from Cloudflare if you're reading this - please keep your UI compact. Your main UI (Dashboard) looks more compact that the docs[1] which are too sparse and terrible for developers who are not average consumers. They can handle the complexity. They also appear to be designed by two separate UI design teams. You guys need an authoritarian designer at the top.

[1] https://developers.cloudflare.com/

s.gif
Their website is more complex, but they offer many more services than before. Not sure what else they could to do streamline things? Prices have gone up a bit as well, but to be expected to a certain degree. Still, I like them for their fixed pricing and no surprise bills at the end of the month. My only complaint is they have a tendency to kill some processes with no warning and no ticket. I'd be fine if they created a ticket to say "Hey, we're killing this process because we're not sure you want to run it, if you do click here to add it to an exclusion list" rather than silently killing it.
s.gif
Digital ocean makes a hundred fifty million more in revenue and 50 million more in profits. I don't think digital ocean will acquire them or merge.
s.gif
This is a meaningless thing to compare. DO has a market cap of $6B and Cloudflare is $37B. This discrepancy is because Cloudflare has insane potential for growth (54% YoY) vs a done and dusted business model of Digital Ocean.
s.gif
What are you comparing? Surely DigitalOcean isn't bigger than Cloudflare or do I have my perception entirely backwards? Guessing that you're comparing DigitalOcean to Linode instead, 50M seems like a very small difference actually.

Btw I find the large split words as numbers ("makes 150M more" -> "makes a hundred fifty million more") quite hard to read, but maybe that's just me.

s.gif
For 2020 (the last full year for both companies):

Cloudflare (NET) has higher revenue: $431M vs $318M

Digital Ocean (DOCN) has higher profit (net income): -$44M to -$119M

Cloudflare is growing revenue faster, but also increasing losses to get there.

s.gif
I tried DO after using Linode for a few years, and DO felt more polished and streamlined. DO's docs are great, too. But I honestly have no complaints with Linode. Note that I'm just running Nginx on a $5/mo VPS.
s.gif
I've had a Linode for many years with no problems, and have had to use their excellent phone service once.

I tried DO and hated it because they used a zillion 3rd-party services on all their internal web pages (after being signed in) so I was in a constant battle enabling things with NoScript.

I have a small Vultr VM I use for testing. Their site didn't require enabling a bunch of 3rd-party domains. They've been rock solid as well, like Linode, and cheap as dirt: I have a small VM with IP4 from years ago for $2.50/mo

s.gif
You may have convinced me to move over to Vultr.
s.gif
That doesn't include a ip4 address and you can't point a domain to it.

I have two grandfathered 2.50 packages with an ip (no domain pointing).. no issues ever but it's a playground not a production box for me.

s.gif
How can you have an ip that can’t be pointed to by a domain?
s.gif
I'm curious about this too. Just... point to it on Namecheap's DNS configurator page?
s.gif
If all you need is web traffic, I've seen some use Cloudflare's proxy, which does support ipv6 as the origin.
s.gif
Linode has better support, DigitalOcean is a lot closer to a cloud-lite setup with better APIs for automation etc.
From Linode blog:
  For the immediate future, we will continue to operate as we always have. Akamai has no intention of changing what has made us successful. This acquisition will propel us both forward — not take anything away. Linode will soon be able to call on the power of Akamai to offer entirely new products, services, expertise, locations, and scale, while Akamai will be able to tap into Linode’s deep expertise in compute, storage, and on-demand infrastructure-as-a-service.
https://www.linode.com/blog/linode/linode-and-akamai/

I like Linode and been with them for ~5 years. Their pricing didn't change much (if at all) during that time but hosting landscape got more competitive since then. Hetzner and OVH offer a better value I believe. I hope Akamai's resources will help them to not lose in the long run.

s.gif
> Akamai has no intention of changing what has made us successful

All the buyers say that now. In 2 years, it will change.

s.gif
That sentence isn’t saying nothing will change, as some letters do.
s.gif
I too, read this as ‘farewell, and thanks for all the fish’.
s.gif
Yeh, why buy them if things aren't going to change…
Congrats Linode. I switched back to Linode after a few years on Digitalocean (I wanted the server to be located in my country and Linode offers that), I was really happy to see how much it has evolved since the last time I had used it circa 2013
Linode is a fairly underrated provider. Getting acquired will hopefully bring them more visibility.

For Akamai, I wonder if this acquisition will have a negative impact on their ability to work with other providers who are now competitors. Similar to how AWS has trouble signing clients in the retail industry.

awww, fuck.

I used Slicehost for years, but they got swallowed by Rackspace and became unusable. Linode has been my home since then. I have no positive experiences with small providers who get swallowed by corporate borgs. :(

I hope nothing changes for the service. I've been a happy customer for more than a decade.
s.gif
I would start looking for a new Linode now. This is a story we all know the ending of. Akamai is nearly 10,000 people and is therefore incapable of operating at the human scale that made Linode good.
I had a sour taste from what I remember being misleading communication around very serious control plane hacks of linode.

A lot of bitcoin theft in 2012 (maybe by their own staff?)

2013 some kind of cold fusion / HTP hack

Another CF / HTP hack here.

2014 brought the MySQL server no password stuff.

2015 ish some kind of total root compromise?

You can get a feel for all this here including the denials / lack of notification.

https://news.ycombinator.com/item?id=10845985

Maybe 2016 same issue?

https://www.zdnet.com/article/cloud-firm-linode-resets-user-...

Not a company I'd put much actual production onto. Imagine if AWS had a hacker running around with total root access, able to reset MFA tokens to their own etc with no notice to customers. I'm not even sure such root access exists on AWS.

s.gif
Look, we are random internet people, and it's a "me vs. you" scenario, but as someone who worked at Linode in 2012, we were a small company, that all worked out of one office, with like 3 admins at the time. Yes, there were various hacks. Yes, there were silly vulns, but positing that one of the employees at that time stole bitcoin is something that I won't stand for.

Could it have happened, sure. Do I think that it was an inside job? No, not at all. 99% of the people there at that time thought Bitcoin was insanely dumb, and I suspect most of us still do.

s.gif
I can't edit my comment.

As I noted elsewhere, there's an opportunity to do the right thing when someone comes to you and says look, someone is coming in on the control plane and resetting my server passwords.

And yes, that includes looking at your staff especially when bitcoin is in the mix as its less traceable to a person.

For some reason, for year after year, there was this pattern. No problem, we have good security, oh wait, we've been rooted for months. Or someone is coming in on the staff admin plane and taking all sorts of action.

It could be outside hackers sure. But linode never seemed that interested in sorting things out.

The takeaway I had was that you might not notice if a staff person OR hacker was messing around.

s.gif
I’m really struggling to see how “the company’s response to security breaches was inappropriate” logically leads to “staff might be stealing Bitcoin.”

It’s a baseless and unfair attack and I think you should consider deleting your original comment.

s.gif
When you're in Linode's position, running people's private and infrastructure, you have a very finite amount of grace. Individual employees at linode had an incentive to violate the company's security. When security violations happened that could plausibly have come from employees following their incentives the company consistently failed to assure customers it was making sure the security issues were being addressed. In my opinion this at least shows a wild disregard the well-being of customers and I think it's pretty normal to treat that kind of wild-disregard as malicious (even though it may not be).
s.gif
Linode would sit on reports for months or not investigate root control plane hacks.

https://wptavern.com/wp-engine-identifies-cloud-infrastructu...

This followed the pagerduty hack. We don't know who else was getting hacked either - these were to high profile ones.

So this just raised the question - what's up that they don't take serious issues seriously? With bitcoin there have been a ton of insider issues with how "trusted" infrastructure providers and exchanges handling bitcoin so that was the question.

Even if external hackers, they just got hacked over and over.

s.gif
If the known facts are consistent with your staff stealing Bitcoin and the reason it can't be confirmed whether or not this happened is because your staff fell short of industry-standard security practices, I think it's entirely fair to say that that might be what happened. Put it this way: from the perspective of someone on the outside, if your staff were stealing Bitcoin this is exactly what it would look like.
s.gif
I wouldn't say it "logically leads to", but it does seem to be a fair question to ask.
s.gif
As someone who commented on Linode hacks earlier, I can vouch for lbotos having worked there and feel the same way as they do. I don't believe any of the attacks were an inside job, because I don't believe anyone would have done that and if they did they knew how not to leave a trail behind.

Also hey lbotos, hope you're doing well!

s.gif
Backing up both Tim and Lee here as a former who overlapped with both of them. I had many issues with Linode as an employee. The idea that anybody I worked with at the time, many of whom I don’t get along with because I was even more of an asshole then than I am now, the idea that any of them would pinch Bitcoin off a Linode is so off-base it’s laughable. It simply didn’t happen. Period. If you believe it did, your logic in getting there is no different than that of political conspiracies that are common today.

I remember that rash of Bitcoin thefts and it was all careless behavior by the Linode owner becoming a secondary consequence of a primary employee compromise, I think. As in what happened to Twitter. Think “admin panel compromised, external actor searches for Linodes known to participate in Bitcoin, methodically compromises them one by one, finds poorly stored wallets and drains them”. That intruder very obviously knew what they were after, if memory serves, but this was almost ten years ago.

Seriously. Linode did one thing well and it was hire (mostly) good people. The comms around security incidents could always use improvement, and I think that led to the loss of trust you’re seeing here. I don’t think it’s just Linode, either, I think a lot of the industry is overly discreet when it comes to what to say publicly about events like this. We see the same with journalism: a lot of methods in reporting are trade skills and most people don’t understand the news gathering process, which leaves room to fill in the gaps with conspiracy. So it is with security, too.

I’d back your speculation, Tim: there were maybe two people, definitely one, maybe two, who could both perform the crime and hide it. One’s an unsavory person to interact with if he doesn’t like you but ultimately ethical and a force for good at his core. The other runs the company. Convince me that either of them did that and you may as well convince me the Earth is flat.

s.gif
I've been on Linode for quite a few years and the support was always top notch. Never got compromised to my knowledge.
s.gif
I think in 2012 the tech community's sentiment was actually that Bitcoin was really cool and definitely useful, since it was new and did things in a practical application that we hadn't seen before. It's interesting that hindsight clouds that, it has been viewed negatively for years now but it's not that old yet. (Either way, your main point stands of course.)
s.gif
> I think in 2012 the tech community's sentiment was actually that Bitcoin was really cool and definitely useful

I don't think there was ever really a consensus on this. Lots of people (myself included, but also quite a few friends) always thought Bitcoin was just kinda useless. It's just that in 2012 there were comparatively low stakes (i.e. no massive energy use, not yet massive amounts of people pouring money in it, no massive amounts of "crypto snakeoil") that it just wasn't worth commenting on.

s.gif
I find this exceedingly hard to believe. Around 2009-2010, btc was definitelynot an "asset" (like the bafoons try to treat it now). It *was a currency*.

Many still believe in the idea that (certain, less well know) crypto can be used as a real currency, but unfortunately the public severely tainted it with ideas of 'being an asset'.

This comment reads as someone who is more aligned with the public's (HN) perception of modern crypto, rather than the use of it pre-2010.

s.gif
Okay, sure, but I don't see how this relates to the fact there was never consensus or broad agreement on Bitcoin in the tech community in general?
s.gif
I understand what you're trying to say, but I'm really curious about how you define "asset" to not include currency.
s.gif
I think the idea of “hodl” bitcoin wasn’t there. One thing that changed in my perspective is that bitcoin transactions were always destined to be more expensive than I had dreamed. In my mind, I thought transactions would be fast and free of cost. In reality, there are reportedly fewer than ten thousand full nodes.

Everyone thinks of bitcoin in terms of “how many USD is it?” I don’t know what the solution is but as long as we think of bitcoin as a perverted asset like housing - apparently people once again believe we will not allow housing prices to fall to any significant degree - there is no reason to use bitcoin as a currency. With so much speculation, the price is too volatile.

I don’t know what the solution is but I believe transaction costs should be minimal if not zero. I don’t know how we will achieve this but apparently there are other projects that try to get much closer to zero transaction fees. I think that is the future

s.gif
Not totally useless. Some of us imagined it would keep some annoying people busy for a while. Seriously, what will we think about it 5 years from now?
s.gif
I was on the mailing list the day the paper was released, and played around with the original PoC.

Lots of us thought bitcoin was dumb back then. Lots of us still do.

s.gif
There were plenty of us who knew – and said – that "cryptocurrency" was borne of technical, political, economic, societal ignorance when it started. Now it's just more obviously terrible.
s.gif
Our luckily, given the way it worked out with crypto.
s.gif
The sentiment I knew at the time was a mix of excitement at the cool new tech, skepticism of the usefulness of it (my camp), and drooling over using the GPU you already had to make easy money.
s.gif
nope, I remember starting my first tech job in 2013, and the only people in tech who cared about it were libertarians, which was a very small subgroup of tech
s.gif
It's still heavily dominated by Libertarians. They want an anarchist revolution, without the effort of actually organizing and sustaining it.
s.gif
"I think money is insanely dumb, and I suspect most of us do ... So it's ok if someone steals money."

I think I get your sentiment(?), but I'm uncertain of whether or not it matters how individuals value 'success' or 'currency' when it comes to personal property. To extend to the logical consequences, I am not sure if people on HN would agree that personal property should be non-existent.

s.gif
No, sorry that I was unclear. My point was that it's my belief that those of us employed at that time did not see value in bitcoin, so we had no motivation to steal bitcoin that would be a small fraction of what we were getting paid.

Now if $some_duder_was_really_into_bitcoin was also on the staff at the time, then sure, maybe they would risk their job to steal some bitcoin because they thought it was cool to do hax0r things with cyberpunk money. I'm not aware of that person existing.

s.gif
A 2012 Bitcoin hack victim was none other than a lead developer of Bitcoin. Back then, they ran a Bitcoin faucet on it that gave out a paltry 0.25 Bitcoin at a time.

I never bothered to jump through those hoops for like a dollar (now about US$10k):

http://gavintech.blogspot.com/2012/03/bitcoin-faucet-hacked....

He only lost 5 bitcoin (like $20 then or $200k today), but another lost 3100, or around… $124 million today:

https://bitcointalk.org/index.php?topic=66916.0

They ran a Bitcoin mining pool and this hack motivated them to create a hardware wallet:

https://blog.trezor.io/how-trezor-was-born-from-a-hacking-at...

s.gif
Wow, that some major root level compromise at linode. It's interesting how quiet they kept these things in those days.
s.gif
Linode reported it the same day:

Manager Security Incident

Ensuring the security of our platform is our top priority. We maintain a strong security policy and aim to communicate openly should it ever be compromised. Thus, we are posting to describe a recent incident affecting the Linode Manager.

Here are the facts:

This morning, an intruder accessed a web-based Linode customer service portal. Suspicious events prompted an immediate investigation and the compromised credentials used by this intruder were then restricted. All activity via the web portal is logged, and an exhaustive audit has provided the following:

All activity by the intruder was limited to a total of eight customers, all of which had references to "bitcoin". The intruder proceeded to compromise those Linode Manager accounts, with the apparent goal of finding and transferring any bitcoins. Those customers affected have been notified. If you have not received a notification then your account is unaffected. Again, only eight accounts were affected.

The portal does not have access to credit card information or Linode Manager user passwords. Only those eight accounts were viewed or manipulated -- no other accounts were viewed or accessed.

Security is our number one priority and has been for over eight years. We depend on and value the trust our customers have placed in us. Now, more than ever, we remain committed to ensuring the safety and security of our customers' accounts, and will be reviewing our policies and procedures to prevent this from ever recurring.

I won't argue Linode is blameless here, but seems like the only reason it had such an outsized impact was because the 8 customers who were targeted evidently didn't do much to protect their assets from someone gaining unauthorized access to their servers--which is always a possibility with any publicly exposed server with or without a breach of the service provider being involved.

s.gif
Take a look at this link re: pagerduty and how linode handled things there.

https://news.ycombinator.com/item?id=10845985

Doesn't it seem kind of crazy that folks get full root control plane on linode so frequently?

s.gif
Hi. This is my comment you keep linking to. Your understanding of what happened is flawed. I do not have signs that Linode was rooted in that compromise. The signals I do have is that they had their database compromised, and likely secret key material. That allowed attackers to crack the hashes offline, and then authenticate using MFA.

IMO, it's plain wrong to categorize that one as "getting full root control plane", where it was instead the compromising of individual accounts that may have had no access to the resources on an account.

s.gif
One of the many reasons every country needs more serious, standard, and mandatory public disclosure laws for cloud infrastructure breaches.
s.gif
Looking from the end user end it seems nice, but will soon be weaponized in all possible mannar, sloppily executed, and too much data to ingest.

For reference there is mandatory disclosure of (serious) data breaches in the GPDR and it's very uncommon that the disclosure actually occurs.

s.gif
Target should have had difficulty surviving as a company as a result of penalties-if-not-prison for their 2013 breach, but we see what happened there.
s.gif
That’s a little over the top, eh? They disclosed within 4 days of discovery and implemented better security controls all over. They are probably the only major retailer with chip and pin payments in the US, for example.

If you think they were unique or egregious in terms of 3rd party access to networks, i am afraid that you will find reality disappointing.

s.gif
I will say I was pleasantly surprised to discover that their store credit card comes without a magstripe on the back
s.gif
Should that same existential penalty be applied to every company who had Log4J running in prod a few months back? That was a much more widespread root compromise...
s.gif
Not comparable. Log4j was a vulnerability in the software, not leaving their shit open and and putting all their cash registers on an intranet available to the internet.

https://krebsonsecurity.com/2015/09/inside-target-corp-days-...

s.gif
The one thing I don't understand is if that many coins were stolen and every transaction is traceable shouldn't there be a trail? The owner has 124 million reasons to find those coins. Is the ability to track past transactions not as possible as it seems?
s.gif
You can track, but large amounts merge with small and disperse as they hit brokers etc. If your claim to someone is that their BTC is 0.01% stolen, it's not so strong. Faster you act, more you can do
s.gif
That makes it a little over $12,000 at the time. Which is likely why there was no big hubbub about it.
s.gif
somebody please explain to me how is it possible that the owner reports losing $124 million, then they casually mention in the reply that: no problem, I'll just cover it with my own money ...

(another recent story on ether hack had the same "resolution" the organization just chose to replaced the losses) ...

where is that money coming from? does not seem real

s.gif
That’s what it would be worth now. Was worth like 1/10000th of that back then. They probably covered it with their previously earned holdings that could have been another fraction of that.
s.gif
The 3,094 BTC stolen happened in 2012. The price of BTC in March of 2012 was ~$3-5 USD, so ~$15,500K on the high end and ~$9,300 on the low end.
s.gif
I imagine they repaid the value of the bitcoins at the time, which would have been a lot less
s.gif
Because back then the price of bitcoin was much lower. It is $124 million in today dollars, like $1,000 in back then dollars.
s.gif
I hate to say it but I recently moved away from Linode after their /64 block in Frankfurt was banned by all Google services. And even though all their kubernetes nodes have a public ipv4 address they were somehow unable to fallback on this when their ipv6 didn't work.

And when I suggested this to their support they acted like I was crazy and said there is no way to switch between ipv4 and ipv6. Well I don't work in networking but I do work for a major telco and I know our networking guys could have done that routing change, easy.

s.gif
We blocked the entire of Linode AS63949 ranges because we were getting attacked from random owned nodes and it was tripping our IDS constantly. Just got fed up with it in the end and decided to hose them.

To note, we have had problems with AWS blocking random addresses as well where we've had staff abroad.

s.gif
Same but also about a hundred others in my block list, except it's not enough because they keep on coming and new ones popping up all the time.
s.gif
Would you mind sharing?

Email address in profile

s.gif
Are you referring to the geoip issue ? https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28769

This was resolved and it didn’t appear something nefarious or any kind of ban was going on, so if you have some references I’d love to see them

s.gif
I never said it was "nefarious", I'm saying it was badly handled and it brought down all services in an entire location for days, with no fallback to ipv4, no status communication from Linode.

It's not malice, it's just incompetence.

Why the block was banned I have no idea and I don't really care, it can happen and they need to be able to handle it.

I literally had a support case asking what happened, no response, eventually I had to say "look, I'm moving now because all services are down, please tell me what is going on", no answer, I had to move to get my services back up.

s.gif
It seems to be a fault on Google's side, and not Linode's. It's like blaming the car when the road is closed down.
s.gif
Yeah but again, no one is blaming anyone for the root cause here. Only for the handling.
s.gif
I am trying to jump through hoops to move to hetzner as I can't get a clean /64 either.
s.gif
The January 2016 thing was them finally acknowledging the attack that had happened many months prior, after WPEngine gave Linode the opportunity to announce they were hacked after they were also compromised using the same vector that hit PagerDuty. If Linode had declined, WPEngine was going to do it on their behalf. I couldn't convince the powers that be to make the same demand months prior, even though I was confident, so if WPEngine hadn't pushed the issue I don't know that Linode would have ever disclosed.

But that doesn't matter anymore. This was nearly a generation ago in tech companies, and they are now part of a bigger one.

s.gif
Yeah, this key issue.

https://wptavern.com/wp-engine-identifies-cloud-infrastructu...

How do you sit on breaches like this for months and cause all this downline pain to your customers is crazy.

s.gif
You can’t just imply that the linode staff assisted or were involved in stealing crypto currency from their customers without actually providing any evidence.
s.gif
Fair point, I can't edit my comment.

My metric though is this: Someone has admin level access to full root on control plane and does things to customer accounts they don't want. A customer complains that the control plane has been used to reset things / asks for logs / etc.

That's your opportunity to identify a root level control plane hack, disclose it, do the password reset things, remediate and move forward.

OR - You don't disclose things till much later, you deny things, you provide scrubbed logs. Add in the bitcoin angle where this type of story is pretty common (trusted entity runs away with coins because they are easy to steal) - and couple that with how they handled the reports -> as I said in my comment, I'd be careful putting sensitive info onto that platform.

s.gif
I ended up dropping Linode as a result of the hacks, after having been a customer for years, both because of the nature of the hacks, and how they communicated.
s.gif
this worry me, any similar compromise in digital ocean for example?
s.gif
Not really a compromise but there was a point where they weren’t wiping block devices by default: https://news.ycombinator.com/item?id=6983097

Not sure if that’s changed. (Hopefully it has!)

s.gif
jeez, a lot has changed since 2013.

DO's the only one out of the major VPS players to go public. It's not amateur hour over there anymore.

s.gif
This was the same era that Linode had the compromises listed. It's a pretty apples-to-apples comparison, though I agree it has been almost a decade since.
Oh, snap. I am a longtime Linode customer. Somehow I fell disturbed by this news. Is there a valuable alternative out there? Just in case something happens with pricing or other things. You know what I mean.
s.gif
I'm going to start looking, for sure. Just assume Akamai will ruin Linode within 24-36 months and plan accordingly.
s.gif
I'm with the other guy, a couple years max. In the meantime maybe digital ocean or vultr? I'm not sure how you're using linode but you might give those two a gander.
Congrats Linode. Been a user since they launched. Even the diehards (like me) have moved a lot to AWS and competitors. Lindode's biz feels stuck in the previous decade, and I think the $900MM purchase price reflects that. Hopefully the founders had a great exit and move on to create new amazing things! Best of luck to the Linode team.
s.gif
Interesting, I disagree but appreciate your perspective. AWS and competitors are just getting way too expensive and they're focusing so much on higher-level products. Some people just need a VM and a DB, and don't want to have to higher an expert just to estimate how much it will cost every month and pray they don't have something go viral that uses up a bunch of egress data.

I don't see bigger companies moving yet (most of them are building their own infra on top of OpenShift and VMware), but I know a few small companies that are.

s.gif
Check out spot instances. An example of how much more competitive and rapidly scaleable aws and friends are vs the old model.
s.gif74 more comments...

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK