|
|
Through a series of connections I know a guy that knows a guy that works at Microsoft that was made aware and the changes have been reverted. Give 'er 30 minutes TTL ;)
|
|
 |
|
192.168.1.1 is gone now, but all authoritative nameservers are still offering 192.168.1.0. Oops.
|
|
|
|
A few are dropping 192.168.1.0 now: as of 1703035296:
ns1-39.azure-dns.com no longer has 192.168.1.0 for microsoft.com
1.1.1.1 still has 192.168.1.0 for microsoft.com
8.8.8.8 still has 192.168.1.0 for microsoft.com
76.76.2.0 no longer has 192.168.1.0 for microsoft.com
9.9.9.9 still has 192.168.1.0 for microsoft.com
208.67.222.222 still has 192.168.1.0 for microsoft.com
185.228.168.9 still has 192.168.1.0 for microsoft.com
76.76.19.19 still has 192.168.1.0 for microsoft.com
94.140.14.14 still has 192.168.1.0 for microsoft.com
|
|
|
|
Only one of those is authoritative. All of the authoritative servers have dropped it. Microsoft has fixed the issue.
|
|
|
|
|
30 minutes minutes or 30 Windows minutes? :P
|
|
|
|
Actually, it’s looking more like 6 days. No wait, 30 seconds.
|
|
|
|
I see them both. My TTL will run out at 16:39 PST, though.
|
|
|
|
TTL appears to be set to an hour. But either way, its been 45 min and the primary ns1-39.azure-dns.com is still offering up 0.1
|
|
|
|
|
|
This isn’t something that I think should be diluted. If it’s that simple for a stray record to be included in the dns round robin it could have been bad if it was an external ip with a machine setup by a phisherman especially since control of a domain is all you need to get an ssl cert now. Couple this with the fact that it’s Microsoft, one of the most relied on companies in our computer world, this is pretty darn horrible.
|
|
|
|
Microsoft also has some of the phishiest looking domains when you are redirected around the O365 cloud.
|
|
|
|
100%. Starting with "onmicrosoft.com". A phisher wouldn't really have to control Microsoft.com to take advantage of confusion.
|
|
|
|
The only thing that competes is the redirecting when you log into any health portal.
|
|
|
|
For all this to work you need to control the domain.
Is that easier than simply breaking into their systems and owning their servers?
|
|
|
|
That's exactly what they're saying. > it could have been bad if it was an external ip with a machine setup by a phisherman I.e. one of the IPs for microsoft.com belongs to $phisher, which means they control (a subset of the traffic going to) the domain. They can't add CNAME records for certificate validation, but LetsEncrypt for example offers HTTP-based validation. Not sure how Microsoft sets up their certificate pinning, it might not be quite that easy.
|
|
|
|
|
I was getting an empty answer for microsoft.com. Turns out my dnsmasq is blocking it: $ dig microsoft.com. | grep EDE
; EDE: 15 (Blocked)
resolver.log:Dec 20 00:43:57 router dnsmasq[8172]: possible DNS-rebind attack detected: microsoft.com
|
|
|
|
Wait wait wait wait. Bunny.net accidentally changed their DNS to 127.0.0.1 and took a bunch of their CDN users down today too. Coincidence? Weird day.
|
|
|
|
microsoft.com is currently IPv6-only on my network, because OpenWrt's DNS rebinding protection filters out the A records: $ ping -4 microsoft.com
ping: microsoft.com: Address family for hostname not supported
$ ping -6 microsoft.com
PING microsoft.com(2603:1030:c02:8::14 (2603:1030:c02:8::14)) 56 data bytes
64 bytes from 2603:1030:c02:8::14 (2603:1030:c02:8::14): icmp_seq=1 ttl=112 time=68.4 ms
|
|
|
|
|
I'm trying to figure out how this could have happened, but I control so few IP addresses that many of my DNS entries are manually assigned. And you'd have to be incompetent if you have access to set DNS records and you set them to RFC 1918 addresses. Anyone have any theories on how this could happen?
|
|
|
|
I'll go with Joseph Conrad on this one. "It's only those who do nothing that make no mistakes, I suppose." Now the persons that did it have some proof that they did something. They will surely put some check in place because there should be another adage somewhere that says that you only learn to use the handrails after you fell in the stairs.
|
|
|
|
|
It continually astounds me that DNAME got standardized. Scary stuff.
|
|
|
|
* Copy/Paste * Copilot told me * Sabotage (internal or external)
|
|
|
|
|
|
Certainly just some automation bug, perhaps a few things strung together, like a dev environment setup that leaked into production. A human in the loop making a mistake probably as well. This is the kind of thing you look at and put up a few guardrails to prevent it happening again.
|
|
|
|
They probably asked copilot to manage their DNS servers
|
|
|
|
that's what happens when you buy address space from the back of a van in the parking lot ;)
|
|
|
|
|
Potential timeouts for clients/workstations trying to reach microsoft.com. Which entry is picked for use is generally random depending on the client. Most systems will retry using another entry though on issues connecting through. That said, if you are on a network that is 192.168 based, trying to get to Microsoft.com may just send you to your local router!
|
|
|
|
|
|
I get it too, with .1.0 as well Name: microsoft.com
Address: 192.168.1.1
Name: microsoft.com
Address: 192.168.1.0
"ooopsie!"
|
|
|
|
1.1 is gone but I'm still seeing the 1.0 entry.
|
|
|
|
|
For the uninitiated, can some traffic get sent to 192.168.1.1. Is it round robin?
|
|
|
|
|
I could be wrong but in my experience OS just selects one random and uses that for some time not round robins it.
|
|
|
|
Even if that is the case, if it is random, some section of DNS would send traffic to it. Maybe it was OK because most resolvers would ignore the local address on the list??
|
|
|
|
for uninitiated (me), why is it bad?
|
|
|
|
Well in my case (and a lot of other people), 192.168.1.1 is the local address of my home router. So if I go to microsoft.com I have a 1 in 7 chance of getting my home router instead (if I ignore the certificate warning). Other random breakage will happen depending on what that local address is assigned to for you. In theory this could be leveraged for hacking, but I think that would require setup in advance.
|
|
|
|
Since 2 out of the 7 IPs are 192.168 (private ips), 2/7 visitors to microsoft.com will load the private ones assumign equal weight and not get the page to load.
|
|
|
|
|
So if you go to microsoft.com with probability 1/7 you'll hit 1.1 on your private network of 192.168 - likely router, and with probability 1/7 you'll hit 1.0 maybe printer
|
|
|
|
What could a TLA do with this if it had time to plan ahead?
|
|
|
|
Serve malicious updates from a locally controlled machine, for one. Lord knows about auth.
|
|
|
|
> Serve malicious updates from a locally controlled machine. Lord knows about auth. Wouldn't they have to break into my local machine first, plant an update service, and an update? That doesn't seem to scale well at all, and wouldn't it be easier to just break into the machine they want to 'update'?
|
|
|
|
A fairly prominent update service already runs from the domain microsoft.com Many machines come with it preinstalled.
|
|
|
|
The erroneous DNS change wouldn't help that sort of exploit. It just redirects attempts to contact microsoft.com to a local address, probably a router.
|
|
|
|
Do most DNS forwarders not block addresses that resolve to a local IP these days? I know dnsmasq does, and NextDNS too I think.
|
|
|
|
|
|
Why? Having local IPs on a public DNS is a legitimate use case.
|
|
|
|
As another reply mentioned, to prevent DNS rebinding attacks. The general expectation is you will whitelist domains from which you expect RFC1918 responses.
|
|
|
|
In fact, some people block domains by routing them to 127.0.0.1 in their host files. I've used private ranges too, in places where loopback might possibly do something funky.
|
|
|
|
Why doesn't windows update use authentication (eg https)?
|
|
|
|
They do: the updates are signed so our hypothetical spies would need to have a zero day in Authenticode or to have compromised the signing keys.
|
|
|
|
|
Good point. Let's watch Microsoft executives & employees closely for signs of panicking over an escaped AGI.
|
|
|
|
So let me see if I understand. With this DNS record, if me or Windows tries to hit “microsoft.com” there’s a 1/7 chance it hit my router instead?
|
|
|
|
|
Wait... Can DNS resolvers be configured so that RFC1918 is respected? I mean: I don't expect anything less from Microsoft than doing stuff like that and it cannot affect me for I nullroute microsoft.com from my unbound server (unboud takes wildcard when nullrouting or NXDOMAINing crap domains like microsoft.com or meta.com etc., which is sweet). However I'd expect my trusty DNS resolver to also prevent me from anyone not on my private LANs to impersonate addresses reserved for private uses. Does anyone know here if it's easily doable?
|
|
|
|
You're looking for DNS rebinding protection, many DNS servers support it. However there are some cases where things do use private IPs in DNS records outside of the local domain, one example is Plex (e.g. https://support.plex.tv/articles/206225077-how-to-use-secure... suggests turning off DNS rebinding protection) -- although in some cases you can allow particular domains which is a much better way than turning it off entirely. (See also the sibling comment about microsoft.com being IPv6 only as a result of a particular implementation of DNS rebinding protection: https://news.ycombinator.com/item?id=38704159)
|
|
|
|
|
My Unbound servers strip RFC out. Public resolvers keep DNS answers intact because they can carry alt data like how dodgy a SMTP server is.
|
|
|
|
An entry-level admin is now unemployed, just before the holidays.
|
|
|
|
> An entry-level admin is now unemployed, just before the holidays. I highly doubt that entry-level admins at Microsoft have access to DNS for their primary domain. My guess is that this incident is a lot more interesting than that.
|
|
|
|
Nah, if it's already reverted, they're good to go. A post-mortem with how something like that got through will definitely be on the table though.
|
|
|
|
I hope not. Failures are on a spectrum and this was unfortunate but probably not malicious. All things considered this should be a lesson learned. There should be more failsafe mechanisms in place so juniors can fail safely and learn from them. The absolute worst thing we can do is shame an individual so they don’t attempt to try new things in fear of ridicule.
|
|
|
|
> There should be more failsafe mechanisms in place so juniors can fail safely and learn from them. And if not, whoever put the junior in that role is the person responsible for the problem.
|
|
|
|
I'm wondering how such a change would get "merged" in to begin with. I imagine even non-network engineers would get this huge itch having a large corporate contain a private IP in the changelist (I'm the non network engineer and can't really explain why it's bad. But it FEELS wrong and sometimes you at least need to use instinct to get another pair of eyes on something).
|
|
|
|
The seniors all go on leave and the interns are left to run the place. If they fired the juniors the seniors would have to come back from holiday!
|
|
|
|
|
|
|
Y'all, instead of the constant confirmed here. Just do an authoritative lookup. dig +trace +short microsoft.com NS a.root-servers.net. from server 100.100.100.100 in 10 ms. NS b.root-servers.net. from server 100.100.100.100 in 10 ms. NS c.root-servers.net. from server 100.100.100.100 in 10 ms. NS d.root-servers.net. from server 100.100.100.100 in 10 ms. NS e.root-servers.net. from server 100.100.100.100 in 10 ms. NS f.root-servers.net. from server 100.100.100.100 in 10 ms. NS g.root-servers.net. from server 100.100.100.100 in 10 ms. NS h.root-servers.net. from server 100.100.100.100 in 10 ms. NS i.root-servers.net. from server 100.100.100.100 in 10 ms. NS j.root-servers.net. from server 100.100.100.100 in 10 ms. NS k.root-servers.net. from server 100.100.100.100 in 10 ms. NS l.root-servers.net. from server 100.100.100.100 in 10 ms. NS m.root-servers.net. from server 100.100.100.100 in 10 ms. RRSIG NS 8 0 518400 20240101050000 20231219040000 46780 . fG/YHtUJu3YMAm9Mlzzvp3xG4UCPG01aYNnlyF1HfAHdZpR+L88CVUcz NFHq9M45KjB7ZTlSFt2JvEyK/8FcavZLOthkXRREbJQswjLCbhiPQCbq tQLF+tKaNYUihqawCfjgZy1i5YwYjmphbjfzwoKo1POtepf0YCIcuLBi nQFw4Lr79O6cjyg6qlYnqaK6z4Xi5qt6ocohJafjs86LuuRo2WvmJ1IK k0ZUoAC6Qyjz4MVhqHMvQGdp7EnzjoL8Y9PTXeUuD6Ixp/Aklj2psLjD TZDPYN1q+zDd1giFyuwNRX9DG1zrxzN2lzQiLWmGKrzP3DvFWL1L2Ts1 FWjy/Q== from server 100.100.100.100 in 10 ms. ;; UDP setup with 2001:502:7094::30#53(2001:502:7094::30) for microsoft.com failed: network unreachable. ;; UDP setup with 2001:502:7094::30#53(2001:502:7094::30) for microsoft.com failed: network unreachable. ;; UDP setup with 2001:502:7094::30#53(2001:502:7094::30) for microsoft.com failed: network unreachable. A 20.112.250.133 from server 150.171.10.39 in 20 ms. A 20.231.239.246 from server 150.171.10.39 in 20 ms. A 20.76.201.171 from server 150.171.10.39 in 20 ms. A 20.70.246.20 from server 150.171.10.39 in 20 ms. A 20.236.44.162 from server 150.171.10.39 in 20 ms. A 192.168.1.0 from server 150.171.10.39 in 20 ms.
|
|
|
|
Or from a bunch of dnses: $ export srch="192.168.1.0"; echo "as of $(date '+%s';):"; for dns in 1.1.1.1 8.8.8.8 76.76.2.0 9.9.9.9 208.67.222.222 185.228.168.9 76.76.19.19 94.140.14.14; do dig @${dns} microsoft.com +short | grep "${srch}" > /dev/null; if [ $? == 0 ]; then echo "${dns} still has ${srch} for microsoft.com"; else echo "${dns} no longer has ${srch} for microsoft.com"; fi; done
as of 1703033639:
1.1.1.1 still has 192.168.1.0 for microsoft.com
8.8.8.8 still has 192.168.1.0 for microsoft.com
76.76.2.0 still has 192.168.1.0 for microsoft.com
9.9.9.9 still has 192.168.1.0 for microsoft.com
208.67.222.222 still has 192.168.1.0 for microsoft.com
185.228.168.9 still has 192.168.1.0 for microsoft.com
76.76.19.19 still has 192.168.1.0 for microsoft.com
94.140.14.14 still has 192.168.1.0 for microsoft.com
$ pbpaste | sed 's;^; ;' | pbcopy
|
|
|
|
I don’t know man, putting microsoft.com on your router sounds like a massive reduction in latency. Congrats on the achievement.
|
|
|
|
|
|
How the hell did that pass any sort of responsible review process at Microsoft? Now Microsoft owns all your home networks, only like the default address on every home router out there...
|
|
|
|
You have the danger of this backwards: this is a very bad security problem for Microsoft, and not a problem for people outside of MS (except to the extent that we're all indirectly reliant on MS being secure). Pointing a domain at an IP address does not give you any power over than IP address, and you can point a domain at anything you want.
|
|
|
|
Any risk here is nearly the opposite of what you seem to think it is
|
|
|
|
|
No they don’t. Going to Microsoft.com will take you to your router.
|
|
|
|
> Now Microsoft owns all your home networks Only if you’re slumming around 192.168.x.x
|
|
|
|
I for one only use Class A CIDR, 10.0.0.0/8
|
|
|
|
No, it's that when you open microsoft.com it could open your router page.
|
|
|
