Working Collision? · Issue #1 · AsuharietYgvar/AppleNeuralHash2ONNX · GitHub
source link: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Can you verify that these two images collide?
Here's what I see from following your directions:
$ python3 nnhash.py NeuralHash/model.onnx neuralhash_128x96_seed1.dat beagle360.png 59a34eabe31910abfb06f308 $ python3 nnhash.py NeuralHash/model.onnx neuralhash_128x96_seed1.dat collision.png 59a34eabe31910abfb06f308
Yes! I can confirm that both images generate the exact same hashes on my iPhone. And they are identical to what you generated here.
Copy link
prusnak commented 5 hours ago
@dxoigmn Can you generate an image for any given hash (preimage attack) or do you need access to the source image first (second preimage attack)?
Copy link
tmechen commented 4 hours ago
If I'm not mistaken a preimage attack was done here
hmmm
"This is so fake it's not even funny. These are just images generated by the model from https://thisartworkdoesnotexist.com . It's hilarious to see so many people falling for it here" (in the comments)
@fuomag9 Interesting, but one ethical question remains -- how did they obtain the CSAM hashes? I was under the impression that the NeuralHash outputs of the NCMEC images were not readily available. This strongly suggests that the authors must have obtained child pornography, hashed it, and then generated spoofs.
"This is so fake it's not even funny. These are just images generated by the model from https://thisartworkdoesnotexist.com . It's hilarious to see so many people falling for it here"
Or they used images generated by that site as the starting point. As I noted in my previous comment, it's impossible to know without having the NeuralHash NCMEC database.
This is not only a collision, it is a pre-image, which breaks the algorithm even more.
Collison:
Find two random images with the same hash.
Pre-image:
Find an image with the same hash as a known, given image.
Copy link
fuomag9 commented 4 hours ago
@fuomag9 Interesting, but one ethical question remains -- how did they obtain the CSAM hashes? I was under the impression that the NeuralHash outputs of the NCMEC images were not readily available. This strongly suggests that the authors must have obtained child pornography, hashed it, and then generated spoofs.
If I'm not mistaken the DB with the hashes is stored locally and you can extract it from the iOS 15 beta
Holy Shit.
@erlenmayr No, this is likely a second preimage attack. In a preimage attack you're just given the hash, not the image.
@fuomag9 so the list of known CP hashes is shipped on every device? Isn't this a huge security issue?
This was untrue. See this comment.
@fuomag9 so the list of known CP hashes is shipped on every device? Isn't this a huge security issue?
This is from apple's pdf on the technical details of their implementation. Feel free to correct me but from my understanding the blinded hash is the CSAM hash DB
Edit: I was wrong and we cannot extract them
Wow. Couldn't they have hashed the image locally, send it to the server and compare it there?
My bad. The hashes on the device have gone through a blinding process, so my train of thought was not correct. Does anyone know why they chose to compare it on the client instead of on the server, seeing as it is only ran on images uploaded to iCloud?
I knew someone already did this :P
@dxoigmn Can you generate an image for any given hash (preimage attack) or do you need access to the source image first (second preimage attack)?
Not sure. My sense is that seed1 seems to be mixing bits from the output of model. But we already know from the literature it is very likely. At least, I am reasonably confident one could generate a noisy gray image that outputs some desired hash value.
@fuomag9 Interesting, but one ethical question remains -- how did they obtain the CSAM hashes? I was under the impression that the NeuralHash outputs of the NCMEC images were not readily available. This strongly suggests that the authors must have obtained child pornography, hashed it, and then generated spoofs.
No one has obtained the CSAM hashes, AFAIK. This repo is just a neural network that takes an input and produces a hash. It's as if someone released a new hash algorithm. (What you do with those hashes is a different story and the comparison of the hash with CSAM hashes is a whole other system.) This just shows that an image of a dog has the same hash value as a noisy gray image. That is, ask the model for the hash of the dog image, then ask the model how to change a gray image to make it output the same hash as the dog image. So it's a second-preimage, which we know has to be possible (by the pigeonhole principle); but it's really just a matter of feasibility. The interesting thing about neural network models (when compared to cryptographic hashes) is that they give you a gradient which tells you how you can change stuff to optimize some objective. This is the basis of deep dream, adversarial examples, and even just plain old training of neural network models.
If I'm not mistaken a preimage attack was done here
hmmm
"This is so fake it's not even funny. These are just images generated by the model from https://thisartworkdoesnotexist.com . It's hilarious to see so many people falling for it here" (in the comments)
From what I understood, the story the HN comment is referring to is AI generated porn, which can match as "real" porn. Aka the "send dunes" story: https://petapixel.com/2017/12/20/uk-police-porn-spotting-ai-gets-confused-desert-photos/
This issue here, however, is a hash collission. This is huge, if confirmed. Really ugly stuff.
Does anyone know why they chose to compare it on the client instead of on the server, seeing as it is only ran on images uploaded to iCloud?
If they compare on the server then Apple is able to know the outcome of the comparison even if there is only a single match. The idea is that Apple can only know that a certain user has matching images in their iCloud account as soon as a certain threshold number of matches is reached. That's my understanding as a non-expert in this field.
Does anyone know why they chose to compare it on the client instead of on the server, seeing as it is only ran on images uploaded to iCloud?
If they compare on the server then Apple is able to know the outcome of the comparison even if there is only a single match. The idea is that Apple can only know that a certain user has matching images in their iCloud account as soon as a certain threshold number of matches is reached. That's my understanding as a non-expert in this field.
I think they also inject false positives for results on device side, so they don't know if there is only a single match. They know only when threshold is reached. It was in the PSI paper.
This issue here, however, is a hash collission. This is huge, if confirmed. Really ugly stuff.
Every practical hashing algorithm is one-to-many. It is only a matter of time when collisions come. This repo contains the model from older release and we don't really know how they have tweaked parameters on Apple side or even how the huge data set is improving the accuracy. NN has many layers so training is rather important.
The goal of the algorithm is to seek matches from specific kind of material (CSAM), and that is used for training. Can we expect that it calculates hashes with similar accuracy for all kind of images? (e.g is it equally hard to make collision for CSAM material than the picture of the dog?)
This issue here, however, is a hash collission. This is huge, if confirmed. Really ugly stuff.
Every practical hashing algorithm is one-to-many. It is only a matter of time when collisions come. This repo contains the model from older release and we don't really know how they have tweaked parameters on Apple side or even how the huge data set is improving the accuracy. NN has many layers so training is rather important.
Let me quote from this article of someone who can explain this better than me:
https://www.hackerfactor.com/blog/index.php?/archives/929-One-Bad-Apple.html
In the six years that I've been using these hashes at FotoForensics, I've only matched 5 of these 3 million MD5 hashes. (They really are not that useful.) In addition, one of them was definitely a false-positive. (The false-positive was a fully clothed man holding a monkey -- I think it's a rhesus macaque. No children, no nudity.)
According to NCMEC, I submitted 608 reports to NCMEC in 2019, and 523 reports in 2020. In those same years, Apple submitted 205 and 265 reports (respectively). It isn't that Apple doesn't receive more picture than my service, or that they don't have more CP than I receive. Rather, it's that they don't seem to notice and therefore, don't report.
Let that sink in.
Now back to our issue:
While hash detection is a really bad idea in general for such purposes, the specific implementation from apple does not matter, because it is closed source. You literally don't know what apple is doing, with what data, and what result comes out of it.
Even if apples specific algorithm is the best in the world and does not have these drawbacks: You would not know. You would have to live in constant fear of your phone "thinking" you may are doing something wrong. That's scary.
Even if apples specific algorithm is the best in the world and does not have this drawbacks: You would not know. You would have to live in constant fear of your phone "thinking" you may are doing something wrong. That's scary.
That is very true. We can either give total trust or nothing at all on closed systems.
@fuomag9 so the list of known CP hashes is shipped on every device? Isn't this a huge security issue?
It's not. It's encrypted using elliptic-curve cryptography first. They keep the actual CSAM neuralhash db on their server.
Why do they do that? A) To prevent reversing the hashes, as mentioned. B) So they can derive the encryption keys for the image locally, which is what the blinded hash table is also used for. I'm leaving out a lot of details that are mentioned in the whitepaper they released, so read that.
@weskerfoot Yeah I edited my other comment with an explanation of the blinded hash.
all we need now is a hash of what is considered 'csam' to iphone, and start making images collide with it,
if we could somehow make collisions with arbitrary content, (eg. like a funny meme that someone would be likely to save to there phone) that would be great-
I'm not sure how to get a hash without having such a file to hash it though ..
Oh just curious, since the hashing is done on the client side, would it be possible to tell iCloud that the hash matched every time? and if so, what's stopping you from just flooding it with random shit?
Oh just curious, since the hashing is done on the client side, would it be possible to tell iCloud that the hash matched every time? and if so, what's stopping you from just flooding it with random shit?
In fact it already does that by design, in order to obscure how many matching images there are before the threshold is crossed where it can decrypt all matching images
So if you can figure out a way to generate synthetic matches on demand, you could make it think there are lots of matches, but it would soon discover they're "fake" once the threshold is crossed, since it wouldn't be able to decrypt them all. Not sure what it would do if you did that repeatedly, maybe it would cause issues.
Edit: to be clear, the inner encryption key is associated with the NeuralHash. So if you have a false positive NeuralHash output, it would trigger manual review, but you would need to actually have that, which is why they keep them a secret.
Copy link
indiv0 commented 1 hour ago
I wonder if this code can be ported so you can generate the collision an Android device...
can we take any given image and then make its hash totally different, despite the image looking basically identical?
just curious if this even works for the one thing its suppost to do.. haha
Copy link
fuomag9 commented 1 hour ago
taking a given image and then make its hash totally different, despite the image looking basically identical. can we do that?
just curious if this even works for the one thing its suppost to do..
cropping the image seems to work since the algorithm (at least what we have now) is vulnerable to that
cropping the image seems to work since the algorithm (at least what we have now) is vulnerable to that
ah yes, because pedos are being so sophisticated to use end to end encryption these days that we need to backdoor everyone's phones, because think of the children but there NOT sophisticated enough to ..checks notes.. crop images,
The cropping sensitivity is likely due to aliasing in convolutional neural networks. [1] is a starting point into that line of research. By starting point I really mean: read the related works and ignore the actual paper.
The cropping sensitivity is likely due to aliasing in convolutional neural networks. [1] is a starting point into that line of research.
So, your saying if you just alias the fuck out of the image it wont be detected ?
can this thing even handle like basic shit like JPEG-ing a png ?
taking a given image and then make its hash totally different, despite the image looking basically identical. can we do that?
just curious if this even works for the one thing its suppost to do..cropping the image seems to work since the algorithm (at least what we have now) is vulnerable to that
Technically though, the cropped image could end up with a colliding hash with another CSAM image, couldn't it?
Technically though, the cropped image could end up with a colliding hash with another CSAM image, couldn't it?
i guess its possible, (though, so could cropping any image, right?)
what are the chances of that happening? can someone whos smarter than me calculate the odds? :?
Technically though, the cropped image could end up with a colliding hash with another CSAM image, couldn't it?
i guess its possible, (though, so could cropping any image, right?)
what are the chances of that happening? can someone whos smarter than me calculate the odds? :?
An illegal image containing something like a vase that gets cropped only to the vase (which by itself it's not illegal) and then gets sent to you as an innocuous image could be considered illegal since it'd match part of a bigger picture?
An illegal image containing something like a vase that gets cropped only to the vase (which by itself it's not illegal) and then gets sent to you as an innocuous image could be considered illegal since it'd match part of a bigger picture?
now imagine another scenario, say you happen to have the same/simular looking vase, and happen to put it on a simular/same table with simular/same looking background and took a photo of it.\ would your completely unrelated but similar looking image now also be flagged?
Apple had good intentions but Jesus this is not the way. I prefer how things currently are. Not this. Fals positives are a real thing and also I don't want a fucking bitcoin mining ai cringe daemon running amock chowint my battery.
A goal for someone producing a second-preimage-image is to construct one that appears to be some arbitary innocuous image. An actual attacker wouldn't use an innocuous image, but would likely instead use nude photographs-- as they would be more likely to be confused for true positives for longer. A random innocuous image would be a reasonable POC that it could also be done with nudes.
Comments above that the target database would be needed by an attacker are mistaken in my view. The whole point of the database they're claiming to use is that it contains millions of highly circulated abuse images. An attacker could probably scan through darknet sites for a few hours and find a collection of sutiable images. They don't have to know for sure exactly what images in the database: Any widely circulated child abuse image is very likely to be in the database. If they obtain, say, 50 of them and use them to create 50 matching second-preimage-images then it should also be likely get the 30 or so hits required to trigger the apple infrastructure.
A real attacker looking to frame someone isn't going to worry too much that the possessing the images to create an attack is a crime-- so would the framing. That fact will just stand in the way of researchers producing POC to show that the system is broken and endangers the public.
I think it's important to understand that even though second-preimage-image attacks exist, concealing the database does not meaningfully protect the users-- as real attackers won't mind handling widely circulated child porn. The database encryption serves only to protect Apple and its (partially undisclosed!) list sources from accountability, and from criticism-via-POC by researchers who aren't out to break the law.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK