很紅的 Stable Diffusion 是寫一串文字 (prompt) 然後產生圖片，而 Riffusion 則是寫一串文字產生音樂。

其中 prompt 轉成音樂其實還在可以預期的範圍 (i.e. 遲早會出現)，但專案的頁面上解釋了 Riffusion 是基於 Stable Fusion 的作品，而且是利用 Stable Fusion 產生出時頻譜 (spectrogram)：

Well, we fine-tuned the model to generate images of spectrograms, like this:

也就是像這樣的圖：

在 Hacker News 上討論時的討論頁可以看看，作者有參與一些討論：「Riffusion – Stable Diffusion fine-tuned to generate music (riffusion.com)」。

其中有人提到這個作法超出想像，因為輸出的圖片只要幾個 pixel 差一點點就會產生出很不同的聲音：

This really is unreasonably effective. Spectrograms are a lot less forgiving of minor errors than a painting. Move a brush stroke up or down a few pixels, you probably won't notice. Move a spectral element up or down a bit and you have a completely different sound. I don't understand how this can possibly be precise enough to generate anything close to a cohesive output.

Absolutely blows my mind.

然後其中一位作者回覆到，他也是做下去後才很意外發現居然可行：

Author here: We were blown away too. This project started with a question in our minds about whether it was even possible for the stable diffusion model architecture to output something with the level of fidelity needed for the resulting audio to sound reasonable.

實際上聽了產生出來的音樂，是真的還 OK 的音樂... 大家都完全沒想到可以這樣搞，然後在 Hacker News 上的 upvote 數量爆炸高 XD

To respond on your own website, enter the URL of your response which should contain a link to this post's permalink URL. Your response will then appear (possibly after moderation) on this page. Want to update or remove your response? Update or delete your post and re-enter your post's URL again. (Learn More)