Ask HN: Is there an ML model that can go from an audio song to sheet music?

uptownfunk | 25 points

I was going to post basic pitch from Spotify but it looks like billconan beat me to it. That said I can give you a bit more advice. The Spotify basic pitch model isn't too good at multi-track input. It's capable of it, but you may actually get better results if you separate out the tracks first and then run them individually through the basic pitch model.

In order to do this you can use a source/stem separation model like spleeter (https://github.com/deezer/spleeter) and then run the basic pitch model (or any other midi transcription model). There's other you can try which may yield better results, for example: (https://github.com/Music-and-Culture-Technology-Lab/omnizart)

Either way the key words you want to be looking for are "midi transcription" and "stem separation", should help you find more models to try for both steps. Good luck! :)

EDIT: Oh it looks like there's even a stem separation leaderboard on papers with code, neat: https://paperswithcode.com/task/music-source-separation

ikhatri | a year ago

Ableton Live has something that can convert audio to midi.

Then there are open source programs that can convert midi to sheet music.

I have tried the process, but you tend to get artifacts in the midi from Ableton.

You have to know a little bit about music theory to clean it up.

tmaly | a year ago

This is called (multitrack) music transcription. There are some commercial solutions (AudioScore, AnthemScore, ...). For OSS, look at Omnizart [1] and magenta/mt3 [2].

I suppose these models are trained on western / pop music, so they may not work nicely on ethnic music.

[1] https://github.com/Music-and-Culture-Technology-Lab/omnizart [2] https://github.com/magenta/mt3

CrypticShift | 10 months ago

I've been working on training models for this problem. Haven't shipped as a product yet because the inference cost is too high (takes 10-15 minutes to transcribe each song). Would love to try it out on the music you're trying to learn and get your feedback. Email address is in my profile.

radq | a year ago

https://www.celemony.com/en/melodyne/what-is-melodyne

This might help. Not exactly what you're looking for.

ndstephens | a year ago

If someone's seriously working on this, I would be happy to help out and build the mobile apps around it.

Imagine how cool, useful, and joyful this tech would be to most people.

moomoo11 | a year ago