Vision-guided Music Source Separation via a Fine-grained Cycle-Separation Network

Music source separation from a sound mixture remains a big challenge because there often exist heavy overlaps and interactions among similar music signals. In order to correctly separate mixed sources, we propose a novel Fine-grained Cycle-Separation Network (FCSN) for vision-guided music source separation. With the guidance of visual features, the proposed FCSN approach preliminarily separated music sources by minimizing the residual spectrogram which is calculated by removing preliminarily separated music spectrograms from the original music mixture. The separation is repeated several times until the residual spectrogram becomes empty or leaves only noise. Extensive experiments are performed on three large-scale datasets, the MUSIC (MUSIC-21), the AudioSet, and the VGGSound. Our approach outperforms state-of-the-art approaches in all datasets, and both separation accuracies and visualization results demonstrate its effectiveness for solving the problem of overlap and interaction in music source separation.