Current video-to-music (V2M) models lack semantic control and fail to penalize instruction violations, largely due to their reliance on reconstruction objectives and the representational bottleneck of static cross-modal conditioning in Diffusion Autoregressive (DAR) architectures. To resolve this, we introduce VIBE, a novel text-and-video-to-music (T+V2M) generation model that leverages: (1) Conditioning Connection, a depth-wise cross-layer conditioning mechanism that dynamically bridges the planning and diffusion refinement heads and (2) a comprehensive reward modeling taxonomy, optimizing for both hard, verifiable constraints (e.g., tempo, key) and soft, subjective qualities (e.g., musicality, multimodal alignment) with a structured 5-stage training curriculum. Upon evaluation using audio-visual alignment, instruction following, and audio quality metrics, along with a subjective human evaluation study, we observe that VIBE demonstrates enhanced controllability and instruction adherence while performing comparably to most evaluated baselines on generation fidelity and multimodal alignment.
Comparison with state-of-the-art models
Sample 1 of 6
VIBE OursVideo + Text
Text prompt
A dark, atmospheric Trap track at 150 BPM in B Minor, featuring a deep, resonant 808 bass, crisp trap drums with tight kicks, snappy snares, and rapid hi-hat rolls in a half-time feel, layered with moody synth pads and a haunting bell-like melody, evoking a brooding, introspective mood that shifts from energetic momentum to subdued melancholy, rooted in a static, hypnotic B Minor tonal center with no chord changes, creating a searching, melancholic atmosphere that aligns with the dark trap sub-genre's cinematic, urban aesthetic.
The same video scored by five other systems
CMTVideo-only
GVMGenVideo-only
VidMuseVideo-only
Video2MusicVideo-only
M2UGenGeneric text
Sample 2 of 6
VIBE OursVideo + Text
Text prompt
A high-energy Trap-Hip-Hop track at 93.75 BPM in F minor, featuring a deep, resonant 808 sub-bass, crisp trap drums with rapid hi-hat rolls, and a bright, melodic synth lead that carries the main hook, layered with a serene and awe-inspiring atmosphere that contrasts the energetic beat, evoking a sense of wonder and tranquility while serving as an ambient backdrop to a nighttime spectacle, blending modern technological elements with universally accessible instrumentation.
The same video scored by five other systems
CMTVideo-only
GVMGenVideo-only
VidMuseVideo-only
Video2MusicVideo-only
M2UGenGeneric text
Sample 3 of 6
VIBE OursVideo + Text
Text prompt
A dark, atmospheric electronic track at 76.9 BPM in C Major with a subtle lean toward minor tonalities, featuring evolving synth pads, a deep resonant synth bass, a sparse electronic drum pattern with soft kick, crisp snare, and subtle hi-hats, and a clean, immersive stereo mix that emphasizes gentle compression and dynamic breathing, conveying an introspective and slightly ominous mood with a hypnotic, steady 4/4 groove that remains detached and non-synchronized, evoking late-night cityscapes and contemplative visual media through a minimalist, genre-blurring fusion of ambient-electronic and downtempo trip-hop sensibilities.
The same video scored by five other systems
CMTVideo-only
GVMGenVideo-only
VidMuseVideo-only
Video2MusicVideo-only
M2UGenGeneric text
Sample 4 of 6
VIBE OursVideo + Text
Text prompt
An ambient electronic track at ~81 BPM in E Major, featuring layered evolving synth pads, a deep resonant synth bass, and a simple repetitive synth melody, creating a serene, introspective atmosphere with a calm and meditative flow, devoid of emotional depth or dynamic contrast, serving as a universally accessible backdrop that evokes a sense of spacious stillness and quiet reflection.
The same video scored by five other systems
CMTVideo-only
GVMGenVideo-only
VidMuseVideo-only
Video2MusicVideo-only
M2UGenGeneric text
Sample 5 of 6
VIBE OursVideo + Text
Text prompt
A relaxed, atmospheric Lo-fi Hip-Hop track at ~83 BPM in F♯ minor, featuring a deep, smooth synth bass, a laid-back drum-machine groove, warm sustained synth pads, and a bright, bell-like synth melody in a subtle minor-key progression (F♯m → C♯m → Amaj7 → Dmaj7 → C♯m7 → F♯m7) that drifts across the stereo field, creating a calm, contemplative mood with a dreamy, introspective vibe, evoking a peaceful late-night atmosphere while subtly layering a sense of melancholy and introspection over its chillwave-inspired, low-fi production.
The same video scored by five other systems
CMTVideo-only
GVMGenVideo-only
VidMuseVideo-only
Video2MusicVideo-only
M2UGenGeneric text
Sample 6 of 6
VIBE OursVideo + Text
Text prompt
A modern ambient-electronic track at ~96.8 BPM in C♯ Major, featuring lush, evolving synth pads, a delicate arpeggiated synth line, and a deep, sustained synth bass, blending ethereal textures with a subtle downtempo groove to evoke a serene, introspective, and slightly wistful East Asian Lunar New Year ambiance, underlining a restrained cultural reverence and traditional reverence without mirroring the visual exuberance, with a calm and contemplative atmosphere layered over a gentle harmonic journey through major-sixth chords and a brief C♯ minor shift, creating a meditative, immersive soundscape rooted in the 2010s-2020s ambient-electronic tradition.
The same video scored by five other systems
CMTVideo-only
GVMGenVideo-only
VidMuseVideo-only
Video2MusicVideo-only
M2UGenGeneric text
Instruction following
Each block holds the input video fixed and varies only the text instruction. The clip on
the left is the silent input; each clip beside it is the same footage scored under a
different instruction.
A deeply sad, melancholic piano ballad ..., quiet sorrow ...
Love
A warm, romantic love theme ..., gentle heartfelt melody ...
Input videoCute dogSame video, 3 prompts
Sad
A deeply sad, melancholic piano ballad ..., quiet sorrow ...
Love
A warm, romantic love theme ..., gentle heartfelt melody ...
125 BPM
An uplifting melodic electronic track at 125 BPM ...
Input videoDogs at playSame video, 2 prompts
Happy
A bright, joyful, upbeat acoustic-pop track ..., pure happiness and fun ...
Motivational
A powerful, motivational cinematic anthem ..., inspiring triumphant build ...
Input videoTaj MahalSame video, 3 prompts
Sad
A deeply sad, melancholic piano ballad ..., quiet sorrow ...
Happy
A bright, joyful, upbeat acoustic-pop track ..., pure happiness and fun ...
Motivational
A powerful, motivational cinematic anthem ..., inspiring triumphant build ...
Method overview
The five training stages
Stage
What it adds
Data
Examples
1Music generation pre-training
Text and music only, no video. Aligns the projectors.
JamendoMaxCaps
~1.6M
2Text-to-music instruction SFT
Instruction-rich captions with tempo and key annotations.
MusicBench
52,768
3Text-to-music preference optimization
DiffusionNFT against CMI-RM plus the hard tempo and key rewards.
CMI-Pref
3,527
4Text plus video instruction SFT
Adds the frozen CLIP encoder and a projection layer trained from scratch.
V2M + HarmonySet
60,000
5Omni-modal preference optimization
DiffusionNFT again, with Qwen2.5-Omni judging video-music alignment too.
V2M + HarmonySet
8,000
Mel-spectrogram comparison
VIBEVideo + Text
CMTVideo-only
GVMGenVideo-only
VidMuseVideo-only
Video2MusicVideo-only
M2UGenGeneric text
Results
All results are reported on ReelBench. The best result among the generated systems is
shown in bold. Ground truth is included for reference and is
excluded from the ranking.
Audio quality and diversity
FAD (↓ lower is better)
3022.5157.50
8.7522
3.5729
2.3022
22.6459
4.5767
1.511
1.5829
CMTGVMGenVidMuseVideo2MusicM2UGenVideo-RobinVIBE
FD (↓ lower is better)
1007550250
37.7945
16.2638
14.5385
73.067
27.4208
10.902
10.1282
CMTGVMGenVidMuseVideo2MusicM2UGenVideo-RobinVIBE
KL (↓ lower is better)
21.510.50
1.7329
1.5573
1.3194
1.8839
1.5301
1.2556
1.3205
CMTGVMGenVidMuseVideo2MusicM2UGenVideo-RobinVIBE
IS (↑ higher is better)
32.251.50.750
1.2243
1.7085
1.4549
1.0233
1.6499
2.0586
2.3578
CMTGVMGenVidMuseVideo2MusicM2UGenVideo-RobinVIBE
IB (↑ higher is better)
0.160.120.080.040
0.1119
0.0957
0.1233
0.0473
0.0722
0.1017
0.0888
CMTGVMGenVidMuseVideo2MusicM2UGenVideo-RobinVIBE
Density (↑ higher is better)
10.750.50.250
0.1084
0.0835
0.1377
0.1647
0.1094
0.1384
0.6741
CMTGVMGenVidMuseVideo2MusicM2UGenVideo-RobinVIBE
Coverage (↑ higher is better)
0.80.60.40.20
0.0614
0.3881
0.5213
0.0084
0.2761
0.5259
0.6095
CMTGVMGenVidMuseVideo2MusicM2UGenVideo-RobinVIBE
Exact figures
Model
FAD↓ (lower is better)
FD↓ (lower is better)
KL↓ (lower is better)
IS↑ (higher is better)
IB↑ (higher is better)
Density↑ (higher is better)
Coverage↑ (higher is better)
Ground truth
-
-
-
-
0.1417
0.9900
0.8800
Video only
CMTDi et al., 2021
8.7522
37.7945
1.7329
1.2243±0.0147
0.1119
0.1084
0.0614
GVMGenZuo et al., 2025
3.5729
16.2638
1.5573
1.7085±0.0281
0.0957
0.0835
0.3881
VidMuseTian et al., 2025
2.3022
14.5385
1.3194
1.4549±0.0281
0.1233
0.1377
0.5213
Video plus auxiliary inputs
Video2MusicKang et al., 2024
22.6459
73.0670
1.8839
1.0233±0.0014
0.0473
0.1647
0.0084
M2UGenLiu et al., 2024
4.5767
27.4208
1.5301
1.6499±0.0567
0.0722
0.1094
0.2761
Text plus video
Video-RobinLokegaonkar et al., 2026
1.5110
10.9020
1.2556
2.0586±0.0472
0.1017
0.1384
0.5259
VIBEours
1.5829
10.1282
1.3205
2.3578±0.0798
0.0888
0.6741
0.6095
VIBE attains the best FD, Inception Score, Density and Coverage among all evaluated
systems. The marginal regression in FAD, KL and ImageBind score relative to Video-Robin is
expected: reinforcement finetuning trades distributional proximity for generative
diversity and instruction adherence. This is reflected in Coverage, which rises from 0.526
to 0.610, and in Density, which rises from 0.138 to 0.674.
Following the instruction
Tempo accuracy, exact (↑ higher is better)
3022.5157.50
22
23.2
24.2
Video-RobinVIBE w/o S5VIBE
Tempo accuracy, octave-equivalent (↑ higher is better)
403020100
29.7
29.6
31.4
Video-RobinVIBE w/o S5VIBE
Key accuracy, exact (↑ higher is better)
43210
2.7
2.6
3.1
Video-RobinVIBE w/o S5VIBE
Key accuracy, loose (↑ higher is better)
1612840
10.1
9.9
10.9
Video-RobinVIBE w/o S5VIBE
Tempo error, MAE in BPM (↓ lower is better)
403020100
33.43
32.77
32.42
Video-RobinVIBE w/o S5VIBE
Exact figures
Metric
Video-Robin
VIBE without stage 5
VIBE
Tempo accuracy, exact↑ (higher is better)
22.0
23.2
24.2
Tempo accuracy, octave-equivalent↑ (higher is better)
29.7
29.6
31.4
Key accuracy, exact↑ (higher is better)
2.7
2.6
3.1
Key accuracy, loose↑ (higher is better)
10.1
9.9
10.9
Tempo error, MAE in BPM↓ (lower is better)
33.43
32.77
32.42
Absolute values are low across all systems, and exact key identification in particular
remains difficult for every model evaluated. The informative comparison is therefore a
relative one: VIBE improves over Video-Robin on all five metrics and over its own Stage 5
ablation on all five, which isolates omni-modal preference tuning, rather than the
architecture alone, as the source of the gain. Tempo is scored correct within 10 percent
of the target; the loose key criterion additionally accepts relative and parallel keys.
Human A/B testing results
Each cell is the win rate of the row model against the column model: the percentage of
paired comparisons in which listeners preferred the row model's output over the column
model's, so 50 is parity and above 50 is a win for the row.
Audio quality
Audio quality: percentage of comparisons the row model won against the column model
wins vs
CMT
GVM
M2U
V2M
VdM
V-R
VIBE
CMT
51
57
38
34
30
29
GVMGen
49
43
37
42
23
21
M2UGen
43
57
41
27
34
32
Video2Music
62
63
59
49
30
28
VidMuse
66
58
73
51
48
46
Video-Robin
70
77
66
70
52
30
VIBE
71
79
68
72
54
70
Musicality
Musicality: percentage of comparisons the row model won against the column model
wins vs
CMT
GVM
M2U
V2M
VdM
V-R
VIBE
CMT
47
59
40
40
25
24
GVMGen
53
39
37
38
33
31
M2UGen
41
61
41
29
38
38
Video2Music
60
63
59
37
34
32
VidMuse
60
62
71
63
43
41
Video-Robin
75
67
62
66
57
47
VIBE
76
69
62
68
59
53
Video-music alignment
Video-music alignment: percentage of comparisons the row model won against the column model
wins vs
CMT
GVM
M2U
V2M
VdM
V-R
VIBE
CMT
51
45
57
28
17
16
GVMGen
49
43
53
36
25
23
M2UGen
55
57
55
27
31
30
Video2Music
43
47
45
29
27
25
VidMuse
72
64
73
71
41
39
Video-Robin
83
75
69
73
59
31
VIBE
84
77
70
75
61
69
Overall
Overall: percentage of comparisons the row model won against the column model
wins vs
CMT
GVM
M2U
V2M
VdM
V-R
VIBE
CMT
53
53
50
34
19
18
GVMGen
47
43
44
34
30
28
M2UGen
47
57
45
27
31
30
Video2Music
50
56
55
29
30
28
VidMuse
66
66
73
71
41
39
Video-Robin
81
70
69
70
59
35
VIBE
82
72
70
72
61
65
Win rate20%80%
Gemini omni-judge scores
Rhythm (↑ higher is better)
43.532.52
2.49
2.622
2.786
2.816
M2UGenVideo2MusicVideo-RobinVIBE
Theme (↑ higher is better)
43.532.52
2.731
2.744
2.89
2.91
M2UGenVideo2MusicVideo-RobinVIBE
Emotion (↑ higher is better)
43.532.52
3
2.744
3.1
3.028
M2UGenVideo2MusicVideo-RobinVIBE
Culture (↑ higher is better)
43.532.52
3.255
3.293
3.565
3.581
M2UGenVideo2MusicVideo-RobinVIBE
Temporal (↑ higher is better)
32.752.52.252
2.143
2.268
2.472
2.589
M2UGenVideo2MusicVideo-RobinVIBE
Instr. fit (↑ higher is better)
43.532.52
3.1
3.09
3.2
3.24
M2UGenVideo2MusicVideo-RobinVIBE
Overall (↑ higher is better)
43.532.52
2.724
2.677
2.753
2.843
M2UGenVideo2MusicVideo-RobinVIBE
Exact figures
Model
Rhythm
Theme
Emotion
Culture
Temporal
Instr. fit
Overall
M2UGen
2.490
2.731
3.000
3.255
2.143
3.100
2.724
Video2Music
2.622
2.744
2.744
3.293
2.268
3.090
2.677
Video-Robin
2.786
2.890
3.100
3.565
2.472
3.200
2.753
VIBE
2.816
2.910
3.028
3.581
2.589
3.240
2.843
Gemini is employed as an omni-modal judge, observing the video and the generated music and
scoring seven axes of audio-visual alignment. VIBE obtains the highest score on six of the
seven axes and on overall alignment; Video-Robin retains emotion alignment. An automatic
judge is not a substitute for human assessment, which is why the A/B study above is
reported alongside it.
Acknowledgements
Research at the University of Maryland is partially supported by Adobe, Amazon, NVIDIA
and Sesame.
Please cite the paper as below. If you build on the released code, please also consider
citing VoxCPM (arXiv:2509.24650), which VIBE extends, along with the projects listed
under Acknowledgements.
@misc{bhosale2026vibevideoinstructionalignedbackground,
title={VIBE: Video Instruction-aligned Background music gEneration},
author={Aryan Vijay Bhosale and Vaibhavi Lokegaonkar and Vishnu Raj and Gouthaman KV and Sreyan Ghosh and Ramani Duraiswami and Lie Lu and Dinesh Manocha},
year={2026},
eprint={2608.30125},
archivePrefix={arXiv},
primaryClass={cs.SD},
url={https://arxiv.org/abs/2608.30125},
}
