VioTech — Frame-Level Violin Playing Technique Detection with Multi-Scale Mixture-of-Experts

3-Minute Talk

A three-minute overview of VioTech for ISMIR 2026. Narration generated with Gemini text-to-speech. Download MP4

Abstract

Automatic detection of violin playing techniques is central to computational performance analysis, yet most systems first segment audio into notes and then classify each note — a cascade whose onset/offset errors propagate downstream. VioTech detects violin playing techniques directly at the frame level (10 ms / frame), removing the note pre-segmentation stage entirely. It predicts three technique dimensions in parallel — tonal technique (none / pizzicato / harmonics), articulation (none / spiccato), and legato (bow-change vs. sustained, framed as a binary regression problem). The proposed model couples an HRPT transcription backbone (providing onset/offset/frame cues) with a frozen MERT-v1-330M acoustic backbone, on top of which N multi-scale temporal experts (default N = 8, receptive fields 20 ms–2560 ms) are routed by per-task gating. On a cross-dataset (RWC) test set, the Mixture-of-Experts routing yields large gains on the expressive, less frequent classes (pizzicato, harmonics, spiccato), reaching a macro accuracy of 86.5%.

Highlights

⏲

Frame-level & cascade-free

Technique labels are predicted every 10 ms with no note onset/offset segmentation stage, avoiding cascading errors.

🧠

Multi-Scale Mixture-of-Experts

8 temporal experts over a frozen MERT-v1-330M backbone, each with its own MERT layer weights and receptive field (20 ms–2560 ms).

🎯

Per-task gating

Independent dense soft routing for tonal, articulation, and legato — revealing which time scales matter for each technique.

🎧

Legato as regression

The first computational model of legato bow-change timing, via a Gaussian-peak binary regression target.

📜

VioTech120 dataset

120 manually annotated violin solo etudes (Paganini Op.1, Wohlfahrt Op.35, Kayser Op.20) with a dedicated annotation interface.

🔄

Transcription cues help

Jointly fine-tuned onset/offset/frame cues from the HRPT backbone measurably improve technique detection.

Method

VioTech three-stage architecture: Feature Extraction (Log-Mel + frozen MERT), Transcription Cue Generation (frame-level CRNN + GRU producing onset/frame/offset), and Mixture-of-Experts Prediction (N experts routed by a gate network to a weighted sum for tonal/articulation/legato prediction). — The three-stage VioTech pipeline: **Feature Extraction** (Log-Mel spectrogram + frozen MERT), **Transcription Cue Generation** (frame-level CRNNs + GRUs producing onset/frame/offset cues), and **Mixture-of-Experts Prediction** (N temporal experts routed by a gate network into a weighted sum for tonal / articulation / legato).

Results

Per-class accuracy on the RWC cross-dataset test set. Columns: Tonal none/pizz/harm (TN/TP/TH), Articulation none/spiccato (AN/AS), Legato sustained/bow-change (LS/LB).

Model	TN	TP	TH	AN	AS	LS	LB	Macro
VioTech (Ours)	79.07	89.60	88.84	97.78	97.16	96.49	56.73	86.52
w/o MoE	85.98	40.85	69.62	94.43	61.41	98.72	68.96	74.28
w/o Layer Weight	86.11	93.68	65.44	98.74	83.66	97.00	57.65	83.18
w/o Transcription Cue	79.83	95.28	83.28	99.57	64.51	97.00	64.21	83.38

The MoE routing yields large gains on the expressive, less frequent classes (pizzicato, harmonics, spiccato), especially under cross-dataset transfer.

Per-class accuracy

Dataset:

Task-level metrics

Dataset:

Expert routing — which time scales each task prefers

Global gate weights (%) over the 8 temporal experts (M0 = 20 ms … M7 = 2560 ms). Legato relies on mid-to-long scales, tonal on short-to-mid scales — a direct, interpretable readout of the model.

Dataset:

VioTech120 Dataset

120annotated etudes

55k+labelled notes

10 msframe resolution

3technique dimensions

Coming Soon...