Multiparty turn-taking · VoxConverse · Multi-TPC

DualTurn: Leveraging Dual-Process Theory for Multimodal Multiparty Turn-Taking Prediction

Rutherford A. Patamia1, Ming Liu1, Wei Luo1, Favour Ekong2, Rehan Mohammed1, Gang Liu3, Akan Cosgun4
1Deakin University · 2Griffith University · 3Harbin Engineering University · 4Arm

Listen to the system at work

Five VoxConverse TEST recordings with the final system's real output. Press play and the timeline follows the audio.
VoxConverse TEST

10 s
Tolerance
Space play/pause · ←/→ 1 s · Shift+←/→ 0.1 s · +/− zoom
Reference floor transfer Missed transfer System 1 candidate (height = System 2 score) Above the 0.45 threshold Correct prediction False alarm
Transfers0
Predicted0
TP0
FP / FN0 / 0
P / R0 / 0
Event F10

Reference transfers are derived from the official RTTM labels with the paper's event rules. Candidate bar height is the System 2 score (threshold 0.45). Final predictions come after timing correction and a 0.5 s cooldown, so they can sit slightly off the candidate that produced them. Audio and labels: VoxConverse (Chung et al., 2020), CC BY 4.0.

Overview

In multiparty conversation, most speech does not hand over the floor. Pauses, backchannels, brief interjections and overlap can look like transitions without being one. DualTurn splits the problem into two stages, inspired by dual-process theory.

System 1

Discover

A fast, recall-oriented process scans the audio continuously and proposes possible transition points.

System 2

Verify

A selective process checks each candidate with acoustic, linguistic, future-activity, speaker-change, ranking and agreement cues.

Output

Localize

Accepted candidates become floor-transfer events, matched one-to-one to the reference within a time tolerance.

Three evaluation tasks

A

Supplied-point classification

A decision time is given. Is the next event a SHIFT or a HOLD?

B

Candidate verification

Is an automatically proposed candidate a genuine floor transfer?

C

Continuous discovery

Find and localize floor transfers on the timeline, matched at ±0.50, ±0.70 or ±1.00 s.

Turn-taking events

VoxConverse provides diarization only. Events are derived from adjacent speaker segments using a 0.06 s overlap threshold and a 0.50 s incoming-duration threshold.

EventPatternLabel
Normal turn transfer (NTT)New speaker, no substantial overlap, speaks > 0.50 sSHIFT
Interruption (ITT)New speaker, overlap ≥ 0.06 s, speaks > 0.50 sSHIFT
Backchannel (BC)New speaker, overlap ≥ 0.06 s, speaks ≤ 0.50 sHOLD
Brief interjection (BI)New speaker, no substantial overlap, speaks ≤ 0.50 sHOLD
ContinuationSame speaker continuesHOLD

Results

VoxConverse, official TEST partition (232 sessions)

AnalysisMetricResult
Task A: supplied-point classificationMacro-F10.849
Task B: candidate verificationEvent F10.455
Task C: continuous discovery, ±0.50 sEvent F10.552
Task C: continuous discovery, ±0.70 sEvent F10.597
Task C: continuous discovery, ±1.00 sEvent F10.620

System 1 proposes 80,012 TEST candidates covering 6,092 of 6,348 reference transfers (0.960). At ±0.50 s, Task C reaches P = 0.536 and R = 0.569.

Multi-TPC (seven triadic groups, 23 sessions)

ProtocolTask ATask BC, ±0.50 sC, ±1.00 s
Group-disjoint held-out0.5000.5630.2600.445
Seven-group nested (mean ± SD)0.572 ± 0.0450.607 ± 0.0210.343 ± 0.0400.504 ± 0.050

The seven-group protocol evaluates a revised system and adds robustness evidence; it is not a second held-out test of the same model.

Reproduce

Eight notebooks reproduce the reported numbers: five for VoxConverse and three for Multi-TPC. Setup, data layout and run order are in the repository.

Citation

@misc{patamia2026dualturn,
  title  = {Leveraging Dual-Process Theory for Multimodal Multiparty
            Turn-Taking Prediction},
  author = {Patamia, Rutherford A. and Liu, Ming and Luo, Wei and
            Ekong, Favour and Mohammed, Rehan and Liu, Gang and Cosgun, Akan},
  year   = {2026},
  note   = {Submitted to IEEE Transactions on Audio, Speech and Language Processing, 2026}
}