Overview
In multiparty conversation, most speech does not hand over the floor. Pauses, backchannels, brief interjections and overlap can look like transitions without being one. DualTurn splits the problem into two stages, inspired by dual-process theory.
Discover
A fast, recall-oriented process scans the audio continuously and proposes possible transition points.
Verify
A selective process checks each candidate with acoustic, linguistic, future-activity, speaker-change, ranking and agreement cues.
Localize
Accepted candidates become floor-transfer events, matched one-to-one to the reference within a time tolerance.
Three evaluation tasks
Supplied-point classification
A decision time is given. Is the next event a SHIFT or a HOLD?
Candidate verification
Is an automatically proposed candidate a genuine floor transfer?
Continuous discovery
Find and localize floor transfers on the timeline, matched at ±0.50, ±0.70 or ±1.00 s.
Turn-taking events
VoxConverse provides diarization only. Events are derived from adjacent speaker segments using a 0.06 s overlap threshold and a 0.50 s incoming-duration threshold.
| Event | Pattern | Label |
|---|---|---|
| Normal turn transfer (NTT) | New speaker, no substantial overlap, speaks > 0.50 s | SHIFT |
| Interruption (ITT) | New speaker, overlap ≥ 0.06 s, speaks > 0.50 s | SHIFT |
| Backchannel (BC) | New speaker, overlap ≥ 0.06 s, speaks ≤ 0.50 s | HOLD |
| Brief interjection (BI) | New speaker, no substantial overlap, speaks ≤ 0.50 s | HOLD |
| Continuation | Same speaker continues | HOLD |
Results
VoxConverse, official TEST partition (232 sessions)
| Analysis | Metric | Result |
|---|---|---|
| Task A: supplied-point classification | Macro-F1 | 0.849 |
| Task B: candidate verification | Event F1 | 0.455 |
| Task C: continuous discovery, ±0.50 s | Event F1 | 0.552 |
| Task C: continuous discovery, ±0.70 s | Event F1 | 0.597 |
| Task C: continuous discovery, ±1.00 s | Event F1 | 0.620 |
System 1 proposes 80,012 TEST candidates covering 6,092 of 6,348 reference transfers (0.960). At ±0.50 s, Task C reaches P = 0.536 and R = 0.569.
Multi-TPC (seven triadic groups, 23 sessions)
| Protocol | Task A | Task B | C, ±0.50 s | C, ±1.00 s |
|---|---|---|---|---|
| Group-disjoint held-out | 0.500 | 0.563 | 0.260 | 0.445 |
| Seven-group nested (mean ± SD) | 0.572 ± 0.045 | 0.607 ± 0.021 | 0.343 ± 0.040 | 0.504 ± 0.050 |
The seven-group protocol evaluates a revised system and adds robustness evidence; it is not a second held-out test of the same model.
Reproduce
Eight notebooks reproduce the reported numbers: five for VoxConverse and three for Multi-TPC. Setup, data layout and run order are in the repository.
Citation
@misc{patamia2026dualturn,
title = {Leveraging Dual-Process Theory for Multimodal Multiparty
Turn-Taking Prediction},
author = {Patamia, Rutherford A. and Liu, Ming and Luo, Wei and
Ekong, Favour and Mohammed, Rehan and Liu, Gang and Cosgun, Akan},
year = {2026},
note = {Submitted to IEEE Transactions on Audio, Speech and Language Processing, 2026}
}