ChatDirector: Enhancing Video Conferencing with Space-Aware Scene Rendering and Speech-Driven Layout Transition

Initial full-view scene of ChatDirector (b) Conversations with Alice (c) Conversations between Alice and Bob (d) Full-view scene when Charlie speaks to all

Figure 1: Screenshots of ChatDirector, captured from the local user, Sean's laptop during a remote meeting with Alice (left), Bob (center), and Charlie (right). (a) Using an off-the-shelf laptop or workstation equipped with an RGB camera, ChatDirector depicts remote participants as 3D portrait avatars and renders them in a shared virtual meeting environment. Sean starts his progress update to the team. (b) When Sean inquires about a feature update from Alice, ChatDirector recognizes the speech activity and automatically focuses the camera on Alice, facilitating a more personal one-on-one discussion. (c) Later, Bob steps in and asks Alice further questions, ChatDirector arranges their avatars in a pairwise layout and simulates direct eye contact by orienting their 3D avatars towards each other. (d) When Charlie updates his progress to everyone, the camera is zoomed out with other avatars turning to Charlie, to provide Sean with a visual cue of the speech transition.