PROJECT OVERVIEW
Pickford set out to pioneer a next-generation, AI-driven television broadcast designed for live streaming on Twitch. The goal was to showcase how artificial intelligence could enable real-time, audience-responsive storytelling as a new form of media. Robot Sea Monster was engaged to architect and build the core technical systems that would bring this vision to life.
the results
Robot Sea Monster built a platform that combined a custom Unity-based rendering engine with a real-time, AI-driven storytelling system. Using pre-generated combinatorial assets and dynamically generated scripts, the platform brought colorful scenes to life during live Twitch broadcasts. Behind the scenes, multiple AI agents coordinated to create and interpret the script scene-by- scene, adjusting story flow based on viewer sentiment, and keeping the narrative aligned with familiar storytelling frameworks like the heroʼs journey. As audiences interacted through chat and emotes, their feedback shaped the unfolding story—unlocking new branches, triggering rewinds, and shifting plot points in real time.
PROJECT OVERVIEW
Lack of Tools for Dynamic, Sentiment-Responsive Storytelling
No existing system could create and adapt narrative storylines in real time based on live audience sentiment.
Traditional engines weren’t designed to link sentiment to specific elements within structured narrative progression.
The client needed a way to align viewer feedback with narrative milestones such as turning points in the hero’s journey and dynamic character and story content without manual intervention.
Limited Real-Time Scripting and Rendering for Broadcast Use
Off-the-shelf tools lacked the control and flexibility needed to choreograph scenes, characters, and environments based only on a Hollywood-style script input.
The rendering system required deep scriptability to sync visuals with evolving narrative states.
Existing pipelines weren’t designed for AI-driven scene transitions or the pacing needed for interactive, broadcast-quality storytelling.
Challenges Translating Live Audience Input Into Narrative Decisions
Twitch APIs provided raw data, but no framework for converting viewer interaction into structured narrative decisions.
Sentiment analysis had to run continuously and be mapped to specific characters, scenes, and plot branches.
The system needed to incorporate changes like rewinds or writing entire alternate paths based on real-time audience feedback in a seamless way that didn’t break the flow of the broadcast.
the results
System Design Approach
The platform was designed with a clear separation between content generation and visual rendering. Script generation and audience feedback were handled by a network of AI agents, while the rendering system focused on interpreting structured JSON inputs and assembling final video output using Unity. This modular approach supported scalability and future integration with more advanced content creation and audience interaction pipelines.
Custom Unity Rendering Engine
We built a broadcast-capable rendering system in Unity, designed to process scripted scenes using a structured JSON format. Our engine was responsible for orchestrating the final output animating characters, environments, cameras, and audio in sync with story progression, and streaming high-quality, Hollywood-style visuals directly to a live Twitch broadcast.
AI-Driven Narrative Engine
We designed a completely new storytelling engine powered by a network of AI agents. These agents created a story that followed structured narrative frameworks (like the hero’s journey) using branching trees, logic flows, and vector memory, while also monitoring audience response and adapting the story in real time.
Real-Time Sentiment Integration via Twitch
We created an AI based system to determine not only the sentiment of Twitch chat (including emotes and viewer behavior), but also the specific elements of the story the sentiment applied to. This sentiment in aggregate directly influenced the story, triggering new plot branches, rewinding the story to try something different, or making subtle forward-based directional shifts based on emotional engagement.
While experimental in scope, the Pickford AI Narrative Engine demonstrated that live audience sentiment can directly influence the flow of AI-driven narratives in real time. The project delivered a compelling proof of concept for interactive, AI-assisted storytelling and laid the groundwork for new formats in live-streamed media. The client was pleased with the outcome, and key components, particularly the rendering engine and integration layer which are well-positioned for reuse in future projects.
