An AI-Native Desktop Where Outcomes Become Software
FromYouProduct Development
When we set out to ask what a personal computer would look like if it were designed for the agent era rather than retrofitted to it, we built FromYou, an "infinite creation" desktop that hides code behind a visual workspace, lets a multi-agent orchestrator route work between specialized models, and materializes user-described software inside a sandboxed WebContainer in real time. The result is a host environment where AI is not a sidecar feature but the way the operating layer itself behaves: users describe outcomes, the agent bar selects tools, apps appear, media persists, desktops can be published and visited, and the workspace travels with the user across devices.
Industry
Consumer AI / Personal Computing
Timeline
Internal R&D: Phased orchestrator migration
FromYou
Today's "AI-powered" desktops are conventional operating systems with a chat sidebar bolted on; the underlying interaction model, files, windows, manual app installation, still assumes a human is doing the work.
A user who wants to build something today is asked to coordinate a fragmented set of tools, a code editor, a terminal, a package manager, a hosting provider, a database console, an AI chat in a separate tab, a media generator in another, and a notes app to track what they meant to do. Each context switch is a tax. We wanted to collapse that surface into a single workspace where the user describes the outcome, the system selects the tools, and the artifacts, apps, files, media, layout, persist as a coherent desktop rather than as scattered tabs.
Rebuilding the Desktop for the Agent Era
The Product Discovery Gap
Users could not describe what they wanted in the language of files and components, the things AI tools demanded. The gap between "I want a place where my friends and I can leave each other notes" and the seven-step recipe of scaffolding, installing, wiring, and deploying was where most ideas died. We needed an interface that accepted intent as the unit of input.
Workflow Fragmentation
The status quo for a creator-developer was five-plus open tabs at any moment: editor, terminal, hosting dashboard, AI chat, asset generator. State lived nowhere consolidated, and every new project re-opened the same tabs. The desktop itself had to become the integration layer instead of yet another window inside it.
Single-Agent Ceiling
Our first cut routed every turn through one large model with the full tool belt. It worked, but conversational responses stalled behind long-running tool calls, prompts ballooned with irrelevant tool descriptions, and a single failure in any tool path would break the whole turn. We needed specialization without losing the feel of one assistant.
Cross-Device Persistence
A desktop that only exists in one browser tab is not a desktop. But the workspace contained a live filesystem, an app registry, generated media, and window/icon layout, none of which fits cleanly into a single document. Restoring a session had to feel instant despite shipping a compressed snapshot of an entire user environment.
Sandbox Fidelity
The apps users build needed to actually run, install packages, hit APIs, render real UI, without us trusting arbitrary code on the host page. WebContainer gave us the sandbox, but seeding it from a snapshot, keeping it in sync with the app registry, and validating the template after each edit added a layer of build-system discipline most chat products never need.
Memory Coherence
Across long sessions, agents needed a shared, durable picture of what had already been done, files touched, apps installed, media generated, prior context summaries, without each turn re-discovering the world. Without that, every long-running task became a stateless re-explanation, and tokens went to waste.
User-First, AI-Native Development
Discovery & Research
- •Mapped the real "build something on a Tuesday night" workflow and counted the tab-switches and dead-ends, that count, not a feature list, became our north star
- •Interviewed creator-developers about where existing AI tools broke down: long tool calls blocking chat, context evaporating between sessions, and assets getting orphaned across services
- •Studied LangGraph-style orchestration patterns and existing WebContainer deployments to identify which primitives we could borrow and which we'd have to build ourselves
Architecture Design
- •Chose Next.js 15 + React 19 as the host so we could ship a server-rendered shell while embedding a fully client-side WebContainer iframe for the user workspace
- •Selected Convex as the persistence layer for typed, reactive backend state, chat threads, messages, agent state, app registry, so that every agent could read and write the same source of truth without bespoke sync code
- •Selected R2 for the heavy artifacts (compressed desktop snapshots) and kept Convex pointers to R2 keys, so we paid object-storage prices for blobs and database prices for queries
- •Adopted a LangGraph-style orchestrator pattern with a classifier and specialized sub-agents (ChatResponder, Builder, DesktopEnhancer, MediaAgent, MemoryManager) instead of one monolithic agent with the full tool belt
- •Standardized on the Vercel AI SDK for streaming + tool-calling, with grok-code-fast-1 as the current model for fast tool-heavy turns
- •Kept the existing tool-invocation API (web_fs_*, web_exec, code_edit_ast, ai_generate, etc.) stable so the WebContainer client could keep operating while we swapped orchestrator internals behind it
Core Technical Decisions
1. Multi-Agent Orchestration Over a Single Super-Agent
A root controller ingests each user turn, runs a lightweight classifier to produce { intent, confidence, evidence, slots }, and dispatches work to the specialized agent best suited to it. We traded the simplicity of one prompt for leaner per-agent contexts, parallel long-running work, and failure isolation, if MediaAgent errors, ChatResponder still talks to the user.
2. Shared, Typed AgentState Keyed by Thread
Rather than re-deriving context every turn, we persist one Convex document per thread containing intent, task queue, files-touched, apps-touched, media assets, validation status, per-agent scratchpads, and a memory-checkpoint pointer. We traded write amplification for handoff clarity: any agent reads one document and knows what's already been done.
3. WebContainer-as-Workspace, Not as Feature
The user's actual desktop, files, app registry under public/apps/registry.json, generated apps under src/apps/*, lives inside a WebContainer iframe seeded from a binary snapshot in templates/webcontainer. We traded raw boot speed for true execution fidelity: the apps users build actually run, install packages, and serve real UI rather than rendering a mock.
4. Snapshot-and-Restore Over Per-File Sync
Desktop state (including layout JSON written to /public/_fyos/desktop-state.json on save) is gzipped, uploaded to R2, and indexed in desktops_private. Restore is a single fetch + hydrate on boot, with IndexedDB as a warm-cache fallback and the default snapshot as a cold-start fallback. We traded fine-grained sync for restoration that's a single round-trip and survives partial outages.
5. Persona Translator on Top of Tool Output
Tool calls produce dense, technical text; the user shouldn't see it. The orchestrator pipes raw tool output through a persona pass that surfaces what changed in the user's language ("works on mobile and desktop") rather than the implementation's ("responsive flexbox"). We traded output throughput for an interface that doesn't read like a developer console.
6. Feature-Flagged Orchestrator Rollout
We kept the original single-agent path live behind a flag while migrating to the orchestrator, and we mirrored token usage and tool logs across both paths so we could compare parity before decommissioning. We traded migration speed for the ability to roll back instantly if any specialized agent regressed.
Iterative Development Phases
- 1.Foundations: Convex schema for AgentState; extracted prompt fragments per planned agent in src/lib/prompts.ts; built classifier prompt and response schema
- 2.Single-loop orchestrator, Replaced the direct streamText call with an orchestrator loop that fetches state, classifies, and dispatches ChatResponder first to mirror existing behavior
- 3.Builder agent extraction: Moved filesystem and code-edit tool calls into a dedicated Builder path; ensured handoff via the task queue without breaking the WebContainer client API
- 4.Desktop and Media agents: Split app management and ai_generate into their own workflows so they could run in parallel with chat when safe
- 5.Memory manager: Triggered consolidation on topic shifts and task completion, keeping memoryCheckpoint in sync with Convex chat memories
- 6.Persistence + R2 wiring: Implemented desktops_private.saveDesktopStart → PUT → saveDesktopFinalize flow with throttled cloud saves and IndexedDB fallback
- 7.Hardening: Added regression tests against mocked tool interfaces, instrumented per-agent latency and queue depth via agentLogger, and shipped the feature flag
- 8.Internal pilot and rollout: Ran orchestrator and monolith side-by-side, compared token and tool logs, then migrated all traffic
Key Features Delivered
AIAgentBar: Outcome-First Input
- •Accepts natural-language intent rather than commands or file paths
- •Routes each turn through a classifier that decides whether the message is chat, build, desktop, media, or memory work
- •Streams the conversational reply immediately while longer-running tasks (code edits, media generation) work in parallel underneath
- •Surfaces task progress as inline status rather than blocking the chat thread
- •Falls back gracefully to a single-agent path if the orchestrator errors, so the user never sees a dead chat
Materializing Apps: Sandboxed, Live, Real
- •Apps the user describes appear as real, runnable applications inside a WebContainer iframe, not previews or mockups
- •Installed apps and the registry restore on login from the user's private snapshot
- •Visual desktop layer hides raw source from the experience; the user sees a desktop, not a code editor
- •The default desktop bundle and template live in templates/webcontainer/, validated by pnpm run verify:webcontainer after every template change
- •New apps slot into public/apps/registry.json and src/apps/* automatically, no manual install step
Persistent Personal Desktop
- •Workspace files, app registry, and desktop layout (icon positions, window geometry, window tabs) persist across devices for authenticated users
- •Anonymous users get IndexedDB local persistence and the default snapshot, the experience degrades gracefully rather than requiring sign-up upfront
- •Cloud saves are throttled to once per 60 seconds with a final attempt on page unload, keeping write costs predictable without losing work
- •Restore on boot tries cloud first, IndexedDB second, default snapshot third, three layers of fallback before the user sees an empty screen
Multi-Agent Orchestration Under the Hood
- •A classifier produces a typed intent payload that decides the route for every turn
- •ChatResponder handles conversation; Builder handles filesystem and code edits; DesktopEnhancer handles app management and layout; MediaAgent handles ai_generate and asset persistence; MemoryManager keeps the long-term context checkpoint coherent
- •All agents read and write the same AgentState document, so handoffs don't repeat tool calls
- •Failures in one specialized agent don't break the others, the user gets a status update from ChatResponder while the failed task is retried or surfaced for review
Social Layer: Publish, Visit, Friend
- •Users can publish their desktops and install apps shared by others
- •Visiting another user's desktop gives a real, sandboxed preview rather than a screenshot
- •Friends and direct messages live in Convex alongside the rest of user state
- •Shared apps move through the same registry path as user-created ones, so installation is uniform
Chat History That Survives the Tab
- •Per-user chat threads (chat_threads) and messages (chat_messages) persist server-side in Convex
- •The chat UI creates a default thread, stores its id in localStorage('agent.threadId'), and seeds useChat with recent messages on mount
- •Authenticated user/assistant messages are written through /api/agent whenever a threadId is present
- •Threads can be renamed and deleted; messages restore in order on reload
Media Ingestion That Doesn't Pollute the Workspace
- •Generated and uploaded assets persist outside the WebContainer filesystem so they don't bloat the snapshot
- •The MediaAgent owns ingestion end-to-end: generate, persist, register asset metadata in AgentState.resources.mediaAssets
- •Assets remain available to other agents (Builder can reference a generated image by id without re-fetching it) via the shared resource map
Technical Highlights
Frontend
Next.js 15 (App Router, Turbopack), React 19, TypeScript, Tailwind, WebContainer API
Backend
Convex (typed reactive backend, queries + mutations + actions), Cloudflare R2 for desktop snapshot blobs, Clerk for authentication and JWT issuance
AI & Machine Learning
Vercel AI SDK, grok-code-fast-1, LangGraph-style multi-agent pattern (classifier + ChatResponder + Builder + DesktopEnhancer + MediaAgent + MemoryManager), shared typed AgentState per thread
APIs & Integration
WebContainer iframe seeded from compressed binary snapshot, default desktop bundle in templates/webcontainer/, app registry at public/apps/registry.json, user-created apps under src/apps/*
DevOps
pnpm workspace; pnpm dev (Next.js + Turbopack) alongside pnpm convex dev; pnpm run verify:webcontainer and pnpm run generate:snapshot; per-agent latency and queue-depth telemetry via agentLogger
Production-Ready Success
Specialized Agents Replaced the Monolith, moved from a single streamText call carrying every tool to a classifier-routed graph of focused agents, each loading only the context and tool subset it needs
Conversation No Longer Blocks on Tool Work: ChatResponder streams immediately while Builder and Media agents run in parallel underneath, eliminating the perceived stall on long turns
One Shared State Document Per Thread, agents handing off no longer re-discover files-touched, apps-touched, or media assets, removing redundant tool calls between turns
Workspace Travels With the User, desktops (files, app registry, layout) restore across devices in a single round-trip from R2, with IndexedDB and the default snapshot as graceful fallbacks
Failure Isolation in the Agent Graph, when a specialized agent errors, the others still communicate status to the user instead of the whole turn dying
Feature-Flagged Migration With Zero Regression Window, the original single-agent path stayed live behind a flag during rollout, with token usage and tool logs mirrored across both paths for parity comparison before decommissioning
Apps the User Describes Actually Run: WebContainer execution fidelity means generated apps install dependencies, hit APIs, and serve real UI rather than rendering a mock
Ready to Build the Next Operating Layer?
The shift from chat-as-feature to chat-as-kernel changes what an interface is for. If you're building a product where the agent is the substrate, not a sidebar, the lessons from FromYou apply directly: typed shared state beats prompt-stuffing, specialized agents beat one super-agent, and persistence has to be a first-class architectural concern, not an afterthought.