When you build an open-source tool, it’s easy to fall into the trap of asking for feedback. You ask, "What do you think of the architecture?" or "Is the documentation clear?" This is mostly fake validation. People will be polite, glance at the code, star the repository, and forget about it.
Real validation happens when another team takes your unfinished tool and runs it on their production codebase—without asking you. That’s exactly what happened with my project, codex-skills-collection.
The Context: Forcing a Workflow on an LLM
The tool was born out of my own needs. It is not another chat UI wrapper. It is a strict process contract for OpenAI Codex, injected directly into a target repository via a .codex/ directory. It is heavily optimized for Codex and explicitly incompatible with other models like Claude.
Instead of letting the model hallucinate code based on loose prompts, the system enforces a strict software delivery pipeline:
Research -> Plan -> Implement -> Review
I built it using over 60 specific "skills" (like nestjs, tauri-window-shell, or sql-and-database). Loading them all at once would destroy the context window. Instead, Codex has to navigate a markdown-based routing tree to find what it needs:
AGENTS.md (Root Router)
└── skills/routing/
├── FRONTEND.md → REACT.md → react-nextjs
├── BACKEND.md → nestjs
└── WORKFLOW.md → cross-cutting (e.g., test-strategy, debug-trace)
I tailored the tool to my own workflow, my understanding of Codex's limitations, and my knowledge of how to navigate its routing tree.
Then, an external commercial team found the project and started using it.
The Problem: Workflow vs. Reality
The collision between my highly structured pipeline and the common "just write a prompt and see what happens" mentality generated massive friction. The team had less experience with AI agents, and navigating a multi-step skill pipeline proved harder than using a standard chat interface.
Their failures became my telemetry. I quickly noticed two major architectural flaws.
Issue 1: Token Burn and the "Lost" Agent
For a developer used to writing "fix this bug," using a command like /debug was jarring. In my system, /debug triggers the debug-trace skill, forcing Codex to reproduce, isolate, trace, and hypothesize before writing a single line of code.
When users bypassed the commands and wrote vague prompts, Codex got lost in the routing tree. It would load irrelevant skills, burn through tokens, and enter frustrating failure loops that the team had to kill manually.
Issue 2: Version Drift
Using codex-skills-collection relies on bootstrapping—copying the skills into the target repository's .codex/ folder. When I analyzed the failure logs, found a logic bug in a skill, and fixed it in the source repo, the external team was still running the broken copy. The tool was fragmenting across the company.
Decisions and Trade-offs
I couldn't blame the users for "using it wrong." If the tool failed in the wild, the Developer Experience (DX) and distribution architecture were flawed.
Decision 1: Flock Management over Manual Updates
Options: I could either ask developers to manually pull new files from the source repository (inefficient), or build a central skill registry (which required a backend, breaking the "local-first" design).
Decision: I stopped treating skills as static markdown files and started treating them like a package ecosystem. I wrote bash scripts to manage the fleet of .codex/ installations.
# Scan the environment for projects containing a .codex/ directory
./scripts/discover-codex-projects.sh --root /path/to/projects --write codex-targets.txt
# Push patches only to projects that are behind the source version
./scripts/sync-skills.sh --changed
Trade-off: This significantly raised the barrier to entry for developing the tool itself. I had to introduce deployment scripts and state files (.codex/skills-sync.json with commit hashes and timestamps). However, it drastically lowered the maintenance burden for the end users.
Decision 2: Cross-Domain Recipes
Problem: Codex generated the most navigation errors on tasks spanning multiple domains (e.g., adding a database table + updating a NestJS endpoint + modifying the UI).
Decision: I hardcoded specific execution paths (Recipes) directly into the root AGENTS.md. If the model detects a task crossing BACKEND and DATA, it is no longer allowed to guess the routing path. It must load the skills in an exact order:
technical-context-discoverysql-and-databasenestjssession-learning
Decision 3: Explicit Onboarding
No agentic system can fix a missing mental model. I had to accept that a process-driven pipeline has a steeper learning curve. I expanded the README to explicitly document the architecture and the hard constraints. Users needed to understand why the system forces Codex to generate an artifact (like /docs-flow) before it is allowed to touch production code.
The Result
The errors generated by that premature, unprompted usage on an external codebase were critical to stabilizing codex-skills-collection.
They provided a clear signal: enforcing a rigid contract on an LLM works exceptionally well, provided the infrastructure can maintain that contract and distribute updates without developer intervention.
The true test of a tool's architecture is the moment someone types /implement without reading the docs. Before this incident, I assumed the user would always know what they were doing. Now, the system assumes it has to defend the process itself.