1 / 5
What a software factory has to do
- Lifecycle traceability — Maintained relationships explain what addresses, implements and verifies each requirement.
- Versioned system design — See the whole intended design before implementation finishes, and distinguish it from what each code version implements.
- Tracked drift resolution — Identify possible disagreement, assign a review or correction, and track it through recheck.
- Change impact and alternatives — Use connected requirements and constraints to assess affected parts, alternatives and trade-offs.
- Requirement test results — Executable tests demonstrate required behaviour in the delivered code version. Passed, failed and unverified requirements are visible.
- Delivery audit trail — Identify the delivered build, governing design, reviews, approvals and waivers. Reference its test results.
- Process portability — Continue running the workflow, orchestration and checks on other supported systems without the original hosted factory service.
Scripts check links and freshness; tests check behaviour. AI and people review meaning.
Writing documents and code is widely available. Keeping them aligned takes more.
Speaker notes
Agents now write requirements, specs, code and tests quickly. For a system that has to be maintained and audited for years, that is not the hard part. The hard part is that all of these documents still agree with each other and with the code a year later.
The first property is meaningful integration: a design decision addresses requirements, code implements the design, and tests verify required behaviour. IDs and links need maintained meaning and versions. The second is a complete intended design for a selected version, available before implementation finishes. Proposed, approved but unimplemented, and implemented changes must be distinguishable. A delivered code version must identify the design it implements.
Third, drift detection must lead to tracked reconciliation. A freshness or suspect-link flag is a prompt to review, not proof that code is wrong. A finding needs an affected area, a review or correction, and a recheck. Automatic document rewriting is not required. Fourth, the connected information should support assessing changes and comparing design alternatives against the same requirements, including constraints, assumptions and trade-offs.
Fifth, executable tests must check the implementation against its requirements at the end of the work, with results bound to the tested code version. Test-file links and code coverage alone are insufficient. Sixth, delivery provenance identifies the build, governing documents, reviews, approvals and waivers. It references test results without counting test adequacy a second time.
Seventh, process portability means the factory can continue running elsewhere. Exportable code and Markdown alone do not qualify. A proprietary licensed runtime can qualify if it can be deployed on different supported systems. Changing engines is not required, and model-service dependencies are stated separately.
For every check, distinguish available capability, trigger, blocking enforcement and tracked resolution. Structural scripts, executable tests, AI review and human decisions supply different evidence. Flexible configuration is not a weakness. Governance is discussed in the report without a score. Deployment, operation and measured effectiveness are outside the scored comparison.
2 / 5
What exists today
| Type | Examples | Good at | Missing |
|---|---|---|---|
| Issue-to-PR agents | Copilot coding agent, Jules, Devin, Azure Boards → Copilot | A ticket becomes a pull request | A persistent specification is not required |
| Spec workflows | Kiro, Spec Kit, OpenSpec, BMAD | Requirements and design before code | Completeness, version alignment and check enforcement vary |
| Requirements tools | Jama, Polarion; Azure DevOps + GitHub Copilot | Requirements and test traceability; Copilot writes code | No dedicated design step in the Azure Boards → Copilot handoff |
| Atlassian suite | Confluence, Jira, Rovo Dev | Documents, work tracking and coding assistance | Full cross-layer reconciliation was not established |
| Agent orchestration | Factory.ai, Augment Cosmos | Parallel agents and executable QA | Maintained requirement/design model not established |
Writing, maintaining and tracing documents are different capabilities. The reviewed products cover different parts.
Speaker notes
The market splits into five groups. Issue-to-PR agents such as GitHub Copilot coding agent take a ticket and open a pull request, and a person merges it. Azure Boards can hand a work item to Copilot the same way. The reviewed issue-to-pull-request workflow does not require a persistent specification between the ticket and code.
Spec workflows such as AWS Kiro, GitHub Spec Kit, OpenSpec and BMAD write requirements and a design before the code. Kiro writes requirements, design and tasks per feature, and those files stay behind as history. Spec Kit offers a living-spec option. OpenSpec merges each change into a current spec. BMAD keeps a PRD, UX, architecture and specs up to date through its own workflows. Checks can be invoked directly or connected to supported triggers. Their scope and enforcement vary; an integrated intended design and a governing version for each delivered build need separate evidence.
Requirements tools such as Jama Connect, Siemens Polarion and Azure DevOps Test Plans have done the long-term part for years. Jama and Polarion keep living requirements, approved baselines, coverage reports and suspect links that flag downstream items after an upstream change. Azure DevOps shows which requirements have no tests and links work items to commits, builds and releases. Azure Boards can send a work item to GitHub Copilot, which writes code and opens a linked draft pull request. That handoff has no dedicated design stage. Jama and Polarion connect to development tools through integrations.
Atlassian shows the problem inside one vendor. Confluence holds PRDs and decision pages, Jira holds the work, and Rovo Dev writes code from Jira items. Test management comes from marketplace apps. Page links and issue keys do not establish a maintained cross-layer model. Rovo’s AI review can flag possible acceptance-criteria mismatches, but tracked design reconciliation remains unestablished. Atlassian is also retiring standalone Rovo Dev and moving its features into Jira.
Orchestration products such as Factory.ai and Augment Cosmos run many agents in parallel with AI review. Factory's Software Factory dashboard is in private preview. Vercel's internal factory for its AI SDK reports 25 to 35% of merged pull requests, and a person merges every one.
Sources: About Copilot coding agent; Azure Boards and GitHub Copilot; Kiro specs; Spec Kit: evolving specs; OpenSpec quickstart; BMAD: plan inside an organization; Jama traceability; Jama electronic signatures; Polarion quality assurance; Azure DevOps requirements traceability; Azure DevOps end-to-end traceability; Atlassian: Rovo Dev licensing; Rovo Dev for architecting solutions; Factory Software Factory; Augment Cosmos; Vercel: building a software factory.
3 / 5
How they compare
| Product | Lifecycle traceability | Versioned system design | Tracked drift resolution | Change impact and alternatives | Requirement test results | Delivery audit trail | Process portability |
|---|---|---|---|---|---|---|---|
| GitHub Copilot coding agent | ? | ? | |||||
| Azure DevOps + Copilot | ? | ||||||
| Atlassian suite (Confluence, Jira, Rovo Dev) | ? | ? | |||||
| AWS Kiro | ? | ? | ? | ||||
| GitHub Spec Kit | ? | ? | |||||
| OpenSpec | ? | ? | |||||
| BMAD Method + Test Architect | |||||||
| Tessl | ? | ? | |||||
| Jama Connect | |||||||
| Siemens Polarion | |||||||
| Factory.ai (Droid + Missions + QA) | ? | ||||||
| Google Jules | ? | ? | |||||
| Cognition Devin | ? | ? | ? | ||||
| OpenHands | ? | ? | ? |
Ratings describe documented support, not percentages. ? Unverified means evidence is insufficient. Not supported requires a documented restriction. Whether checks start automatically and whether failures block delivery are described separately. No overall product scores are calculated.
Product names link to detailed evidence in the report. Azure–Copilot and Rovo require hosted services. Factory’s airgapped Droid runtime is documented; continuity of its complete connected QA workflow remains unverified.
Coverage depends on the complete criterion, the configured checks and the retained evidence.
Speaker notes
The columns use the full definitions on slide 1. The labels show coverage: zero means a documented restriction, one limited support, two substantial support with a specific remaining gap, and three complete documented coverage in the stated configuration. A question mark means the evidence cannot justify a coverage level. These are ordinal documentation assessments, not percentages, vendor totals or hands-on trials.
The report distinguishes the trigger, check mechanism, blocking enforcement and tracked resolution. Flexible configuration receives no penalty. Edition and availability remain chapter details.
Azure DevOps and configured Polarion test/build integrations can connect executable test results to requirements and a build. Jama test-run history is useful, but the tested code-version association was not established in the reviewed native test-run documentation. None of these ratings substitutes test-file links or generic code coverage for implementation verification.
BMAD includes the optional Test Architect module here. Its trace matrix measures requirement coverage; passing tests remain a separate check. Persistent BMAD/Tessl documents and merged OpenSpec capability specs are partial under the stricter requirement for a complete approved target and an explicit governing design version for delivered code.
Process portability scores execution, not file retention. The Azure–Copilot and Atlassian–Rovo bundles require hosted services, so their self-managed core products earn no portability credit for the complete bundle. Kiro and Tessl have unresolved service dependencies. Jama and Polarion have deployable runtimes. Factory documents an airgapped Droid runtime, but complete connected QA continuity remains unverified; its private-preview dashboard is excluded.
A freshness warning, failed scenario or AI review finding needs a tracked decision, correction and recheck. The appropriate resolution may change code, documentation or both; automatic rewriting is not required.
Sources: Kiro correctness; Spec Kit: evolving specs; OpenSpec quickstart; OpenSpec with ADRs; BMAD: plan inside an organization; Tessl spec-driven development; Tessl spec verification; Jama baselines; Jama coverage report; Polarion collections; Azure DevOps automated tests and test cases; Cucumber overview.
4 / 5
Sample custom factory built in ~4 man-months
| Level | Where | Linked by | Count |
|---|---|---|---|
| Requirements | requirements/*.md | REQ-* IDs | 305 |
| Architecture | architecture/ADR-*.md, overview | ADR-* IDs | 14 ADRs |
| Specs | specs/spec-*.md | front matter implements: [REQ-…], adrs: [ADR-…], depends_on: [spec-…] | 46 |
| UI design | ui-design/mockups/*.html + manifest + inventory | screen IDs; criterion IDs in manifests | 16 screens |
| Work items | code/todo.md | TODO-CODE-*; criterion IDs in “Done when” | |
| Tests | test files | @covers AC-…; test named in each “Done when” |
Example: spec-021 (account transfers) cites 7 requirements and 4 ADRs and defines 32 acceptance criteria. Documents name their upstream requirements and decisions. This illustrates traceability structure, not full satisfaction of all seven criteria.
Agents wrote most of the scripts and workflows. The versioned, connected design can outlast the tools that maintain it.
Speaker notes
This is an example of a factory built for one product, a personal finance app, by one person with agents over about four man-months. It is not a product. It illustrates linked documents and local checks built for that product.
There are six levels. Requirements carry IDs such as REQ-TRF-001. Architecture decisions are numbered ADRs. Each spec states in its header which requirements it implements, which ADRs it follows and which specs it depends on. Each UI screen is a mockup built from specs, with a manifest listing the acceptance criteria it shows. Work items name the acceptance criteria they deliver, and tests name the criteria they cover.
Account transfers is a typical spec. It implements seven requirements, follows four architecture decisions and defines thirty-two acceptance criteria. Because every link is an ID in a fixed place, a script can follow the chain from a requirement to its tests and back.
The factory itself is not the asset. Its scripts and workflows were built with agents, and the next generation of agents will build them faster and better, as with skills and workflows today. What carries over is the structured, linked knowledge: requirements, decisions, screens and specs that any new tool can read and check.
5 / 5
Custom factory (slide 4): keeping levels in sync
| Problem | Mechanism | How |
|---|---|---|
| Finding everything about one item | Every requirement, architecture decision, screen, spec, acceptance criterion and work item has a unique ID. A tool or an AI agent can search the whole repository for it. | Structure |
| A requirement or criterion is edited after work was built on it | Each document that refers to a requirement or criterion records which version it was checked against. When the text changes, every document that refers to it is flagged for review. | Deterministic tool |
| What does a change affect? | An impact tool lists everything connected to an ID: the requirements and decisions it rests on, and the specs, screens, work items and tests built on it. | Deterministic tool |
| Acceptance criteria without linked tests | A report lists criteria with no test citation. This identifies missing links, not passed implementation tests. | Deterministic tool |
| Work marked done without proof | A closure check requires each completion condition to name an existing passing test. This checks the work item, not complete requirement verification for a delivered build. | Deterministic tool |
| Architecture decision ignored in code | Each architecture decision lists code rules that a checker enforces. Each rule is itself tested. | Deterministic tool |
| A spec changes after delivery | A workflow reopens dependent work for review and correction. The required resolution is decided before implementation continues. | AI workflow with judgment |
| Requirement and spec decisions | A separate AI role decides on requirement and spec changes and never writes code. | AI workflow with judgment, person approves |
Freshness flags identify a review need. Tracked decisions and corrections resolve it.
Speaker notes
This is the custom framework shown on slide 4. IDs alone do not keep its documents true. Something has to notice when they stop agreeing.
The base is the unique ID. Every requirement, architecture decision, screen, spec, acceptance criterion and work item has one, and every document that relies on another names it by that ID. That makes the whole hierarchy searchable, by a deterministic tool or by an AI agent.
On top of the IDs sit deterministic tools. When a requirement or criterion is edited, everything that refers to it is flagged for review, the same idea as suspect links in Jama or Polarion. An impact tool shows everything connected to an item, in both directions. A coverage tool lists acceptance criteria without test citations. The closure check requires a passing test for each work-item completion condition. Neither alone is a complete requirement-verification report for the delivered build. Architecture decisions are enforced in code by rules, and each rule is tested so it cannot silently stop working.
Some steps need judgment, and those are AI workflows. When a delivered spec changes, the work that depends on it is reopened. A discrepancy requires a decision about whether code, the spec or both need correction. The approved resolution governs the next implementation step. Decisions about requirements and specs go to a separate AI role that never writes code, and a person approves them. The tools find the problems. AI workflows and people fix them.