zk-data-agent/docs/project-history.md

# Project history and decisions

This record summarizes the rebuild that began on 2026-07-26. Git remains the
source of truth for exact changes; this document captures product intent and
architecture decisions that are otherwise difficult to reconstruct from
individual commits.

## 1. Clean-slate web Agent

The existing business/data-specific application was declared a new project
with no backward-compatibility requirement. Business skills, domain workflows,
and user-configurable provider settings were removed from scope. The retained
core requirement was general coding-Agent capability with an independently
evolvable loop.

Decision:

- build one general multi-user Web Agent;
- preserve coding, shell, file, Git, process, planning, memory, and delegation
  capabilities;
- keep infrastructure and product shell separate from Agent intelligence.

## 2. Open WebUI as product shell

Instead of maintaining a second full account/chat frontend, Open WebUI v0.9.6
was pinned and lightly patched. It supplies registration, login, RBAC,
administrator approval, conversation history, and familiar chat interaction.

K1412 kept ownership of the Agent loop, provider routing, memory, tools,
scheduling, events, and workspace execution. This reduced the cost of Agent
experiments without making the loop dependent on Open WebUI internals.

## 3. One Agent path, not Chat versus Work

An early design considered a switch between Open WebUI's native Chat loop and
the K1412 Work loop. The product was simplified to one Agent path because:

- users should not need to understand two orchestration engines;
- conversation continuation between modes creates ambiguous state;
- two loops double frontend states and verification behavior;
- the custom loop is the main research asset.

The remaining interface is an Agent-first chat with server-owned model
selection.

## 4. Model and thinking semantics

The original “light / medium / high reasoning” labels conflated different
models with adjustable effort. The UI and backend now distinguish model
identity from thinking capability:

- Luna, Terra, and Sol are separate local K1412/Ollama models;
- each exposes thinking as enabled but without adjustable effort levels;
- DeepSeek V4 Pro is the cloud extreme tier with maximum reasoning effort.

Provider model IDs, URLs, credentials, and tuning remain server-side.

## 5. Multi-user remote workspaces

Each user received a dedicated Docker container, network, and persistent volume
derived from a hashed immutable user ID. Execution moved from the NAS to the
physical `home-node-itx` through SSH Docker.

The physical node was prepared with:

- an ext4 500 GB data disk mounted at `/srv/k1412-data`;
- Docker root under `/srv/k1412-data/docker`;
- strict SSH host verification;
- no application/database secrets.

The local and SSH execution providers retain one interface so the host can be
replaced without changing Agent behavior.

## 6. File delivery and focused UI

The frontend was simplified around the Agent:

- K1412 product title and assets;
- a compact model selector with separate thinking status;
- no exposed provider/tool/system-prompt settings;
- an authenticated workspace file browser;
- single-file download and streamed directory archive.

Open WebUI attribution and license requirements remain documented.

## 7. Loop hardening through failed-task analysis

Real sorting-script/report tasks exposed model weaknesses and platform bugs.
The loop was progressively hardened to:

- require actual files rather than textual claims;
- require executable source separately from reports;
- run the source before writing a report;
- require measured benchmark values;
- reject invalid Python and escaped-layout corruption;
- recover from failed verification;
- prevent unchanged retry loops and duplicate batches;
- preserve failures even when output is piped through another command;
- bound iteration, tool-batch, model-output, and retry budgets.

These constraints are not assumed permanently optimal. They are the current
baseline and should be removed or changed only through measured experiments.

## 8. Python environment and long-run reliability

Production debugging found three platform problems:

1. the container root was read-only while `pip` defaulted to `/home/agent`;
2. `/tmp` was `noexec`;
3. shell pipelines returned the last command's status, hiding failed installs.

The fix moved home, caches, user packages, and virtual environments to the
persistent workspace, enabled Bash `pipefail`, extended tool timeouts to 900
seconds, and taught the Agent the supported `.venv` workflow.

The same investigation found that a five-minute identity token was reused for
every tool in a long run. Runtime now signs a fresh short-lived identity for
each Gateway call after the original request identity has been verified.

Production validation created a real `.venv`, installed and imported
`psutil 7.2.2`, downloaded the generated proof file through the Web API, and
then removed the temporary proof file.

## 9. Documentation and experiment portal

The repository documentation was reorganized around architecture, Agent
implementation, security, operations, history, development, and experiments.
The curated public portal is shipped inside the existing Web image at
`/doc/`, avoiding a second service or proxy rule.

## Current direction

The platform is now suitable for controlled Agent-loop experimentation. The
next work should prioritize evaluation data, context/memory experiments,
scheduler design, delegation contracts, cost accounting, and hardening for
hostile multi-tenant use rather than adding business-specific workflows.