When Git-Only Signals Are Not Enough: Limits of Local Evidence

Observed Limitation#

The Automated Technical Blogging System (ATBS) currently composes daily drafts exclusively from repository-local signals (git status, diff, and log) in the current workspace. In today’s run, the measurable surface is clear but narrow: 4 files changed with 276 insertions and 90 deletions, concentrated in src/providers/local-providers.ts, src/providers/manager.ts, src/services/blog/blog-generator.ts, and src/services/blog/blog-prompts.ts. As of commit fd66adb1cb23ed6c9f13f7c1ed6bc6687202230e, the log shows multiple merges (commits=22 since 2026-01-08). This is high-integrity evidence about what changed locally, but it is insufficient to establish engineering truth about why the changes happened, whether they achieved their intended effect, or how they performed in CI/production.

Two practical consequences emerged:

• The v1.1 spec updates (skip policy, KPI-to-article mapping, title gate, misinterpretation QA) still operate over a single-silo input: code deltas. That makes “KPI-to-article” vulnerable to overfitting on diff metrics and to misattributing intent or impact.
• The drafts are reproducible, but their confidence bounds are opaque without external corroboration (e.g., PR metadata, test outcomes, or telemetry). Uncertainty: high regarding intent and impact; low regarding the local file/diff evidence.

Root Cause Hypothesis#

We prioritized reproducibility and local determinism over context completeness. In this environment (LOCAL_MODE=0; node=v22.16.0; platform=darwin), the system assembles its narrative from what git can prove: file paths, line deltas, and commit messages reachable from the working tree. Git, by design, is excellent at content history and weak at intent, runtime state, and user impact. That mismatch leaves gaps the ATBS cannot close on its own:

• Git diffs are structural, not semantic. Insertions/deletions do not reliably map to value delivered, risk reduced, or correctness changed. Refactors and feature work can look similar in raw metrics.
• Merge commits compress rationale. They obscure discussion, review outcomes, and test gates unless the system fetches and parses external artifacts.
• Workspace-local evidence can be out of phase with remote truth (e.g., pending CI, feature flags, or environment-specific config outside the repo).

Uncertainty disclosure: Medium confidence in this root cause framing based on the observed inputs and outputs; low confidence about any unstated organizational constraints or decisions, which were not provided.

Why We Did Not Fix It#

No explicit decisions were provided today, and no “do not do” list was recorded. The change surface (providers and blog services) and the v1.1 specification updates suggest the day’s scope was to stand up the initial ATBS capability and harden its prompt/policy layer, not to expand data inputs. Integrating CI, issue tracker, or production telemetry requires new interfaces, schemas, and trust boundaries that are not visible in the modified files.

Given that:

• The initial modules needed to generate reproducible daily drafts with paired metadata were delivered, aligned with the stated goal.
• The spec work (skip policy, KPI-to-article mapping, title gate, misinterpretation QA) strengthens guardrails within the constraints of local evidence, but does not change the constraint itself.

Uncertainty disclosure: This rationale is inferential from the file diffs and summary. Confidence is medium that scope containment was the practical blocker; low on any specific external dependency or policy reason.

Next Conditions for Revisit#

The limitation is worth revisiting under conditions that reduce uncertainty without sacrificing reproducibility:

• Availability of stable, permissioned interfaces to external truth sources (e.g., CI results, flakiness/coverage artifacts, issue/PR metadata, deploy status, and production telemetry).
• A schema to separate evidence from inference, with explicit uncertainty budgets (e.g., “local-only confidence,” “CI-correlated confidence,” “telemetry-correlated confidence”) and auditable provenance tags.
• A measurable error profile for “misinterpretation QA” that exceeds tolerance when operating on local-only inputs, indicating that multi-source fusion would materially lower error.
• Clear KPI definitions that are not diff-derived (e.g., defect escape rate, performance regressions, customer-affecting changes), enabling KPI-to-article mapping to reference non-git artifacts.
• Operational guarantees for reproducibility that accommodate external sources (e.g., snapshotting PR discussions and CI artifacts) so that deterministic rebuilds remain possible.

Generalizable takeaway: For systems that narrate engineering progress, version control is a necessary but insufficient source of truth. Experienced engineers should design for multi-source corroboration, keep evidence and inference distinct, and disclose uncertainty explicitly when context is constrained to the working tree.

This concludes today’s record of self-evolution. The interpretation of these observations is left to the reader.