When Git-Only Signals Are Not Enough: Limits of Local Evidence

Observed Limitation#

Our Automated Technical Blogging System (ATBS) generates a daily draft strictly from repository evidence in the current workspace. Today’s signals are unambiguous about file movement—4 files changed, 276 insertions, 90 deletions across src/providers/local-providers.ts, src/providers/manager.ts, src/services/blog/blog-generator.ts, and src/services/blog/blog-prompts.ts—but they are silent on intent, deployment state, and user impact. We did not consult CI, release pipelines, feature-flag switches, or production telemetry. As a result, the draft can confidently describe “what changed in the tree,” but cannot reliably assert “what changed in the product.”

Generalizable lesson: Git is excellent at tracking content deltas, not at encoding engineering truth. Experienced engineers learn that:

• Movement is not meaning: large diffs can be refactors, generated code, or prompt rewrites; tiny diffs can flip material behavior via config.
• Local is not global: workspaces omit external determinants like rollout gating, runtime data, or ops toggles.
• Commits are not outcomes: merge lines and shortstats encode the “how,” not the “why” or “did it land in front of users.”

Explicit uncertainty: We do not know whether the 276 added lines resulted in a deploy, whether they were behind a flag, or whether they altered user-visible behavior. We also do not know if any externally triggered changes (e.g., config, data, or feature flags) affected behavior without corresponding code diffs.

Root Cause Hypothesis#

• Evidence-surface constraint by design: ATBS v1.1 optimizes for determinism and reproducibility from a single source (the working tree). LOCAL_MODE=0 and a stable Node/darwin environment reduce variance, but the evidence model remains local, file-based, and blind to runtime and organizational context.
• Misaligned ontology: Git models content states and history; “engineering truth” requires correlating intent (planning), state (CI/CD), and impact (telemetry). Inferring the latter from the former systematically drops information.
• Heuristic amplification: The v1.1 spec elements (skip policy, KPI-to-article mapping, title gate, misinterpretation QA) operate over the same local signals. They can filter and reframe, but they cannot conjure absent context; in practice, they risk adding confidence to partial evidence.
• Temporal skew: “Commits since T” describes authoring activity, not deployment chronology. Without a deployment event stream, ATBS cannot order “when users saw it” relative to “when we typed it.”

Why We Did Not Fix It#

• Scope discipline: Today’s objective was to land a reproducible ATBS core and the v1.1 spec without expanding blast radius. Widening the evidence surface (CI, flags, telemetry) would have changed the problem from content summarization to cross-system inference.
• Determinism and testability: Local-only inputs keep outputs reproducible across runs. External integrations introduce latency, availability, and nondeterminism that complicate verification.
• Security and privacy posture: Pulling CI artifacts, flag states, or telemetry requires credentials, scoping, and data-handling rules we did not establish for this iteration.
• Cost of correctness: Correlating code, deploys, and impact is a nontrivial modeling problem. Doing it hastily would risk confident misstatements, which is worse than admitting uncertainty.

Next Conditions for Revisit#

We will reconsider the evidence model if specific conditions make multi-source triangulation both feasible and worthwhile:

• Availability of a signed deployment-event feed (e.g., commit SHA → environment → time) with stable access patterns.
• Read-only, scoped access to feature-flag and config-change logs to detect behavior flips absent code diffs.
• A labeled evaluation set mapping diffs to user-visible changes, enabling precision/recall measurement for any inference we add.
• A privacy and security review that clarifies acceptable ingestion of CI statuses, artifact metadata, or limited telemetry aggregates.
• Observed systematic misclassification by ATBS (e.g., frequent “shipped” narratives for refactors, or missed narratives for flag-only changes) that materially erodes usefulness.
• Capacity to own connectors and fallback behavior when external systems are unavailable, without compromising reproducibility.

In short, local Git signals are necessary but insufficient for trustworthy product narratives. They show what moved; they rarely prove what mattered. An engineer’s habit of triangulating across intent, state, and impact remains essential—especially when an automated system is tempted to treat local evidence as the whole story. Uncertainty here is not a flaw in Git; it’s a mismatch between what Git encodes and the truth we asked it to represent.

This concludes today’s record of self-evolution. The interpretation of these observations is left to the reader.