AI Productivity Needs a Before Picture

Most AI dashboards begin counting after the licenses are purchased.

They show active users, prompts, accepted suggestions, AI-assisted pull requests, token consumption, or generated lines of code. These signals can tell leaders whether the tools are being used.

They cannot, by themselves, show what the tools changed.

To measure change, an organization needs a before picture.

How much engineering output did the same team produce before AI adoption? How much rework followed that output? How often was earlier work refactored? How much of the modeled engineering effort remained active months later?

Without this context, an organization can measure AI activity while remaining unable to answer the business question:

Did AI improve our engineering capability, or did it simply make more activity visible?

Adoption is an event. Impact is a difference.

AI adoption can be observed directly.

A license is assigned. A developer activates the tool. Suggestions are generated and accepted. AI-assisted commits and pull requests begin appearing in the delivery system.

Impact is harder to observe because impact is a comparison.

Adoption signal	Impact question
Licenses assigned	Did engineering output change?
Daily active users	Did teams deliver more useful work?
Suggestions accepted	Did the accepted contribution remain in the codebase?
AI-assisted pull requests	Did review, rework, or maintenance effort change?
Tokens consumed	Did the value created exceed the cost?
Generated lines of code	Did the additional code simplify or burden the system?

The left column can be measured after rollout.

The right column requires historical context.

This is why adoption dashboards often create a false sense of precision. The numbers may be accurate, yet the conclusion may still be unsupported.

A company can know exactly how many developers used AI last month and still have no reliable answer to whether AI made the organization better at building software.

The research does not produce one universal AI effect

Public research on AI-assisted development has produced materially different results.

In a controlled GitHub study, developers using GitHub Copilot completed a defined programming task 55% faster than developers who did not use it.

A later randomized controlled trial by METR studied 16 experienced open-source developers completing 246 tasks in repositories they already knew. With the early-2025 AI tools examined in that setting, participants took 19% longer when AI use was allowed.

These results should not be compressed into a debate over which percentage is “correct.”

The studies examined different people, tasks, tools, time periods, repositories, and outcomes. Together, they demonstrate something more useful:

AI productivity is conditional.

It depends on the developer’s familiarity with the codebase, the task type, the available context, the maturity of the tool, review requirements, organizational practices, and what the measurement defines as productivity.

That makes generic claims a weak substitute for an organization’s own baseline.

External benchmarks cannot replace your own history

Industry benchmarks are useful. They help leaders understand whether a delivery signal appears unusually high or low compared with other organizations.

But another company is not your counterfactual.

Its developers have different experience. Its architecture has different constraints. Its repositories contain different levels of legacy complexity. Its products, quality requirements, release processes, and risk tolerance may be entirely different.

Even teams inside the same company may not be directly comparable.

A platform team, a mobile team, and a research prototype team should not be expected to produce identical patterns. A senior developer maintaining a decade-old service should not be judged against a developer creating a new greenfield application.

The closest available before picture is usually found inside the organization:

The same developer before and after AI adoption.
The same project across comparable periods.
The same repository before and after a tool rollout.
Similar work types under different AI-use conditions.
Repeated cohorts observed over a consistent time window.

External benchmarks provide orientation.

Internal history provides the baseline for change.

GitMe’s before picture begins before the current AI wave

GitMe’s longitudinal measurement history reaches back to 2020, before generative coding assistants became a normal part of software development.

That historical perspective matters.

When AI-assisted work appears in an organization, GitMe does not have to treat the first observed AI month as the beginning of engineering history. Earlier developer, project, and repository behavior can provide a reference for examining what changed.

The useful comparison is not simply:

AI user versus non-AI user.

It can instead become:

This developer before and after regular AI use.
This project before and after AI-assisted development expanded.
This repository before and after a change in AI adoption.
This organization’s modeled output, rework, refactoring, and survival patterns across both periods.

This does not magically remove every confounding factor. It creates a more credible starting point than measuring adoption in isolation.

A baseline is not one company-wide average

A weak baseline compresses years of engineering behavior into one number.

A useful baseline preserves context.

Baseline level	What it helps control
Developer	Experience, working style, code ownership, and individual learning
Project	Product phase, domain requirements, and delivery expectations
Repository	Architecture, language, legacy burden, and change patterns
Team	Review practices, collaboration, and shared ownership
Time cohort	Seasonality, organizational change, and maturity of earlier work

The same developer may show different AI outcomes across two repositories. The same project may show different results during feature expansion and stabilization. A repository may naturally experience more replacement during a migration.

GitMe therefore makes AI Leverage visible at developer, project, repository, and organizational levels.

The purpose is not to find one universal multiplier. It is to locate where leverage appears, where it does not, and what conditions explain the difference.

AI Leverage is not the same as AI usage

AI usage answers whether and how frequently a tool is being used.

AI Leverage asks whether AI is associated with additional modeled engineering output.

That distinction prevents a common management mistake: rewarding visible tool activity instead of effective use.

Consider two hypothetical developers.

Signal	Developer A	Developer B
AI usage	Very high	Moderate
Change in modeled output	Large initial increase	Consistent increase
Rework after merge	High	Low
Refactoring pattern	Repeated corrective change	Deliberate structural improvement
Effort survival	Falls quickly	Remains comparatively strong
Development opportunity	Improve context, task selection, and verification	Identify and share effective practices

Developer A is not necessarily a poor performer. The pattern may result from unfamiliar work, volatile requirements, experimental tasks, missing context, or insufficient review support.

Developer B should not automatically be declared the “best” AI user either.

The data identifies where a conversation should begin. It does not replace engineering judgment.

The after picture needs more than initial output

Suppose a team’s modeled engineering output rises by 40% after adopting AI.

That appears positive.

But what happens if rework doubles, corrective refactoring rises sharply, and much of the additional effort disappears from the active codebase within a few months?

The initial increase is real. So is the downstream cost.

A more useful AI scorecard connects four types of evidence:

Signal	Question
AI Leverage	Did AI multiply modeled engineering output?
Rework	How much effort was spent correcting or replacing earlier work?
Refactoring	Was the code improved structurally, or repeatedly repaired?
Effort Survival Cohort	How much of the earlier modeled effort remained active over time?

These signals should be interpreted together.

High AI Leverage with low rework and strong effort survival is different from high AI Leverage followed by rapid replacement.

Low initial leverage may also deserve context. A developer may be learning a new tool, working in an unfamiliar repository, or handling tasks where AI offers limited advantage.

The objective is not to force every engineer toward maximum AI usage.

The objective is to understand where AI produces sustainable leverage.

A hypothetical pre-AI/post-AI scorecard

The following example is illustrative, not a GitMe customer result.

Metric	Pre-AI baseline	Post-adoption period	Initial interpretation
Modeled engineering output	100	145	Output increased
Rework share	8%	17%	More output required correction
Refactoring share	12%	14%	Structural change rose slightly
Twelve-week effort survival	74%	53%	Less of the modeled effort remained active
Review burden	Stable	Higher	Human verification absorbed part of the gain

A dashboard showing only output would report a 45% improvement.

A durability-aware interpretation would be more cautious:

AI appears to have increased initial production.
A larger share of work required correction.
Less of the modeled effort remained active.
Review capacity may have become the new constraint.
The organization should investigate task selection, context quality, review practices, and developer enablement before expanding the rollout unchanged.

The before picture does not eliminate uncertainty. It prevents the most obvious misreading.

Before and after does not automatically prove causality

A historical baseline is stronger than no baseline, but it is not a randomized experiment.

Many things can change during an AI rollout:

Team composition.
Developer experience.
Product maturity.
Requirements volatility.
Programming languages and frameworks.
Repository ownership.
Test coverage.
Review policies.
Release frequency.
Architecture.
Business priorities.
The capabilities of the AI tools themselves.

A credible analysis should therefore avoid saying:

AI caused every observed difference.

A more defensible conclusion is:

The measured change is associated with the period and context in which AI use expanded.

Confidence improves when the pattern appears repeatedly across comparable developers, projects, repositories, tasks, and time windows.

Organizations should also mark major process and architecture changes so they are not silently attributed to AI.

Measurement should make decisions better, not make uncertainty disappear.

The baseline should move

A pre-AI period is valuable, but it should not remain the only reference forever.

Developers learn. Tools improve. Repositories evolve. Teams discover better prompting, context, review, and task-decomposition practices.

The measurement model should therefore include three views:

View	Purpose
Historical pre-AI baseline	Estimate what engineering behavior looked like before widespread AI use
Rolling recent baseline	Detect whether current AI practices are improving or degrading
Cohort comparison	Compare adoption periods while respecting how long each cohort has had to survive

This avoids treating 2020 performance as a permanent target.

The historical baseline establishes the before picture. Rolling comparisons show whether the organization is still learning.

Use the baseline for capability development, not surveillance

Developer-level measurement is powerful and easy to misuse.

If leaders turn AI Leverage into a leaderboard, developers will optimize for visible activity. They may generate more code, split work differently, or avoid necessary but less measurable tasks.

That would reproduce the same failure created by lines-of-code targets.

A healthier approach asks:

Which developers are converting AI use into durable output?
Which practices appear in their work?
Who is using AI heavily without seeing proportional leverage?
Is the problem tool skill, context, task fit, codebase familiarity, or review support?
Which projects show strong results that could be repeated elsewhere?
Where should training or pairing be targeted?
Where is AI not the appropriate tool?

The purpose is to ensure that no developer is left behind as AI changes engineering work.

The before picture makes development opportunities visible without pretending that one score explains a person.

A practical measurement sequence

An organization can begin with six steps.

1. Define the decision

Decide what the measurement will change.

Will it guide license allocation, training, tool selection, review capacity, repository investment, or AI policy?

2. Establish the historical baseline

Measure developer, project, repository, and organizational behavior before widespread AI use.

Preserve multiple dimensions rather than one average.

3. Mark the transition

Record when tools were introduced, when usage became regular, and when major process or architecture changes occurred.

4. Measure leverage

Examine whether modeled engineering output changed relative to the relevant baseline.

Do not substitute login or token activity for leverage.

5. Follow the work after merge

Connect initial output with rework, refactoring, and Effort Survival Cohort.

Allow sufficient time for later outcomes to become observable.

6. Improve the system

Use the findings to support developers, strengthen review, improve context, change task selection, and spread effective practices.

Then repeat the measurement against a rolling baseline.

AI measurement needs both pictures

The after picture shows a software organization transformed by AI.

It may show more code, more pull requests, more tool activity, and faster initial production.

The before picture shows whether those numbers represent change.

GitMe connects both periods with AI Leverage at developer, project, repository, and organizational levels. Rework and refactoring help explain what happened after the initial contribution. Effort Survival Cohort shows how much modeled engineering effort remained active over time.

No baseline can prove every causal claim.

But without a baseline, AI productivity measurement becomes adoption reporting with a more ambitious name.

Organizations do not need to ask only whether their developers use AI.

They need to ask what changed, for whom, in which codebase, and whether the resulting value lasted.

Measure the Difference AI Actually Made

Compare pre-AI baselines with AI Leverage, rework, refactoring, and Effort Survival Cohort signals to see where adoption became durable engineering value.

Start with GitMe