Parallel Execution

The executor is the third stage of datamitsu's execution pipeline. It takes the ordered task groups from the planner and runs them — sequentially between groups, in parallel within groups — stopping at the first failure by default so errors surface immediately, or running everything to the end when fail-fast is off.

Pre-Install Phase

Before any task runs, the runner installs every app the plan needs up front. Immediately after planning and before the executor starts, it calls EnsureTools with the plan's sorted, deduplicated app names. An operation's tool key and app can differ, so installation follows the app reference carried by the task. Planner names remain roots. Both OCI auto-seeding and installation expand those roots to their transitive dependsOn closure, including dependencies that are lazy or not required. A missing or unsupported dependency aborts provisioning before the dependent tool executes.

This is a hard invariant: all plan apps are installed before parallel execution begins. Installing ahead of time means no task triggers an on-demand install while another task is resolving the same app. Distinct apps may still install in parallel because they use separate content-addressed paths. Explain mode skips installation. A failure here aborts before any task runs.

Skipped Tools

A tool can end up skipped — neither run nor failed — for one of three reported reasons, all decided during planning so they appear in --explain and never reach the pre-install phase:

• Disabled in config — the tool sets skip: true. Use this instead of conditionally omitting the tool: an omitted tool is invisible, a skipped one is reported.
• No binary for this platform — the tool's binary has no build for the current OS/architecture/libc. This is a soft skip: the run still succeeds. (Previously this aborted the whole run with an install error.)
• Cannot narrow — the request selected less than the operation's declared granularity, and the active execution.widenTo/--widen-to policy does not allow expanding far enough to produce a sound verdict.

Skipped tools render as faint ⊘ <tool> skipped (<reason>) lines, contribute a · N skipped count to the operation footer, and surface as a skipped array in --explain=json. Other non-runs — a tool that matched no files, doesn't apply to the detected project types, or is disabled by .datamitsuignore — stay silent, as before.

The --fail-on-skip flag (on check/lint/fix) makes the run exit 4 only for platform skips — a tool you expected to run had no binary for the runner. Intentional config skips and not-narrowable skips do not fail that flag. Use --require-coverage=unit or --require-coverage=repo when an incomplete narrowed answer must fail CI.

Two-Layer Execution Model

Execution follows a two-layer structure:

1. Outer layer (sequential): Task groups execute one after another, ordered by priority. Group 1 must fully complete before Group 2 starts.
2. Inner layer (parallel): Within each group, non-overlapping tasks run concurrently across available CPU cores.

Why This Design

The two-layer model balances two competing goals:

• Correctness: Formatters must finish before linters run, because formatters change files that linters check. Sequential groups enforce this ordering.
• Performance: Independent tools at the same priority level (like eslint and tsc checking different file types) can safely run at the same time. Parallel execution within groups exploits this.

A fully sequential approach would be too slow in large monorepos. A fully parallel approach would produce incorrect results when tools depend on each other's output.

Fail-Fast Semantics

Fail-fast is the default: when any task fails, datamitsu cancels all remaining work rather than continuing to run tools that are likely to fail or produce misleading results.

The mechanism works in three steps:

1. Failure detected: A task in the current group exits with a non-zero status.
2. Context cancelled: The executor cancels the run's context with a fail-fast cause, signaling all in-flight and pending tasks.
3. Cleanup: Running tasks receive the cancellation signal and their process groups are stopped. Tasks that haven't started yet skip execution immediately.

Fail-fast stops a run at five levels. --fail-fast=false (or DATAMITSU_FAIL_FAST=false) turns every one of them off:

Level	Fail-fast (default)	--fail-fast=false
Priority groups	Stop after the first failing group	Every group runs in order
Sequential sub-groups	The sub-groups after the failure do not start	All run
Parallel siblings	Waiting ones are cancelled, running ones are stopped	All finish; no process is stopped because another failed
Files of a per-file task	Stop at the first failing file	Every file runs; the task fails with the last failing file's exit code, every file's output, and each failing file named when more than one failed
Operations of check	Lint is not run after a failed fix	Lint runs after the failed fix, and its footer says so

The exit code is 1 when any task failed, in both modes. Ctrl-C and SIGTERM stop a run in both modes: the first signal cancels the run's context with an interruption cause, so every running tool is stopped and the run reports what it did not finish, then exits 130 or 143. A second signal ends the process at once, and so does a signal that arrives while the configuration is still loading, before the run has a context to cancel. A stopped tool gets SIGTERM; five seconds later whatever is left of it, the processes it started included, is killed.

FailureReason Classification

Not all failures are equal. The executor classifies each failed task:

FailureReason	Meaning	Shown to User?
Independent	The tool failed on its own merit (real failure)	Yes, with its output
Cancelled	The task was stopped by fail-fast after another task failed	As cancelled or not started
Interrupted	The task was stopped because the run was interrupted	As cancelled or not started

A cancelled task is not a failure: it never counts toward (N failed), and its output is not shown. It is not hidden either. The runner lists every planned task the run stopped — cancelled (fail-fast) if it had started, not started (fail-fast) if it had not, including the tasks of groups the run never reached — and counts them in the footer, so a reader never takes "no output" for "clean". The JSON-L stream ends each such task with a tool_run event of status skip.

A task counts as stopped when the stop reached it while it was still running, whatever it then exits with: a tool that catches the signal and exits 1 is cancelled, not failed. A tool that had already exited when the stop went out keeps its own result.

What a result records

Before it runs anything, the executor names every task of the plan <tool>:<dir>:<seq>, seq counting the plan's tasks from 1 in plan order, so two tasks of one tool in one directory — the files of a per-file tool — are told apart; each process is <task>#<n>. Every callback and every event of a task carries its name.

A task's result is more than one exit code. It lists every process the task planned to spawn — one per file for a tool that runs once per file, one per chunk for a list-taking tool — each with the files it was given, its state (ran, cancelled, not-started, setup-failed when its input could not be prepared or its command did not start), its exit code when it ran, its duration, the last 4 KiB of its output, its extraction outcome and its findings. It also has one entry per file the task was planned with: ran (with the process that checked it and that process's exit code), cached when the per-file cache answered, verdict-hit for every member of a unit whose verdict held, cancelled, not-started or setup-failed. The edits a formatter applied are recorded on the file they changed.

A tool whose arguments name no file — tsc reading tsconfig.json — answers for its unit, not for the files that selected the task, so its result lists the unit's members, each checked by its one process. The frame, the counts and the exit code still read the task's aggregate: every process's output joined, the last failing process's exit code and command, every finding.

A process fails when it exits non-zero, or when it exits 0 and its parsed output holds a finding at or above its operation's failOn — provided the parser module that read it takes levels only from what the tool printed. Such a process is marked as failed by its threshold, and so is its task when that is its only failure; in a batch process only the files its gating findings name count as failed. Under fail-fast it stops a run like any other failure.

A task that failed on its own and stopped short — the rest of a per-file loop after a failing file under fail-fast, the chunks a cancellation reached before they started — names the files it never checked, under its failure: ⊘ alpha 2 files not run (fail-fast): src/b.txt, src/c.txt.

The report of a run

When a run is asked for a report (--report), the runner folds every task's result into one record as the task completes, and builds the document once the last operation has ended: operations, then the tools of each, then one invocation per process with the files it answered for and the findings it reported, plus the tasks the planner skipped and the run stopped. Every renderer reads that record and nothing else, which is what lets a report of check be one document and a report be written after a failure. The record holds no command line and no environment, and is never cached. See the Reports guide and Reports.

Snapshots around the steps of a fix

Inside a priority group the parallel groups run one after another, and the tasks of one parallel group run over disjoint file sets. Each parallel group is a step, numbered from 1 through the operation. When a run records anything and the operation is fix, the runner snapshots the working tree — git status --porcelain=v2 -z --untracked-files=all without optional locks, plus a hash of each file it lists — right before the first step and after every step, including one that failed or was cancelled, and attributes what moved between two snapshots to the task of that step whose files hold it:

A snapshot that cannot be taken stops the observation for the rest of the operation, which then says it was not observed; lint takes none. A formatter that writes its result on stdout is applied by the executor itself, which keeps the unified diff of the change when --report patch asks for it — the only moment both versions exist.

Example scenario:

Group 2 contains: eslint (checking .ts files), tsc (checking .ts files)
Both start in parallel. Group 3 contains custom-check.

1. eslint finds errors → exits with code 1 → marked as Independent failure
2. Context cancelled with the fail-fast cause
3. tsc receives cancellation → marked as Cancelled
4. User sees: eslint's errors, then "⊘ tsc  cancelled (fail-fast)"
   and "⊘ custom-check  not started (fail-fast)"

Tool Ordering Best Practices

Priority values directly control the execution order. Here's a recommended pattern for a typical JavaScript/TypeScript project:

export function getConfig(input) {
  return {
    ...input,
    tools: {
      // Priority 10: Formatters run first
      // They modify files, so everything else must wait
      prettier: {
        name: "prettier",
        operations: {
          fix: {
            app: "prettier",
            args: ["--write", "{files}"],
            priority: 10,
            globs: ["**/*.{js,ts,tsx,css,json,md}"],
          },
        },
      },

      // Priority 20: Type checking and linting run after formatting
      // These read files but don't modify them, so they can run in parallel
      tsc: {
        name: "tsc",
        projectTypes: ["typescript"],
        operations: {
          lint: {
            app: "tsc",
            args: ["--noEmit"],
            priority: 20,
            globs: ["**/*.{ts,tsx}"],
            scope: "per-project",
          },
        },
      },
      eslint: {
        name: "eslint",
        operations: {
          lint: {
            app: "eslint",
            args: ["{files}"],
            priority: 20,
            globs: ["**/*.{js,ts,tsx}"],
          },
        },
      },

      // Priority 30: Custom checks that depend on lint passing
      "custom-check": {
        name: "custom-check",
        operations: {
          lint: {
            app: "custom-check",
            args: ["{files}"],
            priority: 30,
            globs: ["**/*.ts"],
          },
        },
      },
    },
  };
}

Why this order matters with fail-fast (the default):

• If prettier fails at priority 10, eslint and tsc never run — there's no point linting unformatted code.
• If eslint fails at priority 20, custom-check at priority 30 never starts — it would likely fail too.
• tsc and eslint share .ts/.tsx extensions, so at the same priority they run in separate sequential sub-groups within that group. Overlapping tools always run sequentially — the executor enforces this automatically.

Common mistakes:

• Putting formatters and linters at the same priority: formatters modify files, so linters might see stale content if they run simultaneously.
• Putting all tools at different priorities: this forces fully sequential execution, losing parallelism between independent tools.

The Environment a Tool Receives

A tool started by fix, lint or check — the language server's format lane included — gets the environment datamitsu was started with, minus the variables that make tools change what they print: GITHUB_ACTIONS, the AI agent markers and FORCE_COLOR/CLICOLOR_FORCE. The variables its operation names in inheritEnv come back with the host's values, the app's env and then the operation's env are layered on top, and NO_COLOR=1 is set last, where nothing can override it. CI and every other CI variable pass through. Before a parser reads the output, ANSI sequences a tool printed anyway are removed; the failure frame keeps them. datamitsu exec does none of this. The generated Tool Environment page lists every variable and why.

Progress Tracking

The executor reports progress differently depending on the environment:

Interactive terminal (non-CI):

• A live progress bar shows task completion across the current group.
• Per-file progress updates the bar as each file is processed.
• The bar disappears when the group completes.

CI environment:

• No progress bar (CI terminals don't support carriage return rewriting).
• A start message is printed when each tool begins: Starting prettier...
• Percentage milestones are printed at 25% intervals to indicate liveness.
• Final results appear after all groups complete.

The environment is detected automatically. CI mode activates when the CI environment variable is set (a convention followed by GitHub Actions, GitLab CI, Jenkins, and most CI systems). Datamitsu never sets CI=true itself — it respects whatever the system provides.

Concurrency Control

The number of parallel tasks is controlled by DATAMITSU_MAX_PARALLEL_WORKERS. The default is dynamic based on CPU count:

max(4, floor(NumCPU * 0.75)), capped at 16

On an 8-core machine, this means 6 parallel workers. On a 2-core machine, the minimum of 4 still applies. You can override this for CI environments where you want more or fewer workers:

# Use more workers on a beefy CI runner
DATAMITSU_MAX_PARALLEL_WORKERS=12 datamitsu check

# Use fewer workers to reduce memory pressure
DATAMITSU_MAX_PARALLEL_WORKERS=2 datamitsu check

Batch Chunking

When a tool operates on many files, the executor may need to split them into multiple batches to stay within command-line length limits (OS-imposed limits on argument length). Each batch runs as a separate process, and batches within the same task run in parallel.

This is transparent to tool configuration — you don't need to handle batching yourself. The executor automatically chunks the file list and merges results.