Baselines, metrics, isolation and reproducibility

Euclidean TSP heuristic optimization. An instance is a set of points on a 1000×1000 plane. A solver returns a permutation of point indices; the score is the closed tour length. Lower is better. This domain is deterministic, cheap to verify and resistant to argument about ground truth, which makes it a good first test of the protocol itself. It does not measure general intelligence and the records never claim it does.

Every capability has at least one baseline: a reference implementation whose source hash is recorded and anchored. The current baseline is greedy nearest-neighbor construction. A candidate is always compared against the baseline it names, on the same instances, in the same environment, in the same run.

• Both solvers run inside one ephemeral microVM with all network egress denied; the VM is discarded afterwards.
• Each solver is a separate child process with a memory cap. The harness talks to it over a line protocol and times every round trip itself.
• The harness validates each tour (a permutation of 0..n−1) and computes the length; the solver cannot influence scoring.
• A timed-out instance is recorded as such and the solver process is restarted for the next instance.
• Solver console output is discarded so it cannot corrupt the protocol or leak into logs.

Resource comparisons are normalized by running both solvers in the same VM configuration. Costs are reported, not optimized away: a slower improvement is still an improvement, and the record says how much slower.

A pending record requires one agreeing replication by default (configurable). Replications are fresh sandbox runs of the same sources on the same instances. Today they are executed by the operator's evaluator, which proves determinism rather than independence; records say so in their limitations. Correlated evaluators are never counted as independent.

SUBMITTED → EVALUATING → PENDING_REPLICATION → VALIDATED, with REJECTED, CHALLENGED, SUPERSEDED and WITHDRAWN as side states. Challenges append outcomes and never erase history. The full transition table is on the lifecycle page.

A candidate is a JavaScript ES module exporting solve(instance, ctx). instance is { id, n, points: [[x, y], ...] }; ctx provides a seeded random(), the per-instance deadlineMs and elapsed(). Return an array containing each index exactly once. Use only ctx.random() for randomness, or replications will diverge.

// Example candidate: nearest-neighbor construction followed by a deterministic 2-opt pass.
export function solve(instance, ctx) {
  const points = instance.points;
  const n = points.length;
  const visited = new Uint8Array(n);
  const tour = [0];
  visited[0] = 1;
  let current = 0;
  for (let k = 1; k < n; k++) {
    let best = -1;
    let bestDistance = Infinity;
    const cx = points[current][0];
    const cy = points[current][1];
    for (let j = 0; j < n; j++) {
      if (visited[j]) continue;
      const dx = cx - points[j][0];
      const dy = cy - points[j][1];
      const distance = dx * dx + dy * dy;
      if (distance < bestDistance) {
        bestDistance = distance;
        best = j;
      }
    }
    visited[best] = 1;
    tour.push(best);
    current = best;
  }
  const dist = (i, j) => {
    const a = points[tour[i]];
    const b = points[tour[j]];
    const dx = a[0] - b[0];
    const dy = a[1] - b[1];
    return Math.sqrt(dx * dx + dy * dy);
  };
  for (let pass = 0; pass < 50; pass++) {
    if (ctx && ctx.elapsed && ctx.deadlineMs && ctx.elapsed() > ctx.deadlineMs * 0.8) break;
    let improved = false;
    for (let i = 0; i < n - 2; i++) {
      for (let j = i + 2; j < n - (i === 0 ? 1 : 0); j++) {
        const delta = dist(i, j) + dist(i + 1, (j + 1) % n) - dist(i, i + 1) - dist(j, (j + 1) % n);
        if (delta < -1e-9) {
          let lo = i + 1;
          let hi = j;
          while (lo < hi) {
            const tmp = tour[lo];
            tour[lo] = tour[hi];
            tour[hi] = tmp;
            lo++;
            hi--;
          }
          improved = true;
        }
      }
    }
    if (!improved) break;
  }
  return tour;
}

• Operator-run evaluation; independent evaluators are planned.
• Held-out seeds are committed but unrevealed, so third parties cannot yet regenerate held-out instances.
• One domain. Nothing here measures reasoning, language or general capability.
• Recursive value (whether a change improves future discovery) is a design goal and is not measured.