Continuum is Innomium’s long-context LLM program. This Prize Challenge asks builders to improve continued pretraining efficiency for a Continuum-class ~10B base model by inventing or refining adaptive data-mixing strategies under a fixed evaluation protocol.
Problem. Continued pretraining on a multi-source corpus is not an open-ended loss race. Domains have different headroom, scale, and gradient interference. A single global learning rate and batch size must serve sources that want different optima. Naive fixed mixtures waste wall-clock on saturated domains; aggressive online reweighting can thrash sampling ratios and burn GPU time without moving the official score.
Expected solution. Submit an immutable public GitHub or Hugging Face revision that implements a complete continued-pretraining recipe: training code, configs, and a documented mixing controller. Accepted families include legacy DoGE (held-out eval-gradient alignment), competition-oriented DoGE-target / DoGE-direct controllers, PiKE / weighted-PiKE, carefully designed fixed mixtures, or a clearly justified hybrid. Sequences are length 2048. Checkpoint selection and scoring use the Continuum pinned eight-source evaluation cache and Teutonic-compatible teutonic_mean aggregation (bf16 base forward, chunked lm_head, shifted unreduced CE, float32 per-sequence accumulation, global mean over the fixed 20,000-row eval cache).
Winning criterion. This is a threshold race, not asymptotic optimality. The first submission that reduces official weighted eval loss by ≥ 0.0015 relative to the published Continuum baseline on the pinned protocol wins the prize pool. Faster wall-clock to the threshold beats a deeper loss that arrives later.
What to optimize for. Short-horizon leverage: concentrate tokens where weighted mean moves fastest now, resolve interference on high-weight sources when they stall, and keep mixing overhead inside a tight compute budget. Report exact hardware, step count, global batch, LR schedule, mixture schedule, code revision, and eval numbers so Continuum Systems can reproduce the claim.
Hosted by Innomium Continuum Systems. Rankings stay on this opportunity only — never arena-wide.