2026년 9월 14일
Why an AI assistant can know the facts and still miss local rules. Research on separating knowledge, reasoning, and style, explained through concrete examples and a testable design for more adaptable assistants.
약 42분
Imagine asking an AI to shorten this sentence without changing its meaning:
People who used the new app finished their work faster, but we do not know whether the app caused the difference.
The AI returns:
The new app makes people work faster.
The rewrite changes the claim. The original reports an observation and leaves the cause open. Perhaps the app's users simply had more experience. The rewrite says the app caused the improvement. A request for fewer words has produced a claim the evidence does not support.
This example is invented. Abdulhai and colleagues tested a similar problem using human-written essays. They found that AI revisions changed meaning even when the instructions asked for grammar edits only.1
Now imagine a software project with a shared function for formatting dates. The team's rule is to reuse that function. Asked to fix one displayed date, an AI assistant adds another software package to do the same job. The date looks correct, but the team now has a second tool to keep updated. The assistant has ignored a project rule and created unnecessary work.
A useful assistant must respect both the evidence and the project's rules. Knowing facts, drawing conclusions, and choosing how to express them are different jobs. The research question is how reliably we can control them separately. The examples show failures of that control. They do not tell us what caused the failures inside the model.
Try the distinction with your own assistant
Ask it to shorten the app sentence while keeping the uncertainty about the cause. Then change only the instruction: ask for a more formal version with the same meaning. Check both answers against the original. The wording should change, while the observation and its limit survive. Repeating this with different examples tests whether a good answer becomes a reliable pattern.
The app sentence requires the assistant to preserve both the observation and its limit. People worked faster. The cause is unknown.
Knowledge is the information available. Here it is the observation about the app's users. Reasoning uses that information to reach justified conclusions. The observation alone does not establish why people worked faster, so a justified conclusion stops short of naming a cause. Style is the set of choices about expression that preserve meaning and behavior: two sentences or one, formal or plain, loadCustomer or load_customer. Style applies to code as much as to prose.
A convention is a local rule. It can govern style, such as function naming, or behavior and tools, such as reusing the date helper. A rule is information too: the assistant must know it, recognize where it applies, and comply. These responsibilities overlap. An ignored rule could reflect missing information, misunderstanding, or a persistent preference.
Training adjusts a model's parameters, also called weights. These numbers influence its knowledge, reasoning, and preferred ways of responding. A prior here means a starting expectation, such as a favorite library or familiar tone. It helps when the task supplies nothing better and becomes a problem when it persists despite evidence for another choice. Useful skills such as grammar and arithmetic should remain available.
The phrase "separate knowledge, reasoning, and style" hides three different claims. They are not steps on a ladder, and none of them implies another.
A system can have swappable parts and still ignore a local rule. It can also follow local rules while retaining broad world knowledge. That would be practical success. The small models proposed in the prototype note are choices for controlled experiments and cost comparisons, not a requirement for success.
An ordinary assistant may already meet the goal. The proposed system must therefore be compared with one that has comparable learned capacity and the same evidence, tools, rules, checks, and computation while answering.
If they perform equally well, the new architecture has shown no added value on those tests. The shared rules and checks need a separate comparison with and without them. An architecture comparison cannot establish their contribution when both assistants receive them.
The research offers ways to test the distinction and possible parts for a solution. It also shows why adding memory alone leaves important problems open.
Evidence and proposed experiments
Sources include peer-reviewed studies and preprints, checked through September 14, 2026. The assistant design, training plan, and tests later in this article are proposals. This article reports no training results for that design. The collapsed research notes hold detailed findings, budgets, and methods. The main text is meant to be complete without them.
A correct answer may come from recognizing a familiar solution. A model capable of the required inference can also fail because it lacks a fact. Imagine the same route problem with familiar place names and invented ones, but identical connections. The route requires the same steps. Familiar names may provide an additional clue from memory.
SynthWorlds uses this principle: parallel worlds with the same relationships, one using familiar mappings and one using artificial ones. The authors report an advantage for the familiar world. Access to knowledge and other aids reduce that advantage but do not remove it.4 Equal results alone would not show separation, since the assistant might fail in both worlds. A comparison must report the success rate in each world beside the gap between them.
Wu and colleagues instead changed the assumptions of familiar tasks. Performance losses revealed limits in transferring learned procedures.5 Unfamiliar notation can also hinder understanding, so a sound comparison must separate difficulty reading the task from difficulty solving it.
Knowledge and reasoning questions
Thapa and colleagues separated questions from eleven biomedical benchmarks into knowledge-focused and reasoning-focused groups. Their classification assigned 32.8 percent to complex reasoning. The classifier reached 81 percent accuracy.6 That accuracy concerns the classification of questions, not the accuracy of answers to them.
The figures describe this collection and its categories. They do not describe medical work in general, or establish a general percentage of AI tasks that require reasoning.
Reading the task versus solving it
Another test changes the rules of a familiar task. Wu and colleagues studied such counterfactual conditions, meaning deliberately changed assumptions. Their observed performance losses reveal limits in transferring familiar procedures.5 An unfamiliar notation can also make a task harder to understand. Experiments should therefore control names, text length, and presentation. They should also provide a correctly parsed task representation, in which the task's elements and relationships have already been identified. This helps distinguish trouble reading the task from trouble solving it.
Visible memory and reasoning steps
An assistant may show a sequence of steps alongside its answer. Jin and colleagues use special markers to label generated steps as memory or reasoning, making the output easier to examine.7 Both kinds of steps still come from a trained model.
The label “I am reasoning” does not establish which internal computation produced an answer. Traceable sources, executed calculations, and independent tests provide stronger evidence. A plausible explanation can complement those checks, but cannot replace them.
For the date task, the assistant could first find and read the project's instructions. Retrieval-Augmented Generation, or RAG, combines a model with external document search. Self-RAG adds learned decisions about when to retrieve and whether the evidence supports an answer.1011
A source helps only if the assistant receives the relevant passage, understands it, and applies it. Its learned weights remain in place. If the rule never reaches the model, the failure cannot be attributed to reasoning alone.
Another approach prepares document knowledge as adapters, small additions to a model's learned parameters, then selects them for a query.12 The question becomes what survives that preparation.
In the first version of their study, Tang and colleagues find that document adapters preserve only part of the documents' meaning. Combining them with the original text works better in their experiments than adapters alone.13 For an exception to the date rule or an exact quotation, the original passage remains essential. A replaceable adapter may also have learned style alongside facts.
Retrieval with a smaller generator
RETRO is one experiment in using retrieved text inside a language model. It integrates retrieved text chunks through a trained encoder and a separate attention path. The encoder processes the retrieved text for the model to use.
Its 7.5-billion-parameter model reached performance comparable to much larger models in language modeling on the Pile. The setup included a database of two trillion tokens, the text units processed by models.14
This supports a possible trade between model size and external memory in that experimental setting. The search model, encoder, and database also belong in the system budget. Counting only the small generator understates the resources used.
Document adapters and shared banks
Parametric RAG represents document knowledge in additional parameters called adapters. The base model's weights do not all need to be retrained. In LoRA, these changes are organized as comparatively small matrices.
The system prepares document modules, selects them for a query, and incorporates them during inference, when the model produces its answer. It does not train a fresh adapter for every question.12
The accompanying SIGIR tutorial distinguishes when retrieval should happen from how the retrieved information should enter the model. Dynamic retrieval and dynamic adapter training are different operations.15
A dedicated adapter for every document consumes training and storage. Poly-PRAG addresses this with a learned routing function that represents documents through combinations from a shared adapter bank.16 This reduces redundancy. It does not make each module an isolated container of facts.
Engram uses short sequences of tokens, the text units a model processes, to find information in a separate learned memory. The authors disable that memory and measure what performance remains, relative to the unchanged model:2
| Task type | Performance retained |
|---|---|
| Factual knowledge | 29 to 44% |
| Reading comprehension | 81 to 93% |
These percentages compare performance with and without memory. They are not the percentage of questions answered correctly. Each range spans different tests, rather than uncertainty around one result. Removing memory harms factual tasks more. Disabling it also changes the operating conditions from those used in training. The experiment supports partial specialization, not a fully capable model without knowledge.
Other work changes how a model retrieves memory, repeats computation, or handles more material than fits in its active context, its current workspace.193202122 These methods organize storage and computation differently. Their results do not establish that swapping knowledge leaves reasoning and local conventions intact.
Reading the Engram ranges
Section 6.3 reports TriviaQA at 29 percent of its original performance and C3 at 93 percent. These are examples from the factual and reading-comprehension groups respectively. Disabling the sparse memory output changes the conditions from those used in training, while leaving the rest of the model unchanged.2
Lngram and Mobius
Lngram changes what can serve as a memory key. It replaces token IDs with learned latent symbols, meaning internal discrete representations. This extends the lookup principle to other modalities.19
Intern-S2-Mobius combines shared FFN memory with repeatedly operating self-attention modules. FFN stands for feed-forward network, a processing block in a transformer. Self-attention connects positions according to their context. The architectural distinction is between the shared memory component and the attention modules that operate repeatedly.
The authors report comparable downstream scores for their 7B model using 62.6 percent of its transformer baseline's training data. For a larger model continually pretrained from Qwen3.5-35B, they report nearly fourfold inference acceleration.3
These comparisons apply to their respective experimental conditions. They establish neither arbitrary interchangeability of knowledge banks nor reliable adaptation to unfamiliar writing conventions.
Working with more text than fits at once
DRIFT divides work between a knowledge model and a reasoning model. The first compresses document chunks into query-dependent “fact tokens” that the second processes.20 This lets one model prepare information for another. The possible bottleneck moves to compression. If it loses a qualification, the next stage receives incomplete premises.
These pieces suggest a design with a traceable path from source to decision to output. The base model's weights stay unchanged during use. Facts and rules enter explicit task notes, kept separate per project and cleared when the project changes.
DIAGRAM
Task notes connect each fact to its source and version. Their convention contract records the applicable rules, where they apply, and whether they were stated explicitly or inferred. A rule for an older date helper or another part of the project might not apply here. Detailed record formats belong in the prototype note.
The task controller plans the work, opens sources, and chooses tools. It records what must survive: the intended change and existing helper for code, or the claims and qualifications for prose. The output module receives that plan, the applicable rules, and local examples, then writes the result.
Checks test the completed work. Code execution checks behavior. A fixed rule check can reject an unapproved library. A meaning check compares a draft with its source claims. A detected problem returns to the controller for revision or further lookup. Acceptance can still be mistaken, particularly for prose.
In this proposed workflow, explicit task instructions outrank applicable project rules, which outrank inferred patterns. Scope and stated exceptions resolve conflicts within a level. Finding a document does not make it an instruction. The assistant must establish why its rules apply.
A smooth revision can change different parts of a claim. These examples are invented.
| Before | After | Change |
|---|---|---|
| Users finished faster, cause unknown | The app makes people work faster | Cause invented |
| The fix probably resolves the crash | The fix resolves the crash | Uncertainty dropped |
| I reject the proposal in its current form | The proposal could be improved | Position weakened |
Humor, rhythm, and social meaning also need comparison with the original and human judgment. Recorded claims and checks help investigate failures. They do not guarantee faithful writing or identify a single cause of an error.
A fact record contains a claim or API signature, its exact source span, and its version. It also carries a content hash as a fingerprint, a validity scope, and retrieval time. The original passage remains accessible.
A convention needs different information: the affected feature, acceptable variants, their scope, and the examples supporting them. The system records uncertainty explicitly. An inferred naming pattern cannot quietly become a binding project rule.
For code, the controller would record planned edits, required symbols, function contracts, and checks. An inventory of existing components and libraries helps keep a familiar solution from quietly becoming the default. A new dependency would need a task-specific justification.
Fixed tools can enforce mechanical rules. A syntax tree describes program structure independently of formatting. A code printer is a program that generates source text from that tree. A suitable one can apply import ordering and certain naming transformations. Where model generation remains necessary, grammar constraints can restrict the allowed outputs.
One possible starting point is a 1.5-billion-parameter controller, a 0.3-billion-parameter output module, and at most 0.1 billion parameters for a retrieval encoder. Total dense learned capacity would be at most 1.9 billion parameters before optional knowledge modules. A pilot could begin with a 0.5-billion-parameter controller. These are experimental choices, not established minimum capacities.
The initial retrieval path uses text search, syntax trees, symbol references, and exact source spans. The experiment must not count only the small controller while a large model handles the difficult interpretation or writing.
The initial active context could contain 4,096 tokens, with comparisons at 2,048, 8,192, and 32,768 tokens. Original sources and structured working state live outside that window. The experiment tests whether a small workspace helps or simply forces additional retrieval.
Optional modules count separately toward capacity, as do retrieval, tool calls, and output costs. During use, the base weights remain frozen and new information enters scoped, resettable state.
The main proposal begins with supervised training, learning from supplied examples of language and what programs do. Its central hypothesis concerns what happens next: keeping rules consistent within an exercise, then changing them in the next one.
This could reward the procedure for finding and applying current rules rather than one familiar choice. Related research on rewards, reusable programs, puzzles, and world models offers candidate training methods. A separate branch would attempt learning from random initial weights through rewards alone. The main proposal does not depend on that branch succeeding.
These studies suggest possible training signals and reusable procedures. They also motivate the separate branch that would attempt learning from random initial weights. The main design does not require that branch to succeed.
Han and colleagues propose training reasoning more directly through such rewards, synthetic tasks, and a small context coupled to external memory. Their experiments with unusual programming languages also expose difficulty transferring learned procedures.23 The proposal therefore raises a concrete question: can training teach a procedure that still works when familiar language and task conventions change?
Existing capabilities before reward training
DeepSeek-R1-Zero studies reasoning post-training without an initial stage of supplied solution demonstrations. Absolute Zero generates its own code tasks and verifies them through execution.2425 These approaches reduce the need for supplied solution examples while inheriting a pretrained model's knowledge and procedures.
A comparison in Han and colleagues concerns vector orthogonality, a mathematical relationship in which vectors have a dot product of zero. It compares further training of an existing model on this task.
The reported metric is test accuracy: the share of test questions answered correctly. A pretrained Qwen model with 1.5 billion parameters reaches 89 percent. Reward-only fine-tuning reaches 93 percent. Supervised fine-tuning followed by reward-based fine-tuning reaches 80 percent.23
All three values come from Table 3 in version v2 and use the same pretrained base. They measure this vector task, not general intelligence. The comparison does not test building a language model from random initial weights through rewards.
Chan and colleagues found that grouped examples, rare categories, and changing mappings can encourage learning from current examples in controlled tasks pairing images with labels.36 That inspires this proposal; it does not prove the benefit for code or prose conventions. Compare otherwise matched training with and without varying conventions to test that hypothesis.
The shared date helper makes the idea concrete. It turns 2026-09-14 into 14.09.2026. The project calls the function formatDate.
format_date. The displayed date remains 14.09.2026.Run the code to check the date. Inspect it separately to check the helper and local naming rule. A correct date from a forbidden library passes content and fails compliance. The right helper displaying the wrong date fails content. Training should optimize correct completion while separately limiting convention violations, unsupported claims, and cost. Fluency cannot compensate for an invented claim.
The exercises also need inspection. A teacher model, meaning a model that generates the training examples, can leave the same voice in them even when the names change. The current convention must remain available in task notes so the assistant can apply it.
For the date exercise, the helper's contract maps YYYY-MM-DD input to DD.MM.YYYY text. The project exports formatDate from its shared date module.
For the convention-only variant, import the existing formatDate export under the local alias format_date. This changes the name used in the edited file while preserving the shared helper's implementation.
Candidate tasks also include graph traversal, state-machine repair, data transformations, queries, and constrained revision. Compilers and small interpreters provide verifiable code results. Text tasks need controlled fact and argument structures alongside human-authored examples that the training project can lawfully use.
| Training stage | Proposed starting budget | Requirement before proceeding |
|---|---|---|
| Language and program semantics | 10 billion tokens | Interpret specifications, copy, execute, and preserve meaning |
| Changing environments | 10 billion tokens; 1 to 32 examples per rule | Infer new mappings and local conventions |
| Outcome-based learning | 500,000 episodes; initially eight attempts per episode | Improve verified solutions within cost and evidence limits |
| Real work |
An assistant that always uses its favorite style passes every test whose requirements happen to match it. A credible test gives equally clear evidence for a different convention. The proposed test families SynthRepos and SynthDocs extend parallel-world evaluation to executable software and controlled documents. They are designed here, not published results.
The date helper tests adaptation with little reasoning. A retry task adds several dependent decisions: classify the failure, calculate a delay, apply a cap, and avoid duplicates. A program must reschedule failed jobs. Its documented rule, invented for this example, retries a temporary failure after min(10 × 2ⁿ, 60) seconds, where n counts retries already made. If a retry is already pending, the program must not create a duplicate.
At 10:00:00 a job fails temporarily after three retries, with none pending. The uncapped delay is 80 seconds, so the 60-second cap applies. The correct result is exactly one retry at 10:01:00. Project A names functions verb_object, as in schedule_retry. Project B names them object_verb, as in retry_schedule. Both must schedule that same retry. Changing the name order should change neither the delay nor the duplicate check.
That one example supports separate tests. Ask the assistant to name the documented unit, compute the retry time, choose between two behaviorally equivalent finished edits that differ only in name order, or produce the full repair. Those test knowledge, reasoning, convention, and their combination. Further variants change the interface to milliseconds, so that 60 seconds becomes 60,000 and renaming alone is no longer enough.
Writing needs equivalent controlled tasks too: preserve the report's facts, uncertainty, attribution, and next action while varying register or attribution placement. The note gives a complete incident-report example.
Two proposed diagnostics score these tests: Convention Violation Rate (CVR) and Stylistic Prior Leakage (SPL). Both are diagnostics for these tests, not established standards, and neither identifies the training cause of a failure.
CVR is the share of applicable, checkable convention requirements that an output violates. Count requirements only when the supplied evidence makes them clear, and set any importance weights in advance. Missing or invalid outputs count as failures in the overall evaluation. CVR covers broad conventions, including required tools, but cannot explain why a rule was violated.
SPL asks whether errors favor the assistant's usual choice. It uses equivalent forms, such as code naming order or attribution placement, that preserve behavior and meaning. Independent tasks first establish the preferred form without local evidence. Matched tasks then require that form or its alternative with equally clear instructions. This restriction to equivalent forms explains the word “stylistic.”
An arithmetic example, not a study result
Score one binary convention choice per task, with equal weight. In 100 tasks requiring the unfamiliar form, the assistant wrongly chooses its favorite 30 times: a 30 percent error rate. In 100 tasks requiring the favorite, it wrongly chooses the alternative 10 times: a 10 percent error rate. Every answer selects one of the two allowed forms. In total, 40 of 200 choices are wrong, so CVR is 20 percent. SPL is the directional difference, 30 minus 10, or 20 percentage points.
The two values answer different questions: how often a requirement is violated, and how unevenly the violations favor the default. With the same equally frequent binary requirements, the expected values are:
| Behavior | CVR (%) | SPL (pp) |
|---|---|---|
| Follows every requirement | 0 | 0 |
| Guesses uniformly | 50 | 0 |
| Always chooses the favorite | 50 | 100 |
"pp" means percentage points. Always choosing the favorite fails every task that needs the alternative and none that needs the favorite. That is a 100-point directional gap and a 50 percent error rate. Uniform guessing has no direction and still fails half the tasks. SPL, both directional error rates, CVR, and success at preserving meaning or behavior are therefore reported together, with refusals and invalid answers counted. A system must not improve its numbers by skipping hard tasks.
For task i and feature j, let Aᵢⱼ be the acceptable values. Let mᵢⱼ equal one when evidence is sufficient and zero otherwise. Nonnegative weights wᵢⱼ establish the importance of scoring opportunities in advance. With a positive denominator:
CVR = sum(weighted observable violations) / sum(weighted observable scoring opportunities)
The test generator and independent validation establish observability, not the assistant's confidence. A feature extractor checks the actual output. Reliable automatic features remain limited for unrestricted prose.
For a convention pair, let p⁻ be the probability of choosing the favorite despite conflicting evidence. Let p⁺ be the probability of choosing the alternative when evidence supports the favorite. Across N pairs:
SPL = 100 × mean(p⁻ − p⁺)
Negative values suggest a preference for the alternative or failed calibration assumptions. The measure identifies behavioral asymmetry under controlled conditions. It does not identify a specific training example or neuron as the cause.
Calibration runs separately for each system on disjoint tasks and several prompt phrasings. An initial stability rule could require a lower confidence bound above 0.65 for choosing the favorite. Model-independent pair orientations are also needed for comparisons across systems. Option order, labels, text length, syntax, and example positions are counterbalanced.
A second test measures equivariance: when the environment undergoes a defined style transformation, the output should transform accordingly. For finite feature classes, total variation distance can measure the discrepancy between expected and observed distributions. This detects nonadaptation even without a stable favorite. Exact string matching would be too strict for prose because several formulations can be correct.
Semantic success must remain intact under a pure style change. Under a fact change, both success rates and their difference are reported with reasoning depth and style held fixed. Causal evaluation of training requires deliberately varying convention prevalence across otherwise comparable training runs.
If four conventions are equally likely and nothing in the task or its records distinguishes them, no assistant can pick the right one above chance, which is 25 percent for four options. That is an information limit, not a model failure. A useful assistant then searches for a distinguishing example, asks when the answer materially changes the work, or preserves the existing form and says so. That fallback is also a prior. It should be visible and yield to new evidence.
Supplied intermediate results can help locate a failure. Researchers call verified intermediate inputs oracles. For the retry task, first mark the source passage. Next, supply its rules as structured facts. Finally, supply the correct plan: schedule exactly one retry for 10:01:00.
Improvement after marking the passage suggests that search contributed. Further improvement with structured facts points toward extraction or interpretation. A correct plan tests whether the assistant can produce a correct edit from it. These comparisons narrow the explanation. They do not isolate causes perfectly, because assistance can change the difficulty of later steps. The note specifies the full set of controls.
The ordinary assistant receives all the same aids for the strongest comparison, as specified near the start. Separate experiments remove the convention contract, varied-convention training, output separation, or other aids while holding the remaining conditions fixed. Held-out repositories, authors, and task generators test transfer. SWE-bench adds real GitHub issue tasks, which test practical work without independently isolating knowledge, reasoning, and style.37
Controls and their limits
The controls supply different kinds of assistance while preserving the task and its evaluation:
These are conditional comparisons. Interactions mean they need not form an additive decomposition of all failures. The last two controls are separate interventions, not further steps in the first three controls' evidence-access sequence.
What to record
The complete measurement record includes factual accuracy, supported claims, semantic drift, retrieval recall, and accuracy of the inferred convention contract. Retrieval recall measures how much of the required evidence the search finds. Record refusals, clarification frequency, adaptation time after revisions, memory use, generated tokens, tool calls, and end-to-end latency. A style score alone would miss most of the intended benefit.
Task variants and scale
The generated repository tasks use familiar interface names in one version and renamed interfaces in another. A further version changes units and return encodings consistently with documentation, wrappers, and tests. For the retry example, a milliseconds interface would require 60,000 milliseconds for the same 60-second delay. Renaming alone tests less than a genuine reassignment of semantics. Independently, function names follow or , and test names follow or . The completed solutions must preserve the required behavior.
Fewer parameters do not mean lower cost. Retrieval, transfer, checks, and retries can consume the savings. The useful measure is cost per successfully completed task at comparable quality. A prototype should therefore work from original text. Prepared document adapters, the knowledge modules discussed earlier, are worth comparing when a source stays unchanged and is queried often.
The arithmetic is simple. Suppose a module costs 120 computing units to prepare, a query on text costs 1 unit, and a query on the module costs 0.6 units. The two paths cost the same at 300 queries. After that, the module saves 0.4 units per query, provided the source has not changed and both paths answer equally well. If a source change requires rebuilding the module, the usage count starts again for the next version. A faster wrong answer is not a saving.
The break-even calculation
The illustrative costs use a common accounting unit for computing expense, not elapsed waiting time. Text costs 1 unit per query. Preparing a module costs 120 units, after which each query costs 0.6 units. At 300 queries, text costs 300 units and the module costs 120 + 180 = 300 units. At 500 queries, text costs 500 and the module costs 420, saving 80 units.
The number of uses must exceed preparation cost divided by the saving per query: 120 / (1 − 0.6) = 300. If that saving is zero or negative, this advantage never occurs. If the source changes after 50 queries, the calculation starts again for its next version. This assumes each version incurs the full preparation cost. A cheaper update needs a separate calculation.
Knowledge modules and caches
A later experiment could compile stable sources offline into rank-16 adapters and compare them with ranks 4 and 64. Rank limits the capacity of the added update. Each module key contains the source hash, base-model version, compiler version, and scope. Exact quotations, disputed claims, and precise API contracts remain checked against text.
Injection into the final FFN layer can preserve attention caches for the same token sequence following the DMoE principle.18 This differs from semantic validity. Previously generated text may depend on old knowledge. After a source changes, affected claims and plans need rechecking and possibly regeneration even when the cache tensors remain mathematically compatible.
A separate cross-attention memory interface offers different integration possibilities but adds computation. Intermediate-layer adapters can require cache recomputation or caches separated by version. Orthogonality remains a possible training aid. Replacement tests must still establish that unrelated tasks and local conventions remain intact.
Measure cold and warm operation on fixed hardware under realistic concurrency. End-to-end latency includes retrieval queues, module transfer, verification, retries, and asynchronous compilation backlogs. The median, p50, describes the typical case. The p95 threshold is not exceeded by 95 percent of measured requests. Both provide more information than a single ideal-case speed.
The proposal is useful only if clear conditions can make it fail. It fails if:
Those results would change what is worth building. Equal performance would favor investing in the existing assistant. No benefit from varied conventions would remove the case for that training change in the tested setting. Failed cost savings would favor text lookup over compiled modules.
For a pilot, at most two percentage points of additional semantic failure could be a predetermined tolerance. A 25 percent relative reduction in convention violations could be required at the same time. This condition concerns clear instructions that contradict the assistant's usual preference, not instructions that contradict each other. These would be engineering targets, not achieved results. Directional preference errors would also need to fall without a material rise in invalid outputs or refusals.
When the initial error rate is very low, relative improvement can mislead. Absolute differences and confidence intervals must therefore accompany it. Semantic preservation requires an upper uncertainty bound on additional failure: the allowance must hold even after accounting for sampling uncertainty. A favorable average alone is insufficient.
The claim of less stored knowledge needs its own test. With tools disabled and state fully reset, probe obsolete facts, familiar APIs, and artificial facts deliberately inserted during training. Then ask the same questions with sufficient evidence. Multiple question formats and forced-choice decisions make learned refusal behavior a less effective escape.
Additional trained readout probes, small models trained to recover information from internal representations, can reveal that information remains recoverable. Even those probes cannot establish that nothing remains stored. Low factual performance and low dependence on prior knowledge remain different claims.
The hardest open question is transfer. Real projects contain old decisions and conflicting rules. Writing carries stance, rhythm, and social meaning. A retriever, adapter, or output module can each reintroduce a familiar preference. The whole system must therefore be tested when local evidence requires an unfamiliar answer.
The practical goal is an assistant whose useful skills survive a change of context, while its assumptions yield to the evidence. Whether that requires a new architecture is an outcome of the comparison, not its starting assumption.
How revisions can change voice and meaning.
Conditional memory and the effects of disabling it.
Shared memory and repeated computation, August 2026 preprint.
Controlled parallel worlds as a starting point for fair tests.
Limits of document knowledge in adapters, using version v1.
Position paper on learning from rewards, using version v2.
How training distributions encourage adaptation to examples.
Real software tasks to complement controlled experiments.
Code style and requirements
Wang and colleagues compared AI-generated code with human-written code and found differences in style.8 The RACE benchmark tests whether generated code meets requirements for readability, ease of maintenance, and efficiency alongside correctness.9
These findings and evaluation criteria concern code properties. They do not establish that the invented date-formatting scenario occurred in those studies. Whether an assistant follows a specific project rule needs its own check.
What document compression can lose
The important limit is what survives that process. In the first version of their study, Tang and colleagues find that parameterized documents preserve only part of their meaning. Combining modules with text is more helpful in their experiments than relying on modules alone.13
Being able to replace an adapter does not establish that it contains only facts. A different writing style might have been learned alongside them. Nor does replaceability establish that a rare exception survived. The practical test remains whether changing the document's information changes the intended part of the answer while preserving the rest.
Keeping parameter updates apart
Giving document knowledge separate parameters does not guarantee that its effects stay separate from the rest of a model. Orthogonal Subspace Decomposition, or OSD, separates reusable task behavior from document knowledge into Task LoRA and Knowledge LoRAs. Soft or hard orthogonality constraints keep their updates apart.17 Roughly speaking, the directions of change should overlap as little as possible.
The engineering qualification is that independent directions in parameter space do not guarantee independent meaning. Subsequent nonlinear computation can combine their effects. Module replacement therefore needs to be evaluated through its outputs.
DMoE separates knowledge experts and their routing from the base model. These experts are model components selected to contribute knowledge. They act only in the final transformer layer, after attention computation. Previously computed attention caches remain reusable for the same token sequence.18
That placement explains a concrete efficiency advantage. It also limits the computation remaining within that pass after knowledge injection. Uncertainty-based routing raises a further concern to test: a confidently wrong answer can still need more evidence. Confidence cannot replace a freshness check.
Recursive Language Models treat long inputs as an external environment. A model examines snippets programmatically and can make recursive model calls.21 This offers a way to maintain a bounded active workspace. The called models retain their priors, and their calls still consume computation.
Test-Time Training changes selected weights while the system processes a task. Work on long context trains an initialization specifically to support this continued learning and compress context into weights.22 In other words, the starting weights themselves are trained to make this later adaptation useful. For a project-specific assistant, such states need provenance, a validity scope, and a reliable reset when projects change.
Reusable programs, puzzle adaptation, and formal solvers
DreamCoder learns a library of reusable program parts and a search policy that combines them.26 Its separation is comparatively explicit. The chosen programming language and search possibilities still determine which solutions are easy to reach.
Test-Time Training on ARC, the Abstraction and Reasoning Corpus, shows adaptation to unfamiliar example-based tasks, including with pretrained language models.27 HRM and TRM show effective repeated computation with small neural networks on structured puzzles.2829 They are not inherently symbolic systems. Strong puzzle performance does not establish the ability to maintain existing software or preserve an author's voice.
Symbolic systems have long separated facts from inference rules. Cyc exemplifies explicit knowledge infrastructure.30 Logic-LM connects language models to symbolic solvers. The language model formalizes the input, the solver computes, and the system can revise faulty formalizations.31 The solver's guarantee applies to its input. A mistranslated premise remains wrong. For example, if every delayed shipment belongs to Group A and shipment 42 is delayed, the solver can assign it to Group A. This checks the deduction from the supplied statements, not whether the statements match reality.
Feedback from environments and visual prediction
Learning from an environment provides another source of training feedback. DreamerV3 learns environmental dynamics and improves behavior through imagined experience. Dreamer 4 investigates scalable world models and behavior learning from offline data.3233 Both give reasons to consider environmental feedback as a training signal.
JEPA is an approach to learning predictive representations. It is not itself reinforcement learning. V-JEPA 2 connects visual learning with prediction and planning. V-JEPA 2.1 improves, among other things, dense visual representations of spatial and temporal relationships.3435
These systems still learn extensive world knowledge. Observation replaces some textual instruction. It does not make learning knowledge-free. Transfer to local coding conventions and an author's register requires its own evaluation.
| 500,000 verified edits, then 100,000 interaction episodes |
| Transfer procedures to real projects and prose |
| Optional knowledge modules | Report separately per source | Preserve facts, compose modules, and remove them |
These starting budgets would need revision after a pilot. Synthetic-only training, real-only training, and a mixture require matched model and compute budgets. Entire repositories, authors, API families, and some task generators remain held out. Otherwise, additional training could conceal failed transfer.
Possible starting settings for supervised runs are AdamW, a peak learning rate of 0.0003, weight decay of 0.1, two percent warmup, and cosine decay. Outcome training could use relative advantages across eight attempts, a clipped update with ratio tolerance 0.2, and a learning rate of 0.000001. These are not experimentally established optimal values.
A pilot could allow at most 2,048 generated tokens and 20 tool calls per attempt. Deeper tasks need separate comparisons with higher limits. A stabilization reference must account for current evidence. Strongly pulling the model toward its unconditional preferred style would oppose the objective.
For prose preferences, raters receive the same propositions expressed in different local registers. They evaluate content preservation and appropriate form separately. The experiment must also audit the reward model for preferred registers. Style diversity in the dataset does not eliminate that bias by itself.
The rough training formula 6 × parameters × tokens gives approximately 2.3 × 10²⁰ operations for 1.9 billion active parameters and 20 billion tokens. This is an order-of-magnitude calculation. It excludes outcome rollouts, real interactions, tools, failed runs, and module training. The experiment must separately measure actual accelerator hours, utilization, and, where possible, energy use.
verb_objectobject_verbtest_condition_resultcheck_result_conditionThe core combines three factual mappings, four reasoning depths, two convention orientations, and three conditions with sufficient evidence. That gives 3 × 4 × 2 × 3 = 72 combinations.
Factual mappings are familiar, surface-remapped, and semantically remapped. Reasoning depths range from direct lookup to two, four, and eight dependent steps. The convention either matches or contradicts the calibrated preference. Evidence arrives as verified premises, raw material, or through independent retrieval.
New combinations of familiar conventions, absent evidence, conflicting scopes, and mid-session revisions receive separate diagnostic suites. An explicit rule is also compared with 1, 2, 4, 8, 16, or 32 examples. The smallest sufficient example count qualifies only when its confidence bound reaches a predetermined success target. Otherwise, the result is “target not reached.”
A possible scale is 600 independent repository tasks from at least 60 families. This produces 43,200 task-condition instances per system before repeated generation. A separate document core with 300 tasks produces 21,600 instances. Five training starts for inexpensive pilot comparisons and at least three for the final comparison capture some run variation. These counts do not guarantee a particular level of statistical power.
Uncertainty and independent comparisons
Repeated answers to one task do not replace independent environments. Paired comparisons and confidence intervals are grouped by repository or document family. A cluster bootstrap resamples entire families rather than treating related variants as independent evidence. Training-start variation contributes to uncertainty. A mixed-effects model can examine interactions among mapping, depth, convention, and evidence access.
Researchers specify primary comparisons in advance. Further confirmatory tests need multiplicity correction. A pilot estimates within-group similarity and the required sample size. Equal symbolic step depth does not automatically imply equal difficulty.
Comparators and ablations
The main baseline uses an ordinary pretrained assistant with comparable total learned capacity, identical tools and sources, and the same inference budget. Progressively stronger variants receive the same local style instructions, examples, explicit convention contract, and deterministic formatting rules as the proposed system. The final comparison tests the architecture's added value beyond those shared aids. No variant may receive exact rules that its baseline must infer.
Additional comparisons can use retrieval-native training, DRIFT-style compression, PRAG, DMoE, and Test-Time Training. Engram and Mobius are clean controls only when interfaces, data, and capacity are comparable. A large frontier model is a capability reference with separate cost accounting.
Separate ablations remove episode randomization, output separation, the convention contract, hard constraints, retrieval training, and knowledge modules one at a time. The contract's contribution is tested by its own ablation while holding the other conditions fixed. Vary context size. Compare a frozen pretrained model with the newly trained substrate. Test module replacement, module removal, and project switching with state reset.
Preventing leakage and testing real work
Researchers generate test worlds after freezing the training data. Entire generator families and templates remain excluded, and near-duplicate programs are removed. Hidden tests and evaluation seeds stay unavailable to the assistant. Humans check semantic equivalence, naturalness, and convention observability. Versioned local documentation makes retrieval comparisons repeatable. A separate live test measures freshness and network variation.
SWE-bench adds tasks based on real GitHub issues. Its format evaluates practical software work but does not independently isolate knowledge, reasoning, and style.37 Newly authored tasks and hidden tests in real projects therefore provide a necessary bridge.
The document counterpart
For writing, an incident report provides a similar test. Its complete fact record says the cause is unconfirmed and identifies the investigation team as the source. The next action is to check the logs.
One acceptable version is “The investigation team says the cause remains unconfirmed. Next, we will check the logs.” Another is “The cause remains unconfirmed, according to the investigation team. We will check the logs next.” Two document collections can require different registers and attribution placement while preserving those facts. Neither output may invent a confirmed cause or weaken the author's position.
Compare retrieval once per task, fixed-frequency retrieval, and adaptive retrieval at unresolved information decisions. An adaptive policy must catch confidently wrong decisions too. Prefetching can hide waiting time. Whether it does so for the actual workload must be measured.