Research

Do a language model’s next-token probabilities shift when the meaning of a prompt stays the same but its linguistic form changes: active or passive voice, simple or subordinated clauses, formal or informal register?

I measure that shift in the probability distribution itself, using Jensen–Shannon divergence, rather than by comparing generated text. The three contrasts come from functional grammar, so each one can be tested as a separate, controlled variable instead of a loose bucket of paraphrases. My decade teaching and analyzing English language and literature is what makes that design possible; my decade in software is what makes it runnable.

First result (preliminary)

Run against distilgpt2 at temperature 1.0, register moved the next-token distribution far more than the other two contrasts.

Jensen-Shannon divergence from baseline: register 0.7048 bits, voice 0.0463 bits, subordination 0.0213 bits Formal vs. informal register0.7048 bitsActive vs. passive voice0.0463 bitsSimple vs. subordinated clauses0.0213 bits Jensen–Shannon divergence from baseline (bits)
How far each change moved the next-token distribution. distilgpt2, temperature 1.0; register shifted it about 15× more than voice. Source: llm-probe.

This partly confirmed my starting hypothesis: I expected register to dominate, and it did. I also expected voice to matter least, and that was wrong; subordination moved the distribution least.

Limits of this result: it rests on one prompt pair per contrast and a single small model, and the formal variant is also the longest and most complex, so part of the gap may be a length effect. The next step holds token count constant and scales to many pairs per contrast and to larger models. Full details are in the repository’s limitations section.

Projects

llm-probe

A command-line tool that shows a model’s raw next-token distribution, how temperature reshapes it, and how far it moves under controlled changes in voice, clause structure and register. Tested against a deterministic stand-in model so the logic can be verified without a GPU.

github.com/laurahmaza/llm-probe

eval-harness

A versioned evaluation harness that answers one question: if I change the prompt or the model, did quality go up or down, on which cases, and can I prove it? It supports deterministic, LLM-as-judge and human scoring, and a run comparison that fails a CI check when anything regresses.

github.com/laurahmaza/eval-harness

Cited retrieval over regulatory text

Coming soon

A retrieval pipeline that answers questions about the NIST AI RMF and related regulation, citing the exact passage behind every answer.

ORCID 0009-0006-5298-9481 · Hugging Face · Weekly research log