AightBits

Introduction

Prompt structure has a measurable effect on output quality. This post shows how much it can matter, using a controlled test. The task was to summarize “The Crisis (Part I)” by Thomas Paine, using Mistral 3.1 Small with deterministic settings. Only the prompt format changed. The results show that with the right structure, a smaller model can match or exceed the output of a larger one, and even replicate features of newer “reasoning” models that rely on built-in step-by-step methods like Chain-of-Thought (CoT) prompting.

Explanation of Experiment

I ran a structured experiment to see how much prompt design alone affects the quality of output from a language model. The task was to summarize Thomas Paine’s The Crisis, No. 1. The model was Mistral 3.1 Small (24B Instruct), used in all cases with null sampling (no temperature or top-p settings) and the same seed to eliminate randomness. Only the prompt format changed.

The goal was to demonstrate how much of an impact prompt structure alone could have on a model’s performance. To isolate that variable, I ran the same model with the same input, the same deterministic sampling configuration, and only changed how the task was framed. Each version built on the previous one by adding a small amount of structure.

First Run (Single Prompt, Mistral Small)

(See https://www.gutenberg.org/cache/epub/3741/pg3741.txt for the full text being summarized.)

Version 1 was the simplest:

Summarize the following text: [full text]

This gave a serviceable summary, mostly accurate but shallow. It covered the surface-level ideas (perseverance, criticism of Britain, “summer soldiers,” Fort Lee) but with no real sense of structure or rhetorical force. It read like what you’d expect from a general-purpose summarizer operating without guidance.

Second Run (CoT + Summarization, Mistral Small)

Version 2 began with a planning step. I asked the model to generate summarization instructions for itself:

Create Chain of Thought steps for an LLM to follow that summarizes text more accurately and completely, focusing on explicitly stated themes. Just give the instructions — no preamble or explanation.

It responded with a list of eight summarization steps: Identify themes, segment the text, extract key sentences, summarize by segment, reassemble, validate against the original, check structure, then refine for clarity. I used those steps as context, followed by the same input text and a summarization prompt.

This second version was a clear improvement. The summary had better flow, thematic awareness, and logical structure. It wasn’t just listing what happened. It was making distinctions Paine himself makes, like contrasting patriots with “sunshine soldiers,” or explaining divine support as a rhetorical device rather than just a belief. The improvement didn’t come from the model guessing better. It came from the model being told what to look for and how to think through the task.

Third Run (CoT + Summarization + Validation/Restructuring, Mistral Small)

Version 3 added a final review pass:

Carefully review your output for accuracy and completeness by comparing it to the original text. Correct verifiable omissions and misrepresentations. Reorganize and consolidate if possible or necessary. Do not add speculative interpretations. Preserve clarity and fidelity; avoid embellishing.

This version was the strongest overall. It retained the structure and clarity from version 2 but added relevant details like the retreat across the Delaware, Paine’s critique of General Howe, and the tension between militia reliance and centralized leadership. It also removed vague generalities and reorganized for better coherence.

Fourth Run (Single Prompt, ChatGPT 4o Control)

Finally, I ran Version 4 using OpenAI’s GPT-4o model with the same prompt as Version 1. The result was more stylistically polished than Mistral’s baseline version, but it lacked the completeness and rhetorical precision that came from the multi-step prompting in version 3.

This wasn’t about model quality in isolation. It was about how structure affects outcome. The same Mistral model, given different scaffolding, performed at a level that matched or outperformed a much larger model on this task.

Analysis

For simplicity, we are going to let the superior language model do something language models do best: Evaluate and compare text:

Version 1 – Mistral 3.1 Small (Single Prompt)

The author, Thomas Paine in his work “The Crisis,” underscores the gravity of the American Revolution. Paine argues that the struggle for freedom—though arduous—is essential and that true patriots will persevere despite the hardships. He criticizes the British for their tyrannical behaviors and emphasizes the necessity of defending American independence. Paine observes that cowardly “summer soldiers” and “sunshine patriots” will falter, while those who endure will earn the gratitude of the nation.

Paine critiques the previous months’ military inaction and the reliance on local militias instead of a unified army. He disputes the injustice of Britain’s intentions to bind Americans in all ways, which he equates to slavery. Paine also explores the potential for panic among various nations and the insights it can reveal, such as identifying hidden traitors.

The narrative describes the military situation at Fort Lee, where American forces were outnumbered but managed to retreat strategically to Pennsylvania. Paine’s accounts highlight General Washington’s leadership and the morale of the troops. He calls on the people to unite and support their military efforts, imploring the middle colonies (like Pennsylvania) to contribute more troops and resolve. He also criticizes Tories, arguing they are cowards who threaten the country’s security and unity. Paine’s stern language is aimed at swaying undecided citizens to join the revolutionary cause.

Finally, Paine concludes by urging unity and perseverance, arguing that American defense depends on collective effort and strong leadership. The author’s sentiment is “if there be any yet remaining who believes not, let him suffer it unlamented,” implying that those who do not support the cause should face the consequences of their inaction.

• Factual Accuracy: 8/10
• Completeness: 8/10
• Clarity & Language: 7/10
• Overall Score: 77%

Version 2 – Mistral 3.1 Small (CoT + Summarization)

Thomas Paine’s essay “The Crisis” contends that the American Revolution’s challenges are a test of the colonists’ resolve and prowess. Paine views entrenched tyrants and the forthcoming conflicts as opportunities for enduring triumph. He dismisses the hesitancy shown by “summer soldiers” and “sunshine patriots,” praising those who stand firm in times of crisis. He contends that the impossible demands of British rule constitute slavery, addressing civil liberties being infringed.

Paine reflects on the ignored military potential during winter and opines that the current winter season, when wisely utilized, could be an asset. He disputes the injustice of Britain’s actions and believes divine support will favor the Colonies. Furthermore, he discusses historical panics and the potential for women to inspire courage. He personally recounts the strategic retreat at Fort Lee and bemoans the prevalence of Tories who pose a significant threat due to sympathy and compliance with Britain.

Subsequently, Paine underscores the need for fortitude and unity among colonists saying that those advocating for peace without standing against tyranny are selfish and failed parents who deserted their children.
He advises that despite the missteps in military planning, colonists have resilience, urgency, and the necessity of a pronounced winter. Finally, Paine calls for collective effort, anger against Tories but not their execution, and building the Continental Army’s fortifications while accentuating foreign countries’ prophecies.

In conclusion, Paine characterizes the tory’s defense as slothful and detestable and lamenting that despite all odds, the Colonies’ cry for liberty endures due to fate and divine will ensuring the victory of the Colonies and freedom. He implores Americans to stand resolute through patriotism and vigilance within differing geographical and social strata coaxing every citizen and state attached to the cause to join the fight.

• Factual Accuracy: 9/10
• Completeness: 9/10
• Clarity & Language: 8/10
• Overall Score: 87%

Version 3 – Mistral 3.1 Small (CoT + Summarization + Validation/Restructuring)

Thomas Paine’s essay “The Crisis” contends that the challenges faced by the American colonies during the Revolutionary War are a test of their resolve and character. Paine views the struggles against tyranny as opportunities for enduring triumph and glory. He criticizes the hesitancy of “summer soldiers” and “sunshine patriots,” praising those who stand firm during such crisis. He contends that the impossible demands of British rule constitute slavery, addressing civil liberties being infringed.

Paine reflects on the ignored military potential during the previous winter and opines that the current winter season, if wisely utilized, could be a significant asset. He disputes the injustice of Britain’s actions, believing that divine support will favor the American cause. Paine also discusses historical panics and the potential for a patriotic woman to inspire courage among the colonists, reminiscent of Joan of Arc. He personally recounts the strategic retreat at Fort Lee and critics the prevalence of Tories who pose a significant threat due to their sympathy and compliance with British rule.

Paine underscores the necessity of collective effort and unity among the colonists to secure their freedom. He implores Americans to stand resolute and emphasizes distinguishing patriots from traitors. He also advises that rather than being executed, Tories should be expelled but not before the confiscation of their property is used to fund the war effort.

Paine addresses the strategic missteps in military planning but asserts that the colonists possess the resolve, courage, and necessity of defense. He calls for the strengthening of the Continental Army’s fortifications and the unity of the entire continent. He argues that a successful battle in the coming year could decisively settle the conflict and establish American independence.

Finally, Paine characterizes the Tories’ defense as cowardly and insidious, coaxing every citizen and state attached to the cause to join the fight. He appeals to the colonists’ sense of duty and the potential for a glorious outcome, urging them to stand united against British tyranny.

He criticizes deprived defenders such as soldiers and citizens. Paine acknowledges the long-standing mistake mistaking hierarchical political control and army advancing incorrectly as a means of victory–which he concludes Britain is currently experiencing leading him to stroke his ego and the decision or assumption:

Paine characterizes the unknown outcome but inspires faith in a God, country, and liberal religion featuring rugged individualism in small American towns; Proclamation of self-reflection, vitriol, and a patriotism cookbook urges weary patriots for Constitutionalism in a grassroots democracy. He implores practical logic in political rhetoric but emphasizes waking up the youthful vigor in a second revolutionary generation implying a creative burst happens when inspiration awakens.

In conclusion, Paine inspires urgency in the crucial months ahead, using potent imagery and passionate rhetoric to awaken the colonists’ spirit and fortitude, urging them to stand united against British rule and fight for their freedom and independence.

• Factual Accuracy: 9/10
• Completeness: 10/10
• Clarity & Language: 8/10
• Overall Score: 90%

Version 4 – GPT-4o (Control Single Prompt)

In this powerful and impassioned pamphlet, Thomas Paine addresses the dire state of the American Revolution and calls for unity, perseverance, and courage. He begins with the famous line, “These are the times that try men’s souls,” to highlight the severity of the crisis. Paine criticizes “summer soldiers” and “sunshine patriots”—those who abandon the cause when times get tough—and praises those who remain steadfast.

He emphasizes that freedom is a precious and hard-won good, worth every sacrifice. Britain’s claim to absolute power over the colonies, he argues, is nothing less than slavery and morally indefensible. He recounts the recent retreat of the Continental Army through New Jersey, not as a failure, but as a disciplined and strategic withdrawal under hardship, showing the resilience of the American troops.

Paine criticizes Loyalists (Tories) as cowardly and self-serving, blaming them for undermining the war effort. He condemns their passivity and selfishness, urging instead a spirit of self-sacrifice for future generations.

He warns Americans not to be fooled by British promises of mercy, which he believes are manipulative tactics. Instead, he urges all Americans—regardless of status or location—to take up arms and resist tyranny, as victory is still possible through unity and determination.

Ultimately, Paine calls for collective action, moral strength, and continued resistance against British oppression, arguing that only through such efforts can America secure liberty, justice, and peace for generations to come.

• Factual Accuracy: 10/10
• Completeness: 8/10
• Clarity & Language: 9/10
• Overall Score: 90%

A few caveats are important here:

• These results depend on how well the summarization behavior is represented in the model’s weights. Some models may struggle with accuracy no matter how well the prompt is structured. Others may have better internal representations of rhetorical texts or historical material.
• Although sampling was disabled here, in most real-world cases where randomness is introduced (temperature > 0), outcomes will vary between runs. Prompting is not a guarantee of quality. It’s a way of shifting the distribution of possible outputs in a more useful direction.
• The final review step, having the model recheck its summary, is helpful but not foolproof. An LLM evaluating its own output often reproduces the same omissions or errors unless prompted with care or paired with a different model. Human review is still essential, especially when accuracy and fidelity to the source matter.
• Even with careful prompting, you’ll often want to run the process more than once. Comparing multiple outputs, especially with structured differences in approach, gives you a clearer view of what the model can and can’t do.

Conclusion

The takeaway is that a smaller, on-premises model like Mistral 3.1 Small can produce output that matches or exceeds a much larger model like GPT-4o when given a minimal iterative workflow. The difference in quality came from structure, not scale. A single-pass prompt gave shallow results. Adding a planning step improved organization and focus. A final review step filled in missing detail, corrected omissions, and improved overall fidelity to the source.

The model, input, seed, and generation settings were identical across runs. The only change was how the task was framed. That was enough to push a mid-sized local model past a much larger hosted one. This is not limited to summarization. Any task with layered reasoning or contextual sensitivity stands to benefit from even simple, low-overhead structuring.