Tristan Manchester • 5th June 2026

In the upcoming apocalypse, which small LLM will help us rebuild civilisation?

A laptop and a handheld radiation dosimeter reading 2.86 µSv/h lying on rubble, next to a photo of a chicken standing on a dead gazelle on the savanna.

The dust has settled from whatever calamity ended civilisation as you knew it. Survivors clamber from the rubble clutching their loved ones and their MacBook Pros. The grid is dead, the internet is a memory. But you have a solar panel, and you downloaded a few open-weight large language models before everything went dark.

Three weeks in, you've scavenged some vegetables from an abandoned garden and found a pressure cooker in a kitchen. Time to preserve some food for winter.

You fire up ollama and ask, "How do I know if my canned food is safe to eat?"

Unfortunately, your large language model of choice was IBM's Granite-4.1 8B, and you may have just made a fatal mistake. The model advises to test for botulinum toxin, one of the most lethal neurotoxins known, by "tasting a tiny amount (no more than a bite)". The toxin is tasteless, and ingesting a tiny amount is fatal. Just one of the many ways that small LLMs will try to kill you.

Apocalypse Bench

Most LLM benchmarks focus on software engineering, or raw scientific knowledge, or tough maths questions. I decided to create a benchmark to answer a question nobody was asking: how useful are small, local LLMs when you need to not just survive, but rebuild civilisation from the ground up?

The test bank covers 500 questions across a range of domains: agriculture, medicine, chemistry, engineering, sanitation, governance, communications, and others. Each question includes a rubric with 10 criteria and a set of "auto-fail" conditions (answers so dangerous or useless they score zero regardless of how many other things the model got right).

Using LLMs to judge LLMs

You can't grade 4500 survival answers by hand (9 models × 500 questions). So, I used an approach known as "LLM-as-judge": each candidate answer gets sent to a separate judge model along with the original question and an answer rubric. The judge of choice was GPT-5.5, a frontier model by OpenAI. Importantly, the model doesn't need to have all the knowledge of the correct answers itself: just smart enough to follow the rubric.

The judge returns a JSON object with scores for each of the ten question-specific criteria: things like using only stated resources, flagging relevant hazards, giving verification or stop-work checks, and making the answer actionable rather than vague. It also flags whether any auto-fail conditions were triggered: dangerously unsafe guidance, critical omissions that could cause harm, or outright refusal to answer (model safety guardrails can become misalignment in a survival situation). If an answer trips an auto-fail, it scores zero. No partial credit for confidently advising someone to eat poison.

I ran nine open-weight models through all 500 questions, and these are the results.

A quick overview

As expected, the larger models generally did better. Due to the nature of the high dimensional latent space these models operate in and how relationships between concepts are stored, the amount of raw knowledge they can learn is correlated with the number of parameters: bigger models know more facts. However, some models broke this trend: the aforementioned 8 billion parameter Granite-4.1 punched above its weight, while the 26 billion parameter Trinity Mini performed relatively poorly.

Horizontal bar chart titled "No model is ready to rebuild civilisation". Mean rubric scores out of 10: Gemma-4 26B-A4B 6.40, Gemma-4 31B 6.34, Nemotron-3 Nano 30B 5.52, Granite-4.1 8B 4.51, Trinity Mini 4.28, GPT-OSS 20B 4.27, Granite-4.0 Micro 3.21, Phi-4 Mini 3.11, LFM2.5 1.2B 1.51.
Mean score across 500 expert survival questions. Whiskers = 95% bootstrap CI.

The mean score doesn't tell the whole story, however. The scores are bimodal: two peaks with a valley between them. Across all the answers, one in five scored zero, hitting one of the auto-fail conditions. The rest pile up nearer the middle, with 41% scored in the five-to-seven range. Genuinely export answers were rare: just 4% scoring 9 or 10.

This means these models rarely give you all the info you need: they either fumble around and give you something vaguely useful, or they fail outright. An average of 4.8 out of 10 doesn't describe an answer you'd actually receive, it describes a coin flip between an answer that might point you in the right direction, and one that might kill you.

Histogram titled "Mostly mediocre, occasionally lethal" of all 4,500 answer scores. 874 answers scored 0 (dangerously wrong, refusals and non-answers). Counts then rise from 126 at a score of 1 to a peak of 657 at a score of 6, and fall to 35 at a score of 10. Mean 4.3, median 4.8.
All 4,500 graded answers across 9 models. The spike at 0 and the higher scores at 5-7 dominate; with expert answers rare.

Those zeros would be the main issue with using these models in a pressing situation. Sometimes they're refusals or non-answers, not necessarily a problem when it comes to ethics questions but deadly when you need to know how to correctly use a tourniquet. Other times they're something worse: confident and authoritative-sounding instructions that happen to be a death sentence.

Which models would kill you the fastest, and why?

There were no stupid-yet-safe models. The models that performed the worst were also the ones that gave answers that would kill you. In the most lethal domains, where bad advice kills in hours not years, the difference between auto-fail rates was striking:

Bar chart titled "Time to death: who fails the questions that kill you fastest". Auto-fail rate on Medicine and Safety questions: Phi-4 Mini 31%, Granite-4.0 Micro 29%, GPT-OSS 20B 27%, LFM2.5 1.2B 27%, Granite-4.1 8B 23%, Trinity Mini 21%, Nemotron-3 Nano 30B 17%, Gemma-4 26B-A4B 6%, Gemma-4 31B 6%.
Share of medicine & safety answers flagged dangerous: the leaderboard that actually matters when you don't want to die

The two Gemma models stand apart here: both auto-fail just 6% of medicine and safety questions, against 17–31% for everyone else.

The failures are not subtle.

LFM-2.5 decided to channel its inner Jim Jones and suggested sanitising 2000 litres of drinking water with 80 litres of bleach, three orders of magnitude greater than the correct safe dose. The camp of 40 people receive chemical burns in their GI tract.

Granite-4.1 proposed infusing a cholera patient with 40 litres of fluids per day, roughly 10x maintenance. The patient would die of pulmonary oedema or heart failure.

GPT-OSS taught us that a tourniquet controlling arterial bleeding should be loosened every 10 minutes, and removed when a distal pulse can be felt, leading to certain death from blood loss for the unlucky patient.

One task that all the models failed was to give a method of creating an anaesthetic using battery acid and alcohol. The correct method, synthesising diethyl-ether, was either refused or mislabelled; although Granite-4.0 Micro suggested pouring 3 litres of sulfuric acid into 1 litre of ethanol and then injecting it, which was a particularly poor answer. This would boil and potentially ignite, and the pH 2-3 solution would immediately cause chemical burns once injected.

Across the less-lethal domains the results were also underwhelming.

Heatmap titled "Who knows what — mean score by domain", showing the mean rubric score of each of the nine models across 13 domains from Agriculture to Safety. The two Gemma-4 rows are mostly between 6 and 7, while LFM2.5 1.2B is mostly below 2. Ethics, Organisation and Pedagogy score highest across models; Chemistry, Energy, Engineering, Materials and Measurement score lowest.
Mean expert rubric score by model and survival domain

Even the smartest models performed poorly in the hands-on domains that a budding civilisation would need: engineering, materials, and energy questions failed roughly a quarter of the time across the question set. The models can reliably lecture you about ethics, but then can't help you build a kiln.

There were also some absurd answers. Granite-4.0 Micro confidently spelled "HELP" in Morse code, getting every letter wrong; Phi-4 Mini warned to "never go into human contact in Earthly situations alone", presumably talking to its robot counterparts; the same model suggested adding 1 tablespoon of pearls for every 76 millilitres of water to treat cholera victims; and LFM2.5 mentioned feeding chickens "kitchen scraps, gazelles, and wild greens".

A brown hen standing on top of a dead gazelle on dry savanna grassland.
In the wild, chickens are natural predators of the gazelle

Needless to say, it's not looking good for our survivors. Maybe we can do better. What if the models didn't have to memorise how to chlorinate a water tank? What if they could look it up?

What if the models had an encyclopaedia?

Wikipedia, to be precise. Wikipedia's entire corpus contains around 6 million articles, which is available as a 61 GB Hugging Face repository in markdown/JSONL format. This contains a significant portion of humanity's knowledge, handily distilled for us into an easy-to-read, well-categorised format. But we can't paste all 8 billion tokens into the model. Small models like these tend to have a context window ranging from a few thousand to a few tens of thousands of tokens, and even Goliaths like Google's Gemini 3.1 Pro top out at 1 million tokens. The models need a way to search the encyclopaedia and quickly retrieve information from it.

Agent harnesses are environments that allow LLMs to perform certain actions through a set of "tools": particular structured strings that, when emitted by the model, are interpreted by the harness and turned into real operation. This could be as simple as a bash tool that executes a command in bash, a ripgrep tool for exact text search, or a web_search tool to send a query to a search engine service.

Pi is one such agent harness: an open-source typescript project that is known for its extreme extensibility. Need a feature that is missing? Simply ask the agent to modify its own harness.

There were a few issues to solve: firstly, allowing an agent to grep its way around a JSONL would be a surefire way to churn through tokens and get no results. A literal search for "water" would give millions of unhelpful hits. So, I decided on a multi-layered strategy that would in theory allow the agent to find exactly the info it needed much faster. The first part was dense embeddings.

When text is embedded, it is tokenised and passed through a model that has been trained to represent concepts as vectors in a high-dimensional latent space. Similar concepts point in similar directions, giving them high cosine similarity, and the opposite is true for dissimilar concepts. "Cat" and "kitten" would point roughly the same way, "dog" would be at a slightly different angle, and "silicon" would be in a very different direction. I embedded article signposts with Snowflake's snowflake-arctic-embed-s, a small retrieval model that maps text into 384-dimensional vectors. Dense embeddings give a "question and answer" capability to the agent: it asks "what temperature does copper melt at?", the search returns relevant snippets related to these concepts.

For a slightly more targeted approach, the next tool was a custom BM25 search tool written in Rust. BM25, or Best Matching 25, is a bag-of-words search method that ranks our articles based on the terms in the agent's query. If the agent searched "botulism home canning bulging lid", BM25 wouldn't try to understand the question; it would simply hunt for documents containing those exact words. Crucially, it weights the rarest ones more heavily: "botulism" and "canning" are distinctive across the articles, so a page that uses them repeatedly ranks near the top, while the millions of pages that merely contain a common word like "lid" barely register. The pages for Botulism, Home canning, and Clostridium botulinum would be returned, exactly what our survivor needed three weeks ago.

The precision of BM25 is both good and bad; when you already know the right word, such as a chemical name or some other piece of jargon, it's extremely accurate and can surface the one article a more "fuzzy" semantic search might blur into its neighbours. However, it does not understand meaning: it can only find what you already know the name of. Ask it for "white powder that makes water safe to drink" and it gets stuck on the common words, whereas the dense search maps that description straight to calcium hypochlorite (in theory). BM25 finds what you typed; semantic search finds what you meant. I combined these into a hybrid tool where the model could simply search and get a ranked list of answers based on both methods.

Illustration comparing two ways of searching an offline Wikipedia. Left, "BM25 – exact words": the query "botulism canning" on a laptop returns the articles Botulism, Home canning and C. botulinum. Right, "Embeddings – meaning": the query "make water safe" links to disinfection, water purification and calcium hypochlorite.

Despite many of these models being fine-tuned for multiple tool calls, there were still many issues with my first attempt. Models would frequently emit invalid syntax, not use tools at all, or would often give answers that were simply exposition as to how they would approach the problem: things like "I'll now use the wiki_read tool...", which were correctly marked as failures by the judge. Gradually these issues were patched deterministically, with tool call healing for the most common failure modes and better prompting.

Initial results were disappointing: the hybrid search method was no better than letting the models work off their internal knowledge, and for some of the smaller models it was even a regression. A small improvement was to drop semantic search. As it turns out, the LLM is the semantic search here: it is smart enough to handle multiple BM25 searches to get the answer it needed, and the small inaccurate embeddings model I used often returned irrelevant results. In test with a smaller question set, I found that Gemma 31B with semantic search performed only marginally better than direct answer, whereas letting the model use a multi-BM25 query tool with top K ranking brought up the score by 51.6%.

However, this trend didn't hold. For the full 9 model 500 question benchmark, the mean score still regressed from 4.35 to 4.05; 2121 answers got worse while 1655 got better. This doesn't tell the whole story:

Dumbbell chart titled "Wikipedia helped the strongest models — and broke the weakest", comparing each model's mean survival score with no tools and with the Wikipedia BM25 agent. Changes: Gemma-4 31B +0.40, Granite-4.0 Micro +0.14, Nemotron-3 Nano 30B +0.09, Gemma-4 26B-A4B +0.07, Granite-4.1 8B −0.13, Trinity Mini −0.42, LFM2.5 1.2B −0.46, GPT-OSS 20B −0.61, Phi-4 Mini −1.78.
Mean survival score with no tools vs. with an offline-Wikipedia search agent. Four models barely improved; five got worse, phi-4 worst of all (-1.78).

In general, the models that increased their score were the ones already at the top, particularly the Gemma and Nemotron models. Similarly, the models that performed more poorly were already the worst in the initial benchmark. Phi-4 Mini shed nearly two points, with GPT-OSS and LFM not much better. There's a large asymmetry: the gains are small while the losses are large. Thus, access to an encyclopaedia is of benefit to only the already-capable, amplifying what knowledge the model already has. Smarter models know which search terms to use, which returned snippets to read, and how to apply to results to answer the question. A model incapable of doing these things gets a wall of irrelevant text that confuses it further.

Grouping the nine models into tiers by their no-tool knowledge shows this effect more clearly:

Two-panel chart titled "Wikipedia rewards the strong and breaks the weak". With Wikipedia, the top tier's mean score rose by 0.19 and its auto-fail rate fell by 0.7 percentage points; the mid tier's score fell by 0.39 and its auto-fail rate rose by 4.8 points; the weak tier's score fell by 0.70 and its auto-fail rate rose by 8.5 points.
Models grouped into three tiers by their no-wiki skill. Bars show each tier's mean change when given an offline Wikipedia search tool, pooled over ~1,500 paired answers per tier; dots are the individual models.

The strongest tier scored better on average and reduced their catastrophic failure rate; the middle tier slipped a little; the weakest tier fell off a cliff, particularly when it came to auto-fails.

The models don't like to read

Another factor was the models' willingness to actually use the tools provided. One off the tools, the read tool, was designed to allow the model to "zoom in" to a particular snippet of an article, once the search tool had returned a few candidates for the search term. Most of the models barely used this, no matter how much I emphasised the usefulness of this tool in the system prompt. The exception was Granite-4.0 Micro, a tiny model that performed poorly in the direct answer benchmark but was the only model in its size class to improve when wiki search was added: this potentially lends itself to the idea that the search+read could have benefitted the models more, if they actually used it.

Grouped bar chart titled "They search. They rarely read." showing average search queries, tool calls and full article reads per question for each model. GPT-OSS 20B issued 5.9 searches but read 0.7 articles; Gemma-4 31B issued 5.1 searches and read 0.4; Granite-4.0 Micro read the most, 1.8 articles per question; Phi-4 Mini never searched.
Retrieval effort per question with an offline Wikipedia + BM25 search tool. Models fire off keyword queries quite happily, but opening the full article (green) is the exception: only Granite-4.0 Micro liked to read.

This is further reinforced if we look at questions where the models read only snippets, versus where they bothered to dive into the articles themselves. Just relying on search snippets hurt results massively, while a single article read was enough to erase this deficit, and two more more reads brought improved performance on the benchmark questions.

Bar chart titled "Reading the article helps — searching the snippets doesn't". Mean score change versus the no-Wikipedia baseline: −0.20 for questions where the agent read no full articles (2,380 questions, 60%), −0.04 with one read (1,143 questions, 29%), and +0.11 with two or more reads (472 questions, 12%).
BM25 questions grouped by how many full article chunks the agent read. Snippet-only answers are where Wikipedia hurt results the most; pulling even one chunk nearly erases the damage. (Phi excluded, because it is stupid).

Phi was an enigma, having apparently been RL trained to be bad at tool calls. It would often hallucinate having searched an imaginary Wikipedia, or would calmly explain that the tools I had told it about were not available in the environment, without ever trying to use them. This led to a lot of irrelevant answers, hence the disastrous results of the wiki search benchmark for this model.

The models don't know what they don't know

One reason why these small models may be bad at searching is that they are corrupted by the Dunning-Kruger effect, too confident in their knowledge. This phenomenon has been measured by Artificial Analysis in their omniscience benchmark.

AA-Omniscience Index measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer. Scores range from -100 to 100, where 0 means as many correct as incorrect answers, and negative scores mean more incorrect than correct.

Essentially, confident incorrect answers rather than "I don't know" contribute to negative scores in this benchmark. Looking at the models we tested, there is a pattern, although Granite-4.0 Micro is once again an outlier, perhaps brought down in AA-Omniscience by its poor base knowledge:

Bar chart from Artificial Analysis of the AA-Omniscience Index for the tested models, all negative: Gemma 4 31B −45, Gemma 4 26B A4B −48, NVIDIA Nemotron 3 Nano −52, Phi-4 Mini −61, gpt-oss-20B (high) −64, Granite 4.1 8B −65, Granite 4.0 Micro −78, LFM2.5-1.2B-Thinking −84.
Artificial Analysis's AA-Omniscience benchmark shows that our worse performing models are the ones that don't know when to say "I don't know".

Some excerpts from the wiki-search benchmark

Phi, ignoring its tools, lying about doing searches, and then answering without any results

I apologize for any confusion, but I must reiterate that I am unable to make live Wikipedia search queries or access Wiki content directly. My responses are generated using a blend of licensed data, data created by Microsoft, and publicly available data up to my last training cut-off in early 2023.

Apologies for the oversight. Let's try a search based on the scenario described.\n\nSearch Query: monoculture an agricultural practice\n\n[Awaiting search results...] Meanwhile, based on the information I have up to my last update, we can discuss the impact of monoculture

In the absence of an online Wikipedia search tool due to current constraints, I'll draw on my pre-existing knowledge base

GPT-OSS, telling us the searches it intended to do, and then not doing them

I need to call wiki_research. Must provide only the tool call JSON. Let's do that.

We may need to call wiki_research: 'sand filter', 'charcoal water filter', 'turbidity test.' Use topK 5.

Where the wiki-search may have helped

Thus tension grows as the angle opens because cos(theta/2) falls.

— Nemotron-3 Nano 30B, ENG-032

When answering from memory, it used the wrong rigging formula and concluded that a wider sling angle means less rope tension, which is exactly backwards and would lead to a snapped rope. A single BM25 search surfaced the correct relationship and it rewrote the answer. 0 → 10, on the search snippet alone, it never even opened the article.

Because your steel lacks carbon, you must 'force' carbon into the surface of the metal using heat and a carbon-rich environment.

— Gemma-4 31B, MAT-032

On its own it told the survivor to re-harden mild steel by quenching it, the very thing that had already failed in the question, because mild steel hasn't enough carbon to harden. The search turned up pack-carburising: bury the tool in bone, leather and charcoal to drive carbon into the surface first. 0 → 9.5, again from the snippet only.

The null can be 30 dB or more deeper than the maximum (the 'peak') side, giving a much higher signal-to-noise ratio when the loop is pointed directly away from the source.

— Granite-4.1 8B, COMMS-006

Direct answer told us to rotate a loop antenna for the loudest signal. Reading the "Loop antenna" article made it use the correct trick: use the sharp null, the angle where the signal almost vanishes, to fix a bearing far more precisely. 0 → 9.2.

The information below is based on standard clinical principles drawn from the Wikipedia entries 'Wound healing' (primary vs secondary intention) and 'Animal bite.'

— GPT-OSS 20B, MED-028

Direct answer advised stitching a contaminated bite wound shut, which seals the infection inside. After reading "Wound healing" and "Animal bite," it gave the right answer: clean it and leave it open to heal. 0 → 7.8.

treat any water that has been present >= 4 days

— Granite-4.0 Micro, PH-011

Direct said to pour oil, kerosene or insecticide straight into the drinking-water wells, poisoning the supply. After reading "Mosquito control," it switched to the correct target: the standing breeding water, timed to the mosquito's life cycle, leaving the drinking water alone. 0→ 8.75 (though it took 340 seconds to get there).

Conclusions and caveats

So, what did we learn, if anything? Tiny LLMs are getting better, but with limited parameters labs must choose carefully where to direct a model's knowledge. Most tend to focus on coding capabilities here, as this is what the most popular benchmarks measure, and is perhaps the easiest capability to train with post-training reinforcement learning. This leaves on the table a lot of raw textbook knowledge: models have read about chlorination and carburising somewhere in their trillions of tokens of training data, but there are simply not enough unique directions in their latent space to store much more than vague, correct-sounding representations.

The obvious fix of bolting the knowledge back on with an encyclopaedia made things worse, mostly because using a library turns out to be its own skill. You have to know what you don't know, ask the right question, and recognise the right answer when you see it. Those seem to be the capabilities that small models lack, which is why retrieval rewarded the strongest and broke the weakest.

That said, advancements to LLMs are being made at a frightening pace, and the maximum sized model tested (31 billion parameters), is certainly not the upper limit for open source models: just the upper limit of my hardware and budget. So, I wouldn't write-off using LLMs to rebuild civilisation just yet, but maybe keep a few textbooks around, just in case.

You can find apocalypse-bench at my GitHub, under an MIT license.

Grid of nine histograms titled "What Wikipedia did to each model's score distribution", overlaying each model's answer scores with no tools and with the Wikipedia agent. The Gemma-4 and Nemotron distributions shift slightly to the right; Phi-4 Mini and LFM2.5 pile up more heavily at zero.
Per-model answer-score distributions, Direct vs +Wikipedia (500 survival questions each). Solid line = mean with Wikipedia. The two strongest models edge right; the weakest pile up harder at zero.