In the upcoming apocalypse, which small LLM will help us rebuild civilisation?

The dust has settled from whatever calamity ended civilisation as you knew it. Survivors clamber from the rubble clutching their loved ones and their MacBook Pros. The grid is dead, the internet is a memory. But you have a solar panel, and you downloaded a few open-weight large language models before everything went dark.
Three weeks in, you've scavenged some vegetables from an abandoned garden and found a pressure cooker in a kitchen. Time to preserve some food for winter.
You fire up ollama and ask, "How do I know if my canned food is safe to eat?"
Unfortunately, your large language model of choice was IBM's Granite-4.1 8B, and you may have just made a fatal mistake. The model advises to test for botulinum toxin, one of the most lethal neurotoxins known, by "tasting a tiny amount (no more than a bite)". The toxin is tasteless, and ingesting a tiny amount is fatal. Just one of the many ways that small LLMs will try to kill you.
Apocalypse Bench
Most LLM benchmarks focus on software engineering, or raw scientific knowledge, or tough maths questions. I decided to create a benchmark to answer a question nobody was asking: how useful are small, local LLMs when you need to not just survive, but rebuild civilisation from the ground up?
The test bank covers 500 questions across a range of domains: agriculture, medicine, chemistry, engineering, sanitation, governance, communications, and others. Each question includes a rubric with 10 criteria and a set of "auto-fail" conditions (answers so dangerous or useless they score zero regardless of how many other things the model got right).
Using LLMs to judge LLMs
You can't grade 4500 survival answers by hand (9 models × 500 questions). So, I used an approach known as "LLM-as-judge": each candidate answer gets sent to a separate judge model along with the original question and an answer rubric. The judge of choice was GPT-5.5, a frontier model by OpenAI. Importantly, the model doesn't need to have all the knowledge of the correct answers itself: just smart enough to follow the rubric.
The judge returns a JSON object with scores for each of the ten question-specific criteria: things like using only stated resources, flagging relevant hazards, giving verification or stop-work checks, and making the answer actionable rather than vague. It also flags whether any auto-fail conditions were triggered: dangerously unsafe guidance, critical omissions that could cause harm, or outright refusal to answer (model safety guardrails can become misalignment in a survival situation). If an answer trips an auto-fail, it scores zero. No partial credit for confidently advising someone to eat poison.
I ran nine open-weight models through all 500 questions, and these are the results.
A quick overview
As expected, the larger models generally did better. Due to the nature of the high dimensional latent space these models operate in and how relationships between concepts are stored, the amount of raw knowledge they can learn is correlated with the number of parameters: bigger models know more facts. However, some models broke this trend: the aforementioned 8 billion parameter Granite-4.1 punched above its weight, while the 26 billion parameter Trinity Mini performed relatively poorly.

The mean score doesn't tell the whole story, however. The scores are bimodal: two peaks with a valley between them. Across all the answers, one in five scored zero, hitting one of the auto-fail conditions. The rest pile up nearer the middle, with 41% scored in the five-to-seven range. Genuinely export answers were rare: just 4% scoring 9 or 10.
This means these models rarely give you all the info you need: they either fumble around and give you something vaguely useful, or they fail outright. An average of 4.8 out of 10 doesn't describe an answer you'd actually receive, it describes a coin flip between an answer that might point you in the right direction, and one that might kill you.

Those zeros would be the main issue with using these models in a pressing situation. Sometimes they're refusals or non-answers, not necessarily a problem when it comes to ethics questions but deadly when you need to know how to correctly use a tourniquet. Other times they're something worse: confident and authoritative-sounding instructions that happen to be a death sentence.
Which models would kill you the fastest, and why?
There were no stupid-yet-safe models. The models that performed the worst were also the ones that gave answers that would kill you. In the most lethal domains, where bad advice kills in hours not years, the difference between auto-fail rates was striking:

The two Gemma models stand apart here: both auto-fail just 6% of medicine and safety questions, against 17–31% for everyone else.
The failures are not subtle.
LFM-2.5 decided to channel its inner Jim Jones and suggested sanitising 2000 litres of drinking water with 80 litres of bleach, three orders of magnitude greater than the correct safe dose. The camp of 40 people receive chemical burns in their GI tract.
Granite-4.1 proposed infusing a cholera patient with 40 litres of fluids per day, roughly 10x maintenance. The patient would die of pulmonary oedema or heart failure.
GPT-OSS taught us that a tourniquet controlling arterial bleeding should be loosened every 10 minutes, and removed when a distal pulse can be felt, leading to certain death from blood loss for the unlucky patient.
One task that all the models failed was to give a method of creating an anaesthetic using battery acid and alcohol. The correct method, synthesising diethyl-ether, was either refused or mislabelled; although Granite-4.0 Micro suggested pouring 3 litres of sulfuric acid into 1 litre of ethanol and then injecting it, which was a particularly poor answer. This would boil and potentially ignite, and the pH 2-3 solution would immediately cause chemical burns once injected.
Across the less-lethal domains the results were also underwhelming.

Even the smartest models performed poorly in the hands-on domains that a budding civilisation would need: engineering, materials, and energy questions failed roughly a quarter of the time across the question set. The models can reliably lecture you about ethics, but then can't help you build a kiln.
There were also some absurd answers. Granite-4.0 Micro confidently spelled "HELP" in Morse code, getting every letter wrong; Phi-4 Mini warned to "never go into human contact in Earthly situations alone", presumably talking to its robot counterparts; the same model suggested adding 1 tablespoon of pearls for every 76 millilitres of water to treat cholera victims; and LFM2.5 mentioned feeding chickens "kitchen scraps, gazelles, and wild greens".

Needless to say, it's not looking good for our survivors. Maybe we can do better. What if the models didn't have to memorise how to chlorinate a water tank? What if they could look it up?
What if the models had an encyclopaedia?
Wikipedia, to be precise. Wikipedia's entire corpus contains around 6 million articles, which is available as a 61 GB Hugging Face repository in markdown/JSONL format. This contains a significant portion of humanity's knowledge, handily distilled for us into an easy-to-read, well-categorised format. But we can't paste all 8 billion tokens into the model. Small models like these tend to have a context window ranging from a few thousand to a few tens of thousands of tokens, and even Goliaths like Google's Gemini 3.1 Pro top out at 1 million tokens. The models need a way to search the encyclopaedia and quickly retrieve information from it.
Agent harnesses are environments that allow LLMs to perform certain actions through a set of "tools": particular structured strings that, when emitted by the model, are interpreted by the harness and turned into real operation. This could be as simple as a bash tool that executes a command in bash, a ripgrep tool for exact text search, or a web_search tool to send a query to a search engine service.
Pi is one such agent harness: an open-source typescript project that is known for its extreme extensibility. Need a feature that is missing? Simply ask the agent to modify its own harness.
There were a few issues to solve: firstly, allowing an agent to grep its way around a JSONL would be a surefire way to churn through tokens and get no results. A literal search for "water" would give millions of unhelpful hits. So, I decided on a multi-layered strategy that would in theory allow the agent to find exactly the info it needed much faster. The first part was dense embeddings.
When text is embedded, it is tokenised and passed through a model that has been trained to represent concepts as vectors in a high-dimensional latent space. Similar concepts point in similar directions, giving them high cosine similarity, and the opposite is true for dissimilar concepts. "Cat" and "kitten" would point roughly the same way, "dog" would be at a slightly different angle, and "silicon" would be in a very different direction. I embedded article signposts with Snowflake's snowflake-arctic-embed-s, a small retrieval model that maps text into 384-dimensional vectors. Dense embeddings give a "question and answer" capability to the agent: it asks "what temperature does copper melt at?", the search returns relevant snippets related to these concepts.
For a slightly more targeted approach, the next tool was a custom BM25 search tool written in Rust. BM25, or Best Matching 25, is a bag-of-words search method that ranks our articles based on the terms in the agent's query. If the agent searched "botulism home canning bulging lid", BM25 wouldn't try to understand the question; it would simply hunt for documents containing those exact words. Crucially, it weights the rarest ones more heavily: "botulism" and "canning" are distinctive across the articles, so a page that uses them repeatedly ranks near the top, while the millions of pages that merely contain a common word like "lid" barely register. The pages for Botulism, Home canning, and Clostridium botulinum would be returned, exactly what our survivor needed three weeks ago.
The precision of BM25 is both good and bad; when you already know the right word, such as a chemical name or some other piece of jargon, it's extremely accurate and can surface the one article a more "fuzzy" semantic search might blur into its neighbours. However, it does not understand meaning: it can only find what you already know the name of. Ask it for "white powder that makes water safe to drink" and it gets stuck on the common words, whereas the dense search maps that description straight to calcium hypochlorite (in theory). BM25 finds what you typed; semantic search finds what you meant. I combined these into a hybrid tool where the model could simply search and get a ranked list of answers based on both methods.

Despite many of these models being fine-tuned for multiple tool calls, there were still many issues with my first attempt. Models would frequently emit invalid syntax, not use tools at all, or would often give answers that were simply exposition as to how they would approach the problem: things like "I'll now use the wiki_read tool...", which were correctly marked as failures by the judge. Gradually these issues were patched deterministically, with tool call healing for the most common failure modes and better prompting.
Initial results were disappointing: the hybrid search method was no better than letting the models work off their internal knowledge, and for some of the smaller models it was even a regression. A small improvement was to drop semantic search. As it turns out, the LLM is the semantic search here: it is smart enough to handle multiple BM25 searches to get the answer it needed, and the small inaccurate embeddings model I used often returned irrelevant results. In test with a smaller question set, I found that Gemma 31B with semantic search performed only marginally better than direct answer, whereas letting the model use a multi-BM25 query tool with top K ranking brought up the score by 51.6%.
However, this trend didn't hold. For the full 9 model 500 question benchmark, the mean score still regressed from 4.35 to 4.05; 2121 answers got worse while 1655 got better. This doesn't tell the whole story:

In general, the models that increased their score were the ones already at the top, particularly the Gemma and Nemotron models. Similarly, the models that performed more poorly were already the worst in the initial benchmark. Phi-4 Mini shed nearly two points, with GPT-OSS and LFM not much better. There's a large asymmetry: the gains are small while the losses are large. Thus, access to an encyclopaedia is of benefit to only the already-capable, amplifying what knowledge the model already has. Smarter models know which search terms to use, which returned snippets to read, and how to apply to results to answer the question. A model incapable of doing these things gets a wall of irrelevant text that confuses it further.
Grouping the nine models into tiers by their no-tool knowledge shows this effect more clearly:

The strongest tier scored better on average and reduced their catastrophic failure rate; the middle tier slipped a little; the weakest tier fell off a cliff, particularly when it came to auto-fails.
The models don't like to read
Another factor was the models' willingness to actually use the tools provided. One off the tools, the read tool, was designed to allow the model to "zoom in" to a particular snippet of an article, once the search tool had returned a few candidates for the search term. Most of the models barely used this, no matter how much I emphasised the usefulness of this tool in the system prompt. The exception was Granite-4.0 Micro, a tiny model that performed poorly in the direct answer benchmark but was the only model in its size class to improve when wiki search was added: this potentially lends itself to the idea that the search+read could have benefitted the models more, if they actually used it.

This is further reinforced if we look at questions where the models read only snippets, versus where they bothered to dive into the articles themselves. Just relying on search snippets hurt results massively, while a single article read was enough to erase this deficit, and two more more reads brought improved performance on the benchmark questions.

Phi was an enigma, having apparently been RL trained to be bad at tool calls. It would often hallucinate having searched an imaginary Wikipedia, or would calmly explain that the tools I had told it about were not available in the environment, without ever trying to use them. This led to a lot of irrelevant answers, hence the disastrous results of the wiki search benchmark for this model.
The models don't know what they don't know
One reason why these small models may be bad at searching is that they are corrupted by the Dunning-Kruger effect, too confident in their knowledge. This phenomenon has been measured by Artificial Analysis in their omniscience benchmark.
AA-Omniscience Index measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer. Scores range from -100 to 100, where 0 means as many correct as incorrect answers, and negative scores mean more incorrect than correct.
Essentially, confident incorrect answers rather than "I don't know" contribute to negative scores in this benchmark. Looking at the models we tested, there is a pattern, although Granite-4.0 Micro is once again an outlier, perhaps brought down in AA-Omniscience by its poor base knowledge:

Some excerpts from the wiki-search benchmark
Phi, ignoring its tools, lying about doing searches, and then answering without any results
I apologize for any confusion, but I must reiterate that I am unable to make live Wikipedia search queries or access Wiki content directly. My responses are generated using a blend of licensed data, data created by Microsoft, and publicly available data up to my last training cut-off in early 2023.
Apologies for the oversight. Let's try a search based on the scenario described.\n\nSearch Query: monoculture an agricultural practice\n\n[Awaiting search results...] Meanwhile, based on the information I have up to my last update, we can discuss the impact of monoculture
In the absence of an online Wikipedia search tool due to current constraints, I'll draw on my pre-existing knowledge base
GPT-OSS, telling us the searches it intended to do, and then not doing them
I need to call wiki_research. Must provide only the tool call JSON. Let's do that.
We may need to call wiki_research: 'sand filter', 'charcoal water filter', 'turbidity test.' Use topK 5.
Where the wiki-search may have helped
Thus tension grows as the angle opens because cos(theta/2) falls.
When answering from memory, it used the wrong rigging formula and concluded that a wider sling angle means less rope tension, which is exactly backwards and would lead to a snapped rope. A single BM25 search surfaced the correct relationship and it rewrote the answer. 0 → 10, on the search snippet alone, it never even opened the article.
Because your steel lacks carbon, you must 'force' carbon into the surface of the metal using heat and a carbon-rich environment.
On its own it told the survivor to re-harden mild steel by quenching it, the very thing that had already failed in the question, because mild steel hasn't enough carbon to harden. The search turned up pack-carburising: bury the tool in bone, leather and charcoal to drive carbon into the surface first. 0 → 9.5, again from the snippet only.
The null can be 30 dB or more deeper than the maximum (the 'peak') side, giving a much higher signal-to-noise ratio when the loop is pointed directly away from the source.
Direct answer told us to rotate a loop antenna for the loudest signal. Reading the "Loop antenna" article made it use the correct trick: use the sharp null, the angle where the signal almost vanishes, to fix a bearing far more precisely. 0 → 9.2.
The information below is based on standard clinical principles drawn from the Wikipedia entries 'Wound healing' (primary vs secondary intention) and 'Animal bite.'
Direct answer advised stitching a contaminated bite wound shut, which seals the infection inside. After reading "Wound healing" and "Animal bite," it gave the right answer: clean it and leave it open to heal. 0 → 7.8.
treat any water that has been present >= 4 days
Direct said to pour oil, kerosene or insecticide straight into the drinking-water wells, poisoning the supply. After reading "Mosquito control," it switched to the correct target: the standing breeding water, timed to the mosquito's life cycle, leaving the drinking water alone. 0→ 8.75 (though it took 340 seconds to get there).
Conclusions and caveats
So, what did we learn, if anything? Tiny LLMs are getting better, but with limited parameters labs must choose carefully where to direct a model's knowledge. Most tend to focus on coding capabilities here, as this is what the most popular benchmarks measure, and is perhaps the easiest capability to train with post-training reinforcement learning. This leaves on the table a lot of raw textbook knowledge: models have read about chlorination and carburising somewhere in their trillions of tokens of training data, but there are simply not enough unique directions in their latent space to store much more than vague, correct-sounding representations.
The obvious fix of bolting the knowledge back on with an encyclopaedia made things worse, mostly because using a library turns out to be its own skill. You have to know what you don't know, ask the right question, and recognise the right answer when you see it. Those seem to be the capabilities that small models lack, which is why retrieval rewarded the strongest and broke the weakest.
That said, advancements to LLMs are being made at a frightening pace, and the maximum sized model tested (31 billion parameters), is certainly not the upper limit for open source models: just the upper limit of my hardware and budget. So, I wouldn't write-off using LLMs to rebuild civilisation just yet, but maybe keep a few textbooks around, just in case.
You can find apocalypse-bench at my GitHub, under an MIT license.
