How Dalalat’s Levels Get Made

Dalalat (دلالات) is an Arabic crossword game for phones. Each level is a small grid of about eight words. You tap an entry, read its clue, and build the answer from letter tiles: the missing letters plus a few decoys. Solve 75% of the grid and the next level opens.

The app is a fairly standard stack: Expo and React Native, NativeWind for styling, Reanimated for the grid and tiles, and Convex as the backend. I’ll come back to it at the end. This post is about the part that took the most work: the levels.

A crossword lives or dies on its content. A clue that fits three answers, a word nobody uses, a fact that’s only true in one country: each one makes the player feel cheated, and no animation makes up for it. I needed dozens of levels, then hundreds, and I’m one person. So I used AI. The interesting part turned out to be everything around the AI.

The first version: ask the model for everything

The first pipeline was the obvious one:

1. Ask Claude for 16 Arabic words on a theme.
2. Pick a clue type for each word at random (description, riddle, saying with a blank, trivia, emoji) and ask for a clue of that type.
3. Place the words in a grid with backtracking, try 10 orderings, keep the densest.
4. Validate and write the level.

It ran end to end and produced levels that looked right. They weren’t good.

The validator checked what code can check: no two letters fighting over a cell, nothing outside the grid, no duplicate words, a connected grid, a clue that doesn’t contain its answer. It had no idea whether a word was real Modern Standard Arabic, whether a clue pointed to one answer or five, or whether a level was easy or hard. A word could be forced into a riddle whether or not a good riddle existed. Even my prompts had problems I only saw later: the example trivia clue had English in it (رمزه الكيميائي Au, “its chemical symbol is Au”), and the prompt for sayings invited Quran verses. The pipeline now rejects both.

The model was fine at producing text. Nothing in the loop could tell a good level from a bad one.

So I rebuilt it around that.

The shape of the new pipeline

The rule I settled on: code does search, arithmetic, and rules. Models do judgment. Nothing ships on one score.

1. Build a word pool from a frequency list, vetted by a classifier model and a policy written in code.
2. Search many grids per level and score their structure.
3. Write several clues for every answer, at different difficulty levels.
4. Have a blind solver guess each clue, then a judge rate it, then apply hard gates.
5. Pick each answer’s clue in the context of its grid, and simulate players solving the level.
6. Have a reviewer play the finished level and approve it or not.
7. Validate and publish.

A command line drives it: design:plan, design:clues, design:eval, design:assemble, design:qa, design:write. Every step reads and writes JSON, so I can stop anywhere, edit something by hand, and pick up again. I built most of it with Claude Code, and Claude also does the writing and reviewing inside it.

Step 1: Don’t let the model invent words

The first change was to stop asking the model for words. They come from an Arabic frequency list instead: 8,833 rows, each with a part of speech, a learner level, and a vocalized spelling.

A deterministic filter runs first. It drops untagged rows, function words, words outside 3 to 8 letters, and words carrying the definite article ال or an attached preposition. That leaves 7,839 candidates.

Arabic adds details you only find by hitting them. The plain spelling column had lost its hamza seats on about 400 rows (ابداع instead of إبداع), so spellings come from the vocalized form with the vowel marks stripped. And a grid cell holds one letter, which the app compares exactly, so أ and ا, or ة and ه, are different letters. Two words only cross on identical letters.

Then every candidate gets a set of questions, put to a classification model that answers with a probability for each option instead of free text. Each word gets:

• form: dictionary headword, plural, conjugated verb, word with something attached, or other;
• register: standard Arabic, colloquial, or misspelled;
• clueability (0–3): can a setter write a short clue that has only this answer?
• familiarity (0–3): how widely do adult native speakers know it?
• category: animals, food, places, and so on;
• six safety questions: sexual, insult, vice, sectarian, political, violence;
• for proper nouns, what kind of name: a place, a famous person, a month, a common word, or just a first name.

A question looks like this, shortened:

const REGISTER = choice(
  {
    question: "Is the Arabic word `word` correct Modern Standard Arabic as spelled?",
    context:
      "Arabic is written without vowel marks, and a doubled letter (shadda) is written once, " +
      "so short words such as حدد، مد، صمم، قرر are complete and correctly spelled.",
  },
  {
    standard: {
      what: "Modern Standard Arabic, spelled correctly. Established loanwords count.",
      examples: ["نافذة", "تلفاز", "راديو", "بنك", "حدد", "صمم"],
    },
    colloquial: {
      what: "Used only in a spoken dialect, or a dialect word that has a different standard equivalent.",
      examples: ["دلوقتي", "هيك", "بدي", "كتير"],
    },
    misspelled: {
      what: "Not a correct word as spelled: a typo, a missing or extra letter, a wrong hamza seat...",
      examples: ["مدرصة", "مسؤل", "شاطيء", "مدرسه"],
    },
  },
);

The decision is code, not the model. The raw probabilities are stored, so I can change a threshold and rerun the policy without paying for another model run. A word passes a question when its acceptable answers hold enough probability together: headword and plural count together, at 0.75, so a word the model splits between the two still passes. At 0.25 or below it fails. In between, it comes to me for review. Familiarity plus half the clueability gives an “ease” score, which sets the word’s tier: easy, medium, or hard.

To tune all this I hand-labelled a gold set of 77 words and ran the questions against it after every change. A few things I learned:

• One “unsafe” question didn’t work. It gave Palestine, war, and fleas (فلسطين، حرب، برغوث) the same probability as real problems, so no threshold could split them. Six narrow topics, each told to judge the word’s common Arabic meaning, worked much better.
• Some fixes are context, not thresholds. Short verbs like برر and زود kept coming back “misspelled”. The fix was the note in the question above: a doubled letter is written once.
• Some errors don’t move with thresholds at all. Real words the model doubts (نحل bees, تنس tennis) and true misspellings land in the same 0.3–0.55 band. That band goes to review, and a hand-edited overrides file has the last word: 220 words forced in, 201 forced out.

Of the 7,839 candidates, 4,769 made it: 682 easy, 2,672 medium, 1,415 hard. The most common reasons for rejection were “hard to clue” (1,548), “weak” on both familiarity and clueability (1,100), and the wrong form (946). With that many words, I cared more about precision than recall. Losing a good word costs nothing. Shipping a bad one costs a player.

Step 2: Search grids, don’t build one

The old grid builder tried 10 orderings and kept the densest. The new one is a beam search.

For each level it draws 60 words from the pool, in the level’s mix of easy, medium, and hard, favoring words whose letters cross well. It starts from a few seed words and grows layouts one word at a time, trying every legal placement and keeping the best 160 at each step. It repeats this for 8 different draws, then keeps the best layout plus a few alternatives that share at most half their words.

To score a layout, it treats it as a graph: words are nodes and crossings are edges. That gives useful numbers:

• cycles (crossings − words + 1): loops give every region more than one way in;
• leaves: words crossed only once;
• bridge cut: the most words a single crossing cuts off;
• hub cut: the most words a single word cuts off;
• density: how much of the bounding box is filled.

The score rewards cycles, crossings, and density. It penalizes leaves, bridges, and hubs; grids that are too wide or too tall (at most 1.6 times taller than wide, so it fits a phone); hard answers without easy neighbours to give them letters; and two hard answers crossing each other.

Then there are rules the search can’t break:

• No shared roots. Arabic words grow from consonant roots, so كتاب (book), كاتب (writer), and مكتبة (library) are nearly the same word to a solver. The check compares consonant skeletons, with special handling for hollow roots like نوم and منام (sleep, a place of sleep).
• Roots rest. A root stays out for six levels on either side, so it doesn’t come back while players still remember it.
• No look-alikes in one level: same-length words one letter apart, or anagrams (إعانة and إعاقة; سماء and مساء). Reviewers read them as typos of each other.
• Caps on categories, verbs, adjectives, plurals, and three-letter words, and at most one number. Numbers also sit out the level after one that had a number. They’re easy and cross well, so without that rule the search put one in most levels.

Step 3: Define difficulty before writing clues

I wanted levels to get harder at a steady pace, so “harder” needed a number.

A clue gets a level from 1 to 5, based on how much work it leaves the solver, not how rare the knowledge is. For قلم (pen):

1. Direct: «نكتب به», “we write with it”.
2. Clear: «يحتاج إلى مبراة», “it needs a sharpener”.
3. Indirect: «رصاص أو حبر», “lead or ink”.
4. Tricky: «لسان الكاتب», “the writer’s tongue”.
5. Expert: «أحد اللسانين», “one of the two tongues”, from a classical saying.

An entry’s difficulty is its clue level plus an adjustment for the word. Less familiar words add up to about a point. Three-letter words lose half a point, because the letter tiles nearly give them away. A level’s difficulty is the average of its entries.

Each level has a target from a curve that climbs quickly at first and levels off below 4.5. Every fifth level is a spike, 0.4 harder, and the one after it is a breather. A level more than 0.4 off its target can’t be written. When I played through and found the start too easy, I dropped the first 22 levels, most of them from an earlier version of the pipeline, and let the curve start further along.

The same number drives the tiles. Harder levels get more decoys, from 4 up to 8, and more of them are dot-lookalikes of the missing letters, like ج for ح or ف for ق. So the stored difficulty has to be right. It isn’t just a label.

Step 4: Write several clues per answer

The clue writer is Claude, running in Claude Code next to the pipeline, writing against a clue guide I can edit.

For every planned answer it writes 3 to 5 candidates at different levels and with different strategies: a definition, a trait, a function, a context, an association, a contrast, wordplay, a fact, a saying, or emoji. A worksheet tells it what each answer’s place in the grid calls for, so it can aim. So far that’s 1,975 candidates for 443 answers.

Cheap checks run before any model sees a clue:

• at most 48 characters, about three lines on the clue screen;
• no word from the answer’s root, using the same skeleton check as the grid;
• a saying has a ... blank and doesn’t start with أكمل (“complete:”), because the app adds that;
• emoji clues contain no text;
• no < or >, because right-to-left text mirrors them, so < can show up as >.

A note on how the model steps run: there’s no API call in the pipeline. design:eval writes batches of JSON into a work folder, each naming its instructions file and its output file. An agent reads a batch and writes the output, and the next run of design:eval takes the results in. In practice the agents are Claude Code subagents, but anything that can read and write a file could fill those chairs. A clue’s id is a hash of the answer, type, and text, so editing a clue gives it a new id and a fresh evaluation.

Step 5: Check correctness with a blind solver and a judge

If I had to throw away everything else, this is the part I’d keep.

Once you know the answer, almost every clue looks fair. That’s hindsight bias, and a model rating clues with the answer in front of it has it too. So the first reviewer doesn’t get the answer.

The blind solver sees the clue, its type, and the answer’s length, nothing else. It plays a quick, honest player and gives up to three guesses with a confidence each. Batches carry only hashed ids, and the instructions tell it not to open other files in the repo, because those hold the answers. An item looks like this:

{ "id": "k3f9a1b2", "length": 5, "type": "descriptive", "clue": "يبنيها الرياضيون في الصالة" }

The judge sees everything: the answer, the clue, and the blind guesses. It doesn’t see the level the writer aimed for. It rates correctness, natural Arabic, grammar match, difficulty, ambiguity, fairness, interest, specialist knowledge, and leakage, then gives a verdict. The blind guesses are its evidence. A confident right guess means the clue is direct. Confident wrong guesses that fit the length mean it’s ambiguous.

Then come hard gates, in code. A clue that fails any of them is out, however good its other scores are:

const GATES = [
  { test: (j) => j.verdict !== "reject", problem: "judge rejected it" },
  { test: (j) => j.correctness >= 4, problem: "not accurate" },
  { test: (j) => j.natural >= 3.5, problem: "unnatural Arabic" },
  { test: (j) => j.grammar, problem: "doesn't match the answer's form" },
  { test: (j) => j.fairness >= 3.5, problem: "unfair once revealed" },
  { test: (j) => j.ambiguity <= 3.5, problem: "too ambiguous" },
  { test: (j) => j.specialist <= 3.5, problem: "needs specialist knowledge" },
  { test: (j) => j.leakage <= 3, problem: "gives the answer away" },
];

Of 2,014 judged clues, 240 (12%) failed at least one gate. The judge rejected 155 outright, 146 were too ambiguous, 107 unfair, and 90 not accurate enough; a clue can fail several. Some examples:

• أجساد (bodies) ← «يبنيها الرياضيون في الصالة», “athletes build them at the gym”. That’s the item above. The blind solver answered عضلات (muscles), also five letters, at 0.9 confidence.
• خريف (autumn) ← «… العمر», “the ___ of life”. Without letters, ربيع (spring), شريك (partner), and فرصة (chance) all fit four boxes.
• رياح (winds) ← «هواء يهب بقوة», “air blowing hard”. A singular clue for a plural answer.
• مدارس (schools) ← «تفتح أبوابها في سبتمبر», “they open their doors in September”. Nothing marks it plural, so مدرسة and جامعة fit too. The judge added that schools in the Gulf open in August.

That last note turned into a pattern. Correctness in Arabic is regional, and the Arab world isn’t one audience.

The judge also rates each clue’s difficulty, and it runs easier than the writer by a steady margin: about 0.71 + 0.56 × the writer’s level (r = 0.75 over the first 204 clues). That makes sense, since it sees the answer. So its rating is mapped back onto the writer’s scale, and the two are averaged. Two independent estimates beat one confident one. The pipeline prints the current fit on every run, so I can see when it drifts.

Step 6: Choose clues in context, and simulate the players

A clue isn’t good or bad on its own. It depends on where it sits. A hard clue is fine on a familiar answer with three crossings. The same clue on a rare word that hangs by one crossing is a wall.

So clues are chosen per level, not per word. The assembler hill-climbs from 40 random starts, changing one answer’s clue at a time, and scores whole assignments:

• target: the level’s average close to its target;
• texture: the sorted entry difficulties spread around the target like a bell curve, so there are easy ways in, a solid middle, and a challenge or two;
• starters: at least two entries most players get with no letters;
• fairness: tricky clues only on familiar answers;
• support: hard entries need crossings;
• quality: the judge’s interest, fairness, naturalness, and ambiguity scores;
• variety: at least three clues that aren’t plain descriptions, at most one emoji clue, two riddles, or two sayings, and no two clues starting with the same word.

The best 40 assignments then go through a simulation that plays the level 300 times. Each run draws a player with some ability. An entry’s chance of being solved falls with its difficulty and rises with every letter already in place from crossings:

const gain =
  recall * PLAYER.letterGain * Math.pow(share, 0.9) +
  (firstKnown ? PLAYER.firstLetterGain : 0) +
  (length - known === 1 ? PLAYER.lastLetterGain : 0);
return 1 / (1 + Math.exp((entry - gain - ability) / PLAYER.slope));

recall is lower for obscure words, so letters help less there: a word you don’t know stays hard with half its letters in. The simulated player solves everything they can in one wave, which fills in crossings, and repeats. When nothing gives, they take a hint on the entry they were closest to.

That produces the numbers that matter: how often a player finds no way in at all, how many entries fall cold, how many hints a run needs, and which entry blocks people. A level is penalized when more than 45% of players get stuck on one entry.

Here’s level 8, a spike level with a target of 3.5. Eight words on a 5×8 grid, 10 crossings, 3 loops, and no word hanging by a single crossing. Four of its clues:

• نظافة (cleanliness) ← «غاية الماء والصابون», “what soap and water are for”. A starter: 81% of simulated players solve it cold.
• مدينة (city) ← «قرية كبرت», “a village that grew up”. A riddle.
• إفطار (iftar) ← «يُعلن موعده مدفع رمضان», “the Ramadan cannon announces its time”.
• شمس (sun) ← «لها قرص ولا تُبلع», “it has a disc but you can’t swallow it”. قرص means both a disc and a pill. The clue is rated 4.5 and only 19% get it cold, which is fine: everyone knows the word, and two crossings feed it.

For إفطار, the writer had also offered two direct definitions: “the fasting person’s meal at sunset” and “the first meal of the day”. The level was running under its target, so the assembler took the harder cannon clue. The judge had noted that the cannon also marks إمساك, the start of the fast, which has the same length. The crossings settle it.

The simulation found no run without a way in, about four entries falling cold, and 83% of runs finishing without a hint. The level landed at 3.14, inside the tolerance.

One more thing the assembler does: it also builds each level with ideal, imaginary clues at every half level. If the real clues score clearly worse on difficulty and flow, it lists the answers whose clue is far from the ideal. That’s my list of clues to write next.

Step 7: A reviewer plays the level

Everything so far looks at clues and numbers. The last check looks at the level the way a player meets it.

design:qa renders each level as text: the grid with empty boxes, each entry with its direction, length, clue type, and clue, which letters it shares with which entry, and the answers, so the reviewer can check them. A separate reviewer solves it in their head, then looks for wrong clues, obscure answers, deadlocks, missing ways in, repetition, giveaways, sensitive content, and flow. Each issue is a blocker, a major, or a minor. Only a level with no blocker or major issue is approved.

The review carries a signature: a hash of every word’s position and every clue’s id. Change one clue and the review goes stale. design:write refuses the level until it’s reviewed again.

All 49 shipped levels were approved, with 137 minor issues logged. The minor ones are where the reviewers earned their place:

• خميس (Thursday) clued as the last workday. In Morocco, Tunisia, Lebanon, and the UAE it’s Friday, and جمعة also has four letters.
• قهوة (coffee) clued as what’s served with dates to a guest. In the Maghreb that’s often milk, also four letters.
• أستاذة, the title for a lawyer, is mostly Egyptian usage.
• دماء (blood) next to a clue about the butcher sharpening his knives. Fine alone, grim together.
• Three clues in one level used the same bare “X and Y” formula, and they were also its three driest.

Each review also writes one sentence about the flow. For level 3: “Easy starters (بلد, جدار, خمسون, نبات) fill the spine quickly, and «قطن السماء» and the Olympic record then land as small aha moments.” قطن السماء is “the sky’s cotton”. That sentence is what I read first.

Reviews become rules

Model reviewers are inconsistent. One catches something, and the next one misses it. So when a complaint comes back more than once, it stops being a note and becomes code:

• A level needs at least three clues that aren’t plain descriptions. Reviewers kept calling all-description levels flat.
• No clue may carry another answer’s root. QA caught الطفلة (the girl) in a clue next to the answer أطفال (children), and مالي (my money) next to أموال (money).
• No look-alike answers in one level, after إعانة and إعاقة (aid and disability) ended up together.
• No < or > in clues.
• The root matcher learned the ست of words like استجابة (response), after QA found it in the level next to أجاب (he answered).
• No number answers in back-to-back levels.

That loop is what makes the pipeline better over time. A reviewer’s note fixes one level. A rule fixes every level after it.

Validation and shipping

design:write is the last gate. A level is written only if it follows the last written level, has a draft, has a current approved review with no blocker, and sits within 0.4 of its target. The structural checks from the very first version still run too: no cell conflicts, nothing out of bounds, no duplicate words, a connected grid.

Every written level also keeps its full design record: the layout and its structure, the simulation, every clue candidate with its blind guesses and judge scores, and the review.

Levels don’t need an app update. bun levels:publish sends new or changed levels to Convex, each with a revision number. The app downloads them on launch and whenever it comes back to the foreground, and keeps whichever copy, bundled or downloaded, has the higher revision. Saved progress resets only if a level’s answers or grid changed. A level that needs features older builds lack gets a higher format number, and those builds skip it. Removed levels are marked as removed, and apps drop them.

What I don’t trust yet

The reviewers rated almost every level “easy”: 44 of 49 got a 2 out of 5, whatever their target. When I played them, the curve felt about right. The reviewers see the answers, so I read that as hindsight bias rather than a reason to make things harder. It does mean the reviewer’s difficulty score is close to useless.

The difficulty model is also a set of guesses. The simulated player’s ability, how much a letter helps, the judge’s rescaling, the width of the bell curve: all hand-set. That’s why every level keeps its design record. Once there’s real data on solve times, hints, and where people give up, those constants can be fitted instead of guessed.

The rest of the app

The game around the levels has its own interesting parts:

• A right-to-left grid. Touch handlers flip the columns, so a finger on the left edge maps to the last column. You can slide across cells to pick a word.
• Credits and ads. Hints cost credits, and a rewarded ad earns some. Ads pay only through AdMob’s server-side verification: Google calls a Convex HTTP endpoint, a Node action checks Google’s signature, and each transaction pays once. No public endpoint hands out credits.
• Identity without accounts. An anonymous device ID, also kept in the iOS Keychain so it survives a reinstall.
• Over-the-air updates for JavaScript fixes, separate from level publishing.

What I took from it

Generating the content was never the hard part. Any model will write you a hundred clues in a minute. The work was in deciding which ones are good, and that took:

• Separate roles. Writer, blind solver, judge, reviewer: the same kind of model in different chairs, each with different information.
• Hiding information on purpose. The blind solver is the most useful check in the pipeline because it can’t see the answer.
• Hard gates over weighted scores. A great clue that’s wrong is still wrong.
• Stored raw scores. Thresholds change, and model runs cost money.
• Rules from reviews. Every repeated complaint becomes code.

The model writes the clues. Everything else is checking.