Every Sound, Labelled: Inside Our Phoneme Annotation Journey

20 October 2026 · Taimoor Haider · 6 min read

Recording children’s voices was hard. Labelling them, sound by sound, was harder. Of everything we have done to build Makharij, phoneme-level annotation was the most demanding task we took on, and it took us many months to get right.

In our last post, we shared how we collected our children’s voice dataset. But raw audio alone teaches a model very little. To give a child useful feedback, our AI must know not only what word was said, but which individual sounds were pronounced correctly and which were not. That means every recording needs to be marked at the level of its smallest units of sound: phonemes.

This is the story of how we did it, and what we learned along the way.

First, decide what a “sound” is

Before anyone could label anything, we had to answer a deceptively simple question: which sounds are we labelling?

The standard sound systems used by language experts were not made with Quran recitation in mind. Some of them miss differences that really matter in tajweed, such as heavy versus light letters, or two letters that both come from the throat but sound slightly different. At the same time, they include tiny details, like small changes in accent, that make no difference to whether a child recited correctly.

So we designed our own phoneme set. It had to be precise enough to capture the mistakes children actually make, yet compact enough to label reliably at scale. Every addition or removal was debated with people who understand both tajweed and speech technology, then tested against real recordings. The set went through several revisions before we froze it, and we treat its final form as part of our core work.

A language our labellers could read

Then we hit a problem we hadn’t fully anticipated. A phoneme set that makes perfect sense to a speech scientist means very little to the people who would actually be doing the labelling. Technical phonetic symbols are unfamiliar, easy to confuse, and slow to type.

Our answer was to build a bridge: a Roman-letter mapping in which every phoneme has a simple, readable equivalent that a labeller can recognise at a glance. Designing it was a balancing act. Each symbol had to be:

Intuitive, so labellers could guess a sound from how it is written.
Unambiguous, so no two phonemes could ever be mistaken for each other.
Fast to work with, so labelling stayed accurate even after hours of focus.

Behind the scenes, the Roman labels map back cleanly to the underlying phoneme set. Labellers work in a form they understand, and our models receive the precise representation they need.

Two people, one recording, two answers

Give the same recording to two careful people, and they may still label it differently. One hears a sound as slightly off; the other hears it as acceptable. One marks a hesitation; the other ignores it. With children’s speech, where pronunciation often falls somewhere between clearly right and clearly wrong, these disagreements multiply.

For a model, inconsistent labels are worse than noise. They teach it contradictory lessons. So consistency became a central goal, not an afterthought. We wrote clear guidelines for the difficult cases, built shared reference examples for the sounds people most often disagreed on, and kept refining both as new edge cases appeared. But we quickly realised that guidelines alone would not be enough. The tools themselves had to make consistency the natural outcome.

We built the tool before we labelled a single file

It would have been faster to start labelling straight away with generic annotation software. We chose not to. Instead, before the first real label was made, we invested time in building our own web application designed specifically for this job.

The idea was simple: the labeller should never start from a blank page. Every recording opens with its canonical labels already in place, showing how the word should be pronounced if it were said perfectly. The labeller’s task becomes listening carefully and marking where the child’s speech differs from that reference.

When something does differ, labellers don’t type free text. They pick from a set of pre-defined choices that cover the variations we see most often in children’s speech. This one design decision changed everything:

Labelling became faster, because most decisions are a single selection.
Typing errors and invented labels largely disappeared.
Different labellers naturally converged on the same answers for the same sounds.
New labellers became productive far sooner.

For the rare cases the choices don’t cover, there is a path to flag the recording for expert review rather than forcing a guess. Building the tool delayed our start, but it paid for itself many times over in speed and quality.

Finding the right ears

Even the best tool is only as good as the people using it. And the people we needed were rare: those with a trained ear for Arabic pronunciation, the patience to listen to the same short clip again and again, and the discipline to apply the same rules on their hundredth recording as on their first.

Recruitment took longer than we expected. Many candidates had strong knowledge of tajweed but found detailed, repetitive listening exhausting. Others were careful workers but struggled to hear the finer distinctions between sounds. We screened carefully and looked for the combination of both.

Everyone who joined then went through a structured training process. They learned the phoneme set through its Roman mapping, practised on recordings we had already reviewed, compared their answers against reference labels, and discussed every disagreement until the reasoning was clear. Only once a labeller was consistently accurate did they move on to live work, and training continued long after that.

Quality is checked, then checked again

We never trusted a single pair of ears. Every label passes through multiple layers of quality assurance before it reaches our models:

Labelling by a trained labeller, guided by canonical labels and pre-defined choices.
Review of completed work by more experienced team members, who check samples and catch errors early.
Expert supervision for difficult and flagged cases, where pronunciation specialists make the final call.
Ongoing monitoring of agreement between labellers, so we can spot drift, retrain where needed and update the guidelines.

These protocols added time and cost. They also gave us something we consider non-negotiable: confidence that when our model learns a sound is right or wrong, it is learning from a label we trust.

Many months later

From defining the phoneme set to the last reviewed label, this work took us many months. There were weeks where progress felt painfully slow, guidelines that had to be rewritten, and recordings that sparked long debates over a single sound.

But the result is something we are proud of: a children’s speech dataset labelled at the phoneme level, consistently and carefully, by people who understand what they are listening for. It is what allows Makharij to point a child to the exact sound that needs work, instead of simply saying “try again.”

We owe a huge thank you to our labellers, reviewers and supervisors. Their patience and precision are built into every piece of feedback a child receives.