One NER task, three generations of taggers: HMM → LSTM → BERT
The companion page builds a Hidden Markov Model from scratch to tag named entities. This page picks up where that one ends and walks up two more rungs of the modelling ladder - a recurrent LSTM tagger, and a fine-tuned BERT/DistilBERT tagger - all three solving the same problem so they can be compared apples-to-apples.
The task is fixed throughout: named entity recognition (NER) over the CoNLL-2003 English corpus (data/eng.train → data/eng.val). Every token gets one of a small set of entity tags; the question each model answers differently is how much context, and in what representation, do we bring to bear on each tagging decision?
The arc is a single idea: we keep widening the window of context a tagging decision can see, and we keep enriching how a word is represented - from atomic word-IDs (HMM), to learned dense vectors read left-to-right (LSTM), to pretrained contextual subword vectors that see the whole sentence at once (BERT).
Provenance: all three live in the CMSI 5370 (NLP) problem-set-2 archive. The HMM and LSTM were the submitted assignment; the BERT tagger is the post-submission “kept going” branch (copy_ps2-scottn66/) that adds HuggingFace fine-tuning.
Full treatment lives on the HMM page; here is the one-paragraph recap so the comparison stands on its own.
The HMM is a generative, count-based model. Training is a single pass of maximum-likelihood counting over the labelled corpus that fills three tables:
P(tag₁) - how often each tag starts a sentence.P(tagₜ | tagₜ₋₁) - the order-1 Markov assumption: a tag depends only on the tag immediately before it.P(word | tag) - how likely each word is, given a tag.All three tables are initialised to ones rather than zeros, which folds add-one (Laplace) smoothing straight into the model so no unseen event gets zero probability. Decoding finds the best tag sequence two ways: greedy (pick the locally best tag at each step) or Viterbi (a dynamic program in log-space that recovers the single most-likely whole-sequence tagging via a trellis + backpointers).
Right: trains in seconds on a CPU, is fully interpretable (every number is a count), and Viterbi gives a provably optimal decode under the model. The in-code benchmark hint targets ~90% token accuracy with greedy and ~91% with Viterbi.
Caps out because: a word is an atomic ID - “Paris” and “London” share nothing, and any word unseen in training collapses to <UNK>. The Markov-1 assumption means a tag sees only the single previous tag, never the words around it. Those two limits are exactly what the next two models attack.
The LSTM tagger (nn.py / lstm.py, PyTorch) swaps counting for learning. It is a small discriminative network with three layers:
word IDs ─▶ nn.Embedding(vocab, embed_dim=32) # learn a dense vector per word
─▶ nn.LSTM(embed_dim=32, hidden_dim=12) # carry a memory cell across the sentence
─▶ nn.Linear(hidden_dim=12, num_tags) # project each step's state to tag scoresTwo things change versus the HMM. First, words become learned dense vectors instead of atomic IDs, so the model can discover that “Paris” and “London” behave alike. Second, the recurrent cell means the tag for word i is informed by a hidden state that has absorbed every word to its left - far more context than the HMM's single previous tag. Training is gradient descent (Adam, lr=0.001, 10 epochs), one sentence per backward pass.
hidden_dim=12) - an instructional baseline, not a tuned model.forward() returns raw logits, but the loss is nn.NLLLoss(), which expects log-probabilities (the log_softmax line is commented out). The network still trains and converges, but the printed loss is not interpretable as cross-entropy.The LSTM fixes the HMM's representation and left-context limits, but it still reads strictly left-to-right and learns its embeddings from scratch on a small corpus. The next model removes both of those constraints at once.
The transformer tagger (bert.py) fine-tunes DistilBERT - a distilled, ~66M-parameter BERT - for token classification through the HuggingFace Trainer API:
datasets = load_dataset('conll2003') # gold NER data
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-cased')
model = AutoModelForTokenClassification.from_pretrained(
'distilbert-base-cased', num_labels=len(label_list))
args = TrainingArguments(lr=2e-5, batch=16, epochs=3, weight_decay=0.01)
trainer = Trainer(model, args, train_dataset=..., data_collator=...)
trainer.train()Two qualitative leaps over the LSTM:
<UNK>.This branch isn't just code: a fine-tuned model was pushed to the HuggingFace Hub as scottn66/distilbert-base-uncased-finetuned-ner, and trans4mers.py wires it into live POS / NER / question-answering pipelines on sample text.
The original archived bert.py couldn't be trusted to report a number: it had a train/eval leak (train_dataset == eval_dataset) and fed word-level labels to the Trainer without aligning them to DistilBERT's subword tokens. A hardened rewrite (proper subword label alignment, a held-out split, and seqeval scoring - see the roadmap below) fixes all of that and produces real, reproducible numbers.
Fine-tuned DistilBERT-cased, 3 epochs on CoNLL-2003 (HF 9-label BIO version) scores, on the held-out test set: entity-level F1 = 0.888 (precision 0.880, recall 0.896) and token accuracy = 0.978. On the validation set the best epoch reaches 0.934 entity-F1 / 0.989 token accuracy. The ~88.8% test entity-F1 is right in the expected band for DistilBERT on CoNLL-2003.
For a strict apples-to-apples check against the HMM and LSTM - the same local 5-class data, the same preprocessing - the fine-tuned model reaches 98.5% token accuracy (macro token-F1 0.933) versus the HMM's ~91% Viterbi token accuracy. Same task, same labels: +7 points from contextual pretraining.
The leaky baseline was rewritten into a properly evaluated DistilBERT NER model. Each fix below is now implemented:
is_split_into_words=True and use word_ids() to spread each word's gold tag onto its first subword while masking continuation pieces and specials with -100 (ignored by the loss). This is the single change that makes the model correct. Verified against the live tokenizer: Kubernetes splits to Ku/##ber/##net/##es, and only the first piece keeps the word's tag (the rest become -100).compute_metrics runs seqeval for entity-level precision / recall / F1 (the standard CoNLL score) on the BIO data, and falls back to token-level scoring for the bare 5-class data.datasets object; the unused BERTPoSTagger, CustomDataset, and precomputed-then-discarded encoded_inputs are gone.transformers import (no trans4mers shim) and a complete requirements.txt. Going further than planned: the rewrite runs on transformers 5.x (processing_class replaces the removed tokenizer= arg) and datasets 5.x (script loading was removed, so it pulls a parquet CoNLL-2003 mirror).bert-base-cased vs DistilBERT; reconcile the cased/uncased split with the published uncased Hub model; optionally add a CRF decode head so the transformer, like the HMM, optimizes the whole tag sequence rather than each token independently.Result: 0.888 test entity-F1 (0.934 on validation) - squarely in the expected band, and now trustworthy because the evaluation is honest.
The same task seen through three lenses. Where a cell would require a number the source never trustworthily produced, it says so rather than inventing one.
| Dimension | HMM | LSTM | BERT / DistilBERT |
|---|---|---|---|
| Paradigm | Generative, count-based | Discriminative, trained from scratch | Pretrained + fine-tuned (transfer learning) |
| Word representation | Atomic ID; <UNK> for unseen |
Learned dense vector (32-dim) | Pretrained contextual subword vectors (768-dim) |
| Context per decision | Previous tag only (Markov-1) | All words to the left (unidirectional) | Entire sentence, both directions (self-attention) |
| Out-of-vocabulary words | Brittle (smoothed <UNK>) |
Single <UNK> embedding |
Graceful (WordPiece subwords) |
| Training | One MLE counting pass; seconds, CPU | Backprop, Adam, 10 epochs, 1 sentence/step | Fine-tune 3 epochs from pretrained; GPU-friendly |
| Parameters | Count tables, O(tags² + tags·|V|) | Tiny (embed 32, hidden 12) | ~66M (DistilBERT) |
| Sequence decoding | Viterbi (global optimum) or greedy | Per-token argmax | Per-token argmax (CRF head = stretch goal) |
| Interpretability | High - every value is a count | Low | Low |
| Documented result | ~90% greedy / ~91% Viterbi token acc. (in-code hint) | Not recorded (+ NLLLoss/logits caveat) | 0.888 entity-F1 (CoNLL test) · 98.5% token-acc on local 5-class (vs HMM ~91%) |
| Main limitation | Markov-1 + atomic words | Left-only, tiny, no pretraining | Per-token decode (no CRF); cased/uncased to reconcile |
Read top to bottom, the three models tell one story: each generation buys richer context and richer word representations at the cost of interpretability and compute. The HMM is transparent and instant but myopic; the LSTM learns representations and remembers its left context; BERT brings world knowledge from pretraining and sees the whole sentence at once. The HMM still wins on transparency and on globally optimal decoding - which is why a CRF head on top of BERT (giving it Viterbi-style sequence optimization) is the natural place these two ends of the ladder meet.
The HMM at the base of this ladder is the same row-stochastic transition matrix that the conscious-agents page repurposes as the substrate of a perception theory. This page climbs the other direction - from that Markov baseline toward modern neural taggers.