Texts! In! Space!

John Ladd, jrladd.com/slides/texts-in-space

spacebar for next slide, arrow keys to navigate, esc/menu for overview

What makes texts similar?

Representing texts as numerical points

Exploring text corpora

Finding similar texts by content

  • Are texts similar when they use the same words? (Bag of Words representation)
  • Are texts similar when they use similar words? (Word & Document embeddings)

Text Similarity Workflow

  1. Normalization and regularization
  2. Encode texts with TF–IDF or Document Embeddings
  3. Calculate similarities with Cosine Distance
  4. Visualize with t-SNE: t-distributed Stochastic Neighbor Embedding

Do we need to normalize spelling?

From EarlyPrint Corpus Search

For TF–IDF, yes! For document embedding, probably not.

TF–IDF, or: Word Counts

Let’s count some words!

austen = “It is a truth universally acknowledged, that a single man in possession of a good fortune must be in want of a wife.”

wharton = “I had the story, bit by bit, from various people, and, as generally happens in such cases, each time it was a different story.”

From the Programming Historian.

We care if a word is distinctive.

Term Frequency–Inverse Document Frequency

\[TF = \frac{Number\,of\,times\,a\,word\,appears}{Total\,words\,in\,the\,text}\]

\[IDF = \ln({\frac{Total\,number\,of\,texts+1}{Number\,of\,texts\,the\,word\,appears\,in}})+1\]

\[TF \times IDF\]

Document embeddings, or: AI ✨

Word vs. Context Embedding

From Jay Alammar’s The Illustrated Word2Vec

From AI for Humanists, Measuring Word Similarity with BERT

Combine context embeddings into document embeddings.

Uses a specialized LLM called an embedding model.

From the Ollama documentation

Let’s compare approaches.

Tf-Idf Wordcounts

LLM document embeddings

Which to use?

TF–IDF:

  • Calculated quickly and cheaply
  • Explainable results at every step
  • Ignores similar words
  • Spelling inconsistencies create spurious connections

Embedding Model:

  • Calculations are longer/more expensive
  • More difficult to explain/interpret
  • Accounts for similarities among words
  • More accurate similarities across a longer span of time

In my project, I used both!

Each approach can validate and support the other.

Tf-Idf Wordcounts

LLM document embeddings