Run Real Linguistic Analysis

Tokenization, lexical diversity (TTR, MATTR, MTLD, HD-D), lexical density, repeated n-grams, adjacent-sentence content-word overlap, and connective density. Lexical density uses a function-word list rather than a POS tagger.

MTLD & HD-D: McCarthy & Jarvis (2010). MATTR: Covington & McFall (2010). Cohesion indices in the spirit of TAACO: Crossley, Kyle & McNamara (2016).

Sentence embeddings from a BERT-family transformer, cosine similarity between two texts, a 2-D PCA projection of sentence embeddings, and a token-level similarity heatmap.

Model: Xenova/all-MiniLM-L6-v2 (BERT-family sentence transformer), 8-bit quantized, via transformers.js. WebGPU with WASM fallback.

Part-of-speech tags and a dependency parse with an arc diagram, plus live dependency-based complexity measures: tree depth and mean dependency distance.

Parser: spaCy en_core_web_sm running in a single stateless serverless function. If the function is unavailable, a hand-annotated UD-style example is shown and labeled as such.

Keyword-in-context concordance: every occurrence of your keyword, aligned in a column with its left and right context, the reading view corpus linguistics is built on.

Concordancing is standard corpus-linguistic practice; see e.g. Sinclair (1991), Corpus, Concordance, Collocation.

Which words are characteristic of corpus A versus corpus B? Log-likelihood compares each word's frequency against its expected frequency under a shared model; higher values mean stronger keyness.

Dunning (1993); Rayson & Garside (2000). Words with combined frequency < 3 are skipped.

Collocates of a node word within a ±4 window, ranked by mutual information, with t-scores alongside. MI favors exclusive pairings; t favors frequent ones.

Church & Hanks (1990). Collocates with fewer than 2 co-occurrences are skipped.

Flesch Reading Ease and Flesch-Kincaid grade level from word, sentence, and syllable counts. Syllables are counted with a vowel-group heuristic, so scores are approximate.

Kincaid, Fishburne, Rogers & Chissom (1975).

Lexicon- and rule-based sentiment scoring tuned for social media text: compound score plus positive/neutral/negative proportions.

Hutto & Gilbert (2014). Runs via the Apache-2.0 vader-sentiment JS port, loaded on demand.

Word frequency against frequency rank on log-log axes. Natural language famously falls near a straight line with slope −1: Zipf's law.

Zipf (1949), Human Behavior and the Principle of Least Effort.

Turns a text into a labeled vector over the 17 Universal Dependencies UPOS categories: the register fingerprint used in multi-feature analyses. Toggle raw counts, per-100-tokens, or relative proportion; compare two texts side by side with cosine similarity.

Multi-feature register analysis: Biber (1988). Tagger: spaCy en_core_web_sm.

The same treatment over syntactic relations: a vector over the top ~20 dependency labels by frequency (rest binned as 'other'). Toggle normalization; compare two texts with cosine similarity over the normalized vectors.

Multi-feature register analysis: Biber (1988). Parser: spaCy en_core_web_sm.