Hebrew TenTen web corpus.

The corpus is cleaned by jusText, deduplicated by onion, tokenised by modified Yoav Goldberg’s Hebrew tokeniser and annotated with morphological categories by Meni Adler‘s tool. With thanks to Noam Ordan.

Structural attributes

Common TenTen corpora attributes

Token attributes

Hebrew is a language with a rich morphology. The tagger output in Hebrew has 27 different attributes, some of which are relevant only to certain parts of speech. Apart from usual attributes such as word, tag and lemma, there are transcriptions to latin alphabet, various morphology categories (affixes), various grammatical categories (gender, number, person, case). The tagset reference can be found below. More about the annotation.

Whenever an attribute is irrelevant or missing, its value is string ‘NULL’. For example, to search for all nouns in the corpus which are prefixed with the definite article, issue the following CQL query: [tag=”noun” & prefdefinite=”NULL”].

Tokens, lemmas, prefixes and suffixes were transliterated according to the following key:
‫ת ש ר ק צ פ ע ס נ מ ל כ י ט ח ז ו ה ד ג ב א‬
a b g d h w z x v i k l m n S y p c q r e T
Whereas the token and lemma attributes are available both in Hebrew alphabet and romanised transliteration, the prefix string and the suffix string appear only in transliteration.

Full name

Attribute

Values

Word word
Transliteration of the word trans
Lemma lemma
Transliteration of the lemma transl
Part of speech tag adjective, adverb, conjunction, copula, existential, foreign, interjection, interrogative, modal, negation, noun, numberExpression, numeral, participle, preposition, pronoun, properName, punctuation, quantifier, title, url, verb, wPrefix
Part of speech type postype amount and arithmetic-operation, bracket-end, bracket-start, colon, comma, coordinating, demonstrative, determiner, dot, exclamation-mark, gematria, hyphen, impersonal, literal-number, numeral-cardinal, numeral-fractional, numeral-ordinal or other, partitive, personal, proadverb, prodet, pronoun, question-mark, quote, reflexive, relativizing, semicolon, slash, subordinating, yesno
Prefix string prestring ‫ב בכ ו וב ובכ וכ וכש וכשל ול ומ ומכ ומש‬ ‫וש ושב ושל ושמ כ ככ כש כשב כשל כשמ‬ ‫ל לכ לכש מ מכ מש משב משכ משל משמ‬ ‫ש שב שכ שכש שכשמ של שמ שמש‬
Base string basestring
Suffix string sufstring ‫גם ה הם הן ו י ך כם כן ם ן נו‬
Gender gender feminine, masculine, masculine-and-feminine
Number number dual, dual-and-plural, plural, singular, singular-and-plural
 Status status absolute, construct
Polarity polarity negative, positive
Person person 1, 2, 3, any
Tense tense beinoni, future, imperative, infinitive, past
Binyan binyan Hifil, Hitpael, Hufal, Nifal, Paal, Piel, Pual
Prefix conjunction prefconj conjunction
Prefix definite article prefdefinite definiteArticle
Prefix interrogative prefinterrog
Prefix preposition prefprep preposition
Prefix subordination conjunction / relativizer relativizer relativizer / subordinatingConjunction
Prefix temporal subordinating conjunction preftemp temporalSubConj
Prefix adverb prefadv adverb
Suffix function suffunction accusative-or-nominative, possessive, pronomial
Suffix number sufnum plural, singular
Suffix gender sufgender feminine, masculine, masculine-and-feminine
Suffix person sufper 1, 2, 3

Changelog

v. 0 (September 2014)

v. 1 (January 2015)

PoS tagged & annotated with morphological categories