The parallel corpora available here have been collected, prepared and aligned by Joerg Tiedermann in the OPUS project (see , where the source data is also available). We are most grateful to Joerg Tiedermann for his great work and co-operation. The data was prepared for the Sketch Engine using a range of lemmatisers, part-of-speech taggers and Sketch Grammars.

The OPUS project in Sketch Engine contains 40 languages: Afrikaans, Albanian, Arabic, Bosnian, Bulgarian, Chinese Simplified, Chinese Traditional, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hebrew, Hindi, Hungarian, Italian, Japanese, Korean, Latvian, Lithuanian, Macedonian, Norwegian, Persian, Polish, Portuguese, Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swedish, Turkish, Ukrainian.

List of Sub-corpora:

  • ECB – European Central Bank corpus (v.0.1)
  • EMEA – European Medicines Agency documents (v.0.3)
  • EUconst – The European constitution (v.0.1)
  • EUROPARL – European Parliament Proceedings (v.3)
  • OpenOffice 3 corpus
  • Opensubs – Open Subtitles corpus (v.2)
  • KDE4 – KDE4 localization files (v.2)
  • KDEdoc – KDE manual corpus
  • MBS – Belgisch Staatsblad corpus (v.0.1)
  • PHP – PHP manual corpus
  • RF – Regeringsförklaringen – Declarations of Government Policy by the Swedish Government
  • SETIMES – A parallel corpus of the Balkan languages (v.1)
  • SPC – Stockholm Parallel Corpora (v.1)
  • TEP – The Tehran English-Persian subtitle corpus (v.0.1)
  • Wikisource

List of tools & grammar used for various languages

Language Tools Used Grammar
Arabic Stanford tagger using Faster Arabic model trained on the Penn Arabic Treebank with  Tagset Universal Sketch Grammar with AMIRA Tagset
Bulgarian TreeTagger using UTF-8 Bulgarian parameter file trained on Tagset Yes
Chinese (Traditional, Simplified, mixed) Segmented using Stanford Segmenter modelled on segmentation standards by Chinese Penn Treebank. Tagged using Stanford tagger trained on a combination of Chinese Treebank texts from Chinese and Hong Kong sources with Tagset Universal Sketch Grammar with tags
Dutch TreeTagger using UTF-8 Italian parameter file trained on Tagset Dutch Sketch Grammar (NLWAC tagset) v4.0 by Carole Tiberius
English TreeTagger using UTF-8 English parameter file trained on Tagset English Sketch Grammar v.2.0 (Penn Treebank tagset) by Niels Ott
Estonian TreeTagger using UTF-8 Estonian parameter file trained on Tagset Estonian Sketch Grammar v1.2 by Maria and Jelena
French TreeTagger using UTF-8 French parameter file trained on Tagset French Sketch Grammar v1.0 by Adam Kilgarriff
German TreeTagger using UTF-8 German parameter file trained on Tagset Sketch Grammar for German by Matej Durco v3.3
Italian TreeTagger using UTF-8 Italian parameter file trained on Tagset Sketch Grammar for Italian v1.2 by Marco Baroni
Portuguese TreeTagger using Pablo Gamalo’s parameter file Portuguese Wordsketches (Linguateca parsed data) v1.0 by Adam Kilgarriff & DP
Russian TreeTagger using Serge Sharoff’s Russian parameter file trained on Tagset Russian Wordsketches v1.0 by Maria Khokhlova
Spanish TreeTagger using UTF-8 Spanish parameter file trained on Tagset Spanish Wordsketches v1.0 by Nuria Bel and Hada Ross Salazar (Pompeu Fabra University, Barcelona)

All other languages without tagging tools were just tokenised using Universal tokenizer built by Jan Pomikalek and inspired by Laurent Pointal’s TreeTagger wrapper.