Manually Annotated Sub-Corpus

Manually Annotated Sub-Corpus (MASC) is a balanced subset of 500K words of written texts and transcribed speech drawn primarily from the Open American National Corpus (OANC). The OANC is a 15 million word (and growing) corpus of American English produced since 1990, all of which is in the public domain or otherwise free of usage and redistribution restrictions.

All of MASC includes manually validated annotations for logical structure (headings, sections, paragraphs, etc.), sentence boundaries, three different tokenizations with associated part of speech tags, shallow parse (noun and verb chunks), named entities (person, location, organization, date and time), and Penn Treebank syntax. Additional manually produced or validated annotations have been produced by the MASC project for portions of the sub-corpus, including full-text annotation for FrameNet frame elements and a 100K+ sentence corpus with WordNet 3.1 sense tags, of which one-tenth are also annotated for FrameNet frame elements. Annotations of all or portions of the sub-corpus for a wide variety of other linguistic phenomena have been contributed by other projects, including PropBank, TimeBank, MPQA opinion, and several others. Co-reference annotations and clause boundaries of the entire MASC corpus are scheduled to be released by the end of 2016.

WordNet sense annotations for all occurrences of 114 words are also included in the MASC distribution, as well as FrameNet annotations for 50-100 occurrences of each of the 114 words. The sentences with WordNet and FrameNet annotations are also distributed as a part of the MASC Sentence Corpus.

Genres

Unlike most freely available corpora including a wide variety of linguistic annotations, MASC contains a balanced selection of texts from a broad range of genres:

Genre No. files No. words Pct corpusCourt transcript 2 30052 6%Debate transcript 2 32325 6%Email 78 27642 6%Essay 7 25590 5%Fiction 5 31518 6%Gov’t documents 5 24578 5%Journal 10 25635 5%Letters 40 23325 5%Newspaper 41 23545 5%Non-fiction 4 25182 5%Spoken 11 25783 5%Technical 8 27895 6%Travel guides 7 26708 5%Twitter 2 24180 5%Blog 21 28199 6%Ficlets 5 26299 5%Movie script 2 28240 6%Spam 110 23490 5%Jokes 16 26582 5%TOTAL 376 506768

Annotations

At present, MASC includes seventeen different types of linguistic annotation (* = in production; ** currently available in original format only):

Annotation type No. wordsLogical 506768Token 506768Sentence 506768POS/lemma (GATE) 506768POS (Penn Treebank) 506768POS (FrameNet) 506768Noun chunks 506768Verb chunks 506768Named Entities (person, org, loc, date) 506768Penn Treebank syntax 506768Coreference *506768Clause boundaries, nucleus/satellite distinctions, discourse markers *506768FrameNet frames/frame elements 39160PropBank **88530Opinion 51243TimeBank *55599Committed Belief 4614Event 4614Dependency treebank **5434Lexical substitution **35,547

All MASC annotations, whether contributed or produced in-house, are transduced to the Graph Annotation Format (GrAF) defined by ISO TC37 SC4's Linguistic Annotation Framework (LAF). The online tool ANC2Go can transduce annotations over all or parts of MASC to any of several other formats, including CONLL IOB format and formats for use in UIMA and General Architecture for Text Engineering.

Distribution

MASC is an open data resource that can be used by anyone for any purpose. At the same time, it is a collaborative community resource that is sustained by community contributions of annotations and derived data. It is freely downloadable from the MASC download page or through the Linguistic Data Consortium.

MASC is also distributed in part-of-speech-tagged form with the Natural Language Toolkit.

Manually Annotated Sub-Corpus

Bron en licentie