CQP-based feature extraction for ICE corpora in QuanTOR

Requirements

You need a working installation of the CWB with cqp available on your PATH. To verify, run cqp -v in the command line. You should see version and copyright information printed. Afterwards, encode your corpus by following the tutorial. It should contain at least one s-attribute (e.g., text_id), which will be used for aggregation. The feature annotation (see below) additionally requires the CWB Perl bindings.

Quick Start

Feature Extraction

The feature extraction scripts output tables in TSV format with one row per s-attribute and one column per feature. The first column is the unique ID of the s-attribute used for aggregation. The first feature is always n_token, which is automatically derived form the s-attribute used in the init_corpus and init_subcorpus macros.

Python version

Run featex.py (we strongly recommend using uv) and provide the CQP file as the first argument. It automatically distinguishes between script and notebook formats. All further arguments (such as -D for specifying the corpus) are forwarded as-is to CQP. For example:

uv run featex.py featex.cqp -D ICE9 > features.tsv

Perl version

Run cqp as a child process with -c and provide the CQP file with -f (for scripts) or -n (for notebooks). Then pipe its output into featex.perl using |. For example:

cqp -c -D ICE9 -f featex.cqp | perl featex.perl features.tsv

How it works

A minimal CQP file should contain the following (see featex.cqp for details):

define macro < "macros/featex.m";
define macro < "macros/mwus.m";
/init_corpus[text_id];
# or: Corpus = ...;
# /init_subcorpus[Corpus, text_id];

macros/featex.m defines the core macros for set operations and feature outputs. The CQP file may only produce output using the feature[...] macro for individual spans and feature_count[...] macro for aggregated counts (this is more efficient when spans are not needed).

The wordlists and mwulists folders contain lists of words and multi-word units used in some of the CQP queries. Both allow regular expression syntax. You can normalize, sort, and validate the word lists with uv run wordlists.py wordlists. The MWU lists are used to automatically generate macros for sequence queries in macros/mwus.m. As such, do not edit this file yourself. Instead update the macro using uv run mwulists.py mwulists > macros/mwus.m.

Feature Annotation

featex-annotate.perl expects CQP output on standard input and writes token-level feature spans back into a CWB corpus as s-attributes instead of generating a matrix. The CQP script must emit token spans for the features you want to write back into the corpus. In practice, that means using token-level feature macros rather than count-only features for the attributes you want to annotate. Avoid including extremely high-frequency features (such as all nouns) unless you explicitly need them.

cqp -c -D ICE9 -f featex.cqp | perl featex-annotate.perl ICE9