598 lines
20 KiB
HTML
598 lines
20 KiB
HTML
|
<! $Id: ngram-count.1,v 1.51 2019/09/09 22:35:37 stolcke Exp $>
|
|||
|
|
<HTML>
|
|||
|
|
<HEADER>
|
|||
|
|
<TITLE>ngram-count</TITLE>
|
|||
|
|
<BODY>
|
|||
|
|
<H1>ngram-count</H1>
|
|||
|
|
<H2> NAME </H2>
|
|||
|
|
ngram-count - count N-grams and estimate language models
|
|||
|
|
<H2> SYNOPSIS </H2>
|
|||
|
|
<PRE>
|
|||
|
|
<B>ngram-count</B> [ <B>-help</B> ] <I>option</I> ...
|
|||
|
|
</PRE>
|
|||
|
|
<H2> DESCRIPTION </H2>
|
|||
|
|
<B> ngram-count </B>
|
|||
|
|
generates and manipulates N-gram counts, and estimates N-gram language
|
|||
|
|
models from them.
|
|||
|
|
The program first builds an internal N-gram count set, either
|
|||
|
|
by reading counts from a file, or by scanning text input.
|
|||
|
|
Following that, the resulting counts can be output back to a file
|
|||
|
|
or used for building an N-gram language model in ARPA
|
|||
|
|
<A HREF="ngram-format.5.html">ngram-format(5)</A>.
|
|||
|
|
Each of these actions is triggered by corresponding options, as
|
|||
|
|
described below.
|
|||
|
|
<H2> OPTIONS </H2>
|
|||
|
|
<P>
|
|||
|
|
Each filename argument can be an ASCII file, or a
|
|||
|
|
compressed file (name ending in .Z or .gz), or ``-'' to indicate
|
|||
|
|
stdin/stdout.
|
|||
|
|
<DL>
|
|||
|
|
<DT><B> -help </B>
|
|||
|
|
<DD>
|
|||
|
|
Print option summary.
|
|||
|
|
<DT><B> -version </B>
|
|||
|
|
<DD>
|
|||
|
|
Print version information.
|
|||
|
|
<DT><B>-order</B><I> n</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Set the maximal order (length) of N-grams to count.
|
|||
|
|
This also determines the order of the estimated LM, if any.
|
|||
|
|
The default order is 3.
|
|||
|
|
<DT><B>-vocab</B><I> file</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Read a vocabulary from file.
|
|||
|
|
Subsequently, out-of-vocabulary words in both counts or text are
|
|||
|
|
replaced with the unknown-word token.
|
|||
|
|
If this option is not specified all words found are implicitly added
|
|||
|
|
to the vocabulary.
|
|||
|
|
<DT><B>-vocab-aliases</B><I> file</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Reads vocabulary alias definitions from
|
|||
|
|
<I>file</I>,<I></I><I></I><I></I>
|
|||
|
|
consisting of lines of the form
|
|||
|
|
<PRE>
|
|||
|
|
<I>alias</I> <I>word</I>
|
|||
|
|
</PRE>
|
|||
|
|
This causes all tokens
|
|||
|
|
<I> alias </I>
|
|||
|
|
to be mapped to
|
|||
|
|
<I>word</I>.<I></I><I></I><I></I>
|
|||
|
|
<DT><B>-write-vocab</B><I> file</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Write the vocabulary built in the counting process to
|
|||
|
|
<I>file</I>.<I></I><I></I><I></I>
|
|||
|
|
<DT><B>-write-vocab-index</B><I> file</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Write the internal word-to-integer mapping to
|
|||
|
|
<I>file</I>.<I></I><I></I><I></I>
|
|||
|
|
<DT><B> -tagged </B>
|
|||
|
|
<DD>
|
|||
|
|
Interpret text and N-grams as consisting of word/tag pairs.
|
|||
|
|
<DT><B> -tolower </B>
|
|||
|
|
<DD>
|
|||
|
|
Map all vocabulary to lowercase.
|
|||
|
|
<DT><B> -memuse </B>
|
|||
|
|
<DD>
|
|||
|
|
Print memory usage statistics.
|
|||
|
|
</DD>
|
|||
|
|
</DL>
|
|||
|
|
<H3> Counting Options </H3>
|
|||
|
|
<DL>
|
|||
|
|
<DT><B>-text</B><I> textfile</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Generate N-gram counts from text file.
|
|||
|
|
<I> textfile </I>
|
|||
|
|
should contain one sentence unit per line.
|
|||
|
|
Begin/end sentence tokens are added if not already present.
|
|||
|
|
Empty lines are ignored.
|
|||
|
|
<DT><B> -text-has-weights </B>
|
|||
|
|
<DD>
|
|||
|
|
Treat the first field in each text input line as a weight factor by
|
|||
|
|
which the N-gram counts for that line are to be multiplied.
|
|||
|
|
<DT><B> -text-has-weights-last </B>
|
|||
|
|
<DD>
|
|||
|
|
Similar to
|
|||
|
|
<B> -text-has-weights </B>
|
|||
|
|
but with weights at the ends of lines.
|
|||
|
|
<DT><B> -no-sos </B>
|
|||
|
|
<DD>
|
|||
|
|
Disable the automatic insertion of start-of-sentence tokens
|
|||
|
|
in N-gram counting.
|
|||
|
|
<DT><B> -no-eos </B>
|
|||
|
|
<DD>
|
|||
|
|
Disable the automatic insertion of end-of-sentence tokens
|
|||
|
|
in N-gram counting.
|
|||
|
|
<DT><B>-read</B><I> countsfile</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Read N-gram counts from a file.
|
|||
|
|
Ascii count files contain one N-gram of
|
|||
|
|
words per line, followed by an integer count, all separated by whitespace.
|
|||
|
|
Repeated counts for the same N-gram are added.
|
|||
|
|
Thus several count files can be merged by using
|
|||
|
|
<A HREF="cat.1.html">cat(1)</A>
|
|||
|
|
and feeding the result to
|
|||
|
|
<B>ngram-count -read -</B><B></B><B></B><B></B>
|
|||
|
|
(but see
|
|||
|
|
<A HREF="ngram-merge.1.html">ngram-merge(1)</A>
|
|||
|
|
for merging counts that exceed available memory).
|
|||
|
|
Counts collected by
|
|||
|
|
<B> -text </B>
|
|||
|
|
and
|
|||
|
|
<B> -read </B>
|
|||
|
|
are additive as well.
|
|||
|
|
Binary count files (see below) are also recognized.
|
|||
|
|
<DT><B>-read-google</B><I> dir</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Read N-grams counts from an indexed directory structure rooted in
|
|||
|
|
<B>dir</B>,<B></B><B></B><B></B>
|
|||
|
|
in a format developed by
|
|||
|
|
Google to store very large N-gram collections.
|
|||
|
|
The corresponding directory structure can be created using the script
|
|||
|
|
<B> make-google-ngrams </B>
|
|||
|
|
described in
|
|||
|
|
<A HREF="training-scripts.1.html">training-scripts(1)</A>.
|
|||
|
|
<DT><B>-intersect</B><I> file</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Limit reading of counts via
|
|||
|
|
<B> -read </B>
|
|||
|
|
and
|
|||
|
|
<B> -read-google </B>
|
|||
|
|
to a subset of N-grams defined by another count
|
|||
|
|
<I>file</I>.<I></I><I></I><I></I>
|
|||
|
|
(The counts in
|
|||
|
|
<I> file </I>
|
|||
|
|
are ignored, only the N-grams are relevant.)
|
|||
|
|
<DT><B>-write</B><I> file</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Write total counts to
|
|||
|
|
<I>file</I>.<I></I><I></I><I></I>
|
|||
|
|
<DT><B>-write-binary</B><I> file</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Write total counts to
|
|||
|
|
<I> file </I>
|
|||
|
|
in binary format.
|
|||
|
|
Binary count files cannot be compressed and are typically
|
|||
|
|
larger than compressed ascii count files.
|
|||
|
|
However, they can be loaded faster, especially when the
|
|||
|
|
<B> -limit-vocab </B>
|
|||
|
|
option is used.
|
|||
|
|
<DT><B>-write-order</B><I> n</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Order of counts to write.
|
|||
|
|
The default is 0, which stands for N-grams of all lengths.
|
|||
|
|
<DT><B>-write</B><I>n file</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
where
|
|||
|
|
<I> n </I>
|
|||
|
|
is 1, 2, 3, 4, 5, 6, 7, 8, or 9.
|
|||
|
|
Writes only counts of the indicated order to
|
|||
|
|
<I>file</I>.<I></I><I></I><I></I>
|
|||
|
|
This is convenient to generate counts of different orders
|
|||
|
|
separately in a single pass.
|
|||
|
|
<DT><B> -sort </B>
|
|||
|
|
<DD>
|
|||
|
|
Output counts in lexicographic order, as required for
|
|||
|
|
<A HREF="ngram-merge.1.html">ngram-merge(1)</A>.
|
|||
|
|
<DT><B> -recompute </B>
|
|||
|
|
<DD>
|
|||
|
|
Regenerate lower-order counts by summing the highest-order counts for
|
|||
|
|
each N-gram prefix.
|
|||
|
|
<DT><B> -limit-vocab </B>
|
|||
|
|
<DD>
|
|||
|
|
Discard N-gram counts on reading that do not pertain to the words
|
|||
|
|
specified in the vocabulary.
|
|||
|
|
The default is that words used in the count files are automatically added to
|
|||
|
|
the vocabulary.
|
|||
|
|
</DD>
|
|||
|
|
</DL>
|
|||
|
|
<H3> LM Options </H3>
|
|||
|
|
<DL>
|
|||
|
|
<DT><B>-lm</B><I> lmfile</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Estimate a language model from the total counts and write it to
|
|||
|
|
<I>lmfile</I>.<I></I><I></I><I></I>
|
|||
|
|
This option applies to several LM model types (see below) but the default is to
|
|||
|
|
estimate a backoff N-gram model, and write it in
|
|||
|
|
<A HREF="ngram-format.5.html">ngram-format(5)</A>.
|
|||
|
|
<DT><B> -write-binary-lm </B>
|
|||
|
|
<DD>
|
|||
|
|
Output
|
|||
|
|
<I> lmfile </I>
|
|||
|
|
in binary format.
|
|||
|
|
If an LM class does not provide a binary format the default (text) format
|
|||
|
|
will be output instead.
|
|||
|
|
<DT><B>-nonevents</B><I> file</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Read a list of words from
|
|||
|
|
<I> file </I>
|
|||
|
|
that are to be considered non-events, i.e., that
|
|||
|
|
can only occur in the context of an N-gram.
|
|||
|
|
Such words are given zero probability mass in model estimation.
|
|||
|
|
<DT><B> -float-counts </B>
|
|||
|
|
<DD>
|
|||
|
|
Enable manipulation of fractional counts.
|
|||
|
|
Only certain discounting methods support non-integer counts.
|
|||
|
|
<DT><B> -skip </B>
|
|||
|
|
<DD>
|
|||
|
|
Estimate a ``skip'' N-gram model, which predicts a word by
|
|||
|
|
an interpolation of the immediate context and the context one word prior.
|
|||
|
|
This also triggers N-gram counts to be generated that are one word longer
|
|||
|
|
than the indicated order.
|
|||
|
|
The following four options control the EM estimation algorithm used for
|
|||
|
|
skip-N-grams.
|
|||
|
|
<DT><B>-init-lm</B><I> lmfile</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Load an LM to initialize the parameters of the skip-N-gram.
|
|||
|
|
<DT><B>-skip-init</B><I> value</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
The initial skip probability for all words.
|
|||
|
|
<DT><B>-em-iters</B><I> n</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
The maximum number of EM iterations.
|
|||
|
|
<DT><B>-em-delta</B><I> d</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
The convergence criterion for EM: if the relative change in log likelihood
|
|||
|
|
falls below the given value, iteration stops.
|
|||
|
|
<DT><B> -count-lm </B>
|
|||
|
|
<DD>
|
|||
|
|
Estimate a count-based interpolated LM using Jelinek-Mercer smoothing
|
|||
|
|
(Chen & Goodman, 1998).
|
|||
|
|
Several of the options for skip-N-gram LMs (above) apply.
|
|||
|
|
An initial count-LM in the format described in
|
|||
|
|
<A HREF="ngram.1.html">ngram(1)</A>
|
|||
|
|
needs to be specified using
|
|||
|
|
<B>-init-lm</B>.<B></B><B></B><B></B>
|
|||
|
|
The options
|
|||
|
|
<B> -em-iters </B>
|
|||
|
|
and
|
|||
|
|
<B> -em-delta </B>
|
|||
|
|
control termination of the EM algorithm.
|
|||
|
|
Note that the N-gram counts used to estimate the maximum-likelihood
|
|||
|
|
estimates come from the
|
|||
|
|
<B> -init-lm </B>
|
|||
|
|
model.
|
|||
|
|
The counts specified with
|
|||
|
|
<B> -read </B>
|
|||
|
|
or
|
|||
|
|
<B> -text </B>
|
|||
|
|
are used only to estimate the smoothing (interpolation weights).
|
|||
|
|
<DT><B> -maxent </B>
|
|||
|
|
<DD>
|
|||
|
|
Estimate a maximum entropy N-gram model.
|
|||
|
|
The model is written to the file specifed by the
|
|||
|
|
<B> -lm </B>
|
|||
|
|
option.
|
|||
|
|
<BR>
|
|||
|
|
If
|
|||
|
|
<B>-init-lm</B><I> file</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
is also specified then use an existing maxent model from
|
|||
|
|
<I> file </I>
|
|||
|
|
as prior and adapt it using the given data.
|
|||
|
|
<DT><B>-maxent-alpha</B><I> A</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Use the L1 regularization constant
|
|||
|
|
<I> A </I>
|
|||
|
|
for maxent estimation.
|
|||
|
|
The default value is 0.5.
|
|||
|
|
<DT><B>-maxent-sigma2</B><I> S</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Use the L2 regularization constant
|
|||
|
|
<I> S </I>
|
|||
|
|
for maxent estimation.
|
|||
|
|
The default value is 6 for estimation and 0.5 for adaptation.
|
|||
|
|
<DT><B> -maxent-convert-to-arpa </B>
|
|||
|
|
<DD>
|
|||
|
|
Convert the maxent model to
|
|||
|
|
<A HREF="ngram-format.5.html">ngram-format(5)</A>
|
|||
|
|
before writing it out, using the algorithm by Wu (2002).
|
|||
|
|
<DT><B> -unk </B>
|
|||
|
|
<DD>
|
|||
|
|
Build an ``open vocabulary'' LM, i.e., one that contains the unknown-word
|
|||
|
|
token as a regular word.
|
|||
|
|
The default is to remove the unknown word.
|
|||
|
|
<DT><B>-map-unk</B><I> word</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Map out-of-vocabulary words to
|
|||
|
|
<I>word</I>,<I></I><I></I><I></I>
|
|||
|
|
rather than the default
|
|||
|
|
<B> <unk> </B>
|
|||
|
|
tag.
|
|||
|
|
<DT><B> -trust-totals </B>
|
|||
|
|
<DD>
|
|||
|
|
Force the lower-order counts to be used as total counts in estimating
|
|||
|
|
N-gram probabilities.
|
|||
|
|
Usually these totals are recomputed from the higher-order counts.
|
|||
|
|
<DT><B>-prune</B><I> threshold</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Prune N-gram probabilities if their removal causes (training set)
|
|||
|
|
perplexity of the model to increase by less than
|
|||
|
|
<I> threshold </I>
|
|||
|
|
relative.
|
|||
|
|
<DT><B>-minprune</B><I> n</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Only prune N-grams of length at least
|
|||
|
|
<I>n</I>.<I></I><I></I><I></I>
|
|||
|
|
The default (and minimum allowed value) is 2, i.e., only unigrams are excluded
|
|||
|
|
from pruning.
|
|||
|
|
<DT><B>-debug</B><I> level</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Set debugging output from estimated LM at
|
|||
|
|
<I>level</I>.<I></I><I></I><I></I>
|
|||
|
|
Level 0 means no debugging.
|
|||
|
|
Debugging messages are written to stderr.
|
|||
|
|
<DT><B>-gt<I>n</I>min</B><I> count</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
where
|
|||
|
|
<I> n </I>
|
|||
|
|
is 1, 2, 3, 4, 5, 6, 7, 8, or 9.
|
|||
|
|
Set the minimal count of N-grams of order
|
|||
|
|
<I> n </I>
|
|||
|
|
that will be included in the LM.
|
|||
|
|
All N-grams with frequency lower than that will effectively be discounted to 0.
|
|||
|
|
If
|
|||
|
|
<I> n </I>
|
|||
|
|
is omitted the parameter for N-grams of order > 9 is set.
|
|||
|
|
<BR>
|
|||
|
|
NOTE: This option affects not only the default Good-Turing discounting
|
|||
|
|
but the alternative discounting methods described below as well.
|
|||
|
|
<DT><B>-gt<I>n</I>max</B><I> count</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
where
|
|||
|
|
<I> n </I>
|
|||
|
|
is 1, 2, 3, 4, 5, 6, 7, 8, or 9.
|
|||
|
|
Set the maximal count of N-grams of order
|
|||
|
|
<I> n </I>
|
|||
|
|
that are discounted under Good-Turing.
|
|||
|
|
All N-grams more frequent than that will receive
|
|||
|
|
maximum likelihood estimates.
|
|||
|
|
Discounting can be effectively disabled by setting this to 0.
|
|||
|
|
If
|
|||
|
|
<I> n </I>
|
|||
|
|
is omitted the parameter for N-grams of order > 9 is set.
|
|||
|
|
</DD>
|
|||
|
|
</DL>
|
|||
|
|
<P>
|
|||
|
|
In the following discounting parameter options, the order
|
|||
|
|
<I> n </I>
|
|||
|
|
may be omitted, in which case a default for all N-gram orders is
|
|||
|
|
set.
|
|||
|
|
The corresponding discounting method then becomes the default method
|
|||
|
|
for all orders, unless specifically overridden by an option with
|
|||
|
|
<I>n</I>.<I></I><I></I><I></I>
|
|||
|
|
If no discounting method is specified, Good-Turing is used.
|
|||
|
|
<DL>
|
|||
|
|
<DT><B>-gt<I>n</I></B><I> gtfile</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
where
|
|||
|
|
<I> n </I>
|
|||
|
|
is 1, 2, 3, 4, 5, 6, 7, 8, or 9.
|
|||
|
|
Save or retrieve Good-Turing parameters
|
|||
|
|
(cutoffs and discounting factors) in/from
|
|||
|
|
<I>gtfile</I>.<I></I><I></I><I></I>
|
|||
|
|
This is useful as GT parameters should always be determined from
|
|||
|
|
unlimited vocabulary counts, whereas the eventual LM may use a
|
|||
|
|
limited vocabulary.
|
|||
|
|
The parameter files may also be hand-edited.
|
|||
|
|
If an
|
|||
|
|
<B> -lm </B>
|
|||
|
|
option is specified the GT parameters are read from
|
|||
|
|
<I>gtfile</I>,<I></I><I></I><I></I>
|
|||
|
|
otherwise they are computed from the current counts and saved in
|
|||
|
|
<I>gtfile</I>.<I></I><I></I><I></I>
|
|||
|
|
<DT><B>-cdiscount<I>n</I></B><I> discount</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
where
|
|||
|
|
<I> n </I>
|
|||
|
|
is 1, 2, 3, 4, 5, 6, 7, 8, or 9.
|
|||
|
|
Use Ney's absolute discounting for N-grams of
|
|||
|
|
order
|
|||
|
|
<I>n</I>,<I></I><I></I><I></I>
|
|||
|
|
using
|
|||
|
|
<I> discount </I>
|
|||
|
|
as the constant to subtract.
|
|||
|
|
<DT><B> -wbdiscount<I>n</I> </B>
|
|||
|
|
<DD>
|
|||
|
|
where
|
|||
|
|
<I> n </I>
|
|||
|
|
is 1, 2, 3, 4, 5, 6, 7, 8, or 9.
|
|||
|
|
Use Witten-Bell discounting for N-grams of order
|
|||
|
|
<I>n</I>.<I></I><I></I><I></I>
|
|||
|
|
(This is the estimator where the first occurrence of each word is
|
|||
|
|
taken to be a sample for the ``unseen'' event.)
|
|||
|
|
<DT><B> -ndiscount<I>n</I> </B>
|
|||
|
|
<DD>
|
|||
|
|
where
|
|||
|
|
<I> n </I>
|
|||
|
|
is 1, 2, 3, 4, 5, 6, 7, 8, or 9.
|
|||
|
|
Use Ristad's natural discounting law for N-grams of order
|
|||
|
|
<I>n</I>.<I></I><I></I><I></I>
|
|||
|
|
<DT><B>-addsmooth<I>n</I></B><I> delta</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
where
|
|||
|
|
<I> n </I>
|
|||
|
|
is 1, 2, 3, 4, 5, 6, 7, 8, or 9.
|
|||
|
|
Smooth by adding
|
|||
|
|
<I> delta </I>
|
|||
|
|
to each N-gram count.
|
|||
|
|
This is usually a poor smoothing method and included mainly for instructional
|
|||
|
|
purposes.
|
|||
|
|
<DT><B> -kndiscount<I>n</I> </B>
|
|||
|
|
<DD>
|
|||
|
|
where
|
|||
|
|
<I> n </I>
|
|||
|
|
is 1, 2, 3, 4, 5, 6, 7, 8, or 9.
|
|||
|
|
Use Chen and Goodman's modified Kneser-Ney discounting for N-grams of order
|
|||
|
|
<I>n</I>.<I></I><I></I><I></I>
|
|||
|
|
<DT><B> -kn-counts-modified </B>
|
|||
|
|
<DD>
|
|||
|
|
Indicates that input counts have already been modified for Kneser-Ney
|
|||
|
|
smoothing.
|
|||
|
|
If this option is not given, the KN discounting method modifies counts
|
|||
|
|
(except those of highest order) in order to estimate the backoff distributions.
|
|||
|
|
When using the
|
|||
|
|
<B> -write </B>
|
|||
|
|
and related options the output will reflect the modified counts.
|
|||
|
|
<DT><B> -kn-modify-counts-at-end </B>
|
|||
|
|
<DD>
|
|||
|
|
Modify Kneser-Ney counts after estimating discounting constants, rather than
|
|||
|
|
before as is the default.
|
|||
|
|
<DT><B>-kn<I>n</I></B><I> knfile</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
where
|
|||
|
|
<I> n </I>
|
|||
|
|
is 1, 2, 3, 4, 5, 6, 7, 8, or 9.
|
|||
|
|
Save or retrieve Kneser-Ney parameters
|
|||
|
|
(cutoff and discounting constants) in/from
|
|||
|
|
<I>knfile</I>.<I></I><I></I><I></I>
|
|||
|
|
This is useful as smoothing parameters should always be determined from
|
|||
|
|
unlimited vocabulary counts, whereas the eventual LM may use a
|
|||
|
|
limited vocabulary.
|
|||
|
|
The parameter files may also be hand-edited.
|
|||
|
|
If an
|
|||
|
|
<B> -lm </B>
|
|||
|
|
option is specified the KN parameters are read from
|
|||
|
|
<I>knfile</I>,<I></I><I></I><I></I>
|
|||
|
|
otherwise they are computed from the current counts and saved in
|
|||
|
|
<I>knfile</I>.<I></I><I></I><I></I>
|
|||
|
|
<DT><B> -ukndiscount<I>n</I> </B>
|
|||
|
|
<DD>
|
|||
|
|
where
|
|||
|
|
<I> n </I>
|
|||
|
|
is 1, 2, 3, 4, 5, 6, 7, 8, or 9.
|
|||
|
|
Use the original (unmodified) Kneser-Ney discounting method for N-grams of
|
|||
|
|
order
|
|||
|
|
<I>n</I>.<I></I><I></I><I></I>
|
|||
|
|
<DT><B> -interpolate<I>n</I> </B>
|
|||
|
|
<DD>
|
|||
|
|
where
|
|||
|
|
<I> n </I>
|
|||
|
|
is 1, 2, 3, 4, 5, 6, 7, 8, or 9.
|
|||
|
|
Causes the discounted N-gram probability estimates at the specified order
|
|||
|
|
<I> n </I>
|
|||
|
|
to be interpolated with lower-order estimates.
|
|||
|
|
(The result of the interpolation is encoded as a standard backoff
|
|||
|
|
model and can be evaluated as such -- the interpolation happens at
|
|||
|
|
estimation time.)
|
|||
|
|
This sometimes yields better models with some smoothing methods
|
|||
|
|
(see Chen & Goodman, 1998).
|
|||
|
|
Only Witten-Bell, absolute discounting, and
|
|||
|
|
(original or modified) Kneser-Ney smoothing
|
|||
|
|
currently support interpolation.
|
|||
|
|
<DT><B>-meta-tag</B><I> string</I><B></B><I></I><B></B><I></I><B></B>
|
|||
|
|
<DD>
|
|||
|
|
Interpret words starting with
|
|||
|
|
<I> string </I>
|
|||
|
|
as count-of-count (meta-count) tags.
|
|||
|
|
For example, an N-gram
|
|||
|
|
<PRE>
|
|||
|
|
a b <I>string</I>3 4
|
|||
|
|
</PRE>
|
|||
|
|
means that there were 4 trigrams starting with "a b"
|
|||
|
|
that occurred 3 times each.
|
|||
|
|
Meta-tags are only allowed in the last position of an N-gram.
|
|||
|
|
<BR>
|
|||
|
|
Note: when using
|
|||
|
|
<B> -tolower </B>
|
|||
|
|
the meta-tag
|
|||
|
|
<I> string </I>
|
|||
|
|
must not contain any uppercase characters.
|
|||
|
|
<DT><B> -read-with-mincounts </B>
|
|||
|
|
<DD>
|
|||
|
|
Save memory by eliminating N-grams with counts that fall below the thresholds
|
|||
|
|
set by
|
|||
|
|
<B>-gt</B><I>N</I><B>min</B><I></I><B></B><I></I><B></B>
|
|||
|
|
options during
|
|||
|
|
<B> -read </B>
|
|||
|
|
operation
|
|||
|
|
(this assumes the input counts contain no duplicate N-grams).
|
|||
|
|
Also, if
|
|||
|
|
<B> -meta-tag </B>
|
|||
|
|
is defined,
|
|||
|
|
these low-count N-grams will be converted to count-of-count N-grams,
|
|||
|
|
so that smoothing methods that need this information still work correctly.
|
|||
|
|
</DD>
|
|||
|
|
</DL>
|
|||
|
|
<H2> SEE ALSO </H2>
|
|||
|
|
<A HREF="ngram-merge.1.html">ngram-merge(1)</A>, <A HREF="ngram.1.html">ngram(1)</A>, <A HREF="ngram-class.1.html">ngram-class(1)</A>, <A HREF="training-scripts.1.html">training-scripts(1)</A>, <A HREF="lm-scripts.1.html">lm-scripts(1)</A>,
|
|||
|
|
<A HREF="ngram-format.5.html">ngram-format(5)</A>.
|
|||
|
|
<BR>
|
|||
|
|
T. Alum<75>e and M. Kurimo, ``Efficient Estimation of Maximum Entropy
|
|||
|
|
Language Models with N-gram features: an SRILM extension,''
|
|||
|
|
<I>Proc. Interspeech</I>, 1820-1823, 2010.
|
|||
|
|
<BR>
|
|||
|
|
S. F. Chen and J. Goodman, ``An Empirical Study of Smoothing Techniques for
|
|||
|
|
Language Modeling,'' TR-10-98, Computer Science Group, Harvard Univ., 1998.
|
|||
|
|
<BR>
|
|||
|
|
S. M. Katz, ``Estimation of Probabilities from Sparse Data for the
|
|||
|
|
Language Model Component of a Speech Recognizer,'' <I>IEEE Trans. ASSP</I> 35(3),
|
|||
|
|
400-401, 1987.
|
|||
|
|
<BR>
|
|||
|
|
R. Kneser and H. Ney, ``Improved backing-off for M-gram language modeling,''
|
|||
|
|
<I>Proc. ICASSP</I>, 181-184, 1995.
|
|||
|
|
<BR>
|
|||
|
|
H. Ney and U. Essen, ``On Smoothing Techniques for Bigram-based Natural
|
|||
|
|
Language Modelling,'' <I>Proc. ICASSP</I>, 825-828, 1991.
|
|||
|
|
<BR>
|
|||
|
|
E. S. Ristad, ``A Natural Law of Succession,'' CS-TR-495-95,
|
|||
|
|
Comp. Sci. Dept., Princeton Univ., 1995.
|
|||
|
|
<BR>
|
|||
|
|
I. H. Witten and T. C. Bell, ``The Zero-Frequency Problem: Estimating the
|
|||
|
|
Probabilities of Novel Events in Adaptive Text Compression,''
|
|||
|
|
<I>IEEE Trans. Information Theory</I> 37(4), 1085-1094, 1991.
|
|||
|
|
<BR>
|
|||
|
|
J. Wu (2002), ``Maximum Entropy Language Modeling with Non-Local Dependencies,''
|
|||
|
|
doctoral dissertation, Johns Hopkins University, 2002.
|
|||
|
|
<H2> BUGS </H2>
|
|||
|
|
Several of the LM types supported by
|
|||
|
|
<A HREF="ngram.1.html">ngram(1)</A>
|
|||
|
|
don't have explicit support in
|
|||
|
|
<B>ngram-count</B>.<B></B><B></B><B></B>
|
|||
|
|
Instead, they are built by separately manipulating N-gram counts,
|
|||
|
|
followed by standard N-gram model estimation.
|
|||
|
|
<BR>
|
|||
|
|
LM support for tagged words is incomplete.
|
|||
|
|
<BR>
|
|||
|
|
Only absolute and Witten-Bell discounting currently support fractional counts.
|
|||
|
|
<BR>
|
|||
|
|
The combination of
|
|||
|
|
<B> -read-with-mincounts </B>
|
|||
|
|
and
|
|||
|
|
<B> -meta-tag </B>
|
|||
|
|
preserves enough count-of-count information for
|
|||
|
|
<I> applying </I>
|
|||
|
|
discounting parameters to the input counts, but it does not
|
|||
|
|
necessarily allow the parameters to be correctly
|
|||
|
|
<I>estimated</I>.<I></I><I></I><I></I>
|
|||
|
|
Therefore, discounting parameters should always be estimated from full
|
|||
|
|
counts (e.g., using the helper
|
|||
|
|
<A HREF="training-scripts.1.html">training-scripts(1)</A>),
|
|||
|
|
and then read from files.
|
|||
|
|
<BR>
|
|||
|
|
Smoothing with
|
|||
|
|
<B> -kndiscount </B>
|
|||
|
|
or
|
|||
|
|
<B> -ukndiscount </B>
|
|||
|
|
has a side-effect on the N-gram counts, since
|
|||
|
|
the lower-order counts are destructively modified according to the KN smoothing approach
|
|||
|
|
(Kneser & Ney, 1995).
|
|||
|
|
The
|
|||
|
|
<B>-write</B><B></B><B></B><B></B>
|
|||
|
|
option will output these modified counts, and count cutoffs specified by
|
|||
|
|
<B>-gt</B><I>N</I><B>min</B><I></I><B></B><I></I><B></B>
|
|||
|
|
operate on the modified counts, potentially leading to a different set of N-grams
|
|||
|
|
being retained than with other discounting methods.
|
|||
|
|
This can be considered either a feature or a bug.
|
|||
|
|
<H2> AUTHOR </H2>
|
|||
|
|
Andreas Stolcke <stolcke@icsi.berkeley.edu>,
|
|||
|
|
Tanel Alum<75>e <tanel.alumae@phon.ioc.ee>
|
|||
|
|
<BR>
|
|||
|
|
Copyright (c) 1995-2011 SRI International
|
|||
|
|
<BR>
|
|||
|
|
Copyright (c) 2009-2013 Tanel Alum<75>e
|
|||
|
|
<BR>
|
|||
|
|
Copyright (c) 2011-2017 Andreas Stolcke
|
|||
|
|
<BR>
|
|||
|
|
Copyright (c) 2012-2017 Microsoft Corp.
|
|||
|
|
</BODY>
|
|||
|
|
</HTML>
|