Skip to content

Instantly share code, notes, and snippets.

@x-yuri
Last active October 31, 2024 12:25
Show Gist options
  • Select an option

  • Save x-yuri/d6e3d01204977400fe64a0a835badb69 to your computer and use it in GitHub Desktop.

Select an option

Save x-yuri/d6e3d01204977400fe64a0a835badb69 to your computer and use it in GitHub Desktop.
mecab

mecab

Dockerfile (git):

FROM alpine:3.20
RUN apk add git build-base curl \
    && git clone https://github.com/taku910/mecab \
    && cd mecab/mecab \
    && ./configure \
    && make install \
    && cd ../mecab-ipadic \
    && ./configure --with-charset=utf8 \
    && make install \
    && cd ../mecab-jumandic \
    && ./configure --with-charset=utf8 \
    && make install \
    \
    && cd / \
    && curl -LOJ https://clrd.ninjal.ac.jp/unidic_archive/2302/unidic-cwj-202302.zip \
    && unzip -d unidic-cwj-202302 unidic-cwj-202302.zip \
    \
    && curl -LOJ https://clrd.ninjal.ac.jp/unidic_archive/2302/unidic-csj-202302.zip \
    && unzip -d unidic-csj-202302 unidic-csj-202302.zip

Dockerfile (tar):

FROM alpine:3.20
RUN apk add build-base curl \
    && curl -LOJ 'https://drive.google.com/uc?export=download&id=0B4y35FiV1wh7cENtOXlicTFaRUE' \
    && tar xf mecab-0.996.tar.gz \
    && cd mecab-0.996 \
    && ./configure \
    && make install \
    \
    && cd .. \
    && curl -LOJ 'https://drive.google.com/uc?export=download&id=0B4y35FiV1wh7MWVlSDBCSXZMTXM' \
    && tar xf mecab-ipadic-2.7.0-20070801.tar.gz \
    && cd mecab-ipadic-2.7.0-20070801 \
    && ./configure --with-charset=utf8 \
    && make install \
    \
    && cd .. \
    && curl 'https://drive.usercontent.google.com/download?id=0B4y35FiV1wh7X2pESGlLREpxdXM&export=download&confirm=t' -o mecab-jumandic-7.0-20130310.tar.gz \
    && tar xf mecab-jumandic-7.0-20130310.tar.gz \
    && cd mecab-jumandic-7.0-20130310 \
    && ./configure --with-charset=utf8 \
    && make install \
    \
    && cd / \
    && curl -LOJ https://clrd.ninjal.ac.jp/unidic_archive/2302/unidic-cwj-202302.zip \
    && unzip -d unidic-cwj-202302 unidic-cwj-202302.zip \
    && curl -LOJ https://clrd.ninjal.ac.jp/unidic_archive/2302/unidic-csj-202302.zip \
    && unzip -d unidic-csj-202302 unidic-csj-202302.zip

mecab installs:

usr/local/bin/mecab
usr/local/bin/mecab-config
usr/local/etc/mecabrc
usr/local/include/mecab.h
usr/local/lib/libmecab.a
usr/local/lib/libmecab.la
usr/local/lib/libmecab.so
usr/local/lib/libmecab.so.2
usr/local/lib/libmecab.so.2.0.0
usr/local/libexec/mecab/mecab-cost-train
usr/local/libexec/mecab/mecab-dict-gen
usr/local/libexec/mecab/mecab-dict-index
usr/local/libexec/mecab/mecab-system-eval
usr/local/libexec/mecab/mecab-test-gen
usr/local/share/man/man1/mecab.1

The dictionaries (ipadic, jumandic) are installed to /usr/local/lib/mecab/dic.

mecab needs at least one dictionary, by default (according to the config, at /usr/local/etc/mecabrc) ipadic. One can override the dictionary location with (-d, --dicdir=DIR):

$ mecab -d /usr/local/lib/mecab/dic/jumandic
$ mecab -d unidic-cwj-202302
$ mecab -d unidic-csj-202302

Show information about the dictionary (-D, --dictionary-info):

$ mecab -D
$ mecab -d /usr/local/lib/mecab/dic/jumandic -D
$ mecab -d unidic-cwj-202302 -D
$ mecab -d unidic-csj-202302 -D

There are 3 builtin output formats: the default one, wakati, dump. And none. Dictionaries can define their own custom formats in dicrc. Settings defining a format are suffixed with the format name (e.g. node-format-yomi). To choose a format use -O, --output-format-type=TYPE:

$ mecab -O wakati

Output N best results (-N, --nbest=INT):

$ mecab -N2

Override path to the config (-r, --rcfile=FILE):

$ mecab -r path/to/config

Output all morphemes (-a, --all-morphs):

$ mecab -a

Print the settings (-P, --dump-config):

$ mecab -P

Output to a file (-o, --output=FILE):

$ mecab -o path/to/file

Show version (-v, --version):

$ mecab -v

Output format can be specified from the command line:

-F, --node-format=STR
-U, --unk-format=STR
-B, --bos-format=STR
-E, --eos-format=STR
-S, --eon-format=STR

Categorize unknown words as STR, don't try to guess (-x, --unk-feature=STR):

mecab -x "未知語"

Installing perl bindings:

$ cd perl
$ perl Makefile.PL
$ make install

It installs:

/usr/lib/perl5/core_perl/perllocal.pod
/usr/local/lib/perl5/site_perl/MeCab.pm
/usr/local/lib/perl5/site_perl/auto/MeCab/.packlist
/usr/local/lib/perl5/site_perl/auto/MeCab/MeCab.so
use strict;
use warnings;
use MeCab;
use Text::CSV;
use utf8;
use Encode;
binmode(STDOUT, 'encoding(UTF-8)');

my $csv = new Text::CSV({auto_diag => 1});

my $m = new MeCab::Model;
my $t = $m->createTagger() or die "failed to create a tagger\n";
my $n = $t->parseToNode($ARGV[0]) or die 'failed to parse the text: ' . $t->what() . "\n";
for (; $n ; $n = $n->{next}) {
    next unless $n->{surface};
    print decode('UTF-8', $n->{surface}), "\n";
    $csv->parse(decode 'UTF-8', $n->{feature}) or die "failed to parse a feature\n";
    my @els = $csv->fields;
    print '  part of speech: ', join(', ', @els[0..3]), "\n";
    print '  conjugation type: ', $els[4], "\n";
    print '  conjugated form: ', $els[5], "\n";
    print '  base form: ', $els[6], "\n";
    print '  reading: ', $els[7], "\n";
    print '  pronunciation: ', $els[8], "\n";
}

Examples:

example.cpp in perl
example_lattice.cpp in perl
dictionary fields
kana forms
kana forms
best kana forms
kana forms using Lattice
best kana forms using Lattice

Documentation:

index (gt)
bindings (gt)
doxygen
format (gt)

MeCab (Wikipedia)
juman (gt)
JUMAN Information

Download links:

mecab
ipa
juman
unidic (gt)

hayashibe.jp:

example.cpp translated into perl. It's meant to be used with ipadic.

new MeCab::Tagger() may throw exceptions:

new MeCab::Tagger() -> MeCabc::new_Tagger()
MeCabc::new_Tagger() -> _wrap_new_Tagger()
_wrap_new_Tagger() -> _wrap_new_Tagger__SWIG_1()
_wrap_new_Tagger__SWIG_1() -> new_MeCab_Tagger()
new_MeCab_Tagger() -> throw MeCab::getLastError()

MeCab::Tagger::parse() may return 0, use MeCab::Tagger::what():

MeCab::Tagger::parse() -> MeCabc::Tagger_parse()
MeCabc::Tagger_parse() -> _wrap_Tagger_parse()
_wrap_Tagger_parse() -> _wrap_Tagger_parse__SWIG_2()
_wrap_Tagger_parse__SWIG_2() -> MeCab::TaggerImpl::parse()
MeCab::TaggerImpl::parse() -> MeCab::TaggerImpl::parse()
MeCab::TaggerImpl::parse() -> return 0

MeCab::Tagger::parseNBest() may return 0, use MeCab::Tagger::what():

MeCab::Tagger::parseNBest() -> MeCabc::Tagger_parseNBest()
MeCabc::Tagger_parseNBest() -> _wrap_Tagger_parseNBest()
_wrap_Tagger_parseNBest() -> MeCab::TaggerImpl::parseNBest()
MeCab::TaggerImpl::parseNBest() -> MeCab::TaggerImpl::parseNBest()
MeCab::TaggerImpl::parseNBest() -> return 0

MeCab::Tagger::parseNBestInit() may return false, use MeCab::Tagger::what():

MeCab::Tagger::parseNBestInit() -> MeCabc::Tagger_parseNBestInit()
MeCabc::Tagger_parseNBestInit() -> _wrap_Tagger_parseNBestInit()
_wrap_Tagger_parseNBestInit() -> MeCab::TaggerImpl::parseNBestInit()
MeCab::TaggerImpl::parseNBestInit() -> MeCab::TaggerImpl::parseNBestInit()
MeCab::TaggerImpl::parseNBestInit() -> return false

MeCab::Tagger::parseToNode() may return 0, use MeCab::Tagger::what():

MeCab::Tagger::parseToNode() -> MeCabc::Tagger_parseToNode()
MeCabc::Tagger_parseToNode() -> _wrap_Tagger_parseToNode()
_wrap_Tagger_parseToNode() -> MeCab::TaggerImpl::parseToNode()
MeCab::TaggerImpl::parseToNode() -> MeCab::TaggerImpl::parseToNode()
MeCab::TaggerImpl::parseToNode() -> return 0

MeCab::Tagger::dictionary_info() may return 0:

MeCab::Tagger::dictionary_info() -> MeCabc::Tagger_dictionary_info()
MeCabc::Tagger_dictionary_info() -> _wrap_Tagger_dictionary_info()
_wrap_Tagger_dictionary_info() -> MeCab::TaggerImpl::dictionary_info()
MeCab::TaggerImpl::dictionary_info() -> MeCab::ModelImpl::dictionary_info()
MeCab::ModelImpl::dictionary_info() -> return ... ? ... : 0

use strict;
use warnings;
use MeCab;
use Text::CSV;
use utf8;
use Encode;
binmode(STDOUT, 'encoding(UTF-8)');

my $csv = new Text::CSV({auto_diag => 1});

use subs ('describe_feature');

sub describe_lines {
    my $lines = shift;
    foreach my $line (split /\n/, $lines) {
        my ($surface, $feature) = split /\t/, $line, 2;
        print "($surface)\n";
        next unless length $feature;
        describe_feature $feature;
    }
}

sub describe_feature {
    my $feature = shift;
    $csv->parse($feature) or die "failed to parse a feature\n";
    my @features = $csv->fields;
    printf "  part of speech: (%s)\n", join ', ', @features[0..3];
    printf "  conjugation type: (%s)\n", $features[4];
    printf "  conjugated form: (%s)\n", $features[5];
    printf "  base form: (%s)\n", $features[6];
    printf "  reading: (%s)\n", $features[7];
    printf "  pronunciation: (%s)\n", $features[8];
}

my $t = new MeCab::Tagger;
my $r = $t->parse($ARGV[0]) or die 'failed to parse the text: ' . $t->what() . "\n";
$r = decode 'UTF-8', $r;
print "result: ($r)\n";
describe_lines $r;

print "\n";
print "-- nbest\n";
$r = $t->parseNBest(3, $ARGV[0]) or die 'failed to parse the text: ' . $t->what() . "\n";
$r = decode 'UTF-8', $r;
print "result: ($r)\n";
describe_lines $r;

print "\n";
$t->parseNBestInit($ARGV[0]) or die 'failed to parse the text: ' . $t->what() . "\n";
for (my $i = 0; $i < 3; $i++) {
    my $r = $t->next;
    $r = decode 'UTF-8', $r;
    print "$i: ($r)\n";
    describe_lines $r;
}

print "\n";
my $n = $t->parseToNode($ARGV[0]) or die 'failed to parse the text: ' . $t->what() . "\n";
for (; $n; $n = $n->{next}) {
    print '$n->{id}: ', $n->{id}, "\n";
    if ($n->{stat} == $MeCab::MECAB_BOS_NODE) {
        print "BOS\n";
    } elsif ($n->{stat} == $MeCab::MECAB_EOS_NODE) {
        print "EOS\n";
    } else {
        print '$n->{surface}: ', decode('UTF-8', $n->{surface}), "\n";
        print '$n->{length}: ', $n->{length}, "\n";
        print '$n->{feature}: ', decode('UTF-8', $n->{feature}), "\n";
        describe_feature decode('UTF-8', $n->{feature});
        print '$n->{rcAttr}: ', $n->{rcAttr}, "\n";
        print '$n->{lcAttr}: ', $n->{lcAttr}, "\n";
        print '$n->{posid}: ', $n->{posid}, "\n";
        print '$n->{char_type}: ', $n->{char_type}, "\n";
        print '$n->{stat}: ', $n->{stat}, "\n";
        print '$n->{isbest}: ', $n->{isbest}, "\n";
        print '$n->{alpha}: ', $n->{alpha}, "\n";
        print '$n->{beta}: ', $n->{beta}, "\n";
        print '$n->{prob}: ', $n->{prob}, "\n";
        print '$n->{cost}: ', $n->{cost}, "\n";
    }
}

print "\n";
print "-- dictionary info\n";
for (my $d = $t->dictionary_info; $d; $d = $d->{next}) {
    print '$d->{filename}: ', $d->{filename}, "\n";
    print '$d->{charset}: ', $d->{charset}, "\n";
    print '$d->{size}: ', $d->{size}, "\n";
    print '$d->{type}: ', $d->{type}, "\n";
    print '$d->{lsize}: ', $d->{lsize}, "\n";
    print '$d->{rsize}: ', $d->{rsize}, "\n";
    print '$d->{version}: ', $d->{version}, "\n";
}
$ perl b.pl 未然形
result: (未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
形	名詞,接尾,一般,*,*,*,形,ガタ,ガタ
EOS
)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ガタ)
  pronunciation: (ガタ)
(EOS)

-- nbest
result: (未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
形	名詞,接尾,一般,*,*,*,形,ガタ,ガタ
EOS
未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
形	名詞,一般,*,*,*,*,形,カタチ,カタチ
EOS
未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
形	名詞,接尾,一般,*,*,*,形,ケイ,ケイ
EOS
)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ガタ)
  pronunciation: (ガタ)
(EOS)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (カタチ)
  pronunciation: (カタチ)
(EOS)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ケイ)
  pronunciation: (ケイ)
(EOS)

0: (未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
形	名詞,接尾,一般,*,*,*,形,ガタ,ガタ
EOS
)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ガタ)
  pronunciation: (ガタ)
(EOS)
1: (未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
形	名詞,一般,*,*,*,*,形,カタチ,カタチ
EOS
)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (カタチ)
  pronunciation: (カタチ)
(EOS)
2: (未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
形	名詞,接尾,一般,*,*,*,形,ケイ,ケイ
EOS
)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ケイ)
  pronunciation: (ケイ)
(EOS)

$n->{id}: 0
BOS
$n->{id}: 4
$n->{surface}: 未然
$n->{length}: 6
$n->{feature}: 名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
$n->{rcAttr}: 1285
$n->{lcAttr}: 1285
$n->{posid}: 38
$n->{char_type}: 2
$n->{stat}: 0
$n->{isbest}: 1
$n->{alpha}: 0
$n->{beta}: 0
$n->{prob}: 0
$n->{cost}: 5336
$n->{id}: 6
$n->{surface}: 形
$n->{length}: 3
$n->{feature}: 名詞,接尾,一般,*,*,*,形,ガタ,ガタ
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ガタ)
  pronunciation: (ガタ)
$n->{rcAttr}: 1298
$n->{lcAttr}: 1298
$n->{posid}: 51
$n->{char_type}: 2
$n->{stat}: 0
$n->{isbest}: 1
$n->{alpha}: 0
$n->{beta}: 0
$n->{prob}: 0
$n->{cost}: 9495
$n->{id}: 11
EOS

-- dictionary info
$d->{filename}: /usr/local/lib/mecab/dic/ipadic/sys.dic
$d->{charset}: utf8
$d->{size}: 392126
$d->{type}: 0
$d->{lsize}: 1316
$d->{rsize}: 1316
$d->{version}: 102

example_lattice.cpp translated into perl. It's meant to be used with ipadic:

new MeCab::Model() may throw exceptions:

new MeCab::Model() -> MeCabc::new_Model()
MeCabc::new_Model() -> _wrap_new_Model()
_wrap_new_Model() -> _wrap_new_Model__SWIG_1()
_wrap_new_Model__SWIG_1() -> MeCab::ModelImpl::new_MeCab_Model()
MeCab::ModelImpl::new_MeCab_Model() -> MeCab::createModel()
MeCab::createModel() -> MeCab::ModelImpl::open()
MeCab::ModelImpl::open() -> setGlobalError(...)
MeCab::ModelImpl::new_MeCab_Model() -> throw MeCab::getLastError()

MeCab::Model::createTagger() may return 0:

MeCab::Model::createTagger() -> MeCabc::Model_createTagger()
MeCabc::Model_createTagger() -> _wrap_Model_createTagger()
_wrap_Model_createTagger() -> MeCab::ModelImpl::createTagger()
MeCab::ModelImpl::createTagger() -> return 0

MeCab::Model::createLattice() may return 0:

MeCab::Model::createLattice() -> MeCabc::Model_createLattice()
MeCabc::Model_createLattice() -> _wrap_Model_createLattice()
_wrap_Model_createLattice() -> MeCab::ModelImpl::createLattice()
MeCab::ModelImpl::createLattice() -> return 0

MeCab::Tagger::parse() may return 0, use MeCab::Tagger::what():

MeCab::Tagger::parse() -> MeCabc::Tagger_parse()
MeCabc::Tagger_parse() -> _wrap_Tagger_parse()
_wrap_Tagger_parse() -> _wrap_Tagger_parse__SWIG_2()
_wrap_Tagger_parse__SWIG_2() -> MeCab::TaggerImpl::parse()
MeCab::TaggerImpl::parse() -> MeCab::TaggerImpl::parse()
MeCab::TaggerImpl::parse() -> return 0

MeCab::Lattice::set_request_type():

MeCab::Lattice::set_request_type() -> MeCabc::Lattice_set_request_type()
MeCabc::Lattice_set_request_type() -> _wrap_Lattice_set_request_type()
_wrap_Lattice_set_request_type() -> MeCab::LatticeImpl::set_request_type()
MeCab::LatticeImpl::set_request_type() -> request_type_ = request_type

MeCab::Lattice::remove_request_type():

MeCab::Lattice::remove_request_type() -> MeCabc::Lattice_remove_request_type()
MeCabc::Lattice_remove_request_type() -> _wrap_Lattice_remove_request_type()
_wrap_Lattice_remove_request_type() -> MeCab::LatticeImpl::remove_request_type()
MeCab::LatticeImpl::remove_request_type() -> request_type_ &= ~request_type

MeCab::Lattice::set_sentence():

MeCab::Lattice::set_sentence() -> MeCabc::Lattice_set_sentence()
MeCabc::Lattice_set_sentence() -> _wrap_Lattice_set_sentence()
_wrap_Lattice_set_sentence() -> MeCab::LatticeImpl::set_sentence()
MeCab::LatticeImpl::set_sentence() -> MeCab::LatticeImpl::set_sentence()
MeCab::LatticeImpl::set_sentence() -> ...

MeCab::Lattice::bos_node():

MeCab::Lattice::bos_node() -> MeCabc::Lattice_bos_node()
MeCabc::Lattice_bos_node() -> _wrap_Lattice_bos_node()
_wrap_Lattice_bos_node() -> MeCab::LatticeImpl::bos_node()
MeCab::LatticeImpl::bos_node() -> return end_nodes_[0]

use strict;
use warnings;
use MeCab;
use Text::CSV;
use utf8;
use Encode;
binmode(STDOUT, 'encoding(UTF-8)');

my $csv = new Text::CSV({auto_diag => 1});

use subs ('describe_feature');

sub describe_lines {
    my $lines = shift;
    foreach my $line (split /\n/, $lines) {
        my ($surface, $feature) = split /\t/, $line, 2;
        print "($surface)\n";
        next unless length $feature;
        describe_feature $feature;
    }
}

sub describe_feature {
    my $feature = shift;
    $csv->parse($feature) or die "failed to parse a feature\n";
    my @features = $csv->fields;
    printf "  part of speech: (%s)\n", join ', ', @features[0..3];
    printf "  conjugation type: (%s)\n", $features[4];
    printf "  conjugated form: (%s)\n", $features[5];
    printf "  base form: (%s)\n", $features[6];
    printf "  reading: (%s)\n", $features[7];
    printf "  pronunciation: (%s)\n", $features[8];
}

my $m = new MeCab::Model();
# my $m = new MeCab::Model('-d /usr/local/lib/mecab/dic/jumandic');
# my $m = new MeCab::Model('-d ../unidic-cwj-202302');
# my $m = new MeCab::Model('-d ../unidic-csj-202302');
my $t = $m->createTagger or die "failed to create a tagger\n";
my $l = $m->createLattice or die "failed to create a lattice\n";
$l->set_sentence($ARGV[0]);
$t->parse($l) or die 'failed to parse a lattice: ' . $t->what() . "\n";
print '$l->toString: (', decode('UTF-8', $l->toString), ")\n";
describe_lines decode 'UTF-8', $l->toString;
for (my $n = $l->bos_node; $n; $n = $n->{next}) {
    print '- iteration', "\n";
    print '$n->{id}: ', $n->{id}, "\n";
    if ($n->{stat} == $MeCab::MECAB_BOS_NODE) {
        print "BOS\n";
    } elsif ($n->{stat} == $MeCab::MECAB_EOS_NODE) {
        print "EOS\n";
    } else {
        print '$n->{surface}: (', decode('UTF-8', $n->{surface}), ")\n";
        print '$n->{length}: ', $n->{length}, "\n";
        print '$n->{feature}: (', decode('UTF-8', $n->{feature}), ")\n";
        describe_feature decode 'UTF-8', $n->{feature};
        print '$n->{rcAttr}: ', $n->{rcAttr}, "\n";
        print '$n->{lcAttr}: ', $n->{lcAttr}, "\n";
        print '$n->{posid}: ', $n->{posid}, "\n";
        print '$n->{char_type}: ', $n->{char_type}, "\n";
        print '$n->{stat}: ', $n->{stat}, "\n";
        print '$n->{isbest}: ', $n->{isbest}, "\n";
        print '$n->{alpha}: ', $n->{alpha}, "\n";
        print '$n->{beta}: ', $n->{beta}, "\n";
        print '$n->{prob}: ', $n->{prob}, "\n";
        print '$n->{cost}: ', $n->{cost}, "\n";
    }
}

print "\n";
print "-- begin_nodes/end_nodes\n";
my $size = $l->size;
for (my $i = 0; $i <= $size; $i++) {
    print "- i: $i\n";
    for (my $b = $l->begin_nodes($i); $b; $b = $b->{bnext}) {
        printf "B[%d]: (%s\t%s)\n", $i, decode('UTF-8', $b->{surface}), decode('UTF-8', $b->{feature});
        describe_feature decode 'UTF-8', $b->{feature};
    }
    for (my $e = $l->end_nodes($i); $e; $e = $e->{enext}) {
        printf "E[%d]: (%s\t%s)\n", $i, decode('UTF-8', $e->{surface}), decode('UTF-8', $e->{feature});
        describe_feature decode 'UTF-8', $e->{feature};
    }
}

print "\n";
print "-- best results\n";
$l->set_request_type($MeCab::MECAB_NBEST);
$l->set_sentence($ARGV[0]);
$t->parse($l) or die 'failed to parse a lattice: ' . $t->what(). "\n";
for (my $i = 0; $i < 10; $i++) {
    print "- i: $i\n";
    print '$l->toString: (', decode('UTF-8', $l->toString), ")\n";
    describe_lines decode 'UTF-8', $l->toString;
    last unless $l->next;
}

print "\n";
print "-- marginal probabilities\n";
$l->remove_request_type($MeCab::MECAB_NBEST);
$l->set_request_type($MeCab::MECAB_MARGINAL_PROB);
$l->set_sentence($ARGV[0]);
$t->parse($l) or die 'failed to parse a lattice: ' . $t->what() . "\n";
print '$l->theta: ', $l->theta, "\n";
for (my $n = $l->bos_node; $n; $n = $n->{next}) {
    print "- iteration\n";
    print '$n->{surface}: (', decode('UTF-8', $n->{surface}), ")\n";
    print '$n->{length}: ', $n->{length}, "\n";
    print '$n->{feature}: (', decode('UTF-8', $n->{feature}), ")\n";
    describe_feature decode 'UTF-8', $n->{feature};
    print '$n->{prob}: ', $n->{prob}, "\n";
}

print "\n";
print "-- dictionary info\n";
for (my $d = $m->dictionary_info; $d; $d = $d->{next}) {
    print "- iteration\n";
    print '$d->{filename}: (', $d->{filename}, ")\n";
    print '$d->{charset}: (', $d->{charset}, ")\n";
    print '$d->{size}: ', $d->{size}, "\n";
    print '$d->{type}: ', $d->{type}, "\n";
    print '$d->{lsize}: ', $d->{lsize}, "\n";
    print '$d->{rsize}: ', $d->{rsize}, "\n";
    print '$d->{version}: ', $d->{version}, "\n";
}

# print "\n";
# print "-- swap a model atomically\n";
# my $m2 = new MeCab::Model;
# $m->swap($m2);  # causes SIGSEGV
$ perl c.pl 未然形
$l->toString: (未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
形	名詞,接尾,一般,*,*,*,形,ガタ,ガタ
EOS
)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ガタ)
  pronunciation: (ガタ)
(EOS)
- iteration
$n->{id}: 0
BOS
- iteration
$n->{id}: 4
$n->{surface}: (未然)
$n->{length}: 6
$n->{feature}: (名詞,一般,*,*,*,*,未然,ミゼン,ミゼン)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
$n->{rcAttr}: 1285
$n->{lcAttr}: 1285
$n->{posid}: 38
$n->{char_type}: 2
$n->{stat}: 0
$n->{isbest}: 1
$n->{alpha}: 0
$n->{beta}: 0
$n->{prob}: 0
$n->{cost}: 5336
- iteration
$n->{id}: 6
$n->{surface}: (形)
$n->{length}: 3
$n->{feature}: (名詞,接尾,一般,*,*,*,形,ガタ,ガタ)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ガタ)
  pronunciation: (ガタ)
$n->{rcAttr}: 1298
$n->{lcAttr}: 1298
$n->{posid}: 51
$n->{char_type}: 2
$n->{stat}: 0
$n->{isbest}: 1
$n->{alpha}: 0
$n->{beta}: 0
$n->{prob}: 0
$n->{cost}: 9495
- iteration
$n->{id}: 11
EOS

-- begin_nodes/end_nodes
- i: 0
B[0]: (未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
B[0]: (未	接頭詞,名詞接続,*,*,*,*,未,ミ,ミ)
  part of speech: (接頭詞, 名詞接続, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未)
  reading: (ミ)
  pronunciation: (ミ)
B[0]: (未	名詞,一般,*,*,*,*,未,ヒツジ,ヒツジ)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未)
  reading: (ヒツジ)
  pronunciation: (ヒツジ)
B[0]: (未	名詞,固有名詞,地域,一般,*,*,未,ヒツジ,ヒツジ)
  part of speech: (名詞, 固有名詞, 地域, 一般)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未)
  reading: (ヒツジ)
  pronunciation: (ヒツジ)
E[0]: (	BOS/EOS,*,*,*,*,*,*,*,*)
  part of speech: (BOS/EOS, *, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (*)
  reading: (*)
  pronunciation: (*)
- i: 1
- i: 2
- i: 3
B[3]: (然	副詞,一般,*,*,*,*,然,シカ,シカ)
  part of speech: (副詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (然)
  reading: (シカ)
  pronunciation: (シカ)
E[3]: (未	名詞,固有名詞,地域,一般,*,*,未,ヒツジ,ヒツジ)
  part of speech: (名詞, 固有名詞, 地域, 一般)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未)
  reading: (ヒツジ)
  pronunciation: (ヒツジ)
E[3]: (未	名詞,一般,*,*,*,*,未,ヒツジ,ヒツジ)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未)
  reading: (ヒツジ)
  pronunciation: (ヒツジ)
E[3]: (未	接頭詞,名詞接続,*,*,*,*,未,ミ,ミ)
  part of speech: (接頭詞, 名詞接続, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未)
  reading: (ミ)
  pronunciation: (ミ)
- i: 4
- i: 5
- i: 6
B[6]: (形	名詞,一般,*,*,*,*,形,ナリ,ナリ)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ナリ)
  pronunciation: (ナリ)
B[6]: (形	名詞,一般,*,*,*,*,形,カタ,カタ)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (カタ)
  pronunciation: (カタ)
B[6]: (形	名詞,一般,*,*,*,*,形,カタチ,カタチ)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (カタチ)
  pronunciation: (カタチ)
B[6]: (形	名詞,接尾,一般,*,*,*,形,ケイ,ケイ)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ケイ)
  pronunciation: (ケイ)
B[6]: (形	名詞,接尾,一般,*,*,*,形,ガタ,ガタ)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ガタ)
  pronunciation: (ガタ)
E[6]: (然	副詞,一般,*,*,*,*,然,シカ,シカ)
  part of speech: (副詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (然)
  reading: (シカ)
  pronunciation: (シカ)
E[6]: (未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
- i: 7
- i: 8
- i: 9
B[9]: (	BOS/EOS,*,*,*,*,*,*,*,*)
  part of speech: (BOS/EOS, *, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (*)
  reading: (*)
  pronunciation: (*)
E[9]: (	BOS/EOS,*,*,*,*,*,*,*,*)
  part of speech: (BOS/EOS, *, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (*)
  reading: (*)
  pronunciation: (*)
E[9]: (形	名詞,接尾,一般,*,*,*,形,ガタ,ガタ)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ガタ)
  pronunciation: (ガタ)
E[9]: (形	名詞,接尾,一般,*,*,*,形,ケイ,ケイ)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ケイ)
  pronunciation: (ケイ)
E[9]: (形	名詞,一般,*,*,*,*,形,カタチ,カタチ)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (カタチ)
  pronunciation: (カタチ)
E[9]: (形	名詞,一般,*,*,*,*,形,カタ,カタ)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (カタ)
  pronunciation: (カタ)
E[9]: (形	名詞,一般,*,*,*,*,形,ナリ,ナリ)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ナリ)
  pronunciation: (ナリ)

-- best results
- i: 0
$l->toString: (未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
形	名詞,接尾,一般,*,*,*,形,ガタ,ガタ
EOS
)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ガタ)
  pronunciation: (ガタ)
(EOS)
- i: 1
$l->toString: (未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
形	名詞,接尾,一般,*,*,*,形,ガタ,ガタ
EOS
)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ガタ)
  pronunciation: (ガタ)
(EOS)
- i: 2
$l->toString: (未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
形	名詞,一般,*,*,*,*,形,カタチ,カタチ
EOS
)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (カタチ)
  pronunciation: (カタチ)
(EOS)
- i: 3
$l->toString: (未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
形	名詞,接尾,一般,*,*,*,形,ケイ,ケイ
EOS
)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ケイ)
  pronunciation: (ケイ)
(EOS)
- i: 4
$l->toString: (未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
形	名詞,一般,*,*,*,*,形,ナリ,ナリ
EOS
)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ナリ)
  pronunciation: (ナリ)
(EOS)
- i: 5
$l->toString: (未然	名詞,一般,*,*,*,*,未然,ミゼン,ミゼン
形	名詞,一般,*,*,*,*,形,カタ,カタ
EOS
)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (カタ)
  pronunciation: (カタ)
(EOS)
- i: 6
$l->toString: (未	接頭詞,名詞接続,*,*,*,*,未,ミ,ミ
然	副詞,一般,*,*,*,*,然,シカ,シカ
形	名詞,一般,*,*,*,*,形,カタチ,カタチ
EOS
)
(未)
  part of speech: (接頭詞, 名詞接続, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未)
  reading: (ミ)
  pronunciation: (ミ)
(然)
  part of speech: (副詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (然)
  reading: (シカ)
  pronunciation: (シカ)
(形)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (カタチ)
  pronunciation: (カタチ)
(EOS)
- i: 7
$l->toString: (未	名詞,一般,*,*,*,*,未,ヒツジ,ヒツジ
然	副詞,一般,*,*,*,*,然,シカ,シカ
形	名詞,一般,*,*,*,*,形,カタチ,カタチ
EOS
)
(未)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未)
  reading: (ヒツジ)
  pronunciation: (ヒツジ)
(然)
  part of speech: (副詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (然)
  reading: (シカ)
  pronunciation: (シカ)
(形)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (カタチ)
  pronunciation: (カタチ)
(EOS)
- i: 8
$l->toString: (未	接頭詞,名詞接続,*,*,*,*,未,ミ,ミ
然	副詞,一般,*,*,*,*,然,シカ,シカ
形	名詞,接尾,一般,*,*,*,形,ガタ,ガタ
EOS
)
(未)
  part of speech: (接頭詞, 名詞接続, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未)
  reading: (ミ)
  pronunciation: (ミ)
(然)
  part of speech: (副詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (然)
  reading: (シカ)
  pronunciation: (シカ)
(形)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ガタ)
  pronunciation: (ガタ)
(EOS)
- i: 9
$l->toString: (未	接頭詞,名詞接続,*,*,*,*,未,ミ,ミ
然	副詞,一般,*,*,*,*,然,シカ,シカ
形	名詞,一般,*,*,*,*,形,ナリ,ナリ
EOS
)
(未)
  part of speech: (接頭詞, 名詞接続, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未)
  reading: (ミ)
  pronunciation: (ミ)
(然)
  part of speech: (副詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (然)
  reading: (シカ)
  pronunciation: (シカ)
(形)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ナリ)
  pronunciation: (ナリ)
(EOS)

-- marginal probabilities
$l->theta: 0.75
- iteration
$n->{surface}: ()
$n->{length}: 0
$n->{feature}: (BOS/EOS,*,*,*,*,*,*,*,*)
  part of speech: (BOS/EOS, *, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (*)
  reading: (*)
  pronunciation: (*)
$n->{prob}: 0
- iteration
$n->{surface}: (未然)
$n->{length}: 6
$n->{feature}: (名詞,一般,*,*,*,*,未然,ミゼン,ミゼン)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
$n->{prob}: 1
- iteration
$n->{surface}: (形)
$n->{length}: 3
$n->{feature}: (名詞,接尾,一般,*,*,*,形,ガタ,ガタ)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ガタ)
  pronunciation: (ガタ)
$n->{prob}: 1
- iteration
$n->{surface}: ()
$n->{length}: 0
$n->{feature}: (BOS/EOS,*,*,*,*,*,*,*,*)
  part of speech: (BOS/EOS, *, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (*)
  reading: (*)
  pronunciation: (*)
$n->{prob}: 1

-- dictionary info
- iteration
$d->{filename}: (/usr/local/lib/mecab/dic/ipadic/sys.dic)
$d->{charset}: (utf8)
$d->{size}: 392126
$d->{type}: 0
$d->{lsize}: 1316
$d->{rsize}: 1316
$d->{version}: 102

Display dictionary fields:

d.pl:

use strict;
use warnings;
use MeCab;
use Text::CSV;
use utf8;
use Encode;
binmode(STDOUT, 'encoding(UTF-8)');

my $csv = new Text::CSV({auto_diag => 1});

sub describe_lines {
    my ($dict, $lines) = @_;
    foreach my $line (split /\n/, $lines) {
        my ($surface, $feature) = split /\t/, $line, 2;
        print "($surface)\n";
        next unless length $feature;
        __PACKAGE__->can('describe_feature_' . $dict)->($feature);
    }
}

# https://taku910.github.io/mecab/#parse
# https://hayashibe.jp/tr/mecab/dictionary/ipadic
# https://hayashibe-jp.translate.goog/tr/mecab/dictionary/ipadic?_x_tr_sl=ja&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=sc
sub describe_feature_ipa {
    my $feature = shift;
    $csv->parse($feature) or die "failed to parse a feature\n";
    my @features = $csv->fields;
    printf "  part of speech: (%s)\n", join ', ', @features[0..3];
    printf "  conjugation type: (%s)\n", $features[4];
    printf "  conjugated form: (%s)\n", $features[5];
    printf "  base form: (%s)\n", $features[6];
    printf "  reading: (%s)\n", $features[7];
    printf "  pronunciation: (%s)\n", $features[8];
}

print "-- ipa\n";
print "fields:\n";
print "  part of speech (品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3)\n";
print "  conjugation type (活用型 ctype)\n";
print "  conjugated form (活用形 cform)\n";
print "  base form (原形)\n";
print "  reading (読み)\n";
print "  pronunciation (発音)\n";

my $m = new MeCab::Model;
my $t = $m->createTagger() or die "failed to create a tagger\n";
my $r = $t->parse($ARGV[0]) or die 'failed to parse the text: ' . $t->what() . "\n";
describe_lines 'ipa', decode 'UTF-8', $r;

# https://hayashibe.jp/tr/mecab/dictionary/juman
# https://hayashibe-jp.translate.goog/tr/mecab/dictionary/juman?_x_tr_sl=ja&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=sc
# https://github.com/taku910/mecab/blob/05481e751dd5aa536a2bace46715ce54568b972a/mecab-jumandic/script/juman2mecab.pl#L66-L67
# https://github.com/taku910/mecab/blob/05481e751dd5aa536a2bace46715ce54568b972a/mecab-jumandic/script/juman2mecab.pl#L100
sub describe_feature_juman {
    my $feature = shift;
    my @features = split /,/, $feature;
    printf "  part of speech: (%s)\n", join ', ', @features[0..1];
    printf "  conjugation type: (%s)\n", $features[2];
    printf "  conjugated form: (%s)\n", $features[3];
    printf "  headword: (%s)\n", $features[4];
    printf "  reading: (%s)\n", $features[5];
    printf "  semantic information and standard representation: (%s)\n", $features[6];
}

print "\n";
print "-- juman\n";
print "fields:\n";
print "  part of speech (品詞, 品詞細分類)\n";
print "  conjugation type (活用型 ctype)\n";
print "  conjugated form (活用形 cform)\n";
print "  headword (見出し語)\n";
print "  reading (読み)\n";
print "  semantic information and standard representation (意味情報・代表表記)\n";
print "if ctype is set, cform is set and headword might differ from the surface\n";

$m = new MeCab::Model('-d /usr/local/lib/mecab/dic/jumandic');
$t = $m->createTagger() or die "failed to create a tagger\n";
$r = $t->parse($ARGV[0]) or die 'failed to parse the text: ' . $t->what() . "\n";
describe_lines 'juman', decode 'UTF-8', $r;

# https://hayashibe.jp/tr/mecab/dictionary/unidic/field
# https://hayashibe-jp.translate.goog/tr/mecab/dictionary/unidic/field?_x_tr_sl=ja&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=sc
sub describe_feature_unidic {
    my $feature = shift;
    my @features = split /,/, $feature;
    printf "  part of speech: (%s)\n", join ', ', @features[0..3];
    printf "  conjugation type: (%s)\n", $features[4];
    printf "  conjugated form: (%s)\n", $features[5];
    printf "  lexical form reading: (%s)\n", $features[6];
    printf "  lemma: (%s)\n", $features[7];
    printf "  orthographic surface form: (%s)\n", $features[8];
    printf "  pronunciation surface form: (%s)\n", $features[9];
    printf "  orthographic base form: (%s)\n", $features[10];
    printf "  pronunciation: base form: (%s)\n", $features[11];
    printf "  word class: (%s)\n", $features[12];
    printf "  initial inflection type: (%s)\n", $features[13];
    printf "  initial inflection form: (%s)\n", $features[14];
    printf "  final inflection type: (%s)\n", $features[15];
    printf "  final inflection form: (%s)\n", $features[16];
    printf "  initial inflection connection type: (%s)\n", $features[17];
    printf "  final inflection connection type: (%s)\n", $features[18];
    printf "  lexical type: (%s)\n", $features[19];
    printf "  kana surface form: (%s)\n", $features[20];
    printf "  kana base form: (%s)\n", $features[21];
    printf "  word form: (%s)\n", $features[22];
    printf "  word base form: (%s)\n", $features[23];
    printf "  accent type: (%s)\n", $features[24];
    printf "  accent connection type: (%s)\n", $features[25];
    printf "  accent modification type: (%s)\n", $features[26];
    printf "  lexicon id: %s\n", $features[27];
    printf "  lemma id: %s\n", $features[28];
}

print "\n";
print "-- unidic\n";
print "fields:\n";
print "  part of speech (品詞大分類 pos1, 品詞中分類 pos2, 品詞小分類 pos3, 品詞細分類 pos4)\n";
print "  conjugation type (活用型 ctype)\n";
print "  conjugated form (活用形 cform)\n";
print "  lexical form reading (語彙素読み lform)\n";
print "  lemma (語彙素表記 lemma)\n";
print "  orthographic surface form (書字形出現形 orth)\n";
print "  pronunciation surface form (発音形出現形 pron)\n";
print "  orthographic base form (書字形基本形 orthbase)\n";
print "  pronunciation base form (発音形基本形 pronbase)\n";
print "  word class (語種 goshu)\n";
print "  initial inflection type (語頭変化型 itype)\n";
print "  initial inflection form (語頭変化形 iform)\n";
print "  final inflection type (語末変化型 ftype)\n";
print "  final inflection form (語末変化形 fform)\n";
print "  initial inflection connection type (語頭変化結合形 icontype)\n";
print "  final inflection connection type (語末変化結合形 fcontype)\n";
print "  lexical type (語彙素類 ltype)\n";
print "  kana surface form (仮名形出現形 kana)\n";
print "  kana base form (仮名形基本形 kanabase)\n";
print "  word form (語形 form)\n";
print "  word base form (語形基本形 formbase)\n";
print "  accent type (アクセント型 atype)\n";
print "  accent connection type (アクセント結合型 acontype)\n";
print "  accent modification type (アクセント修飾型 amodtype)\n";
print "  lexicon id (語彙表ID lidtype)\n";
print "  lemma id (語彙素ID lemma_id)\n";

print "\n";
print "-- unidic-cwj\n";
$m = new MeCab::Model('-d ../unidic-cwj-202302');
$t = $m->createTagger() or die "failed to create a tagger\n";
$r = $t->parse($ARGV[0]) or die 'failed to parse the text: ' . $t->what() . "\n";
describe_lines 'unidic', decode 'UTF-8', $r;

print "\n";
print "-- unidic-csj\n";
$m = new MeCab::Model('-d ../unidic-csj-202302');
$t = $m->createTagger() or die "failed to create a tagger\n";
$r = $t->parse($ARGV[0]) or die 'failed to parse the text: ' . $t->what() . "\n";
describe_lines 'unidic', decode 'UTF-8', $r;
$ perl d.pl 未然形
-- ipa
fields:
  part of speech (品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3)
  conjugation type (活用型 ctype)
  conjugated form (活用形 cform)
  base form (原形)
  reading (読み)
  pronunciation (発音)
(未然)
  part of speech: (名詞, 一般, *, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (未然)
  reading: (ミゼン)
  pronunciation: (ミゼン)
(形)
  part of speech: (名詞, 接尾, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  base form: (形)
  reading: (ガタ)
  pronunciation: (ガタ)
(EOS)

-- juman
fields:
  part of speech (品詞, 品詞細分類)
  conjugation type (活用型 ctype)
  conjugated form (活用形 cform)
  headword (見出し語)
  reading (読み)
  semantic information and standard representation (意味情報・代表表記)
if ctype is set, cform is set and headword might differ from the surface
(未然)
  part of speech: (名詞, 普通名詞)
  conjugation type: (*)
  conjugated form: (*)
  headword: (未然)
  reading: (みぜん)
  semantic information and standard representation: (代表表記:未然/みぜん カテゴリ:抽象物)
(形)
  part of speech: (接尾辞, 名詞性名詞接尾辞)
  conjugation type: (*)
  conjugated form: (*)
  headword: (形)
  reading: (かた)
  semantic information and standard representation: (代表表記:型/かた 内容語 カテゴリ:形・模様;抽象物)
(EOS)

-- unidic
fields:
  part of speech (品詞大分類 pos1, 品詞中分類 pos2, 品詞小分類 pos3, 品詞細分類 pos4)
  conjugation type (活用型 ctype)
  conjugated form (活用形 cform)
  lexical form reading (語彙素読み lform)
  lemma (語彙素表記 lemma)
  orthographic surface form (書字形出現形 orth)
  pronunciation surface form (発音形出現形 pron)
  orthographic base form (書字形基本形 orthbase)
  pronunciation base form (発音形基本形 pronbase)
  word class (語種 goshu)
  initial inflection type (語頭変化型 itype)
  initial inflection form (語頭変化形 iform)
  final inflection type (語末変化型 ftype)
  final inflection form (語末変化形 fform)
  initial inflection connection type (語頭変化結合形 icontype)
  final inflection connection type (語末変化結合形 fcontype)
  lexical type (語彙素類 ltype)
  kana surface form (仮名形出現形 kana)
  kana base form (仮名形基本形 kanabase)
  word form (語形 form)
  word base form (語形基本形 formbase)
  accent type (アクセント型 atype)
  accent connection type (アクセント結合型 acontype)
  accent modification type (アクセント修飾型 amodtype)
  lexicon id (語彙表ID lidtype)
  lemma id (語彙素ID lemma_id)

-- unidic-cwj
(未然)
  part of speech: (名詞, 普通名詞, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  lexical form reading: (ミゼン)
  lemma: (未然)
  orthographic surface form: (未然)
  pronunciation surface form: (ミゼン)
  orthographic base form: (未然)
  pronunciation: base form: (ミゼン)
  word class: (漢)
  initial inflection type: (*)
  initial inflection form: (*)
  final inflection type: (*)
  final inflection form: (*)
  initial inflection connection type: (*)
  final inflection connection type: (*)
  lexical type: (体)
  kana surface form: (ミゼン)
  kana base form: (ミゼン)
  word form: (ミゼン)
  word base form: (ミゼン)
  accent type: (0)
  accent connection type: (C2)
  accent modification type: (*)
  lexicon id: 10028104424759808
  lemma id: 36482
(形)
  part of speech: (接尾辞, 名詞的, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  lexical form reading: (ケイ)
  lemma: (形)
  orthographic surface form: (形)
  pronunciation surface form: (ケー)
  orthographic base form: (形)
  pronunciation: base form: (ケー)
  word class: (漢)
  initial inflection type: (*)
  initial inflection form: (*)
  final inflection type: (*)
  final inflection form: (*)
  initial inflection connection type: (*)
  final inflection connection type: (*)
  lexical type: (接尾体)
  kana surface form: (ケイ)
  kana base form: (ケイ)
  word form: (ケイ)
  word base form: (ケイ)
  accent type: (*)
  accent connection type: (C4)
  accent modification type: (*)
  lexicon id: 2989305861644800
  lemma id: 10875
(EOS)

-- unidic-csj
(未然)
  part of speech: (名詞, 普通名詞, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  lexical form reading: (ミゼン)
  lemma: (未然)
  orthographic surface form: (未然)
  pronunciation surface form: (ミゼン)
  orthographic base form: (未然)
  pronunciation: base form: (ミゼン)
  word class: (漢)
  initial inflection type: (*)
  initial inflection form: (*)
  final inflection type: (*)
  final inflection form: (*)
  initial inflection connection type: (*)
  final inflection connection type: (*)
  lexical type: (体)
  kana surface form: (ミゼン)
  kana base form: (ミゼン)
  word form: (ミゼン)
  word base form: (ミゼン)
  accent type: (0)
  accent connection type: (C2)
  accent modification type: (*)
  lexicon id: 10028104424759808
  lemma id: 36482
(形)
  part of speech: (接尾辞, 名詞的, 一般, *)
  conjugation type: (*)
  conjugated form: (*)
  lexical form reading: (ケイ)
  lemma: (形)
  orthographic surface form: (形)
  pronunciation surface form: (ケー)
  orthographic base form: (形)
  pronunciation: base form: (ケー)
  word class: (漢)
  initial inflection type: (*)
  initial inflection form: (*)
  final inflection type: (*)
  final inflection form: (*)
  initial inflection connection type: (*)
  final inflection connection type: (*)
  lexical type: (接尾体)
  kana surface form: (ケイ)
  kana base form: (ケイ)
  word form: (ケイ)
  word base form: (ケイ)
  accent type: (*)
  accent connection type: (C4)
  accent modification type: (*)
  lexicon id: 2989305861644800
  lemma id: 10875
(EOS)

Display the kana forms:

e.pl:

use strict;
use warnings;
use MeCab;
use Text::CSV;
use utf8;
use Encode;
binmode(STDOUT, 'encoding(UTF-8)');

my $csv = new Text::CSV({auto_diag => 1});

use subs ('parse_lines', 'parse_line');

sub process_dict {
    my ($args, $dict, $i_kana) = @_;
    my $m = new MeCab::Model($args);
    my $t = $m->createTagger() or die "failed to create a tagger\n";
    my $r = $t->parse($ARGV[0]) or die 'failed to parse the text: ' . $t->what() . "\n";
    print "\n";
    print "-- $dict\n";
    foreach my $line (parse_lines decode 'UTF-8', $r) {
        print $$line[0], ': ', $$line[$i_kana], "\n";
    }
}

sub parse_lines {
    my $lines = shift;
    grep scalar @$_,
        map [parse_line $_], split /\n/, $lines;
}

sub parse_line {
    my $line = shift;
    my ($surface, $feature) = split /\t/, $line, 2;
    return unless length $feature;
    $csv->parse($feature) or die "failed to parse a feature\n";
    return $surface, $csv->fields;
}

process_dict '', 'ipa', 9;
process_dict '-d /usr/local/lib/mecab/dic/jumandic', 'juman', 6;
process_dict '-d ../unidic-cwj-202302', 'unidic-cwj', 21;
process_dict '-d ../unidic-cwj-202302', 'unidic-csj', 21;
$ perl e.pl 未然形

-- ipa
未然: ミゼン
形: ガタ

-- juman
未然: みぜん
形: かた

-- unidic-cwj
未然: ミゼン
形: ケイ

-- unidic-csj
未然: ミゼン
形: ケイ

Display the kana forms:

f.pl:

use strict;
use warnings;
use MeCab;
use Text::CSV;
use utf8;
use Encode;
binmode(STDOUT, 'encoding(UTF-8)');

my $csv = new Text::CSV({auto_diag => 1});

use subs ('parse_feature');

sub process_dict {
    my ($args, $dict, $i_kana) = @_;
    my $m = new MeCab::Model($args);
    my $t = $m->createTagger() or die "failed to create a tagger\n";
    my $n = $t->parseToNode($ARGV[0]) or die 'failed to parse the text: ' . $t->what() . "\n";
    print "\n";
    print "-- $dict\n";
    for (; $n; $n = $n->{next}) {
        if ($n->{stat} != $MeCab::MECAB_BOS_NODE && $n->{stat} != $MeCab::MECAB_EOS_NODE) {
            my @feature = parse_feature decode 'UTF-8', $n->{feature};
            print decode('UTF-8', $n->{surface}), ': ', $feature[$i_kana], "\n";
        }
    }
}

sub parse_feature {
    my $feature = shift;
    $csv->parse($feature) or die "failed to parse a feature\n";
    return $csv->fields;
}

process_dict '', 'ipa', 8;
process_dict '-d /usr/local/lib/mecab/dic/jumandic', 'juman', 5;
process_dict '-d ../unidic-cwj-202302', 'unidic-cwj', 20;
process_dict '-d ../unidic-cwj-202302', 'unidic-csj', 20;
$ perl f.pl 未然形

-- ipa
未然: ミゼン
形: ガタ

-- juman
未然: みぜん
形: かた

-- unidic-cwj
未然: ミゼン
形: ケイ

-- unidic-csj
未然: ミゼン
形: ケイ

Display the best kana forms (MeCab::Tagger::parseNBest() is deprecated):

f.pl:

use strict;
use warnings;
use MeCab;
use Text::CSV;
use utf8;
use Encode;
binmode(STDOUT, 'encoding(UTF-8)');

my $csv = new Text::CSV({auto_diag => 1});

use subs ('parse_lines', 'parse_line');

sub process_dict {
    my ($args, $dict, $i_kana) = @_;
    my $m = new MeCab::Model($args);
    my $t = $m->createTagger() or die "failed to create a tagger\n";
    my $r = $t->parseNBest(3, $ARGV[0]) or die 'failed to parse the text: ' . $t->what() . "\n";
    print "\n";
    print "-- $dict\n";
    foreach my $line (parse_lines decode 'UTF-8', $r) {
        print $$line[0], ': ', $$line[$i_kana], "\n";
    }
}

sub parse_lines {
    my $lines = shift;
    grep scalar @$_,
        map [parse_line $_], split /\n/, $lines;
}

sub parse_line {
    my $line = shift;
    my ($surface, $feature) = split /\t/, $line, 2;
    return unless length $feature;
    $csv->parse($feature) or die "failed to parse a feature\n";
    return $surface, $csv->fields;
}

process_dict '', 'ipa', 9;
process_dict '-d /usr/local/lib/mecab/dic/jumandic', 'juman', 6;
process_dict '-d ../unidic-cwj-202302', 'unidic-cwj', 21;
process_dict '-d ../unidic-cwj-202302', 'unidic-csj', 21;
$ perl g.pl 未然形

-- ipa
未然: ミゼン
形: ガタ
未然: ミゼン
形: カタチ
未然: ミゼン
形: ケイ

-- juman
未然: みぜん
形: けい
未然: みぜん
形: かた
未然: みぜん
形: かたち

-- unidic-cwj
未然: ミゼン
形: ケイ
未然: ミゼン
形: ガタ
未然: ミゼン
形: カタチ

-- unidic-csj
未然: ミゼン
形: ケイ
未然: ミゼン
形: ガタ
未然: ミゼン
形: カタチ

Display the kana forms using Lattice:

g.pl:

use strict;
use warnings;
use MeCab;
use Text::CSV;
use utf8;
use Encode;
binmode(STDOUT, 'encoding(UTF-8)');

my $csv = new Text::CSV({auto_diag => 1});

use subs ('parse_feature');

sub process_dict {
    my ($args, $dict, $i_kana) = @_;
    my $m = new MeCab::Model($args);
    my $t = $m->createTagger() or die "failed to create a tagger\n";
    my $l = $m->createLattice() or die "failed to create a lattice\n";
    $l->set_sentence($ARGV[0]);
    $t->parse($l) or die 'failed to parse a lattice: ' . $t->what() . "\n";
    print "\n";
    print "-- $dict\n";
    my $n = $l->bos_node;
    for (; $n; $n = $n->{next}) {
        if ($n->{stat} != $MeCab::MECAB_BOS_NODE && $n->{stat} != $MeCab::MECAB_EOS_NODE) {
            my @feature = parse_feature decode 'UTF-8', $n->{feature};
            print decode('UTF-8', $n->{surface}), ': ', $feature[$i_kana], "\n";
        }
    }
}

sub parse_feature {
    my $feature = shift;
    $csv->parse($feature) or die "failed to parse a feature\n";
    return $csv->fields;
}

process_dict '', 'ipa', 8;
process_dict '-d /usr/local/lib/mecab/dic/jumandic', 'juman', 5;
process_dict '-d ../unidic-cwj-202302', 'unidic-cwj', 20;
process_dict '-d ../unidic-cwj-202302', 'unidic-csj', 20;
$ perl h.pl 未然形

-- ipa
未然: ミゼン
形: ガタ

-- juman
未然: みぜん
形: かた

-- unidic-cwj
未然: ミゼン
形: ケイ

-- unidic-csj
未然: ミゼン
形: ケイ

Display best kana forms using Lattice:

h.pl:

use strict;
use warnings;
use MeCab;
use Text::CSV;
use utf8;
use Encode;
binmode(STDOUT, 'encoding(UTF-8)');

my $csv = new Text::CSV({auto_diag => 1});

use subs ('parse_lines', 'parse_line');

sub process_dict {
    my ($args, $dict, $i_kana) = @_;
    my $m = new MeCab::Model($args);
    my $t = $m->createTagger() or die "failed to create a tagger\n";
    my $l = $m->createLattice() or die "failed to create a lattice\n";
    $l->set_request_type($MeCab::MECAB_NBEST);
    $l->set_sentence($ARGV[0]);
    $t->parse($l) or die 'failed to parse a lattice: ' . $t->what() . "\n";
    print "\n";
    print "-- $dict\n";
    for (my $i = 0; $i < 3; $i++) {
        print "- i: $i\n";
        foreach my $line (parse_lines decode 'UTF-8', $l->toString) {
            print $$line[0], ': ', $$line[$i_kana], "\n";
        };
        last unless $l->next;
    }
}

sub parse_lines {
    my $lines = shift;
    grep scalar @$_,
        map [parse_line $_], split /\n/, $lines;
}

sub parse_line {
    my $line = shift;
    my ($surface, $feature) = split /\t/, $line, 2;
    return unless length $feature;
    $csv->parse($feature) or die "failed to parse a feature\n";
    return $surface, $csv->fields;
}

process_dict '', 'ipa', 9;
process_dict '-d /usr/local/lib/mecab/dic/jumandic', 'juman', 6;
process_dict '-d ../unidic-cwj-202302', 'unidic-cwj', 21;
process_dict '-d ../unidic-cwj-202302', 'unidic-csj', 21;
$ perl i.pl 未然形

-- ipa
- i: 0
未然: ミゼン
形: ガタ
- i: 1
未然: ミゼン
形: ガタ
- i: 2
未然: ミゼン
形: カタチ

-- juman
- i: 0
未然: みぜん
形: かた
- i: 1
未然: みぜん
形: けい
- i: 2
未然: みぜん
形: かた

-- unidic-cwj
- i: 0
未然: ミゼン
形: ケイ
- i: 1
未然: ミゼン
形: ケイ
- i: 2
未然: ミゼン
形: ガタ

-- unidic-csj
- i: 0
未然: ミゼン
形: ケイ
- i: 1
未然: ミゼン
形: ケイ
- i: 2
未然: ミゼン
形: ガタ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment