Cambridge SMT System
main.hifst.hpp
Go to the documentation of this file.
1 // Licensed under the Apache License, Version 2.0 (the "License");
2 // you may not use these files except in compliance with the License.
3 // You may obtain a copy of the License at
4 //
5 // http://www.apache.org/licenses/LICENSE-2.0
6 //
7 // Unless required by applicable law or agreed to in writing, software
8 // distributed under the License is distributed on an "AS IS" BASIS,
9 // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
10 // See the License for the specific language governing permissions and
11 // limitations under the License.
12 
13 // Copyright 2012 - Gonzalo Iglesias, AdriĆ  de Gispert, William Byrne
14 
21 #ifndef HIFST_H
22 #define HIFST_H
23 
24 namespace ucam {
25 namespace util {
26 
27 extern bool user_check_ok;
28 extern const bool detailed;
29 
30 }
31 } // end namespaces
32 
34 
35 #include "global_incls.hpp"
36 #include "custom_assert.hpp"
37 #include "global_decls.hpp"
38 #include "global_funcs.hpp"
39 
40 #include <fst/fstlib.h>
41 #include <fst/script/print.h>
42 #include <fst/extensions/pdt/pdtlib.h>
43 
44 #include "logger.hpp"
45 #include "szfstream.hpp"
46 
47 #include "registrypo.hpp"
48 #include "taskinterface.hpp"
49 #include "range.hpp"
50 #include "addresshandler.hpp"
52 
53 #include <constants-fsttools.hpp>
54 #include <constants-hifst.hpp>
55 
57 
58 #include "params.hpp"
59 
60 #include "tokenizer.osr.hpp"
61 
62 //KenLM
63 #define KENLM_MAX_ORDER 6
64 #include "lm/model.hh"
65 
67 //#include "lexicographic-tropical-tropical-decls.h"
70 #include <tropical-sparse-tuple-weight.h>
71 #include <tropical-sparse-tuple-weight-decls.h>
73 
74 #include "fstio.hpp"
75 #include "fstutils.hpp"
78 #include "fstutils.mapper.hpp"
79 #include "fstutils.multiunion.hpp"
80 #include "fstutils.ftcompose.hpp"
81 
82 #include "params.hpp"
83 
84 #include <defs.grammar.hpp>
85 #include <defs.ssgrammar.hpp>
86 #include <defs.cykparser.hpp>
87 
88 #include <data.stats.hpp>
89 #include <data.grammar.hpp>
90 #include <data.lm.hpp>
91 #include <data.ssgrammar.hpp>
92 #include <data.cykparser.hpp>
93 
95 #include <data-main.hifst.hpp>
96 
97 #include <task.readfst.hpp>
98 #include <task.writefst.hpp>
99 
100 #include <task.loadwordmap.hpp>
101 
102 #include <task.grammar.hpp>
103 #include <task.loadlm.hpp>
104 #include <task.prepro.hpp>
106 #include <task.ssgrammar.hpp>
107 #include <task.cykparser.hpp>
108 #include <task.referencefilter.hpp>
109 #include <task.hifst.hpp>
110 #include <task.optimizefst.hpp>
111 
112 #include <task.loadunimap.hpp>
113 #include <task.disambig.hpp>
114 
115 #include <task.postpro.hpp>
116 #include <task.hifst-stats.hpp>
117 
119 
120 #endif
Handles simple wildcard expansion for strings.
Convenience functors/functions for lexicographic<tropical,tropical> semiring.
Contains convenience functions to write and read fsts.
Stream wrapper for pipe/text/compressed files.
Relative to Stats across the pipeline.
Contains patterns to instance-patterns implementation.
Data object for hifst or related tools.
Implementation of different type of compositions (i.e. failure transitions)
Contains structures and classes for GrammarData.
Lower casing/Tokenization/Detokenization not available for open source release.
Data object for alilats to sparse weight lats binary.
Provides smarter assert methods.
Interfaces with basic methods for iteration.
To initialize boost parameter options for hifst tool.
Implementations of multiple fst unions.
Implementation of a Fst writer taking the fst from data object.
Describes class GrammarTask.
Contains structures and classes for hifst task (target lattice building)
Task that dumps statistics related specifically to the tool hifst.
Implementation of a language model task.
const bool detailed
Contains implementation of ApplyLanguageModelOnTheFly.
Implements a class that loads the grammar sparseweight flower lattice and stores a pointer on the dat...
Contains structures and classes for GrammarData.
Utilites to extract vocabulary, pseudo-determinize lattices and build substring transducers.
Headers for standalone shared library.
Implementation of a language model data structure using kenlm.
Implemention of class DisambigTask.
Convenience functions to parse parameters from a string.
Implementation of a Fst reader to data structure.
Task that writes translation to a text file. This translation might be wordmapped and tokenized...
Wrapper around WordMapper loader.
Contains implementation for sentence-specific grammar task.
Contains definitions for cykparser data and task.
Contains definitions for cykparser data and task.
Contains cyk parser implementation.
General functions.
Generalized weight mapper functor.
Contains definitions for sentence-specific grammar data and task.
Convenience functors that allow transparent handling for weights within hifst.
Handles different type of integer ranges.
Contains sentence-specific grammar data.
Describes class PreProTask, which preprocesses (tokenizes and maps to integers with WordMapper) sourc...
Files to include for the tropical sparse tuple semiring.
Describes class ReferenceFilterTask (builds unweighted substring fst for lattice alignment ) ...
General typedefs, defines...
Multiepsilon composition.
Contains wrapper class RegistryPO, which uses boost::program_options to parse parameters, and provides methods to access them.
Implementation of a Fst writer taking the fst from data object.
Definition: bleu.hpp:14
All included standard headers, boost headers,...
Implementation of a unigram transduction model loader task.