Maximum Entropy Modeling Toolkit for Python and C++

News

• 2011-09-02 Source code moved to github.com
  Please download the latest source code of maxent from github.com: https://github.com/lzhang10/maxent
  The new release incorporated important bug fixes by 
  Jiri Marsik.
• version 20061005
  Fix building problem on GCC 4, upgrade python binary package to Python 2.5. Linux binary is the same so not updated.
• 20060810
  Haven't touched the code for two years and I need to read the manual to see how to use it. Well, I put a self-explain note about Maximum Entropy here.
• 20060129
  
  Upgrade python binary package to Python 2.4.
• version 20040930
  Maintenance update. Some users encountered troubles in building the toolkit (and the python extension) under Debian Linux. This version fixes all those frustrations, plus a new installable python extension for Win32!.
• version 20040709
  In response to popular request this version can now be compiled with Microsoft Visual C++ 7.1, along with several other 
  win32 compilers. The building system has switched to Autoconf/Jam, a faster solution to the SCons used before. This version also fixes several serious bugs so all users are recommended to upgrade.
• version 20040315
  bug fixed version, the license is now changed to LGPL.
• version 20031006
  real feature supported, compressed binary model and faster IO.

Introduction

The Maximum Entropy Toolkit provides a set of tools and library for constructing maximum entropy (maxent) model in either Python or C++.

Maxent Entropy Model is a general purpose machine learning framework that has proved to be highly expressive and powerful in statistical natural language processing, statistical physics, computer vision and many other fields. Please see my maxent page for more information on maxent. right place.
If you do not like this Toolkit, you can still try several other ME implementations.

Features

Current version has the following highlights:

• Conditional Maximum Entropy Model
• L-BFGS Parameter Estimation
• GIS Parameter Estimation
• Gaussian Prior Smoothing
• C++ API
• Python Extension module written in C++
• Command line utility: maxent
• Documentation and Tutorial ;-)

Supported Platforms and Compilers

The toolkit is written in ISO C++ with speed and portability in mind. It has been tested under the following OS/Compilers:

The toolkit is written in ISO C++ with speed and portability in mind. It has been tested under the following OS/Compilers:

• GNU/Linux (kernel 2.4.x) with GCC 3.2/3.3
• FreeBSD 4.8/4.9 with GCC 3.2
• FreeBSD 4.8/4.9 with MinGW Cross Compiler (aka. cross-mingw)
• NetBSD 1.6.2 with GCC 3.2
• SunOS 5.9 with GCC 3.4.3
• Win32/Cygwin with GCC 3.2
• Win32/MinGW with GCC 3.2
• Win32/Borland Free C++ Compiler 5.5 with STLPort 4.5.3
• Win32/Microsoft Visual C++ 7.1
• Win32/Microsoft Visual C++ 7.1 with STLPort 5.0-beta
• Win32/Intel C++ 8.0 with MSVC7.1's runtime lib

License

This software is freeware and is released under LGPL license. Please consult the LICENSE file in source package for more information.

The adoption of LGPL is in accord with the license of java maxent project: http://maxent.sourceforge.net, from which the toolkit was derived. LGPL makes it easier to share source code, as well as new ideas, between both projects.

Download

Here's a list of files you can download. Please note:

• Binaries for FreeBSD and NetBSD are removed to save web space. Since both OSs can run Linux binary without any difficulity (through Linux binary compatibility mode), users on these platforms are advised to use linux binary (see below) or build from source package).
• Binary files may be a little out of date (but they work fine)
• You still need to get document from the source tar ball, even if you only download the binary.

Document

Here is the PDF manual. Alternatively, you can generate HTML reference document by running "doxygen" in the doc/ directory if you have Doxygen installed.

FAQ

• I got a runtime-error while running the L-BFGS trainer. The error is:
  

   IFLAG= -1 
   LINE SEARCH FAILED. SEE DOCUMENTATION OF ROUTINE MCSRCH
   ERROR RETURN OF LINE SEARCH: INFO=  3
   POSSIBLE CAUSES: FUNCTION OR GRADIENT ARE INCORRECT
   OR INCORRECT TOLERANCES
  std::runtime_error caught:lbfgs routine stops with an error
  

  This is a rare event and most users do not encounter this problem. However, when this does happen it is usually an indication of "over-training", which can be caused by either using a very small dataset, or trying to perform too many training iterations on a large dataset. The ad-hoc solution is to use a larger dataset, or simply use fewer training iterations. Future releases will probably turn this bug into a useful feature:-)

• Missing limits.hpp when compiling the source code?
  

  You will get this error if you try to compile the source code with an old version of gcc (gcc 2.9x). This can be fixed by using gcc 3.x instead. If your new gcc binaries are named gcc32 and g++32. You can run the following command to get the building system configured properly:

      CC=gcc32 CXX=g++32 ./configure
  

• Can you explain the format of the model file? I want to use it in my own application.
  

  There is no document on the model file format at this time. Since it is likely to be changed in future releases. You are encouraged to look at src/modelfile.cpp, if you are really curious.

• I got an error regarding -fPIC when building on an AMD64 machine, help.
  

  A user Zhihui Jin suggests adding '-fPIC' into FORTRANFLAG to compile lbfgs.f will solve the problem.

  Also Vee Satayamas made a patch so that the code can be compiled on AMD64.

History

Back to the year 2002, as I got more knowledge on maxent I decided to do some experiments to assess the usefulness of the framework. Unfortunately, unlike other popular machine learning techniques such as SVMs, which have many off-the-shelf implementations on the net, only few maxent packages can be found from the internet. Partly because implementing a maxent model is a non-trival work, especially the iterative parameter estimating procedure.

After trying several maxent implementations, I found the java maxent package is a high quality, easy use one. The software is part of the OpenNLP project and is written in Java. But I want a C++ or Python solution that can be integrated into my existing code. Then, I began rewriting the java code in C++. Hopefully, it is relatively easy to translate java code into C++ (but not vice versa) and by the end of 2002 I finished the first C++ version. The speed of C++ is impressive: optimized C++ code ourperforms java code by a wide margin.

estimate ME model's parameters. His experiment showed L-BFGS was much faster than GIS and IIS. So, I added L-BFGS estimating code in May. 2003. Meanwhile, I swithed the project form autoconf/automake to SCons, a much better make replacement written in Python. Later, in April. 2003 I came across Curran and Clark's paper, which proves a correction-free version of GIS algorithm (originally pointed out by [Goodman, 2002]), elimiting the need of correction feature in GIS. I readily adopted the idea and greatly simplified the GIS code.

In September, 2003, documentation was added and the toolkit was put to public release.

Acknowledgment

• developers of maxent.sf.net, the java implementation of Maxent with GIS training algorithm. Actually, this toolkit evolves from an early attempt to port java maxent to C++.
• Robert Malouf. Dr. Malouf kindly answered my questions on maxent and provides his excellent implementation on four maxent parameter estimation algorithms.
• Jorge Nocedal, for the excellent Fortran L-BFGS implementation.

Contact

This software is freeware and as such there will be no support from the author.

Wow, you read everything, cool!