Error loading ukb with Python API on Windows 10

Submitted by PaulBreugnot on Thu, 08/23/2018 - 15:08
Forums
Installation

Hello,
I'm using Freeling Python API since 3 months now, mainly on Linux, and for other users I needed to use it on Windows.
So I compiled Freeling 4.1 as described in the doc, using Microsoft Visual Studio 2015 (pre-compiled boost as mentioned in the doc are not available for Microsoft Visual Studio 2017) and it compiled without error, running the following CMake command :

cmake .. -DCMAKE_INSTALL_PREFIX=C:\Freeling\freeling -DZLIB_INCLUDE_DIR=C:\Freeling\dependencies\zlib\include -DZLIB_LIBRARY=C:\Freeling\dependencies\zlib\lib\zlibd.lib -DBOOST_ROOT=C:\Freeling\dependencies\boost -DICU_ROOT=C:\Freeling\dependencies\icu -DSWIG_DIR="C:\Program Files\swigwin" -DSWIG_EXECUTABLE="C:\Program Files\swigwin\swig.exe" -DPYTHON3_API=ON -DJAVA_API=ON -G "NMake Makefiles"

However, when I try to load ukb, for example running this code from the APIs\Python3 folder :

import pyfreeling

# Init locales
pyfreeling.util_init_locale("default");

LANG="es";
op= pyfreeling.maco_options(LANG);
DATA = "C:\\Freeling\\freeling\\share\\freeling\\"
op.set_data_files( "",
DATA + "common\\punct.dat",
DATA + LANG + "\\dicc.src",
DATA + LANG + "\\afixos.dat",
"",
DATA + LANG + "\\locucions.dat",
DATA + LANG + "\\np.dat",
DATA + LANG + "\\quantities.dat",
DATA + LANG + "\\probabilitats.dat");

wsd=pyfreeling.ukb(DATA + LANG + "\\ukb.dat")

after a few minutes a window popup with the following error message :

Debug Assertion Failed!

Program: C:\windows\SYSTEM32\MSVCP140D.dll
File: C:\Program Files (x86)\Microsoft Visual Studio14.0\VC\INCLUDE\vector
Line: 1234

Expression: vector subscript out of range

For information on how your program can cause an assertion failure, see the Visual C++ documentation on asserts.

(press Retry to debug the application)

with 3 buttons, "give up", "retry", "ignore"...

This can't be a source code problem because it works on Linux, on I am nearly sure (I can't be sure about anything right now) that I already used the Python API with ukb on Windows. So this probably come from my configuration, but I re-compiled everything, re-installed Visual Studio, nothing works and now I just don't know what to do.

Any idea would be welcomed.

Is it crashing when loading the ukb module?  or when calling it?

Note that to call UKB module, you need to run first the tagger and the "senses" module.

You can debug this further using the following steps:

  • build freeling with cmake option -DTRACES=ON
  • add the following lines at the beggining of your python program

        pyfreeling.traces.TraceLevel = 6   # the higher, the more verbose
        pyfreeling.traces.TraceModule = 0x00400000   # see src/include/freeling/morfo/traces.h to find the bitmask for each module

 In this way you may be able to spot with more precision what is going on.

 

Hello,
firstly thank you for your quick answer.

The problem is loading the module, I also have problem with my full functional freeling codes. Also, even if sample.py works, it's like very, very slow...

I compiled Freeling with the -DTRACES=ON option :
cmake .. -DCMAKE_INSTALL_PREFIX=C:\Freeling2\freeling -DZLIB_INCLUDE_DIR=C:\Freeling2\dependencies\zlib\include -DZLIB_LIBRARY=C:\Freeling2\dependencies\zlib\lib\zlibd.lib -DBOOST_ROOT=C:\Freeling2\dependencies\boost -DICU_ROOT=C:\Freeling2\dependencies\icu -DSWIG_DIR="C:\Program Files\swigwin" -DSWIG_EXECUTABLE="C:\Program Files\swigwin\swig.exe" -DPYTHON3_API=ON -DTRACES=ON -G "NMake Makefiles" > cmake_output.txt

Outputs for cmake and nmake are there :
https://drive.google.com/open?id=1Br0gGYz7WYiQlDfMpOIXu91NR__8fE9K

But I'm not sure it changed something.
I also had a screenshot of the window that pop up.

Also, this Python code :

import pyfreeling

pyfreeling.traces.TraceLevel = 6 # the higher, the more verbose
pyfreeling.traces.TraceModule = 0x00400000 # see src/include/freeling/morfo/traces.h to find the bitmask for each module

# Init locales
pyfreeling.util_init_locale("default");

LANG="es";
op= pyfreeling.maco_options(LANG);
DATA = "C:\\Freeling2\\freeling\\share\\freeling\\"
op.set_data_files( "",
DATA + "common\\punct.dat",
DATA + LANG + "\\dicc.src",
DATA + LANG + "\\afixos.dat",
"",
DATA + LANG + "\\locucions.dat",
DATA + LANG + "\\np.dat",
DATA + LANG + "\\quantities.dat",
DATA + LANG + "\\probabilitats.dat");

wsd=pyfreeling.ukb(DATA + LANG + "\\ukb.dat")

doesn't print anything. Am I doing something wrong?

PaulBreugnot

Sat, 08/25/2018 - 01:53

Permalink

Today I compiled Freeling 4.1 on an other Windows, and same problem : incredibly slow processing with sample.py, and crash when trying to load ukb.

Maybe a problem with some version of Visual Studio / CMake / SwigWin ...?
(For example the program didn't compiled with CMake 3.12 but it worked with CMake 3.11...)

Is it equally slow when running analyzer.bat?

If that is faster, then yes, it points to something with the API...

[edit] I've been checking, and I see that the ukb module stores the WN graph in a CSR (Compressed Row Storage) structure ( see https://en.wikipedia.org/wiki/Sparse_matrix for details).

This is implemented in the class csr_kb.cc, which relies on several STL lists and maps... Maybe the STL implementation is not as efficient in MSVC... or maybe there is some kind of operations where it is buggy...  :S

You can try adding traces to this module too. 

 

Actually, the initialization only writes traces when it ends.  If it crashes while loading, it may be you don't see anything.

You may try to add some trace lines to src/libfreeling/ukb.cc in the constructor method.

The method is quite straightforward, and the format for traces is also easy to see from other lines with TRACE(...)