Usage

Python API usage for coreference, semantic graph and NERC

mvillanueva — Thu, 21 Sep 2023 07:18:30 +0000

Python API usage for coreference, semantic graph and NERC mvillanueva Thu, 09/21/2023 - 09:18

Forums

----
Intro
----
Hi, I have been using freeling for a few months now to extract triplets. So far I have succeded in doing so by using the dependency tree and the full parse tree, but I am trying to improve my approach by using coreference, semantic graph and NERC.

----
My work so far
---
I checked the tutorial for python, but I couldn't find anything beyond depdency parsing. So I went through the class list (since the same classes should be available for python and c++) and I decided to use relaxcor, but the analyze method of this class only accepts a document as a parameter.

----
Problems
----
So what I'm asking if anyone can help me with is the following:
1. Document creation problem: How do you create a document using the python API? All examples use a tokenizer then a splitter which returns a list of sentences. I checked some c++ examples provided in the simple_examples folder (https://github.com/TALP-UPC/FreeLing/blob/master/src/main/simple_exampl…) and I tried using the document method "insert" just like the example but after several different attempts I couldn't find a way to provide the right parameters, aparently it expects:
std::list< freeling::paragraph >::insert(
std::list< freeling::paragraph >::iterator,
std::list< freeling::paragraph >::value_type const &)
So i tried the method append. Which didn't throw any errors but I haven't fully checked if this is correct, since the only thing i checked is the number of words and it didn't seem correct, my text was a dummy test of the likes "Sobre la mesa María ve y coge una manzana, un sombrero, una llave y dos paraguas rojo." and after the append method the document had 80 words.
Here is my code so far for this:
# tokenize input line into a list of words lw = tk.tokenize(text) print(lw) # split list of words in sentences, return list of sentences ls = sp.split(lw)

paragraphs = pyfreeling.paragraph(ls)
# list_paragraphs = pyfreeling.ListParagraph([paragraphs])

doc = pyfreeling.document()
doc.append(paragraphs)

2. Doubt about entities: Going back to the example "Sobre la mesa María ve y coge una manzana, un sombrero, una llave y dos paraguas rojo." I realized that working with capitalized words and lowercase produce different results, but by making it all lowercase the entity recognition stops recognizing "maría" as a person. Is there are workaround for this or am I going in the wrong direction? The main problem is that "maría" not recognized as a named entity (which i need it to be by the way) results in "maría" not being the subject of the sentence anymore. Here is how im getting this:
neclass = pyfreeling.ner(lpath + "/nerc/ner/ner-ab-rich.dat") ls = morfo.analyze_sentence_list(ls) ls = tagger.analyze_sentence_list(ls) ls = sen.analyze_sentence_list(ls) ls = neclass.analyze_sentence_list(ls) ls = wsd.analyze_sentence_list(ls)

ls = srl_parser.analyze_sentence_list(ls)
ls = dep.analyze_sentence_list(ls)
ls = parser.analyze_sentence_list(ls)
3. How to retrieve named entities: Kind of a follow up of the previous question, how do I get the named entities? I couldn't find any code related to this.

Problem 1 half solved

I don't need a solution for problem 1 I just managed to create the document. Yet im unnable to retrieve any groups, according to the semantic graph it is 0
doc.get_semantic_graph().get_num_groups()

Problem 1About document…

Problem 1

About document creation: You are doing it right, create the paragraph from ls, and then the document from the list of paragraphs. I am not sure using [ ] will create a proper list.
Keep in mind that Freeling is a C++ library, so you have to adapt to C++ methods, not all python power can be translated into C++ easily.

To get the coreference groups, you need to run relaxcor. For relaxcor to run properly, you need to have previously run the document through the whole pipeline (tagger, NERC, wsd, parser, SRL, ...)

Also, just running relaxcor will add coreference info to your document, but not a semantic graph. For that you need to run the semantic graph extractor after relaxcor.

Problem 2&3

There are 3 different NERC modules in freeling, one rule-based and two ML-based. However, all of them use capitalization as a feature, and since both models are trained on standard text, all NEs seen in training are capitalized. So, it is very unlikely that the CRF or the adaBoost model will tag as NE something in lowercase.

Entities are identfied by their PoS tag. If a word (or multiword) has a pos-tag starting with "NP", then it means it was recognized by the NERC module. No other modules assign this tag.

You mentioned that in order…

You mentioned that in order to run the relaxcor i need to go through the whole pipeline. Well I am calling the analyzer like this:
doc = pyfreeling.document() analyzer.analyze(text, doc)

output = pyfreeling.output_freeling()
output.output_senses(False)
output.output_dep_tree(False)
output.output_corefs(False)
output.output_semgraph(True)

And configuring the analyzer like this but i get no coreference output:
# define creation options for morphological analyzer modules opt.config_opt.Lang = lang opt.config_opt.TOK_TokenizerFile = f"{data}/{lang}" + "/tokenizer.dat" opt.config_opt.SPLIT_SplitterFile = f"{data}/{lang}" + "/splitter.dat" opt.config_opt.MACO_Decimal = "." opt.config_opt.MACO_Thousand = "," opt.config_opt.MACO_LocutionsFile = f"{data}/{lang}" + "/locucions.dat" opt.config_opt.MACO_QuantitiesFile = f"{data}/{lang}" + "/quantities.dat" opt.config_opt.MACO_AffixFile = f"{data}/{lang}" + "/afixos.dat" opt.config_opt.MACO_ProbabilityFile = f"{data}/{lang}" + "/probabilitats.dat" opt.config_opt.MACO_DictionaryFile = f"{data}/{lang}" + "/dicc.src" opt.config_opt.MACO_NPDataFile = f"{data}/{lang}" + "/np.dat" opt.config_opt.MACO_PunctuationFile = data + "/common/punct.dat" opt.config_opt.MACO_ProbabilityThreshold = 0.001

opt.config_opt.NEC_NECFile = f"{data}/{lang}" + "/nerc/nec/nec-ab-poor1.dat"
opt.config_opt.SENSE_ConfigFile = f"{data}/{lang}" + "/senses.dat"
opt.config_opt.UKB_ConfigFile = f"{data}/{lang}" + "/ukb.dat"
opt.config_opt.TAGGER_HMMFile = f"{data}/{lang}" + "/tagger.dat"
opt.config_opt.PARSER_GrammarFile = f"{data}/{lang}" + "/chunker/grammar-chunk.dat"
opt.config_opt.DEP_TxalaFile = f"{data}/{lang}" + "/dep_txala/dependences.dat"
opt.config_opt.DEP_TreelerFile = f"{data}/{lang}" + "/treeler/dependences.dat"

opt.config_opt.MACO_CompoundFile = f"{data}/{lang}" + "/compounds.dat"
opt.config_opt.COREF_CorefFile = f"{data}/{lang}" + "/coref/relaxcor_constit/relaxcor.dat"

opt.invoke_opt.TAGGER_ForceSelect = pyfreeling.RETOK
opt.invoke_opt.SENSE_WSD_which = pyfreeling.UKB
opt.invoke_opt.TAGGER_which = pyfreeling.HMM
opt.invoke_opt.DEP_which = pyfreeling.TXALA
opt.invoke_opt.SRL_which = pyfreeling.TREELER
opt.invoke_opt.InputLevel = pyfreeling.TEXT
opt.invoke_opt.OutputLevel = pyfreeling.SEMGRAPH

# choose which modules among those available will be used by default
opt.invoke_opt.MACO_UserMap=False
opt.invoke_opt.MACO_AffixAnalysis = True
opt.invoke_opt.MACO_MultiwordsDetection = True
opt.invoke_opt.MACO_NumbersDetection = True
opt.invoke_opt.MACO_PunctuationDetection = True
opt.invoke_opt.MACO_DatesDetection = True
opt.invoke_opt.MACO_QuantitiesDetection = True
opt.invoke_opt.MACO_DictionarySearch = True
opt.invoke_opt.MACO_ProbabilityAssignment = True
opt.invoke_opt.MACO_CompoundAnalysis = False
opt.invoke_opt.MACO_NERecognition = True
opt.invoke_opt.MACO_RetokContractions = False

opt.invoke_opt.NEC_NEClassification = True
opt.invoke_opt.PHON_Phonetics = False

Relaxcor works better on the…

Relaxcor works better on the output of the lstm parser, not txala parser (lstm parser is much more accurate)

So, you need to set:
opt.config_opt.DEP_LSTMFile = f"{data}/{lang}" + f"/dep_lstm/params-{lang}.dat"
opt.invoke_opt.DEP_which = pyfreeling.LSTM

And since you are using LSTM parser and not the chart parser neither Txala parser, the following options are not necessary:
opt.config_opt.PARSER_GrammarFile = f"{data}/{lang}"+ "/chunker/grammar-chunk.dat"
opt.config_opt.DEP_TxalaFile = f"{data}/{lang}" + "/dep_txala/dependences.dat"

You also need to provide a config file for the SRL
opt.config_opt.SRL_TreelerFile = f"{data}/{lang}" + " treeler/srl.dat"

as well as for the semgraph extractor
opt.config_opt.SEMGRAPH_SemGraphFile = f"{data}/{lang}" + "/semgraph/semgraph-SRL.dat"

You can check file /usr/share/freeling/config/es.cfg to see which options are available for each module, and which config files may be set for each option.

Thank you for your quick…

Thank you for your quick response.
When i use relaxcor_dep i get the following message:
RELAXCOR_FEX: Requested feature RCF_DEMONYM not implemented. It will be ignored.
RELAXCOR_FEX: Requested feature RCF_LOC_MATCH not implemented. It will be ignored.

but if I change it to relaxcor_constit i get a MENTION_DETECTOR: Error: document is not parsed.

The first two are warnings,…

The first two are warnings, you can ignore them

relaxcor_constit is to be used if the parser was a constituency parser. (i.e the "PARSER" module in freeling, which I don't recommend since it is rule based and rather obsolete).

Since you are using LSTM parser, which is a dependency parser, you need to use relaxcor_dep

remove entirely the use of "_" (underscore) from the tagging output

kloro2006 — Thu, 10 Nov 2022 04:19:20 +0000

remove entirely the use of "_" (underscore) from the tagging output kloro2006 Thu, 11/10/2022 - 05:19

Forums

Usage

I would like to remove entirely the use of "_" (underscore) from the tagging output and have each word in the group tagged separately.

Here are some examples of what I mean, along with the sections of the original text:

LaMuerte.txt:A_el_poco_rato al_poco_rato RG 1 [Al poco rato empezaron a sentirse mal]

abanderado.txt:por_medio_de por_medio_de SP 1 [le señalan las faltas del juego por medio de banderas]

abajo.txt:En_dirección_a en_dirección_a SP 1 [En dirección a la parte más baja]

calabazo cala_bazo NCFS000 0.575281 [un enorme calabazo lleno de piedras]

How can I do this?

There are several reasons…

There are several reasons why you get this underscores, depending on the involved words, and the solution is different in each case

Named entities, dates, physical quantities, and multiword expressions, are detected by some modules in FreeLing, and they are glued together. The responsible modules can be selectively deactivated.

The solutions described below assume you have FreeLing installed in your computer, and that you have access to the user manual https://freeling-user-manual.readthedocs.io/en/v4.2/ for details about specific options and how to use it.

If you are using FreeLing via textserver, then the answer is shorter: Textserver offers pre-canned services based on FreeLing, but it does not expose all possible options or behaviours of the analyzer. If you want to customize FreeLing behaviour, you need to download and install it locally.

So, assuming you have FreeLing in your computer, and you input the sentece:
"Jose Perez vino el martes a las doce con treinta y dos naranjas, al poco rato se bebió dos litros de agua."

you get:

Jose_Perez jose_perez NP00SP0 1
vino venir VMIS3S0 0.1875
el el DA0MS0 1
martes_a_las_doce [M:??/??/??:12.00:??] W 1
con con SP 1
treinta_y_dos 32 Z 1
naranjas naranja NCCP000 0.638706
, , Fc 1
a_el_poco_rato al_poco_rato RG 1
se se P00CN00 0.494509
bebió beber VMIS3S0 1
dos_litros VL_l:2 Zu 1
de de SP 0.999961
agua agua NCCS000 0.997446
. . Fp 1

"jose_perez" is detected by the named entity recognizer. You can deactivate it with option --noner, but that will cause that proper nouns are not marked as such. An alternative is to set the option <SplitMultiwords> to "yes" in the named entity recognizer configuration file (e.g. /usr/share/freeling/es/np.dat)

"martes_a_las_doce" is detected by the date time recognizer. You can deactivate it with option --nodate. That will cause that time expressions are not marked.

"treinta_y_dos" is detected by the numbers recognizer. You can deactivate it with option --nonumb. That will cause that numerical expressions are not recognized as such (not even single-word numbers or numbers in figures)

"al_poco_rato" is detected by the locutions recognized. You can deactivate it with option --noloc

"dos_litros" is detected by the quantitites recognized. You can deactivate it with option --noquant

In you examples there is also "calabazo" -> "cala_bazo"
"calabazo" is not a Spanish word (or if it is, it is not in FreeLing dictionary). So the analyzer tries to see if it is a compound word (like "aguafiestas", "verdeazulado", "tierra-aire", etc. In this case, it finds a match with two nouns "cala" and "bazo" and marks that.
This is done by the compound detection module. You can deactivate it with option --nocomp
You can include new words in the dictionary editing the file /usr/share/freeling/es/dicc.src

Error while compiling Example #1

BenDiet — Sun, 14 Aug 2022 07:42:48 +0000

Error while compiling Example #1 BenDiet Sun, 08/14/2022 - 09:42

Forums

Usage

Hello everyone,

I'm a computer science student and working on a project, which involves
the usage of a NLP-tool like FreeLing. My question is probably based on a
easy to see mistake I made, but I can't solve it my self. So I was hoping
someone here could help me with that.

I'm currently trying to compile example 1 on my Linux system
and get the following error:

FAILED: CMakeFiles/test_freeling.dir/main.cpp.o
/usr/bin/c++ -g -std=gnu++17 -MD -MT CMakeFiles/test_freeling.dir/main.cpp.o -MF CMakeFiles/test_freeling.dir/main.cpp.o.d -o CMakeFiles/test_freeling.dir/main.cpp.o -c /home/ben/Schreibtisch/Bachelor_Projekt/test_freeling/main.cpp
In file included from /usr/include/freeling/morfo/idioma.h:45,
from /usr/include/freeling/morfo/lang_ident.h:44,
from /usr/include/freeling.h:35,
from /home/ben/Schreibtisch/Bachelor_Projekt/test_freeling/main.cpp:2:
/usr/include/freeling/morfo/smoothingLD.h: In constructor ‘freeling::smoothingLD::smoothingLD(const wstring&, const std::map, E>&)’:
/usr/include/freeling/morfo/smoothingLD.h:129:73: error: there are no arguments to ‘log’ that depend on a template parameter, so a declaration of ‘log’ must be available [-fpermissive]
129 | if (name==L"LinearDiscountAlpha") { double a; sin>>a; alpha = log(a); notalpha=log(1-a); }
| ^~~
/usr/include/freeling/morfo/smoothingLD.h:129:73: note: (if you use ‘-fpermissive’, G++ will accept your code, but allowing the use of an undeclared name is deprecated)
/usr/include/freeling/morfo/smoothingLD.h:129:90: error: there are no arguments to ‘log’ that depend on a template parameter, so a declaration of ‘log’ must be available [-fpermissive]
129 | if (name==L"LinearDiscountAlpha") { double a; sin>>a; alpha = log(a); notalpha=log(1-a); }
| ^~~
/usr/include/freeling/morfo/smoothingLD.h:179:18: error: there are no arguments to ‘log’ that depend on a template parameter, so a declaration of ‘log’ must be available [-fpermissive]
179 | pUnseen = -log(vsize-ntypes); // log version of 1/(vsize-ntypes)
| ^~~
/usr/include/freeling/morfo/smoothingLD.h:180:14: error: there are no arguments to ‘log’ that depend on a template parameter, so a declaration of ‘log’ must be available [-fpermissive]
180 | nobs = log(nobs);
| ^~~
ninja: build stopped: subcommand failed.

I didn't change anything of the example and don't really know what I did wrong.
I also tried to compile it with the terminal and used the command:

g++ example.cc -o example -lfreeling -std=c++0x

The same error appeared.
Did I install FreeLing wrong in any way or should I link another lib or something?
Thanks to everyone helping me.

To give you additional…

To give you additional Information:
I installed 'freeling-4.2-focal-amd64.deb' on 'Linux Mint 20.3 Cinnamon'.
I've also installed all the dependencies mentioned here:
https://freeling-user-manual.readthedocs.io/en/v4.2/installation/requir…

I've solved the problem by…

I've solved the problem by including to my main file and used using std::log. Is this supposed to be like this?

That is ok. The error is…

That is ok. The error is caused by a missing "using std::log" in smoothingLD.h, so adding it somewhere is the proper solution.

It is already fixed in the git master branch, smoothingLD.h already has the "using std:log" line

thanks!

Dependency parser generated info after analysis

Josenp1311 — Fri, 21 Jan 2022 09:23:08 +0000

Dependency parser generated info after analysis Josenp1311 Fri, 01/21/2022 - 10:23

Forums

Usage

Hello everyone.

I'm trying to replicate the dependency parsing results from the demo 4.2 on the Java 4.0 version, but I'm having some troubles doing so.

In the 4.2 demo, in the CONLL format, there are two new columns generated after selecting the dependency parsing, that's the information I'm looking for.

First. I can't find where that information is stored after doing the analysis with the Rule-based Dependecy Parser, I'm assuming the information is stored on the DepTree generated in the sentence after analyze it, but I don't know exactly where. The closest thing I found is the label generated in the nodes, but the labels generated don't match with the ones on the 4.2 demo.

I don't know if I'm looking at the wrong place, the labels are different on the 4.0 version, I'm doing the wrong dependency parsing method, or what I'm trying to do is impossible in the 4.0 version.

Thank you for your time!

You question is a bit…

You question is a bit confusing... "two new columns"... which two exactly? columns are always there, but depending on the used analyzers they have just dashes...

I'm going to assume that you mean columns 10 and 11 as shown in the demo, which contain the dependency head and function for each token.

If you use FreeLing API, you'll get a dep_tree with the same information. Only that the head number is not a number, but just the parent of each node (so you don't need the numbers, just accessing the parent node does the trick). The function is the attribute 'label' in the node, that can be obtained with get_label().

About labels matching between versions, that depends on which parser you use. The rule-based parser uses a custom set of labels. Statistical parsers (treeler and lstm) use the label set from the training corpus (which is CoNLL-X shared task corpus).
If you use the same parser in both versions, labels should be the same.

Notice that the web demo uses the lstm parser. If you use the rule-based parser, labels will not be the same.

Finally, notice also that the rule-based parser is a rather obsolete and its performance is below the state-of-the-art. You'll get better results with the lstm parser in 4.2.
If you want to stick with 4.0 (not recommended, it is over 5 years old), you should use dep_treeler parser, not the rule based one.

If all this does not answer your question, please be clear in what is the problem. An example may help.

JRE Crashing when using Freeling

Ayto — Mon, 16 Aug 2021 11:56:43 +0000

JRE Crashing when using Freeling Ayto Mon, 08/16/2021 - 13:56

Forums

Usage

Hello everyone, hope you're well.

I am using the Freeling library for my Master's Degree thesis on Java, so using the JFreeling API.
Whilst it is working perfectly well on its own, I need to combine it with the ExtJWNL library to extract hyponyms and other semantic relationships.
Since introducing the JWNL library, a SIGSEGV error has been thrown at me every time:

--------------------------------------------------------------------
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f9035e9d308, pid=4378, tid=0x00007f9072ba7700
#
# JRE version: OpenJDK Runtime Environment (8.0_292-b10) (build 1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.292-b10 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libJfreeling.so+0x14d308] std::_Rb_tree, std::less, std::allocator >::_M_begin() const+0xc
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/ayto/IdeaProjects/Lexical/hs_err_pid4378.log
Compiled method (nm) 98787 953 n 0 edu.upc.Jfreeling.JfreelingJNI::Word_getSensesString__SWIG_1 (native)
total in heap [0x00007f905c4f4790,0x00007f905c4f4b20] = 912
relocation [0x00007f905c4f48b8,0x00007f905c4f4900] = 72
main code [0x00007f905c4f4900,0x00007f905c4f4b18] = 536
oops [0x00007f905c4f4b18,0x00007f905c4f4b20] = 8
Compiled method (c1) 98787 862 3 sun.nio.cs.UTF_8$Encoder::encodeLoop (28 bytes)
total in heap [0x00007f905c4a5790,0x00007f905c4a5fb8] = 2088
relocation [0x00007f905c4a58b8,0x00007f905c4a5928] = 112
main code [0x00007f905c4a5940,0x00007f905c4a5dc0] = 1152
stub code [0x00007f905c4a5dc0,0x00007f905c4a5e78] = 184
metadata [0x00007f905c4a5e78,0x00007f905c4a5e90] = 24
scopes data [0x00007f905c4a5e90,0x00007f905c4a5f08] = 120
scopes pcs [0x00007f905c4a5f08,0x00007f905c4a5f98] = 144
dependencies [0x00007f905c4a5f98,0x00007f905c4a5fa0] = 8
nul chk table [0x00007f905c4a5fa0,0x00007f905c4a5fb8] = 24
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
-----------------------------------------------------------------------------------------------

I am using two document databases, A large one (~7200 docs), and a tiny sample(20) for tests. I'm using four semantic fields: hypernym, hyponyms, holonyms and meronyms.
Running the program trying to extract each at a run, on the test database all are working but for the hyponyms. No matter the depth choosen.
On the large main database, none of them is working.

What could be the problem here? I have been scratching my head over it on my own for a month now and haven't made any progress, any help is welcomed, thank you.

INFOS: Using Ubuntu and coding using Netbeans (tried changing IDE moving to IntelliJ too).
Here is the Java Code:
https://justpaste.it/2k0kb

I am not familiar with…

I am not familiar with ExtJWNL, so I am not sure of what may be the problem.

Also, it is not clear how are you comibing ExtJWNL and FreeLing. It may be something as simple as a missing synset in FL sense database.

You should try to narrow down the issue: Does it happen with a specific word or text? which? I see that the problem is with getSensesString... What parameters make it crash?

Apologies for the very late…

Apologies for the very late reply, I hope it's not too late to get help though.
Unfortunately I have been unable to narrow down the error. I have tried pinning down a possibly problematic file, or even word, but it's not it. The error doesn't always happen on the same file, in fact it may happen with previously working files, nor does it happen with specific words. Sometimes the error occurs just before FL start treating a new text.

What I have been able to find out is that on the 20-documents mini corpus I use for testing purposes, I am able to use FL to extract synsets (as is the case with the main corpus) as well as extract their hypernyms (crash on the main corpus), but not they hyponyms, unless the depth is set at 1. Meaning I can only extract the direct hyponyms of each word present in the mini corpus.

To give more precision as to how I proceed: I use a SimpleFileVisitor to walk through the files that are present in different folders (the folders are the labels for my classification). First I treat each text one by one with FL: Tokenization, disambiguation etc. I also capture each word and extract its synset. The goal is to write a Sparse Matrix for my SVM algorithm to train/test.
So basically I end up with a file written like this:
LABEL n:m n:m n:m
LABEL n:m n:m n:m (...)

With Label, the name of the folder containing the file, n the synset of the word, m the number of its occurences.

When I was done with this step, I introduced JWNL to extract hypernyms, hyponyms, meronyms etc. and see if it improves the Classification.
To do so, I simply made it so that in the middle of a FL analysis, I would take the currently processed word and use JWNL to get its hypernyms from its synset, up to a certain depth. I would then add those hypernyms' synsets to the Matrix.

But I have struggled with this process ever since without finding a solution.
Here's a copy of a bug report written by the JVM, I am seeing that the problem indeed occurs with getSensesString and during the WalkFileTree process, but I haven't been able to understand what the issue is exactly.
https://jpst.it/2CkKz

Thousand thanks for your time.

I do not have enough…

I do not have enough information to help you... You should activate the debug options in freeling to see what is going on. (build freeling with cmake option -DTRACES=ON, and then set TraceLevel and TraceModule variables in your program. See user manual for details

However, it may be easier to split your job in two stages: First you run a simpler program that uses freeling to produce your file with "LABEL n:m n:m n:m" etc

Then you write a separate program that reads this file and uses JWNL to look for hipernyms, hiponyms or whatnot.

In this way, you get rid of any possible interaction between FL and JWNL (which seems to be what is causing problems), and having them separated it may be easier to spot what the problem (is if it persists)

We thought of the same thing…

We thought of the same thing. I did just that. Separated the two in the same manner you described. Works all fine now. Thank you so much for taking the time to answer and for your help!

I also, somehow, forgot to…

I also, somehow, forgot to mention but during my tests, for some weird reasons the last word just before a crash would be printed out like this:
"CURRENT WORD: 챀翧emmed"
For some reason, no matter the word, Some letters are swapped with Kanji characters. My documents are all in English only.

Differences in dependenency parses across outlvs

maniol — Mon, 31 May 2021 09:46:14 +0000

Differences in dependenency parses across outlvs maniol Mon, 05/31/2021 - 11:46

Forums

Usage

Hello, I've been observing some inconsistencies wrt dependecy parser. I obtain different dependency parses in "dep" output level and in "semgraph" output level with Freeling 4.2 and lstm dep parser.
Frase: "atm withdrawal five hundred dollars on june seventeenth"
--outlvl dep
"dependencies" : [ {"token" : "t1.2", "function" : "ROOT", "word" :"withdrawal", "children" : [ {"token" : "t1.1", "function" : "NMOD", "word" : "atm"}, {"token" : "t1.3", "function" : "ADV", "word" : "five_hundred_dollars"}, {"token" : "t1.4", "function" : "TMP", "word" : "on", "children" : [ {"token" : "t1.5", "function" : "PMOD", "word" : "june_seventeenth"} ]}
--outlvl semgraph
"dependencies" : [ {"token" : "t1.3", "function" : "ROOT", "word" : "five_hundred_dollars", "children" : [ {"token" : "t1.2", "function" : "NMOD", "word" : "withdrawal", "children" : [ {"token" : "t1.1", "function" : "NMOD", "word" : "atm"} ]}, {"token" : "t1.4", "function" : "LOC", "word" : "on", "children" : [ {"token" : "t1.5", "function" : "PMOD", "word" : "june_seventeenth"} ]} ]}]
Also, this same sentence yields yet another dep parse in the online demo (with semgraph outout level):
dependencies: depnode function="ROOT" token="t1.2" word="withdrawal" depnode function="NMOD" token="t1.1" word="atm"/ depnode function="NMOD" token="t1.3" word="five_hundred_dollars" depnode function="TMP" token="t1.4" word="on" depnode function="PMOD" token="t1.5" word="june_seventeenth"

The demo parse is the most correct parse.
Could you please explain how I can reproduce the online demo pipeline locally? What parameters should I set to what values so that I can obtain the same parse locally?
Thank you.

FreeLing version running in…

FreeLing version running in the demo is not necessarily 4.2, it may be some later development version, so that explains one of the differences.

But if you set --dep lstm in both output levels, the tree should be the same.
This may be some bug in 4.2 maybe fixed in the development version.

I'll check if I can reproduce the problem in master branch and if so, try to find out the cause.

I found the problem. There…

I found the problem. There was a bug in the "analyzer" program that affected the interaction between the selected output level and the parser to be used. For "semgraph", it ended up not using the lstm parser, but treeler.

I fixed the option management, so it will work now. You can replace your main program in:

src/main/sample_analyzer/main.cc

with the one in master
https://github.com/TALP-UPC/FreeLing/blob/master/src/main/sample_analyzer/main.cc

then "make install" again, and it should work. Let me know if it doesn't

Adding new named entities to Freeling

UDL — Sat, 23 Jan 2021 00:23:24 +0000

Adding new named entities to Freeling UDL Sat, 01/23/2021 - 01:23

Forums

Usage

I've been using Freeling for a while and it seems like a great project. What I have used it the most for is to work with named entities. Now I am interested in adding new entities to the recognition, so I wonder if someone here has already done this and could help me a little, I have been looking at the code and it is not as simple as using a corpus labeled with the new entities, you have to change various things in the scripts.
In short, someone from here in the community has already added new entities to the recognition in freeling, could you give me some advice on which files to modify and how?

(Additionally I have another question about how to obtain the necessary elements to create a corpus in full format, specifically lemma, PoS tag and the list of triple lemma-tag-probabilty for all possible word readings.)

To alter the set of classes…

To alter the set of classes, there are indeed a few changes to do to the scripts.

Basically, you should look for the places where the freeling-like tags (NP00G00, NP00SP0, etc) are used or mapped to generic class names (LOC, PER, etc)

This happens in:

    corpus/bin/extract-gaz.sh
    nec/bin/extract-gaz.sh
       Several places mapping one set of labels to the other:
      nom["NP00SP0"]="PER"; nom["NP00G00"]="LOC";
      nom["NP00O00"]="ORG"; nom["NP00V00"]="MISC";

    corpus/bin/nec2full.awk
      Same agaain
     c["NP00SP0"]="PER"; c["NP00G00"]="LOC"    c["NP00O00"]="ORG"; c["NP00V00"]="MISC"

   nec/bin/nec-adaboost.sh
   nec/bin/nec-svm.sh
      Some places with a list of freeling labels for the trainer:
      "0 NP00SP0 1 NP00G00 2 NP00O00 3 NP00V00"

Those should be the only things you need to change

(Additionally I have another question about how to obtain the necessary elements to create a corpus in full format, specifically lemma, PoS tag and the list of triple lemma-tag-probabilty for all possible word readings.)

You can use FreeLing to produce the Pos tag column. You can use FreeLign with option "--outlv morfo" to produce the list of possible pos tags. You can then use "paste" (or a small python script) to put this toghether. The column with the NE bio class, you'll have to add it manually.

retraining ner using freeling corpus

UDL — Thu, 19 Nov 2020 21:58:54 +0000

retraining ner using freeling corpus UDL Thu, 11/19/2020 - 22:58

Forums

Usage

Hello, I would like to extend the classification of named entities of Freeling, improving the ones that already exist and adding some new ones, where can I get the corpus with which the model that comes by default in Freeling was trained? the language in which I want to do this is in Spanish.

FreeLing is trained on…

FreeLing is trained on public corpora.

NERC is trained with CoNLL 2002 shared task data: https://www.clips.uantwerpen.be/conll2002/ner/

Thank you

Thank you very much, so what you want to tell me is that, for example, freeling for the recognition of entities in Spanish used the data from this corpus (https://www.clips.uantwerpen.be/conll2002/ner/data/esp.train )
That's 270k lines, I guess that's enough. Then I need to transform this corpus to full corpus format before processing, right?
Now a question, if I wanted to add new classes to the classification of named entities, could I use that same corpus with more elements to keep the previous classifications?

Thank you very much, so what…

Thank you very much, so what you want to tell me is that, for example, freeling for the recognition of entities in Spanish used the data from this corpus (https://www.clips.uantwerpen.be/conll2002/ner/data/esp.train )

yes, that's it

That's 270k lines, I guess that's enough.

Enough or not, it is what is available. I think there is a train set, a development set, and a test set, so the total may be more than that

Then I need to transform this corpus to full corpus format before processing, right?

Yes, it should be put in the format expected by the training scripts.

Now a question, if I wanted to add new classes to the classification of named entities, could I use that same corpus with more elements to keep the previous classifications?

Yes, as long as you go over the whole corpus and manually mark any instances of your new classes as B-NEWCLASS or I-NEWCLASS

I am not sure whether you may need to change something in the training script to make it aware of which are the expected classes (you can check that with a quick look at the script).

I will ask you because it…

Thank you very much for the help so quickly, I will take all that into account.

I will ask you because it seems that you have experience in the matter but in case you do not know how to answer me, what I originally planned was to do another post. Wouldn't it be possible to define in Freeling a set of rules that help in the classification? for example, if you wanted to add a new entity called Installation, like hospitals, museums, etc. , where the structure would be [word] of [Named entity] and [word] belongs to a list of words such as (hospital, factory, building, etc.) and the set of words that this structure has would be classified as Installation. Do you think there is a way to do something like this?

FreeLing does not provide…

FreeLing does not provide that functionality out of the box.

However, you can use FreeLing to process a text, tokenize it, get the PoS of each word, identify NEs, etc. and then write a small program that detects the patterns you are interested in using the information in the processed text.

That is the main goal of FreeLing: provide a tool that converts text into structured data, so you can use it for your desired application (extracting patterns in your case).

One possible workaround could be using the chart parser: You could define a simple grammar that identifies the patterns you are interested in (check the user manual for chart_parser module).
Since the parser is devised to perform shallow syntactic analysis, depending on the pattterns you want to find, it may not have the right expression power.. In any case, it may be worth a try.

I've been reading about the…

I've been reading about the chart_parser module and I still don't really understand it at all, Freeling is an excellent project but unfortunately the documentation still needs to improve a bit. I will continue investigating to see if I can understand how to use it well. I wanted to ask you a question, do you know if there is a script that converts the CoNLL 2002 corpus (https://www.clips.uantwerpen.be/conll2002/ner/data/esp.train) to full corpus format to be able to do the training with the "prepare-corpus.sh" script?

I've been reading about the…

I've been reading about the chart_parser module and I still don't really understand it at all,

A chart parser is a parser that expects a CFG grammar. The patterns you suggested (e.g. "(hospital|bank|government) of NE ") can be described with a CFG grammar.

You should not use this module until you understand what a CFG grammar is and can write one.

Freeling is an excellent project but unfortunately the documentation still needs to improve a bit.

FreeLing is free software, developed with very low resources, and provided "as is".
You are welcome to contribute anytime.

do you know if there is a script that converts the CoNLL 2002 corpus to full corpus format to be able to do the training with the "prepare-corpus.sh" script?

There is none, but shouldn't be difficult to build one, since it is a similar format, and FreeLing produces it (e.g. if you want to add pos tag columns to the file)

Yes, I know it is a free…

Yes, I know it is a free project, and it is great, I am just learning to use Freeling well, when I feel more experienced I will try to help as I can. Thank you so much for your help

iOS (terminal): Library not loaded. Image not found

Sebastià Salvà i Puig — Fri, 30 Oct 2020 11:54:53 +0000

iOS (terminal): Library not loaded. Image not found Sebastià Salvà… Fri, 10/30/2020 - 12:54

Forums

Usage

Dear all,

I have just installed FreeLing 4.2 in a MacBook (iOS Catalina) and the language pack.
I amb new on FreeLing and I am not very used to use Terminal.

I'd like to tag a text in Catalan. The file TXT (UTF 8) is in the Documents directory.

When I try some command, I'll get this:

Nou-MacBook-Air:~ sebastia$ cd ~/documents
Nou-MacBook-Air:documents sebastia$ analyze -f myconfig.cfg --outlv tagged prova_preproc.tag
dyld: Library not loaded: /usr/local/opt/boost/lib/libboost_regex-mt.dylib
Referenced from: /usr/local/bin/analyzer
Reason: image not found
/usr/local/bin/analyze: line 39: 24951 Abort trap: 6 $FREELING/bin/analyzer $param
Nou-MacBook-Air:documents sebastia$

What I should do, in order to load that library? FreeLing is installed in the default setup-up directories for MAC.

I would apreciate any help.

Did you installed required…

Did you installed required libraries, as described in the user manual ?

https://freeling-user-manual.readthedocs.io/en/latest/installation/requirements-mac/#install-dependencies

Symbol not found

Permalink

Thanks for the answer. I have just installed XCode, HomeBrew and I have run the "brew install boost icu4c" command.

However, now I get this other message:

Nou-MacBook-Air:documents sebastia$ analyze -f myconfig.cfg --outlv tagged mytext.tag
dyld: lazy symbol binding failed: Symbol not found: __ZN5boost16re_detail_10730011raw_storage6resizeEm
Referenced from: /usr/local/lib/libfreeling.dylib
Expected in: /usr/local/lib/libboost_regex-mt.dylib

dyld: Symbol not found: __ZN5boost16re_detail_10730011raw_storage6resizeEm
Referenced from: /usr/local/lib/libfreeling.dylib
Expected in: /usr/local/lib/libboost_regex-mt.dylib

/usr/local/bin/analyze: line 39: 7433 Abort trap: 6 $FREELING/bin/analyzer $param
Nou-MacBook-Air:documents sebastia$

And another question, in…

And another question, in case we can manage to solve the previous problem: how should I pick up the Catalan language package, which is already downloaded and installed? I mean, when running an instruction such as "analyze -f myconfig.cfg --outlv tagged mytext.tag", how does FreeLing now that this text is in Catalan?

Dear Lluís, I installed…

Dear Lluís,

I installed XCode, HomeBrew and I ran the "brew install boost icu4c" command.

However, I get this other message:

sebastia@Nou-MacBook-Air documents % analyze -f myconfig.cfg --outlv tagged rond1_preproc.tag
dyld: lazy symbol binding failed: Symbol not found: __ZN5boost16re_detail_10730011raw_storage6resizeEm
Referenced from: /usr/local/lib/libfreeling.dylib
Expected in: /usr/local/lib/libboost_regex-mt.dylib

dyld: Symbol not found: __ZN5boost16re_detail_10730011raw_storage6resizeEm
Referenced from: /usr/local/lib/libfreeling.dylib
Expected in: /usr/local/lib/libboost_regex-mt.dylib

/usr/local/bin/analyze: line 39: 4739 Abort trap: 6 $FREELING/bin/analyzer $param
sebastia@Nou-MacBook-Air documents %

What could I do?

Besides this, in case we can manage to solve the previous problem: how should I pick up the Catalan language package, which is already downloaded and installed? I mean, when running an instruction such as "analyze -f myconfig.cfg --outlv tagged mytext.tag", how does FreeLing now that this text is in Catalan?

I would really apreciate your help.

Symbol not found

Thanks for the answer. I have just installed XCode, HomeBrew and I have run the "brew install boost icu4c" command.

However, now I get this other message:

dyld: Symbol not found: __ZN5boost16re_detail_10730011raw_storage6resizeEm
Referenced from: /usr/local/lib/libfreeling.dylib
Expected in: /usr/local/lib/libboost_regex-mt.dylib

/usr/local/bin/analyze: line 39: 7433 Abort trap: 6 $FREELING/bin/analyzer $param
Nou-MacBook-Air:documents sebastia$

MAybe it is related to the…

MAybe it is related to the boost version... can you check which version did brew install ?

boost 1.74.0

Thanks you for the answer.

sebastia@Nou-MacBook-Air ~ % brew list --versions
boost 1.74.0
cmake 3.18.4
gdbm 1.18.1_1
icu4c 67.1
node 15.0.1
openssl@1.1 1.1.1h
pcre 8.44
perl 5.32.0
python@3.9 3.9.0_1
readline 8.0.4
sqlite 3.33.0
swig 4.0.2
xz 5.2.5
sebastia@Nou-MacBook-Air ~ %

What should I do now?

Dear Lluís, I am still not…

Dear Lluís,

I am still not able to properly run FreeLing aftern installing it. I'd apreciate some help.

VERSIONS:
boost 1.74.0
cmake 3.18.4
gdbm 1.18.1_1
icu4c 67.1
node 15.0.1
openssl@1.1 1.1.1h
pcre 8.44
perl 5.32.0
python@3.9 3.9.0_1
readline 8.0.4
sqlite 3.33.0
swig 4.0.2
xz 5.2.5

semantic graph

alondra831002 — Wed, 09 Sep 2020 05:21:40 +0000

semantic graph alondra831002 Wed, 09/09/2020 - 07:21

Forums

Usage

How can I obtain a semantic graph using Freeling with ruby?

I use this code line in ruby…

I use this code line in ruby and I obtain a plain text with the POS information of my input text but I want to obtain a semantic graph in an XML format does somebody know how to do it?

system("C:/freeling_win22/freeling/bin/analyzer.exe -f C:/freeling_win22/freeling/data/config/es.cfg <"+file+">"+ file2)

You need to call "analyzer…

You need to call "analyzer.exe" with the appropriate options. (though it would be better to call "analyze.bat" which takes care of solving paths )

User manual on "analyzer.exe" and "analyze.bat": https://freeling-user-manual.readthedocs.io/en/v4.2/analyzer/

User manual on options you can use (you need to look for "output level" and "output format"): https://freeling-user-manual.readthedocs.io/en/v4.2/analyzer/#valid-options