Linguistic Data

Version of sense dictionary for Spanish?

ulrike.henny — Tue, 14 Dec 2021 14:41:59 +0000

Version of sense dictionary for Spanish? ulrike.henny Tue, 12/14/2021 - 15:41

Forums

Hello,
I have been using FreeLing for some time now to annotate literary texts in Spanish. Thank you very much for this great and very useful tool!
Today I have a question about the version of the sense dictionary for Spanish which is used in FreeLing. In the documentation I found the information that the sense dictionary from the MRC 3.0 project is used. However, I noticed that some words that I would have expected to be annotated with sense information are not.
For example, the lemma "callejuela" is contained in MRC 3.0 as it can be found through the web interface at https://adimen.si.ehu.es/cgi-bin/wei/public/wei.consult.perl?item=calle…. However, this word is not annotated with sense information when I use the analyzer of FreeLing. An example for a sentence where it occurs is "Ibamos a salir de una callejuela formada con sacos de harina y cajas de fideos."
I used the following command to call the analyzer: analyze -f es.cfg --outlv tagged --sense ukb --nec --output xml < nh0010.txt > nh0010.xml
My version of FreeLing is 4.0 and I use it on Ubuntu 20.04.3 LTS.
I was wondering whether the latest version of the MRC is used in FreeLing or maybe an older version with fewer senses. Of course I consider to upgrade to the latest version of FreeLing if that would make a difference but did not do so yet because I could not find any comments on this issue in the news about the latest versions.
Best regards,
Ulrike

4.0 is a bit outdated... It…

4.0 is a bit outdated... It is very likely that it includes an older version of MCR that does not contain "callejuela".

An easy way to fix it is just to replace the file senses30.src with the one from a newer FreeLing version

Also, upgrading to a more recent version will solve this and many other issues you may have.

Thank you very much! I…

Thank you very much! I removed version 4.0 and installed 4.2 and now indeed more senses are annotated than before, including "callejuela". It is also good to know where the file with the senses is.

Just for information, now I get some warnings of the following type:

Unknown synset 80000054-a ignored. Please check consistency between sense dictionary and KB

But I already found your discussion on this issue (https://githubmemory.com/repo/TALP-UPC/FreeLing/issues/109).

Thank you for your quick response and help!

That synset corresponds to …

That synset corresponds to "human" when used as an adjective, which it seems to have been added to esWN, but not to other languages:

$ grep 80000054 data/es/senses30.src
80000054-a humano

The graph structure is common to all languages, and does not include any relation for that sysnset, thus the WSD algorithm complains about that.

$ grep 80000054 data/common/xwn.dat
01835496-v 80000054-v
08524735-n 80000054-n

Verbs suddenly missing in Freeling dictionary (PT)?

pbholmen — Wed, 09 Dec 2020 18:39:18 +0000

Verbs suddenly missing in Freeling dictionary (PT)? pbholmen Wed, 12/09/2020 - 19:39

Forums

Linguistic Data

I'm using the Portuguese Freeling dictionary data file for a tool for personal use. I just upgraded Freeling using Homebrew on macOS, the current version is stated as 4.2_1

A very common verb, "resolver" is suddenly missing. I looked in the GitHub repo, and see that it is also missing in FreeLing/data/pt/dictionary/entries/verbs.

I looked at the History for the file and found three commits, one named "Initial load version 4.0 (beta1)", in this version the verb is there. The second commit is named "new pt data", also here it is included, and the third commit, made on May 4 2016, named "improves in dict solving" does not include the verb.

I understand that it's not meant to be a data source for arbitrary tools, so I wonder is this a regression or meant to be that way? Is the verb handled some other place in the Freeling system? Or have I accidentally installed a development version and not an official release?

it may just be it was…

it may just be it was deleted by mistake.

If you improve the dictionary entries in data/pt/dictionary/entries, you're welcome to pull request them

thanks!

Error in the Portuguese dictionary?

pbholmen — Wed, 09 Dec 2020 12:04:28 +0000

Error in the Portuguese dictionary? pbholmen Wed, 12/09/2020 - 13:04

Forums

Linguistic Data

I think there might be an error in the dictionary found under:
share/freeling/pt/dicc.src

Line 232708 reads:

dar dar VMN0000 dar VMN01S0 dar VMN03S0 dar VMSF1S0 dar VMSF3S0

I believe this means that "dar" is supposed to be the future subjunctive of the verb "dar" for third and first person singular. This would be the case if the verb was regular, however it is irregular. Here is an authoritative source for the correct conjugation:

http://www.portaldalinguaportuguesa.org/index.php?action=lemma&lemma=35…

I don't know enough about Freeling development to create a pull request myself, in fact I even seem to recall the file is generated during install(?), however I have made a diff between the installed dicc.src, (Portuguese, Freeling version 4.2.1), and my own where I have corrected it:

232708c232708 < dar dar VMN0000 dar VMN01S0 dar VMN03S0 dar VMSF1S0 dar VMSF3S0 --- > dar dar VMN0000 dar VMN01S0 dar VMN03S0 232831c232831 < dardes dar VMN02P0 dar VMSF2P0 dardar VMM02S0 dardar VMSP2S0 --- > dardes dar VMN02P0 dardar VMM02S0 dardar VMSP2S0 232855c232855 < darem dar VMN03P0 dar VMSF3P0 --- > darem dar VMN03P0 232857c232857 < dares dar VMN02S0 dar VMSF2S0 dares NCMP000 --- > dares dar VMN02S0 dares NCMP000 232864c232864 < darmos dar VMN01P0 dar VMSF1P0 --- > darmos dar VMN01P0 244702a244703 > der dar VMSF1S0 dar VMSF3S0 244711a244713,244715 > derdes dar VMSF2P0 > derem dar VMSF3P0 > deres dar VMSF2S0 244831a244836 > dermos dar VMSF1P0

I hope this comes out correctly in the post. More info: I'm using the Homebrew installation of Freeling on macOS.

these files are generated…

these files are generated during install, putting toghether the entries in data/pt/dictionary/entries/verbs

If you update that file, you can do a pull request

However, make sure you do not remove forms that are valid in some Protuguese variant (Portugal or Brasil)

thanks!

Lemas file

Sebastià Salvà i Puig — Mon, 16 Nov 2020 19:21:36 +0000

Lemas file Sebastià Salvà… Mon, 11/16/2020 - 20:21

Forums

Linguistic Data

Dear all,

Do you know how to get, from a dicc.dat/dicc.txt file, a file with all the lemas and all the tokens forms, in oder to load it in a concordance program such as AntConc?

casa -> casa, cases
menjar -> menjo, menjava, menjaré, mengin...

Could yo recommend me some kind of instruction, such as awk...?

Actually, the proper format…

Actually, the proper format should be:

casa TAB -> TAB casa TAB cases
menjar TAB -> TAB menjo TAB menjava TAB menjaré TAB mengin

(etcetera...)

Once you install FreeLing, a…

Once you install FreeLing, a file is created in /usr/local/share/freeling/ca/dicc.src, which contains something very close to what you ask, and that should be easy to adapt with a simple awk command, a small python program, or even loading the file in a spreadsheet

Yes, an AWK command would be…

Yes, an AWK command would be great. But I have no idea of which would be the one.

AWK is an option for what…

AWK is an option for what toy want to do, not a requirement.

If you don't know AWK, you can use a small python script. Or Perl. Or any other programming language

If you are not a programmer, you probably can achieve the same loading the data in a spreadsheet like excel or openoffice.

Personal articles in Catalan: "En Pere" & "na Maria"

Sebastià Salvà i Puig — Mon, 16 Nov 2020 19:14:22 +0000

Personal articles in Catalan: "En Pere" & "na Maria" Sebastià Salvà… Mon, 11/16/2020 - 20:14

Forums

Linguistic Data

Dear all,

Any idea or suggestion for modifying some catalan FreeLing language DAT file(s) in order to get the proper tokenization and labeling for the personal articles before a proper a name in Catalan?

For instance: "en Pere" should be labeled "en DA0MS | Pere NP00SP0", not as a preposicional phrase ("en SP | Pere NP00SP0"). And "na Maria" should be tokenized in two words and tagged such as "na DA0FS | Maria_NP, not as "na_Maria_NP00SP0" (with a "_" merging the article and the proper name).

The tagger is statistical,…

The tagger is statistical, and the frequency of "en" as a DA is 0.08%, which makes it very unlikely that this reading is selected.

However, you may try biassing that decision as follows:

edit the file /usr/local/share/freeling/ca/tagger.dat, and in section <Forbidden> add a new line:

*.SP<en>.NP

However, be aware that this will cause that "en" is *never* tagged as a preposition before a NP, which migth introduce errors in other sentences.

Regarding "Na_Marta", that happens because the default NE recognizer is also a ML model, and has probably not seen that case often. You can solve that using an alternative NE recognizer: Add "--fnp /usr/local/share/freeling/ca/np.dat" to your analyzer command.

However: If "Na" is capitalized and not at the beginning of the sentence, it will be considered part of the NP by any NER system.

Thanks for your answer…

Thanks for your answer.

Could I also try to introduce this other exception: "*.SP.NP00SP0", in order just to exclude the person proper names (not the place proper names)?

Demo is not working

Sara_Saad — Tue, 24 Jul 2018 13:12:33 +0000

Demo is not working Sara_Saad Tue, 07/24/2018 - 15:12

Forums

Linguistic Data

Hi All,

Please I need your kind assistance as the online demo is not working. I have about 5 days trying and I always have the below error message:

Fatal error: Uncaught Exception: String could not be parsed as XML in /home/operador/public_html/freeling-8.5.0/demo/demo.php:265 Stack trace: #0 /home/operador/public_html/freeling-8.5.0/demo/demo.php(265): SimpleXMLElement->__construct('') #1 {main} thrown in /home/operador/public_html/freeling-8.5.0/demo/demo.php on line 265

would you please advise?

Thanks you!
Sara

Our servers were down some…

Our servers were down some days for summer maintenance.

It should be working now (though there may be other cuts during August)

Example 07: Extracting Triples with Semantic Information

kashyap — Mon, 09 Jul 2018 12:51:18 +0000

Example 07: Extracting Triples with Semantic Information kashyap Mon, 07/09/2018 - 14:51

Forums

Linguistic Data

As a part of my current task, I am trying to feed(/extracted) knowledge(/triples) from Freeling to the Ontology. I did go through Freeling tutorials and I did try to run the example 7(https://talp-upc.gitbooks.io/freeling-tutorial/content/code/example07.p…) but when I run the programme it does gives below error:

FREELINGDIR environment variable not defined, trying /usr/local
Text language is: en
Traceback (most recent call last):
File "pyfreeling_triple_extraction.py", line 172, in
ProcessSentences(ls, sdb)
File "pyfreeling_triple_extraction.py", line 46, in ProcessSentences
if lsubj!="" and ldobj!="" :
UnboundLocalError: local variable 'lsubj' referenced before assignment

Even if I comment those lines and only try to print get_predicates() it gives "Text language is: en" as output nothing else.

Here, goes my code:

#! /usr/bin/python

import pyfreeling
import sys, os

##---------------------------------------------
## Extract lemma and sense of word 'w' and store them
## in 'lem' and 'sens' respectively
##---------------------------------------------
def extract_lemma_and_sense(w) :
lem = w.get_lemma()
sens=""
if len(w.get_senses())>0 :
sens = w.get_senses()[0][0]
return lem, sens

## -----------------------------------------------
## Do whatever is needed with analyzed sentences
## -----------------------------------------------
def ProcessSentences(ls, sdb) :

# for each sentence in list
for s in ls :

# for each predicate in sentence
for pred in s.get_predicates() :
lsubj=""; ssubj=""; ldobj=""; sdobj=""
# for each argument of the predicate
for arg in pred :
# if the argument is A1, store lemma and synset in ldobj, sdobj
if arg.get_role()=="A1" :
(ldobj,sdobj) = extract_lemma_and_sense(s[arg.get_position()])
# if the argument is A0, store lemma and synset in lsubj, subj
elif arg.get_role()=="A0" :
(lsubj,ssubj) = extract_lemma_and_sense(s[arg.get_position()])
# Get tree node corresponding to the word marked as argument head
head = s.get_dep_tree().get_node_by_pos(arg.get_position())
# check if the node has dependency is "by" in passive structure
if lsubj=="by" and head.get_label=="LGS" :
# get first (and only) child, and use it as actual subject
head = head.get_nth_child(0)
(lsubj,ssubj) = extract_lemma_and_sense(head.get_word())

#if the predicate had both A0 and A1, we found a complete SVO triple. Let's output it.
if lsubj!="" and ldobj!="" :
(lpred,spred) = extract_lemma_and_sense(s[pred.get_position()])
# if we found a synset for the predicate, obtain lemma synonyms and SUMO link
if (spred!="") :
ipred = sdb.get_sense_info(spred);
lpred = "/".join(ipred.words) + " [" + ipred.sumo + "]"
# if we found a synset for the subject, obtain lemma synonyms and SUMO link
if (ssubj!="") :
isubj = sdb.get_sense_info(ssubj);
lsubj = "/".join(isubj.words) + " [" + isubj.sumo + "]"

# if we found a synset for the object, obtain lemma synonyms and SUMO link
if (sdobj!="") :
idobj = sdb.get_sense_info(sdobj);
ldobj = "/".join(idobj.words) + " [" + idobj.sumo + "]"

print ("SVO : (pred: " , lpred, "[" + spred + "]")
print (" subject:" , lsubj, "[" + ssubj + "]")
print (" dobject:" , ldobj, "[" + sdobj + "]")
print (" )")

## -----------------------------------------------
## Set desired options for morphological analyzer
## -----------------------------------------------
def my_maco_options(lang) :

lpath = DATA + LANG + "/"

# create options holder
opt = pyfreeling.maco_options(lang);

# Provide files for morphological submodules. Note that it is not
# necessary to set file for modules that will not be used.
opt.UserMapFile = "";
opt.LocutionsFile = lpath + "locucions.dat";
opt.AffixFile = lpath + "afixos.dat";
opt.ProbabilityFile = lpath + "probabilitats.dat";
opt.DictionaryFile = lpath + "dicc.src";
opt.NPdataFile = lpath + "np.dat";
opt.PunctuationFile = lpath + "../common/punct.dat";
return opt;

## ----------------------------------------------
## ------------- MAIN PROGRAM ---------------
## ----------------------------------------------

## Check whether we know where to find FreeLing data files
if "FREELINGDIR" not in os.environ :
if sys.platform == "win32" or sys.platform == "win64" : os.environ["FREELINGDIR"] = "C:\\Program Files"
else : os.environ["FREELINGDIR"] = "/usr/local"
print >> sys.stderr, "FREELINGDIR environment variable not defined, trying ", os.environ["FREELINGDIR"]

if not os.path.exists(os.environ["FREELINGDIR"]+"/share/freeling") :
print >> sys.stderr, "Folder",os.environ["FREELINGDIR"]+"/share/freeling", "not found.\nPlease set FREELINGDIR environment variable to FreeLing installation directory"
sys.exit(1)

# Location of FreeLing configuration files.
DATA = os.environ["FREELINGDIR"]+"/share/freeling/";

# Init locales
pyfreeling.util_init_locale("default");

# create language detector. Used just to show it. Results are printed
# but ignored (after, it is assumed language is LANG)
la=pyfreeling.lang_ident(DATA+"common/lang_ident/ident-few.dat");

# create options set for maco analyzer. Default values are Ok, except for data files.
LANG="en";

op= pyfreeling.maco_options(LANG);
op.set_data_files( "",
DATA + "common/punct.dat",
DATA + LANG + "/dicc.src",
DATA + LANG + "/afixos.dat",
"",
DATA + LANG + "/locucions.dat",
DATA + LANG + "/np.dat",
DATA + LANG + "/quantities.dat",
DATA + LANG + "/probabilitats.dat");

# create analyzers
tk=pyfreeling.tokenizer(DATA+LANG+"/tokenizer.dat");
sp=pyfreeling.splitter(DATA+LANG+"/splitter.dat");
sid=sp.open_session();
mf=pyfreeling.maco(op);

# activate mmorpho odules to be used in next call
mf.set_active_options(False, True, True, True, # select which among created
True, True, False, True, # submodules are to be used.
True, True, True, True ); # default: all created submodules are used

# create tagger, sense anotator, and parsers
tg=pyfreeling.hmm_tagger(DATA+LANG+"/tagger.dat",True,2);
sen=pyfreeling.senses(DATA+LANG+"/senses.dat");
parser= pyfreeling.chart_parser(DATA+LANG+"/chunker/grammar-chunk.dat");
dep=pyfreeling.dep_txala(DATA+LANG+"/dep_txala/dependences.dat", parser.get_start_symbol());

# create semantic DB module
sdb = pyfreeling.semanticDB(DATA+LANG+"/semdb.dat");

# process input text
lin=sys.stdin.readline();

print "Text language is: "+la.identify_language(lin)

while (lin) :

l = tk.tokenize(lin);
ls = sp.split(sid,l,False);

ls = mf.analyze(ls);
ls = tg.analyze(ls);
ls = sen.analyze(ls);
ls = parser.analyze(ls);
ls = dep.analyze(ls);

lin=sys.stdin.readline();

# do whatever is needed with processed sentences
ProcessSentences(ls, sdb)

# clean up
sp.close_session(sid);

it is resolved. but when we…

it is resolved. but when we have a sentence like A0, AM-TMP and AM-LOC it does not generate any output.

Of course it doesn't. The…

Of course it doesn't. The program extracts SVO triples, which means it looks for sentences with A0 *and* A1. If the sentence has only A0, there is no direct object, thus no triple is extracted.

However, this program is just an example. You should adapt the code to your needs to output whatever you want.

I have the same issue here…

I have the same issue here.
get_predicates() method returns an empty list. It has been solved? Do I have to clone and re-compile Freeling again?

get_predicates will only…

get_predicates will only return something if you called the SRL

If you are using version 4.1, that task is performed by dep_treeler module (and not for all languages, so check that your target language has this feature available).

If you are using master version, this has changed, and SRL is a separate module that needs to be called explicitily.

Check details for SRL module in 'master' version of the manual.
https://talp-upc.gitbook.io/freeling-4-1-user-manual/v/master/

Different behavior with Catalan numbers

jmfc90 — Fri, 10 Feb 2017 12:03:44 +0000

Different behavior with Catalan numbers jmfc90 Fri, 02/10/2017 - 13:03

Forums

Linguistic Data

Hello.
I am having some troubles working with numbers in Catalan. In your demo page I use the text "tres mil" with Catalan language and using the default configuration and I get the following output:
tres_mil 3000 Z

But If I use the text "nou mil" with exactly the same configuration I get the output:
nou mil nou 1000 AQ0MS00 Z

Why in the second example did I not get the same output structure with one entity with the complete number (9000) as in the first example?
Thanks.

that is an error

That is caused by "nou" being interpreted as an adjective instead of number.
This happens in version 4.0 stable, which is the one used by the demo,

It is fixed in the latest development version.

NER and IOB sequential classification

flopezbello — Sun, 20 Nov 2016 18:27:32 +0000

NER and IOB sequential classification flopezbello Sun, 11/20/2016 - 19:27

Forums

Linguistic Data

Hello.

Consider the following statement:
"La sentencia dictada por la Sra. Juez Letrado de Primera Instancia de Montevideo"

When you use Freeling to parse for NER/NEC, you get:

which is not correct for line 6. One would expect, for line 6, something like:

Sra. B-PER
Juez I-PER
Letrado I-PER
de I-PER
Primera I-PER
Instancia I-PER
de
Montevideo B-LOC

I've been trying to tune configuration files tw*dat and gen*dat with no luck.

Any ideas? Am I missing something?

Thanks!

FreeLing is a library

Freeling is a library which produces a data structure as a result.
"analyzer" program is just a sample of how to use this library. This program has many options and output formats, but does not have any possible format or imaginable combination.

As written in the manual,

Thus, the question is not why this program doesn't offer functionality X?, why it doesn't output information Y?, or why it doesn't present results in format Z?, but How should I use FreeLing library to write a program that does exactly what I need?.

So, if you need some output or processing not offered by the sample program, you need to write your own main program (or to modify "analyzer" to get it)

In your case, if all you need is breaking NEs, you can write a dummy python script (or perl, awk, or whatever you prefer) to do the job.

Thanks Lluis; actually my

Thanks Lluis; actually my question was towards solving this gap through configuration, but I got your point.

Regards

it is not configurable

Current configuration options do not allow that.
That is why I told you the alternatives ;-)

incorrect language on Semantic Graph Frame lemma

carlesg — Fri, 29 Jul 2016 12:37:58 +0000

incorrect language on Semantic Graph Frame lemma carlesg Fri, 07/29/2016 - 14:37

Forums

Linguistic Data

Hello,

When I use the SemanticGraph to ask for the lemmas of the Frames on the Graph, I get the lemma in English, although I configure Freeling in Spanish, and I ask a question in Spanish.
If I try the Spanish sentence 'Dime el valor del coche.', the tagger says:

-------- TAGGER results -----------
Di decir VMM02S0
me me PP1CS00
el el DA0MS0
valor valor NCMS000
de de SP
el el DA0MS0
coche coche NCMS000
. . Fp

But the semantic Graph says (look at the lemma on Frame F1):
-------- SEMANTIC GRAPH results -----------
ENTITY W2 : me
ENTITY W3 : valor
FRAME F1 : speak.01|talk.01 : 1 : 1 : 00941990-v
ARG A2:Co-Agent : W2
ARG A1:Topic : W3

I'm using a Java class on Windows to invoke freeling via the JNI library.

I I use the online demo, it works right and it says the lemma is 'decir.00'.
Which is the problem? Is there any configuration problem?

By the way, what does the final number means on the frame lemma? (.00, .01...)

that is not the lemma

Those codes are not the lemma, but the semantic code for the verb meaning in propbank (http://propbank.github.io/)

E.g for the verb "bear", propbank has two senses, bear.01 and bear.02:
http://verbs.colorado.edu/propbank/framesets-english-aliases/bear.html

The idea is that you get a language-independent information on the verb frame. So if the sentence was in English instead of Spanish, the verb codes would be the same. Also if the verb was "tell" instead of "say".
In this way, you get a language independent semantic graph that can be usefult to compare texts that express the same meaning using different words, or even in different languages

If you get codes such as "decir.00" is because it could not disambiguate properly or the sense was not found in propbank (that is why you get ".00", which is not in propbank)

If you want to recover the lemma for a frame in the graph, you need to be aware that the results as presented by analyzer are very simple.
If you use a json or XML output you will have the full graph, with links among components (e.g. in the semantic graph frame you will have the semantic code, but if you want the lemma, the frame will contain the ID for the token that originated the frame, which will contain the lemma).
If you call the library yourself, you can navigate the document data structure to locate the token that originated the frame and find out its lemma.

Thank you.

Thank you.
So the next question is, why does the demo shows different semantic code for the same word from the one I get in a local execution? Why does the demo can not disambiguate the meaning, but local execution can do that, in the same sentence?

And the second question is,
How can I get the lemma from root word in the frame (in this case, 'Dime'->'decir') traversing the semantic graph? (the last option you say) How can I locate the original Word that originate that token? Which is the matching id?
Is there any way to map the objects in the Semantic graph (doc.getSemanticGraph().getEntities() and doc.getSemanticGraph().getFrames() (root) and doc.getSemanticGraph().getFrames().getArguments()) to the objects in the Trees or Lists (Word, etc.) ?

The demo is probably not

The demo is probably not running the same revision, but an older one, so there may be some differences.

For The sentence "Dime el valor del coche." you get the semantic graph:
<semantic_graph>
<entity id="W2" lemma="me">
<mention id="t1.2" words="me"/>
</entity>
<entity id="W3" lemma="valor" sense="05856388-n">
<mention id="t1.4" words="el valor de el coche"/>
<synonym lemma="valor"/>
<URI URI="http://wordnet-rdf.princeton.edu/wn30/05856388-n" knowledgeBase="WordNet"/>
<URI URI="http://ontologyportal.org/SUMO.owl#Quantity" knowledgeBase="SUMO"/>
</entity>
<frame id="F1" lemma="decir.00" sense="" token="t1.1">
<argument entity="W2" role="A2"/>
<argument entity="W3" role="A1"/>
</frame>
</semantic_graph>

You can see that the frame "F1" corresponds to token "t1.1"

Then, you can navigate the XML tree looking for a "<token>" with id="t1.1" and you will get
<token ctag="VMM" form="Di" id="t1.1" lemma="decir" mood="imperative" num="singular" person="2" pos="verb" tag="VMM02S0" type="main">

where you can extract that "lemma" is "decir"

If you are not using XML, but directly calling the library, you need to go to token 1 in sentence 1 of the document (that is what t1.1 means)