folia-idf yielded empty output

[pirolen]

I got empty output when running folia-idf in the foliautils container.

/data # FoLiA-idf --class FoLiA-txt  -O myidf bla 
Processed :bla/TRAINING_VALIDATION_SET_Combined_VKS_2_Silvestrovskij_0_01_GT.folia.xml with 0 unique words. still 0 files to go.
start calculating the results
created IDF list 'myidf.idf.tsv'
done: 

[kosloot]

@pirolen it would be helpful if you could send me the file:
bla/TRAINING_VALIDATION_SET_Combined_VKS_2_Silvestrovskij_0_01_GT.folia.xml
so I can try to reproduce the problem

(does it indeed have words with class=FoLiA-txt ?)

[pirolen]

Here you are.
Yes, it has that text class, since the file was generated using FoLiA-txt.

TRAINING_VALIDATION_SET_Combined_VKS_2_Silvestrovskij_0_01_GT.folia.xml.txt

[kosloot]

AHA.
FoLiA-idf extracts its information from either <w> nodes or <str> nodes.
In this case you have no <w>, but you DO have <str>.
To handle this, you must provide the --strings option:

$ FoLiA-idf -O grr --strings --class=FoLiA-txt IDFbug.xml 
Processed :IDFbug.xml with 5355 unique words. still 0 files to go.

Hope this helps.

[pirolen]

Brilliant, thanks! Will try out asap, right now I can't...

[kosloot]

Just a tought: I think you should be better off running ucto on this file first.
That will create <w> nodes AND remove unwanted hyphens. (And also handle punctuation)

NOTE: ucto will by default create words with <t> nodes in textclass current, so you will need to specify that class when running FoLiA-idf
It does not use current as the default. (which is a bug. fixed now in Git)