Hyphenation exeption list

[KariRudjord]

Make a solution for hyphen exeptions, e.g. (never to hyphenate the word)
de-tte
de-nne
di-sse
kj-eks

[bertfrees]

The method that we used before to enter exception words required that the base table did not contain any patterns with 9 in it. This is because 9 is the highest priority, so it's not possible to undo certain hyphenation points with exceptions. It might work fine but you must realize it's not guaranteed.

There are of course other ways to implement an exception list. For example by checking for exception words before passing words to libhyphen. That sounds very straightforward but is not so great for a number of reasons so we have to think it through.

By the way, I noticed that you inserted your exception list of non-standard hyphenations in hyph_nb_NO.dic directly, whereas we used to concat the two files while building. Not really anything wrong with that, except that there is duplication now.

One thing that bothers me is that running substrings.pl on your new hyph_nb_NO.dic yields changes to the file. I expected it to add some new patterns because of the addition of the non-standard hyphenation patterns, but I didn't expect that it would change the whole file (and not only the ordering). As far as I understand that means that the file was not prepared correctly.

Finally, what is that last line: "Binærfil (standard inndata) samsvarer"?

[josteinaj]

Would it be possible to for instance replace all the standard hyphenation 8s with 7, replace all the non-standard hyphenation 9s with 8, and then use 9 for the exception words? Do you think that would break things?

We could go back to how they were concatenated before. I didn't think about that.

Yeah I have no idea how it was prepared, only that it works better than what we had before.

Oh, whoops. "Binærfil (standard inndata) samsvarer" is from a diff and means "Binary file (standard in) matches" (or however it's formulated in the english locale). I suppose I performed a diff at some point. That should be removed.

[bertfrees]

The fact that the non-standard patterns have 9 in them is not a problem because these already represent full words, so they can't and don't need to be overwritten by exceptions.

The problem is that the base table has 9 in it as well. You can't change any of the numbers because all levels are taken.

[josteinaj]

Yeah. I don't know how the implementation works but would it be a problem to move patterns with 9s to 8 even though there already are other patterns using 8? I would expect the behavior to change slightly but maybe not that much?

[bertfrees]

I think you need to read this.

[bertfrees]

As always I think it would be better to start from the source code if possible instead of patching up the resulting table. It was the case when we used the "spell-norwegian" project and it is the case now. Of course there needs to be source code in the first place, and you need to be able to get hold of it. As you know it is not always possible to get in touch with the author. There are a bunch of people mentioned in the header of the No.pm file of the Text-Hyphen-No project, but it is not immediately clear who you would contact.

By the way maybe you should include the header in our copy too. In case anyone else stumbles upon our copy it would be helpful for them.

One thing that bothers me is that running substrings.pl on your new hyph_nb_NO.dic yields changes to the file.

Thinking about this again, this is of course because the patterns in the Text-Hyphen-No project are TeX patterns, not Hyphen patterns. The Text::Hyphen perl module has nothing to do with the Hyphen library.

[josteinaj]

I grep'ed all the files in spell-norwegian-2.2.tar.gz and found e-mail addresses for three norwegians. I sent a request to each of them now. 🤞.

[josteinaj]

I got a response from Rune Kleveland (translated by me):

The hyphenation files are made with a program called patgen which is bundled with TeX. There are many rules that can be used to generate patterns in the Makefile in the patterns folder in the file

https://sourceforge.net/projects/spell-no/files/ispell-norsk/ispell-norsk%202.0/

To build the list from source you need a lot of things, among other things ispell. There are many complicated rules in the Makefile, and you'll probably have to use Linux for it to work.

The challenge with hyphenation is mainly compound words. To improve this one needs several compound words that are actually used. If you add it to the list you'll get better patterns with the build scripts in the distribution.

There are two levels for hyphenation. One file that divides word into compounds/parts (barnehage-tante, barne-hage), and one that divides into components (bar-ne-hage-tan-te). This can be used to collect compund words because the rules that divide into compound words can be used to divide compound words that can be collected. And then you'll have to avoid sydame-rikaner, pils-piss and similar divisions.

There is actually quite a bit of documentation in the file. And there are newer versions as well, but I don't really think they have worked a lot with hyphenation.

I've downloaded the CVS repo for spell-no from sourceforge, converted it to git, and uploaded it here in case we want to make changes to it:

https://github.com/nlbdev/spell-no

So as I understand it, it's the norsk.words file that is the input (which we already tried), but we need to get the Makefile working so that it generates the word components.

[bertfrees]

Cool! Yes, this is kind of what I remember from reading the README some years ago. I never tried to build it because the Makefile seemed super complicated. But maybe now that we can ask Rune Kleveland for help we should give it a try.

Do you know what the connection is between this project and the TeX file that you found in the Perl project?

[bertfrees]

The things we should definitely try to do after we manage to build it are:

• To include rules in the Makefile to build a version of the table for Libhyphen. It looks like there is already a "hyph_no_NO.dic" rule for OpenOffice.org but it there is no trace of the substrings.pl tool. Maybe this is from before the time that OpenOffice.org started using Libhyphen? Needs to be investigated.
• To add non-standard hyphenation patterns.
• Would be nice to have some tests.

[bertfrees]

The Makefile in the spell-norwegian-2.1 project looks a bit newer than the one in spell-no that you got from CVS.

[bertfrees]

I've been looking at the code a bit. The part of the build that interests us the most is in the patterns subdir. It says:

The purpose of this script is to generate hyphenation patterns for use with TeX based on a dictionary hyphenated at compound points and a pattern file which handles non-compound words.

In other words, it looks like the basis of the whole thing is an existing patterns file. But the patterns themselves are not used in the final output, rather they are used to hyphenate the word list, and from this new patterns are generated. This means we could add our own exception words to the process. Also we can decide the hyphen levels in such a way that at least one level is available to add the non-standard patterns at the end. (We can't add them sooner in the process because this part is specific to Libhyphen.)

The README suggests two solutions for when hyphenation fails on words (not in the dictionary). The first one I don't fully understand. The second solution basically adds the word to a list of exceptions which are checked at runtime (TeX's \hyphenation{...} command). I'd much rather generate proper patterns though, and it looks like this is possible.

This project solves non-standard hyphenation too, but apparently it is done via some TeX configuration file. I don't think we have to reevaluate our approach, but it's something to keep in mind.

At the top of the Makefile it says that you need a patgen with enough capacity. Luckily we can build patgen from source in case we need to make adjustments to some parameters in the code.

[josteinaj]

Do you know what the connection is between this project and the TeX file that you found in the Perl project?

No, I don't.

Maybe we could add a Dockerfile to https://github.com/nlbdev/spell-no with a build environment?

[bertfrees]

OK, sure. Although the build prerequisites are almost non-existing. You only need common Unix tools like awk, sed, gzip, etc., and we will probably build patgen ourselves, and the patgen build is also fully self-contained.

By the way I compared the "patterns" directories inside the spell-no repo and the newer spell-norwegian-2.1 and the conclusion is that the differences are negligible, so we can proceed with the spell-no repo.

[bertfrees]

The norsk.words file however has become much bigger in spell-norwegian-2.1, so it's a good idea to update it in spell-no.

[bertfrees]

I have a good understanding now of how the build works. I had to do a few modifications in order to get it working on my Mac OS machine and with my version of patgen. I also had to add a rule to create a patterns file for Libhyphen. Before I proceed with the other planned changes, like adding exception words, support for non-standard hyphenation, etc., we should discuss a few things (see Slack).

[bertfrees]

I did a lot of work for this issue last year in December, but there is still some work to do before we can use it in Pipeline notably:

• Add non-standard hyphenation

I also would like to:

• Add tests to the build
• Possibly update to the "spell-norwegian-2.1" version

Has anyone tried my tool to check for mistakes or missing words in the norsk.words and norsk.singlewords files?

[josteinaj]

I really like the tool you made. One thing is that we need to be sure that we keep it in sync with our latest build, in case we make changes to the norsk.words/norsk.singlewords/etc. files. Maybe it could be built together with the rest of the system (maybe it already is, I haven't looked at it since last year), and then I could expose the file on the docker container running Pipeline 2 here. Ideally, it would be almost interactive (edit file - check results - edit again - etc.), but I suspect it's not worth spending time on that. Non-standard hyphenation is something we certainly need yes.