Skip to content

Commit

Permalink
Docs for v3.6.0 (#12792)
Browse files Browse the repository at this point in the history
* Docs for v3.6.0

* Add sl performance

* Add da trf note
  • Loading branch information
adrianeboyd committed Jul 6, 2023
1 parent 830dcca commit 4e19ec7
Show file tree
Hide file tree
Showing 4 changed files with 154 additions and 5 deletions.
143 changes: 143 additions & 0 deletions website/docs/usage/v3-6.mdx
@@ -0,0 +1,143 @@
---
title: What's New in v3.6
teaser: New features and how to upgrade
menu:
- ['New Features', 'features']
- ['Upgrading Notes', 'upgrading']
---

## New features {id="features",hidden="true"}

spaCy v3.6 adds the new [`SpanFinder`](/api/spanfinder) component to the core
spaCy library and new trained pipelines for Slovenian.

### SpanFinder {id="spanfinder"}

The [`SpanFinder`](/api/spanfinder) component identifies potentially
overlapping, unlabeled spans by identifying span start and end tokens. It is
intended for use in combination with a component like
[`SpanCategorizer`](/api/spancategorizer) that may further filter or label the
spans. See our
[Spancat blog post](https://explosion.ai/blog/spancat#span-finder) for a more
detailed introduction to the span finder.

To train a pipeline with `span_finder` + `spancat`, remember to add
`span_finder` (and its `tok2vec` or `transformer` if required) to
`[training.annotating_components]` so that the `spancat` component can be
trained directly from its predictions:

```ini
[nlp]
pipeline = ["tok2vec","span_finder","spancat"]

[training]
annotating_components = ["tok2vec","span_finder"]
```

In practice it can be helpful to initially train the `span_finder` separately
before [sourcing](/usage/processing-pipelines#sourced-components) it (along with
its `tok2vec`) into the `spancat` pipeline for further training. Otherwise the
memory usage can spike for `spancat` in the first few training steps if the
`span_finder` makes a large number of predictions.

### Additional features and improvements {id="additional-features-and-improvements"}

- Language updates:
- Add initial support for Malay.
- Update Latin defaults to support noun chunks, update lexical/tokenizer
settings and add example sentences.
- Support `spancat_singlelabel` in `spacy debug data` CLI.
- Add `doc.spans` rendering to `spacy evaluate` CLI displaCy output.
- Support custom token/lexeme attribute for vectors.
- Add option to return scores separately keyed by component name with
`spacy evaluate --per-component`, `Language.evaluate(per_component=True)` and
`Scorer.score(per_component=True)`. This is useful when the pipeline contains
more than one of the same component like `textcat` that may have overlapping
scores keys.
- Typing updates for `PhraseMatcher` and `SpanGroup`.

## Trained pipelines {id="pipelines"}

### New trained pipelines {id="new-pipelines"}

v3.6 introduces new pipelines for Slovenian, which use the trainable lemmatizer
and [floret vectors](https://github.com/explosion/floret).

| Package | UPOS | Parser LAS | NER F |
| ------------------------------------------------- | ---: | ---------: | ----: |
| [`sl_core_news_sm`](/models/sl#sl_core_news_sm) | 96.9 | 82.1 | 62.9 |
| [`sl_core_news_md`](/models/sl#sl_core_news_md) | 97.6 | 84.3 | 73.5 |
| [`sl_core_news_lg`](/models/sl#sl_core_news_lg) | 97.7 | 84.3 | 79.0 |
| [`sl_core_news_trf`](/models/sl#sl_core_news_trf) | 99.0 | 91.7 | 90.0 |

### Pipeline updates {id="pipeline-updates"}

The English pipelines have been updated to improve handling of contractions with
various apostrophes and to lemmatize "get" as a passive auxiliary.

The Danish pipeline `da_core_news_trf` has been updated to use
[`vesteinn/DanskBERT`](https://huggingface.co/vesteinn/DanskBERT) with
performance improvements across the board.

## Notes about upgrading from v3.5 {id="upgrading"}

### SpanGroup spans are now required to be from the same doc {id="spangroup-spans"}

When initializing a `SpanGroup`, there is a new check to verify that all added
spans refer to the current doc. Without this check, it was possible to run into
string store or other errors.

One place this may crop up is when creating `Example` objects for training with
custom spans:

```diff
doc = Doc(nlp.vocab, words=tokens) # predicted doc
example = Example.from_dict(doc, {"ner": iob_tags})
# use the reference doc when creating reference spans
- span = Span(doc, 0, 5, "ORG")
+ span = Span(example.reference, 0, 5, "ORG")
example.reference.spans[spans_key] = [span]
```

### Pipeline package version compatibility {id="version-compat"}

> #### Using legacy implementations
>
> In spaCy v3, you'll still be able to load and reference legacy implementations
> via [`spacy-legacy`](https://github.com/explosion/spacy-legacy), even if the
> components or architectures change and newer versions are available in the
> core library.
When you're loading a pipeline package trained with an earlier version of spaCy
v3, you will see a warning telling you that the pipeline may be incompatible.
This doesn't necessarily have to be true, but we recommend running your
pipelines against your test suite or evaluation data to make sure there are no
unexpected results.

If you're using one of the [trained pipelines](/models) we provide, you should
run [`spacy download`](/api/cli#download) to update to the latest version. To
see an overview of all installed packages and their compatibility, you can run
[`spacy validate`](/api/cli#validate).

If you've trained your own custom pipeline and you've confirmed that it's still
working as expected, you can update the spaCy version requirements in the
[`meta.json`](/api/data-formats#meta):

```diff
- "spacy_version": ">=3.5.0,<3.6.0",
+ "spacy_version": ">=3.5.0,<3.7.0",
```

### Updating v3.5 configs

To update a config from spaCy v3.5 with the new v3.6 settings, run
[`init fill-config`](/api/cli#init-fill-config):

```cli
$ python -m spacy init fill-config config-v3.5.cfg config-v3.6.cfg
```

In many cases ([`spacy train`](/api/cli#train),
[`spacy.load`](/api/top-level#spacy.load)), the new defaults will be filled in
automatically, but you'll need to fill in the new settings to run
[`debug config`](/api/cli#debug) and [`debug data`](/api/cli#debug-data).
9 changes: 7 additions & 2 deletions website/meta/languages.json
Expand Up @@ -222,7 +222,9 @@
},
{
"code": "la",
"name": "Latin"
"name": "Latin",
"example": "In principio creavit Deus caelum et terram.",
"has_examples": true
},
{
"code": "lb",
Expand Down Expand Up @@ -339,7 +341,10 @@
},
{
"code": "sl",
"name": "Slovenian"
"name": "Slovenian",
"example": "France Pre拧eren je umrl 8. februarja 1849 v Kranju",
"has_examples": true,
"models": ["sl_core_news_sm", "sl_core_news_md", "sl_core_news_lg", "sl_core_news_trf"]
},
{
"code": "sq",
Expand Down
3 changes: 2 additions & 1 deletion website/meta/sidebars.json
Expand Up @@ -14,7 +14,8 @@
{ "text": "New in v3.2", "url": "/usage/v3-2" },
{ "text": "New in v3.3", "url": "/usage/v3-3" },
{ "text": "New in v3.4", "url": "/usage/v3-4" },
{ "text": "New in v3.5", "url": "/usage/v3-5" }
{ "text": "New in v3.5", "url": "/usage/v3-5" },
{ "text": "New in v3.6", "url": "/usage/v3-6" }
]
},
{
Expand Down
4 changes: 2 additions & 2 deletions website/src/templates/index.js
Expand Up @@ -58,8 +58,8 @@ const AlertSpace = ({ nightly, legacy }) => {
}

const navAlert = (
<Link to="/usage/v3-5" noLinkLayout>
<strong>馃挜 Out now:</strong> spaCy v3.5
<Link to="/usage/v3-6" noLinkLayout>
<strong>馃挜 Out now:</strong> spaCy v3.6
</Link>
)

Expand Down

0 comments on commit 4e19ec7

Please sign in to comment.