quantization for transformer based models

[abhinaynagpal]

Hi

I have been using the transformer based models (pertained shipped with spacy) (primarily for NER), and need to improve upon the inference speed.

1. Is it possible to use quantization:
   Due tot the thinc wrapper I was not able to figure out how to load the pytorch model and use quantization, any pointers will help
   https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html

2. Use turbotransformers inference engine
   https://github.com/Tencent/TurboTransformers
   Any pointers on how I could use this would be very helpful!

Thanks!

[abhinaynagpal]

I was able to get it working by updating spacy_transformers/pipeline_component.py and it gives a 20-25% improvement in inference speed , of course this might cause loss of precision so we need to make it configurable.

def from_disk(
        self, path: Union[str, Path], *, exclude: Iterable[str] = tuple()
    ) -> "Transformer":
    def load_model(p):
            try:
                with open(p, "rb") as mfile:
                    self.model.from_bytes(mfile.read())
                    import torch
                    qconfig_dict = {
                      torch.nn.EmbeddingBag : torch.quantization.float_qparams_weight_only_qconfig,
                      torch.nn.Linear: torch.quantization.default_dynamic_qconfig
                    }
       
                    self.model._transformer = torch.quantization.quantize_dynamic(self.model.transformer, qconfig_dict)
             

[rennanvoa2]

I was able to get it working by updating spacy_transformers/pipeline_component.py and it gives a 20-25% improvement in inference speed , of course this might cause loss of precision so we need to make it configurable.

def from_disk(
        self, path: Union[str, Path], *, exclude: Iterable[str] = tuple()
    ) -> "Transformer":
    def load_model(p):
            try:
                with open(p, "rb") as mfile:
                    self.model.from_bytes(mfile.read())
                    import torch
                    qconfig_dict = {
                      torch.nn.EmbeddingBag : torch.quantization.float_qparams_weight_only_qconfig,
                      torch.nn.Linear: torch.quantization.default_dynamic_qconfig
                    }
       
                    self.model._transformer = torch.quantization.quantize_dynamic(self.model.transformer, qconfig_dict)
             

Is it possible to mutate the Spacy model in runtime (inference) to add the quantization or the only way is to change the code in the pipeline_component file?

[danieldk]

Thanks for reporting that this works and benchmarking the performance improvement!

Quantization is certainly something we want to support out of the box in the future. We are working on support for model distillation to be able extract smaller and faster models from large transformer models. We also plan to experiment with quantization during distillation, which may result in a lower impact on accuracy than quantizing the model after training, since the model is then optimized with lower precision.

[imohitmayank]

+1 for the quantization and distillation features. Do we have anything yet?

[danieldk]

It's on our radar. Since distillation is computationally-intensive, we first focused on removing various training and inference bottlenecks. Many of them shipped in recent Thinc and spaCy versions and we have some more in the pipeline for the next releases.

[rennanvoa2]

@danieldk Any news about the quantization support? I am looking forward to using it with Spacy :D

[saitej123]

Want to use pretrained model trf

[saitej123]

Document size: 40 sentences and 10k chars

[danieldk]

We provide beta models for curated transformers here:

#12848

Like our current transformer models, not all models support NER, but for instance the English model does. It's probably best to benchmark the model for your particular data and hardware.

Circling back to the original topic: these models are not quantized. We do support quantization, but decided to disable it for the upcoming 3.7 release. Curated Transformers originally used TorchScript scripting, but we decided to switch to tracing and we have to redo the quantization support around that.

[saitej123]

How to get speed of spacy LG model for trf? Wps is more for lg model ...we have been using trf model for production but processing very less articles because of inference speed ... please share hacks to accelerate this process

[danieldk]

The transformer models will be slower than our convolution models (sm/md/lg), because they have to do more computation. If you want to use the transformer models and speed is an issue, you should consider using GPUs for inference.

[celsowm]

any news about quantization ? my 3060ti is not able to run mistral without 8bit mode