Intel® Extension for PyTorch* v2.3.0+cpu Release Notes

We are excited to announce the release of Intel® Extension for PyTorch* 2.3.0+cpu which accompanies PyTorch 2.3. This release mainly brings you the new feature on Large Language Model (LLM) called module level LLM optimization API, which provides module level optimizations for commonly used LLM modules and functionalities, and targets to optimize customized LLM modeling for scenarios like private models, self-customized models, LLM serving frameworks, etc. This release also extends the list of optimized LLM models to a broader level and includes a set of bug fixing and small optimizations. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try this release and feedback as to improve further on this product.

Highlights

• Large Language Model (LLM) optimization

  Intel® Extension for PyTorch* provides a new feature called module level LLM optimization API, which provides module level optimizations for commonly used LLM modules and functionalities. LLM creators can then use this new API set to replace related parts in models by themselves, with which to reach peak performance.

  There are 3 categories of module level LLM optimization APIs in general:

  • Linear post-op APIs

  # using module init and forward
  ipex.llm.modules.linearMul
  ipex.llm.modules.linearGelu
  ipex.llm.modules.linearNewGelu
  ipex.llm.modules.linearAdd
  ipex.llm.modules.linearAddAdd
  ipex.llm.modules.linearSilu
  ipex.llm.modules.linearSiluMul
  ipex.llm.modules.linear2SiluMul
  ipex.llm.modules.linearRelu

  • Attention related APIs

  # using module init and forward
  ipex.llm.modules.RotaryEmbedding
  ipex.llm.modules.RMSNorm
  ipex.llm.modules.FastLayerNorm
  ipex.llm.modules.VarlenAttention
  ipex.llm.modules.PagedAttention
  ipex.llm.modules.IndirectAccessKVCacheAttention
  
  # using as functions
  ipex.llm.functional.rotary_embedding
  ipex.llm.functional.rms_norm
  ipex.llm.functional.fast_layer_norm
  ipex.llm.functional.indirect_access_kv_cache_attention
  ipex.llm.functional.varlen_attention

  • Generation related APIs

  # using for optimizing huggingface generation APIs with prompt sharing
  ipex.llm.generation.hf_beam_sample
  ipex.llm.generation.hf_beam_search
  ipex.llm.generation.hf_greedy_search
  ipex.llm.generation.hf_sample

  More detailed introduction on how to apply this API set and example code walking you through can be found here.

• Bug fixing and other optimization

  • Optimized the performance of LLM #2561 #2584 #2617 #2663 #2733
  • Supported Act Order of GPTQ #2550 #2568
  • Improved the warning and the logging information for better user experience #2641 #2675
  • Added TorchServe CPU Example #2613
  • Upgraded oneDNN to v3.4.1 #2747
  • Misc fix and enhancement #2468 #2627 #2631 #2704

Full Changelog: v2.2.0+cpu...v2.3.0+cpu

Intel® Extension for PyTorch* v2.1.30+xpu Release Notes

2.1.30+xpu

We are excited to announce the release of Intel® Extension for PyTorch* v2.1.30+xpu. This is an update release which supports Intel® GPU platforms (Intel® Data Center GPU Flex Series, Intel® Data Center GPU Max Series and Intel® Arc™ A-Series Graphics) based on PyTorch* 2.1.0.

Highlights

• Intel® oneDNN v3.4.1 integration

• Intel® oneAPI Base Toolkit 2024.1 compatibility

• Large Language Model (LLM) optimizations for FP16 inference on Intel® Data Center GPU Max Series (Beta): Intel® Extension for PyTorch* provides a lot of specific optimizations for LLM workloads in this release on Intel® Data Center GPU Max Series. In operator level, we provide highly efficient GEMM kernel to speed up Linear layer and customized fused operators to reduce HBM access/kernel launch overhead. To reduce memory footprint, we define a segment KV Cache policy to save device memory and improve the throughput. Such optimizations are added in this release to enhance existing optimized LLM FP16 models and more Chinese LLM models such as Baichuan2-13B, ChatGLM3-6B and Qwen-7B.

• LLM optimizations for INT4 inference on Intel® Data Center GPU Max Series and Intel® Arc™ A-Series Graphics (Prototype): Intel® Extension for PyTorch* shows remarkable performance when executing LLM models on Intel® GPU. However, deploying such models on GPUs with limited resources is challenging due to their high computational and memory requirements. To achieve a better trade-off, a low-precision solution, e.g., weight-only-quantization for INT4 is enabled to allow Llama 2-7B, GPT-J-6B and Qwen-7B to be executed efficiently on Intel® Arc™ A-Series Graphics. The same optimization makes INT4 models achieve 1.5x speeded up in total latency performance compared with FP16 models with the same configuration and parameters on Intel® Data Center GPU Max Series.

• Opt-in collective performance optimization with oneCCL Bindings for Pytorch*: This opt-in feature can be enabled by setting TORCH_LLM_ALLREDUCE=1 to provide better scale-up performance by enabling optimized collectives such as allreduce, allgather, reducescatter algorithms in Intel® oneCCL. This feature requires XeLink enabled for cross-cards communication.

Known Issues

Please refer to Known Issues webpage.

Intel® Extension for PyTorch* v2.1.20+xpu Release Notes

2.1.20+xpu

We are excited to announce the release of Intel® Extension for PyTorch* v2.1.20+xpu. This is a minor release which supports Intel® GPU platforms (Intel® Data Center GPU Flex Series, Intel® Data Center GPU Max Series and Intel® Arc™ A-Series Graphics) based on PyTorch* 2.1.0.

Highlights

• Intel® oneAPI Base Toolkit 2024.1 compatibility
• Intel® oneDNN v3.4 integration
• LLM inference scaling optimization based on Intel® oneCCL 2021.12 (Prototype)
• Bug fixing and other optimization
  • Uplift XeTLA to v0.3.4.1 #3696
  • [SDP] Fallback unsupported bias size to native impl #3706
  • Error handling enhancement #3788, #3841
  • Fix beam search accuracy issue in workgroup reduce #3796
  • Support int32 index tensor in index operator #3808
  • Add deepspeed in LLM dockerfile #3829
  • Fix batch norm accuracy issue #3882
  • Prebuilt wheel dockerfile update #3887, #3970
  • Fix windows build failure with Intel® oneMKL 2024.1 in torch_patches #18
  • Fix FFT core dump issue with Intel® oneMKL 2024.1 in torch_patches #20, #21

Known Issues

Please refer to Known Issues webpage.

Intel® Extension for PyTorch* v2.2.0+cpu Release Notes

We are excited to announce the release of Intel® Extension for PyTorch* 2.2.0+cpu which accompanies PyTorch 2.2. This release mainly brings in our latest optimization on Large Language Model (LLM) including new dedicated API set (ipex.llm), new capability for auto-tuning accuracy recipe for LLM, and a broader list of optimized LLM models, together with a set of bug fixing and small optimization. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try this release and feedback as to improve further on this product.

Highlights

• Large Language Model (LLM) optimization

  Intel® Extension for PyTorch* provides a new dedicated module, ipex.llm, to host for Large Language Models (LLMs) specific APIs. With ipex.llm, Intel® Extension for PyTorch* provides comprehensive LLM optimization cross various popular datatypes including FP32/BF16/INT8/INT4. Specifically for low precision, both SmoothQuant and Weight-Only quantization are supported for various scenarios. And user can also run Intel® Extension for PyTorch* with Tensor Parallel to fit in the multiple ranks or multiple nodes scenarios to get even better performance.

  A typical API under this new module is ipex.llm.optimize, which is designed to optimize transformer-based models within frontend Python modules, with a particular focus on Large Language Models (LLMs). It provides optimizations for both model-wise and content-generation-wise. ipex.llm.optimize is an upgrade API to replace previous ipex.optimize_transformers, which will bring you more consistent LLM experience and performance. Below shows a simple example of ipex.llm.optimize for fp32 or bf16 inference:

  import torch
  import intel_extension_for_pytorch as ipex
  import transformers
  
  model= transformers.AutoModelForCausalLM(model_name_or_path).eval()
  
  dtype = torch.float # or torch.bfloat16
  model = ipex.llm.optimize(model, dtype=dtype)
  
  model.generate(YOUR_GENERATION_PARAMS)

  More examples of this API can be found at LLM optimization API.

  Besides the new optimization API for LLM inference, Intel® Extension for PyTorch* also provides new capability for users to auto-tune a good quantization recipe for running SmoothQuant INT8 with good accuracy. SmoothQuant is a popular method to improve the accuracy of int8 quantization. The new auto-tune API allows automatic global alpha tuning, and automatic layer-by-layer alpha tuning provided by Intel® Neural Compressor for the best INT8 accuracy. More details can be found at SmoothQuant Recipe Tuning API Introduction.

  Intel® Extension for PyTorch* newly optimized many more LLM models including more llama2 variance like llama2-13b/llama2-70b, encoder-decoder model like T5, code generation models like starcoder/codegen, and more like Baichuan, Baichuan2, ChatGLM2, ChatGLM3, mistral, mpt, dolly, etc.. A full list of optimized models can be found at LLM Optimization.

• Bug fixing and other optimization

  • Further optimized the performance of LLMs #2349 #2412, #2469, #2476
  • Optimized the Flash Attention Operator #2317 #2334 #2392 #2480
  • Fixed the static quantization of the ELSER model #2491
  • Switched deepspeed to the public release version on PyPI #2473 #2511
  • Upgrade oneDNN to v3.3.4 #2433

Full Changelog: v2.1.100+cpu...v2.2.0+cpu

Intel® Extension for PyTorch* v2.1.100+cpu Release Notes

Highlights

• Improved the performance of BF16 LLM generation inference: #2253 #2251 #2236 #2278
• Added the optimization for Codegen: #2257
• Provided the dockerfile and updated the related doc to improve the UX for LLM users: #2229 #2195 #2299 #2315 #2283
• Improved the accuracy of the quantization path of LLMs: #2280 #2292 #2275 #2319
• Misc fix and enhancement: #2198 #2264 #2290

Full Changelog: v2.1.0+cpu...v2.1.100+cpu

Intel® Extension for PyTorch* v2.1.10+xpu Release Notes

2.1.10+xpu

We are pleased to announce the release of Intel® Extension for PyTorch* v2.1.10+xpu. This is the new Intel® Extension for PyTorch* release supports both CPU platforms and GPU platforms (Intel® Data Center GPU Flex Series, Intel® Data Center GPU Max Series and Intel® Arc™ A-Series Graphics) based on PyTorch* 2.1.0. It extends PyTorch* 2.1.0 with up-to-date features and optimizations on xpu for an extra performance boost on Intel hardware. Optimizations take advantage of AVX-512 Vector Neural Network Instructions (AVX512 VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel Xe Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, through PyTorch* xpu device, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs with PyTorch*.

Please refer to the Installation Guide for the system requirements and steps to install and use Intel® Extension for PyTorch* 2.1.10+xpu. For more detailed tutorials and documentations describing features, APIs and technical details, please refer to Intel® Extension for PyTorch* Documentation.

Highlights

This release provides the following features:

• Large Language Model (LLM) optimizations for FP16 inference on Intel® Data Center GPU Max Series (Experimental): Intel® Extension for PyTorch* provides a lot of specific optimizations for LLM workloads on Intel® Data Center GPU Max Series in this release. In operator level, we provide highly efficient GEMM kernel to speedup Linear layer and customized fused operators to reduce HBM access and kernel launch overhead. To reduce memory footprint, we define a segment KV Cache policy to save device memory and improve the throughput. To better trade-off the performance and accuracy, low-precision solution e.g., weight-only-quantization for INT4 is enabled. Besides, tensor parallel can also be adopted to get lower latency for LLMs.

  • A new API function, ipex.optimize_transformers, is designed to optimize transformer-based models within frontend Python modules, with a particular focus on LLMs. It provides optimizations for both model-wise and content-generation-wise. You just need to invoke the ipex.optimize_transformers API instead of the ipex.optimize API to apply all optimizations transparently. More detailed information can be found at Large Language Model optimizations overview.
  • A typical usage of this new feature is quite simple as below:

    import torch
    import intel_extension_for_pytorch as ipex
    ...
    model = ipex.optimize_transformers(model, dtype=dtype)
    

• Torch.compile functionality on Intel® Data Center GPU Max Series (Experimental): Extends Intel® Extension for PyTorch* capabilities to support torch.compile APIs on Intel® Data Center GPU Max Series. And provides Intel GPU support on top of Triton* compiler to reach competitive performance speed-up over eager mode by default "inductor" backend of Intel® Extension for PyTorch*.

• Intel® Arc™ A-Series Graphics on WSL2, native Windows and native Linux are officially supported in this release. Intel® Arc™ A770 Graphic card has been used as primary verification vehicle for product level test.

• Other features are listed as following, more detailed information can be found in public documentation:

  • FP8 datatype support (Experimental): Add basic data type and FP8 Linear operator support based on emulation kernel.
  • Kineto Profiling (Experimental): An extension of PyTorch* profiler for profiling operators on Intel® GPU devices.
  • Fully Sharded Data Parallel (FSDP): Support new PyTorch* FSDP API which provides an industry-grade solution for large-scale model training.
  • Asymmetric INT8 quantization: Support asymmetric quantization to align with stock PyTorch* and provide better accuracy in INT8.

• CPU support has been merged in this release. CPU features and optimizations are equivalent to what has been released in Intel® Extension for PyTorch* v2.1.0+cpu release that was made publicly available in Oct 2023. For customers who would like to evaluate workloads on both GPU and CPU, they can use this package. For customers who are focusing on CPU only, we still recommend them to use Intel® Extension for PyTorch* v2.1.0+cpu release for smaller footprint, less dependencies and broader OS support.

Known Issues

Please refer to Known Issues webpage.

Intel® Extension for PyTorch* v2.1.0+cpu Release Notes

We are excited to announce the release of Intel® Extension for PyTorch* 2.1.0+cpu which accompanies PyTorch 2.1. This release mainly brings in our latest optimization on Large Language Model (LLM), torch.compile backend optimization as to leverage TorchInductor’s capability, performance optimization of static quantization under dynamic shape, together with a set of bug fixing and small optimization. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try this release and feedback as to improve further on this product.

Highlights

• Large Language Model (LLM) optimization (Experimental): Intel® Extension for PyTorch* provides a lot of specific optimizations for LLMs in this new release. In operator level, we provide highly efficient GEMM kernel to speedup Linear layer and customized operators to reduce the memory footprint. To better trade-off the performance and accuracy, different low-precision solutions e.g., smoothQuant for INT8 and weight-only-quantization for INT4 and INT8 are also enabled. Besides, tensor parallel can also be adopt to get lower latency for LLMs.

  A new API function, ipex.optimize_transformers, is designed to optimize transformer-based models within frontend Python modules, with a particular focus on Large Language Models (LLMs). It provides optimizations for both model-wise and content-generation-wise. You just need to invoke the ipex.optimize_transformers function instead of the ipex.optimize function to apply all optimizations transparently. More detailed information can be found at Large Language Model optimizations overview.

  Specifically, this new release includes the support of SmoothQuant and weight only quantization (both INT8 weight and INT4 weight) as to provide better performance and accuracy for low precision scenarios.

  A typical usage of this new feature is quite simple as below:

  import torch
  import intel_extension_for_pytorch as ipex
  ...
  model = ipex.optimize_transformers(model, dtype=dtype)

• torch.compile backend optimization with PyTorch Inductor (Experimental): We optimized Intel® Extension for PyTorch to leverage PyTorch Inductor’s capability when working as a backend of torch.compile, which can better utilize torch.compile’s power of graph capture, Inductor’s scalable fusion capability, and still keep customized optimization from Intel® Extension for PyTorch.

• performance optimization of static quantization under dynamic shape: We optimized the static quantization performance of Intel® Extension for PyTorch for dynamic shapes. The usage is the same as the workflow of running static shapes while inputs of variable shapes could be provided during runtime.

• Bug fixing and other optimization

  • Optimized the runtime memory usage #1563
  • Fixed the excessive size of the saved model #1677 #1688
  • Supported shared parameters in ipex.optimize #1664
  • Enabled the optimization of LARS fusion #1695
  • Supported dictionary input in ipex.quantization.prepare #1682
  • Updated oneDNN to v3.3 #2137

Intel® Extension for PyTorch* v2.0.110+xpu Release Notes

2.0.110+xpu

We are pleased to announce the release of Intel® Extension for PyTorch* v2.0.110+xpu. This is the new Intel® Extension for PyTorch* release supports both CPU platforms and GPU platforms (Intel® Data Center GPU Flex Series and Intel® Data Center GPU Max Series) based on PyTorch* 2.0.1. It extends PyTorch* 2.0.1 with up-to-date features and optimizations on xpu for an extra performance boost on Intel hardware. Optimizations take advantage of AVX-512 Vector Neural Network Instructions (AVX512 VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel Xe Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, through PyTorch* xpu device, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs with PyTorch*.

Please refer to the Installation Guide for the system requirements and steps to install and use Intel® Extension for PyTorch* 2.0.110+xpu. For more detailed tutorials and documentations describing features, APIs and technical details, please refer to Intel® Extension for PyTorch* Documentation.

Highlights

This release introduces specific XPU solution optimizations on Intel discrete GPUs which include Intel® Data Center GPU Flex Series and Intel® Data Center GPU Max Series. Optimized operators and kernels are implemented and registered through PyTorch* dispatching mechanism for the xpu device. These operators and kernels are accelerated on Intel GPU hardware from the corresponding native vectorization and matrix calculation features. In graph mode, additional operator fusions are supported to reduce operator/kernel invocation overheads, and thus increase performance.

This release provides the following features:

• oneDNN 3.3 API integration and adoption
• Libtorch support
• ARC support on Windows, WSL2 and Ubuntu (Experimental)
• OOB models improvement
  • More fusion patterns enabled for optimizing OOB models
• CPU support is merged in this release:
  • CPU features and optimizations are equivalent to what has been released in Intel® Extension for PyTorch* v2.0.100+cpu release that was made publicly available in May 2023. For customers who would like to evaluate workloads on both GPU and CPU, they can use this package. For customers who are focusing on CPU only, we still recommend them to use Intel® Extension for PyTorch* v2.0.100+cpu release for smaller footprint, less dependencies and broader OS support.

This release adds the following fusion patterns in PyTorch* JIT mode for Intel GPU:

• add + softmax
• add + view + softmax

Known Issues

Please refer to Known Issues webpage.

Intel® Extension for PyTorch* v2.0.100+cpu Release Notes

Highlights

• Enhanced the functionality of Intel® Extension for PyTorch as a backend of torch.compile: #1568 #1585 #1590
• Fixed the Stable Diffusion fine-tuning accuracy issue #1587 #1594
• Fixed the ISA check on old hypervisor based VM #1513
• Addressed the excessive memory usage in weight prepack #1593
• Fixed the weight prepack of convolution when padding_mode is not 'zeros' #1580
• Optimized the INT8 LSTM performance #1566
• Fixed TransNetV2 calibration failure #1564
• Fixed BF16 RNN-T inference when AVX512_CORE_VNNI ISA is used #1592
• Fixed the ROIAlign operator #1589
• Enabled execution on designated numa nodes with launch script #1517

Full Changelog: v2.0.0+cpu...v2.0.100+cpu

Intel® Extension for PyTorch* v1.13.120+xpu Release Notes

1.13.120+xpu

We are pleased to announce the release of Intel® Extension for PyTorch* 1.13.120+xpu. This is the updated IPEX XPU release supports both CPU platforms and GPU platforms (Intel® Data Center GPU Flex Series and Intel® Data Center GPU Max Series) based on PyTorch 1.13.1. It extends PyTorch* 1.13.1 with up-to-date features and optimizations on xpu for an extra performance boost on Intel hardware. The optimizations take advantage of AVX-512 Vector Neural Network Instructions (AVX512 VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel Xe Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, through PyTorch* xpu device, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs with PyTorch*.

Please refer to the Installation Guide for the system requirements and steps to install and use Intel® Extension for PyTorch* 1.13.120+xpu. For more detailed tutorials and documentations describing features, APIs and technical details, please refer to Intel® Extension for PyTorch* Documentation.

Highlights

This release introduces specific XPU solution optimizations on Intel discrete GPUs which include Intel® Data Center GPU Flex Series and Intel® Data Center GPU Max Series. Optimized operators and kernels are implemented and registered through PyTorch* dispatching mechanism for the xpu device. These operators and kernels are accelerated on Intel GPU hardware from the corresponding native vectorization and matrix calculation features. In graph mode, additional operator fusions are supported to reduce operator/kernel invocation overheads, and thus increase performance.

This release provides the following features:

• oneDNN 3.1 API integration and adoption
• OOB models improvement
  • More fusion patterns enabled for optimizing OOB models
• CPU support is merged in this release:
  • CPU features and optimizations are equivalent to what has been released in Intel® Extension for PyTorch* v1.13.100+cpu release that was made publicly available in Feb 2023. For customers who would like to evaluate workloads on both GPU and CPU, they can use this package. For customers who are focusing on CPU only, we still recommend them to use Intel® Extension for PyTorch* v1.13.100+cpu release for smaller footprint, less dependencies and broader OS support.

This release adds the following fusion patterns in PyTorch* JIT mode for Intel GPU:

• Matmul + UnaryOp(abs, sqrt, square, exp, log, round, Log_Sigmoid, Hardswish, HardSigmoid, Pow, ELU, SiLU, hardtanh, Leaky_relu)
• Conv2d + BinaryOp(add, sub, mul, div, max, min, eq, ne, ge, gt, le, lt)
• Linear + BinaryOp(add, sub, mul, div, max, min)
• Conv2d + mul + add
• Conv2d + mul + add + relu
• Conv2d + sigmoid + mul + add
• Conv2d + sigmoid + mul + add + relu

Known Issues

Please refer to Known Issues webpage.