v0.7.0

[qinxuye]

This is the release notes of v0.7.0. See here for the complete list of solved issues and merged PRs.

This release note only covers the difference from v0.7.0rc2; for all highlights and changes, please refer to the release notes of the pre-releases:

alpha1
alpha2
alpha3
alpha4
alpha5
alpha6
alpha7
alpha8
beta1
beta2
rc1
rc2

Changes that break compatibility

v0.7.0 has unified local and distributed execution layer, local thread-based scheduling has been removed, instead, the unified runtime is based on multiprocess-based scheduling which could get rid of infamous GIL problem .

Thus, for local usage, please new a local default session via:

import mars

mars.new_session()  # create a default local session

If not doing so, it will be initialized once in the background, however, keep in mind that the multiprocess scheduling consumes more time compared to multithread one.

We tried our best to keep other compatibilities, if you find any incompatible place, please open an issue to reach out to us.

Highlights

v0.7.0 implements a unified execution layer, all deployment including bare metal, Kubernetes, Ray as well as Yarn shares the same fundamental components. This unified execution layer optimized many aspects compare to the old one including:

• Better serialization based on pickle5 protocol, which is 5-7x faster than old version.
• Completely rewritten execution layer which has better performance, even 20%-50% faster than the old version on a laptop.
• Based on multiprocess scheduling which avoids infamous GIL issue.
• Mars on Ray is way more better due to the reason that Ray actor is leveraged to build the Ray backend of Oscar which is a lightweight actor framework that is the fundamental part of the entire execution layer.

New Features

• Tensor
  • Add partial support of bessel functions ([BACKPORT] Add partial support of bessel functions (#2258) #2274, thanks @JuntaoMa!)
  • Implements mars.tensor.in1d ([BACKPORT] Implements mars.tensor.in1d (#2297) #2301)
• Learn
  • Implements mars.learn.utils.multiclass.unique_label ([BACKPORT] Implements mars.learn.utils.multiclass.unique_label (#2295) #2300)
• Services
  • Add get_storage_level_info api ([BACKPORT] Add get_storage_level_info api (#2228) #2242)
  • Add API to fetch tileable graph as JSON ([BACKPORT] Add API to fetch tileable graph as JSON (#2253) #2271, thanks @RandomY-2!)
  • Enable running on GPU for oscar ([BACKPORT] Enable running on GPU for oscar (#2284) #2306)
• Others
  • Add support for seek method in memory cases ([BACKPORT] Add support for seek method in memory cases (#2250) #2264)

Enhancements

• Add support for stateless actors ([BACKPORT] Add support for stateless actors (#2218) #2220)
• Add status filters for Cluster service ([BACKPORT] Add status filters for Cluster service (#2214) #2221)
• Pass logging config file name into sub pools ([BACKPORT] Pass logging config file name into sub pools (#2222) #2225)
• Support choosing aggregation algorithm at runtime ([BACKPORT] Support choosing aggregation algorithm at runtime (#2213) #2226)
• Add method to session to get web endpoint ([BACKPORT] Add method to session to get web endpoint (#2236) #2238)
• Use Kubernetes Service to discover Mars Supervisors ([BACKPORT] Use Kubernetes Service to discover Mars Supervisors (#2227) #2240)
• Ensure range index incremental for data source op like md.read_csv ([BACKPORT] Ensure range index incremental for data source op like md.read_csv (#2232) #2244)
• Record mapper meta for shuffle task ([BACKPORT] Record mapper meta for shuffle task (#2248) #2255)
• Support data dependency for run_script ([BACKPORT] Support data dependency for run_script (#2251) #2256)
• Refine oscar debugging ([BACKPORT] Refine oscar debugging (#2252) #2261)
• Support fetch_log for web session ([BACKPORT] Support fetch_log for web session (#2257) #2262)
• Allow turning off actor killing ([BACKPORT] Allow turning off actor killing (#2273) #2277)
• Use batch method to reduce transferring cost for shuffle tasks ([BACKPORT] Use batch method to reduce transferring cost for shuffle tasks (#2233) #2279)
• Assign bands given devices of subtasks ([BACKPORT] Assign bands given devices of subtasks (#2276) #2278)
• Add bind method to facilitate extracting batch args ([BACKPORT] Add bind method to facilitate extracting batch args (#2280) #2281)
• Reduce memory estimation for specific operands ([BACKPORT] Reduce memory estimation for specific operands (#2283) #2285)

Bug fixes

• Fix NoDataToSpill when multiple storage quota requests happen simultaneously ([BACKPORT] Fix NoDataToSpill when multiple storage quota requests happen simultaneously (#2203) #2223)
• Stop using thread local to store default session ([BACKPORT] Stop using thread local to store default session (#2217) #2243)
• Fix service errors in Windows ([BACKPORT] Fix service errors in Windows (#2237) #2247)

Documentation

• Doc refinement for Oscar ([BACKPORT] Doc refinement for Oscar (#2234) #2291)
• Add docs for batch methods ([BACKPORT] Add docs for batch methods (#2293) #2298)

Installation

• Merge default & distributed requirements ([BACKPORT] Merge default & distributed requirements (#2263) #2270)

Tests

• Add separate check pipeline ([BACKPORT] Add separate check pipeline (#2299) #2302)

---

This discussion was created from the release v0.7.0.