2.8.5

Fixed:

luigi

• (Doc) Fix example of "summary_length" #2700
• Fix __init__ error when using TOML config #2702
• add callback to metric collector #2704

luigi.contrib

• Fix BigQueryTarget parsing in beam_dataflow module #2705

Changed:

luigi.contrib

• aws batch : job queue as parameter #2689

2.8.4 (BROKEN)

This release is broken due to #2628 .

Added:

luigi

• Added support for a detailed LuigiRunResult instead of a plain Boolean (#2630)
• Add worker option 'max_keep_alive_idle_duration (#2654)
• Added worker-id commandline parameter (#2655)

luigi.contrib

• Add support for specifying kubernetes namespace (#2629)
• Add a Task wrapper for MicroSoft OpenPAI (#2531)
• Provide automatic URL tracking for Spark applications (#2661) (#2669)
• add beam_dataflow_task to luigi/contrib (#2675)

Both

• Add Prometheus contrib for monitoring purpose (#2628)

Fixed:

luigi

• setup.py: Support older setuptools (<=20.1.1) (#2623)
• Fix broken aws tests (#2658)
• Accept pathlib based path as argument for LocalTarget (#2548)
• Fix durations in D3 graph (fixes #2620) (#2624)
• Import collections ABCs from collections.abc, not collections (#2683)
• Configuration documentation: remove deprecated/wrong [core]max_reschedules entry (#2692)

luigi.contrib

• fixes #2223 HdfsTarget is not working with snakebite (#2572)
• Add port field for PostgresQuery (fixes #2625) (#2627)

Both

• Fix flake errors after moving to python 3 (#2695)

Changed:

luigi

• Simplify implementation of temporary_path() (#2652)
• Prevent range-tasks from autoloading (#2656)
• Replace Python MapReduce example with Spark example. (#2668)
• Minor improvements to single-worker-timeout support (#2667)
• Require at least python-dateutil version 2.7.5 instead of only 2.7.5 (fixes #2662) (#2679)
• RangeMonthly should deal with whole months (#2666)
• Reconcile underscore/dash config style handling (#2688) #2691

luigi.contrib

• Add autodetect parameter to BigQueryLoadTask (#2363) (#2575)

2.8.3

Added:

luigi

• Add BaseTIS to the company list #2607
• Add Hopper to the company list #2614
• give a few default values to opts when setting up logging #2612
• Add range functionality for monthly cadence. #2601

luigi.contrib

• Added port to PostgresTarget #2615
• Support for Azure Blob Storage Target #2585
• Add Datadog contrib for monitoring purpose #2434

Fixed:

luigi

• Docs: Fixed a mistake with @inherits syntax in luigi/util.py #2613
• Check type of column before migrating schemas for task db history for postgres dialect (fixes #2563) #2564

luigi.contrib

• S3: Fix call to message from TypeError not working with Python 3.6 #2617
• Use proper API call in bigtable.py's make_dataset #2618
• Sqla: Fix the table name when reflect is True in sqla.CopyToTable (fixes #2604) #2605

Changed:

luigi.contrib

• Changed to buffered reads when using GCSTarget #2588

2.8.2 (Hotfix)

• fix logging setup in Python 2.7 (#2593)

2.8.1(broken)

Note: Broken due to a runtime error in LuigiConfigParser. See #2592.

Added:

luigi

• Add some docs to interface.run #2582
• Configure logging via TOML config #2483

luigi.contrib

• Added port property to CopyToTable #2561
• Make it so we can do from luigi.contrib.hdfs import HdfsFlagTarget #2594
• contrib: Add ExternalDailySnapshot #2591

Fixed:

luigi

• (docstring) Update task.py #2589
• Docs: Fixed "Github" to fit to the rest of the doc #2596
• Fix inspect.getargspec() DeprecationWarning in PY3 #2579

luigi.contrib

• Fix Travis Moto Test Failures #2586
• Reduce TravisCI Test Runtime #2541

Changed:

luigi

• Make Worker parameter task_process_context an OptionalParameter #2574

luigi.contrib

• S3Client improvements #2569

2.8.0

This is a minor version bump, due to:

• Dropping Python 3.4 and 3.5 from CI, which means no automated tests to ensure compatibility for those versions
• [Security Patch] CORS being disabled by default. A new section of configuration [cors] is introduced to enable custom settings. For details, refer to user group topic: https://groups.google.com/forum/#!topic/luigi-user/ZgfRTpBsVUY

Added

luigi:

• Add Python 3.7 compatibility (#2466)
  This also drops 3.4 and 3.5 from CI.
• Interpolate environment variables in .cfg config files (#2527)

luigi.contrib:

• Add CopyToTable task for MySQL (#2553)
• Add HdfsFlagTarget (#2559)

luigi.contrib:

Fixed

luigi:

• Fix ReadTheDocs build (#2546)
• Make capture_output non-positional in ExternalProgramTask (#2547)

luigi.contrib:

• Fix S3Client's _path_to_bucket_and_key to support keys with question marks (#2534)
• Fix S3Client.remove - add max batch size (#2529)
• Small fix to logging in contrib/ecs.py (#2556)
• FIX HdfsAtomicWriteDirPipe.close() when using snakebite and the file do not exist. (#2549)

Changed:

luigi:

• [ImgBot] optimizes images (#2555)

luigi.contrib:

• Remove s3 bucket validation prior to file upload (#2528)
• Refactor s3 copy into sub-methods (#2508)

2.7.9

Added

luigi.contrib:

• Added optional choice for hdfs clients (#2487)
• s3client check for deprecated host keyword and raise error with the details (#2493)
• Add a "capture_output" parameter to ExternalProgramTask (#2430)

Fixed

luigi:

• Fix exception when toml lib is not installed (#2506)
• Replace direct attribute accessing by using built-n function getattr (#2509)
• set upper bound of python-daemon (#2536)

luigi.contrib:

• Fix S3Client.copy return value consistency (#2488)
• Update MockTarget mode to accept r* or w* (#2519)

Version 2.7.8 (hotfix)

• revert tornado upgrade (#2504)
  Upgrading tornado unfortunately breaks older version of Python

Version 2.7.7 (partly broken)

Added

luigi:

• Add Data Revenue to the blogged list (#2472)
• Add default reviewers in CODEOWNERS (#2465)
• Optional TOML configs support (#2457)
• Add support for multiple requires and inherits arguments (#2475)
• Add a visiblity level for luigi.Parameters (#2278)
• Make logging of RPC retries configurable #2486
• Added a new event 'progress' (#2498)

luigi.contrib:

• Additions to provide support for the Load Sharing Facility (LSF) job scheduler (#2373)
• Added default port behaviour for Redshift (#2474)
• Add metadata columns to the RDBMS contrib (#2440)
• Use passed password when create a redis connection (#2489)

Changed

luigi:

• Update supplementary github files to improve repo organization and maintenance (#2463)
• Use task_id in Task.eq comparison (#2462)
• Replace luigi.Task by RunOnceTask in scheduler_visualisation_test (#2476)
• (Breaking change) Bump tornado milestone version (#2490)
  This changes requires Python version 2.7.9+ and 3.4+

luigi.contrib:

• S3 client refactor (#2482)
• Update moto to 1.x milestone version (#2471)

Fixed

luigi:

• Fix Scheduler.add_task to overwrite accepts_messages attribute. (#2469)
• Fix race condition (#2477)
• Fix attribute forwarding for tasks with dynamic dependencies (#2478)

luigi.contrib:

• Fix transfer config import (#2458)

Removed

luigi:

• Remove long-deprecated scheduler config variable alternatives (#2491)

Version 2.7.6

Added

luigi:

• Add a configuration parameter to force multiprocessing (#2401)
• Add a configuration parameter to enable/disable the pause button (#2399)
• Send messages from scheduler to tasks (via "Send message" UI button) (#2426)
• Allow to inject a context manager around TaskProcess.run (via task_process_context configuration parameter) (#2449)

luigi.contrib:

• S3: use Boto3 for the S3Client (#2423, #2149)
• GCS: add method to push files using multiprocessing (#2376)
• HDFS: add get_merge to snakebite client (#2410)
• Redshift: add schema to DB if it doesn't exist (#2439)
• Redshift: add table constraints support (#2435)

Fixed

luigi:

• Allow long parameters in task history DB SQL result store (#2404)
• Fix MissingParameterException when generating execution summary (#2415)
• Fix luigid crash due to configuration file parsing (#2394)
• Allow explicit parsing of BoolParameters (via luigi.BoolParameter.parsing variable) (#2427)
• Make ChoiceParameter check if option is valid within .normalize (#2454)
• ...and a good deal of documentation fixes and similar.

luigi.contrib:

• BigQuery: fix bulk_complete failing when argument is a generator (#2441)
• Kubernetes: prevent KeyError in KubernetesJobTask (#2433)
• Kubernetes: don't set activeDeadlineSeconds by default (#2452)