Airflow 1.10.7, 2019-12-24

New Features

• [AIRFLOW-5088] [AIP-24] Persisting serialized DAG in DB for webserver scalability (#5992)
• [AIRFLOW-6083] Adding ability to pass custom configuration to AWS Lambda client. (#6678)
• [AIRFLOW-5117] Automatically refresh EKS API tokens when needed (#5731)
• [AIRFLOW-5118] Add ability to specify optional components in DataprocClusterCreateOperator (#5821)
• [AIRFLOW-5681] Allow specification of a tag or hash for the git_sync init container (#6350)
• [AIRFLOW-6025] Add label to uniquely identify creator of Pod (#6621)
• [AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator) (#5489)
• [AIRFLOW-5751] add get_uri method to Connection (#6426)
• [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id (#6655)
• [AIRFLOW-2694] Declare permissions in DAG definition (#4642)
• [AIRFLOW-4940] Add DynamoDB to S3 operator (#5663)
• [AIRFLOW-4161] BigQuery to Mysql Operator (#5711)
• [AIRFLOW-6041] Add user agent to the Discovery API client (#6636)
• [AIRFLOW-6089] Reorder setup.py dependencies and add ci (#6681)
• [AIRFLOW-5921] Add bulk_load_custom to MySqlHook (#6575)
• [AIRFLOW-5854] Add support for tty parameter in Docker related operators (#6542)
• [AIRFLOW-4758] Add GcsToGDriveOperator operator (#5822)

Improvements

• [AIRFLOW-3656] Show doc link for the current installed version (#6690)
• [AIRFLOW-5665] Add path_exists method to SFTPHook (#6344)
• [AIRFLOW-5729] Make InputDataConfig optional in Sagemaker's training config (#6398)
• [AIRFLOW-5045] Add ability to create Google Dataproc cluster with custom image from a different project (#5752)
• [AIRFLOW-6132] Allow to pass in tags for the AzureContainerInstancesOperator (#6694)
• [AIRFLOW-5945] Make inbuilt OperatorLinks work when using Serialization (#6715)
• [AIRFLOW-5947] Make the json backend pluggable for DAG Serialization (#6630)
• [AIRFLOW-6239] Filter dags return by last_dagruns (to only select visible dags, not all dags) (#6804)
• [AIRFLOW-6095] Filter dags returned by task_stats (to only select visible dags, not all dags) (#6684)
• [AIRFLOW-4482] Add execution_date to "trigger DagRun" API response (#5260)
• [AIRFLOW-1076] Add get method for template variable accessor (#6793)
• [AIRFLOW-5194] Add error handler to action log (#5883)
• [AIRFLOW-5936] Allow explicit get_pty in SSHOperator (#6586)
• [AIRFLOW-5474] Add Basic auth to Druid hook (#6095)
• [AIRFLOW-5726] Allow custom filename in RedshiftToS3Transfer (#6396)
• [AIRFLOW-5834] Option to skip serve_logs process with airflow worker (#6709)
• [AIRFLOW-5583] Extend the 'DAG Details' page to display the start_date / end_date (#6235)
• [AIRFLOW-6250] Ensure on_failure_callback always has a populated context (#6812)
• [AIRFLOW-6222] http hook logs response body for any failure (#6779)
• [AIRFLOW-6260] Drive _cmd config option by env var (AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD for example) (#6801)
• [AIRFLOW-6168] Allow proxy_fix middleware of webserver to be configurable (#6723)
• [AIRFLOW-5931] Use os.fork when appropriate to speed up task execution. (#6627)
• [AIRFLOW-4145] Allow RBAC roles permissions, ViewMenu to be over-ridable (#4960)
• [AIRFLOW-5928] Hive hooks load_file short circuit (#6582)
• [AIRFLOW-5313] Add params support for awsbatch_operator (#5900)
• [AIRFLOW-2227] Add delete method to Variable class (#4963)
• [AIRFLOW-5082] Add subject in AwsSnsHook (#5694)
• [AIRFLOW-5715] Make email, owner context available (#6385)
• [AIRFLOW-5345] Allow SqlSensor's hook to be customized by subclasses (#5946)
• [AIRFLOW-5417] Fix DB disconnects during webserver startup (#6023)
• [AIRFLOW-5730] Enable get_pandas_df on PinotDbApiHook (#6399)
• [AIRFLOW-3235] Add list function in AzureDataLakeHook (#4070)
• [AIRFLOW-5442] implementing get_pandas_df method for druid broker hook (#6057)
• [AIRFLOW-5883] Improve count() queries in a few places (#6532)
• [AIRFLOW-5811] Add metric for externally killed task count (#6466)
• [AIRFLOW-5758] Support the custom cursor classes for the PostgreSQL hook (#6432)
• [AIRFLOW-5766] Use httpbin.org in http_default (#6438)
• [AIRFLOW-5798] Set default ExternalTaskSensor.external_task_id (#6431)
• [AIRFLOW-5643] Reduce duplicated logic in S3Hook (#6313)
• [AIRFLOW-5562] Skip grant single DAG permissions for Admin role. (#6199)
• [AIRFLOW-6192] Stop creating Hook from SFTPSensor.init (#6748)
• [AIRFLOW-5749][AIRFLOW-4162] Support the "blocks" component for the Slack operators (#6418)
• [AIRFLOW-5693] Support the "blocks" component for the Slack messages (#6364)
• [AIRFLOW-5714] Collect SLA miss emails only from tasks missed SLA (#6384)
• [AIRFLOW-5049] Add validation for src_fmt_configs in bigquery hook (#5671)
• [AIRFLOW-6177] Log DAG processors timeout event at error level, not info (#6731)
• [AIRFLOW-6180] Improve kerberos init in pytest conftest (#6735)
• [AIRFLOW-6159] Change logging level of the heartbeat message to DEBUG (#6716)
• [AIRFLOW-6144] Improve the log message of airflow scheduler (#6710)
• [AIRFLOW-6099] Add host name to task runner log (#6688)
• [AIRFLOW-6045] Error on failed execution of compile_assets (#6640)
• [AIRFLOW-5144] Add confirmation on delete button click (#6745)
• [AIRFLOW-6099] Add host name to task runner log (#6688)
• [AIRFLOW-5915] Add support for the new documentation theme (#6563)
• [AIRFLOW-5897] Allow setting -1 as pool slots value in webserver (#6550)
• [AIRFLOW-5888] Use psycopg2-binary for postgres operations (#6533)
• [AIRFLOW-5870] Allow -1 for pool size and optimise pool query (#6520)

Bug Fixes

• [AIRFLOW-XXX] Bump Jira version to fix issue with async
• [AIRFLOW-XXX] Add encoding to fix Cyrillic output when reading back task logs (#6631)
• [AIRFLOW-5304] Fix extra links in BigQueryOperator with multiple queries (#5906)
• [AIRFLOW-6268] Prevent (expensive) ajax calls on home page when no dags visible (#6839)
• [AIRFLOW-6259] Reset page to 1 with each new search for dags (#6828)
• [AIRFLOW-6185] SQLAlchemy Connection model schema not aligned with Alembic schema (#6754)
• [AIRFLOW-3632] Only replace microseconds if execution_date is None in trigger_dag REST API (#6380)
• [AIRFLOW-5458] Bump Flask-AppBuilder to 2.2.0 (for Python >= 3.6) (#6607)
• [AIRFLOW-5072] gcs_hook should download files once (#5685)
• [AIRFLOW-5744] Environment variables not correctly set in Spark submit operator (#6796)
• [AIRFLOW-3189] Remove schema from DbHook.get_uri response if None (#6833)
• [AIRFLOW-6195] Fixed TaskInstance attrs not correct on UI (#6758)
• [AIRFLOW-5889] Make polling for AWS Batch job status more resillient (#6765)
• [AIRFLOW-6043] Fix bug in UI when "filtering by root" to display section of dag (#6638)
• [AIRFLOW-6033] Fix UI Crash at "Landing Times" when task_id is changed (#6635)
• [AIRFLOW-3745] Fix viewer not able to view dag details (#4569)
• [AIRFLOW-6175] Fixes bug when tasks get stuck in "scheduled" state (#6732)
• [AIRFLOW-5463] Make Variable.set when replacing an atomic operation (#6807)
• [AIRFLOW-5582] Add get_autocommit to JdbcHook (#6232)
• [AIRFLOW-5867] Fix webserver unit_test_mode data type (#6517)
• [AIRFLOW-5819] Update AWSBatchOperator default value (#6473)
• [AIRFLOW-5709] Fix regression in setting custom operator resources. (#6331)
• [AIRFLOW-5658] Fix broken navigation links (#6374)
• [AIRFLOW-5727] SqoopHook: Build --connect parameter only if port/schema are defined (#6397)
• [AIRFLOW-5695] use RUNNING_DEPS to check run from UI (#6367)
• [AIRFLOW-6254] obscure conn extra in logs (#6817)
• [AIRFLOW-4824] Add charset handling for SqlAlchemy engine for MySqlHook (#6816)
• [AIRFLOW-6091] Add flushing in execute method for BigQueryCursor (#6683)
• [AIRFLOW-6256] Ensure Jobs table is cleared when resetting DB (#6818)
• [AIRFLOW-5224] Add encoding parameter to GoogleCloudStorageToBigQuery (#6297)
• [AIRFLOW-5179] Remove top level init.py (#5818)
• [AIRFLOW-5660] Attempt to find the task in DB from Kubernetes pod labels (#6340)
• [AIRFLOW-6241] Fix typo in airflow/gcp/operator/dataflow.py (#6806)
• [AIRFLOW-6171] Apply .airflowignore to correct subdirectories (#6784)
• [AIRFLOW-6018] Display task instance in table during backfilling (#6612)
• [AIRFLOW-6189] Reduce the maximum test duration to 8 minutes (#6744)
• [AIRFLOW-6141] Remove ReadyToRescheduleDep if sensor mode == poke (#6704)
• [AIRFLOW-6054] Add a command that starts the database consoles (#6653)
• [AIRFLOW-6047] Simplify the logging configuration template (#6644)
• [AIRFLOW-6017] Exclude PULL_REQUEST_TEMPLATE.md from RAT check (#6611)
• [AIRFLOW-4560] Fix Tez queue parameter name in mapred_queue (#5315)
• [AIRFLOW-2143] Fix TaskTries graph counts off-by-1 (#6526)
• [AIRFLOW-5873] KubernetesPodOperator fixes and test (#6523)
• [AIRFLOW-5869] BugFix: Some Deserialized tasks have no start_date (#6519)
• [AIRFLOW-4020] Remove DAG edit permissions from Viewer role (#4845)
• [AIRFLOW-6263] Fix broken WinRM integration (#6832)
• [AIRFLOW-5836] Pin azure-storage-blob version to <12 (#6486)
• [AIRFLOW-4488] Fix typo for non-RBAC UI in max_active_runs_per_dag (#6778)
• [AIRFLOW-5942] Pin PyMSSQL to <3.0 (#6592)
• [AIRFLOW-5451] SparkSubmitHook don't set default namespace (#6072)
• [AIRFLOW-6271] Printing log files read during load_test_config (#6842)
• [AIRFLOW-6308] Unpin Kombu for Python 3

Misc/Internal

• [AIRFLOW-6009] Switch off travis_wait for regular tests (#6600)
• [AIRFLOW-6226] Always reset warnings in tests
• [AIRFLOW-XXX] Remove cyclic imports and pylint hacks in Serialization (#6601)
• [AIRFLOW-XXX] Bump npm from 6.4.1 to 6.13.4 in /airflow/www (#6815)
• [AIRFLOW-XXX] Remove executable permission from file
• [AIRFLOW-XXX] Group AWS & Azure dependencies (old [emr] etc. extra still work)
• [AIRFLOW-5487] Fix unused warning var (#6111)
• [AIRFLOW-5925] Relax funcsigs and psutil version requirements (#6580)
• [AIRFLOW-...

Airflow 1.10.6rc1, 2019-10-18

New Features

• [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546)
• [AIRFLOW-4741] Optionally report task errors to Sentry (#5407)
• [AIRFLOW-4939] Add default_task_retries config (#5570)
• [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130)
• [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789)
• [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743)

Improvements

• [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738)
• [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737)
• [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736)
• [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330)
• [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034)
• [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194)
• [AIRFLOW-5657] Update the upper bound for dill dependency (#6334)
• [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891)
• [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233)
• [AIRFLOW-5126] Read aws_session_token in extra_config of the aws hook (#6303)
• [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302)
• [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304)
• [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309)
• [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153)
• [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) (#6163)
• [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012)
• [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062)
• [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059)
• [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948)
• [AIRFLOW-3888] HA for Hive metastore connection (#4708)
• [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873)
• [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768)
• [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214)
• [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696)
• [AIRFLOW-2891] allow configurable docker_operator container name (#5689)
• [AIRFLOW-4285] Update task dependency context definition and usage (#5079)
• [AIRFLOW-5142] Fixed flaky Cassandra test (#5758)
• [AIRFLOW-5218] Less polling of AWS Batch job status (#5825)
• [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589)
• [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339)
• [AIRFLOW-5560] Allow no confirmation on reset dags in airflow backfill command (#6195)
• [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879)
• [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159)
• [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154)
• [AIRFLOW-4835] Refactor operator render_template (#5461)

Bug Fixes

• [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator (#6078)
• [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635)
• [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747)
• [AIRFLOW-5497] Update docstring in airflow/utils/dag_processing.py (#6314)
• Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance (#5908)
• [AIRFLOW-5634] Don't allow editing of DagModelView (#6308)
• [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (#6102)
• [AIRFLOW-5387] Fix "show paused" pagination bug (#6100)
• [AIRFLOW-5489] Remove unneeded assignment of variable (#6106)
• [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108)
• [AIRFLOW-5492] added missing docstrings (#6107)
• [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125)
• [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101)
• [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678)
• [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909)
• [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185)
• [AIRFLOW-5306] Fix the display of links when they contain special characters (#5904)
• [AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841)
• [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237)
• [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307)
• [AIRFLOW-4833] Allow to set Jinja env options in DAG declaration (#5943)
• [AIRFLOW-5408] Fix env variable name in Kubernetes template (#6016)
• [AIRFLOW-5102] Worker jobs should terminate themselves if they can't heartbeat (#6284)
• [AIRFLOW-5572] Clear task reschedules when clearing task instances (#6217)
• [AIRFLOW-5543] Fix tooltip disappears in tree and graph view (RBAC UI) (#6174)
• [AIRFLOW-5444] Fix action_logging so that request.form for POST is logged (#6064)
• [AIRFLOW-5484] fix PigCliHook has incorrect named parameter (#6112)
• [AIRFLOW-5342] Fix MSSQL breaking task_instance db migration (#6014)
• [AIRFLOW-5556] Add separate config for timeout from scheduler dag processing (#6186)
• [AIRFLOW-4858] Deprecate "Historical convenience functions" in airflow.configuration (#5495) (#6144)
• [AIRFLOW-774] Fix long-broken DAG parsing Statsd metrics (#6157)
• [AIRFLOW-5419] Use sudo to kill cleared tasks when running with impersonation (#6026) (#6176)
• [AIRFLOW-5537] Yamllint is not needed as dependency on host
• [AIRFLOW-5536] Better handling of temporary output files
• [AIRFLOW-5535] Fix name of VERBOSE parameter
• [AIRFLOW-5519] Fix sql_to_gcs operator missing multi-level default args by adding apply_defaults decorator (#6146)
• [AIRFLOW-5210] Make finding template files more efficient (#5815)
• [AIRFLOW-5447] Scheduler stalls because second watcher thread in default args (#6129)

Doc-only changes

• [AIRFLOW-5574] Fix Google Analytics script loading (#6218)
• [AIRFLOW-5588] Add Celery's architecture diagram (#6247)
• [AIRFLOW-5521] Fix link to GCP documentation (#6150)
• [AIRFLOW-5398] Update contrib example DAGs to context manager (#5998)
• [AIRFLOW-5268] Apply same DAG naming conventions as in literature (#5874)
• [AIRFLOW-5101] Fix inconsistent owner value in examples (#5712)
• [AIRFLOW-XXX] Fix typo - AWS DynamoDB Hook (#6319)
• [AIRFLOW-XXX] Fix Documentation for adding extra Operator Links (#6301)
• [AIRFLOW-XXX] Add section on task lifecycle & correct casing in docs (#4681)
• [AIRFLOW-XXX] Make it clear that 1.10.5 wasn't accidentally omitted from UPDATING.md (#6240)
• [AIRFLOW-XXX] Improve format in code-block directives (#6242)
• [AIRFLOW-XXX] Format Sendgrid docs (#6245)
• [AIRFLOW-XXX] Update to new logo (#6066)
• [AIRFLOW-XXX] Typo in FAQ - schedule_interval (#6291)
• [AIRFLOW-XXX] Add message about breaking change in DAG#get_task_instances in 1.10.4 (#6226)
• [AIRFLOW-XXX] Fix incorrect units in docs for metrics using Timers (#6152)
• [AIRFLOW-XXX] Fix backtick issues in .rst files & Add Precommit hook (#6162)
• [AIRFLOW-XXX] Update documentation about variables forcing answer (#6158)
• [AIRFLOW-XXX] Add a third way to configure authorization (#6134)
• [AIRFLOW-XXX] Add example of running pre-commit hooks on single file (#6143)
• [AIRFLOW-XXX] Add information about default pool to docs (#6019)
• [AIRFLOW-XXX] Make Breeze The default integration test environment (#6001)

Misc/Internal

• [AIRFLOW-5687] Upgrade pip to 19.0.2 in CI build pipeline (#6358) (#6361)
• [AIRFLOW-5533] Fixed failing CRON build (#6167)
• [AIRFLOW-5130] Use GOOGLE_APPLICATION_CREDENTIALS constant from library (#5744)
• [AIRFLOW-5369] Adds interactivity to pre-commits (#5976)
• [AIRFLOW-5531] Replace deprecated log.warn() with log.warning() (#6165)
• [AIRFLOW-4686] Make dags Pylint compatible (#5753)
• [AIRFLOW-4864] Remove calls to load_test_config (#5502)
• [AIRFLOW-XXX] Pin version of mypy so we are stable over time (#6198)
• [AIRFLOW-XXX] Add tests that got missed from #5127
• [AIRFLOW-4928] Move config parses to class properties inside DagBag (#5557)
• [AIRFLOW-5003] Making AWS Hooks pylint compatible (#5627)
• [AIRFLOW-5580] Add base class for system test (#6229)

Airflow 1.10.5, 2019-08-30

New Features

• [AIRFLOW-1498] Add feature for users to add Google Analytics to Airflow UI (#5850)
• [AIRFLOW-4074] Add option to add labels to Dataproc jobs (#5606)
• [AIRFLOW-4846] Allow specification of an existing secret containing git credentials for init containers (#5475)

Improvements

• [AIRFLOW-5335] Update GCSHook methods so they need min IAM perms (#5939)
• [AIRFLOW-2692] Allow AWS Batch Operator to use templates in job_name parameter (#3557)
• [AIRFLOW-4768] Add Timeout parameter in example_gcp_video_intelligence (#5862)
• [AIRFLOW-5165] Make Dataproc highly available (#5781)
• [AIRFLOW-5139] Allow custom ES configs (#5760)
• [AIRFLOW-5340] Fix GCP DLP example (#594)
• [AIRFLOW-5211] Add pass_value to template_fields BigQueryValueCheckOperator (#5816)
• [AIRFLOW-5113] Support icon url in slack web hook (#5724)
• [AIRFLOW-4230] bigquery schema update options should be a list (#5766)
• [AIRFLOW-1523] Clicking on Graph View should display related DAG run (#5866)
• [AIRFLOW-5027] Generalized CloudWatch log grabbing for ECS and SageMaker operators (#5645)
• [AIRFLOW-5244] Add all possible themes to default_webserver_config.py (#5849)
• [AIRFLOW-5245] Add more metrics around the scheduler (#5853)
• [AIRFLOW-5048] Improve display of Kubernetes resources (#5665)
• [AIRFLOW-5284] Replace deprecated log.warn by log.warning (#5881)
• [AIRFLOW-5276] Remove unused helpers from airflow.utils.helpers (#5878)
• [AIRFLOW-4316] Support setting kubernetes_environment_variables config section from env var (#5668)

Bug fixes

• [AIRFLOW-5168] Fix Dataproc operators that failed in 1.10.4 (#5928)
• [AIRFLOW-5136] Fix Bug with Incorrect template_fields in DataProc{*} Operators (#5751)
• [AIRFLOW-5169] Pass GCP Project ID explicitly to StorageClient in GCSHook (#5783)
• [AIRFLOW-5302] Fix bug in none_skipped Trigger Rule (#5902)
• [AIRFLOW-5350] Fix bug in the num_retires field in BigQueryHook (#5955)
• [AIRFLOW-5145] Fix rbac ui presents false choice to encrypt or not encrypt variable values (#5761)
• [AIRFLOW-5104] Set default schedule for GCP Transfer operators (#5726)
• [AIRFLOW-4462] Use datetime2 column types when using MSSQL backend (#5707)
• [AIRFLOW-5282] Add default timeout on kubeclient & catch HTTPError (#5880)
• [AIRFLOW-5315] TaskInstance not updating from DB when user changes executor_config (#5926)
• [AIRFLOW-4013] Mark success/failed is picking all execution date (#5616)
• [AIRFLOW-5152] Fix autodetect default value in GoogleCloudStorageToBigQueryOperator(#5771)
• [AIRFLOW-5100] Airflow scheduler does not respect safe mode setting (#5757)
• [AIRFLOW-4763] Allow list in DockerOperator.command (#5408)
• [AIRFLOW-5260] Allow empty uri arguments in connection strings (#5855)
• [AIRFLOW-5257] Fix ElasticSearch log handler errors when attemping to close logs (#5863)
• [AIRFLOW-1772] Google Updated Sensor doesnt work with CRON expressions (#5730)
• [AIRFLOW-5085] When you run kubernetes git-sync test from TAG, it fails (#5699)
• [AIRFLOW-5258] ElasticSearch log handler, has 2 times of hours (%H and %I) in _clean_execution_dat (#5864)
• [AIRFLOW-5348] Escape Label in deprecated chart view when set via JS (#5952)
• [AIRFLOW-5357] Fix Content-Type for exported variables.json file (#5963)
• [AIRFLOW-5109] Fix process races when killing processes (#5721)
• [AIRFLOW-5240] Latest version of Kombu is breaking airflow for py2

Misc/Internal

• [AIRFLOW-5111] Remove apt-get upgrade from the Dockerfile (#5722)
• [AIRFLOW-5209] Fix Documentation build (#5814)
• [AIRFLOW-5083] Check licence image building can be faster and moved to before-install (#5695)
• [AIRFLOW-5119] Cron job should always rebuild everything from scratch (#5733)
• [AIRFLOW-5108] In the CI local environment long-running kerberos might fail sometimes (#5719)
• [AIRFLOW-5092] Latest python image should be pulled locally in force_pull_and_build (#5705)
• [AIRFLOW-5225] Consistent licences can be added automatically for all JS files (#5827)
• [AIRFLOW-5229] Add licence to all other file types (#5831)
• [AIRFLOW-5227] Consistent licences for all .sql files (#5829)
• [AIRFLOW-5161] Add pre-commit hooks to run static checks for only changed files (#5777)
• [AIRFLOW-5159] Optimise checklicence image build (do not build if not needed) (#5774)
• [AIRFLOW-5263] Show diff on failure of pre-commit checks (#5869)
• [AIRFLOW-5204] Shell files should be checked with shellcheck and have identical licence (#5807)
• [AIRFLOW-5233] Check for consistency in whitespace (tabs/eols) and common problems (#5835)
• [AIRFLOW-5247] Getting all dependencies from NPM can be moved up in Dockerfile (#5870)
• [AIRFLOW-5143] Corrupted rat.jar became part of the Docker image (#5759)
• [AIRFLOW-5226] Consistent licences for all html JINJA templates (#5828)
• [AIRFLOW-5051] Coverage is not properly reported in the new CI system (#5732)
• [AIRFLOW-5239] Small typo and incorrect tests in CONTRIBUTING.md (#5844)
• [AIRFLOW-5287] Checklicence base image is not pulled (#5886)
• [AIRFLOW-5301] Some not-yet-available files from breeze are committed to master (#5901)
• [AIRFLOW-5285] Pre-commit pylint runs over todo files (#5884)
• [AIRFLOW-5288] Temporary container for static checks should be auto-removed (#5887)
• [AIRFLOW-5326] Fix teething proglems for Airflow breeze (#5933)
• [AIRFLOW-5206] All .md files should have all common licence, TOC (where applicable) (#5809)
• [AIRFLOW-5329] Easy way to add local files to docker (#5933)
• [AIRFLOW-4027] Make experimental api tests more stateless (#4854)

Doc-only changes

• [AIRFLOW-XXX] Fixed Azkaban link (#5865)
• [AIRFLOW-XXX] Remove duplicate lines from CONTRIBUTING.md (#5830)
• [AIRFLOW-XXX] Fix incorrect docstring parameter in SchedulerJob (#5729)

1.10.4rc3

[AIRFLOW-XXX] Update changelog for 1.10.4rc3

1.10.3

New Feature

• [AIRFLOW-4232] Add none_skipped trigger rule (#5032)
• [AIRFLOW-3971] Add Google Cloud Natural Language operators (#4980)
• [AIRFLOW-4069] Add Opsgenie Alert Hook and Operator (#4903)
• [AIRFLOW-3552] Fix encoding issue in ImapAttachmentToS3Operator (#5040)
• [AIRFLOW-3552] Add ImapAttachmentToS3Operator (#4476)
• [AIRFLOW-1526] Add dingding hook and operator (#4895)
• [AIRFLOW-3490] Add BigQueryHook's Ability to Patch Table/View (#4299)
• [AIRFLOW-3918] Add ssh private-key support to git-sync for KubernetesExecutor (#4777)
• [AIRFLOW-3659] Create Google Cloud Transfer Service Operators (#4792)
• [AIRFLOW-3939] Add Google Cloud Translate operator (#4755)
• [AIRFLOW-3541] Add Avro logical type conversion to bigquery hook (#4553)
• [AIRFLOW-4106] instrument staving tasks in pool (#4927)
• [AIRFLOW-2568] Azure Container Instances operator (#4121)
• [AIRFLOW-4107] instrument executor (#4928)
• [AIRFLOW-4033] record stats of task duration (#4858)
• [AIRFLOW-3892] Create Redis pub sub sensor (#4712)
• [AIRFLOW-4124] add get_table and get_table_location in aws_glue_hook and tests (#4942)
• [AIRFLOW-1262] Adds missing docs for email configuration (#4557)
• [AIRFLOW-3701] Add Google Cloud Vision Product Search operators (#4665)
• [AIRFLOW-3766] Add support for kubernetes annotations (#4589)
• [AIRFLOW-3741] Add extra config to Oracle hook (#4584)
• [AIRFLOW-1262] Allow configuration of email alert subject and body (#2338)
• [AIRFLOW-2985] Operators for S3 object copying/deleting (#3823)
• [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators (#3828)
• [AIRFLOW-3799] Add compose method to GoogleCloudStorageHook (#4641)
• [AIRFLOW-3218] add support for poking a whole DAG (#4058)
• [AIRFLOW-3315] Add ImapAttachmentSensor (#4161)
• [AIRFLOW-3556] Add cross join set dependency function (#4356)

Improvement

• [AIRFLOW-3823] Exclude branch's downstream tasks from the tasks to skip (#4666)
• [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes (#4648)
• [AIRFLOW-4247] Template Region on the DataprocOperators (#5046)
• [AIRFLOW-4008] add envFrom for Kubernetes Executor (#4952)
• [AIRFLOW-3947] Flash msg for no DAG-level access error (#4767)
• [AIRFLOW-3287] Moving database clean-up code into the CoreTest.tearDown() (#4122)
• [AIRFLOW-4058] Name models test file to get automatically picked up (#4901)
• [AIRFLOW-3830] Remove DagBag from /dag_details (#4831)
• [AIRFLOW-3596] Clean up undefined template variables. (#4401)
• [AIRFLOW-3573] Remove DagStat table (#4378)
• [AIRFLOW-3623] Fix bugs in Download task logs (#5005)
• [AIRFLOW-4173] Improve SchedulerJob.process_file() (#4993)
• [AIRFLOW-3540] Warn if old airflow.cfg file is found (#5006)
• [AIRFLOW-4000] Return response when no file (#4822)
• [AIRFLOW-3383] Rotate fernet keys. (#4225)
• [AIRFLOW-3003] Pull the krb5 image instead of building (#3844)
• [AIRFLOW-3862] Check types with mypy. (#4685)
• [AIRFLOW-251] Add option SQL_ALCHEMY_SCHEMA parameter to specify schema for metadata (#4199)
• [AIRFLOW-1814] Temple PythonOperator {op_args,op_kwargs} fields (#4691)
• [AIRFLOW-3730] Standarization use of logs mechanisms (#4556)
• [AIRFLOW-3770] Validation of documentation on CI] (#4593)
• [AIRFLOW-3866] Run docker-compose pull silently in CI (#4688)
• [AIRFLOW-3685] Move licence header check (#4497)
• [AIRFLOW-3670] Add stages to Travis build (#4477)
• [AIRFLOW-3937] KubernetesPodOperator support for envFrom configMapRef and secretRef (#4772)
• [AIRFLOW-3408] Remove outdated info from Systemd Instructions (#4269)
• [AIRFLOW-3202] add missing documentation for AWS hooks/operator (#4048)
• [AIRFLOW-3908] Add more Google Cloud Vision operators (#4791)
• [AIRFLOW-2915] Add example DAG for GoogleCloudStorageToBigQueryOperator (#3763)
• [AIRFLOW-3062] Add Qubole in integration docs (#3946)
• [AIRFLOW-3288] Add SNS integration (#4123)
• [AIRFLOW-3148] Remove unnecessary arg "parameters" in RedshiftToS3Transfer (#3995)
• [AIRFLOW-3049] Add extra operations for Mongo hook (#3890)
• [AIRFLOW-3559] Add missing options to DatadogHook. (#4362)
• [AIRFLOW-1191] Simplify override of spark submit command. (#4360)
• [AIRFLOW-3155] Add ability to filter by a last modified time in GCS Operator (#4008)
• [AIRFLOW-2864] Fix docstrings for SubDagOperator (#3712)
• [AIRFLOW-4062] Improve docs on install extra package commands (#4966)
• [AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME (#4705)
• [AIRFLOW-4002] Option to open debugger on errors in airflow test. (#4828)
• [AIRFLOW-3997] Extend Variable.get so it can return None when var not found (#4819)
• [AIRFLOW-4009] Fix docstring issue in GCSToBQOperator (#4836)
• [AIRFLOW-3980] Unify logger (#4804)
• [AIRFLOW-4076] Correct port type of beeline_default in init_db (#4908)
• [AIRFLOW-4046] Add validations for poke_interval & timeout for Sensor (#4878)
• [AIRFLOW-3744] Abandon the use of obsolete aliases of methods (#4568)
• [AIRFLOW-3865] Add API endpoint to get python code of dag by id (#4687)
• [AIRFLOW-3516] Support to create k8 worker pods in batches (#4434)
• [AIRFLOW-2843] Add flag in ExternalTaskSensor to check if external DAG/task exists (#4547)
• [AIRFLOW-2224] Add support CSV files in MySqlToGoogleCloudStorageOperator (#4738)
• [AIRFLOW-3895] GoogleCloudStorageHook/Op create_bucket takes optional resource params (#4717)
• [AIRFLOW-3950] Improve AirflowSecurityManager.update_admin_perm_view (#4774)
• [AIRFLOW-4006] Make better use of Set in AirflowSecurityManager (#4833)
• [AIRFLOW-3917] Specify alternate kube config file/context when running out of cluster (#4859)
• [AIRFLOW-3911] Change Harvesting DAG parsing results to DEBUG log level (#4729)
• [AIRFLOW-3584] Use ORM DAGs for index view. (#4390)
• [AIRFLOW-2821] Refine Doc "Plugins" (#3664)
• [AIRFLOW-3561] Improve queries (#4368)
• [AIRFLOW-3600] Remove dagbag from trigger (#4407)
• [AIRFLOW-3713] Updated documentation for GCP optional project_id (#4541)
• [AIRFLOW-2767] Upgrade gunicorn to 19.5.0 to avoid moderate-severity CVE (#4795)
• [AIRFLOW-3795] provide_context param is now used (#4735)
• [AIRFLOW-4012] Upgrade tabulate to 0.8.3 (#4838)
• [AIRFLOW-3623] Support download logs by attempts from UI (#4425)
• [AIRFLOW-2715] Use region setting when launching Dataflow templates (#4139)
• [AIRFLOW-3932] Update unit tests and documentation for safe mode flag. (#4760)
• [AIRFLOW-3932] Optionally skip dag discovery heuristic. (#4746)
• [AIRFLOW-3258] K8S executor environment variables section. (#4627)
• [AIRFLOW-3931] set network, subnetwork when launching dataflow template (#4744)
• [AIRFLOW-4095] Add template_fields for S3CopyObjectOperator & S3DeleteObjectsOperator (#4920)
• [AIRFLOW-2798] Remove needless code from models.py
• [AIRFLOW-3731] Constrain mysqlclient to <1.4 (#4558)
• [AIRFLOW-3139] include parameters into log.info in SQL operators, if any (#3986)
• [AIRFLOW-3174] Refine Docstring for SQL Operators & Hooks (#4043)
• [AIRFLOW-3933] Fix various typos (#4747)
• [AIRFLOW-3905] Allow using "parameters" in SqlSensor (#4723)
• [AIRFLOW-2761] Parallelize enqueue in celery executor (#4234)
• [AIRFLOW-3540] Respect environment config when looking up config file. (#4340)
• [AIRFLOW-2156] Parallelize Celery Executor task state fetching (#3830)
• [AIRFLOW-3702] Add backfill option to run backwards (#4676)
• [AIRFLOW-3821] Add replicas logic to GCP SQL example DAG (#4662)
• [AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator (#4347)
• [AIRFLOW-3647] Add archives config option to SparkSubmitOperator (#4467)
• [AIRFLOW-3802] Updated documentation for HiveServer2Hook (#4647)
• [AIRFLOW-3817] Corrected task ids returned by BranchPythonOperator to match the dummy operator ids (#4659)
• [AIRFLOW-3782] Clarify docs around celery worker_autoscale in default_airflow.cfg (#4609)
• [AIRFLOW-1945] Add Autoscale config for Celery workers (#3989)
• [AIRFLOW-3590] Change log message of executor exit status (#4616)
• [AIRFLOW-3591] Fix start date, end date, duration for rescheduled tasks (#4502)
• [AIRFLOW-3709] Validate allowed_states for ExternalTaskSensor (#4536)
• [AIRFLOW-3522] Add support for sending Slack attachments (#4332)
• [AIRFLOW-3569] Add "Trigger DAG" button in DAG page (/www only) (#4373)
• [AIRFLOW-3569] Add "Trigger DAG" button in DAG page (/www_rbac only) (#4373)
• [AIRFLOW-3044] Dataflow operators accept templated job_name param (#3887)
• [AIRFLOW-3023] Fix docstring datatypes
• [AIRFLOW-2928] Use uuid4 instead of uuid1 (#3779)
• [AIRFLOW-2988] Run specifically python2 for dataflow (#3826)
• [AIRFLOW-3697] Vendorize nvd3 and slugify (#4513)
• [AIRFLOW-3692] Remove ENV variables to avoid GPL (#4506)
• [AIRFLOW-3907] Upgrade flask and set cookie security flags. (#4725)
• [AIRFLOW-3698] Add documentation for AWS Connection (#4514)
• [AIRFLOW-3616][AIRFLOW-1215] Add aliases for schema with underscore (#4523)
• [AIRFLOW-3375] Support returning multiple tasks with BranchPythonOperator (#4215)
• [AIRFLOW-3742] Fix handling of "fallback" for AirflowConfigParsxer.getint/boolean (#4674)
• [AIRFLOW-3742] Respect the fallback arg in airflow.configuration.get (#4567)
• [AIRFLOW-3789] Fix flake8 3.7 errors. (#4617)
• [AIRFLOW-3602] Improve ImapHook handling of retrieving no attachments (#4475)
• [AIRFLOW-3631] Update flake8 and fix lint. (#4436)

Bug fixes

• [AIRFLOW-4248] Fix 'FileExistsError' makedirs race in file_processor_handler (#5047)
• [AIRFLOW-4240] State-changing actions should be POST requests (#5039)
• [AIRFLOW-4246] Flask-Oauthlib needs downstream dependencies pinning due to breaking changes (#5045)
• [AIRFLOW-3887] Downgrade dagre-d3 to 0.4.18 (#4713)
• [AIRFLOW-3419] Fix S3Hook.select_key on Python3 (#4970)
• [AIRFLOW-4127] Correct AzureContainerInstanceHook._get_instance_view's return (#4945)
• [AIRFLOW-4172] Fix changes for driver class path option in Spark Submi...

1.7.0

Raw changelog

for the record, generated with:
~/node_modules/github-changes/bin/index.js -o airbnb -r airflow --only-pulls --use-commit-body --between-tags 1.6.2...1.7.0rc1 --token {your_gh_token_here}

1.7.0

• #1147 Enhance CLI Test command to accept a JSON-formatted dictionary of par… (@r39132)
• #975 statuses column on /admin shows only active or most recent dag_runs (@jtschoonhoven)
• #1127 FAQ entry about start_date (@mistercrunch)
• #1129 Showing active dag runs as in (3/16) in tooltip (@mistercrunch)
• #1144 More explicit zindex on modals (@mistercrunch)
• #1143 small fixes to previous bq project inclusion pr. (@mtagle)
• #1139 Allow specificiation of project in BigQuery Hook methods (@mtagle)
• #1110 Add date support to MySQL to GCS operator (@criccomini)
• #1135 Added start_date initialization for DagRun creation within schedule_dag(self, dag_id) (@RvN76)
• #1140 License check (@bolkedebruin)
• #1138 Add support for BigQuery User Defined Functions in BigQuery operator (@LeBlanc)
• #1132 Add custom email backends. (@jmcarp)
• #1119 Add gcloud-based GCSHook (@jlowin)
• #1134 Parameterizing DagBag import timeouts (@mistercrunch)
• #1045 Add startup scripts for upstart based systems (@d-lee)
• #1131 Adding ssh connection type to webform (@hyperborea)
• #792 add error handling for slack api (@abridgett)
• #1115 This patch adds license checking for Airflow. (@bolkedebruin)
• #1116 Add WePay and committer list to README.md (@criccomini)
• #1118 Don't force installation of GCP API client dependencies (@jlowin)
• #1090 Adding template support in qbol operator (@msumit)
• #1108 new company + link to pitfalls (@kretes)
• #1104 Add INT24 (MEDIUMINT) support to MySQL to Google cloud storage operator (@criccomini)
• #1076 SubDAG docs and examples (@nicktrav)
• #1101 modify datastore hook so that authorization is maintained for the lif… (@mtagle)
• #1099 Adding @mention in Airbnb and links to airbnb open source projects (@artwr)
• #1096 Make SqlAlchemy pool_recycle and pool_size configurable (@amread)
• #1070 Proper sqlalchemy syntax for desc (@mistercrunch)
• #1079 SID Oracle DB connection support (@biln)
• #1094 Adding to inits (@artwr)
• #1093 Add MySQL to BQ support for TINYINT (@criccomini)
• #1088 Fix MySQL to Google cloud storage scoping. (@criccomini)
• #1075 Pig hook and operator stub (@artwr)
• #1084 Replace deprecated flask.ext.* with flask_* (@jeffwidman)
• #1082 Cleanup Contributing.md (@jeffwidman)
• #1086 Default to 0 if no rows loaded in GCS to BQ operator. (@criccomini)
• #1080 Add MySQL->GCS, GCS->BQ operators (@criccomini)
• #1073 Fixing small issue with qbol operator and hook (@msumit)
• #1068 Add a new hook for google datastore (@mtagle)
• #1065 dag pausing should pause queued tasks as well (@mistercrunch)
• #1067 Add two methods to BigQueryBaseCursor: (@mtagle)
• #1064 Add Thumbtack to organizations using Airflow (@natekupp)
• #1063 Documenting task details doc_ feature (@mistercrunch)
• #1049 Allow the use of the autoconfig client (@bolkedebruin)
• #1056 Add output encoding option to BashOperator (@Attsun1031)
• #996 Added the dst filename to template_fields GoogleCloudStorageDownloadOperator (@0xR)
• #1051 Running unit tests with local executor (@mistercrunch)
• #1008 Add FTPSHook class. (@geeknam)
• #1055 Improve [Hide/Show all series] perforamce (@lumengxi)
• #1054 CLI's trigger_dag now accepts --conf as json (@mistercrunch)
• #1053 Add Kogan.com to the list of users (@geeknam)
• #1046 Documenting the cluster policy feature (@mistercrunch)
• #1044 Updating the Readme with a link to the TriggerDagRunOperator post (@r39132)
• #1043 Adding an example to illustrate the TriggerDagRunOperator (@r39132)
• #1039 Only set headers and delimiters for CSV exports in Google BigQuery hook (@criccomini)
• #1026 Minor documentation tweaks to the FAQ under the fernet key section (@r39132)
• #1025 Add notes on connection password encryption (@d-lee)
• #1018 add SSL support for SMTP (@bbrumi)
• #1020 Add BigQuery copy operator. (@criccomini)
• #964 Clean test database out in between unit test runs (@nicktrav)
• #1019 Add direct dependencies for Google cloud contribs (@criccomini)
• #985 [airflow][presto] Gracefully handle 503 errors and avoid eval() (@airbnb)
• #1013 Fixed issue 1012: pool not used with celery executor (@rdavison)
• #1011 drop the tmp table after ingestion (@dayzzz)
• #995 Fix LDAP error messages when login fails. (@criccomini)
• #1006 Fix forgetting to expose yesterday_ds_nodash and tomorrow_ds_nodash (@0xR)
• #1003 Update to the README such as adding Hootsuite (@r39132)
• #998 Pedantic documentation tweaks (@joshmarlow)
• #940 Refresh border coloring in graph view without having to refresh the entire page (@dinocow)
• #987 Add logout button to Airflow (@criccomini)
• #930 When either data_profiler_filter or superuser_filter aren't defined,… (@mtagle)
• #994 there could be dot in the key string, which is illegal in the hive ta… (@dayzzz)
• #993 Added yesterday_ds_nodash and tommorow_ds_nodash (@0xR)
• #992 Rename BashOperator.xcom_push member to avoid collision with parent class (@seregasheypak)
• #978 Fixing conflicting params in default_args (@mistercrunch, @nicktrav)
• #984 Add tests for params handling in Dag construction (@nicktrav)
• #986 [documentation] document that max_active_runs can prevent a DAG from running (@aoen)
• #983 Added destination_dataset_table to template_fieds of bigquery_operator (@0xR)
• #981 Add BigQuery PEP 249 support (@criccomini)
• #980 Support user-defined macros and params in dry-run backfills with task… (@r39132)
• #976 More verbose logging for are_dependencies_met when called from run (@mistercrunch)
• #944 Fix statuses in dag list for dags with dag_ids prefixed by a number (@wil5for)
• #942 Set effective user of (web)hdfs hooks using connection config, optionally overridden by constructor (@xadhoom)
• #965 Adding mock lib to devel extras_require (@mistercrunch)
• #963 Only schedule DagRuns between start and end dates (@nicktrav)
• [#948](https://git...

1.6.2

1.6.2 is mostly a release with bug fixes and a few relatively minor features. Thanks to all contributors!

v1.6.1

• Scheduler bugfix

v1.6.0

v1.6.0 brings:

• [scheduler] the notion of DAG runs allows for more parallelization, and controls around scheduling (max number of running task instance per DAG, max number of DAG runs to be evaluated for scheduling, ...)
• [scheduler] support for "externally triggered" DAG runs, or DAGs that run on demand as opposed to on a schedule
• [scheduler] support for cron-like syntax (as in: "0 0 * * *") and macros (as in "@montlhly", "@hourly", "@Weekly", ...)
• UI changes related to new scheduler features
• LDAP authentication for the web UI, more extensible authentication backend
• UI activity logging
• WebHdfsSensor and Hook for HDFS interactions that are py3 compatible
• Continuous integration with Travis-UI and Coveralls
• ShortCircuitOperator
• python3 compatibility!
• Tons of bug fixes and incremental improvements
• + all the things I'm forgetting while browsing through an infinite list of commits!

Thanks to everyone in the community for all the PRs (stellar contributions!), comments and issue reporting.

v1.5.2

This is probably missing some important items, but most of it should be captured

• Initial setup on Travis CI provides continuous Integration, automated testing, no Hadoop unit tests just yet, but that is coming up
• Unit tests coverage reports with coverals
• Better py3 compatibility, unit tests run against both 2.7 ad 3.4 and we're now using the from __future__ imports to prevent regression
• A MesosExecutor to run your tasks on Mesos
• Some Kerberos integration for Hive / Hadoop
• DAG's graph view legend for states are now toggles to highlight tasks in specific states
• Automated zombie task instance killing as part of the scheduler's routine. The process looks for running tasks that don't have a heartbeat and kills them
• MySqlHook bulk load option
• More options in the UI's Mark Success form
• MySQL uses mysqlclient lib instead of mysql-python
• Using gunicorn instead of tornado as the wsgi web server
• OracleHook
• FTPHook
• Much more! tons of bug fixes and usability improvements.