2.8.1

What's Changed

• [2.8.x] Add newline before ORDER BY clause in BranchIterator (#9566) by @albscui in #9568
• [2.8.x] Backport generic watcher for commits by @albscui in #9569
• bump console 2.8.1 by @molinamelendezj in #9570

Full Changelog: v2.8.0...v2.8.1

2.9.0-nightly.20231201

What's Changed

• Refactor the way we watch for events related to commits. by @albscui in #9465
• remove mount-server from jupyterlab-pachyderm extension by @kevyang in #9563
• Add newline before ORDER BY clause in BranchIterator by @albscui in #9566

Full Changelog: v2.8.0-nightly.20231129...v2.9.0-nightly.20231201

2.8.0

What's Changed

CORE

• Spout commits should be closed by PPS - #9353
• exchange PACH_ID_TOKEN env var for session token during client creation - #9397
• Add new clusterstate for v2.8.0 - #9199
• Add new watcher for new postgres tables - #9200
• Migrate data from collections.branches to pfs.branches - #9260
• Add GetBranch and UpsertBranch DB APIs - #9305
• Refactor clusterstate v2.8.0 - #9308
• Refactor pfsdb API - #9307
• pfsdb: add branch provenance related functionality - #9317
• Generic pfsdb list and filtering applied to branches - #9329
• pfsdb new DeleteBranch function - #9346
• Add support for branch triggers in pfsdb - #9349
• Add ability to sort by a set of columns in pfsdb.BranchIterator - #9356
• pfsdb improvements for branches. - #9440
• Update service callsites for Commits and Branches - #9400
• Fix error handling in pipeline controller - #9485
• Skip missing branches during migration - #9540
• Propagate force delete through PFS APIs - #9541
• Sort CommitSets in descending order - #9548
• Ensure BranchIterator implements iterator interface - #9552
• Update branch deletion logic - #9507
• Collection migration optimization - #9194
• generate grpc-gateway typescript client - #9383
• Add RerunPipeline RPC - #9287
• debug: avoid retrieving binaries by default - #9114
• use go 1.21 - #9184
• Increase pgbouncer connection limit - #9190
• pachctl: misc generate-download-url: show a full clickable link if the server knows its address - #9218
• client: don't use a timeout context unless the timeout is > 0 - #8800
• update readiness and liveness probe endpoints - #9225
• Move archiveserver to a general "PachHTTP" server - #9213
• feat: Generate JSON schema for PPS proto. - #9252
• proxy: discover endpoints before k8s health check passes - #9235
• Serve JSON schemas from pachd - #9257
• pps/cmds: send auth credentials for inspect/create defaults - #9256
• proto: Invoke the proto compiler exactly once; namespace JSON schemas - #9269
• Return paused status from InspectCluster; warn in pachctl when seen - #9248
• Add JSON schema during pachctl edit - #9258
• Allow both proto names and JSON names in generated schemas - #9259
• Warn when stopping/deleting a pipeline that doesn't exist - #9284
• pfs/pps: make everything take a context - #9283
• Bump gopkg.in/yaml.v3 from 3.0.0-20200313102051-9f266ea9e77c to 3.0.0 in /etc/helm - #9276
• pachctl: add a "misc grpc" command to make arbitrary RPCs - #9281
• Fix Master After Merge Race - #9295
• Name all example pipelines *.pipeline.json - #9298
• real_env: log server messages to a file in /tmp instead of t.Log - #9293
• Generate documentation from protos - #9271
• Allow only protojson field names in example pipelines - #9304
• Fix panic in CreateFileSet.CopyFile - #9301
• Remove a bunch of NUL bytes from the end of RPC panic stack traces - #9303
• pachctl: Add a command to decode protos - #9285
• Add Starlark-running infrastructure - #9331
• Add the ability to run Starlark during debug dumps, and run debug dumps locally - #9338
• Dynamic k8s API for starlark scripts - #9350
• Add a /pfs HTTP endpoint that allows browsing files stored in PFS - #9187
• ignore generated go proto files in codecov - #9438
• handle errors during restgateway mux creation - #9452
• ignore grpc gateway generated files for code coverage - #9454
• remove snowflake and pachtf - #9539
• Switch over to using pachyderm-sdk over python-pachyderm in jupyter extension - #9428
• added global registry support - #9178
• 0.26.2 det helm update - #9467
• Merge cluster defaults post-2.7 - #9138
• Cluster-defaults-aware CreatePipelineV2 - #9098
• Implement server-side dry run support - #9201
• Save user spec - #9167
• pachctl create pipeline --dry-run - #9205
• Include user and effective specs in output of pachctl inspect pipeline - #9210
• Warn, not error, on nonexistent contextual project - #9215
• Propagate cluster defaults - #9020
• Synthesize cluster defaults from environment during migration - #9220
• Support regenerate & reprocess when updating and deleting defaults - #9234
• Support dry run in SetClusterDefaults - #9239
• Switch pachctl pipeline create/update to use a spec reader - #9231
• Allow access to PauseStatus when auth is disabled - #9249
• Support dry run in pachctl [create|delete|update] defaults - #9243
• Support user spec in pachctl edit pipeline - #9236
• Add migration to synthesize user and effective spec from details - #9241
• Use log.Proto for logging gRPC details - #9275
• Include cluster defaults in debug dumps - #9268
• Add some simple benchmarking of effective spec calculation - #9335
• Do not check project when auth is enabled but credentials are missing - #9333
• Unhide pachctl [inspect|create|delete|update] defaults - #9373
• Allow YAML cluster defaults in pachctl - #9371
• Validate effective specs - #9352
• Remove repeating header when listing files - #9405
• Switch from UnimplementedAPIServer to UnsafeAPIServer - #9409
• Add project defaults collection & associated migration - #9423
• Add project defaults messages - #9424
• Rerun previous pipeline spec - #9448
• Add GetProjectDefaults RPC - #9442
• Update CreatePipeline for project defaults - #9445
• Remove mention of project defaults - #9480
• Add SetProjectDefaults - #9468
• Add pachctl update project defaults - #9471
• Add pachctl inspect project defaults - #9470
• Implement pachctl delete project defaults - #9472
• Implement pachctl create project defaults - #9473
• Add squash to project defaults migration - #9519
• Correct auth check for project defaults - #9521
• Update to Go 1.21.4 - #9526
• Add singular default and defaults global help - #9514
• Generate REST API from Pachyderm gRPC API using grpc-gateway - #9380

PFS

• migrations: add Squash - #9186
• debug/server: add Env struct for dependencies - #9208
• break dependency server/pfs -> server/pps - #9212
• Remove PFS from proto owners - #9286
• Moves EnvFromServiceEnv functions into internal/pachd. - #9290
• pachd: add PreflightEnv for preflight mode dependencies - #9296
• bump go to 1.21.1 to patch vulnerabilities - #9315
• pfs: adds WorkerEnv, WorkerConfig, and Worker - #9302
• pfs: separate master and remove background goroutine from APIServer - #9314
• pps: add Worker and WorkerEnv - #9355
• pfs: replace AuthServer with PFSAuth interface - #9362
• internal/client: add functions for file methods - #9372
• debug: worker only needs PFS and TaskService - #9376
• pps: worker depends on PFS directly - #9375
• transactionenv: remove serviceenv dependency - #9387
• Initial PJS protos - #9415
• obj: don't clear staging in local fs client - #9427
• More trigger fixes - #9246
• New squash / drop commit RPCs and implementations - #9492
• Add ctx To Methods That Will Call or Depend on the New Repos Data Model - #9196
• Migrate Repos from collections to relational model - #9015
• update project_id type in repoRow - #9221
• Add errors.Cause() - #9266
• Expand function signature to take ctx. - #9270
• Add not null violation - #9288
• Create Initial Schema Migration for Commits - #9297
• Add New Data Model for pfs.commits - #9312
• Migrate collections.commits to pfs.commits - #9328
• Squash all applied migrations in 2.8.0 - #9340
• Update Commits Functions for Callsite Integration - #9432
• Handle Potentially Nil Branches - #9436
• Use Generic Page Iterator for List Commits - #9457
• Use Paged Iterator for the Repo Iterator - #9462
• Refactor List Project to Use the Generic Page Iterator - #9476
• Implement Updated ListBranch and ForEachBranch - #9477
• Implement ForEachRepo and ListRepos - #9478
• Implement ForEachProject() and ListProject() - #9481
• Commit.branches should be a copy and not a reference - #9487
• Update Error Handling Code - #9490
• Fix Scenario Where Commits Are Not Migrated - #9528
• Add Batching Code to Commits Migration - #9520
• lock repos, branches, and commits - #9536
• Add ForEachCommit() and ListCommit() - #9496
• Rename some collections tables post-migration - #9553

INT

• Proposed: Log HTTP responses - #9430
• Avoid double-logging request ID - #9453
• Remove bash from JupyterLab-Pachyderm - #9408
• Exclude the PJS API from pachyderm-sdk (for now) - #9486
• [Jupyter] Enable networking over UNIX domain sockets with pycurl - #9366
• Auto-detect if non-priv mode should be used - #9401
• Remove stack trace on console by disabling nbclassic - #9426
• Expand Integrations CODEOWNERS - #9206
• [Python-SDK] Fix CI linter - #9217
• Unprivileged jupyter lab extension -- enable /pfs folder to link - #8985
• [Jupyter] Paginated Filebrowser - #9413
• [Python-SDK] upgrade grpcio dependency - #9498
• [Python-SDK] containerize documentation generation - #9497
• Update jupyter_server version to 2.7.0 to match Determined AI - #9351
• Fix HPC nonpriv containers + mount-server --sock-path - #9389
• [Mount Server] Implement functionality to mount specific commit - #9310
• [Mount Server] Fixes mounting datum bug in loopback.go - #9374
• Add mounting multiple files from same repo - #9402
• Improve mounting datums performance - #9393
• Fix auth login - #9464
• [Mount Server] Missing io import - #9466

CONSOLE

• Add post-repo-create message
• Timestamps should use 0-23 hour notation instead of 1-24
• Datum Card has incorrect rounded edges
• Add a dedicated health check endpoint
• Add message field to commit in right panel
• Global ID: Restore ability to see point-in-time side panel data
• Add timestamp test coverage
• Hide side panel if user does not have correct access
• Job IDs in pipeline sidebar should be click-to-copy
• Going back from applying global id applied wrong URL params
• Datum Viewer a...

2.8.0-nightly.20231129

What's Changed

• check for not found errors by @djanicekpach in #9546
• pipeline worker alerts and check status by @molinamelendezj in #9461
• Sort CommitSets in descending order by @albscui in #9548
• Rename some collections tables post-migration by @FahadBSyed in #9553
• Ensure BranchIterator implements iterator interface by @albscui in #9552
• [CORE-1659] Add context to client.IsAPIActive by @robert-uhl in #9558
• Remove conditional build on go1.20 by @jrockway in #9561
• [CORE-1724] pachctl project defaults lifecycle tests by @robert-uhl in #9547
• go mod tidy by @albscui in #9564

Full Changelog: v2.8.0-nightly.20231115...v2.8.0-nightly.20231129

2.8.0-rc.4

What's Changed

• Backport ForEach Commit by @FahadBSyed in #9551
• Backport: Rename some collections tables post-migration by @FahadBSyed in #9555
• bump console version 2.8.0-rc.4 by @molinamelendezj in #9557

Full Changelog: v2.8.0-rc.3...v2.8.0-rc.4

2.8.0-nightly.20231120

What's Changed

• check for not found errors by @djanicekpach in #9546
• pipeline worker alerts and check status by @molinamelendezj in #9461
• Sort CommitSets in descending order by @albscui in #9548
• Rename some collections tables post-migration by @FahadBSyed in #9553
• Ensure BranchIterator implements iterator interface by @albscui in #9552

Full Changelog: v2.8.0-nightly.20231115...v2.8.0-nightly.20231120

2.8.0-rc.3

What's Changed

• Backport PFS-169 into 2.8 by @albscui in #9535
• Backport Commit Migration Batcher to 2.8.x by @FahadBSyed in #9534
• 2.8.x Backport Locking During Migration by @FahadBSyed in #9538
• [2.8.x] Propagate force delete through PFS APIs (#9541) by @albscui in #9544
• [2.8.x] Skip missing branches during migration (#9540) by @albscui in #9542
• [2.8.x] remove snowflake and pachtf by @jrockway in #9545
• Djanicek - PFS-175 - debug rollback err (#9537) by @djanicekpach in #9543
• bump console 2.8.0-rc.3 by @molinamelendezj in #9550

Full Changelog: v2.8.0-rc.2...v2.8.0-rc.3

2.8.0-nightly.20231115

What's Changed

• [CORE-2036] Add singular default and defaults global help by @robert-uhl in #9514
• Update branch deletion logic by @albscui in #9507
• [PFS-172] Add Batching Code to Commits Migration by @FahadBSyed in #9520
• [PFS-173] lock repos, branches, and commits by @FahadBSyed in #9536
• remove snowflake and pachtf by @jrockway in #9539
• Skip missing branches during migration by @albscui in #9540
• Djanicek - PFS-175 - debug rollback err by @djanicekpach in #9537
• Propagate force delete through PFS APIs by @albscui in #9541

Full Changelog: v2.8.0-nightly.20231113...v2.8.0-nightly.20231115

2.8.0-rc.2

What's Changed

• [Python-SDK] upgrade grpcio dependency (#9498) by @bbonenfant in #9508
• Djanicek/allow upload timeout (#9516) by @djanicekpach in #9518
• [CORE-2040] Backport squash migration by @robert-uhl in #9523
• [CORE-2037] Correct auth check for project defaults (backport #9521) by @robert-uhl in #9524
• [CORE-2042] Update Go to v1.21.4 (backport #9526) by @robert-uhl in #9527
• Backport Changes for CORE-2038 by @FahadBSyed in #9529
• [Python-SDK] Bugfix: AuthServiceNotActivated errors not being raised by @bbonenfant in #9530
• [CORE-2036] Add singular default and defaults global help by @robert-uhl in #9515
• bump console 2.8.0-rc.2 by @molinamelendezj in #9533

Full Changelog: v2.8.0-rc.1...v2.8.0-rc.2

2.8.0-nightly.20231113

What's Changed

• Collection migration optimization by @dgeorg42 in #9194
• Generate REST API from Pachyderm gRPC API using grpc-gateway by @zmajeed in #9380
• Proposed: Log HTTP responses by @msteffen in #9430
• remove underscores from description frontmatter by @lbliii in #9449
• pfsdb improvements for branches. by @albscui in #9440
• [CORE-2015] Rerun previous pipeline spec by @robert-uhl in #9448
• handle errors during restgateway mux creation by @jrockway in #9452
• Avoid double-logging request ID by @msteffen in #9453
• [CORE-1711] Add GetProjectDefaults RPC by @robert-uhl in #9442
• ignore grpc gateway generated files for code coverage by @jrockway in #9454
• Update service callsites for Commits and Branches by @albscui in #9400
• Disable test_mount_datums by @msteffen in #9456
• [Jupyter] Paginated Filebrowser by @bbonenfant in #9413
• [Jupyter] Upload cypress artifacts on e2e test failure by @bbonenfant in #9455
• [PFS-94] Use Generic Page Iterator for List Commits by @FahadBSyed in #9457
• Handle memory requests/limits that are a fraction of a byte by @jrockway in #9460
• [Notebooks] Improve mounting datums performance by @smalyala in #9393
• [Jupyter] Fix auth login by @smalyala in #9464
• [Mount Server] Missing io import by @smalyala in #9466
• Switch over to using pachyderm-sdk over python-pachyderm in jupyter extension by @kevyang in #9428
• [PFS-158] Use Paged Iterator for the Repo Iterator by @FahadBSyed in #9462
• Increase rate limit spec in TestTrigger/Or to avoid prematurely firing the trigger [PFS-163] by @albscui in #9469
• Refactor List Project to Use the Generic Page Iterator by @FahadBSyed in #9476
• 0.26.2 det helm update by @molinamelendezj in #9467
• [PFS-158-branches] Implement Updated ListBranch and ForEachBranch by @FahadBSyed in #9477
• dockertestenv: expose configuration for direct and pgbouncer connections by @brendoncarroll in #9459
• [CORE-1777] Update CreatePipeline for project defaults by @robert-uhl in #9445
• [CORE-2022] Remove mention of project defaults by @robert-uhl in #9480
• Implement ForEachRepo and ListRepos by @FahadBSyed in #9478
• Remove bash from JupyterLab-Pachyderm by @msteffen in #9408
• [CORE-1712] Add SetProjectDefaults by @robert-uhl in #9468
• [PFS-158-projects] Implement ForEachProject() and ListProject() by @FahadBSyed in #9481
• [PFS-166] Commit.branches should be a copy and not a reference by @FahadBSyed in #9487
• Fix error handling in pipeline controller by @albscui in #9485
• fix unknown directive toolchain issue for iac test envs by @molinamelendezj in #9479
• Exclude the PJS API from pachyderm-sdk (for now) by @msteffen in #9486
• [CORE-1720] Add pachctl update project defaults by @robert-uhl in #9471
• [CORE-1719] Add pachctl inspect project defaults by @robert-uhl in #9470
• [CORE-1721] Implement pachctl delete project defaults by @robert-uhl in #9472
• [CORE-1718] Implement pachctl create project defaults by @robert-uhl in #9473
• Update Error Handling Code by @FahadBSyed in #9490
• [PFS-151][PFS-152][PFS-157] New squash / drop commit RPCs and implementations by @brycemcanally in #9492
• example pipelines *.pipeline.json in release tests by @molinamelendezj in #9493
• [CORE-2035] Update gRPC by @robert-uhl in #9501
• [Python-SDK] upgrade grpcio dependency by @bbonenfant in #9498
• [Python-SDK] containerize documentation generation by @bbonenfant in #9497
• [Jupyter] Better handling of PFSConentManager errors by @bbonenfant in #9494
• Djanicek/allow upload timeout by @djanicekpach in #9516
• [Mount Server] Emulate pipeline validation for input spec when mounting datums by @smalyala in #9458
• [CORE-2040] Add squash to project defaults migration by @robert-uhl in #9519
• [CORE-2037] Correct auth check for project defaults by @robert-uhl in #9521
• make qa1 iac test env bit bigger by @molinamelendezj in #9525
• Make sure self reference is impossible in edges. by @albscui in #9510
• [CORE-2942] Update to Go 1.21.4 by @robert-uhl in #9526
• [PFS-158] Add ForEachCommit() and ListCommit() by @FahadBSyed in #9496
• [CORE-2038] Fix Scenario Where Commits Are Not Migrated by @FahadBSyed in #9528
• [Python-SDK] Bugfix: AuthServiceNotActivated errors not being raised by @bbonenfant in #9522
• [CORE-1723] Add/update tests for project defaults by @robert-uhl in #9500
• Remove details argument from ListPipelineRequest by @robert-uhl in #9509

New Contributors

• @dgeorg42 made their first contribution in #9194

Full Changelog: v2.8.0-alpha.5...v2.8.0-nightly.20231113