Changelog

1.25.0 (2024-05-06)

Features

monitor warehouse databases (#4619) (e550a11)
onboard new generic destination sftp (#4601) (c6a28bb)
onboard Yandex Metrica Offline Events destination (#4534) (904d8fb)

Miscellaneous

add support for live events in internal batch endpoint (#4627) (26ecd98)
cleanup *jobsdb.Handle.checkIfFullDSInTx(...) (#4634) (c64dd3a)
deps: bump golangci/golangci-lint-action from 4 to 5 (#4620) (566f430)
release 1.24.0 (#4617) (#4625) (581d5aa)
remove composite indices on rt, batch_rt (#4632) (8828cc2)

1.24.0 (2024-04-25)

Features

adaptations of additional helpers packages from go-kit (#4562) (3deadee)
adapting to helpers in go-kit (#4560) (3c515dd)
additional of some helpers fot go-kit (3c515dd)
append only tables for redshift (#4596) (9a89780)
introduce tx idle timeout in Postgres connections (#4598) (043b4c2)

Bug Fixes

actions cancelling on push to master (#4578) (c47bd91)
correctly quote connection on subscription creation (#4606) (ef35f90)
ever increasing idle http connections while fetching transformer features (#4594) (a44f92e)
health endpoint backwards compatibility (#4614) (87a2dfa)
update to check for supported scopes for OAuth destinations (#4585) (f1a8b8c)

Miscellaneous

adapt internal endpoint to ingest msg in new format (#4587) (e7d034c)
add error logs for oauthv2 errors (#4608) (ba98beb)
add sourceCategory label to event_delivery_time metric (#4569) (1a5da41)
configure histogram buckets and add sourceId label on event_delivery_time metric (#4559) (be5d501)
deps: bump github.com/rudderlabs/rudder-go-kit from 0.23.3 to 0.25.0 (#4565) (d7e9a9e)
deps: bump google.golang.org/grpc from 1.62.1 to 1.63.0 (#4544) (e738a20)
deps: bump the go-deps group across 1 directory with 3 updates (#4602) (5f9dd27)
deps: bump the go-deps group across 1 directory with 8 updates (#4595) (c7398de)
deps: bump the go-deps group with 2 updates (#4580) (a9e94b1)
deps: bump the go-deps group with 5 updates (#4577) (72d63ea)
group minor, patch dependabot updates (#4576) (c5d1943)
log config init errors using exposed methods (#4541) (ac14cdb)
merge release 1.23.0 in main branch (#4558) (6d5be81)
merge release 1.23.0 in main branch (#4604) (fb6737c)
refactor onConfigDataEvent (#4540) (88ebb15)
release 1.23.0 (#4557) (6d5be81)
remove recovery module (#4584) (5c15c7e)
remove unused code (#4579) (c41ae57)
revert "feat: adapting to helpers in go-kit" (#4561) (03eb2ab)
workflow concurrency=1 (#4573) (3555d1f)

1.23.0 (2024-04-08)

Features

add support to build arm64 image (#4467) (d1cf2c2)
optimise reporting module queries (#4472) (0da9f28)
update oauth module in server to interceptor pattern (#4329) (e154286)

Bug Fixes

filter success events before sending it to error reporting table (6b2e957)
filter success events before sending it to error reporting table (#4505) (1961019)
health check for ssh server (#4551) (4c512bf)
lint issues (559d701)
optimise WH Syncs page query (#4507) (edf8624)
robust dangling table deletion during recovery (#4519) (6ea8ff3)
terminate goroutines if reporting mainLoop panics (#4457) (2c46967)

Miscellaneous

added integration test for config BE un-availability (#4473) (b88fdf2)
addressing minor comment (1bf68b4)
code review changes (d757132)
code review changes (cd9a37f)
code review changes (915a697)
deps: bump cloud.google.com/go/bigquery from 1.59.1 to 1.60.0 (#4531) (7dbd0a2)
deps: bump cloud.google.com/go/pubsub from 1.36.1 to 1.37.0 (#4545) (352647b)
deps: bump docker/login-action from 2.1.0 to 3.1.0 (#4543) (f3a550f)
deps: bump docker/setup-buildx-action from 2 to 3 (#4542) (c02d0b2)
deps: bump github.com/apache/pulsar-client-go from 0.12.0 to 0.12.1 (#4493) (f51b714)
deps: bump github.com/aws/aws-sdk-go from 1.50.38 to 1.51.6 (#4518) (66342e6)
deps: bump github.com/bugsnag/bugsnag-go/v2 from 2.3.0 to 2.3.1 (#4530) (0987a20)
deps: bump github.com/gomodule/redigo from 1.8.9 to 1.9.2 (#4494) (3c8ed38)
deps: bump github.com/marcboeker/go-duckdb from 1.6.1 to 1.6.2 (#4529) (c756d2f)
deps: bump github.com/onsi/ginkgo/v2 from 2.15.0 to 2.17.1 (#4532) (5f2e0e6)
deps: bump github.com/onsi/gomega from 1.31.1 to 1.32.0 (#4522) (d3f9970)
deps: bump github.com/redis/go-redis/v9 from 9.4.0 to 9.5.1 (#4485) (31f368b)
deps: bump github.com/rudderlabs/rudder-go-kit from 0.23.2 to 0.23.3 (#4533) (a8aa5ed)
deps: bump github.com/snowflakedb/gosnowflake from 1.8.0 to 1.9.0 (#4527) (1197b6d)
deps: bump go.etcd.io/etcd/api/v3 from 3.5.12 to 3.5.13 (#4548) (b8cab35)
deps: bump google.golang.org/api from 0.167.0 to 0.172.0 (#4528) (d06d711)
disable IPv6 as temp fix for port conflict in our tests (#4523) (b98f747)
fetch pileup counts concurrently (#4517) (f27a0f9)
lint fixes (c934ee7)
lint issues fix (37686c6)
merge 1.22.0 release to master (#4503) (ee993e3)
remove rudder id from internal batch endpoint (#4549) (889e33f)
retry cron tracker with backoff (#4526) (2428c80)
script to migrate old backup files to new format (#4397) (abdb873)
upgrade to go version 1.22.1 and tools (#4521) (a705299)
version handler cleanup (#4496) (8ef80d1)

1.22.1 (2024-03-20)

Bug Fixes

optimise WH Syncs page query (#4507) (e0de4b3)

1.22.0 (2024-03-19)

Bug Fixes

decrease async destination pending events (229e9d3)
destType issue in eloqua stats (#4475) (229e9d3)
eloqua: destType issue (229e9d3)
too many connections in router transform (#4492) (3da9179)

Miscellaneous

deps: bump github.com/bugsnag/bugsnag-go/v2 from 2.2.1 to 2.3.0 (229e9d3)
deps: bump github.com/marcboeker/go-duckdb from 1.5.6 to 1.6.1 (#4469) (229e9d3)
deps: bump github.com/marcboeker/go-duckdb from 1.5.6 to 1.6.1 (#4469) (5438a85)
deps: bump github.com/prometheus/client_model from 0.5.0 to 0.6.0 (229e9d3)
deps: bump github.com/prometheus/client_model from 0.5.0 to 0.6.0 (#4448) (229e9d3)
deps: bump github.com/tidwall/gjson from 1.17.0 to 1.17.1 (229e9d3)
deps: bump github.com/tidwall/gjson from 1.17.0 to 1.17.1 (#4449) (229e9d3)
deps: bump google.golang.org/grpc from 1.61.0 to 1.62.1 (#4454) (6baa913)
deps: bump google.golang.org/protobuf from 1.32.0 to 1.33.0 (#4474) (229e9d3)
deps: bump google.golang.org/protobuf from 1.32.0 to 1.33.0 (#4474) (66dcdb6)
fix databricks merge query (#4471) (b17cd48)
forcing major release 1.22.0 (2065907)
go-kit bump 0.20.2 -> 0.23.2 (#4459) (f4185e2)
sending blank event to event schemas (#4484) (858c251)
vacuum reports table when size > 5GB (229e9d3)

1.21.1 (2024-03-14)

Bug Fixes

always look for list item error for bingads audience (#4430) (b9a02e9)

1.21.0 (2024-03-06)

Features

internal batch endpoint (#4394) (3f87930)
openAPI spec for rudder-server (#4376) (a606282)
toggle event ordering for workspace/destination (#4278) (3d43b69)

Bug Fixes

dontbatch-not-persisted (f13b550)
empty customVal in rt tables (#4437) (5abe242)
enable archiver for use rudder storage for dtaging file entries (#4433) (ea78ee5)
send throttling rate limit per second (#4419) (3501578)
wrong processor out of order stats (#4426) (00b43d7)

Miscellaneous

deps: bump github.com/aws/aws-sdk-go from 1.50.13 to 1.50.23 (#4412) (019b128)
deps: bump github.com/aws/aws-sdk-go from 1.50.23 to 1.50.29 (#4438) (8886f17)
deps: bump github.com/bugsnag/bugsnag-go/v2 from 2.2.0 to 2.2.1 (#4422) (fe95e15)
deps: bump github.com/minio/minio-go/v7 from 7.0.66 to 7.0.67 (#4415) (b569f78)
deps: bump github.com/snowflakedb/gosnowflake from 1.7.2 to 1.8.0 (a044053)
deps: bump github.com/snowflakedb/gosnowflake from 1.7.2 to 1.8.0 (#4442) (4070a17)
deps: bump golang.org/x/oauth2 from 0.16.0 to 0.17.0 (#4386) (2f5c7ae)
deps: bump golangci/golangci-lint-action from 3 to 4 (#4393) (c9928f0)
fast http client in processor transformer (#4432) (1bc759b)
fix snyk reported security issues with benchmark k8s deployment (#4400) (ba93342)
merge release 1.20.0 in main branch (#4391) (99db4e2)
processor transformer uses parent ctx to create new request (#4431) (8dde92c)
send source name in metadata to transformer (#4443) (3088a10)
set 1000 as default max limit (#4315) (02fbd08)
some non-exhaustive processor drop count stats (52a828e)
some non-exhaustive processor drop count stats (#4446) (61a820d)
ununsed code (#4408) (d8c413e)

1.20.0 (2024-02-12)

Features

add 'config' field to proxy request for v0 (#4352) (09b9f71)
adding sourceId and destinationId in pipeline info metrics (#4332) (c2d7a42)
update error table with new columns (#4356) (9e8ebc6)

Bug Fixes

batchrouter async dest pending event counts aren't being decreased (#4346) (10c653c)
handle consent management configuration fallback for gcm (#4355) (a418e03)
ignore gwJobs' destinationID in processor rsources.statCollector (#4321) (ec91612)
jobs not draining as fast as we'd like because of rate-limiting etc. (#4327) (ed301a3)
marketo bulk upload's upload url preparation fix (#4358) (c58b486)
nil load statistics for bigquery (#4328) (ec91612)

Miscellaneous

add tests for aiokafka consumer (#4372) (dba6d31)
deps: bump arduino/setup-protoc from 2 to 3 (#4363) (14b84bf)
deps: bump cloud.google.com/go/bigquery from 1.58.0 to 1.59.0 (#4374) (0b88164)
deps: bump cloud.google.com/go/pubsub from 1.36.0 to 1.36.1 (#4369) (6d26af0)
deps: bump codecov/codecov-action from 3 to 4 (#4362) (1cb7af1)
deps: bump github.com/aws/aws-sdk-go from 1.49.21 to 1.50.0 (#4341) (08aa4ae)
deps: bump github.com/aws/aws-sdk-go from 1.50.10 to 1.50.13 (#4378) (bb321c8)
deps: bump github.com/aws/aws-sdk-go from 1.50.6 to 1.50.10 (#4365) (95ba15b)
deps: bump github.com/golang-migrate/migrate/v4 from 4.16.2 to 4.17.0 (#4302) (a086bfa)
deps: bump github.com/grpc-ecosystem/grpc-gateway/v2 from 2.18.1 to 2.19.0 (#4336) (3c63569)
deps: bump github.com/onsi/ginkgo/v2 from 2.13.2 to 2.15.0 (#4345) (592fea7)
deps: bump github.com/onsi/gomega from 1.30.0 to 1.31.1 (#4348) (5d23637)
deps: bump github.com/rudderlabs/bing-ads-go-sdk from 0.2.0 to 0.2.1 (#4339) (c23c6c3)
deps: bump github.com/rudderlabs/rudder-go-kit from 0.19.0 to 0.19.1 (#4338) (abae730)
deps: bump github.com/samber/lo from 1.38.1 to 1.39.0 (#4320) (a46cf47)
deps: bump github.com/segmentio/kafka-go from 0.4.44 to 0.4.47 (#4343) (03e6be9)
deps: bump github.com/urfave/cli/v2 from 2.25.7 to 2.27.1 (#4318) (5a0493f)
deps: bump go.etcd.io/etcd/api/v3 from 3.5.11 to 3.5.12 (#4367) (ea66769)
deps: bump go.etcd.io/etcd/client/v3 from 3.5.11 to 3.5.12 (#4368) (3acd5e6)
deps: bump golang.org/x/oauth2 from 0.15.0 to 0.16.0 (#4319) (cbf1b0f)
deps: bump google.golang.org/api from 0.156.0 to 0.157.0 (#4342) (3e448f9)
deps: bump google.golang.org/api from 0.160.0 to 0.161.0 (#4366) (b917316)
deps: bump peter-evans/repository-dispatch from 2 to 3 (#4354) (166de8a)
fix code cov breaking change (#4379) (888a354)
integration test for rETL flow (#4324) (8c92520)
reduce cardinality of warehouse metrics (#4364) (2cd917c)
revert update error table with new columns (#4380) (8119fe3)
set container and account name for load file does not exist test for databricks (#4361) (59ecda8)
using new kit kafka client and docker resources (#4350) (4fa88cb)

1.19.2 (2024-01-18)

Bug Fixes

nil load statistics for bigquery (#4328) (48cd75c)

1.19.1 (2024-01-16)

Bug Fixes

ignore gwJobs' destinationID in processor rsources.statCollector (#4321) (8d58bd5)

1.19.0 (2024-01-15)

Features

add 'config' field to proxy request (#4264) (1c3fb01)
add destinationID to rETL endpoint (#4234) (2c02dc2)
add support for regulation from features.json (#4304) (3c2fcf6)
add version deprecation error-code identification logic (#3995) (866393d)
added sprig in supportedDestionations for user deletion (#4285) (e3b8018)
filter destinations for events from rETL in processor (#4247) (2b82920)
lower retry limits(fail-fast) for sources jobs in batchrouter (#4310) (d7cbc8c)
support for adaptive rate limiting [PIPE-481] (#4160) (40586e5)
tracing support for gateway, processor and router (#4248) (7dc7747)
warehouse append vs merge (#4139) (40586e5)

Bug Fixes

adapt connection timeout for sql for redshift (#4253) (1ff6f90)
adaptiveWindow default value should be more than 0 (#4312) (5108d45)
add wait for get supported destinations in regulation worker (#4311) (cfed04a)
bigquery job statistics for rate limit (#4272) (9842f64)
changing eventNames longer than configured max length to ":max-length-exceeded:" before sending to reporting (#4244) (719396f)
ci: fix distpatch event action typo (#4252) (1057ee1)
exhaust error index jobs when work is called (#4295) (b7b75ad)
gateway stat tags when source is disabled (#4215) (85235bb)
ignore destinationID in gateway rsources.statCollector (#4299) (eece12b)
processor: duplicate rsources out stats can be recorded in case of a retry due to an operation timeout (#4223) (a7c373a)
pubsub channel not closing during close if no value has been published (#4269) (ab0e0d9)
return only the latest namespace entry when fetching tables from warehouse (#4287) (8f1e6b1)
server panics during shutdown with reporting metrics: failed to store jobs: context canceled (#4228) (68e52d5)

Miscellaneous

adapt rudder-go-kit v0.19.0 changes (#4227) (08c0864)
add env to enable throttlerV2 (#4313) (139d46e)
additional error mappings for deltalake (#4265) (35cc8e9)
autofix corrupted job-status stats (#4224) (781f24f)
backend config calls metric (#4306) (b03aaba)
backend config response size metric (#4309) (2fbf4c5)
change the max limit to take multiplier of the existing limits (#4263) (7c4cc03)
ci: automated deployment (875cc54)
custom buckets for gw request size (#4275) (6466a9b)
deps: bump actions/download-artifact from 3 to 4 (#4240) (d8d980d)
deps: bump actions/labeler from 4 to 5 (#4192) (21c5fab)
deps: bump actions/setup-go from 4 to 5 (#4212) (c306dff)
deps: bump actions/stale from 8 to 9 (#4216) (6f88c3f)
deps: bump actions/upload-artifact from 3 to 4 (#4239) (426d3bf)
deps: bump cloud.google.com/go/storage from 1.34.1 to 1.36.0 (#4236) (db00c39)
deps: bump github.com/aws/aws-sdk-go from 1.48.11 to 1.49.15 (#4283) (78a64d3)
deps: bump github.com/aws/aws-sdk-go from 1.49.15 to 1.49.17 (#4294) (3088bba)
deps: bump github.com/dvsekhvalnov/jose2go from 1.5.0 to 1.6.0 (#4260) (bfa407f)
deps: bump github.com/marcboeker/go-duckdb from 1.5.1 to 1.5.6 (#4277) (854242c)
deps: bump github.com/mkmik/multierror from 0.3.0 to 0.4.0 (#4276) (47a2d14)
deps: bump github.com/redis/go-redis/v9 from 9.3.0 to 9.4.0 (#4292) (5bdc43c)
deps: bump github.com/snowflakedb/gosnowflake from 1.6.25 to 1.7.0 (#4190) (40586e5)
deps: bump github.com/snowflakedb/gosnowflake from 1.6.25 to 1.7.0 (#4190) (2072e2e)
deps: bump github.com/snowflakedb/gosnowflake from 1.7.0 to 1.7.1 (#4300) (cad4c62)
deps: bump github.com/spf13/cast from 1.5.1 to 1.6.0 (#4191) (8aa4294)
deps: bump go.etcd.io/etcd/client/v3 from 3.5.10 to 3.5.11 (#4254) (c1758a6)
deps: bump golang.org/x/sync from 0.5.0 to 0.6.0 (#4291) (266fb11)
deps: bump google.golang.org/api from 0.153.0 to 0.155.0 (#4282) (3ecc0a6)
enabling gateway to ingest events even when sharedDB is down (#4262) (bd365b5)
error index file path collision (#4297) (37b2e35)
fix flaky test for tables for connections from wh schemas as ordering for tables can be different (#4307) (ac2e155)
flaky docker test (#4286) (b928bfe)
flaky oauth test (#4280) (d078c93)
improve error logging (#4288) (9cb9caf)
improve error logs during jobsdb backup table test (#4308) (aea58d7)
improve error logs during jobsdb backup table test: (aea58d7)
include the component in application_name of postgres connection (#4225) (dcf73f0)
remove events schemas v1 (#3923) (40586e5)
remove events schemas v1 (#3923) (14466dc)
remove namespace tag from error_detail_reporting_failures metric (#4305) (6c2d900)
source category set to event stream as default (#4226) (af82a6f)
support concurrent addition of failed keys for the same target (#4241) (95ff5ae)
tune shared db connection pooling (#4213) (cfe458a)
upgrade 'pinned' go package versions (#4270) (befbc96)
upgrade sql tunnels to v0.1.6 (#4296) (47dc055)
use local db in rsources handler for verifying job completed status (#4245) (d53aa80)

1.18.2 (2023-12-12)

Bug Fixes

server panics during shutdown with reporting metrics: failed to store jobs: context canceled (#4228) (c5cb5a8)

1.18.1 (2023-12-07)

Miscellaneous

tune shared db connection pooling (#4213) (01c90fc)

1.18.0 (2023-12-05)

Features

a table for all drain configuration(jobRunID for now) (#4153) (3d605d3)
append only tables config for snowflake (#4186) (09a1ab0)
capture delayed events (#4104) (f9f8782)
generic consent management (#4056) (0f202e8)
include error codes in v2 failed-records response payload (#4116) (e803bf9)
introduce random sleep before clickhouse loads (#4193) (85cfdcf)
partial failure support for delivery via transformer proxy (#4131) (a7e2e81)

Bug Fixes

add autovacuum_vacuum_cost_limit to the reports table [PIPE-512] (#4136) (690aeb0)
align gha pr comment with workflow config (#4194) (1355c96)
clickhouse zookeeper table metadata (#4121) (41e060a)
error reporting handling and config changes (#4195) (70af46e)
gateway responds with http status 500 and body pq: invalid byte sequence for encoding UTF8: 0x00 (#4161) (7b118b9)
graceful termination for cron tracker (#4128) (cb33412)
snowflake delete (#4179) (906e5b0)
source no pending jobs (#4197) (79f8551)
warehouse archiver integration tests (#4135) (cb33412)
warehouse: crash recovery without panics (#4182) (a97436e)

Miscellaneous

add a dial timeout of 10 seconds in etcd client during tests (#4127) (cb33412)
deps: bump actions/checkout from 3 to 4 (#4151) (1621628)
deps: bump actions/stale from 5 to 8 (#4149) (0534c38)
deps: bump amannn/action-semantic-pull-request from 4 to 5 (#4168) (324e9d9)
deps: bump arduino/setup-protoc from 1 to 2 (#4150) (f97b69c)
deps: bump beatlabs/delete-old-branches-action from 0.0.9 to 0.0.10 (#4147) (95f5ec2)
deps: bump docker/build-push-action from 3 to 5 (#4148) (cebce15)
deps: bump docker/login-action from 2 to 3 (#4155) (38986ed)
deps: bump docker/metadata-action from 4 to 5 (#4167) (a61ab5f)
deps: bump github.com/aws/aws-sdk-go from 1.46.4 to 1.47.10 (#4134) (cb33412)
deps: bump github.com/aws/aws-sdk-go from 1.47.10 to 1.47.11 (#4143) (3e103d3)
deps: bump github.com/grpc-ecosystem/grpc-gateway/v2 from 2.16.0 to 2.18.1 (#4091) (cb33412)
deps: bump github.com/hashicorp/go-retryablehttp from 0.7.4 to 0.7.5 (#4107) (2400b7a)
deps: bump github.com/minio/minio-go/v7 from 7.0.63 to 7.0.64 (#4177) (6da3150)
deps: bump github.com/onsi/ginkgo/v2 from 2.13.0 to 2.13.2 (#4180) (58d1a97)
deps: bump github.com/onsi/gomega from 1.29.0 to 1.30.0 (#4114) (cb33412)
deps: bump golang.org/x/oauth2 from 0.14.0 to 0.15.0 (#4185) (6dbccb3)
deps: bump google-github-actions/release-please-action from 3 to 4 (#4184) (5845a02)
deps: revert google-github-actions/release-please-action from 4 to 3 (#4203) (26ef4b1)
enforce slices instead of exp/slices pkg (#4047) (cb33412)
error logs for unmarshal error debugging (#4181) (852e1b9)
filtered and error stats separated in processor (#4137) (22e4944)
improve fatal log during panic capture (#4189) (7ff4103)
improve rsources service table setup procedure (#4165) (42d5130)
include all aborted jobs in failed records (362f65a)
include all aborted jobs in failed records (#4154) (ba5b80f)
include github-actions and docker updates (PIPE-539) (#4144) (61f3957)
increasing health timeout (6c28e25)
introduce loadfiles GetByID (#4093) (cb33412)
kafka batch integration test (#4146) (6c28e25)
linter fixes for enterprise and event-schema in rudder-server (#4021) (cb33412)
populate total events (#4099) (cb33412)
processor: include partition tag in stats (#4170) (1eecaa9)
revert data bricks sql client to 1.4.0 (#4157) (ea455ca)
router: fine grained job iterator configuration (#4169) (b1bbf7a)
router: interruption of iterator should cause router to sleep (#4171) (d29f819)
router: interuption of iterator should cause router to sleep (d29f819)
split integration tests (#4138) (cb33412)
trim down error responses stored in job status to a maximum of 10KB (#4166) (44f7456)
uncomment clickhouse integration test (#4156) (554acfa)

1.17.3 (2023-12-05)

Bug Fixes

source no pending jobs (#4197) (221d7da)

1.17.2 (2023-11-20)

Bug Fixes

gateway responds with http status 500 and body pq: invalid byte sequence for encoding UTF8: 0x00 (#4161) (2c168ef)

1.17.1 (2023-11-17)

Miscellaneous

revert data bricks sql client to 1.4.0 (#4157) (c42f289)

1.17.0 (2023-11-14)

Features

introducing warehouse repo withTx (#4042) (6973c75)
ketch consent manager cloud mode filtering (#4077) (47b2f92)
new event type - record (#3879) (6973c75)
new event type - record (#3879) (ecae511)
new sources endpoints (#4076) (07a7db3)
new transformer service to fetch and serve transformer features (#4007) (f95fa51)
router: disable event order for users with event volume above threshold (#4067) (6973c75)
router: disable event order for users with event volume above threshold (#4067) (266a256)
track event delivery stats (#3974) (24294a8)

Bug Fixes

destination: oauth stats prefix (#4033) (6973c75)
don't send error in stats (#4055) (6973c75)
error index sortKey filter based test (#4102) (836b0f0)
invalid input syntax for type json for table esch_job_status (#4090) (6973c75)
minio heathcheck (#4068) (6973c75)
sanitize messageID from \u0000 and irregular utf8 runes (#4063) (6973c75)
webhook source transformation failed with error: cannot find module (#4120) (f4686aa)

Miscellaneous

adapt health.WaitUntilReady (#4081) (6973c75)
apk usage in Dockerfile (#3835) (6973c75)
deps: bump cloud.google.com/go/bigquery from 1.55.0 to 1.56.0 (#4039) (6973c75)
deps: bump cloud.google.com/go/bigquery from 1.56.0 to 1.57.1 (#4085) (6973c75)
deps: bump cloud.google.com/go/storage from 1.33.0 to 1.34.1 (#4086) (6973c75)
deps: bump github.com/apache/pulsar-client-go from 0.11.0 to 0.11.1 (#4097) (476f223)
deps: bump github.com/databricks/databricks-sql-go from 1.4.0 to 1.5.1 (#4084) (6973c75)
deps: bump github.com/onsi/gomega from 1.27.10 to 1.29.0 (#4038) (6973c75)
deps: bump github.com/oschwald/maxminddb-golang from 1.11.0 to 1.12.0 (#4073) (6973c75)
deps: bump github.com/redis/go-redis/v9 from 9.1.0 to 9.3.0 (#4074) (6973c75)
deps: bump github.com/segmentio/kafka-go from 0.4.42 to 0.4.44 (#4083) (6973c75)
deps: bump github.com/tidwall/gjson from 1.16.0 to 1.17.0 (#4035) (6973c75)
deps: bump go.etcd.io/etcd/client/v3 from 3.5.9 to 3.5.10 (#4036) (6973c75)
deps: bump golang.org/x/sync from 0.4.0 to 0.5.0 (#4095) (4b5e926)
deps: bump google.golang.org/api from 0.148.0 to 0.149.0 (#4072) (6973c75)
deps: bump google.golang.org/grpc from 1.58.3 to 1.59.0 (#4075) (6973c75)
have processor's rsources dropped stats to include only source info and not destination (#4087) (6973c75)
increasing retries for health check for trino (#4105) (4e4adcf)
meaningful name for receivers (#4064) (6973c75)
minor formatting changes (#4079) (6973c75)
minor processor cleanup (#3876) (6973c75)
remove apply hot fixes to master action (#4054) (6973c75)
remove panic and trigger full update on incremental update failure (#4109) (fcb7e87)
remove workspace ack in multitenant setup [PIPE-474] (#4066) (6973c75)
revert append vs merge (#4129) (9c860c6)
set gw rate limits at event level (#4069) (6973c75)
sources async job (#4008) (6973c75)
split state machine (#4058) (11efbd3)
state for uploads (#4057) (6973c75)
support generic rules to have routers drain events (#3856) (6973c75)
upgrade net library (#4065) (6973c75)
uploads default logFields and repo load files queries (#4089) (6973c75)
uploads updates (#4045) (6973c75)
use latest dependencies for tests (#4092) (6973c75)
use latest dependencies for tests (#4092) (d76bdb9)

1.16.3 (2023-11-09)

Bug Fixes

error index sortKey filter based test (#4102) (836b0f0)

Miscellaneous

remove panic and trigger full update on incremental update failure (#4109) (92e0918)

1.16.2 (2023-11-08)

Bug Fixes

preferAppend defaults to false if not defined (#4098) (941caba)

1.16.1 (2023-11-06)

Bug Fixes

sorting key for error indexing during parquet (#4078) (a029a81)

1.16.0 (2023-10-31)

Features

error index reporting implementation (#3948) (51220da)
grpc middleware for stats (#4030) (a524cbc)
introduce pagination in failed-keys endpoint (#3967) (aa8044a)
introducing chi middleware for warehouse (#4010) (4d9f9b3)
oauth access denied handling (#3960) (a53a127)
processor: added ability for geolocation enrichment during pipeline processing (#3866) (28497cf)
push error index metadata (#4002) (72423dd)
report failed messages in processor, router and batchrouter (#3914) (51220da)
use append vs merge option from backend config (#3965) (6d2db45)

Bug Fixes

add error log for reporting metrics (#3978) (0963193)
config for stash payload limit (#4001) (f4c93ce)
corrupted rsources stats captured by processor for dropped jobs (#3999) (e74cd7d)
error index filtering for timestamp fields use int64 (#4062) (17590a6)
flaky validations tests (#4012) (3b157e3)
invalid memory address or nil pointer dereference in googlecloudfunction (#4003) (37690ed)
merge error blocks in gcf (#4004) (03a4c26)
update error parsing of eloqua (#3996) (e74cd7d)
validations tests (3b157e3)

Miscellaneous

add workspaceID to router discarded stats (#3977) (51220da)
advertise gzip support to transformer through X-Feature-Gzip-Support header (#3990) (10c0ffe)
avoid using global conf during tests (#4046) (4e3d477)
bump rudder-go-kit to 1.16.2 (#4026) (be29d5b)
collect only drained failed keys at router (#3930) (51220da)
deps: bump github.com/confluentinc/confluent-kafka-go/v2 from 2.2.0 to 2.3.0 (#4024) (5192a09)
deps: bump github.com/google/uuid from 1.3.1 to 1.4.0 (#4022) (a4dd910)
deps: bump github.com/hashicorp/golang-lru/v2 from 2.0.6 to 2.0.7 (#4023) (8e0796a)
deps: bump github.com/rs/cors from 1.10.0 to 1.10.1 (#4017) (4010776)
deps: bump github.com/snowflakedb/gosnowflake from 1.6.24 to 1.6.25 (#4025) (43add27)
deps: bump github.com/trinodb/trino-go-client from 0.312.0 to 0.313.0 (#4016) (1ee9f56)
deps: bump go.uber.org/goleak from 1.2.1 to 1.3.0 (#4019) (f9c9615)
deps: bump google.golang.org/grpc from 1.58.2 to 1.58.3 (#4011) (3840612)
enable errcheck and unparam linters for warehouse (#3970) (51220da)
error index reporter improvements (#3994) (55f0737)
fix remove warehouse jobs panic (#3982) (6e0729c)
flaky error index report test (#3988) (9b7157f)
flaky tests (#3989) (4db35e9)
flaky validations test (#4027) (3ee4c7b)
go kit v1.16.0 (#4014) (f200683)
increase archiver postgres shm size (#4040) (bd855f6)
migrate to minio resource from rudder-go kit (#4028) (3ba0260)
minor scheduler cleanup (#4032) (2cc9470)
minor tunneling cleanup (#4034) (877eb70)
replace golang.org/x/exp/slices to slices (#4031) (f014c01)
replace varcheck and deadcode with unused linter (#3968) (51220da)
scheduler cleanup: avoid globals (2cc9470)
track long-running transformations in processor (#3976) (51220da)
tunneling cleanup (877eb70)
upgrade go version to 1.21.3 (#3973) (11c3f28)
upgrade golangci-lint (#4029) (554a04a)
upgrade urfave/cli v2 for rudder-cli (#3980) (d0d99bc)
use a normalised data model for storing failed keys (#3961) (aa8044a)

1.15.4 (2023-10-23)

Bug Fixes

invalid memory address or nil pointer dereference in googlecloudfunction (#4003) (ccb6068)

1.15.3 (2023-10-19)

Bug Fixes

corrupted rsources stats captured by processor for dropped jobs (#3999) (e7b829d)
update error parsing of eloqua (#3996) (978c292)

1.15.2 (2023-10-18)

Miscellaneous

advertise gzip support to transformer through X-Feature-Gzip-Support header (#3990) (bce584d)

1.15.1 (2023-10-13)

Bug Fixes

create schema before even fetching for deltalake (#3971) (50f9758)

Miscellaneous

send filtered events in ut as dropped (#3972) (b7590d2)

1.15.0 (2023-10-11)

Features

add gzip bodyFormat support (#3904) (b050450)
added flags for event audit (#3859) (890ca68)
filter events support (#3882) (7ead8a9)
health dashboard (#3906) (dbd4ea6)

Bug Fixes

access_denied error handling for OAuth destinations (#3853) (0d30d3b)
change jobsdb pathPrefix config (a747653)
event schema versions without a sample event (#3913) (ea92a2e)
googlecloudfunction destination implementation and refactor code (#3907) (44f5c23)
increasing default marketo bulk upload timeout (#3940) (bd9166b)
increasing default marketo upload timeout (bd9166b)
invalid JobsDB.backup.pathPrefix configuration (#3921) (a747653)
replay request fails with 400 - request neither has anonymousId nor userId (#3911) (85adbbf)
revert fixes for access_denied error handling for OAuth destinations (#3853) (#3959) (247414b)
router transformer client fails with error connect: cannot assign requested address (#3898) (f6c51b7)
rsources dropped jobs at processor (#3905) (1c4fc5e)
set local schema when syncing from remote (#3956) (6dd93d4)
skip snakecasing for namespace if skipNamespaceSnakeCasing is set to true (#3932) (6da163b)
test ssh (#3915) (36cdb44)
use dedup on new records for deltalake (#3927) (9192975)
wh query types (#3951) (1e415e0)

Miscellaneous

add rsources stats for dropped events at processor (#3852) (f5b8e7b)
adding missing query types (#3917) (be24be3)
app for warehouse (#3862) (5c9789f)
applying 1.14.3 hotfixes to main branch (#3926) (e08decb)
backmerge 1.14.5 to master (#3935) (c7a485f)
cleanup JobsDB.useSingleGetJobsQuery config option (#3893) (be0a62e)
cleanup notifier (#3867) (98498ee)
different router retry limits for sources jobs (#3944) (f13dbc3)
empty batch payload (#3947) (1e20655)
enable user deletion for supported destinations (#3946) (6ceb4e5)
fix archiver test - inconsistent number of files (#3902) (01a0737)
introduce load table total rows (#3851) (1fcabdd)
moving uploadSchema into UploadJob (#3888) (2a5547f)
omit failedMessages from reporting json object (#3936) (0969b31)
populate error category (#3949) (37bbbbe)
reporting feature (#3912) (37b9cbc)
rsources tests - postgres resources use different creds (#3919) (7ca721c)
sampling errors if no message in the batch succeeds (#3918) (69be777)
split warehouse package (#3937) (c94c953)
support replay with new file format (#3834) (5c76185)
upgrade go mod replace versions (#3880) (3c3e407)
upgrade go mod replace versions (#3880) (5c3aefb)
use common PAT instead of rudder-server-bot (#3952) (08d036e)
use memoized payloadFunc for dedup (#3780) (74f5f11)
use new reloadable config api for warehouse (#3920) (6f2b7b9)
use publicly available transformer docker image (#3916) (5f3820f)
use the new reloadable config api in reporting (#3909) (741fd74)
warehouse schema race and cleanup (#3900) (61883be)

1.14.6 (2023-10-04)

Bug Fixes

skip snakecasing for namespace if skipNamespaceSnakeCasing is set to true (#3932) (8b3cb32)

1.14.5 (2023-10-03)

Miscellaneous

fix backup prefix (#3933) (206e20e)

1.14.4 (2023-09-29)

Bug Fixes

use dedup on new records for deltalake (#3927) (5656ab3)

1.14.3 (2023-09-28)

Miscellaneous

added flags for event audit (#3924) (6908b0a)

1.14.2 (2023-09-25)

Bug Fixes

googlecloudfunction destination implementation and refactor code (#3907) (0676b0a)

1.14.1 (2023-09-21)

Bug Fixes

router transformer client fails with error connect: cannot assign requested address (#3898) (1cb5d5d)

1.14.0 (2023-09-20)

Features

onboard bulk upload destination eloqua (#3779) (0a9954d)
onboard destination google cloud function (#3794) (3b616a5)

Bug Fixes

detected illegal job sequence during barrier enter/wait (#3881) (7891da3)
error message for deltalake during test connection (#3883) (1fa2f45)
error while cleaning up old jobs: sql: no rows in result set (#3850) (1f87a26)
jobsdb ds max table size (#3887) (38e604a)
redis cluster mode (#3831) (3b90266)
remove table name for load file (#3837) (3b90266)
sample duplicate messagesids for snowflake (#3884) (b06dc36)
send reportedAt in milliseconds for error reporting (#3857) (7ec96c9)
update error processing of eloqua (#3877) (c22f22e)

Miscellaneous

add new logo (#3865) (7f290f4)
add rudder-cli source and rm binaries (#3872) (c7ab148)
add stats for archiver (#3836) (3b90266)
alerting on reports pileup (#3755) (4bf9ce7)
batchrouter: decouple upload frequency from main loop frequency (#3889) (e188b94)
batchrouter: honour upload frequency when limitsReached if destination is failing (#3874) (ae989cd)
better log message (#3878) (fcd6676)
bump rudderlabs/rudder-go-kit from 0.15.7 to 0.15.8 (#3863) (785577d)
check list permissions for replay (#3873) (fddbc0d)
cleanup grpc (#3814) (594828e)
deps: bump github.com/aws/aws-sdk-go from 1.45.1 to 1.45.3 (#3842) (3b31818)
deps: bump github.com/rs/cors from 1.9.0 to 1.10.0 (#3840) (c1969e1)
deps: bump github.com/rudderlabs/sql-tunnels from 0.1.4 to 0.1.5 (#3841) (a7136c5)
deps: bump golang.org/x/oauth2 from 0.11.0 to 0.12.0 (#3839) (c4c49fe)
don't create custom destination clients for disabled destination (#3885) (aa07262)
go-kit v0.15.9 (#3864) (60749af)
merge release 1.13.0 in main branch (#3833) (0e5a477)
new reloadable config in jobsdb (#3868) (2c655f9)
update joborderlog function (#3848) (3bfe354)
upgrade compose test (#3847) (3a34e06)
upgrade go version 1.21 (#3838) (a1075da)
uploader can append (#3805) (56ab090)
use new reloadable config api in processor and gateway (#3875) (b0576d6)
use new reloadable config api in router and batchrouter (#3871) (12f4d83)
use single query for getting jobs (#3820) (2aff3b2)
use structured logging in jobsdb backup (#3786) (d254bc2)

1.13.2 (2023-09-12)

Bug Fixes

send reportedAt for error-reporting in milliseconds (29ea342)
send reportedAt in milliseconds for error reporting (#3857) (29ea342)

Miscellaneous

fix reporting tests (b241a79)
upgrade compose test (#3847) (8333c35)

1.13.1 (2023-09-06)

Bug Fixes

redis cluster mode (#3831) (3a5e974)
remove table name for load file (#3837) (54bf914)

Miscellaneous

add stats for archiver (#3836) (7b559a0)

1.13.0 (2023-09-05)

Features

archive jobs to object storage (#3721) (c65ded3)
snowflake appendmode (#3745) (2fc1389)
support HSET in redis (#3768) (fd2b341)

Bug Fixes

adding the poll failure response details to db (#3826) (f3d9d11)
correct jobsdb table count stat (#3808) (ebc918e)
do not delete staging files when opted for rudderstorge during archival (#3801) (3dff32d)
don't query db if archival is disabled, set parameters in jobStatus (#3810) (fdfb4f6)
extend sql middleware for mssql, azure_synapse and clickhouse (#3771) (3bfc7e1)
increased postgres cpu usage after switching to a single get jobs query (#3812) (e4a65f3)
marketo bulk upload edits (#3778) (94df125)

Miscellaneous

add gw failure stats (#3809) (437b8d5)
add support for logging duplicate messageIDs (#3759) (890ccb6)
change some worker pool logs to debug (#3744) (6c3fcca)
cleanup http handlers (#3767) (5dacdcf)
cloud extract should always merge (#3800) (7d2446b)
deltalake error message length (#3766) (3bfc7e1)
deps: bump cloud.google.com/go/bigquery from 1.53.0 to 1.54.0 (#3763) (0612cc1)
deps: bump cloud.google.com/go/storage from 1.31.0 to 1.32.0 (#3754) (8ab2ac2)
deps: bump github.com/aws/aws-sdk-go from 1.44.319 to 1.44.323 (#3748) (7d10800)
deps: bump github.com/aws/aws-sdk-go from 1.44.323 to 1.44.324 (#3753) (a3f28c7)
deps: bump github.com/aws/aws-sdk-go from 1.44.324 to 1.44.326 (#3761) (9fcead7)
deps: bump github.com/aws/aws-sdk-go from 1.44.326 to 1.44.327 (#3769) (930c914)
deps: bump github.com/aws/aws-sdk-go from 1.44.327 to 1.44.328 (#3774) (901c69e)
deps: bump github.com/aws/aws-sdk-go from 1.44.328 to 1.44.329 (#3777) (f2184ad)
deps: bump github.com/aws/aws-sdk-go from 1.44.329 to 1.44.330 (#3781) (7da6d0a)
deps: bump github.com/aws/aws-sdk-go from 1.44.330 to 1.44.331 (#3785) (a7d7c74)
deps: bump github.com/aws/aws-sdk-go from 1.44.331 to 1.44.332 (#3798) (6abd76e)
deps: bump github.com/aws/aws-sdk-go from 1.44.332 to 1.45.1 (#3818) (96588bb)
deps: bump github.com/google/uuid from 1.3.0 to 1.3.1 (#3775) (91889ad)
deps: bump github.com/hashicorp/golang-lru/v2 from 2.0.5 to 2.0.6 (#3789) (4e5b6a7)
deps: bump github.com/minio/minio-go/v7 from 7.0.61 to 7.0.62 (#3762) (417c981)
deps: bump github.com/minio/minio-go/v7 from 7.0.62 to 7.0.63 (#3802) (ae66f4f)
deps: bump github.com/onsi/ginkgo/v2 from 2.11.0 to 2.12.0 (#3783) (0e382b4)
deps: bump github.com/rudderlabs/rudder-go-kit from 0.15.5 to 0.15.6 (#3782) (f5a7e6e)
deps: bump github.com/rudderlabs/sql-tunnels from 0.1.3 to 0.1.4 (#3797) (1083cfe)
deps: bump github.com/snowflakedb/gosnowflake from 1.6.23 to 1.6.24 (#3776) (ca74e38)
deps: bump github.com/spf13/cast from 1.5.0 to 1.5.1 (#3770) (fd30711)
deps: bump google.golang.org/api from 0.136.0 to 0.137.0 (#3749) (3aff568)
deps: bump google.golang.org/api from 0.137.0 to 0.138.0 (#3760) (862a0b9)
gateway: authentication middlewares and context information (#3736) (264c52a)
introduce encoding factory (#3740) (188b95c)
jobsdb: unify get processed and unprocessed queries (#3742) (66c7e2e)
License update (#3821) (dd4dd3d)
option to pass *config.Config to jobsdb (#3764) (a04142e)
populate context with validation errors when context is nil (#3815) (dcb6a15)
readme updates (#3822) (4c258a3)
remove archival for jobsdb journal tables (#3758) (ff80a1d)
remove Init and exported struct variables from Archiver (#3735) (6c3fcca)
replay internal endpoint (#3746) (cd7557f)
revert warehouse tests race detection (#3825) (d08198e)
router_response_counts stat now shows if it's a retry attempt (#3799) (ec07777)
router: provide more debug info when detecting an illegal job sequence (#3787) (9f1c5fb)
slave cleanup and introduce tests (#3728) (6c3fcca)
update license to ELv2 (#3823) (4648ea3)
uploads cleanup (#3743) (ca52403)
warehouse tests race detection (#3773) (ce2ed33)

1.12.2 (2023-08-22)

Bug Fixes

extend sql middleware for mssql, azure_synapse and clickhouse (#3771) (e8809bb)

Miscellaneous

deltalake error message length (#3766) (fa09173)

1.12.1 (2023-08-16)

Miscellaneous

bingads safetypecast (#3750) (6204c3e)

1.12.0 (2023-08-14)

Features

bing ads bulk upload (#3371) (574d0a5)
parse replay configuration from backend config (#3703) (35f55e0)

Bug Fixes

add index on reported_at column of reports table (#3606) (4171517)
cleanup for warehouse integration tests (#3596) (4171517)
ignore workspace if no backup config is found (#3685) (dd3f42e)
illegal job sequence during barrier enter in router pickup (#3737) (5017146)
jobsdb: when running in embedded mode migration fails with column workspace_id does not exist (#3714) (817e97f)
minio manager race (#3672) (54e3055)
redshift dedup pick latest entry (#3603) (4171517)
send last error response for aborted jobs to reporting (#3692) (cfbeee9)
user transformations timeout (#3583) (4171517)
warehouse stringmap configs are lower case w.r.t viper (#3592) (4171517)

Miscellaneous

add custom buckets to gateway.user_suppression_age (#3679) (bef1280)
add max connections to warehouse and pgnotifier (#3597) (4171517)
add provision to disable tracking event names from a source for reporting (#3632) (4171517)
added trino and spark test (#3525) (4171517)
addressing review coments from #3602 (#3713) (141d109)
archiver cleanup (#3726) (e9b6a69)
cleanup integrations package (#3635) (4171517)
deps: bump cloud.google.com/go/bigquery from 1.52.0 to 1.53.0 (#3671) (8be90cb)
deps: bump cloud.google.com/go/pubsub from 1.32.0 to 1.33.0 (#3668) (c35d127)
deps: bump cloud.google.com/go/storage from 1.30.1 to 1.31.0 (#3625) (4171517)
deps: bump github.com/apache/pulsar-client-go from 0.10.0 to 0.11.0 (#3622) (4171517)
deps: bump github.com/aws/aws-sdk-go from 1.44.295 to 1.44.299 (#3615) (4171517)
deps: bump github.com/aws/aws-sdk-go from 1.44.299 to 1.44.300 (#3636) (4171517)
deps: bump github.com/aws/aws-sdk-go from 1.44.300 to 1.44.301 (#3640) (4171517)
deps: bump github.com/aws/aws-sdk-go from 1.44.301 to 1.44.302 (#3646) (4171517)
deps: bump github.com/aws/aws-sdk-go from 1.44.301 to 1.44.302 (#3646) (b05d2d6)
deps: bump github.com/aws/aws-sdk-go from 1.44.302 to 1.44.304 (#3650) (4171517)
deps: bump github.com/aws/aws-sdk-go from 1.44.302 to 1.44.304 (#3650) (edc9396)
deps: bump github.com/aws/aws-sdk-go from 1.44.304 to 1.44.305 (#3654) (4171517)
deps: bump github.com/aws/aws-sdk-go from 1.44.304 to 1.44.305 (#3654) (366393e)
deps: bump github.com/aws/aws-sdk-go from 1.44.305 to 1.44.306 (#3663) (f3cd10c)
deps: bump github.com/aws/aws-sdk-go from 1.44.306 to 1.44.307 (#3669) (e179725)
deps: bump github.com/aws/aws-sdk-go from 1.44.307 to 1.44.312 (#3686) (a4edf6c)
deps: bump github.com/aws/aws-sdk-go from 1.44.312 to 1.44.314 (#3693) (2bc503d)
deps: bump github.com/aws/aws-sdk-go from 1.44.314 to 1.44.315 (#3699) (2467eab)
deps: bump github.com/aws/aws-sdk-go from 1.44.315 to 1.44.317 (#3712) (de33fc3)
deps: bump github.com/aws/aws-sdk-go from 1.44.317 to 1.44.318 (#3718) (7719719)
deps: bump github.com/aws/aws-sdk-go from 1.44.318 to 1.44.319 (#3722) (06477fc)
deps: bump github.com/confluentinc/confluent-kafka-go/v2 from 2.1.1 to 2.2.0 (#3628) (4171517)
deps: bump github.com/databricks/databricks-sql-go from 1.3.1 to 1.4.0 (#3734) (c948122)
deps: bump github.com/dgraph-io/badger/v4 from 4.1.0 to 4.2.0 (#3711) (6a1c49b)
deps: bump github.com/go-chi/chi/v5 from 5.0.8 to 5.0.10 (#3637) (4171517)
deps: bump github.com/hashicorp/golang-lru/v2 from 2.0.4 to 2.0.5 (#3725) (5a36137)
deps: bump github.com/iancoleman/strcase from 0.2.0 to 0.3.0 (#3626) (4171517)
deps: bump github.com/minio/minio-go/v7 from 7.0.59 to 7.0.60 (#3621) (4171517)
deps: bump github.com/minio/minio-go/v7 from 7.0.60 to 7.0.61 (#3639) (4171517)
deps: bump github.com/onsi/gomega from 1.27.8 to 1.27.9 (#3664) (197ee3f)
deps: bump github.com/onsi/gomega from 1.27.9 to 1.27.10 (#3667) (1c3cd4f)
deps: bump github.com/rudderlabs/rudder-go-kit from 0.15.4 to 0.15.5 (#3724) (b199d1d)
deps: bump github.com/segmentio/kafka-go from 0.4.40 to 0.4.42 (#3620) (4171517)
deps: bump github.com/snowflakedb/gosnowflake from 1.6.22 to 1.6.23 (#3673) (a22ea8c)
deps: bump github.com/tidwall/gjson from 1.14.4 to 1.15.0 (#3681) (b25173e)
deps: bump github.com/tidwall/gjson from 1.15.0 to 1.16.0 (#3732) (48f1717)
deps: bump go.uber.org/atomic from 1.10.0 to 1.11.0 (#3587) (4171517)
deps: bump go.uber.org/automaxprocs from 1.5.2 to 1.5.3 (#3651) (4171517)
deps: bump go.uber.org/automaxprocs from 1.5.2 to 1.5.3 (#3651) (f9a19f3)
deps: bump golang.org/x/oauth2 from 0.10.0 to 0.11.0 (#3710) (92e1bf2)
deps: bump golang.org/x/oauth2 from 0.9.0 to 0.10.0 (#3585) (4171517)
deps: bump google.golang.org/api from 0.129.0 to 0.130.0 (#3588) (4171517)
deps: bump google.golang.org/api from 0.130.0 to 0.131.0 (#3627) (4171517)
deps: bump google.golang.org/api from 0.131.0 to 0.132.0 (#3647) (4171517)
deps: bump google.golang.org/api from 0.131.0 to 0.132.0 (#3647) (ffb993e)
deps: bump google.golang.org/api from 0.132.0 to 0.133.0 (#3670) (5c3a0d0)
deps: bump google.golang.org/api from 0.133.0 to 0.134.0 (#3677) (06b1e7b)
deps: bump google.golang.org/api from 0.134.0 to 0.135.0 (#3717) (bd329f8)
deps: bump google.golang.org/api from 0.135.0 to 0.136.0 (#3723) (e045ef3)
deps: bump google.golang.org/grpc from 1.56.1 to 1.56.2 (#3599) (4171517)
deps: bump google.golang.org/grpc from 1.56.2 to 1.57.0 (#3675) (277767b)
drop failing backups after few retries (#3695) (c81c66f)
gateway (#3708) (3cc17f3)
gateway backend config initialisation improvements (#3688) (3cca234)
increase transformer request timeouts (#3701) (998ce44)
introduce source category for deduplication (#3730) (00f994b)
jobsdb: increase maintenance operation timeouts and make them hot-reloadable (#3601) (1c94c53)
move warehouse handle to router (#3687) (e227046)
remove google cloud storage dependency for gcs datalake test using fake-gcs-server (#3576) (4171517)
remove notion artefacts (#3694) (fbd99c7)
remove unused ginkgo suites (#3662) (66a863a)
standardize warehouse timeout config name (#3553) (1cdc6d1)
warehouse backend config refactoring (#3602) (e48c98e)
warehouse sql stats (#3638) (4171517)
warehouse sql stats (#3638) (e20976d)

1.11.4 (2023-08-08)

Bug Fixes

jobsdb: when running in embedded mode migration fails with column workspace_id does not exist (#3714) (744cfd6)

1.11.3 (2023-07-28)

Miscellaneous

add custom buckets to gateway.user_suppression_age (#3679) (d1bc95c)

1.11.2 (2023-07-19)

Miscellaneous

fix stat table prefix (#3644) (204a577)

1.11.1 (2023-07-19)

Bug Fixes

destination: fix: add max-open connections & close deleted rows for reporting modules (#3643) (8e5a8c5)

Miscellaneous

fix negative pending event count (#3641) (74d98fd)

1.11.0 (2023-07-13)

Features

save webhook (/source transformation) failures to proc errors (#3522) (45a1802)
send stats for the time taken for suppression (#3542) (45955e2)

Bug Fixes

jobsdb panics during recovery after backup failure(s) (#3580) (abd9c8c)
transformation does not retry indefinitely when control plane is down (#3581) (1c6fc24)

Miscellaneous

deps: bump cloud.google.com/go/bigquery from 1.51.2 to 1.52.0 (#3556) (2936c04)
deps: bump cloud.google.com/go/pubsub from 1.31.0 to 1.32.0 (#3563) (496aca7)
deps: bump github.com/aws/aws-sdk-go from 1.44.282 to 1.44.283 (#3509) (a69c422)
deps: bump github.com/aws/aws-sdk-go from 1.44.283 to 1.44.284 (#3515) (a69c422)
deps: bump github.com/aws/aws-sdk-go from 1.44.284 to 1.44.285 (#3520) (a69c422)
deps: bump github.com/aws/aws-sdk-go from 1.44.285 to 1.44.287 (#3536) (a69c422)
deps: bump github.com/aws/aws-sdk-go from 1.44.287 to 1.44.288 (#3545) (486a7e5)
deps: bump github.com/aws/aws-sdk-go from 1.44.288 to 1.44.289 (#3550) (4d1e3e6)
deps: bump github.com/aws/aws-sdk-go from 1.44.289 to 1.44.290 (#3555) (0dbb328)
deps: bump github.com/aws/aws-sdk-go from 1.44.290 to 1.44.294 (#3575) (6cb7bf1)
deps: bump github.com/aws/aws-sdk-go from 1.44.294 to 1.44.295 (#3579) (5c442c0)
deps: bump github.com/databricks/databricks-sql-go from 1.3.0 to 1.3.1 (#3558) (4e661dd)
deps: bump github.com/hashicorp/golang-lru/v2 from 2.0.3 to 2.0.4 (#3538) (a69c422)
deps: bump github.com/minio/minio-go/v7 from 7.0.56 to 7.0.57 (#3511) (a69c422)
deps: bump github.com/minio/minio-go/v7 from 7.0.57 to 7.0.58 (#3546) (9b15bb9)
deps: bump github.com/minio/minio-go/v7 from 7.0.58 to 7.0.59 (#3569) (3bdd305)
deps: bump github.com/onsi/ginkgo/v2 from 2.10.0 to 2.11.0 (#3516) (a69c422)
deps: bump github.com/rudderlabs/compose-test from 0.1.1 to 0.1.2 (#3547) (55bde71)
deps: bump github.com/rudderlabs/rudder-go-kit from 0.15.0 to 0.15.1 (#3530) (a69c422)
deps: bump github.com/urfave/cli/v2 from 2.25.6 to 2.25.7 (#3514) (a69c422)
deps: bump google.golang.org/api from 0.127.0 to 0.128.0 (#3508) (a69c422)
deps: bump google.golang.org/api from 0.128.0 to 0.129.0 (#3564) (6cda94d)
deps: bump google.golang.org/grpc from 1.55.0 to 1.56.0 (#3510) (a69c422)
deps: bump google.golang.org/grpc from 1.56.0 to 1.56.1 (#3537) (a69c422)
deps: bump google.golang.org/protobuf from 1.30.0 to 1.31.0 (#3557) (6f42c97)
drop attempt_number tag from metrics (#3551) (af3fec1)
drop multitenant (573d6ff)
drop union query, fair pickup and multitenant handle (#3521) (573d6ff)
exclude attempt from metrics (#3549) (75d08dc)
gateway_response_time buckets (#3554) (bed100d)
gateway: move warehouse fetch tables under internal (#3540) (a69c422)
increase golangci lint execution timeout (#3539) (a69c422)
introduce goimports (#3507) (a69c422)
jobsdb: dsList lock performance improvements (#3559) (702ce95)
jobsdb: tuning and improvements (#3584) (cfa6132)
load file upload (#3552) (a8abd1d)
minor processor refactor for error handling and readability (#3534) (8f4635d)
minor refactor for readability (8f4635d)
move filemanager to rudder-go-kit (#3517) (9b82187)
remove unused code from warehouse (#3567) (a07cf9e)
retry on 408 response events (#3617) (6c44758)
router: penalise continuous non-productive pickup loops by sleeping (#3571) (b3050a7)
router: stop pickup iteration conditions (#3582) (9e7f117)
support replay with iam role (#3560) (f471fb0)
typo for workspaceId in stats (#3566) (8975dcc)

1.10.0 (2023-06-23)

Features

add config param for batch router for name customize (#3461) (c16e692)
error detail reporting (#3265) (34f4c0d)
router: support for isolation modes using limiters (#3379) (fbe109f)
warehouse: staging file schema consolidation (#3088) (b245915)

Bug Fixes

batchrouter: batchrouter stops processing events for destinations where a destType-specific config option is set (#3421) (ee87636)
clickhouse flaky test (#3392) (a89ef27)
error extraction for errors in destination transformation (#3499) (5c43457)
flag for warehouse altering (#3460) (c23ffb2)
gateway flaky test (#3356) (b245915)
gateway: use get method for /fetch-tables (#3528) (08cd99c)
gw transient errors crash (#3397) (6ad51e8)
jobsforwarder panics with slice bounds out of range (#3414) (0bda13f)
kafka flaky batching test (#3447) (fcd49e4)
migration fails with error pq: VACUUM cannot run inside a transaction block (#3464) (9e32802)
regulation-worker flaky test (#3374) (b245915)
respect user schema modification by user in warehouse (#3419) (fda4baf)
router panics with limit for rt_pickup must be greater than 0 (#3467) (cc56b56)
router: using wrong partition tag in limiter stats (#3518) (893504f)
stats duplicated labels (#3411) (0bda13f)
warehouse flaky test (#3402) (3f88f50)
warehouse proxy endpoints (#3476) (c23ffb2)

Miscellaneous

add missing index warehouse load files table (#3519) (9dcea7b)
add resp body status check (#3446) (e0d7b6d)
add support for detecting bot events in gateway (#3504) (51a4c45)
applying 1.9.3 hotfixes to main branch (#3423) (ee1630d)
avoid querying a dataset in case AfterJobID falls after said dataset (#3478) (f612aae)
batch event schema messages for faster processing (#3406) (0bda13f)
clean up warehouse indexes and queries (#3448) (7a3589a)
cleanup for integration tests for warehouse (#3412) (3323427)
deps: bump cloud.google.com/go/pubsub from 1.30.1 to 1.31.0 (#3399) (ef8e86c)
deps: bump github.com/aws/aws-sdk-go from 1.44.264 to 1.44.265 (#3361) (b245915)
deps: bump github.com/aws/aws-sdk-go from 1.44.265 to 1.44.266 (#3368) (b245915)
deps: bump github.com/aws/aws-sdk-go from 1.44.266 to 1.44.271 (#3409) (fc48d93)
deps: bump github.com/aws/aws-sdk-go from 1.44.271 to 1.44.275 (#3442) (8b179b6)
deps: bump github.com/aws/aws-sdk-go from 1.44.275 to 1.44.280 (#3481) (9c0a4e7)
deps: bump github.com/aws/aws-sdk-go from 1.44.280 to 1.44.281 (#3488) (af72e90)
deps: bump github.com/aws/aws-sdk-go from 1.44.281 to 1.44.282 (#3494) (aa6e194)
deps: bump github.com/databricks/databricks-sql-go from 1.2.0 to 1.3.0 (#3466) (3f187d9)
deps: bump github.com/golang-migrate/migrate/v4 from 4.15.2 to 4.16.1 (#3441) (ef16663)
deps: bump github.com/golang-migrate/migrate/v4 from 4.16.1 to 4.16.2 (#3480) (7465dd4)
deps: bump github.com/hashicorp/go-retryablehttp from 0.7.2 to 0.7.4 (#3457) (b3cd0f6)
deps: bump github.com/hashicorp/golang-lru/v2 from 2.0.2 to 2.0.3 (#3475) (74b4163)
deps: bump github.com/minio/minio-go/v7 from 7.0.52 to 7.0.53 (#3370) (b245915)
deps: bump github.com/minio/minio-go/v7 from 7.0.53 to 7.0.56 (#3437) (dbd9bf1)
deps: bump github.com/onsi/ginkgo/v2 from 2.9.5 to 2.9.7 (#3449) (1af6b90)
deps: bump github.com/onsi/ginkgo/v2 from 2.9.7 to 2.10.0 (#3458) (dacfdc8)
deps: bump github.com/onsi/gomega from 1.27.6 to 1.27.7 (#3360) (b245915)
deps: bump github.com/onsi/gomega from 1.27.7 to 1.27.8 (#3459) (d64c405)
deps: bump github.com/rudderlabs/rudder-go-kit from 0.14.3 to 0.15.0 (#3473) (6eea667)
deps: bump github.com/snowflakedb/gosnowflake from 1.6.20 to 1.6.21 (#3385) (4a06f44)
deps: bump github.com/snowflakedb/gosnowflake from 1.6.21 to 1.6.22 (#3503) (0f9c816)
deps: bump github.com/stretchr/testify from 1.8.2 to 1.8.3 (#3359) (b245915)
deps: bump github.com/stretchr/testify from 1.8.3 to 1.8.4 (#3450) (f5367ed)
deps: bump github.com/urfave/cli/v2 from 2.25.3 to 2.25.5 (#3418) (63a24eb)
deps: bump github.com/urfave/cli/v2 from 2.25.5 to 2.25.6 (#3489) (3a00ee6)
deps: bump golang.org/x/oauth2 from 0.8.0 to 0.9.0 (#3495) (5021989)
deps: bump golang.org/x/sync from 0.2.0 to 0.3.0 (#3502) (f3391d7)
deps: bump google.golang.org/api from 0.122.0 to 0.123.0 (#3362) (b245915)
deps: bump google.golang.org/api from 0.123.0 to 0.124.0 (#3386) (da6215c)
deps: bump google.golang.org/api from 0.124.0 to 0.125.0 (#3427) (a81b684)
deps: bump google.golang.org/api from 0.125.0 to 0.126.0 (#3472) (0f573e4)
deps: bump google.golang.org/api from 0.126.0 to 0.127.0 (#3487) (b525949)
drop merged schema column (#3493) (ba797a4)
fix flaky dedup test (#3417) (2f0a684)
fix idle query (#3430) (3cf342d)
gateway health endpoint should return 503 if db is down (#3351) (e22c790)
gateway stores singular event batches (#3256) (1ccec6e)
getUploadsToProcess error handling (#3380) (b245915)
jobsdb max age for jobs, cleanup routine (#3420) (5025a72)
kafka manager errors (#3465) (21487c2)
make tests required for passing (#3347) (b245915)
move suppression-backup-service from rudderlabs to rudderstack directory in dockerhub (#3505) (6c9b25b)
nil check for health handler (#3500) (195e2a0)
pass context (#3326) (990a405)
periodically push zero output measurement for detecting stuck processing pipelines (#3453) (4116f37)
processor: support multiple jobsdb writers when source isolation is enabled (#3428) (b25003d)
remove deprecated merged schema field (#3482) (39a0915)
remove namespace tag from measurements (#3468) (a6ac7bd)
replace announcement header with data learning centre link (#3358) (b245915)
revert drop merged schema column (#3501) (c8861ee)
router: observability on job iterator discards & stop iteration if throttled with destination isolation enabled #3491 (66e32ad)
router: periodic flush during pickup (#3497) (1193acc)
set limits for event schema messages and discard messages above these limits (#3435) (b70c075)
source transformation failures stat (#3524) (1146217)
source transformation failures stat tag correction (#3541) (b17dec9)
split unit tests (#3416) (471a562)
split unit tests (#3492) (1d0c720)
tests coverage (#3349) (b245915)
timeout for warehouse sql queries (#3433) (ca512a0)
upgrade golangci version and lint fixes (#3443) (3d03653)
upgrade to badgerV4 (#3340) (3ecea32)
using parquet-rudderlabs-go (#3490) (fbbf101)
vaccum status tables if they cross threshold (#3434) (8d35882)

1.9.6 (2023-06-09)

Bug Fixes

warehouse proxy endpoints (#3476) (fda977f)

1.9.5 (2023-06-07)

Bug Fixes

flag for warehouse altering (#3460) (472d310)

1.9.4 (2023-06-06)

Miscellaneous

add resp body status check (#3446) (b251b21)

1.9.3 (2023-05-31)

Bug Fixes

batchrouter: batchrouter stops processing events for destinations where a destType-specific config option is set #3421 (f2dc53c)

1.9.2 (2023-05-29)

Bug Fixes

jobsforwarder panics with slice bounds out of range (#3414) (99b7e62)

1.9.1 (2023-05-29)

Bug Fixes

stats duplicated labels (#3411) (0beb455)

Miscellaneous

batch event schema messages for faster processing (#3406) (e9f2174)

1.9.0 (2023-05-25)

Features

batchrouter: introduce isolation levels and concurrency limiters (#3248) (d90ea68)
support for incremental updates while fetching backend config (#3175) (9de904d)
wh Endpoint To Fetch Tables Per Connection (#3279) (ea7d5ce)

Bug Fixes

batchrouter acquiring a read lock twice (#3363) (6fd8552)
deltalake parquet datatype support (#3393) (0a6c85e)
external table support for databricks validations (#3378) (acf0c47)
include source definition type when destination has UT (#3338) (1ab2f55)
kafka secret comes first (#3307) (19ceebb)
limiter not respecting WithLimiterTags option (#3367) (9a3e6fc)
max connections for warehouse slaves (#3341) (1ab2f55)
processor panicking during shutdown (#3396) (4e3981f)
schema forwarder records invalid json in statuses (#3350) (91b1902)
staging files status when insert (#3332) (fb7277f)
stash sleep (#3312) (19ceebb)

Miscellaneous

adaptive processor worker sleep time (#3334) (4a4f293)
add toggle for backendconfig db caching (#3320) (0d198b8)
added logs to help debug suppression backup service issue (#3249) (a28984a)
cleanup fetch schema (#3242) (a28984a)
dedup based on message ID (#3289) (a28984a)
dedup based on message ID (#3289) (a2679d5)
deps: bump cloud.google.com/go/bigquery from 1.51.0 to 1.51.1 (#3303) (a28984a)
deps: bump cloud.google.com/go/bigquery from 1.51.1 to 1.51.2 (#3309) (465ad41)
deps: bump github.com/aws/aws-sdk-go from 1.44.256 to 1.44.259 (#3302) (a28984a)
deps: bump github.com/aws/aws-sdk-go from 1.44.256 to 1.44.259 (#3302) (2476215)
deps: bump github.com/aws/aws-sdk-go from 1.44.259 to 1.44.261 (#3316) (f4abcb1)
deps: bump github.com/aws/aws-sdk-go from 1.44.261 to 1.44.262 (#3322) (a2dd313)
deps: bump github.com/aws/aws-sdk-go from 1.44.262 to 1.44.264 (#3343) (de5c605)
deps: bump github.com/confluentinc/confluent-kafka-go/v2 from 2.1.0 to 2.1.1 (#3266) (a28984a)
deps: bump github.com/onsi/ginkgo/v2 from 2.9.4 to 2.9.5 (#3336) (8aefa7a)
deps: bump github.com/prometheus/common from 0.42.0 to 0.43.0 (#3293) (a28984a)
deps: bump github.com/rudderlabs/rudder-go-kit from 0.13.0 to 0.13.1 (#3284) (a28984a)
deps: bump github.com/rudderlabs/rudder-go-kit from 0.13.1 to 0.13.3 (#3296) (a28984a)
deps: bump github.com/segmentio/kafka-go from 0.4.39 to 0.4.40 (#3294) (a28984a)
deps: bump go.etcd.io/etcd/client/v3 from 3.5.8 to 3.5.9 (#3323) (c54f7d4)
deps: bump golang.org/x/oauth2 from 0.7.0 to 0.8.0 (#3300) (a28984a)
deps: bump golang.org/x/sync from 0.1.0 to 0.2.0 (#3301) (a28984a)
deps: bump golang.org/x/sync from 0.1.0 to 0.2.0 (#3301) (feb07aa)
deps: bump google.golang.org/api from 0.120.0 to 0.121.0 (#3286) (a28984a)
deps: bump google.golang.org/api from 0.121.0 to 0.122.0 (#3310) (d5a506a)
deps: bump google.golang.org/grpc from 1.54.0 to 1.55.0 (#3283) (a28984a)
fix flaky test (#3325) (22f2510)
go-kit v0.13.5 (#3375) (7352e64)
include error message in error log (#3348) (bb01437)
increase default jobsdb operation timeouts (#3172) (a28984a)
jobsdb: granular table count and cache hit stats (#3372) (a481131)
moved from gorilla to chi (#3263) (57231be)
rudder-go-kit v0.13.4 (#3365) (dfb8745)
upgraded chi v1 middleware to v5 (#3353) (a1b37d1)
warehouse integration test improvements (#3264) (4f8f54d)

1.8.5 (2023-05-17)

Bug Fixes

max connections for warehouse slaves (#3341) (b355eb2)

1.8.4 (2023-05-17)

Bug Fixes

include source definition type when destination has UT (#3338) (3343017)

1.8.3 (2023-05-12)

Miscellaneous

add toggle for backendconfig db caching (#3320) (e75fea0)

1.8.2 (2023-05-10)

Bug Fixes

stash loop not sleeping when no jobs are present (#3312) (b446ae7)

1.8.1 (2023-05-10)

Bug Fixes

kafka secret comes first (#3307) (757f608)

1.8.0 (2023-05-09)

Features

databricks compatible go driver (#3148) (7660520)
destination: support event filtering in a scalable way for hybrid & cloud modes (#3188) (2c2c0e1)
embed avro schema id (#3118) (8c98631)
kafka compression (#3179) (acb3918)
modifed kafka to use config received from config-be. (#3205) (0d1c75f)
processor: enhance reports to hold transformation and tracking plan metrics (#3138) (865ad6c)
processor: forward events to new events-schema jobsDB (#3201) (5a69ce9)
router: limit number of jobs with the same ordering key in worker's queue (#3243) (ed5722d)
rudder-server modification to use suppression-backup service. (#3116) (daf3e26)
schema forwarder (#3146) (9accebf)
source level isolation at processor (#3187) (a3f687e)
suppression backup service (#2910) (5b68307)
update transformer image to rudderstack org (#3115) (8f74bbb)
warehouse: middleware for sql warehouse integrations queries (#3190) (85f064a)
warehouse: warehouse schema repository (#3087) (7becd87)

Bug Fixes

add externalID to S3 file uploader config (#3153) (d7ac3a8)
add mandatory env for shopify oss (#3227) (460890d)
add numeric scale precision to snowflake (#3239) (ef127ef)
controller conn (#3142) (f65a67a)
databricks retries (#3305) (6cacb2e)
error messages map !initialized in gateway (#3191) (03cd911)
identity tables load files are not generated (0aab933)
illegal job order with maxRetriesLimitReached (#3163) (f328e43)
kafka with CP admin client (#3297) (c9fcb8c)
no need for setup during warehouse crash recovery (#3203) (1f7d542)
postgres support for loading users and identifies in same transactions (#3237) (2c3697a)
redshift staging table creation (#3180) (8ec737d)
remove logging in messages potentially leaking creds (#3195) (04a31a0)
roleBasedAuth parameter wasn't being set correctly (#3134) (c2f6dd8)
stash loop is not sleeping even though query limits are not being reached (#3291) (5a0947e)
stats come first (#3159) (c7210fa)
update lo to slices in eventfilter package (#3238) (0a91574)
warehouse: all staging files processing failed (#3137) (9dc4b8f)
warehouse: table uploads were not getting updated (6dec777)

Miscellaneous

access denied for s3 datalake (#3186) (29c818a)
add metrics for file sizes for badgerDB (#3092) (f659b49)
add sql middleware to for warehouse (#3212) (b8bc531)
added postgres legacy and new implementation tests (#3216) (e22428c)
added threshold for commit for sqlwrapper (#3273) (a4f3b56)
added transactions to sqlquerywrapper (#3231) (bba4894)
adding BeginTx to sqlwrapper (#3260) (d4b30d6)
addressing snyk vulnerabilities (#3224) (9f5dc99)
bigquery query rate limited (#3185) (60437df)
bq column count error mappings (#3184) (0b47f65)
bump github.com/rudderlabs/sql-tunnels from 0.1.2 to 0.1.3 (#3225) (698fe5e)
bump kafka-go and gosnowflake (#3217) (1b4698e)
change stash defaults (#3136) (e2f019d)
cleanup of unnecessary connection-tester (#3208) (4204f7a)
cleanup warehouse handlers (#3211) (f65f0e7)
deltalake parquet support (#3135) (f54b9fd)
deps: bump cloud.google.com/go/bigquery from 1.46.0 to 1.50.0 (#3171) (07cde26)
deps: bump cloud.google.com/go/pubsub from 1.30.0 to 1.30.1 (#3277) (c9d12c1)
deps: bump github.com/aws/aws-sdk-go from 1.44.245 to 1.44.246 (#3229) (cae0093)
deps: bump github.com/aws/aws-sdk-go from 1.44.246 to 1.44.252 (#3254) (42ee1b5)
deps: bump github.com/aws/aws-sdk-go from 1.44.252 to 1.44.253 (#3258) (be1fef9)
deps: bump github.com/aws/aws-sdk-go from 1.44.253 to 1.44.254 (#3261) (0f87463)
deps: bump github.com/aws/aws-sdk-go from 1.44.254 to 1.44.256 (#3276) (804ec2c)
deps: bump github.com/databricks/databricks-sql-go from 1.1.0 to 1.2.0 (#3234) (7b7cfe7)
deps: bump github.com/docker/docker from 20.10.21+incompatible to 23.0.3+incompatible (#3167) (0600834)
deps: bump github.com/lib/pq from 1.10.8 to 1.10.9 (#3247) (15baff9)
deps: bump github.com/onsi/ginkgo/v2 from 2.9.2 to 2.9.4 (#3278) (114f710)
deps: bump github.com/onsi/gomega from 1.27.1 to 1.27.6 (#3213) (48e6913)
deps: bump github.com/opencontainers/runc from 1.1.4 to 1.1.5 (#3140) (e3bd948)
deps: bump github.com/rudderlabs/sql-tunnels from 0.1.2 to 0.1.3 (698fe5e)
deps: bump github.com/samber/lo from 1.37.0 to 1.38.1 (#3194) (d66b63f)
deps: bump github.com/shirou/gopsutil/v3 from 3.23.1 to 3.23.3 (#3170) (e9bb277)
deps: bump github.com/shirou/gopsutil/v3 from 3.23.3 to 3.23.4 (#3259) (4eb3a64)
deps: bump github.com/stretchr/testify from 1.8.1 to 1.8.2 (#3082) (109b652)
deps: bump github.com/urfave/cli/v2 from 2.25.1 to 2.25.3 (#3262) (b1bab9a)
deps: bump go.uber.org/automaxprocs from 1.5.1 to 1.5.2 (#3193) (833a4f7)
deps: bump golang.org/x/oauth2 from 0.6.0 to 0.7.0 (#3196) (10b62ea)
deps: bump google.golang.org/api from 0.110.0 to 0.116.0 (#3177) (4227912)
deps: bump google.golang.org/api from 0.118.0 to 0.120.0 (#3246) (a05ae14)
disable KeepAlives by default in transformer HTTP clients (#3147) (6c308c4)
enable backoff when event ordering is disabled (#3121) (bbc14c1)
enable setting of embedAvroSchemaID from config (#3282) (e2d729d)
error enrichment for redshift (#3182) (3e8a962)
fix deadlock issue uploader (#3272) (71b5ac8)
fix typecast panic (#3269) (e0280e4)
improve pending events api (#3253) (3704619)
initiate db handle warehouse slave (#3270) (bb897b4)
kafka batching (#3151) (cdbbc7f)
kit v0.12.0 (#3209) (1475ece)
lock bitnami test image versions (#3232) (7087f63)
otel buckets (#3120) (803ac03)
refactor regulation worker integration test (#3230) (381ca72)
remove global pkglogger from warehouse integrations (#3202) (1211e51)
remove jobsdb status handler (#3192) (73c295a)
remove readonlyjobsdb and some status handlers (#3168) (7d82354)
replace readme header link with slack link (#3183) (db9fdc5)
restore sample configuration files (#3122) (0ac3a6e)
router: don't include job in metadata during transformation requests (#3111) (d2910d2)
router: skip full workers during the generator loop (#3206) (bed02f7)
start server in normal mode when there is no instruction by scheduler (#3103) (bad6a82)
stop warehouse panics (#3105) (7becd87)
updating error mappings (#3150) (0851cb9)
upgrade direct dependencies (#3223) (309727d)
upgrading workflow to use action v3. (#3164) (f7cf6c9)
use rudder-go-kit (#3104) (7becd87)
use the common PAT for release-please (#3204) (14e3b91)
warehouse: cleanup for test connection (#3226) (51c1ac0)
warehouse: snowflake application identifier (#3124) (cdff792)

1.7.3 (2023-04-04)

Bug Fixes

add externalID to S3 file uploader config (#3153) (355dda2)

1.7.2 (2023-03-30)

Bug Fixes

identity tables load files are not generated (be569c1)

1.7.1 (2023-03-24)

Bug Fixes

warehouse: table uploads were not getting updated (abb8bfb)

1.7.0 (2023-03-17)

Features

default retention period set to 7 days for rudder backups (#3038) (0d9af35)
gateway: support new event type extract (#2999) (63dc940)
kafka over ssh (#3007) (99262c3)
warehouse: added duplicates stats for snowflake (#3097) (0eeaeb5)
warehouse: snowflake roles support. (#3031) (98a413c)
warehouse: temp table support for postgres (#2964) (9a80f45)

Bug Fixes

enabled data race (#3041) (7136be4)
inconsistent and leaky retry delay logic in router (#3002) (20c8644)
kafka create topic (#3102) (10ccbf3)
limiter deadlock while trying to notify a dynamic priority item (#3056) (ade2e4d)
minio error while creating bucket if already exists (#3109) (2abecaa)
payload limiter gauge reports invalid value type (#3048) (828f70d)
pending events all (#3075) (1d9f63b)
router: retry backoff causing out-of-order job processing (#3098) (eda4525)
user/anonymous id read at gateway (#3051) (828f70d)
warehouse: added support for filtering on the uploads and calculating aborted events for task_run_id (#2975) (8ab58b8)
warehouse: grouping uploads for processing pickup (#3039) (4832630)
warehouse: infinites retries (#3050) (d3320fa)
warehouse: snowflake default timestamp to timestamp with time zone (#3100) (cef4a18)
warehouse: snowflakes user identifies table getting skipped (#3113) (dd626b3)

Miscellaneous

abort job after 5 attempts at deletion-worker (#3012) (ed83eaa)
add live events stats (#2930) (7ca1b0b)
add transformations challenge announcement (#3072) (f128b93)
badgerDB improvements for debugger (#3101) (0c525d0)
debugger's badgerdb cache optimisations (#3042) (e6e9933)
deps: bump github.com/minio/minio-go/v7 from 7.0.48 to 7.0.49 (#3018) (f4ea7b3)
deps: bump github.com/onsi/ginkgo/v2 from 2.1.6 to 2.9.0 (#3068) (6bdbb7a)
deps: bump github.com/prometheus/common from 0.37.0 to 0.41.0 (#3062) (bb04a8b)
deps: bump github.com/urfave/cli/v2 from 2.20.3 to 2.25.0 (#3067) (6b429b7)
deps: bump go.uber.org/goleak from 1.2.0 to 1.2.1 (#3017) (2eb92ca)
don't include prereleases in docker latest tag (#3006) (20c8644)
enable pipeline level sync frequency (#3094) (ea3bbd5)
fix namespace bug (#3110) (7b6fa35)
improve regulation-worker status capture (#2837) (6f1d07d)
increase parallel loads (#3073) (7dcc756)
jobsdb: omit workspaceId tag when it doesn't correspond to an actual workspace (#3057) (f936260)
migrate stats to otel (#2989) (a4243de)
perform rss/wss-aware cgroups memory usage calculation (#3052) (1b6af85)
prefer using lo.BufferWithTimeout (#2998) (20c8644)
reduce parquet file size datalake (#3035) (4cb5907)
remove workerID tag (#3055) (b732919)
upgrade all dependencies (#2996) (20c8644)
use gcra rate limiter gateway (#3086) (61d9275)
use official bitnami images for arm64 (#3047) (3aeb4f6)
use token on protoc setup to avoid rate-limit (#3083) (0f89b26)
use upload_id for staging files (#3066) (3ec2433)
warehouse: added support for observability for loading tables and made dedup optional for Redshift (#3037) (63fd288)
warehouse: added warehouse handling for s3 with glue and other improvements (#2940) (3495797)
warehouse: additional error mappings (#2994) (20c8644)
warehouse: allow empty properties schema for event models (#3095) (b9deb4a)
warehouse: default warehouse priority set to 100 (#3026) (20c8644)
warehouse: encoding package with readers, loaders, writers (#3077) (06c0a71)
warehouse: naming conventions (#3076) (f2e99c7)
warehouse: use first_event_at while pickup for warehouse processing jobs (#3036) (7aeed3b)
warehouse: warehouse integration tests improvements (#3059) (e57362e)

1.6.3 (2023-03-01)

Bug Fixes

limiter deadlock while trying to notify a dynamic priority item (#3056) (5f967dc)

1.6.2 (2023-02-28)

Bug Fixes

payload limiter gauge reports invalid value type (#3048) (5605abc)
user/anonymous id read at gateway (#3051) (b87cc25)

1.6.1 (2023-02-28)

Miscellaneous

debugger's badgerdb cache optimisations (#3042) (34602c3)

1.6.0 (2023-02-23)

Features

adaptive payload size limits (#2949) (fbfd845)
add webhook source error logging (#2958) (042d9b3)
filter events based on destination consent management rules (#2973) (47a12bd)
introduce upload id to associate staging files to uploads (#2908) (d127a82)
processor: add support for workspace-level isolation (#2891) (e8dfff3)
warehouse: added use rudder storage option to deltalake (#2929) (6885ba1)
warehouse: clickhouse s3 table engine for load tables (#2739) (074e789)
warehouse: databricks unity catalog (#2878) (074e789)
warehouse: error tagging (#2956) (3131b96)
warehouse: glue partitions (#2899) (9a928d4)

Bug Fixes

adding throttling stats in router (#2923) (61109e2)
adding throttling stats in router (#2923) (5a36692)
create index concurrently cannot run inside a transaction block (#3008) (a9f0192)
handle NULL on upload timings (#2967) (58381db)
job ordering inconsistencies with router destination isolation (#3030) (a77c382)
limit entire transformationStatus struct when caching them (#2928) (2283aef)
move recovery from scheduler to upload (#2959) (df7a491)
processor read of dest consent categories (#3028) (c83582c)
router: wrong job order keys used in batching job order logic (#3019) (e9314cf)
suppression syncer using wrong credentials in multi-tenant mode (#2936) (0eb3abb)
upload missing metric (#2947) (b1fc5ee)
warehouse: added check for nil warehouse manager during error mapping (#2981) (a258f74)
warehouse: consolidaiton for merged schema to give preference for text datatype (#2983) (b262f1d)
warehouse: deltalake client crashing when failing on connectionstep (#2961) (3373144)
warehouse: fix db migration to add upload_id to wh_staging_files… (#2948) (7fa1c9e)
warehouse: fix upload error and add additional logs (#2972) (d1d2338)
warehouse: redshift dedup window (#3013) (1dbe505)
warehouse: skipping deprecated columns (#3000) (82a35d6)
warehouse: use single protocol source category while doing dedup for new record (#2937) (c46ba75)
warehouse: warehouse successful upload exists (#2931) (c46ba75)
warehouse: warehouse validations fixed to lookup secrets using sshKeyId (#2950) (017d8ba)

Miscellaneous

add workspace id in wh_async_jobs table (#2848) (074e789)
capture cgroup-aware memory stats (#2945) (4e6cf3b)
decouple loadfile logic and repo (#2753) (074e789)
dedup destination metadata in router based on job id (#2982) (dfc2273)
deps: bump github.com/gofrs/uuid from 4.2.0+incompatible to 4.4.0+incompatible (#2906) (e8ecf32)
deps: bump github.com/samber/lo from 1.35.0 to 1.37.0 (#2913) (6718048)
drop column only if exists (#2979) (03e7390)
enhance data sent to reporting (#2914) (c58100c)
introduce stat for skip upload syncs (#2938) (b1fc5ee)
jobsdb: support query filtering by workspaceID (#2911) (d049f2b)
make GC discard ratio configurable (#3023) (0801818)
processor: stop using global variables (#2881) (074e789)
protect measurement & tag names from empty strings (#2946) (f20a6f3)
refactored gateway marshalling (#2915) (2625f1a)
remove gorate throttling algorithm (#2918) (772187a)
remove reports down migrations (#2920) (2fa72d6)
replace readme v1 announcement header (#2893) (074e789)
staging files to uploads (#2863) (692f55c)
tunable badgerDB config params (#3027) (0742fd7)
use a partition lock when updating badger live events cache (#2902) (a4d1a71)
use badgerdb for live events cache (#2833) (074e789)
use gauge for capturing pending events (#2960) (169ab96)
warehouse: alter handling for redshift (#2888) (e1918c1)
warehouse: filemanager nil fix for validations (#2986) (306d55b)
warehouse: log improvements for total count in warehouse integrations (#2976) (994c89b)
warehouse: move warehouse destinations to integrations package (#2885) (074e789)
warehouse: remove unused stats from deltalake integration (#2977) (d8639ee)
warehouse: use warn while getting total count in warehouse (#2944) (4f225f5)
warmup gcra algorithms in throttling test (#2909) (2754027)

1.5.5 (2023-02-07)

Bug Fixes

upload missing metric (#2947) (cbb4b1a)

1.5.4 (2023-02-06)

Miscellaneous

introduce stat for skip upload syncs (#2938) (9ac5006)

1.5.3 (2023-02-03)

Bug Fixes

warehouse: use single protocol source category while doing dedup for new record (#2937) (8087e46)
warehouse: warehouse successful upload exists (#2931) (bcb2bb2)

1.5.2 (2023-02-02)

Bug Fixes

runtime error: slice bounds out of range [:-1] (#2932) (07dd59e)

1.5.1 (2023-02-01)

Bug Fixes

adding throttling stats in router (#2923) (6de8862)

1.5.0 (2023-01-24)

Features

add total_bytes field in staging file model (#2853) (cc74fcb)
added MT support for regulation worker (#2831) (f1d3d03)
changes to make rudder-scheduler work with HA gateway (#2823) (04a8559)
controlplane client support for destination history (#2747) (736a70b)
distributed throttling (#2606) (736a70b)
warehouse: added support for workspaceID's configuration (#2760) (736a70b)
warehouse: implement ssh tunnelling (#2804) (1cbb1e7)
warehouse: upload job processing metrics (#2780) (736a70b)

Bug Fixes

degraded workspace ids should be excluded by upload job allocator (#2773) (736a70b)
destination: add proper stats for rt/batch transformation & proxy (#2855) (b7aa800)
failing executing jobs(except gateway) instead of deleting them (#2736) (736a70b)
processor: wrong event filter in count stat captured (#2889) (a8a92c6)
regulation-worker panics during startup (#2877) (9ad84be)
shallow copy config in processor (#2856) (b7aa800)
stash reporting back less error jobs after store (#2882) (5b8f5c0)
stats.Timer#End panics with nil pointer (#2866) (17c078b)
unnecessary router pending event counts (#2849) (6c5d928)
warehouse: added support for testing ssh tunnelling using postgres (#2836) (333310d)
warehouse: control plane client needs to be initialized for master mode (#2826) (5ad7ee7)
warehouse: grcp cp-router logs (#2766) (736a70b)
warehouse: include exclusion window in status tracker (#2812) (0459aff)
warehouse: processing available workers to be gauge stats (#2822) (dfc4b23)

Miscellaneous

abort jobs in regulation worker for unsupported destinations (#2821) (791c235)
add logs when backup is skipped due to preferences (#2867) (1d9320d)
add tags(sourceType, sdkVersion) to gateway stats (#2896) (078ee91)
added prefix to regulation worker stats (#2879) (bab5a85)
applying 1.4.3 and 1.4.4 hotfixes to main branch (#2794) (736a70b)
close http responses in a way to allow the Transport to re-use the TCP connection (#2718) (736a70b)
completely replace gofrs with google uuid (#2733) (736a70b)
deps: bump cloud.google.com/go/pubsub from 1.19.0 to 1.27.0 (#2755) (736a70b)
disable batch destinations by default in regulation worker (#2886) (2767dfb)
docs update - unlimited event copy (#2813) (7f6c834)
drop support for v1 pending events (#2820) (eadf1da)
gateway stats (#2758) (65774e2)
handling wrapped temporary errors for Kafka destinations (#2898) (0efa116)
introduce region in reporting url (#2832) (87a967e)
limit live events stored in memory (#2803) (3257aac)
prohibit deletion of incomplete job-status stats (#2850) (ea4829a)
remove deepsource badge (#2765) (736a70b)
timerStat.RecordDuration, Since instead of timerStat.Start, End (#2870) (134bb0a)
upgrade postgres image (#2764) (736a70b)
use destination config from config-be than from message (#2830) (28f8750)
using an exponential backoff (#2865) (56f84df)
warehouse: integration test improvements (#2685) (736a70b)
warehouse: use strings.builder for warehouse add columns (#2801) (736a70b)

1.4.6 (2023-01-10)

Bug Fixes

destination: add proper stats for rt/batch transformation & proxy (#2855) (6127684)
shallow copy config in processor (#2856) (ea3bfdc)

1.4.5 (2022-12-15)

Bug Fixes

aws session creation is failing for s3 manager when roles are used (#2799) (1534d64)
destination: alert flag setting for router-aborted-count alert definition (#2808) (f2e9001)
pending events stats incremented twice during startup due to tenantstats being initialized twice (#2798) (49aa37f)

Miscellaneous

enable failed keys by default (#2805) (ba9832c)
gateway: add webhook failure stats (#2811) (60fc5f7)
gateway: enabling source transformation alerts (#2809) (cc14e6c)
quote table identifier (#2810) (58f2e4c)

1.4.4 (2022-12-09)

Bug Fixes

using a wrong datatype for pg_class.reltuples causes internal migration to fail (#2792) (006a203)

1.4.3 (2022-12-08)

Bug Fixes

remove schema from stage file model (#2790) (2bc1f1d)

1.4.2 (2022-12-07)

Miscellaneous

remove shopify autoreload (#2784) (85ba13a)

1.4.1 (2022-12-07)

Miscellaneous

fail transformer timeouts (#2779) (05d8264)
remove hacky commit hash (cf0cfad)

1.4.0 (2022-12-05)

Features

add support for request payloads using gzip content-encoding (#2741) (d9b9084)
regulation api support for Universal Analytics (#2632) (87a5d02)
support using badgerDB as a repository for data regulation suppressions (#2619) (712b6a5)
warehouse: added support for bigquery custom partition for workspaceIDs (#2679) (3bb21c9)
warehouse: added support for warehouse column count limit (#2723) (bed58c5)
warehouse: degraded workspace id (#2627) (3bb21c9)

Bug Fixes

aws role session name (#2712) (87f57f8)
batch router event delivery time (#2711) (3376dc9)
capture router's iterator stats after iteration is complete (#2698) (5d466d3)
destination: delete users support add test-cases and better error strings (#2744) (b69c4bb)
goroutine leak in lock.AsyncLockWithCtx (#2722) (b29da14)
incorrect stats captured at gateway (#2710) (a6c1a16)
init jobsdb logger at the right place (#2729) (ef64bba)
jobsdb schema migration not performed against all datasets (#2737) (4d7fd5a)
metadata in event_schema table has TotalCount which exceeds 32bit integer limits (#2738) (0cf5fd6)
no timezone while parsing (#2671) (9f006e5)
regulation-worker changes for oauth destinations (#2730) (0ed5a82)
removed quote Identifier on sourceDestId (#2763) (2cc87f4)
use local variable for workspaceId in stash backup (#2706) (b3a5fc1)
warehouse sync job queries for snowflake and bigquery (#2740) (d33d8c3)
warehouse: deepsource fix for use of empty error string in errors.New (#2731) (867ce5b)
warehouse: increase async job timeout (#2721) (88f1ec1)
warehouse: syncs issues (#2732) (0941cc0)
warehouse: use correct config for columns batch size (#2720) (0e26b30)
warehouse: warehouse archiver initialize (#2697) (c7af5d9)

Documentation

create SECURITY.md (#2656) (3bb21c9)

Miscellaneous

add workspaceid label to router aborted events (#2724) (8a04871)
advertise server features in version endpoint (#2751) (73fc7ea)
by default enable max concurrent gw request limit. (#2648) (3bb21c9)
compact status tables before proceeding with migration (#2688) (27299f2)
data residency adaptations (#2703) (52d9392)
deps: bump github.com/samber/lo from 1.33.0 to 1.35.0 (#2707) (a01b515)
deps: bump google.golang.org/grpc from 1.50.0 to 1.51.0 (#2717) (29f2de4)
don't use release-please in main branch (bc0d1c7)
enhance transformer error with more metadata (#2742) (e9b75e9)
ensure internal migration of large job-status datasets (#2748) (a0fc4f5)
fixing a flaky rsources test (#2680) (ad2626f)
include missed changelogs due to rebase (3bb21c9)
introduce api & repo components (#2691) (c26844e)
introduce in-memory stats for testing (#2735) (6ac8c31)
jobsdb: add support for schema migration changesets to run always (#2746) (463533e)
jobsdb: latest job status query optimization (#2693) (4e0008c)
regression while starting a standalone warehouse slave (#2743) (4e8df6f)
regulation worker avoid panic in case of timeout (#2657) (3bb21c9)
release 1.3.1 (#2708) (c96169d)
release 1.3.2 (#2713) (bca84b7)
release 1.3.3 (#2726) (bc6afca)
remove redundant/duplicate add_ds distributed lock (#2769) (808f02d)
safe database initialisation and schema migration during startup (#2734) (cf80d67)
set max connections to readonly jobsdbs (#2745) (5e8f73d)
update config backend url domain name (#2699) (9e06351)
use azure-sql image with m1 compatible healthcheck (#2702) (4f0d189)
use bugsnag in router and processor goroutines (#2686) (3bb21c9)
using arm64 compatible images if necessary (#2670) (3bb21c9)
warehouse: handle schema change (#2654) (d01f98b)
warehouse: use fastUUID with google UUID generation (#2598) (07093b1)

1.3.3 (2022-11-23)

Bug Fixes

goroutine leak in lock.AsyncLockWithCtx (#2722) (7597ecf)

1.3.2 (2022-11-17)

Bug Fixes

batch router event delivery time (#2711) (3782597)
incorrect stats captured at gateway (#2710) (52e8fa0)

1.3.1 (2022-11-16)

Bug Fixes

use local variable for workspaceId in stash backup (#2706) (f0d3612)

1.3.0 (2022-11-10)

Features

adding metadata in the proxy request (#2622) (f6f91ea)
destination: bqstream - add support for batch of properties (#2367) (73baf76)
router: avoid worker starvation during job pickup (fee04b7)
router: avoid worker starvation during job pickup (#2379) (0ec74d1)
support for config to custom destinations (#2625) (cb230b0)
support initialisation of producer without topic for kafka, azure event hub and confluent cloud (#2569) (0312c55)
support reporting pii filter based on config backend (#2655) (ef95eba)
user suppressions adaptations for namespaces (#2604) (5c26d1b)
warehouse: added base support for s3 datalake deletion as part of regulation API. (#2515) (4369abf)
warehouse: batching of alter add statements (#2484) (37d32f1)
warehouse: unrecognized schema in warehouse (#2638) (366c1f5)

Bug Fixes

always use a sql safe table name in failed events manager (#2664) (0d061ff)
batchrouter: concurrent modification of job parameters causes panic (#2631) (79e3e34)
changed query to accept user input in prepared sql statement (#2652) (2f956b7)
close init after assigning storage settings (#2678) (4986595)
destination: empty oauth account check (#2675) (1584352)
don't ignore stats middleware template path calculation (#2594) (f589f5f)
gcs manager cancels context (#2662) (0964e83)
golangci-lint issues (#2641) (c9bd5d4)
jobsDB backup query time (#2521) (46b5b3f)
jobsdb: update cache after transaction completes (#2567) (2d70da7)
only close response body in oauth handler in case of no error (#2593) (83ace48)
page size fixed irrespectiveof suppressAPIToken (#2611) (71d8c99)
preferences should be false incase of invalid bucket (#2661) (725e9e7)
properly escape table name when querying for failed events (#2663) (9c009d9)
remove gateway db write panic (#2644) (60bc174)
removing the backup bucket env check to determine if backup is enabled (#2668) (ea8cf4c)
reset job status ids during internal migration (#2684) (3cffe02)
router destinationsMap access (#2582) (3770720)
send router transform failures to live events (#2637) (6931b17)
take workpaceid from config backend key (#2683) (931cb5c)
use enterprise token from env as fallback (#2667) (56f0f23)
use proper status code to handle warehouse process (#2659) (a53657d)
use sum64 to avoid checkptr race bug (#2645) (126cbdf)
warehouse: id resolution index issue (#2676) (8ddfb8e)
warehouse: parquet oom and ftr changes (#2599) (b820a55)
warehouse: read unlock in historic identities (#2592) (09dba21)

Miscellaneous

add github action for prerelease workflow (#2660) (43dde0c)
add source to event_delivery metric (#2642) (fcf9e5e)
added status table cleanup logic before backup with code refactor (#2542) (9196f8f)
adding log entry to multi tenant test (#2565) (8444433)
address snyk vulnerabilities (#2579) (d1b1988)
applying 1.2.2 hotfixes to main branch (#2585) (71c8a15)
applying 1.2.4 hotfixes to main branch (#2597) (0e1da7e)
backup support for new multitenant system (#2549) (4741989)
clean up unused variables (#2647) (5541e7e)
cleanup unused code (#2561) (4de6105)
cleanup unused/obsolete code (#2612) (f13ea33)
fix defect with router destinations map access event order (#2589) (a9d515e)
fix replay bugs (#2653) (8ac05c2)
improve grpc tooling and upgrade setup-go to v3 (#2553) (ab20ad6)
internal migration in postgres (#2559) (3cb2ec6)
jobsdb: use a different advisory lock for different table prefixes (#2575) (26b218c)
modified statsMiddleware to be generic (#2626) (b6ce9b9)
release 1.2.1 (#2573) (eb705df)
release 1.2.5 (#2602) (854e359)
remove leftover flag.Parse() (#2643) (3659e12)
rsources flaky test - start services before all (#2650) (1c49e96)
upgrade GO version to 1.19 (#2635) (768be08)
use gotestsum for test execution (#2586) (a194b0c)
use WriteTimeout instead of Gateway.pendingEventsQueryTimeout (#2649) (e9aff77)
using error from router http client (#2628) (a6e253b)
warehouse: add user transformer url env (#2651) (00ba231)
warehouse: fix log level of async jobs (#2617) (078bf76)
warehouse: verify workspace ID in stats (#2615) (dceaf29)
warehouse: warehouse formatting changes (#2568) (888f6f8)

1.2.5 (2022-10-20)

Bug Fixes

warehouse: parquet oom and ftr changes (#2599) (ff07f37)

1.2.4 (2022-10-19)

Bug Fixes

warehouse: default job type (#2595) (fd2652a)

1.2.3 (2022-10-19)

Bug Fixes

warehouse: remove bad unlock (#2590) (aba7893)

1.2.2 (2022-10-19)

Bug Fixes

jobsdb: race - repeat job count calculation after acquiring migration lock (#2583) (87f9768)

1.2.1 (2022-10-14)

Bug Fixes

jobsdb: update cache after transaction completes (#2567) (6c0db53)

1.2.0 (2022-10-12)

Features

add support for role based auth flag (#2519) (810bb5d)
add support iam role support for warehouse destinations (#2496) (867123a)
addition of Azure SAS Tokens for Azure Object storage destinations (#2358) (cf5afe1)
changes to support variadic keys for deletion from API dest. (#2457) (1950713)
core: logical replication monitoring query routine (#2436) (8271aab)
destination: add new tags to router-aborted-count alert definition (#2514) (60bc1be)
introduce features/settings client (#2377) (711f266)
updating supportedDestinations in api.go (#2468) (3a2e75c)
warehouse_jobs support for async warehouse jobs (#2319) (73cc48a)
warehouse: added warehouse features (#2545) (8ea8be5)
warehouse: introduce workspace id in warehouse (#2523) (2c5fa3d)
warehouse: skipping scanning the entire table for duplicates in redshift (#2518) (104410a)
warehouse: timeout during warehouse table count (#2543) (b2b1c08)

Bug Fixes

core: cache backend config and block processor (#2225) (40fe19d)
json out of bounds in batchrouter (#2465) (2bf0f6a)
only report features on master / standalone warehouse (#2483) (33ba5ef)
remove custom credentials chain to use default (#2485) (de0fe36)
router: preserve event order while draining a previously failed job (#2546) (f0654b0)
silent backup ds drop (#2550) (4fe3038)
to prevent migration of tables every loop after maxdsretention is passed (#2554) (af27add)
use workspace id as aws iam role external id (#2386) (bced8d5)
warehouse: omit view tables while fetching schema (#2482) (74f268c)
warehouse: typo with fetch schema for bigquery (#2556) (ce69ae3)

Documentation

docker fails to parse quotes in env value (#2345) (67584f7)
remove unnecessary bold style in README (#2509) (1392de1)
v1.0 announcement and event link (#2400) (44f3786)

Miscellaneous

add object storage support to replay (#2277) (e9e15b9)
add reason tag in failed requests and failed events stats (#2430) (a6137a9)
add replay support for all object storage (e9e15b9)
add resource tags in metrics (#2462) (2f73704)
added pre_drop_table count metrics (#2441) (869d694)
added rudder-server config metrics with version and other details as tag (#2445) (da55a86)
applying 1.1.1 hotfixes to main branch (#2449) (ac9afbd)
applying 1.1.2 hotfixes to main branch (#2474) (575c9e4)
change base image to alpine for build and prod (#2480) (4593f65)
clean up pathfinder which is related to old migrator code. (#2479) (3230fc6)
config package (#2439) (2ce9210)
dataset index calculation (#2512) (bfba9cf)
deps: bump github.com/mitchellh/mapstructure from 1.4.1 to 1.5.0 (#2442) (79c63e5)
deps: bump github.com/segmentio/kafka-go from 0.4.32 to 0.4.35 (#2455) (3441862)
deps: bump github.com/snowflakedb/gosnowflake from 1.6.3 to 1.6.13 (#2432) (7ae6421)
deps: bump go.etcd.io/etcd/client/v3 from 3.5.2 to 3.5.5 (#2499) (6ece30e)
disable deepsource transformer (0466ec6)
disable semantic PR single commit validation (#2487) (261345e)
drop old migrator (#2417) (870bccf)
excluding destId tag for free customers (#2555) (18b7f7b)
fix defect and use noCache if database setup fails during cache.Start (#2537) (89afba3)
fix error log defect in processor's transformer (#2547) (f49df7f)
fix flaky tests (#2424) (dadb4df)
fix race defect after stats refactoring (#2511) (13e28a0)
fix unused method receiver (#2498) (d17ece5)
gateway middleware refactoring (#2416) (1ea77a0)
include matched template path in gateway's request metrics (#2531) (ded763c)
include status code in pipeline_processed_events metric (#2450) (45936bb)
introduce tooling for local development (#2413) (85b0342)
introduced log & stats for retry of jobsDB execute & query methods (#2434) (c1b90f1)
jobsdb: introduce context aware mutex locks, transactional migrations & cancelable maintenance operations (#2505) (096db9c)
kafka cloud platform CI integration (#2504) (1f1c989)
limit queries to ds (#2446) (082b9f5)
linter fix unused method receiver (#2486) (7ae57b3)
linter fix: unused parameter should be replaced by underscore (#2490) (f1dfdab)
logger package (#2477) (19c08e2)
minor error message change (#2557) (a305aca)
reduce the warehouse slave payload (#2533) (1013a93)
remove unused suite test files (#2429) (be670e1)
reverse proxy warehouse pending-events endpoint (#2438) (8199002)
router: include runtime information during event ordering errors (#2421) (12d3e59)
simplify migrations using go's embed fs (#2535) (41ea78e)
skip json truncating chars (#2491) (e4fb710)
stats package (#2489) (c36b303)
upgrade go dependencies with (#2522) (3147333)
warehouse lint (#2493) (bb4f3d6)
warehouse: add coverage for warehouse configuration validations (#2538) (07dbe71)
warehouse: added integration test for snowflake case-sensitive database (#2501) (cf661fd)
warehouse: bigquery handler restructuring (#2551) (cb6fc27)
warehouse: format warehouse queries (#2540) (9b944c9)
warehouse: integration test changes v3 (#2426) (4e98109)
warehouse: pump more test for warehouse module (#2460) (58671c1)
warehouse: utility for staging table name and some refactoring (#2529) (079449c)
warehouse: validate object storage credentials (#2440) (3d2d87e)
warehouse: warehouse api to capture task run ID when calculating the pending uploads. (#2435) (d43705e)
warehouse: warehouse retry uploads count (#2275) (212db1e)

1.1.2 (2022-09-23)

Bug Fixes

snowflake identifier issue (#2471) (d4b2583)

1.1.1 (2022-09-15)

Bug Fixes

close http response body only if there is no error (#2447) (ef72aee)

1.1.0 (2022-09-12)

Features

add support for sending custom payload to webhook source (#2315) (8c0d811)
cpRouter Multitenant Support (#2233) (3c86dab)
honour anonId only from header (#2323) (cbe74b6)
improvements in rudder-sources integration (#2366) (54133d1)
introduce context, timeout and retries in jobsDB queries (#2217) (32b2f40)
isolate destID in router (#2369) (0708615)
jobsdb: thread safe addNewDSLoop (#2356) (8ef6c39)
perform ADD_DS operation in a single transaction (#2324) (7ee2d11)
thread safe Job Storage (#2403) (a123b10)

Bug Fixes

add trailing slash for licensing link (ca00c32)
add www and trailing slash to RS links (7574509)
safe webhook concurrent map access (#2389) (b72f6be)
use jsoniter when unmarshaling to avoid perfromance issues (#2381) (35ad8e4)

Miscellaneous

add token type for success message (#2408) (92600fd)
added metrics to track failed DS backup (#2370) (1d3cd42)
change proto message names (#2394) (689edd6)
disable parallel test execution and refactor jobsdb tests (#2410) (3bd27ee)
fix cache bug (#2412) (4f83ba7)
fix for TestNewForDeployment test (#2425) (1c4a36d)
log improvements (#2365) (95a24ca)
migrate small tables (#2327) (1ca1faa)
modified regulation-worker response code handling from 404 to 204, when no job is available (#2422) (44e8553)
router: include original client creation error when circuit breaker is open (#2373) (4d8a9bc)
router: new event ordering algorithm with proper aborted job limiting (#2344) (097c61b)
skip double mutex unlock (#2415) (366e1b9)
upgrade golangci and linting fixes (#2347) (6348406)
use generics for contains function (#2396) (2134c54)
use right id to allocate worker (#2399) (312a301)
using go test (#2411) (7eb62d9)
warehouse: remove tokens from logs for redshift and snowflake (#2359) (7409c56)
warehouse: skip warehouse snowflake test (#2404) (09bc408)
warehouse: unnecessary join in fetch schema for postgres and snowflake (#2360) (2de0d68)

1.0.2 (2022-08-25)

Bug Fixes

jobsdb: report correct table count metrics from gateway writer (#2333) (bb445a5)
release-please behaviour (33d1dec)
remove anti pattern issues (#2335) (7468709)
router: event ordering algorithm with proper aborted job limiting support (#2329) (c6f8173)
router: trying to stop the router before the generatorLoop is started causes router to hang (#2321) (cbaef78)
warehouse: added initialization for datalake (#2332) (8fe1bf9)
warehouse: upload validations should happen when validator is set (#2348) (f0a6416)

Miscellaneous

fix lint errors in test (#2352) (d67ef3a)
fix misspell (c5d5060)
include all supported pr types in changelog (#2339) (c074298)
introduce go report card (#2320) (578de21)
remove build tag from test (#2350) (cf6d629)
remove build tags from tests (#2346) (16cc643)
skip v1.0.1 release (d65658d)
upgrade actions runner to ubuntu 20.04 (#2326) (8c970f6)
upgrade GO version to 1.18 (#2328) (edde758)
warehouse: update errorMap for deltalake partition query (#2334) (e011726)

1.0.0 (2022-08-19)

Features

backend config namespace support (#2232) (30da9ec)
core: secure customer s3 bucket access for backups using cross-account role (#2250) (964c2d9)
destination: onboarding Lambda destination (#2229) (581aef4)
introduce retention for datasets (#2214) (beb7be8)
minor changes for namespace config endpoint (#2307) (8c5c2bc)
processor: filter unsupported messages (type & event fields) (#2211) (e701950)
update namespace endpoint according to the latest changes (#2269) (eb67dca)
use different http client timeouts for different components (#2300) (9dc3d48)
warehouse: handle bucket config change (#2084) (7f83f0b)

Bug Fixes

awsutils to support secretAccessKey (#2306) (801355e)
backend config auth (#2259) (4c014a0)
backend config refactoring (#2200) (e242c7d)
core: configSubscriber lock, router_status_events count (#2181) (3c89b96)
core: order inserts(updates) to avoid deadlocks (#2167) (28c8872)
destination: oauth expired secret type change to json.RawMessage from string (#2236) (b001747)
destination: transformer proxy respond with timeout for slow or unresponsive dests. (#2258) (c592462)
dockerfile issues related to scripts (#2298) (7348cb2)
fetch filemanagers timeout from config instead of registering (#2213) (b764798)
gateway: make gateway user logic synonymous to proxy logic (#2205) (8c889ae)
group errors swallowed + honouring ctx (#2309) (ae53e6e)
jobsdb: fix ANALYSE for Negative Indices during internal migration (#2192) (69ae857)
multi_tenant_test.go improvements (#2247) (b644468)
only tag images with :latest if they are part of a release (#2174) (b8c5353)
parse JSON error field as a string (#2246) (fbb9b62)
reinstate bugsnag notifications (#2265) (781862c)
restore / endpoint (#2237) (2316604)
router: handle transformer in/out inconsistencies (#2276) (ecca8ec)
stream manager logging format string (#2263) (5aea319)
warehouse: marshall pg_notifier_queue payload during the time of inserting the data (#2243) (a3d1632)
warehouse: reset priority if earlier upload is already in progress (b9e2674)
warehouse: support skipping escaping of reserved keywords (#2115) (2b4425a)
warehouse: use correct config for destination name (#2221) (666654e)
warehouse: use global config for parquet patallel writers (#2184) (54091ed)

Miscellaneous Chores

prepare v1 release (040b734)

0.2.0 (2022-07-06)

Features

core: add support for managing failed records in rsources.JobService (#2087) (c22541c)
core: introduce new failed-records endpoint (#2120) (70cbdcd)
destination: putting support for AVRO in Kafka. (#2042) (b9677a8)
expose robots.txt to disable crawling (#2090) (506e9aa)
warehouse: enable postgres query execution (#2106) (66a8846)

Bug Fixes

calculate consistent tagStr cache keys in stats (#2108) (fea8e30)
kafka multiple hosts (#2103) (30ee189)
make sure jobsdb is started when using the replay feature (#2071) (b64a37f)
multi-tenant: removed unused mock calls (#2073) (7e33ede)
processor: include test for sources fields in transform event metadata (#2143) (5a0f854)
properly start a new badgerdb transaction on ErrTxnTooBig (#2061) (ea839f4)
removed the output of goRoutine count from the HTTP GET / of the server (#2079) (ca6450b)
update failedJobIDMap after draining a job in router (#2065) (19d1e3d)
use a writer jobsdb for writing to gw tables during replay (#2086) (bad7dea)
warehouse: close connections after validation steps (#2122) (6491d65)

Miscellaneous Chores

change release to 0.2.0 (3cd36f4)

0.1.11 (2022-06-16)

Features

core: pub-sub for sources jobs stats in a multi-tenant scenario (#2022) (81d01ce)
integration tags (#2053) (3ad96b4)

Bug Fixes

jobsdb: use transaction instead of previous prepared statement for executing ANALYZE (#2054) (ef01b16)

0.1.10 (2022-06-14)

Features

calculate aggregated pending_events_count metrics for all workspaces (#1858) (f16a239)
capture rudder-sources job statistics (#1949) (d14deba)
Checks checksum written to the destination ssl folder, rewrite only if checksum changes (214e774)
Checks checksum written to the destination ssl folder, rewrite only if checksum changes (57f73e8)
core: add support for transient sources (#1869) (049b015)
core: default rsources.JobService implementation (#1938) (b08bdee)
core: updating enterprise commit with changes related reporting service (#2035) (e94d313)
enable timeouts for google destinations (#1857) (31b64d7)
gateway back pressure mechanism implementation (#1847) (c9b6f02)
gateway: introduce /job-status endpoint (#1917) (688cf1e)
google sheets batching support (#1764) (497b1d9)
jobsdb: made backup related config hot-reloadable (#1844) (5f69442)
jobsdb: support limiting jobs' pickup based on total payload size (#1884) (699727f)
kafka client (#1875) (f900211)
kafka publishTime stat (#2008) (01454ad)
more kafka stats (#2013) (c2379c1)
multi-tenant: add mode provider in docker-compose file (#1931) (a81bb0a)
multi-tenant: cluster manager gateway int (#1943) (1f2ca4d)
multi-tenant: etcd provider and config backend updates (#1839) (6985721)
multi-tenant: introduce deployment types, enabling multitenant with etcd (#1882) (6c04f99)
multi-tenant: multitenant support for embedded app (#2018) (8ea4ff6)
multi-tenant: use cached backend config (#1996) (8f1cc8b)
router: enable batching config at destType level. (#1902) (d69d489)
router: use a circuit breaker while producing custom destination clients (#2016) (235f3c2)
stats for skipped messages on kafka destinations (#1934) (b58c7e9)
use new client for kafkamanager to support timeouts (#1920) (82db596)
warehouse: added metrics for capturing stats in warehouse scheduling (d2868c7)
warehouse: added support for deltalake partition and external location support. (#1914) (7d85221)
warehouse: added support for multi retry admin api (#1889) (230473b)
warehouse: configurable timewindow layout for gcs datalake (#2015) (931c5fc)
warehouse: json column support for warehouse destinations bq,postgres,snowlake,rs (#1897) (2f877f7)

Bug Fixes

pre_drop table pileup issue (#1799) (e301b56)
added http client timeout (#1873) (83f8041)
added http client timeout in processor transformer (#1887) (7028f15)
admin status handlers data race (#1838) (d4bdb5c)
disable bq dedup by default (#1855) (8c05b37)
disable creation of views when dedup is enabled in BQ (#1885) (414551d)
disable stash backup if master Backup is disabled. (#1874) (ea9921d)
disable support for Google Developers Console client_credentials.json file (#2001) (ef83509)
do not skip stash aborting jobs if backup is disabled (#2003) (498bd93)
enabling earlier usage of stats in other components (#1937) (b71c93d)
EventBus memory issue with slow subscribers (#1802) (d6b85c1)
execute ANALYZE after completing batch jobsdb operations (#1800) (db951aa)
fix repetitive bugsnag alert for a single crash (#1928) (0653956)
Force set lower version during pg notifier sql migration (#1907) (151e308)
Force set lower version during pg notifier sql migration (#1907) (#1908) (a54bc45)
gateway: getUsersPayload optimizations (#2040) (f48693e)
install ginkgo V2 cli (ee904a4)
Jobsdb migration bug (#1785) (b091116)
jobsdb: mark MIGRATE_COPY done and POST_MIGRATE_DS_OP start atomically (#1999) (0ef882f)
jobsdb: sanitize jsonb values before inserting them to the database (#1963) (c74df14)
lock by value (#1842) (777541c)
Makefile: Keep separate cover profiles (823ab85)
move initialisation to setup (23445a4)
multi-tenant: etcd Key case fix (#1954) (304a0b7)
multi-tenant: multitenanat backend config (#1930) (1ed73fa)
ovverride arm64 check (#1936) (e887e23)
rateLimit: Moving Rate Limit Block post Validations (4720e6b)
readme: update developer machine setup link (#1843) (cd32fb7)
remove ginkgo version (87c3d89)
Remove unused Setup and Finish methods (0149a18)
reporting the drained events to the reports table (#1982) (b40dabe)
reporting: gateway events are not reported when source is connected to no destination(or device mode destination) (#1822) (2962bd6)
router: Added/Modified Stats for assistance in debugging for transformerProxy (#1979) (b79d9d9)
router: panic sources grouped together (#1846) (69344c6)
safe concurrent access of metric registry's index values (#1805) (7e672e6)
Save responses from the destinations with odd content-type header (#1749) (67c1f59)
support multiple configvars registration (#1878) (e1ece33)
take care of nullable columns when retrieving dangling connections (#1801) (099b3c0)
tests.yaml: remove go get ginkgo (5ad45bc)
tests.yaml: remove go get ginkgo open source (bac703c)
tooling: add prometheus endpoints for OSS users (867c9c3)
update enterprise commit (ad3dc56)
Update enterprise commit (55a0f83)
use the correct measurement name and tags for aggregated pending events metrics (#1891) (5f45c1f)
warehouse: Add explicit migration for pg notifier priority column (#1898) (224038d)
warehouse: add support for multiline in deltalake. (#1872) (ecace44)
warehouse: added ability to test destination credentials once the upload gets aborted (#1890) (fbc862d)
warehouse: closing file while creating load file for warehouse configuration test (#1919) (8d71648)
warehouse: columns need to be provided during copy command for deltalake (#2033) (75d15c6)
warehouse: deltalake users table getting populated corruptly. (#1976) (#1977) (8ba4182)
warehouse: dsiable reporting setup for slave (#2037) (983338a)
warehouse: fix s3 manager for aws glue region (#1951) (539c0c9)
warehouse: fixing timeout issue for snowflake (#1945) (c82428a)
warehouse: skip backendConfig check for slave (#2034) (e513501)
warehouse: skipping bigquery integration test for now (4bae57b)
warehouse: updated reserved keywords for warehouse mssql and azure destination (#1932) (928d3d4)
warehouse: use correct locks for dateformat in processor (#2020) (972577a)
warehouse: warehouse schema int to float schema fix (#1974) (bf46c85)
wrong routing of warehouse crash alerts (#1926) (e77dc83)

Miscellaneous Chores

new release 0.1.10 (e8fc750)

Files

CHANGELOG.md

Latest commit

History

CHANGELOG.md

File metadata and controls

Changelog

1.25.0 (2024-05-06)

Features

Miscellaneous

1.24.0 (2024-04-25)

Features

Bug Fixes

Miscellaneous

1.23.0 (2024-04-08)

Features

Bug Fixes

Miscellaneous

1.22.1 (2024-03-20)

Bug Fixes

1.22.0 (2024-03-19)

Bug Fixes

Miscellaneous

1.21.1 (2024-03-14)

Bug Fixes

1.21.0 (2024-03-06)

Features

Bug Fixes

Miscellaneous

1.20.0 (2024-02-12)

Features

Bug Fixes

Miscellaneous

1.19.2 (2024-01-18)

Bug Fixes

1.19.1 (2024-01-16)

Bug Fixes

1.19.0 (2024-01-15)

Features

Bug Fixes

Miscellaneous

1.18.2 (2023-12-12)

Bug Fixes

1.18.1 (2023-12-07)

Miscellaneous

1.18.0 (2023-12-05)

Features

Bug Fixes

Miscellaneous

1.17.3 (2023-12-05)

Bug Fixes

1.17.2 (2023-11-20)

Bug Fixes

1.17.1 (2023-11-17)

Miscellaneous

1.17.0 (2023-11-14)

Features

Bug Fixes

Miscellaneous

1.16.3 (2023-11-09)

Bug Fixes

Miscellaneous

1.16.2 (2023-11-08)

Bug Fixes

1.16.1 (2023-11-06)

Bug Fixes

1.16.0 (2023-10-31)

Features

Bug Fixes

Miscellaneous

1.15.4 (2023-10-23)

Bug Fixes

1.15.3 (2023-10-19)

Bug Fixes

1.15.2 (2023-10-18)

Miscellaneous

1.15.1 (2023-10-13)

Bug Fixes

Miscellaneous

1.15.0 (2023-10-11)