Skip to content

Releases: spark-jobserver/spark-jobserver

v0.11.1

06 May 18:25
Compare
Choose a tag to compare
  • fix(build): Switch to JFrog instead of Bintray / (Valentina Glagoleva | 15bff80)
  • feat: Add callbackUrl to jobs / (Marc Zoeller | 3764f9b)
  • refactor: Replace expectNoMsg with expectNoMessage to fix deprecation warnings / (Marc Zoeller | e3c926c)
  • bug: backward compatibility of "python-archive" / (Marc Zoeller | b50667d)
  • refactor(Context Reset): Make context resets async and report exceptions / (Marc Zoeller | 78cc201)

v0.11.0

06 Feb 08:05
Compare
Choose a tag to compare
  • misc(Python): Document wheel support / (Marc Zoeller | 50d4138)
  • misc(Python): Replace python 2 with python 3 / (Marc Zoeller | 22a731e)
  • refractor(auth): Rename "authentication" to "access-control" / (Marc Zoeller | 15f457d)
  • feat(auth): Permission Management / (Marc Zoeller | 49d1580)
  • feat(auth): BasicAuth via Keycloak / (Marc Zoeller | 8c98799)
  • refractor(auth): Cleanup authentication / (Marc Zoeller | 9efaa15)
  • fix(docker): Adapt docker config to updated library versions / (Valentina Glagoleva | 648f9a0)
  • misc: Update scala version in scripts and readme / (Marc Zoeller | a63bfbe)
  • misc: Increase cache timeout / (Marc Zoeller | 7ab8e2d)
  • misc: Remove unsupported jvm opts / (Marc Zoeller | a867380)
  • refractor: Close open ActorSystems / (Marc Zoeller | 0e70e8e)
  • refractor: Replace deprecated slick.driver with slick.jdbc / (Marc Zoeller | e2deb1e)
  • feat: Migrate to scala-2.12 / (Marc Zoeller | 90803ba)
  • Fix Docker image (#1324) / (Valentina Glagoleva | e2aeb9b)
  • fix(dao): Jobs endpoint should return jobs in desc order / (Valentina Glagoleva | b791741)
  • fix(test): fix Python style guide checker / (Prabhu Gnana Sundar | be46be5)
  • fix(test): fix Python style guide checker / (Prabhu Gnana Sundar | 7bd155e)
  • fix(zookeeperdao): Remove duplicate list operation / (Robin Wolters | 99d546b)
  • feat(launcher): Pass extraJavaOptions to executor / (Robin Wolters | da8d6f9)
  • fix(build): Change typesafe repository to https / (Valentina Glagoleva | 60b3133)
  • docs(README): Add Cadenz.ai to the users list (#1322) / (Prabhu Gnana Sundar | bb60528)
  • doc(release): Move into separate file / (Valentina Glagoleva | 669efa1)
  • refactor(readme): Improve commit output format / (Behroz Sikander | 612aae7)
  • doc(readme): Add release process documentation / (Behroz Sikander | bf5ec81)
  • release(jobserver): Ignore untracked files during release / (Behroz Sikander | 83c3477)
  • doc(swagger): Update according to last WebAPI changes / (Valentina Glagoleva | 42daf45)
  • refactor(config): Remove combineddao section from config / (Valentina Glagoleva | 645945a)
  • refactor(dao): Remove JobDAO and InMemoryDAO classes / (Valentina Glagoleva | 0a6b672)
  • feat(dao): Delete CombinedDAO and port its tests / (Valentina Glagoleva | e28f9b4)
  • feat(dao): JobDAOActor uses Meta and Binary DAO / (Valentina Glagoleva | 1940182)
  • feat(webpi): Add counter for internal server error / (Valentina Glagoleva | e08ab4a)
  • doc(readme): Update supported Spark versions table / (Valentina Glagoleva | 65f45a8)
  • fix(jobmanager): Spawn separate thread for onMemberRemoved / (Valentina Glagoleva | 18b58be)
  • fix(akka): Improve shutdown logic / (Valentina Glagoleva | 9906782)
  • Merge pull request #1305 from valan4ik/fix_opensource_tests / (Noorul Islam K M | 6e90f53)
  • fix(tests): Add retry to AkkaClusterSupervisorActorSpec / (Valentina Glagoleva | 68ec697)
  • fix(test): Deduplicate WebApi PUT contexts test / (Valentina Glagoleva | 152a50b)
  • fix(file-cache): Create missing directories / (Valentina Glagoleva | 3cd159e)
  • fix(jobserver): Graceful force kill if master is down / (Behroz Sikander | 1deaaeb)
  • feat(test): Convert in-memory dao to combineddao / (Behroz Sikander | 155997c)
  • doc(jmx): Add example how to down a node from jmx-cli / (Valentina Glagoleva | 187f149)
  • misc(cassandra): Remove CassandraDAO / (Valentina Glagoleva | cf8d7a7)
  • doc(readme): Remove JobSqlDAO from README / (Valentina Glagoleva | 42d7406)
  • misc(dao): Remove JobSqlDAO and SqlCommon / (Valentina Glagoleva | 6fb38c2)
  • tests(cassandradao): Use own test config file / (Valentina Glagoleva | 642a665)
  • fix(tests): Use CombinedDAO in MetaDataSqlDAOSpec / (Valentina Glagoleva | 28d5a20)
  • misc(jobserver): Replace JobSqlDAO checks with CombinedDAO / (Valentina Glagoleva | 2bc16bb)
  • refactor(dao-tests): Remove SqlTestHelpers from MetaDataDAOSpec / (Valentina Glagoleva | 4c985b2)
  • refactor(dao): Remove SqlCommon from MetaDataSqlDAO / (Valentina Glagoleva | 40b1974)
  • feat(dao): Add CombinedDAO support for MySQL / (Valentina Glagoleva | b365e2d)
  • feat(postresql): Add CombinedDAO support for PostgreSQL / (Valentina Glagoleva | 6e283de)
  • doc(combined-dao): Add documentation how to setup SQL DAO / (Valentina Glagoleva | a8261cd)
  • feat(dao): Add BinarySqlDAO / (Valentina Glagoleva | ef99b77)
  • feat(dao): Add in-memory meta and binary dao / (Behroz Sikander | e522b45)
  • refactor(jobserver): Adapt file as per scalastyle / (Behroz Sikander | 0923892)
  • refactor(jobserver): Replace info actor with dao actor / (Behroz Sikander | 392717a)
  • refactor(jobserver): Add GetJobInfo message to DAOActor / (Behroz Sikander | fc432a7)
  • refactor(jobserver): Adapt SaveJobConfig as per JobInfoActor / (Behroz Sikander | ce9dbf0)
  • feat(jobserver): Add status filter in GetJobInfos message / (Behroz Sikander | cead28e)
  • fix(bin): Remove time stamp from gc logs / (Behroz Sikander | 26c5c39)
  • fix(bin): Add date to gc logs / (Behroz Sikander | 3e2e7ec)

v0.10.1

27 Nov 19:40
Compare
Choose a tag to compare
  • fix: Fail postgres migration on exception / (Marc Zoeller | c650a53)
  • fix(build): Change typesafe repository to https / (Valentina Glagoleva | ee63e6e)
  • fix: sort jobs by start time in descending order / (Marc Zoeller | cd11406)
  • refactor(migration): remove temp table / (Marc Zoeller | 8c24a0a)
  • fix: flyway postgres migration to V0.7.9 / (Marc Zoeller | b7a5afd)
  • notes(jobserver): Add release notes for 0.10.0 (#1286) / (Behroz Sikander | e3c3d3c)

v0.10.0

18 Mar 15:19
Compare
Choose a tag to compare

Merge pull request #1283 from bsikander/244_upgrade / (Noorul Islam K M | a448f2d)
feat(python): Disable hive support from Python Spark Session / (Behroz Sikander | 66f6150)
refactor(python): Make subprocess.py PEP8 complaint / (Behroz Sikander | 875407a)
fix(tests): Stop context cleanly / (Sikander | d53f507)
feat(integration-tests): Add subproject for integration tests (#1285) / (Nibooor | d10522c)
refactor(readme): Set stop-context-on-job-error=true (#1284) / (Behroz Sikander | 61598af)
jobserver(python): Enable secure communication / (Behroz Sikander | 31fd78c)
doc(webapi): Add swagger documentation for Jobserver WebApi (#1282) / (Nibooor | bf54e18)
fix(ci): Update CI scripts to 2.4.4 / (Behroz Sikander | da9dfd4)
feat(jobserver): Add missing C* dependency / (Behroz Sikander | d88b956)
feat(jobserver): Disable hive support from SparkSession / (Behroz Sikander | f44aa50)
feat(jobserver): Upgrade spark to 2.4.4 / (Behroz Sikander | 1244df7)
fix(dependent-jars): Dependent-jar-uris for context config / (Valentina Glagoleva | 8911bb1)
misc(webapi): Standardize response functions and message / (Valentina Glagoleva | b9de4e1)
refactor(webApi): Remove unused code / (Valentina Glagoleva | 25d2534)
fix(WebApi): Respond only with status code instead of "OK" / (Valentina Glagoleva | b4f13d3)
refactor(WebApi): Use StatusCode obj for response codes / (Valentina Glagoleva | 90daa98)
ci(jobserver): Build failing due to missing public key / (Behroz Sikander | 7fbf03e)
Update README.md (#1276) / (pgouda89 | 07672b3)
refactor(cassandra): Deprecate Cassandra DAO (#1272) / (Valentina Glagoleva | 611f7e6)
feat(jobserver): Add /binaries/ endpoint (#1273) / (Nibooor | 808010a)
Merge pull request #1267 from valan4ik/dependent_jars / (Noorul Islam K M | 9662313)
test(jobdaoactor): Add test for broken URI case / (Valentina Glagoleva | 02e3c1e)
doc(readme): Update outdated coverage test command (#1270) / (Valentina Glagoleva | 2308463)
fix(cassandra): Fix CassandraDAO config path / (Robin Wolters | 1536a2c)
refactoring(tests): Remove duplicating genJob function / (Valentina Glagoleva | 6b4192f)
fix(python): Improve job validation / (Valentina Glagoleva | 78ec91f)
doc(jobserver): Dependency jars and job submit / (Valentina Glagoleva | 6b08de4)
fix(jobserver): Fix DAOs and refactor JobInfo / (Valentina Glagoleva | 1916d43)
misc(cassandra): Add small readme and settings section / (Valentina Glagoleva | 9fe16d9)
refactor(jobserver): Remove JobFileDAO / (Valentina Glagoleva | 8eef173)
feat(jobdaoactor): Validate schema for URIs / (Valentina Glagoleva | 35d66b9)
fix(binarymanager): Fix exception message type for cp / (Valentina Glagoleva | b74268c)
feat(jobserver): Extend JobInfo to maintain cp BinaryInfo list / (Valentina Glagoleva | 3a12f1c)
feat(jobserver): Add URI BinaryType / (Valentina Glagoleva | a1a44b5)
internal(jobserver): Compatibility code for already running jobs / (Valentina Glagoleva | 5c79c42)
feat(jobserver): Extend dependent-jar-uri feature / (Valentina Glagoleva | 179f794)
test(jobmanager): Add test for dependent-jar-uris feature / (Valentina Glagoleva | 91bf8da)
refactor(jobserver): Refactor CombinedDAO / (Robin Wolters | d69f5d7)
feat(jobserver): Move ZK DAO timers to CombinedDAO / (Robin Wolters | 28ab843)
feat(jobserver): Make stop-context-on-job-error default for streaming jobs / (Behroz Sikander | e068585)
feat(jobserver): Add stop-context-on-job-error feature / (Behroz Sikander | a4ac6ca)
Fixes and multi-jvm tests for split brain resolver (#1264) / (Valentina Glagoleva | 3fb2d94)
feat(webapi): check sjs actors liveness (#1196) / (Srivignesh | a71fe07)
doc(readme): Remove SPARK_CLASSPATH as it's deprecated / (Valentina Glagoleva | 187b7b3)
misc(akka): Upgrade Akka version to 2.4.20 / (Valentina Glagoleva | 832be9c)
fix(tests): Disable akka logging / (Behroz Sikander | 7ab2090)
fix(test): Fix flaky test in JobManagerActorSpec / (Behroz Sikander | bf3bbfe)
fix(test): Stop autopilot to fix flakiness / (Behroz Sikander | 695d0ff)
fix(tests): Cleanly clear SparkEnv object / (Behroz Sikander | e9f1e24)
refactor(jobserver): Add space in log message / (Behroz Sikander | cea3edd)
fix(docu): Add docu for console logging for tests / (Behroz Sikander | 0f18636)
fix(tests): Speed up SessionJobSpec test / (Behroz Sikander | 1e0504a)
fix(sbt): Use evaluated instead of inputTaskValue / (Behroz Sikander | 5c27de6)
fix(readme): Add force=true flag for delete context / (Behroz Sikander | 1f02572)
feat(sbt): Move sbt to 1.2.6 / (Behroz Sikander | 546dff5)
fix(test): Fix spark.driver.host to localhost / (Behroz Sikander | d912972)
refactor(jobserver): Remove outdated TODO / (Robin Wolters | 6cb6810)
fix(akka): Subscribe SBR to cluster events in constructor / (Valentina Glagoleva | f44c81f)
fix(akka): SBR should exit with -1 without hooks / (Valentina Glagoleva | 103e043)
feat(api): Handle all throwables in loadAndValidateJob / (Robin Wolters | 345cc7c)
refactor(jobserver): Address SBT warnings / (Valentina Glagoleva | 9b1addb)
doc(jobserver): Mention jobserver HA in README.md / (Robin Wolters | c5fc2e1)
doc(ha): Add documentation for HA setup / (Robin Wolters | 3433f04)
doc(jobserver): Update DAO setup documentation / (Robin Wolters | 5533ded)
fix(jobserver): Decrease down-removal-margin / (Robin Wolters | c6a5e94)
doc(splitbrain): Add documentation for Jobserver split brain resolver / (Valentina Glagoleva | 368125a)
feat(akka): Add split brain resolver / (Valentina Glagoleva | 832fef9)
fix(jobserver): Add watch to existing contexts automatically / (Robin Wolters | f6082df)
fix(jobserver): Don't allow STOPPING context to restart / (Valentina Glagoleva | 3e3151f)
feat(jobserver): Add HA setup for jobserver (#1241) / (Behroz Sikander | d7f2682)
fix(jobserver): Fix custom exceptions constructor / (Valentina Glagoleva | f975d8b)
feat(jobserver): Make AkkaClusterSupervisorActor a singleton / (Robin Wolters | 9448ee7)
Fixes related to StopContext flow (#1238) / (Nibooor | 8ef2121)
doc(jobserver): Update jobs state machine (#1237) / (Behroz Sikander | 2b6c0ac)
Only respond if cleanup is complete (#1233) / (Behroz Sikander | af63e84)
refactor(webapi): Deprecate WebAPI /jars path / (Valentina Glagoleva | 16eaff5)
feat(sbt): Use cached resolution / (Valentina Glagoleva | 7448975)
CassandraDAOSpec: Fix of flaky test and small refactoring (#1231) / (Valentina Glagoleva | 483840c)
feat(jobserver): Clean non-final jobs at startup / (Behroz Sikander | fa749b7)
fix(jobserver): Repair final states without end time / (Robin Wolters | 0afcc7e)
fix(jobserver): Properly set end_time on ForkedJVMInitTimeout / (Robin Wolters | 75954de)
refactor(jobserver): Improve log messages / (Behroz Sikander | 1a4e55e)
refactor(jobserver): Add logging before terminating JVM / (Behroz Sikander | 3780682)
refactor(jobserver): Explain CombinedDAO binary deletion error / (Robin Wolters | d978b9d)
fix(tests): Make NamedRDD calculation longer for timeout tests / (Valentina Glagoleva | f025a4c)
fix(tests): Add autodown to AkkaClusterSupervisorSpec / (Valentina Glagoleva | 5e995b5)
refactor(tests): AkkaClusterSupervisor compliant with scalastyle / (Valentina Glagoleva | b4f7ad8)
fix(tests): Cleanup SQL database in tests / (Valentina Glagoleva | 51aa2d8)
fix(jobserver): Persist error state sequentially during startup / (Robin Wolters | f4e1edd)
fix(jobserver): Remove refreshBinary for Zookeeper / (Robin Wolters | 302cf92)
fix(tests): Adapt AutoPurgeActorSpec to slow builds / (Robin Wolters | 4e94910)
doc(jobserver): Add documentation of Zookeeper autopurge feature / (Robin Wolters | 9c84c97)
feat(jobserver): Automatically purge Zookeeper data / (Robin Wolters | 5193eaa)
refactor(jobserver): Replace hard-coded dao timers with config values / (Robin Wolters | 94ed79b)
fix(jobserver): Add timers to ZookeeperDAO / (Robin Wolters | cc6a28a)
fix(jobserver): Add timers to WebAPI / (Robin Wolters | f543b00)
ZookeeperDAO fixes (#1216) / (Nibooor | 9b2b600)
doc(jobserver): Add context creation flow diagram (#1215) / (Nibooor | 206dec5)
Minor fixes and improved logging (#1214) / (Behroz Sikander | b054242)
Remove binaries if not used & adding logging to webapi (#1209) / (Behroz Sikander | a2d5eb8)
fix(tests): Fix saveJobConfig tests (#1210) / (Behroz Sikander | 489b45a)
Make zookeeper settings configurable (#1211) / (Valentina Glagoleva | 53d335c)
refactor(jobserver): Remove blocking await from StartJob / (Valentina Glagoleva | d705751)
Downgrade zookeeper version and remove not needed ZK test helper (#1206) / (Valentina Glagoleva | f4f29ed)
feat(jobserver): Force consistent read with manual sync (#1205) / (Nibooor | 14a1d4f)
Minor refactor for WebApi logging (#1200) / (Valentina Glagoleva | dc703ad)
Merge pull request #1201 from valan4ik/combined_dao_doc_conn / (Noorul Islam K M | 399f731)
fix(travis): Update base container tag (#1204) / (Behroz Sikander | 6d3e616)
fix(zookeeper): Scale timeouts to avoid ConnectionLoss exception / (Valentina Glagoleva | 372d510)
doc(jobserver): Add documentation for CombinedDAO / (Valentina Glagoleva | 815ee6a)
fix(jobserver): Create Job with full BinaryInfo information / (Valentina Glagoleva | d339ce3)
fix(jobserver): Return JobInfo with binHash in SQL DAOs / (Valentina Glagoleva | 1ad61b0)
fix(jobserver): keeping unused binaries (#1198) / (Behroz Sikander | ae54f48)
fix(jobserver): Exclude guava from curator dependencies / (Robin Wolters | ca20f24)
fix(tests): Fix build issues with test setup / (Robin Wolters | 4c70b9f)
refactor(jobserver) Improve ZooKeeper logging / (Robin Wolters | c4bba5c)
feat(jobserver): Reuse zookeeper connections within same dao operation / (Robin Wolters | 6c10947)
fix(tests): Fix curator framework import and namespace / (Valentina Glagoleva | 4f4d642)
feat(jobserver): Implement Zook...

Read more

v0.9.0

12 Mar 06:13
Compare
Choose a tag to compare
  • feat(jobserver): Replace manager_start with SparkLauncher (@bsikander) - this is a pretty big deal to stabilize and extend SJS

  • feat(jobserver): Create context table (@ecandreev)

  • feat(jobserver): move data from contexts hash to db (@ecandreev)

  • feat(jobserver): Add CONTEXT_ID and STATE to the JOBS table (@ecandreev)

  • feat(jobserver): Introduce supervise mode (@bsikander)

    • feat(jobserver): Filter jobs by contextId and multiple statuses
    • feat(jobserver): Introduce supervise mode for contexts
    • feat(jobserver): Make ActorSystem remote port configurable in Jobserver
    • fix(jobserver): Improve resolution of JobManagerActor addresses
    • feat(jobserver): Add parameter to ignore akka.remote.netty.tcp.hostname
    • feat(jobserver): Make sure akka tcp port is not random
    • fix(jobserver): Add timeout for DAO failure testcases
    • fix(jobserver): Add documentation and remove unused variable
  • Updated Python documentation to reference PythonSession (@rfrohock)

  • fix(webapi): repair context info route for mode context-per-jvm (@pk-work)

  • feat(webapi): Add application id to route /contexts/ (@pk-work)

  • test(jobserver): Add testcases for AkkaClusterSupervisor (@bsikander)

  • fix(jobserver): Fix randomly failing testcase in AkkaClusterSupervisorSpec (@bsikander)

  • feat(jobserver): Allow valid memory value and add testcases (@bsikander)

  • feat(jobserver): Remove extra states from contextInitInfos (@bsikander)

  • feat(jobserver): Extend zombie killing logic to add Initialize msg (@bsikander)

  • fix(jobserver): Fix testcases which simulate DAO failure (@bsikander)

  • feat(jobserver): Remove extra states from contextInitInfos (@bsikander)

  • Improved documentation in readme for H2 db (@ericraymond)

  • docs(jobserver): Add new logging doc and a line to cluster doc (@LXXIII73)

  • docs(jobserver): Update JMX documentation for Akka Cluster (@bsikander)

  • fix(job-server): use 24h format for cached binary file name (@sgautrin)

  • docs(jobserver): Fix broken links (@VEINHORN)