Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migration for all the model changes for Project Templates #69816

Open
wants to merge 28 commits into
base: master
Choose a base branch
from

Conversation

saponifi3d
Copy link
Contributor

Description

This will create 2 new models, ProjectTemplate and ProjectTemplateOptions. These models will be used to store settings across multiple projects. This PR also creates a 1 to 1 relationship for ProjectTemplates and Projects.

Design Document: https://www.notion.so/sentry/Service-Settings-Templates-ca6fdb38b15f4a6db61d319bc342b86c?pvs=4#60623e7fbaba4e0985f8d4581af18e64

@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Apr 26, 2024

class Meta:
app_label = "sentry"
db_table = "sentry_projecttemplate"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need any unique together constraints like only 1 per org?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, nope, we want to support multiple templates in an org. The feature use case is an organization wants to have a default template for their python backend services and their js FE services.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the names be unique per org?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh snap, i did not read that very well. yeah, they should likely be unique names per org. thanks for the shout Colleen and the catch Dan!

@@ -245,6 +245,7 @@ class Project(Model, PendingDeletionMixin, OptionMixin, SnowflakeIdMixin):
# projects that were created before this field was present
# will have their first_event field set to date_added
first_event = models.DateTimeField(null=True)
project_template = FlexibleForeignKey("sentry.ProjectTemplate", null=True)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Someone like Dan would know better than me but idk if you can add this and create the new table in one go

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, i wasn't sure if i should do all 3 model changes in 1 pr or in 3 PRs. I can ping dan and see if this makes sense.

fwiw the migration did work locally.. unclear if that means it'll work in prod or not though 😅

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah ha! it seems that django uses the operations key to decide the order it applies the migrations: https://github.com/getsentry/sentry/pull/69816/files#diff-705160f3302cfba81a9ad4b3182878d45a36c8a5f3f55d5be85d5ebde083e309R32 - since django automates all this they figured out the application order for me.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine, you can add an FK and the table in the same migration

Copy link
Contributor

github-actions bot commented Apr 29, 2024

This PR has a migration; here is the generated SQL for src/sentry/migrations/0709_project_template_models.py ()

--
-- Create model ProjectTemplate
--
CREATE TABLE "sentry_projecttemplate" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "date_updated" timestamp with time zone NOT NULL, "date_added" timestamp with time zone NULL, "name" varchar(200) NOT NULL, "organization_id" bigint NOT NULL);
ALTER TABLE "sentry_projecttemplate" ADD CONSTRAINT "sentry_projecttempla_organization_id_b072f5c6_fk_sentry_or" FOREIGN KEY ("organization_id") REFERENCES "sentry_organization" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_projecttemplate" VALIDATE CONSTRAINT "sentry_projecttempla_organization_id_b072f5c6_fk_sentry_or";
CREATE INDEX CONCURRENTLY "sentry_projecttemplate_organization_id_b072f5c6" ON "sentry_projecttemplate" ("organization_id");
--
-- Add field template to project
--
ALTER TABLE "sentry_project" ADD COLUMN "template_id" bigint NULL;
--
-- Create model ProjectTemplateOption
--
CREATE TABLE "sentry_projecttemplateoption" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "key" varchar(64) NOT NULL, "value" text NOT NULL, "project_template_id" bigint NOT NULL);
ALTER TABLE "sentry_project" ADD CONSTRAINT "sentry_project_template_id_67f0bec7_fk_sentry_pr" FOREIGN KEY ("template_id") REFERENCES "sentry_projecttemplate" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_project" VALIDATE CONSTRAINT "sentry_project_template_id_67f0bec7_fk_sentry_pr";
CREATE INDEX CONCURRENTLY "sentry_project_template_id_67f0bec7" ON "sentry_project" ("template_id");
ALTER TABLE "sentry_projecttemplateoption" ADD CONSTRAINT "sentry_projecttempla_project_template_id_23fa7e71_fk_sentry_pr" FOREIGN KEY ("project_template_id") REFERENCES "sentry_projecttemplate" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_projecttemplateoption" VALIDATE CONSTRAINT "sentry_projecttempla_project_template_id_23fa7e71_fk_sentry_pr";
CREATE INDEX CONCURRENTLY "sentry_projecttemplateoption_project_template_id_23fa7e71" ON "sentry_projecttemplateoption" ("project_template_id");
--
-- Create constraint unique_projecttemplate_name_per_org on model projecttemplate
--
CREATE UNIQUE INDEX CONCURRENTLY "unique_projecttemplate_name_per_org" ON "sentry_projecttemplate" ("name", "organization_id");
ALTER TABLE "sentry_projecttemplate" ADD CONSTRAINT "unique_projecttemplate_name_per_org" UNIQUE USING INDEX "unique_projecttemplate_name_per_org";
--
-- Alter unique_together for projecttemplateoption (1 constraint(s))
--
CREATE UNIQUE INDEX CONCURRENTLY "sentry_projecttemplateop_project_template_id_key_87ae3e18_uniq" ON "sentry_projecttemplateoption" ("project_template_id", "key");
ALTER TABLE "sentry_projecttemplateoption" ADD CONSTRAINT "sentry_projecttemplateop_project_template_id_key_87ae3e18_uniq" UNIQUE USING INDEX "sentry_projecttemplateop_project_template_id_key_87ae3e18_uniq";

Copy link

codecov bot commented Apr 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.00%. Comparing base (2489a4a) to head (8fdf00e).
Report is 3 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master   #69816       +/-   ##
===========================================
+ Coverage   51.67%   80.00%   +28.32%     
===========================================
  Files        6460     6494       +34     
  Lines      288653   290026     +1373     
  Branches    49758    49963      +205     
===========================================
+ Hits       149164   232028    +82864     
+ Misses     139074    57587    -81487     
+ Partials      415      411        -4     
Files Coverage Δ
src/sentry/backup/comparators.py 98.06% <ø> (+13.25%) ⬆️
src/sentry/models/__init__.py 100.00% <100.00%> (ø)
src/sentry/models/options/__init__.py 100.00% <100.00%> (ø)
...c/sentry/models/options/project_template_option.py 100.00% <100.00%> (ø)
src/sentry/models/project.py 97.35% <100.00%> (+49.01%) ⬆️
src/sentry/models/projecttemplate.py 100.00% <100.00%> (ø)
src/sentry/testutils/helpers/backups.py 99.70% <100.00%> (+2.35%) ⬆️

... and 2084 files with indirect coverage changes


class Meta:
app_label = "sentry"
db_table = "sentry_projecttemplate"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the names be unique per org?

@@ -245,6 +245,7 @@ class Project(Model, PendingDeletionMixin, OptionMixin, SnowflakeIdMixin):
# projects that were created before this field was present
# will have their first_event field set to date_added
first_event = models.DateTimeField(null=True)
project_template = FlexibleForeignKey("sentry.ProjectTemplate", null=True)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine, you can add an FK and the table in the same migration

@@ -245,6 +245,7 @@ class Project(Model, PendingDeletionMixin, OptionMixin, SnowflakeIdMixin):
# projects that were created before this field was present
# will have their first_event field set to date_added
first_event = models.DateTimeField(null=True)
project_template = FlexibleForeignKey("sentry.ProjectTemplate", null=True)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might make sense as just template, otherwise we'll be calling project.project_template a lot

# is a schema change, it's completely safe to run the operation after the code has deployed.
# Once deployed, run these manually via: https://develop.sentry.dev/database-migrations/#migration-deployment

is_post_deployment = True
Copy link
Member

@wedamija wedamija Apr 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Schema changes can't be post deployment, this will take down sentry, as per the comment (this is the only blocking comment)

from sentry.testutils.cases import TestCase


class ProjectTemplateOptionTest(TestCase):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's up to you, but you don't really need to test that the ORM works

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, agree. I just wanted to create a placeholder test for upcoming PRs :)

Copy link
Member

@wedamija wedamija left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

migration lgtm

primary_key=True, serialize=False
),
),
("key", models.CharField(max_length=64)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When querying this table will you always have project_template_id available? If not you'll need an index on key.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we'll always need the project template id to select the settings. I don't see us needing queries like "what are all the values an organization has set in all of their templates" or "Get me all the values for email prefixes in my templates" -- the design right now is more that we'll apply these settings to a project when getting those options, or when directly manipulating the template itself.

from sentry.db.models.fields import PickledObjectField


@region_silo_only_model
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
@region_silo_only_model
@region_silo_model

This is in the process of being renamed.


project_template = FlexibleForeignKey("sentry.ProjectTemplate", related_name="options")
key = models.CharField(max_length=64)
value = PickledObjectField()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we should be adding more pickled data to our database. Using a text field with JSON encoded payloads would be an improvement over the current option schema.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pickled field actually just uses json now, there was a project to convert it a while ago. Not sure why we never renamed it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a fixed set of values, or some sort of schema, we expect for this field, or is it completely freeform? If it's the former, we should at least at add a doc comment with some details (see UserOption for inspiration).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, it's freeform. We'll be aligning to a subset of project options and new template level options. Since it's not clear what options are going to be in the initial version (waiting on design for those areas), this is purposefully left open until we have a bit more clarity on those details.

I was pushing off that work until the model code has been merged, i kinda figured there'd be a lot of test updates / random things and wanted to keep the complexity of this pr as close to sql changes as possible.

src/sentry/models/projecttemplate.py Outdated Show resolved Hide resolved
Copy link
Contributor

This PR has a migration; here is the generated SQL for src/sentry/migrations/0710_project_template_models.py ()

--
-- Create model ProjectTemplate
--
CREATE TABLE "sentry_projecttemplate" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "date_updated" timestamp with time zone NOT NULL, "date_added" timestamp with time zone NULL, "name" varchar(200) NOT NULL, "organization_id" bigint NOT NULL);
ALTER TABLE "sentry_projecttemplate" ADD CONSTRAINT "sentry_projecttempla_organization_id_b072f5c6_fk_sentry_or" FOREIGN KEY ("organization_id") REFERENCES "sentry_organization" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_projecttemplate" VALIDATE CONSTRAINT "sentry_projecttempla_organization_id_b072f5c6_fk_sentry_or";
CREATE INDEX CONCURRENTLY "sentry_projecttemplate_organization_id_b072f5c6" ON "sentry_projecttemplate" ("organization_id");
--
-- Add field template to project
--
ALTER TABLE "sentry_project" ADD COLUMN "template_id" bigint NULL;
--
-- Create model ProjectTemplateOption
--
CREATE TABLE "sentry_projecttemplateoption" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "key" varchar(64) NOT NULL, "value" text NOT NULL, "project_template_id" bigint NOT NULL);
ALTER TABLE "sentry_project" ADD CONSTRAINT "sentry_project_template_id_67f0bec7_fk_sentry_pr" FOREIGN KEY ("template_id") REFERENCES "sentry_projecttemplate" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_project" VALIDATE CONSTRAINT "sentry_project_template_id_67f0bec7_fk_sentry_pr";
CREATE INDEX CONCURRENTLY "sentry_project_template_id_67f0bec7" ON "sentry_project" ("template_id");
ALTER TABLE "sentry_projecttemplateoption" ADD CONSTRAINT "sentry_projecttempla_project_template_id_23fa7e71_fk_sentry_pr" FOREIGN KEY ("project_template_id") REFERENCES "sentry_projecttemplate" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_projecttemplateoption" VALIDATE CONSTRAINT "sentry_projecttempla_project_template_id_23fa7e71_fk_sentry_pr";
CREATE INDEX CONCURRENTLY "sentry_projecttemplateoption_project_template_id_23fa7e71" ON "sentry_projecttemplateoption" ("project_template_id");
--
-- Create constraint unique_projecttemplate_name_per_org on model projecttemplate
--
CREATE UNIQUE INDEX CONCURRENTLY "unique_projecttemplate_name_per_org" ON "sentry_projecttemplate" ("name", "organization_id");
ALTER TABLE "sentry_projecttemplate" ADD CONSTRAINT "unique_projecttemplate_name_per_org" UNIQUE USING INDEX "unique_projecttemplate_name_per_org";
--
-- Alter unique_together for projecttemplateoption (1 constraint(s))
--
CREATE UNIQUE INDEX CONCURRENTLY "sentry_projecttemplateop_project_template_id_key_87ae3e18_uniq" ON "sentry_projecttemplateoption" ("project_template_id", "key");
ALTER TABLE "sentry_projecttemplateoption" ADD CONSTRAINT "sentry_projecttemplateop_project_template_id_key_87ae3e18_uniq" UNIQUE USING INDEX "sentry_projecttemplateop_project_template_id_key_87ae3e18_uniq";

Copy link
Contributor

github-actions bot commented May 2, 2024

This PR has a migration; here is the generated SQL for src/sentry/migrations/0714_project_template_models.py ()

--
-- Create model ProjectTemplate
--
CREATE TABLE "sentry_projecttemplate" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "date_updated" timestamp with time zone NOT NULL, "date_added" timestamp with time zone NULL, "name" varchar(200) NOT NULL, "organization_id" bigint NOT NULL);
ALTER TABLE "sentry_projecttemplate" ADD CONSTRAINT "sentry_projecttempla_organization_id_b072f5c6_fk_sentry_or" FOREIGN KEY ("organization_id") REFERENCES "sentry_organization" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_projecttemplate" VALIDATE CONSTRAINT "sentry_projecttempla_organization_id_b072f5c6_fk_sentry_or";
CREATE INDEX CONCURRENTLY "sentry_projecttemplate_organization_id_b072f5c6" ON "sentry_projecttemplate" ("organization_id");
--
-- Add field template to project
--
ALTER TABLE "sentry_project" ADD COLUMN "template_id" bigint NULL;
--
-- Create model ProjectTemplateOption
--
CREATE TABLE "sentry_projecttemplateoption" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "key" varchar(64) NOT NULL, "value" text NOT NULL, "project_template_id" bigint NOT NULL);
ALTER TABLE "sentry_project" ADD CONSTRAINT "sentry_project_template_id_67f0bec7_fk_sentry_pr" FOREIGN KEY ("template_id") REFERENCES "sentry_projecttemplate" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_project" VALIDATE CONSTRAINT "sentry_project_template_id_67f0bec7_fk_sentry_pr";
CREATE INDEX CONCURRENTLY "sentry_project_template_id_67f0bec7" ON "sentry_project" ("template_id");
ALTER TABLE "sentry_projecttemplateoption" ADD CONSTRAINT "sentry_projecttempla_project_template_id_23fa7e71_fk_sentry_pr" FOREIGN KEY ("project_template_id") REFERENCES "sentry_projecttemplate" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_projecttemplateoption" VALIDATE CONSTRAINT "sentry_projecttempla_project_template_id_23fa7e71_fk_sentry_pr";
CREATE INDEX CONCURRENTLY "sentry_projecttemplateoption_project_template_id_23fa7e71" ON "sentry_projecttemplateoption" ("project_template_id");
--
-- Create constraint unique_projecttemplate_name_per_org on model projecttemplate
--
CREATE UNIQUE INDEX CONCURRENTLY "unique_projecttemplate_name_per_org" ON "sentry_projecttemplate" ("name", "organization_id");
ALTER TABLE "sentry_projecttemplate" ADD CONSTRAINT "unique_projecttemplate_name_per_org" UNIQUE USING INDEX "unique_projecttemplate_name_per_org";
--
-- Alter unique_together for projecttemplateoption (1 constraint(s))
--
CREATE UNIQUE INDEX CONCURRENTLY "sentry_projecttemplateop_project_template_id_key_87ae3e18_uniq" ON "sentry_projecttemplateoption" ("project_template_id", "key");
ALTER TABLE "sentry_projecttemplateoption" ADD CONSTRAINT "sentry_projecttemplateop_project_template_id_key_87ae3e18_uniq" UNIQUE USING INDEX "sentry_projecttemplateop_project_template_id_key_87ae3e18_uniq";

…t_template' to 'template' on the project model
@saponifi3d saponifi3d force-pushed the jcallender/settings-templates-models branch from 43dcaad to 8fdf00e Compare May 7, 2024 22:54
Copy link
Contributor

github-actions bot commented May 7, 2024

This PR has a migration; here is the generated SQL for src/sentry/migrations/0715_project_template_models.py ()

--
-- Create model ProjectTemplate
--
CREATE TABLE "sentry_projecttemplate" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "date_updated" timestamp with time zone NOT NULL, "date_added" timestamp with time zone NULL, "name" varchar(200) NOT NULL, "organization_id" bigint NOT NULL);
ALTER TABLE "sentry_projecttemplate" ADD CONSTRAINT "sentry_projecttempla_organization_id_b072f5c6_fk_sentry_or" FOREIGN KEY ("organization_id") REFERENCES "sentry_organization" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_projecttemplate" VALIDATE CONSTRAINT "sentry_projecttempla_organization_id_b072f5c6_fk_sentry_or";
CREATE INDEX CONCURRENTLY "sentry_projecttemplate_organization_id_b072f5c6" ON "sentry_projecttemplate" ("organization_id");
--
-- Add field template to project
--
ALTER TABLE "sentry_project" ADD COLUMN "template_id" bigint NULL;
--
-- Create model ProjectTemplateOption
--
CREATE TABLE "sentry_projecttemplateoption" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "key" varchar(64) NOT NULL, "value" text NOT NULL, "project_template_id" bigint NOT NULL);
ALTER TABLE "sentry_project" ADD CONSTRAINT "sentry_project_template_id_67f0bec7_fk_sentry_pr" FOREIGN KEY ("template_id") REFERENCES "sentry_projecttemplate" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_project" VALIDATE CONSTRAINT "sentry_project_template_id_67f0bec7_fk_sentry_pr";
CREATE INDEX CONCURRENTLY "sentry_project_template_id_67f0bec7" ON "sentry_project" ("template_id");
ALTER TABLE "sentry_projecttemplateoption" ADD CONSTRAINT "sentry_projecttempla_project_template_id_23fa7e71_fk_sentry_pr" FOREIGN KEY ("project_template_id") REFERENCES "sentry_projecttemplate" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_projecttemplateoption" VALIDATE CONSTRAINT "sentry_projecttempla_project_template_id_23fa7e71_fk_sentry_pr";
CREATE INDEX CONCURRENTLY "sentry_projecttemplateoption_project_template_id_23fa7e71" ON "sentry_projecttemplateoption" ("project_template_id");
--
-- Create constraint unique_projecttemplate_name_per_org on model projecttemplate
--
CREATE UNIQUE INDEX CONCURRENTLY "unique_projecttemplate_name_per_org" ON "sentry_projecttemplate" ("name", "organization_id");
ALTER TABLE "sentry_projecttemplate" ADD CONSTRAINT "unique_projecttemplate_name_per_org" UNIQUE USING INDEX "unique_projecttemplate_name_per_org";
--
-- Alter unique_together for projecttemplateoption (1 constraint(s))
--
CREATE UNIQUE INDEX CONCURRENTLY "sentry_projecttemplateop_project_template_id_key_87ae3e18_uniq" ON "sentry_projecttemplateoption" ("project_template_id", "key");
ALTER TABLE "sentry_projecttemplateoption" ADD CONSTRAINT "sentry_projecttemplateop_project_template_id_key_87ae3e18_uniq" UNIQUE USING INDEX "sentry_projecttemplateop_project_template_id_key_87ae3e18_uniq";

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Scope: Backend Automatically applied to PRs that change backend components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants