Skip to content

Commit

Permalink
fix!: update Stable/3.2.x to work with django 3.2 tests (#710)
Browse files Browse the repository at this point in the history 
* performance files

* test_benchmark

* performance testing changes

* changes in benchmark performance for prod

* changes to number of runs

* adding comments

* linting changes

* 3.2 changes

* adding version change

* lint changes and resmoving performance changes

* version changes

* chore: fix release build (#659)

* chore: fix release build

Fix release build by migrating to secret manager secrets and use templated kokoro configs for docs/ and release/

* fix: fix config names

* chore: add populate secrets script

* docs: fix license

* chore: preserve original year

* chore: revert years in manually committed files

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md

* chore: update lockfile

* chore: add .kokoro/docker directory

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

* feat: Added support for check constraint (#679)

* feat: Added support for check constraint

* fix: change decimal out of scale ProgramingError to ValueError

* fix: skip check_constraints tests when running on emmulator

* fix: remove check constraint for emulator

* docs: update docs to show decimal field support and check constraints but no support for unsigned data type (#683)

* test: Performance Testing (#675)

* performance files

* test_benchmark

* performance testing changes

* changes in benchmark performance for prod

* changes to number of runs

* adding comments

* linting changes

* changes for 3.2

* Revert "changes for 3.2"

This reverts commit 488035c.

* adding licence

* chore: release 2.2.1b2 (#685)

Release-As: 2.2.1b2

* chore: release 2.2.1b2 (#687)

* chore: release 2.2.1b2

* Updated CHANGELOG.md

Corrected the change log msg.

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: Vikash Singh <3116482+vi3k6i5@users.noreply.github.com>

* fix: Bump version number after 2.2.1b2 release (#688)

* chor: Update repo to say beta release instead of alpha (#691)

* fix: Bump version number after 2.2.1b2 release

* Update setup.py

Current release is beta so updating the same in setup.py

* chore: release 2.2.1b3 (#693)

Release-As: 2.2.1b3

* chore: release 2.2.1b3 (#694)

* chore: release 2.2.1b3

* Updated CHANGELOG.md

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: Vikash Singh <3116482+vi3k6i5@users.noreply.github.com>

* fix: Bump version number after 2.2.1b3 release (#696)

* chore: release 2.2.1b3

Release-As: 2.2.1b3

* fix: Bump version number after 2.2.1b3 release

* fix: add test samples script and related kokoro files

* fix: correct repo name from python-spanner to python-spanner-django

* fix: correct license from Apache to BSD style

* docs: lint fix for samples (#697)

* Docs: fix changelog link and sample examples. (#700)

* docs: update docs to show decimal field support and check constraints but no support for unsigned data type

* docs: linked changelog correctly

* docs: fix doc links for sample examples

* fix: skip test cursor_executemany_with_empty_params_list as spanner support is not there

* docs: update dbapi location in overview asset file (#702)

* chore: migrate to main branch (#706)

* chore: migrate to main branch

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* use latest post processor image

* remove obsolete replacements in owlbot.py

* update post processor image

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>

* fix: added fixes for latest feature changes in django 3.2

* fix: fixes for running tests for django3.2

* fix: change django repo path

* test: test fixes for order by nulls first and last

* docs: fix readme link target

* test: set default auto field type in test settings

* fix: update features to skip tests that are not support by spanner

* fix: remove choices module from django3.2 as it has been removed from django 3.2

Co-authored-by: Astha Mohta <asthamohta@gmail.com>
Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com>
Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: Dan Lee <71398022+dandhlee@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
  • Loading branch information
@vi3k6i5 @asthamohta
@busunkim96 @gcf-owl-bot @release-please @dandhlee @parthea
8 people committed Sep 21, 2021
1 parent 736a84a commit b163ed2
Show file tree
Hide file tree
Showing 19 changed files with 531 additions and 204 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/django_tests_against_emulator0.yml
View file
Expand Up @@ -29,4 +29,4 @@ jobs:
GOOGLE_CLOUD_TESTS_CREATE_SPANNER_INSTANCE: true
RUNNING_SPANNER_BACKEND_TESTS: 1
SPANNER_TEST_INSTANCE: google-cloud-django-backend-tests
DJANGO_TEST_APPS: admin_changelist admin_ordering aggregation choices distinct_on_fields expressions_window fixtures_model_package datetimes custom_methods generic_inline_admin field_defaults datatypes empty m2o_recursive many_to_one_null migration_test_data_persistence admin_docs invalid_models_tests migrate_signals model_forms.test_uuid model_forms.test_modelchoicefield syndication_tests view_tests update test_utils select_related_onetoone sessions_tests
DJANGO_TEST_APPS: admin_changelist admin_ordering aggregation distinct_on_fields expressions_window fixtures_model_package datetimes custom_methods generic_inline_admin field_defaults datatypes empty m2o_recursive many_to_one_null migration_test_data_persistence admin_docs invalid_models_tests migrate_signals model_forms.test_uuid model_forms.test_modelchoicefield syndication_tests view_tests update test_utils select_related_onetoone sessions_tests
117 changes: 5 additions & 112 deletions README.rst
View file
Expand Up @@ -253,118 +253,11 @@ Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms. See the `Code
of Conduct <https://github.com/googleapis/python-spanner-django/blob/main/CODE_OF_CONDUCT.md>`_ for more information.

Current limitations
-------------------

``AutoField`` generates random IDs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Spanner doesn't have support for auto-generating primary key values.
Therefore, ``django-google-spanner`` monkey-patches ``AutoField`` to generate a
random UUID4. It generates a default using ``Field``'s ``default`` option which
means ``AutoField``\ s will have a value when a model instance is created. For
example:

::

>>> ExampleModel()
>>> ExampleModel.pk
4229421414948291880

To avoid
`hotspotting <https://cloud.google.com/spanner/docs/schema-design#uuid_primary_key>`__,
these IDs are not monotonically increasing. This means that sorting
models by ID isn't guaranteed to return them in the order in which they
were created.

``ForeignKey`` constraints aren't created (`#313 <https://github.com/googleapis/python-spanner-django/issues/313>`__)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Spanner does not support ``ON DELETE CASCADE`` when creating foreign-key
constraints, so this is not supported in ``django-google-spanner``.

``Unsigned`` datatypes are not supported
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Spanner does not support ``Unsigned`` datatypes so `PositiveIntegerField
<https://docs.djangoproject.com/en/stable/ref/models/fields/#positiveintegerfield>`__
and `PositiveSmallIntegerField
<https://docs.djangoproject.com/en/3.2/ref/models/fields/#positivesmallintegerfield>`__
are both stored as `Integer type
<https://cloud.google.com/spanner/docs/data-types#integer_type>`__
.

``Meta.order_with_respect_to`` model option isn't supported
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This feature uses a column name that starts with an underscore
(``_order``) which Spanner doesn't allow.

Random ``QuerySet`` ordering isn't supported
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Spanner does not support it and will throw an exception. For example:

::

>>> ExampleModel.objects.order_by('?')
...
django.db.utils.ProgrammingError: 400 Function not found: RANDOM ... FROM
example_model ORDER BY RANDOM() ASC

Schema migrations
~~~~~~~~~~~~~~~~~

There are some limitations on schema changes to consider:

- No support for renaming tables and columns;
- A column's type can't be changed;
- A table's primary key can't be altered.

``DurationField`` arithmetic doesn't work with ``DateField`` values (`#253 <https://github.com/googleapis/python-spanner-django/issues/253>`__)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Spanner requires using different functions for arithmetic depending on
the column type:

- ``TIMESTAMP`` columns (``DateTimeField``) require ``TIMESTAMP_ADD``
or ``TIMESTAMP_SUB``
- ``DATE`` columns (``DateField``) require ``DATE_ADD`` or ``DATE_SUB``

Django does not provide ways to determine which database function to
use. ``DatabaseOperations.combine_duration_expression()`` arbitrarily uses
``TIMESTAMP_ADD`` and ``TIMESTAMP_SUB``. Therefore, if you use a
``DateField`` in a ``DurationField`` expression, you'll likely see an error
such as:

::

"No matching signature for function TIMESTAMP\_ADD for argument types:
DATE, INTERVAL INT64 DATE\_TIME\_PART."

Computations that yield FLOAT64 values cannot be assigned to INT64 columns
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Spanner does not support this (`#331
<https://github.com/googleapis/python-spanner-django/issues/331>`__) and will
throw an error:

::

>>> ExampleModel.objects.update(integer=F('integer') / 2)
...
django.db.utils.ProgrammingError: 400 Value of type FLOAT64 cannot be
assigned to integer, which has type INT64 [at 1:46]\nUPDATE
example_model SET integer = (example_model.integer /...

Addition with null values crash
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Additions cannot include ``None`` values. For example:
LIMITATIONS
-----------------

::
Spanner has certain limitations of it's own and a full set of limitations are documented over `here <https://cloud.google.com/spanner/quotas#schema_limits>`_
It is recommended that you go through that list.

>>> Book.objects.annotate(adjusted_rating=F('rating') + None)
...
google.api_core.exceptions.InvalidArgument: 400 Operands of + cannot be literal
NULL ...
Django spanner has a set of limitations as well, please go through the `list <https://googleapis.dev/python/django-google-spanner/latest/limitations.html>`_.
15 changes: 14 additions & 1 deletion django_spanner/__init__.py
View file
Expand Up @@ -12,7 +12,12 @@
from uuid import uuid4

import pkg_resources
from django.db.models.fields import AutoField, Field
from django.db.models.fields import (
AutoField,
SmallAutoField,
BigAutoField,
Field,
)

# Monkey-patch google.DatetimeWithNanoseconds's __eq__ compare against
# datetime.datetime.
Expand Down Expand Up @@ -45,6 +50,14 @@ def autofield_init(self, *args, **kwargs):


AutoField.__init__ = autofield_init
SmallAutoField.__init__ = autofield_init
BigAutoField.__init__ = autofield_init
AutoField.db_returning = False
SmallAutoField.db_returning = False
BigAutoField.db_returning = False
AutoField.validators = []
SmallAutoField.validators = []
BigAutoField.validators = []

old_datetimewithnanoseconds_eq = getattr(
DatetimeWithNanoseconds, "__eq__", None
Expand Down
7 changes: 7 additions & 0 deletions django_spanner/base.py
View file
Expand Up @@ -45,6 +45,7 @@ class DatabaseWrapper(BaseDatabaseWrapper):
"GenericIPAddressField": "STRING(39)",
"NullBooleanField": "BOOL",
"OneToOneField": "INT64",
"PositiveBigIntegerField": "INT64",
"PositiveIntegerField": "INT64",
"PositiveSmallIntegerField": "INT64",
"SlugField": "STRING(%(max_length)s)",
Expand Down Expand Up @@ -96,6 +97,12 @@ class DatabaseWrapper(BaseDatabaseWrapper):
"iendswith": "",
}

data_type_check_constraints = {
"PositiveBigIntegerField": "%(column)s >= 0",
"PositiveIntegerField": "%(column)s >= 0",
"PositiveSmallIntegerField": "%(column)s >= 0",
}

Database = spanner_dbapi
SchemaEditorClass = DatabaseSchemaEditor
creation_class = DatabaseCreation
Expand Down

0 comments on commit b163ed2

Please sign in to comment.