Skip to content
Snippets Groups Projects
Commit 98da8d0b authored by Michael Fladischer's avatar Michael Fladischer
Browse files

New upstream version 2.0.5

parent 470910a0
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 569 additions and 195 deletions
.circleci/config.yml
+ 79
− 13
View file @ 98da8d0b
......@@ -6,7 +6,7 @@ executors:
version:
type: string
docker:
- image: python:<< parameters.version >>-alpine
- image: python:<< parameters.version >>-buster
- image: postgres:11.0
environment:
POSTGRES_DB: 'psqlextra'
......@@ -22,11 +22,11 @@ commands:
steps:
- run:
name: Install packages
command: apk add postgresql-libs gcc musl-dev postgresql-dev git
command: apt-get update && apt-get install -y --no-install-recommends postgresql-client-11 libpq-dev build-essential git
- run:
name: Install Python packages
command: pip install --progress-bar off .[<< parameters.extra >>]
command: pip install --progress-bar off '.[<< parameters.extra >>]'
run-tests:
parameters:
......@@ -78,7 +78,7 @@ jobs:
extra: test
- run-tests:
pyversion: 38
djversions: 20,21,22,30,31,32
djversions: 20,21,22,30,31,32,40
test-python39:
executor:
......@@ -90,7 +90,7 @@ jobs:
extra: test
- run-tests:
pyversion: 39
djversions: 21,22,30,31,32
djversions: 21,22,30,31,32,40
test-python310:
executor:
......@@ -102,7 +102,7 @@ jobs:
extra: test
- run-tests:
pyversion: 310
djversions: 21,22,30,31,32
djversions: 21,22,30,31,32,40
- store_test_results:
path: reports
- run:
......@@ -121,14 +121,80 @@ jobs:
name: Verify
command: python setup.py verify
publish:
executor:
name: python
version: "3.9"
steps:
- checkout
- install-dependencies:
extra: publish
- run:
name: Set version number
command: echo "__version__ = \"${CIRCLE_TAG:1}\"" > psqlextra/_version.py
- run:
name: Build package
command: python -m build
- run:
name: Publish package
command: >
python -m twine upload
--username "${PYPI_REPO_USERNAME}"
--password "${PYPI_REPO_PASSWORD}"
--verbose
--non-interactive
--disable-progress-bar
dist/*
workflows:
version: 2
build:
jobs:
- test-python36
- test-python37
- test-python38
- test-python39
- test-python310
- analysis
- test-python36:
filters:
tags:
only: /.*/
branches:
only: /.*/
- test-python37:
filters:
tags:
only: /.*/
branches:
only: /.*/
- test-python38:
filters:
tags:
only: /.*/
branches:
only: /.*/
- test-python39:
filters:
tags:
only: /.*/
branches:
only: /.*/
- test-python310:
filters:
tags:
only: /.*/
branches:
only: /.*/
- analysis:
filters:
tags:
only: /.*/
branches:
only: /.*/
- publish:
requires:
- test-python36
- test-python37
- test-python38
- test-python39
- test-python310
- analysis
filters:
tags:
only: /^v.*/
branches:
ignore: /.*/
README.md
+ 2
− 2
View file @ 98da8d0b
......@@ -8,7 +8,7 @@
| :memo: | **License** | [![License](https://img.shields.io/:license-mit-blue.svg)](http://doge.mit-license.org) |
| :package: | **PyPi** | [![PyPi](https://badge.fury.io/py/django-postgres-extra.svg)](https://pypi.python.org/pypi/django-postgres-extra) |
| :four_leaf_clover: | **Code coverage** | [![Coverage Status](https://coveralls.io/repos/github/SectorLabs/django-postgres-extra/badge.svg?branch=coveralls)](https://coveralls.io/github/SectorLabs/django-postgres-extra?branch=master) |
| <img src="https://cdn.iconscout.com/icon/free/png-256/django-1-282754.png" width="22px" height="22px" align="center" /> | **Django Versions** | 2.0, 2.1, 2.2, 3.0, 3.1, 3.2 |
| <img src="https://cdn.iconscout.com/icon/free/png-256/django-1-282754.png" width="22px" height="22px" align="center" /> | **Django Versions** | 2.0, 2.1, 2.2, 3.0, 3.1, 3.2, 4.0 |
| <img src="http://www.iconarchive.com/download/i73027/cornmanthe3rd/plex/Other-python.ico" width="22px" height="22px" align="center" /> | **Python Versions** | 3.6, 3.7, 3.8, 3.9, 3.10 |
| :book: | **Documentation** | [Read The Docs](https://django-postgres-extra.readthedocs.io/en/master/) |
| :warning: | **Upgrade** | [Upgrade from v1.x](https://django-postgres-extra.readthedocs.io/en/master/major_releases.html#new-features)
......@@ -59,7 +59,7 @@ With seamless we mean that any features we add will work truly seamlessly. You s
### Prerequisites
* PostgreSQL 10 or newer.
* Django 2.0 or newer (including 3.x).
* Django 2.0 or newer (including 3.x, 4.x).
* Python 3.6 or newer.
### Getting started
......
docs/source/api_reference.rst
+ 6
− 0
View file @ 98da8d0b
......@@ -37,6 +37,12 @@ API Reference
.. autoclass:: ConditionalUniqueIndex
.. autoclass:: CaseInsensitiveUniqueIndex
.. automodule:: psqlextra.partitioning
:members:
.. automodule:: psqlextra.backend.migrations.operations
:members:
.. automodule:: psqlextra.types
:members:
:undoc-members:
......
docs/source/conflict_handling.rst
+ 1
− 1
View file @ 98da8d0b
......@@ -219,7 +219,7 @@ This is preferable when the data you're about to insert is the same as the one t
# obj2 is none! object alreaddy exists
obj2 = MyModel.objects.on_conflict(['name'], ConflictAction.NOTHING).insert(name="me")
This applies to both :meth:`~psqlextra.query.PostgresQuerySet.insert` and :meth:`~psqlextra.query.PostgresQuerySet.bulk_insert`
This applies all methods: :meth:`~psqlextra.query.PostgresQuerySet.insert`, :meth:`~psqlextra.query.PostgresQuerySet.insert_and_get`, :meth:`~psqlextra.query.PostgresQuerySet.bulk_insert`
Bulk
......
docs/source/table_partitioning.rst
+ 188
− 103
View file @ 98da8d0b
......@@ -80,18 +80,131 @@ This will generate a migration that creates the partitioned table with a default
Do not use the standard ``python manage.py makemigrations`` command for partitioned models. Django will issue a standard :class:`~django:django.db.migrations.operations.CreateModel` operation. Doing this will not create a partitioned table and all subsequent operations will fail.
Adding/removing partitions manually
-----------------------------------
Automatically managing partitions
---------------------------------
Postgres does not have support for automatically creating new partitions as needed. Therefore, one must manually add new partitions. Depending on the partitioning method you have chosen, the partition has to be created differently.
The ``python manage.py pgpartition`` command can help you automatically create new partitions ahead of time and delete old ones for time-based partitioning.
Partitions are tables. Each partition must be given a unique name. :class:`~psqlextra.models.PostgresPartitionedModel` does not require you to create a model for each partition because you are not supposed to query partitions directly.
You can run this command manually as needed, schedule to run it periodically or run it every time you release a new version of your app.
.. warning::
We DO NOT recommend that you set up this command to automatically delete partitions without manual review.
Specify ``--skip-delete`` to not delete partitions automatically. Run the command manually periodically without the ``--yes`` flag to review partitions to be deleted.
Command-line options
********************
==================== ============= ================ ==================================================================================================== === === === === === ===
Long flag Short flag Default Description
==================== ============= ================ ==================================================================================================== === === === === === ===
``--yes`` ``-y`` ``False`` Specifies yes to all questions. You will NOT be asked for confirmation before partition deletion.
``--using`` ``-u`` ``'default'`` Name of the database connection to use.
``--skip-create`` ``False`` Whether to skip creating partitions.
``--skip-delete`` ``False`` Whether to skip deleting partitions.
==================== ============= ================ ==================================================================================================== === === === === === ===
Configuration
*************
In order to use the command, you have to declare an instance of :class:`psqlextra.partitioning.PostgresPartitioningManager` and set ``PSQLEXTRA_PARTITIONING_MANAGER`` to a string with the import path to your instance of :class:`psqlextra.partitioning.PostgresPartitioningManager`.
For example:
.. code-block:: python
# myapp/partitioning.py
from psqlextra.partitioning import PostgresPartitioningManager
manager = PostgresPartitioningManager(...)
# myapp/settings.py
PSQLEXTRA_PARTITIONING_MANAGER = 'myapp.partitioning.manager'
Time-based partitioning
~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: python
from dateutil.relativedelta import relativedelta
from psqlextra.partitioning import (
PostgresPartitioningManager,
PostgresCurrentTimePartitioningStrategy,
PostgresTimePartitionSize,
partition_by_current_time,
)
from psqlextra.partitioning.config import PostgresPartitioningConfig
manager = PostgresPartitioningManager([
# 3 partitions ahead, each partition is one month
# delete partitions older than 6 months
# partitions will be named `[table_name]_[year]_[3-letter month name]`.
PostgresPartitioningConfig(
model=MyPartitionedModel,
strategy=PostgresCurrentTimePartitioningStrategy(
size=PostgresTimePartitionSize(months=1),
count=3,
max_age=relativedelta(months=6),
),
),
# 6 partitions ahead, each partition is two weeks
# delete partitions older than 8 months
# partitions will be named `[table_name]_[year]_week_[week number]`.
PostgresPartitioningConfig(
model=MyPartitionedModel,
strategy=PostgresCurrentTimePartitioningStrategy(
size=PostgresTimePartitionSize(weeks=2),
count=6,
max_age=relativedelta(months=8),
),
),
# 12 partitions ahead, each partition is 5 days
# old partitions are never deleted, `max_age` is not set
# partitions will be named `[table_name]_[year]_[month]_[month day number]`.
PostgresPartitioningConfig(
model=MyPartitionedModel,
strategy=PostgresCurrentTimePartitioningStrategy(
size=PostgresTimePartitionSize(days=5),
count=12,
),
),
])
Changing a time partitioning strategy
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When switching partitioning strategies, you might encounter the problem that partitions for part of a particular range already exist.
In order to combat this, you can use the :class:`psqlextra.partitioning.PostgresTimePartitioningStrategy` and specify the `start_datetime` parameter. As a result, no partitions will be created before the given date/time.
Custom strategy
~~~~~~~~~~~~~~~
You can create a custom partitioning strategy by implementing the :class:`psqlextra.partitioning.PostgresPartitioningStrategy` interface.
You can look at :class:`psqlextra.partitioning.PostgresCurrentTimePartitioningStrategy` as an example.
Manually managing partitions
----------------------------
If you are using list or has partitioning, you most likely have a fixed amount of partitions that can be created up front using migrations or using the schema editor.
Using migration operations
**************************
Adding a range partition
~~~~~~~~~~~~~~~~~~~~~~~~
Use the :class:`~psqlextra.backend.migrations.operations.PostgresAddRangePartition` operation to add a new range partition. Only use this operation when your partitioned model uses the :attr:`psqlextra.types.PostgresPartitioningMethod.RANGE`.
Use the :class:`~psqlextra.backend.migrations.operations.PostgresAddRangePartition` operation to add a new range partition. Only use this operation when your partitioned model uses :attr:`psqlextra.types.PostgresPartitioningMethod.RANGE`.
.. code-block:: python
......@@ -113,7 +226,7 @@ Use the :class:`~psqlextra.backend.migrations.operations.PostgresAddRangePartiti
Adding a list partition
~~~~~~~~~~~~~~~~~~~~~~~
Use the :class:`~psqlextra.backend.migrations.operations.PostgresAddListPartition` operation to add a new list partition. Only use this operation when your partitioned model uses the :attr:`psqlextra.types.PostgresPartitioningMethod.LIST`.
Use the :class:`~psqlextra.backend.migrations.operations.PostgresAddListPartition` operation to add a new list partition. Only use this operation when your partitioned model uses :attr:`psqlextra.types.PostgresPartitioningMethod.LIST`.
.. code-block:: python
......@@ -131,12 +244,36 @@ Use the :class:`~psqlextra.backend.migrations.operations.PostgresAddListPartitio
]
Adding a hash partition
~~~~~~~~~~~~~~~~~~~~~~~
Use the :class:`~psqlextra.backend.migrations.operations.PostgresAddHashPartition` operation to add a new list partition. Only use this operation when your partitioned model uses :attr:`psqlextra.types.PostgresPartitioningMethod.HASH`.
.. code-block:: python
from django.db import migrations, models
from psqlextra.backend.migrations.operations import PostgresAddHashPartition
class Migration(migrations.Migration):
operations = [
PostgresAddHashPartition(
model_name="mypartitionedmodel",
name="pt1",
modulus=3,
remainder=1,
),
]
Adding a default partition
~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :class:`~psqlextra.backend.migrations.operations.PostgresAddDefaultPartition` operation to add a new default partition. A default partition is the partition where records get saved that couldn't fit in any other partition.
Use the :class:`~psqlextra.backend.migrations.operations.PostgresAddDefaultPartition` operation to add a new list partition.
Note that you can only have one default partition per partitioned table/model. An error will be thrown if you try to create a second default partition.
Note that you can only have one default partition per partitioned table/model.
If you used ``python manage.py pgmakemigrations`` to generate a migration for your newly created partitioned model, you do not need this operation. This operation is added automatically when you create a new partitioned model.
.. code-block:: python
......@@ -158,6 +295,12 @@ Deleting a default partition
Use the :class:`~psqlextra.backend.migrations.operations.PostgresDeleteDefaultPartition` operation to delete an existing default partition.
.. warning::
Deleting the default partition and leaving your model without a default partition can be dangerous. Rows that do not fit in any other partition will fail to be inserted.
.. code-block:: python
from django.db import migrations, models
......@@ -176,7 +319,7 @@ Use the :class:`~psqlextra.backend.migrations.operations.PostgresDeleteDefaultPa
Deleting a range partition
~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :class:`psqlextra.backend.migrations.operations.PostgresDeleteRangePartition` operation to delete an existing range partition.
Use the :class:`psqlextra.backend.migrations.operations.PostgresDeleteRangePartition` operation to delete an existing range partition. Only use this operation when your partitioned model uses :attr:`psqlextra.types.PostgresPartitioningMethod.RANGE`.
.. code-block:: python
......@@ -196,7 +339,7 @@ Use the :class:`psqlextra.backend.migrations.operations.PostgresDeleteRangeParti
Deleting a list partition
~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :class:`~psqlextra.backend.migrations.operations.PostgresDeleteListPartition` operation to delete an existing list partition.
Use the :class:`psqlextra.backend.migrations.operations.PostgresDeleteListPartition` operation to delete an existing range partition. Only use this operation when your partitioned model uses :attr:`psqlextra.types.PostgresPartitioningMethod.LIST`.
.. code-block:: python
......@@ -213,6 +356,26 @@ Use the :class:`~psqlextra.backend.migrations.operations.PostgresDeleteListParti
]
Deleting a hash partition
~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :class:`psqlextra.backend.migrations.operations.PostgresDeleteHashPartition` operation to delete an existing range partition. Only use this operation when your partitioned model uses :attr:`psqlextra.types.PostgresPartitioningMethod.HASH`.
.. code-block:: python
from django.db import migrations, models
from psqlextra.backend.migrations.operations import PostgresDeleteHashPartition
class Migration(migrations.Migration):
operations = [
PostgresDeleteHashPartition(
model_name="mypartitionedmodel",
name="pt1",
),
]
Using the schema editor
***********************
......@@ -248,120 +411,42 @@ Adding a list partition
)
Adding a default partition
~~~~~~~~~~~~~~~~~~~~~~~~~~
Adding a hash partition
~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: python
from django.db import connection
connection.schema_editor().add_default_partition(
connection.schema_editor().add_hash_partition(
model=MyPartitionedModel,
name="default",
name="pt1",
modulus=3,
remainder=1,
)
Deleting a partition
~~~~~~~~~~~~~~~~~~~~
Adding a default partition
~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: python
from django.db import connection
connection.schema_editor().delete_partition(
connection.schema_editor().add_default_partition(
model=MyPartitionedModel,
name="default",
)
Adding/removing partitions automatically
----------------------------------------
:class:`psqlextra.partitioning.PostgresPartitioningManager` an experimental helper class that can be called periodically to automatically create new partitions if you're using range partitioning.
.. note::
There is currently no scheduler or command to automatically create new partitions. You'll have to run this function in your own cron jobs.
The auto partitioner supports automatically creating yearly, monthly, weekly or daily partitions. Use the ``count`` parameter to configure how many partitions it should create ahead.
Partitioning strategies
***********************
Time-based partitioning
~~~~~~~~~~~~~~~~~~~~~~~
Deleting a partition
~~~~~~~~~~~~~~~~~~~~
.. code-block:: python
from dateutil.relativedelta import relativedelta
from psqlextra.partitioning import (
PostgresPartitioningManager,
PostgresCurrentTimePartitioningStrategy,
PostgresTimePartitionSize,
partition_by_current_time,
)
manager = PostgresPartitioningManager([
# 3 partitions ahead, each partition is one month
# delete partitions older than 6 months
# partitions will be named `[table_name]_[year]_[3-letter month name]`.
PostgresPartitioningConfig(
model=MyPartitionedModel,
strategy=PostgresCurrentTimePartitioningStrategy(
size=PostgresTimePartitionSize(months=1),
count=3,
max_age=relativedelta(months=6),
),
),
# 6 partitions ahead, each partition is two weeks
# delete partitions older than 8 months
# partitions will be named `[table_name]_[year]_week_[week number]`.
PostgresPartitioningConfig(
model=MyPartitionedModel,
strategy=PostgresCurrentTimePartitioningStrategy(
size=PostgresTimePartitionSize(weeks=2),
count=6,
max_age=relativedelta(months=8),
),
),
# 12 partitions ahead, each partition is 5 days
# old partitions are never deleted, `max_age` is not set
# partitions will be named `[table_name]_[year]_[month]_[month day number]`.
PostgresPartitioningConfig(
model=MyPartitionedModel,
strategy=PostgresCurrentTimePartitioningStrategy(
size=PostgresTimePartitionSize(wdyas=5),
count=12,
),
),
])
from django.db import connection
# these are the default arguments
partioning_plan = manager.plan(
skip_create=False,
skip_delete=False,
using='default'
connection.schema_editor().delete_partition(
model=MyPartitionedModel,
name="default",
)
# prints a list of partitions to be created/deleted
partitioning_plan.print()
# apply the plan
partitioning_plan.apply(using='default');
Custom strategy
~~~~~~~~~~~~~~~
You can create a custom partitioning strategy by implementing the :class:`psqlextra.partitioning.PostgresPartitioningStrategy` interface.
You can look at :class:`psqlextra.partitioning.PostgresCurrentTimePartitioningStrategy` as an example.
Switching partitioning strategies
*********************************
When switching partitioning strategies, you might encounter the problem that partitions for part of a particular range already exist. In order to combat this, you can use the :class:`psqlextra.partitioning.PostgresTimePartitioningStrategy` and specify the `start_datetime` parameter. As a result, no partitions will be created before the given date/time.
psqlextra/__init__.py
+ 11
− 0
View file @ 98da8d0b
import django
from ._version import __version__
if django.VERSION < (3, 2): # pragma: no cover
default_app_config = "psqlextra.apps.PostgresExtraAppConfig"
__all__ = [
"default_app_config",
"__version__",
]
else:
__all__ = [
"__version__",
]
psqlextra/_version.py 0 → 100644
+ 1
− 0
View file @ 98da8d0b
__version__ = "2.0.4"
psqlextra/apps.py
+ 3
− 0
View file @ 98da8d0b
......@@ -4,3 +4,6 @@ from django.apps import AppConfig
class PostgresExtraAppConfig(AppConfig):
name = "psqlextra"
verbose_name = "PostgreSQL Extra"
def ready(self) -> None:
from .lookups import InValuesLookup # noqa
psqlextra/backend/migrations/patched_autodetector.py
+ 108
− 42
View file @ 98da8d0b
from contextlib import contextmanager
from unittest import mock
import django
from django.db.migrations import (
AddField,
AlterField,
......@@ -11,7 +13,6 @@ from django.db.migrations import (
)
from django.db.migrations.autodetector import MigrationAutodetector
from django.db.migrations.operations.base import Operation
from django.db.models import Model
from psqlextra.models import (
PostgresMaterializedViewModel,
......@@ -21,6 +22,11 @@ from psqlextra.models import (
from psqlextra.types import PostgresPartitioningMethod
from . import operations
from .state import (
PostgresMaterializedViewModelState,
PostgresPartitionedModelState,
PostgresViewModelState,
)
# original `MigrationAutodetector.add_operation`
# function, saved here so the patched version can
......@@ -88,12 +94,26 @@ class AddOperationHandler:
actually applying it.
"""
model = self.autodetector.new_apps.get_model(
self.app_label, operation.model_name
)
if django.VERSION >= (4, 0):
model_identifier = (self.app_label, operation.model_name.lower())
model_state = (
self.autodetector.to_state.models.get(model_identifier)
or self.autodetector.from_state.models[model_identifier]
)
if issubclass(model, PostgresViewModel):
return self.add(operations.ApplyState(state_operation=operation))
if isinstance(model_state, PostgresViewModelState):
return self.add(
operations.ApplyState(state_operation=operation)
)
else:
model = self.autodetector.new_apps.get_model(
self.app_label, operation.model_name
)
if issubclass(model, PostgresViewModel):
return self.add(
operations.ApplyState(state_operation=operation)
)
return self.add(operation)
......@@ -101,16 +121,28 @@ class AddOperationHandler:
"""Adds the specified :see:CreateModel operation to the list of
operations to execute in the migration."""
model = self.autodetector.new_apps.get_model(
self.app_label, operation.name
)
if django.VERSION >= (4, 0):
model_state = self.autodetector.to_state.models[
self.app_label, operation.name.lower()
]
if isinstance(model_state, PostgresPartitionedModelState):
return self.add_create_partitioned_model(operation)
elif isinstance(model_state, PostgresMaterializedViewModelState):
return self.add_create_materialized_view_model(operation)
elif isinstance(model_state, PostgresViewModelState):
return self.add_create_view_model(operation)
else:
model = self.autodetector.new_apps.get_model(
self.app_label, operation.name
)
if issubclass(model, PostgresPartitionedModel):
return self.add_create_partitioned_model(model, operation)
elif issubclass(model, PostgresMaterializedViewModel):
return self.add_create_materialized_view_model(model, operation)
elif issubclass(model, PostgresViewModel):
return self.add_create_view_model(model, operation)
if issubclass(model, PostgresPartitionedModel):
return self.add_create_partitioned_model(operation)
elif issubclass(model, PostgresMaterializedViewModel):
return self.add_create_materialized_view_model(operation)
elif issubclass(model, PostgresViewModel):
return self.add_create_view_model(operation)
return self.add(operation)
......@@ -118,32 +150,52 @@ class AddOperationHandler:
"""Adds the specified :see:Deletemodel operation to the list of
operations to execute in the migration."""
model = self.autodetector.old_apps.get_model(
self.app_label, operation.name
)
if django.VERSION >= (4, 0):
model_state = self.autodetector.from_state.models[
self.app_label, operation.name.lower()
]
if isinstance(model_state, PostgresPartitionedModelState):
return self.add_delete_partitioned_model(operation)
elif isinstance(model_state, PostgresMaterializedViewModelState):
return self.add_delete_materialized_view_model(operation)
elif isinstance(model_state, PostgresViewModelState):
return self.add_delete_view_model(operation)
else:
model = self.autodetector.old_apps.get_model(
self.app_label, operation.name
)
if issubclass(model, PostgresPartitionedModel):
return self.add_delete_partitioned_model(model, operation)
elif issubclass(model, PostgresMaterializedViewModel):
return self.add_delete_materialized_view_model(model, operation)
elif issubclass(model, PostgresViewModel):
return self.add_delete_view_model(model, operation)
if issubclass(model, PostgresPartitionedModel):
return self.add_delete_partitioned_model(operation)
elif issubclass(model, PostgresMaterializedViewModel):
return self.add_delete_materialized_view_model(operation)
elif issubclass(model, PostgresViewModel):
return self.add_delete_view_model(operation)
return self.add(operation)
def add_create_partitioned_model(
self, model: Model, operation: CreateModel
):
def add_create_partitioned_model(self, operation: CreateModel):
"""Adds a :see:PostgresCreatePartitionedModel operation to the list of
operations to execute in the migration."""
partitioning_options = model._partitioning_meta.original_attrs
if django.VERSION >= (4, 0):
model_state = self.autodetector.to_state.models[
self.app_label, operation.name.lower()
]
partitioning_options = model_state.partitioning_options
else:
model = self.autodetector.new_apps.get_model(
self.app_label, operation.name
)
partitioning_options = model._partitioning_meta.original_attrs
_, args, kwargs = operation.deconstruct()
if partitioning_options["method"] != PostgresPartitioningMethod.HASH:
self.add(
operations.PostgresAddDefaultPartition(
model_name=model.__name__, name="default"
model_name=operation.name, name="default"
)
)
......@@ -153,9 +205,7 @@ class AddOperationHandler:
)
)
def add_delete_partitioned_model(
self, model: Model, operation: DeleteModel
):
def add_delete_partitioned_model(self, operation: DeleteModel):
"""Adds a :see:PostgresDeletePartitionedModel operation to the list of
operations to execute in the migration."""
......@@ -164,11 +214,21 @@ class AddOperationHandler:
operations.PostgresDeletePartitionedModel(*args, **kwargs)
)
def add_create_view_model(self, model: Model, operation: CreateModel):
def add_create_view_model(self, operation: CreateModel):
"""Adds a :see:PostgresCreateViewModel operation to the list of
operations to execute in the migration."""
view_options = model._view_meta.original_attrs
if django.VERSION >= (4, 0):
model_state = self.autodetector.to_state.models[
self.app_label, operation.name.lower()
]
view_options = model_state.view_options
else:
model = self.autodetector.new_apps.get_model(
self.app_label, operation.name
)
view_options = model._view_meta.original_attrs
_, args, kwargs = operation.deconstruct()
self.add(
......@@ -177,20 +237,28 @@ class AddOperationHandler:
)
)
def add_delete_view_model(self, model: Model, operation: DeleteModel):
def add_delete_view_model(self, operation: DeleteModel):
"""Adds a :see:PostgresDeleteViewModel operation to the list of
operations to execute in the migration."""
_, args, kwargs = operation.deconstruct()
return self.add(operations.PostgresDeleteViewModel(*args, **kwargs))
def add_create_materialized_view_model(
self, model: Model, operation: CreateModel
):
def add_create_materialized_view_model(self, operation: CreateModel):
"""Adds a :see:PostgresCreateMaterializedViewModel operation to the
list of operations to execute in the migration."""
view_options = model._view_meta.original_attrs
if django.VERSION >= (4, 0):
model_state = self.autodetector.to_state.models[
self.app_label, operation.name.lower()
]
view_options = model_state.view_options
else:
model = self.autodetector.new_apps.get_model(
self.app_label, operation.name
)
view_options = model._view_meta.original_attrs
_, args, kwargs = operation.deconstruct()
self.add(
......@@ -199,9 +267,7 @@ class AddOperationHandler:
)
)
def add_delete_materialized_view_model(
self, model: Model, operation: DeleteModel
):
def add_delete_materialized_view_model(self, operation: DeleteModel):
"""Adds a :see:PostgresDeleteMaterializedViewModel operation to the
list of operations to execute in the migration."""
......
psqlextra/lookups.py 0 → 100644
+ 34
− 0
View file @ 98da8d0b
from django.db.models import lookups
from django.db.models.fields import Field, related_lookups
from django.db.models.fields.related import ForeignObject
class InValuesLookupMixin:
"""Performs a `lhs IN VALUES ((a), (b), (c))` lookup.
This can be significantly faster then a normal `IN (a, b, c)`. The
latter sometimes causes the Postgres query planner do a sequential
scan.
"""
def as_sql(self, compiler, connection):
if not self.rhs_is_direct_value():
return super().as_sql(compiler, connection)
lhs, lhs_params = self.process_lhs(compiler, connection)
_, rhs_params = self.process_rhs(compiler, connection)
rhs = ",".join([f"(%s)" for _ in rhs_params]) # noqa: F541
return f"{lhs} IN (VALUES {rhs})", lhs_params + list(rhs_params)
@Field.register_lookup
class InValuesLookup(InValuesLookupMixin, lookups.In):
lookup_name = "invalues"
@ForeignObject.register_lookup
class InValuesRelatedLookup(InValuesLookupMixin, related_lookups.RelatedIn):
lookup_name = "invalues"
psqlextra/management/commands/pgpartition.py
+ 5
− 16
View file @ 98da8d0b
......@@ -2,9 +2,6 @@ import sys
from typing import Optional
import colorama
from ansimarkup import ansiprint, ansistring
from django.conf import settings
from django.core.management.base import BaseCommand
from django.utils.module_loading import import_string
......@@ -70,10 +67,6 @@ class Command(BaseCommand):
*args,
**kwargs,
):
# disable coloring if no terminal is attached
if not sys.stdout.isatty():
colorama.init(strip=True)
partitioning_manager = self._partitioning_manager()
plan = partitioning_manager.plan(
......@@ -83,7 +76,7 @@ class Command(BaseCommand):
creations_count = len(plan.creations)
deletions_count = len(plan.deletions)
if creations_count == 0 and deletions_count == 0:
ansiprint("<b><white>Nothing to be done.</white></b>")
print("Nothing to be done.")
return
plan.print()
......@@ -92,18 +85,14 @@ class Command(BaseCommand):
return
if not yes:
sys.stdout.write(
ansistring(
"<b><white>Do you want to proceed? (y/N) </white></b>"
)
)
sys.stdout.write("Do you want to proceed? (y/N) ")
if not self._ask_for_confirmation():
ansiprint("<b><white>Operation aborted.</white></b>")
print("Operation aborted.")
return
plan.apply(using=using)
ansiprint("<b><white>Operations applied.</white></b>")
print("Operations applied.")
@staticmethod
def _ask_for_confirmation() -> bool:
......@@ -119,7 +108,7 @@ class Command(BaseCommand):
@staticmethod
def _partitioning_manager():
partitioning_manager = getattr(
settings, "PSQLEXTRA_PARTITIONING_MANAGER"
settings, "PSQLEXTRA_PARTITIONING_MANAGER", None
)
if not partitioning_manager:
raise PostgresPartitioningError(
......
psqlextra/partitioning/__init__.py
+ 2
− 1
View file @ 98da8d0b
......@@ -5,6 +5,7 @@ from .manager import PostgresPartitioningManager
from .partition import PostgresPartition
from .plan import PostgresModelPartitioningPlan, PostgresPartitioningPlan
from .range_partition import PostgresRangePartition
from .range_strategy import PostgresRangePartitioningStrategy
from .shorthands import partition_by_current_time
from .strategy import PostgresPartitioningStrategy
from .time_partition import PostgresTimePartition
......@@ -22,8 +23,8 @@ __all__ = [
"PostgresTimePartition",
"PostgresPartitioningStrategy",
"PostgresTimePartitioningStrategy",
"PostgresCurrentTimePartitioningStrategy",
"PostgresRangePartitioningStrategy",
"PostgresCurrentTimePartitioningStrategy",
"PostgresPartitioningConfig",
"PostgresTimePartitionSize",
]
psqlextra/partitioning/plan.py
+ 7
− 12
View file @ 98da8d0b
from dataclasses import dataclass, field
from typing import List, Optional
from ansimarkup import ansiprint
from django.db import connections, transaction
from .config import PostgresPartitioningConfig
......@@ -49,17 +48,17 @@ class PostgresModelPartitioningPlan:
def print(self) -> None:
"""Prints this model plan to the terminal in a readable format."""
ansiprint(f"<b><white>{self.config.model.__name__}:</white></b>")
print(f"{self.config.model.__name__}:")
for partition in self.deletions:
ansiprint("<b><red> - %s</red></b>" % partition.name())
print(" - %s" % partition.name())
for key, value in partition.deconstruct().items():
ansiprint(f"<white> <b>{key}</b>: {value}</white>")
print(f" {key}: {value}")
for partition in self.creations:
ansiprint("<b><green> + %s</green></b>" % partition.name())
print(" + %s" % partition.name())
for key, value in partition.deconstruct().items():
ansiprint(f"<white> <b>{key}</b>: {value}</white>")
print(f" {key}: {value}")
@dataclass
......@@ -104,12 +103,8 @@ class PostgresPartitioningPlan:
create_count = len(self.creations)
delete_count = len(self.deletions)
ansiprint(
f"<b><red>{delete_count} partitions will be deleted</red></b>"
)
ansiprint(
f"<b><green>{create_count} partitions will be created</green></b>"
)
print(f"{delete_count} partitions will be deleted")
print(f"{create_count} partitions will be created")
__all__ = ["PostgresPartitioningPlan", "PostgresModelPartitioningPlan"]
setup.py
+ 18
− 4
View file @ 98da8d0b
......@@ -4,6 +4,8 @@ import subprocess
from setuptools import find_packages, setup
exec(open("psqlextra/_version.py").read())
class BaseCommand(distutils.cmd.Command):
user_options = []
......@@ -36,7 +38,7 @@ with open(
setup(
name="django-postgres-extra",
version="2.0.4",
version=__version__,
packages=find_packages(exclude=["tests"]),
include_package_data=True,
license="MIT License",
......@@ -63,9 +65,8 @@ setup(
],
python_requires=">=3.6",
install_requires=[
"Django>=2.0",
"Django>=2.0,<5.0",
"python-dateutil>=2.8.0,<=3.0.0",
"ansimarkup>=1.4.0,<=2.0.0",
],
extras_require={
':python_version <= "3.6"': ["dataclasses"],
......@@ -90,10 +91,23 @@ setup(
"isort==5.10.0",
"docformatter==1.4",
],
"publish": [
"build==0.7.0",
"twine==3.7.1",
],
},
cmdclass={
"lint": create_command(
"Lints the code", [["flake8", "setup.py", "psqlextra", "tests"]]
"Lints the code",
[
[
"flake8",
"--builtin=__version__",
"setup.py",
"psqlextra",
"tests",
]
],
),
"lint_fix": create_command(
"Lints the code",
......
tests/test_lookups.py 0 → 100644
+ 102
− 0
View file @ 98da8d0b
from django.db import models
from .fake_model import get_fake_model
def test_invalues_lookup_text_field():
model = get_fake_model({"name": models.TextField()})
[a, b] = model.objects.bulk_create(
[
model(name="a"),
model(name="b"),
]
)
results = list(model.objects.filter(name__invalues=[a.name, b.name, "c"]))
assert results == [a, b]
def test_invalues_lookup_integer_field():
model = get_fake_model({"number": models.IntegerField()})
[a, b] = model.objects.bulk_create(
[
model(number=1),
model(number=2),
]
)
results = list(
model.objects.filter(number__invalues=[a.number, b.number, 3])
)
assert results == [a, b]
def test_invalues_lookup_uuid_field():
model = get_fake_model({"value": models.UUIDField()})
[a, b] = model.objects.bulk_create(
[
model(value="f8fe0431-29f8-4c4c-839c-8a6bf29f95d5"),
model(value="2fb0f45b-afaf-4e24-8637-2d81ded997bb"),
]
)
results = list(
model.objects.filter(
value__invalues=[
a.value,
b.value,
"d7a8df83-f3f8-487b-b982-547c8f22b0bb",
]
)
)
assert results == [a, b]
def test_invalues_lookup_related_field():
model_1 = get_fake_model({"name": models.TextField()})
model_2 = get_fake_model(
{"relation": models.ForeignKey(model_1, on_delete=models.CASCADE)}
)
[a_relation, b_relation] = model_1.objects.bulk_create(
[
model_1(name="a"),
model_1(name="b"),
]
)
[a, b] = model_2.objects.bulk_create(
[model_2(relation=a_relation), model_2(relation=b_relation)]
)
results = list(
model_2.objects.filter(relation__invalues=[a_relation, b_relation])
)
assert results == [a, b]
def test_invalues_lookup_related_field_subquery():
model_1 = get_fake_model({"name": models.TextField()})
model_2 = get_fake_model(
{"relation": models.ForeignKey(model_1, on_delete=models.CASCADE)}
)
[a_relation, b_relation] = model_1.objects.bulk_create(
[
model_1(name="a"),
model_1(name="b"),
]
)
[a, b] = model_2.objects.bulk_create(
[model_2(relation=a_relation), model_2(relation=b_relation)]
)
results = list(
model_2.objects.filter(
relation__invalues=model_1.objects.all().values_list(
"id", flat=True
)
)
)
assert results == [a, b]
tox.ini
+ 2
− 1
View file @ 98da8d0b
[tox]
envlist = py36-dj{20,21,22,30,31,32}, py37-dj{20,21,22,30,31,32}, py38-dj{20,21,22,30,31,32}, py39-dj{21,22,30,31,32}, py310-dj{21,22,30,31,32}
envlist = py36-dj{20,21,22,30,31,32}, py37-dj{20,21,22,30,31,32}, py38-dj{20,21,22,30,31,32,40}, py39-dj{21,22,30,31,32,40}, py310-dj{21,22,30,31,32,40}
[testenv]
deps =
......@@ -9,6 +9,7 @@ deps =
dj30: Django~=3.0.0
dj31: Django~=3.1.0
dj32: Django~=3.2.0
dj40: Django~=4.0.0
.[test]
setenv =
DJANGO_SETTINGS_MODULE=settings
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment