"Allow moving a model between apps" -- proposal for Google Summer of Code 2023.

proposal.md

"Allow moving a model between apps" proposal for Google Summer of Code 2023.

Table of content

1. Abstract
   • 1.1 Overview
   • 1.2 Goals
   • 1.3 Benefits
2. The solution
   • 2.1 Modified CreateModel operation .
   • 2.2 Modified AlterModelTable operation .
   • 2.3 Modified DeleteModel operation .
   • 2.4 New generate_move_models() method in autodetector.
3. Schedule and milestones
4. About me

1.Abstract

1.1 Overview

In long running projects , there often arrives a situation where one or more model(s) are required to move from one app to the other. For example, Imagine you have a Django project for a blogging platform, and you have two apps: posts and users. Initially, you put all the models related to posts in the posts app, including the Post model, the Comment model, and the Tag model. However, as your project grows, you realize that you need to add some features related to user profiles, such as user avatars, user bios, and social media links.

In this case, you might decide to create a new app called profiles and move the UserProfile model from the users app to the new profiles app. However, the UserProfile model has a foreign key to the Post model, which means you need to update the Post model to use the new UserProfile model without losing any data or breaking any existing functionality.

Before:

- posts
    - Post
    - Comment
    - Tag
- users
    - BaseUser
    - UserProfile

After:

- posts
    - Post
    - Comment
    - Tag
- users
    - BaseUser
- profiles
    - UserProfile

One of the solution is to use custom RunRython operation.The Django docs have some information on this which uses RunPython operation and bulk_create to insert data in newly created model in new app, but this process would take a lot of time when you have a huge database, leading to increased downtime of your application in production.

There is another solution posted on SO which uses the powerful SeparateDatabaseAndState operation to move model between the apps and is better than the latter. But even this process requires a lot of manual work which slows down the developement of an application.

There is a long standing ticket to have migration operations auto-generated which would definitely save a lot of time of developers and improve maintainability.

1.2 Goals

This proposal is about modifying the existing CreateModel , DeleteModel and AlterModelTable operations so that they can work on database conditionally and auto-detect the movement of model(s).

1. The first milestone would be to modify DeleteModel and AlterModelTable migration operation which will be placed in the old app , CreateModel placed in new_app along with tests and documentation.
2. The second milestone would be to write generate_moved_models method for detecting a moved model ,writing tests and better documenting the working and use of SeparateDatabaseAndState and guide on moving a model using SeparateDatabaseAndState.

1.3 Benefits

• Models will be easily moved to any app without any data loss or code break.
• Supports reverse migration in case wrong model is moved.
• Will detect the moved model and auto generate migration for you.
• Turn off old app after moving model to new app with very less manual work.
• Closes ticket #24686

2.The Solution

First of all we will create a flag state_only_op(or maybe some different name) in model options which will allow DeleteModel and CreateModel to work conditionally on database. Means if state_only_op is set to True for DeleteModel or CreateModel, then they will only change the state of the model and not the database. This approach provides control over every step involved in moving a model. Now the whole process of moving a model will basically involve 3 steps in 3 different migration files:

1. Creating a new migration file in old app and add AlterModelTable operation in it to change table name to new one along with setting the state_only_op to True for the model to be moved.
2. Then creating a new migration file in new app and adding CreateModel operation which will create a new model state for moved model in new app. In this model state also we'll set state_only_op to True so that the model is created in state only(as we already have the table).After CreateModel operation AlterField operation will be added (in their own apps) for all the fields(if any) which were pointing to the moved model.
3. Another migration file will be added in old app with Alterfield operation for fields referencing the moved model (if any) in old app and in the end DeleteModel operation will be added to delete state of old model from old app. Now this DeleteModel will be applied only on state due to the state_only_op flag which was set to True during AlterModelTable operation.

2.1 Modified CreateModeloperation

The CreateModel operation will be placed in the app to which the model is being moved.

1. First we are going to add state_only_op to CreateModel as an keyword argument and then initialize it to the value passed(either True or False).

    ...
    def __init__(
            self, name,
            fields, options=None,
            bases=None,
            managers=None,
            state_only_op=None
        ):
            self.fields = fields
            self.state_only_op = state_only_op or False
    ...

2. Then we'll create a function say update_op_type in ProjectState to update the flag state_only_op in state.

    ...
    def update_op_type(self, app_label, model_name, options):
        model_state = self.models[app_label, model_name]
        model_state.options = {**model_state.options, **options}
        self.reload_model(app_label, model_name, delay=True)
    ...

3. Now in state_forwards of CreateModel we'll call update_op_type method after adding model to state.

    ...
    state.update_op_type(app_label, self.name_lower, {
            "state_only_op": self.state_only_op})
    ...

4. Then in database_forwards and database_backwards we'll check if state_only_op is not true, then only we'll perform operation on database.

    ...
    if not model._meta.state_only_op:
            if self.allow_migrate_model(schema_editor.connection.alias, model):
                schema_editor.create_model(model)
    ...

2.2 Modified AlterModelTableoperation

The AlterModelTable will be placed in the old_app to remove old model from state.

1. Similar to CreateModel we'll add state_only_op as a keyword argument and initialize it to the value passed(either True or False).

    ...
    def __init__(self, name, table, state_only_op=None):
        self.table = table
        self.state_only_op = state_only_op or False
    ...

2. Then in state_forwards we'll call update_op_type method to update the flag of old state of moved model.

    ...
    def state_forwards(self, app_label, state):
        state.update_op_type(app_label, self.name_lower, {
            "state_only_op": self.state_only_op})
        state.alter_model_options(app_label, self.name_lower, {"db_table": self.table})
    ...

Yes ,updating state_only_op in AlterModelTable seems to be a bit confusing, but it is the only mandatory operation before DeleteModel for altering old model state and we cannot update it in DeleteModel. We can document this behaviour in the process of moving model between apps.

2.3 Modified DeleteModeloperation

The DeleteModel will be placed in the old_app to remove old model from state.

1. Now for DeleteModel we'll just check the flag state_only_op and only perform database operation when its False. This will delete the old model state of moved model.

    ...
    if not model._meta.state_only_op:
            if self.allow_migrate_model(schema_editor.connection.alias, model):
                schema_editor.delete_model(model)
    ...

2.4 New generate_move_models() method in autodetector

A new method will be written in autodetector.py which will detect the moved model(s) and auto-generate migration operations .

1. First we will find the models added in the new app with the help of model keys and store them in added_models.

   ...
   added_models = self.new_model_keys - self.old_model_keys
   ...
   

2. Then we will loop through the added models and extract model state and model fields definition. we are also going to find and store the model(s) removed from the old app. All this information will be used to compare if the model which was added in the new_app is same as it was in the old_app and not a newly created model.

   Note: Only one model will be detected at a time as it will require multiple operations in multiple files with dependencies.

   ...
   for app_label, model_name in sorted(added_models):
       model_state = self.to_state.models[app_label, model_name]
       model_fields_def = self.only_relation_agnostic_fields(model_state.fields)
       removed_models = self.old_model_keys - self.new_model_keys
   ...
   

3. Then we will create another loop inside the above one to compare added_models and removed_models.If the removed_model and added_model have different app labels but same model field definitions, it means the model is moved from one app to another. We will confirm it with the help of a new questioner as_move_model().

   ...
   for moved_app_label, moved_model_name in removed_models:
       if moved_app_label != app_label:
           moved_model_state = self.from_state.models[
               moved_app_label, moved_model_name
           ]
           moved_model_fields_def = self.only_relation_agnostic_fields(
               moved_model_state.fields
           )
           if model_fields_def == moved_model_fields_def:
               if self.questioner.ask_move_model(
                   moved_model_state, model_state
               ):
               ...
   

4. If the user confirms that a model has been moved, then we will have to construct dependencies for related fields and also to detect the next migrations.

   ...
   if model_fields_def == moved_model_fields_def:
       if self.questioner.ask_move_model(
           moved_model_state, model_state
       ):
           dependencies = []
           fields = list(model_state.fields.values()) + [
               field.remote_field
               for relations in self.to_state.relations[
                   app_label, model_name
               ].values()
               for field in relations.values()
           ]
           for field in fields:
               if field.is_relation:
                   dependencies.extend(
                       self._get_dependencies_for_foreign_key(
                           app_label,
                           model_name,
                           field,
                           self.to_state,
                       )
                   )
                   ...
   

5. Once dependencies are created, we are going to add different operations in the in app specific migration files starting with AlterModelTable operation in old app with state_only_op set to True.

    ...
    self.add_operation(
        rem_app_label,
        operations.AlterModelTable(
            name=model_name,
            table=db_table,
            state_only_op=True
        ),
        dependencies=dependencies,
    )
    ...

6. Then we'll add CreateModel in new app with state_only_op set to True.

    ...
    self.add_operation(
        app_label,
        operations.CreateModel(
            name=model_state.name,
            fields=[
                d
                for d in model_state.fields.items()
            ],
            options=model_state.options,
            bases=model_state.bases,
            managers=model_state.managers,
            state_only_op=True,
        ),
        dependencies=dependencies,
        beginning=True,
    )
    ...

7. After CreateModel, we'll add AlterField operation for each foreign key in their specific app. We have to keep track of all alterfield operations so that generate_altered_fields method in autodetector don't add more of them and we can ignore them.

    ...
    for field in fields:
        if field.is_relation and field.remote_field.related_model != model:
            self.add_operation(
                field.related_model._meta.model_name,
                operations.AlterField(
                    model_name=field.related_model._meta.model_name,
                    name=field.remote_field.name,
                    field=field.remote_field,
                ),
                dependencies=dependencies,
            )
             self.already_alter_fields.add(
                (
                    field.related_model._meta.model_name,
                    field.related_model._meta.model_name,
                    field.remote_field.name
                )
            )
            ...

Note: Generic Foreign keys are ignored and AlterField operation will not be added for them because generic foreign key is not actually a field in the database. Instead, it is implemented using two separate fields: a foreign key to the content type model and another foreign key to the specific object instance within that content type. So we'll add Generic Foreign Keys in CreateModel in new app along with other fields so that it can be applied on empty db in case old app is turned off.

8. In the end, DeleteModel operation will be added in second migration file in old app to remove the old state of moved model. So there will be atleast 3 new migration files created(2 in old app ,1 in new app).

    ...
    self.add_operation(
        rem_app_label,
        operations.DeleteModel(
            name=rem_model_name),
        dependencies=dependencies,
    )
    ...

Note: There can be a situation where we need to turn off the old app after moving a model the new app and apply the migrations on an empty database.This can be acheived with very less manual work as follows:

1. Remove the old app from INSTALLED_APPS and old app dependencies from the migration files in new app.
2. Run squashmigrations on new app.
3. Set state_only_op to False (or remove state_only_op) in CreateModel operation generated after squashing.
4. Run migrate command to apply the changes on empty database.

3.Schedule and milestones

My final exams would be conducted during june (the final date is not decided yet). Till then I have regular offline classes plus training, still I would be able to devote 30-35 hours a week (2-3 hours on weekdays and 4-6 hours on weekends) throughout the GSoC period ( a little less during final exams).

I would like to devote 50% of my time to learning and coding, and 50% of my time to test the changes and write documentation for new stuff. I will write blog posts every weekend to make the community aware of my progress, contributions, and a plan for next week.

3.1 Community Bonding(May 4 - May 28)

• Discuss approach and implementation with senior developers or Mentors.
• Figure out if there is any better approach to the problem or the solution could be improved in any way.

3.2 Modified migration operations -- first milestone

(From May 29 to July 10) During this phase, I will work on modifying CreateModel,DeleteModel and AlterModelTbable migration operation in models.py, update_op_type method in ProjectState .

3.2.1 creating new flag and Writing update_db_type method(1 week)

• Creating state_only_op flag.
• Writing new method for updating flag
• Fixing and Writing tests along with Documentaion.(if required)

3.2.2 Modifying CreateModel operation(1 week)

• Initializing CreateModel method with state_only_op flag.
• Calling update_only_db to update flag.
• Adding condition for schemaEditor to work as per flag.
• Fixing and Writing tests related to CreateModel along with Documentation.(if required)

3.2.3 Modifying AlterMModelTable operation(1 week)

• Initializing CreateModel method with state_only_op flag.
• Calling update_only_db to update flag for DeleteModel to work as per flag..
• Fixing and Writing tests related to AlterModelTable along with Documentation.(if required)

3.2.4 Modifying DeleteModel operation(0.5 week)

• Adding condition for schemaEditor to work as per flag.
• Fixing and Writing tests related to CreateModel along with Documentation.(if required)

3.2.5 Testing and documentation(1.5-2 weeks)

• Fixing and Writing tests related to modified migration operations.
• Documenting the changes required.
• Review Fixing.

3.3 Better documenting use of SeparateStateAndDatabase -- second milestone

(From July 14 to August 21) During this time i'll be writing generate_moved_models method in autodetector.py along with writing test related to auto-detector and better documenting SeparateDatabaseAndState for moving models.

3.3.1 Writing generate_moved_models method in autodetector(2-3 weeks)

• Writing logic for detecting movement of models.
• Creating dependencies and operations for foreign keys.
• Auto-generating operations AlterModelTable,DeleteModel in old app and CreateModel in new app.
• Fixing and Writing tests related to autodetector along with Documentation.(if required)

3.3.2 Documenting and testing moving model using SeparateDatabaseAndState(1-2 week)

• Writing tests for moving models between apps for various cases (if required).
• Better documentation is needed along with an example with detailed explanation.

If time permits or After GSOC...

• I would like to work on the issue Adapt schema editors to operate from model states instead of fake rendered models which was a success in GSOC 2021.

4.About me

My name is Bhuvnesh Sharma and I am a Pre-final year Btech. student from Dr. A.P.J. Abdul Kalam Technical University (India). I started my coding journey when i was in my 11th grade with C++ then shifted to python in about a year. I started learnig Django in my freshman year and created a video calling application during covid.

4.1 Past contributions in Django

I started contributing to django in September 2022 and found it really interesting. The community is very also supportive and so I would like to thank all the community members of Django for helping me out in this journey.

4.1.1 Issues Fixed

• #28987 : Migration changing ManyToManyField target to 'self' doesn't work correctly.
• #33975 : __in doesn't clear selected fields on the RHS when QuerySet.alias() is used after annotate().
• #33995 : Rendering empty_form crashes when empty_permitted is passed to form_kwargs.
• #34019 : "Extending Django's default user" section refers to a deleted note.
• #34112 : Add async interface to Model.
• #34137 : model.refresh_from_db() doesn't clear cached generic foreign keys.
• #34171 : QuerySet.bulk_create() crashes on mixed case columns in unique_fields/update_fields.
• #34217 : Migration removing a CheckConstraint results in ProgrammingError using MySQL < 8.0.16.
• #34250 : Duplicate model names in M2M relationship causes RenameModel migration failure.

4.1.2 Pull Requests (Merged)

• PR-16032 : Fixed #33975 -- Fixed __in lookup when rhs is a queryset with annotate() and alias().
• PR-16034 : Refs #33616 -- Updated BaseDatabaseWrapper.run_on_commit comment.
• PR-16041 : Fixed #33995 -- Fixed FormSet.empty_form crash when empty_permitted is passed to form_kwargs.
• PR-16065 : Fixed #34019 -- Removed obsolete references to "model design considerations" note.
• PR-16242 : Fixed #34112 -- Added async-compatible interface to Model methods.
• PR-16251 : Refs #33646 -- Moved tests of QuerySet async interface into async tests.
• PR-16260 : Fixed #34137 -- Made Model.refresh_from_db() clear cached generic relations.
• PR-16281 : Fixed #28987 -- Fixed altering ManyToManyField when changing to self-referential.
• PR-16315 : Fixed #34171 -- Fixed QuerySet.bulk_create() on fields with db_column in unique_fields/update_fields.
• PR-16405 : Fixed #34217 -- Fixed migration crash when removing check constraints on MySQL < 8.0.16.
• PR-16532 : Fixed #34250 -- Fixed renaming model with m2m relation to a model with the same name.

4.2 Personal Details

• Email : [email protected]
• Timezone : Indian Standard Time (UTC + 5:30)
• Primary Language : English
• Country of Residence : India
• LinkedIn Profile : Bhuvnesh Sharma