gistfile1.md

Proposed Meta interface

Refactoring Options to provide a public, simpler and maintainable API

In the next 3 months as part of my Summer of Code project, I will be refactoring Meta to a stable and public API. Based on this, I have come up with a draft API spec that provides (to my belief) the following:

• Simplicity

  There are too many ways to access fields and attributes, this leads to a lot of internal complexity that must be greatly simplified.g

• Future-proof

  Currently, the getter functions and accessors for fields are tailored to the needs of Django internals, instead of creating functions that only support these, we intruduce the concept of predicates, that can be used to create custom filtering ad-hoc for the single case. This will avoid having to create new functions in the future that increase the complexity of Meta, as get_fields and every predicate will be designed to be loosely coupled.

Current state of Meta

Below is a map of the main endpoints in _meta:

• Redundancy

  There seems to be a lot of redundancy: This is expected as the API has never has a basic refactor. There are obviously differences in each function, but they all return similar data. An example of this are the fields accessors:

• get_fields_with_model
• get_concrete_fields_with_model
• concrete_fields
• fields

• Multiple caching systems

  There seems to be more than 1 way to perform caching. In earlier versions of meta, caching was made by the fill* or init* functions that were called by a parent if no cache was available. Later on, caching has been done through @cached_property and expired through del statements

• Too many instance variables

  Many parts of the system access the same instance variables at the same time, we should try to group them together (ex: app_label, model_name). Said this, I still need to investigate if this is possible and I need to find more evidence that this is really the case.

Proposal

Below are a draft list of changes that can be done to improve the current implementation.

• Public accessor

  If we are formalizing _meta, then there should be a non-private way to access it

   >> User.get_meta()
   <Options for User>

• Predicate System

  Predicates a small functions that provide simple filtes for meta. Each predicate should be dumb and simple, it should do 1 thing and do it well. Dumb predicates combined together can yield complex behavour. This means that instead of providing complex filter functions to filter lists of field, we create a simple function get_field that accepts a number of predicates that do filtering for us.

predicates = [exclude_related, exclude_virtual, exclude_m2m] User.get_meta().get_fields(predicates)

Each predicate should accept a standard input and return a standard output, basically a pipeline system. There should be 3 main types of predicates:

* ###### Core predicates
Core predicates are included in django.core and provide common functionality such as:

 - exclude_local
 - exclude_proxy
 - exclude_m2m

 The importance of core predicates is that this becomes part of a standardized API: any custom meta implementation should expect to be able to filter by these predicates internally. 

* ###### App-related predicates
Custom apps and django.contrib apps can specify their own filters that can be added to the pipeline. Functions such as get_concrete_fields_with_model are only used 3 times therefore should be defined outside of the core predicates

* ###### Future - Custom store predicates
By creating a flexible meta system, we can also allow developers to override the current meta implementation and create their own custom logic around a specific store (NoSQL, Email, LDAP, anything). The sub-classes will always need to implement core predicates, but developers can also implement their own predicates to work tighly coupled with their implementation.

* #### FieldMetaSet class as a return type
Meta is not only used to lookup fields, but some times it is used to count the occurances of a field type being present (such as ForeignKey) or to only read field names. the return type of get_field should be a lazy class of type FieldMetaSet, that will provide instance methods for:
- count()
- count(CharField)
- names()
- uniq()


```python
field_meta_set = User.get_meta().get_fields(only_local)
field_meta_set.count()
>>> 11
field_meta_set.count(CharField)
>>> 4

FieldMetaSet can also be iterated

Community feedback

I will be writing feedback obtained on IRC, any other feedback is welcome! you can comment on this Gist

• Provide examples of using predicates
• FieldMetaSet may increase complexity, reduce it to returning (field_object, model, direct, m2m)
• Are predicates necessary? maybe the interface should be simpler