[Tensorflow 2.0] Development Guideline

Compiled Model & Estimator

Applicability:

	1. model compilation requires implementation of compute output shape
    
    2. model compilation requires homogeneous training data tensor shapes
    
    3. model internal attributes is set by passing a defer tensor through model call
    
	4. model compilation is more akin to tf graph mode
    
Estimators can only be initiated from a compiled model, hence accepting only homogeneous input sources


Keras model comes with two modes:

1. graph mode

2. subclassing mode

subclassing mode lacks input output information, can be converted to graph mode by calling build

now the build is not fully implemented

subclasses 

not compiliant w.r.t nested outputs

Keras Constraints.md

impose hard constraints on weights:

MaxNorm, MinMaxNorm, NonNeg, UnitNorm

set constraint to variable to guarantee properties of variables

Keras Layer.md

Implement a Keras Layer:

_init_ save layer configurations

build() called once from _call_, bootstrap from initial input to the layer

call() called from _call_ implement the logics of the layer

_init_ API:

trainable, name and dtype (configure the dtype of layers variables)

attributes: activity_regularizer dtype

io attributes: input, input_mask, input_shape (not eager compitable)

losses: variable regularization tensors associated with the layer

_call_ API:

wrap call for prepost processing

reserved keywords: trainin, mask

internal build API:

add_loss:

add regularizer for the layer's weights should fill inputs when the regularization depends on inputs

add_update:

add update (batch normalization) for the layer should fill inputs when the update depends on inputs

add_variable (add_weight):

basic information: name, shape, dtype, initializer

training related: regularizer, trainable, contraint

distributed synchroization: synchronization, aggregation

build: create variables (may depends on the input shape etc)

compute_output_shape: evaluate output shape from given input shapes

set_weights: setting weights from a numpy array

Keras Model.md

1. model subclass field names:

   1. field name without underscore prefix

   2. avoid use names in existing properties

   3. define grouped primatives for easy testing

   4. explicit naming everything to avoid order depending variable names

base class properties:

activity_regularizer, input input_mask, input_shape, input_spec layers, name, non_trainable_variables, non_trainable_weights output, output_mask, output_shape, state_updates stateful, trainable_variables, trainable_weights update, uses_learning_phase, variables, weights

useless properties:

applicable only in single input or single output model:

input_mask, input_shape

output_mask, output_shape

special properties:

state_updates: updates for stateful layers stateful: if the model is stateful (contraining stateful components)

training properties:

updates: for batch normalization etc

losses: regularization losses associted with all weights

compilation:

used in the case when: model has single output or model has multiple outputs

multiple outputs & mutiple losses: pass a dictionary of loss functions total loss will be the sum of each individual one

target_tensors: default compilation creates placeholder for target tensor pass a single tensor or a list of tensors here

build: take a list of input shapes, initialized internal shape checking

optional APIs to implement

compute_mask: compute mask for each input compute_output_shape: evalue output shape given a list of input shapes