Dan Zheng dan-zheng

Keras-style Sequential in Swift for TensorFlow.

The current Sequential in Swift for TensorFlow does not support shape inference. Previous layers' output sizes are redundantly repeated as next layers' input sizes. This increases user burden and adds room for error.

let modelOld = Sequential {

tensorflow/swift-apis compilation is painfully slow. It's probably due to type-checking.

Swift compilation debugging tips here. swift build -Xswiftc -Xfrontend -Xswiftc -debug-time-function-bodies results below.

# Worst offenders, time in milliseconds.
(1767, "global function 'gelu'", '/Users/danielzheng/swift-apis/Sources/TensorFlow/Operators/Math.swift:1225:13')
(1767, "global function 'gelu'", '/Users/danielzheng/swift-apis/Sources/TensorFlow/Operators/Math.swift:1225:13')
(1767, "global function 'gelu'", '/Users/danielzheng/swift-apis/Sources/TensorFlow/Operators/Math.swift:1225:13')

Retroactive derivative registration: register derivatives for functions in other modules.

Previously:

If module A defines func foo, then its derivatives must be in the same module.

With retroactive derivative registration:

If module A defines func foo;
And if module B imports module A and defines @differentiating(foo) func derivativeFoo:
Then module C can import modules A and B and differentiate func foo (e.g. via differentiation APIs).

I tried to define a generic Adam optimizer using ElementaryFunctions, Differentiable, and VectorProtocol (latter two are being incubated on tensorflow branch).

It doesn't work due to missing scalar-vector and vector-vector operations.

$ swift adam.swift
adam.swift:82:95: error: binary operator '+' cannot be applied to operands of type 'Model.TangentVector' and 'Model.TangentVector.VectorSpaceScalar'
        model.move(along: -stepSize * firstMoments / (Model.TangentVector.sqrt(secondMoments) + epsilon))
                                                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~
adam.swift:82:95: note: expected an argument list of type '(Self, Self)'

Python interoperability

Check out this tutorial demonstrating Python interoperability in Swift.

Python library version details

`PythonLibrary.useVersion`

This function is intended to set the Python version before any references to the global PythonInterface called Python - it cannot be used to switch Python versions dynamically.

This gist is an in-depth exploration of the Swift parameter update design, focusing on dynamic parameters and how to synthesize allKeyPaths for them.

	import torch
	import tangent
	from tangent import grad

	def nested_loop(x):
	outer = x
	for _ in range(1, 3):
	outer = outer * x
	inner = outer

	import TensorFlow

	extension SGD {
	// Take `Model` metatype as an argument, so it doesn't need to be written out
	// explicitly as a generic parameter.
	// IMPORTANT: This API change might not be desirable, because usage involves
	// getting dynamic type via `type(of:)`.
	convenience init(_ modelType: Model.Type, learningRate: Scalar) {
	self.init(learningRate: learningRate)
	}

	## Building Swift

	./swift/utils/update-checkout --skip-repository swift --clone --scheme tensorflow
	./swift/utils/build-script -x -R --debug-swift --skip-build-benchmarks --stdlib-deployment-targets=macosx-x86_64 2>&1 \| tee log.txt
	./swift/utils/build-script -R --skip-build-benchmarks --stdlib-deployment-targets=macosx-x86_64 --enable-tensorflow --no-swift-stdlib-assertions --reconfigure

	import TensorFlow
	struct Model: Parameterized, Differentiable {
	@TFParameter var w: Tensor<Float>
	func tangentVector(from cotangent: Parameters) -> Parameters {
	return cotangent
	}
	}

	let model = Model(w: Tensor<Float>(zeros: [5]))
	let pb = pullback(at: model) { m in m.w }