tensor-thumbnail's Introduction

Tensor Thumbnail

When working with neural networks in TensorFlow, one of the big challenges is aligning the tensor, matrix and vector dimensions properly. When starting with machine learning, it is particularly easy to loose sense of dimensionality in the middle of a complicated expression. This library is designed to help with debugging and understanding code that makes heavy use of linear algebra.

The main component of the library is TensorThumbnail struct - a compact SwiftUI view. It renders tensor shape as a cube-like diagram annotated with dimensions.

Convention

Diagrams use conventions adopted in virtually all linear algebra textbooks. Unless otherwise stated, vectors (a.k.a. 1D tensors) are displayed as a column. It naturally follows that matrices (a.k.a 2D tensors) use the [rows, columns] shape convention and expand the vector diagram horizontally. Images (a.k.a. 3D tensors with shape [height, width, channels]) are rendered as matrices stacked in the depth-wise channel dimension.

Schematics above shows a few examples and a way to reconstruct tensor shape from the diagram. The width, height and depth of the tensor diagram are scaled relative to each other. So cases when one dimension is much larger than the others are visible at the first sight. The scaling also allows quick judgement if all the dimensions are approximately equal.

Usage

Xcode Debugger - DOESN'T WORK

The main goal of the project is implementation of the Quick Look feature for Tensors in Xcode Debugger. Here's official Apple documentation on the topic which contains explanation how everything works. The library implements an extension of Tensor with debugQuickLookObject() method that returns the tensor thumbnail wrapped in a NSView.

Unfortunately, as of Xcode 12 this doesn't seem to be possible. To the extent of my knowledge the debugQuickLookObject() method is called only for classes. It requires @objc decorator responsible for Objective-C bridging in order to work properly. Since Tensor is a struct there's no way to implement this feature unless Apple decides to allow it and changes the implementation in Xcode. There are more details about the problem in issue #2.

Xcode Playgrounds - WORKS ONLY WITH MOCKUP

Another goal of the project is to implement Tensor previews in Xcode Playgrounds. The library implements an extension that makes Tensor conform to CustomPlaygroundDisplayConvertible protocol. The protocol requirement is playgroundDescription property and the implementation returns the thumbnail wrapped in a NSView. The thumbnail is then visible in the playgrounds notebook.

While the previews themselves are working well, as of 0.11 S4TF toolchain doesn't support Xcode Playgrounds. This should eventually fix itself. My understanding is that the ultimate end goal is to have TensorFlow work as a regular SPM library. The differentiation features should be available in release builds of the Swift compiler. Once this is completed, the user experience there should be great.

At the moment Playgrounds only works with TensorFraud instead of TensorFlow. TensorFraud (a part of this repository) is a small mockup of TensorFlow that may be useful in situations when running the full S4TF toolchain isn't possible.

Examples

The Examples/ directory contains demonstrations how TensorThumbnail view can be used.

Gallery.xcodeproj - An example macOS application that displays diagrams for a few common tensor shapes.
Playground.playground - A playground with a few example tensors to play with. In order to make external Swift packages work, the playground must be opened from within the Workspace.xcworkspace. If opened standalone, the Swift compiler will fail because it won't be able to find the TensorThumbnail module and import it.

Requirements

You may have to experiment a little bit with the versions. This package is in a very early development stage.

Swift for TensorFlow (S4TF) toolchain 0.11+
Xcode 12.2+

Installation

Library is distributed as a standard SPM package. To use it in your project add the following line to your Package.swift. manifest.

.package(url: "https://github.com/vojtamolda/tensor-sensor", from: "0.0.1"),

License

The source code is licensed under the Apache-2.0 license. See the text for more details.

tensor-thumbnail's People

Contributors

Stargazers

Watchers

tensor-thumbnail's Issues

Visualization beyond the 3rd dimension

Generally, visualizing higher-dimensional spaces is a tricky problem that has a tendency to quite a bit mess with one's brain 👻. So most likely there's not going to be a silver bullet solution that pleases everybody.

The key objective of the visualization diagram is clarity. It should be possible to look at the diagram and intuitively reconstruct back the exact shape of the tensor. The diagram should also try to display the dimensions in proportion to each other. So it is clear at the first sight how two tensors can be multiplied or how to feed them to a neural network layer.

Typical use-case of higher dimensional diagram is an input tensor of a 2D convolution layer. Most frequently used convention for a batch of images is [B, C, H, W]. Meaning of the dimensions is the following: B - batch size, C - number of channels, H - height and W - width.

Quick Look in Xcode Debugger doesn't work

Clicking the 👁️ icon in Xcode Debugger displays very useful previews for built-in types. For example it can show a preview of an image, color, cursor, bezier path and many more types.

Here's an example:

Since about 2014 it is possible to implement custom quick look behavior for user defined data types. Here is Apple's official documentation on how to do it.

Unfortunately, as of Xcode 12 custom quick look isn't possible for structs. To the extent of my knowledge the debugQuickLookObject() method is called only for classes. It requires @objc decorator responsible for Objective-C bridging in order to work properly. Since TensorFlow.Tensor is a struct there's no way to implement this feature unless Apple decides to allow it and changes the implementation in Xcode.

Below is an example main.swift file that demonstrates the problem. When debugging the application QuickLookClass shows a nice rectangle view after clicking the quick look icon. Whereas QuickLookStruct doesn't show anything even tough the implementation is identical

import SwiftUI

/// A simple example view.
struct RedRectangleView: View {
    var body: some View {
       Rectangle()
        .fill()
        .foregroundColor(.red)
        .cornerRadius(5)
    }
}

/// A class with working Quick Look in Xcode Debugger.
class QuickLookClass {
    init() { }
    
    @objc
    func debugQuickLookObject() -> NSView {
        let window = NSWindow(contentRect: NSRect(x: 0, y: 0, width: 20, height: 20),
                              styleMask: [.borderless], backing: .buffered, defer: false)
        window.contentView = NSHostingView(rootView: RedRectangleView())
        return window.contentView!
    }
}

/// A struct with broken Quick Look in Xcode Debugger.
struct QuickLookStruct {
    init() { }

    func debugQuickLookObject() -> NSView {
        let window = NSWindow(contentRect: NSRect(x: 0, y: 0, width: 20, height: 20),
                              styleMask: [.borderless], backing: .buffered, defer: false)
        window.contentView = NSHostingView(rootView: RedRectangleView())
        return window.contentView!
    }
}


let qlClass = QuickLookClass()
let qlStruct = QuickLookStruct()

print(qlStruct, qlClass)

Recommend Projects

vojtamolda / tensor-thumbnail Goto Github PK