laurentmazare / tch-rs Goto Github PK

I'm trying to translate https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html#sphx-glr-beginner-examples-autograd-two-layer-net-custom-function-py

In particular, this example reimplements MyReLU by providing it's own implementation for forward/backward passes.

class MyReLU(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
    @staticmethod
    def backward(ctx, grad_output):v

Now, we need to find the tch-rs trait that corresponds to torch.autograd.Function .

However, when I grep for "pub trait", I get:

grep "pub trait" . -R
./wrappers/kind.rs:pub trait T {
./nn/optimizer.rs:pub trait OptimizerConfig
./nn/rnn.rs:pub trait RNN {
./nn/module.rs:pub trait Module: std::fmt::Debug {
./nn/module.rs:pub trait ModuleT: std::fmt::Debug {
./tensor/index.rs:pub trait IndexOp<T> {

Question: how do we implement custom functions (with it's own backward/forward) in tch-rs ?

cargo run --example reinforcement-learning --features=python pg

cargo run --example reinforcement-learning --features=python pg
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `target/debug/examples/reinforcement-learning pg`
action space: 2
observation space: [4]
epoch: 0   episodes: 242   avg reward per episode: 20.73
epoch: 1   episodes: 202   avg reward per episode: 24.76
epoch: 2   episodes: 202   avg reward per episode: 24.77
epoch: 3   episodes: 181   avg reward per episode: 27.86
epoch: 4   episodes: 163   avg reward per episode: 30.69

cargo run --example reinforcement-learning --features=python a2c

cargo run --example reinforcement-learning --features=python a2c
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `target/debug/examples/reinforcement-learning a2c`
action space: 6
observation space: [1, 1, 84, 84]
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib64/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "examples/reinforcement-learning/atari_wrappers.py", line 245, in worker
    ob = env.reset()
  File "/home/x/.local/lib/python3.7/site-packages/gym/core.py", line 257, in reset
    observation = self.env.reset(**kwargs)
  File "/home/x/.local/lib/python3.7/site-packages/gym/core.py", line 271, in reset
    return self.env.reset(**kwargs)
  File "/home/x/.local/lib/python3.7/site-packages/gym/core.py", line 258, in reset
    return self.observation(observation)
  File "/home/x/.local/lib/python3.7/site-packages/gym/core.py", line 266, in observation
    raise NotImplementedError
NotImplementedError

(This only shows one thread, as I change the 16 to 1).

it appears to work running directly from Python:

cat test.py ; echo "==="; python3 test.py 
import gym


env = gym.make('SpaceInvaders-v0')
env.reset()
actions = env.action_space.n
===

Question: what am I doing wrong & how do we fix this?

The ocaml-torch bindings now have a yolo-v3 implementation. Port this to tch-rs as the two frameworks are quite similar.
https://github.com/LaurentMazare/ocaml-torch/blob/master/examples/yolo/yolo.ml

I have loaded a python saved model using the code

let model = tch::CModule::load(model_file)?;

how can i convert this to a model so that i can run the below code.

let train_images = tch::no_grad(|| image_dataset.train_images.apply_t(&model, false));

Currently the error is this

error[E0277]: the trait bound `tch::CModule: tch::nn::Module` is not satisfied
  --> src/main.rs:38:65
   |
38 |     let test_images = tch::no_grad(|| image_dataset.test_images.apply_t(&model, false));
   |                                                                 ^^^^^^^ the trait `tch::nn::Module` is not implemented for `tch::CModule`
   |
   = note: required because of the requirements on the impl of `tch::nn::ModuleT` for `tch::CModule`

It's customary for Rust crates to include both MIT and Apache v2.0. Would it be possible to have MIT license so that a user can choose one depending on their use cases?

Several Tensor methods, for example mse_loss(), has i64 typed reduction parameter. It's less intuitive for users is suggested to replace it with existing enum.

It seems this patch requires a patch on auto-generated source. Suppose we can separate the sys API and wrapper api? For now I find the mapping in this Torch source.

Hi Laurent!

In libtorch it is possible to provide options to an LSTM object, such as with_bias.

However, it is not clear how to do the same in tch: constructor for LSTM seems to have a config parameter, but the config type is generic and doesn't seem to support setting RNN-specific options.

I am trying to translate https://pytorch.org/tutorials/beginner/examples_tensor/two_layer_net_tensor.html#sphx-glr-beginner-examples-tensor-two-layer-net-tensor-py to tch-rs.

It's not clear to me how to translate the following line:

grad_h[h<0] = 0

My understanding is that h<0 is supposed to return a vec of bool, which we use as a mask for which elements of grad_h to modify.

It's not clear to me how to do this in a 'parallel' (i.e. not a single for loop) in tch-rs.

tch-rs does not provide torch.chunk. Look like it's missing in f_* functions. The backend seems to be generated by automated means and I would like to follow the convention, so I don't know how to make proper patches for this. Can you implement it or tell me how to make contributions?

We could use TravisCI to ensure that all commit comply with the formatting rules in rustfmt.toml. This is just a matter of adding the following lines to .travis.yml:

before_script:
  - rustup component add rustfmt

script:
- cargo fmt -- --check

However, currently there are also some warnings when running cargo fmt (some code is not formatted accordingly and it seems like some nightly features are used).

The current max1(&self, other: &Tensor) requires a Tensor argument. It leads to some inconvenience on scalar types. It gets worse if the input tensor varies in devices. To make sure max runs without errors, I have to write:

let max_val = Tensor::from(6_f32).to_device(xs.device());
xs.max1(&max_val)

I rather expect a method allowing arbitrary scalar like maxN<S: Into<Scalar>>(&self, other: S). In fact, I discovered #62 issue when I'm trying to implement this feature.

Any objections to a PR adding windows support?

I raised this question from my experience in Keras/TensorFlow. Keras allows users to specify 'same'/'valid' padding besides integers. That's a great convenience. I see that PyTorch does not provide this feature, and thus it's asked on their forum.

I would propose a solution that may get the best of two. We could provide a method set_padding() on tch::nn::ConvConfigND. Following Rust's convention, we could write:

let config = ConvConfig { /* ... */ }.set_padding("same");

We can let set_padding() to accept an enum over str or int. Likely bias, strides and other fields may apply for this change.

I build the my package with tch dep and pass TORCH_CUDA_VERSION env. I observe that it stucks at compiling torch-sys for more than 10 minutes.

env TORCH_CUDA_VERSION=9.0 cargo build

I made a quick diagnosis by lsof and strace. The build script took very long time to unzip the libtorch zip file. The I/O rate only peaks at 5 MiB/s. The build takes place on an SSD disk so disk I/O should not be bottleneck. In contrast, the command unzip v1.1.0.zip completes in seconds. May we consider replace the zip crate with zip-sys, or faster implementations?

When trying to find the loss getting an interesting error. This is when running cross_entropy_for_logits on the labels.

thread 'main' panicked at 'called `Result::unwrap()` on an `Err`
 value: TorchError { c_error: "[enforce fail at CPUAllocator.cpp:56] posix_memalign(&data, gAlignment, 
nbytes) == 0. 12 vs 0\nframe #0: std::function<std::string ()>::operator()() const + 0x11 (0x7ffbfd74c441 
inn/target/debug/build/torch-sys-8ab344225acfe8de/out/libtorch/libtorch/lib/libc10.so)\nframe #1: 
c10::ThrowEnforceNotMet(char const*, int, char const*, std::string const&, void const*) + 0x49 
(0x7ffbfd74c259 inn/target/debug/build/torch-sys-8ab344225acfe8de/out/libtorch/libtorch
/lib/libc10.so)\nframe #2: c10::alloc_cpu(unsigned long) + 0x65e (0x7ffbfd73546e inn/target/debug
/build/torch-sys-8ab344225acfe8de/out/libtorch/libtorch/lib/libc10.so)\nframe #3: <unknown function> + 
0x13dca (0x7ffbfd736dca inn/target/debug/build/torch-sys-8ab344225acfe8de/out/libtorch/libtorch
/lib/libc10.so)\nframe #4: THStorage_resize + 0x76 (0x7ffbf40aa8b6 inn/target/debug/build/torch-
sys-8ab344225acfe8de/out/libtorch/libtorch/lib/libcaffe2.so)\nframe #5: 
at::native::resize_cpu_(at::Tensor&, c10::ArrayRef<long>) + 0x38f (0x7ffbf3d014df inn/target/debug
/build/torch-sys-8ab344225acfe8de/out/libtorch/libtorch/lib/libcaffe2.so)\nframe #6: <unknown function> 
+ 0xb2cb6e (0x7ffbf3e5ab6e inn/target/debug/build/torch-sys-8ab344225acfe8de/out/libtorch/libtorch
/lib/libcaffe2.so)\nframe #7: at::Tensor::resize_(c10::ArrayRef<long>) + 0x4d (0x55777ea3df2b in 
target/debug/pytorch-image-classification)\nframe #8: <unknown function> + 0xb3d18c 
(0x7ffbf3e6b18c inn/target/debug/build/torch-sys-8ab344225acfe8de/out/libtorch/libtorch
/lib/libcaffe2.so)\nframe #9: at::native::mm(at::Tensor const&, at::Tensor const&) + 0x65 
(0x7ffbf3c7a485 inn/target/debug/build/torch-sys-8ab344225acfe8de/out/libtorch/libtorch
/lib/libcaffe2.so)\nframe #10: at::TypeDefault::mm(at::Tensor const&, at::Tensor const&) const + 0x5d 
(0x7ffbf4013a8d inn/target/debug/build/torch-sys-8ab344225acfe8de/out/libtorch/libtorch
/lib/libcaffe2.so)\nframe #11: torch::autograd::VariableType::mm(at::Tensor const&, at::Tensor const&) 
const + 0x6ea (0x7ffbf27059fa inn/target/debug/build/torch-sys-8ab344225acfe8de/out/libtorch/libtorch
/lib/libtorch.so.1)\nframe #12: <unknown function> + 0x3238ea (0x7ffbf22488ea in target/debug/build
/torch-sys-8ab344225acfe8de/out/libtorch/libtorch/lib/libtorch.so.1)\nframe #13: 
torch::autograd::generated::MmBackward::apply(std::vector<torch::autograd::Variable, 
std::allocator<torch::autograd::Variable> >&&) + 0x170 (0x7ffbf227b750 in target/debug/build/torch-
sys-8ab344225acfe8de/out/libtorch/libtorch/lib/libtorch.so.1)\nframe #14: <unknown function> + 
0x30cd5a (0x7ffbf2231d5a in target/debug/build/torch-sys-8ab344225acfe8de/out/libtorch/libtorch
/lib/libtorch.so.1)\nframe #15: 
torch::autograd::Engine::evaluate_function(torch::autograd::FunctionTask&) + 0x385 (0x7ffbf222ae25 
in /target/debug/build/torch-sys-8ab344225acfe8de/out/libtorch/libtorch/lib/libtorch.so.1)\nframe #16: 
torch::autograd::Engine::thread_main(torch::autograd::GraphTask*) + 0xc0 (0x7ffbf222ce20 in 
target/debug/build/torch-sys-8ab344225acfe8de/out/libtorch/libtorch/lib/libtorch.so.1)\nframe #17: 
torch::autograd::Engine::thread_init(int) + 0x136 (0x7ffbf222a1f6 inn/target/debug/build/torch-
sys-8ab344225acfe8de/out/libtorch/libtorch/lib/libtorch.so.1)\nframe #18: <unknown function> + 
0xbd9e0 (0x7ffbfda229e0 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)\nframe #19: <unknown function> 
+ 0x76db (0x7ffbf19016db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #20: clone + 0x3f 
(0x7ffbf141288f in /lib/x86_64-linux-gnu/libc.so.6)\n" }', src/libcore/result.rs:997:5

Hi

As different contributors are showing interests, having a CI becomes critical.

These examples have been broken when switching to PyTorch v1.1.0.

It's mostly about my thoughts on VarStore.

First it's about the way current impl handles duplicated names. As you see this line, it puts a HashMap size suffix when collision. However, it leads to inconsistent namings depending the order you add the tensors.

Suppose we want to add three tensors named foo, and other three named bar.
If we start from foo tensors and then bar, it results in

foo, foo__1, foo__2, boo, boo__4, boo__5

If we add bar tensors first instead, we have

bar, bar__1, bar__2, foo, foo__4, foo__5

I'm not sure it's expected or not. It's fine in most cases. I observed this when I wrote tensor logging. Imagine you're developing a model and change the order of model building functions. It could break the log reader. We can take TensorFlow's naming as reference. At least, we can make it order independent.

Another scenario is variable reusing. The current impl always append a new tensor whenever you call methods on Path. Also, I see no way to obtain saved tensors from VarStore. That's inconvenient if we want to build two models on shared variables. We may provide an additional "try_reuse" parameter for this.

It happens that I had to implement MobileNetV3 (repo link) for some purpose. I think it's small enough to fit into examples. It provides an example program and library interface. I have plan A and B to publish this work.

• A: Integrate into the examples. It would be better to let users import the model in the code.
• B: Keep it a standalone repo, and I will publish it to crates.io.

If plan A, we would need someone else to verify the correctness of the impl. So far I'm only certain that the precision reaches paper's claim in minutes.

I'm wondering if a convenient slicing function that automatically select(), narrow(), masked_index() or index_select() tensors. Just like that in PyTorch. For the sake of limitations of Index and IndexMut, we could name a polymorphic method tensor.i(), which impl depends on input type. This snipplet illustrates the idea.

trait TensorIndex<T> {
    fn i(&self, index: T) -> Tensor;
}

impl TensorIndex<Range> for Tensor {...}

I looked into how PyTorch handles slice indexes of distinct types, and summarize them into these categories

I think Rust is capable of providing above semantics. However, unlike Python, we cannot have mixed typed slices. We need to play with macros to cope with explosive combinations of mixed-type tuples. So I leave the thought here and seek if anyone knows the best way.

In the ops generated code, ptr_option is used to get a C pointer from an option or null. Rather than pattern matching use unwrap_or.

Here is my scenario: I'm working on a huge custom data set. The iterator code loads images sequentially and convert to tensor by Tensor::of_slice(). However, I observed obvious memory growing and got OOM eventually.

The code can be as simple as:

for array in data_source {
    let tensor = Tensor::of_slice(array.as_slice()).to_device(gpu_device);
    let output = model.forward_t(&tensor);
}

Although I comment out my model code and leave of_slice() alone, the memory still grows even the tensor goes out of scope. Hence, I got some wondering:

• What's is canonical way to feed custom data set?
• When and which action triggers the deallocation of tensor? It is done automatically or manually?
• Does the second forward_t() double the memory?

I ran into this issue when I'm working with multi-GPU training. Surprisingly, this test failed unexpectedly. The device() returns Cuda(1) after to_device(Cuda(0)). Suppose it an off-by-1 bug?

use tch::{Tensor, Device};

#[test]
fn test_device() {
    let x = Tensor::from(1).to_device(Device::Cuda(0));
    assert_eq!(x.device(), Device::Cuda(0)); // actually returns Device::Cuda(1)
}

The forward function only allows single tensor. It cannot fit the need of multiple input models. I looked PyTorch example and it allows to pass a list of tensors. For example, we can provide forward(&[&Tensor]) for this purpose.

Consider the following code:

extern crate tch;
use tch::{Cuda, Tensor};

pub fn main() {
    println!("cuda: {:?}", Cuda::is_available());

    let opts = (tch::Kind::Float, tch::Device::Cuda(1));

    let start = std::time::Instant::now();
    let x_empty = Tensor::empty(&[5, 3], opts);
    let mid = std::time::Instant::now();

    let x_rand = Tensor::rand(&[5, 3], opts);
    let x_zeros = Tensor::zeros(&[5, 3], opts);
    let t = Tensor::of_slice(&[5, 3]);

    let end = std::time::Instant::now();

    println!("time to create 1st tensor: {:?}", mid - start);
    println!("time to create next 3 tensor: {:?}", end - mid);

    println!("start: {:?}", start);
    println!("mid: {:?}", mid);
    println!("end: {:?}", end);
}

I get results of:

cuda: true
time to create 1st tensor: 4.124049426s
time to create next 3 tensor: 907.468µs
start: Instant { tv_sec: 28481, tv_nsec: 825629454 }
mid: Instant { tv_sec: 28485, tv_nsec: 949678880 }
end: Instant { tv_sec: 28485, tv_nsec: 950586348 }

Clearly I am doing something wrong, as it should not take 4 seconds to initialize CUDA. What am I doing wrong?

in this example for loading pretrained files, .ot files are used while the general pytorch convention is to save in .pth files. Whats the difference between the two?

I have a tensor T and I would like to make changes in the tensor. One way I can do that is by multiplying the tensor with a scalar. Is there a way to do that?

This may work out of the box but we should probably test this, fix if necessary, and add some specific CI integration if this starts becoming important.

Hi

First of all, thank you for the binding initiations. Rust needs to have it eventually :)

Running cargo run --example basics fails using stable(1.0) cpu libtorch

--- stderr
CMake Error at /home/ehsan/LibTorch/libtorch/share/cmake/Caffe2/public/utils.cmake:17 (add_dependencies):
  add_dependencies Cannot add target-level dependencies to INTERFACE library
  target "caffe2_library".

Call Stack (most recent call first):
  /home/ehsan/LibTorch/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:121 (caffe2_interface_library)
  /home/ehsan/LibTorch/libtorch/share/cmake/Torch/TorchConfig.cmake:39 (find_package)
  CMakeLists.txt:4 (find_package)

I think, we should consider adding a CI.

This example could be based on the following Python tutorial, e.g. by using an implementation close to the ocaml one.
Once this is done, it may be worth writing some tutorial around the example.

The Tensor methods double_value() and int64_value() can possibly segfault if idx parameter is nonsense. For example:

let t: Tensor = (0.0).into(); // t is a scalar tensor
t.double_value(&[0]); // segfault!

The code segfaults due to irrelevant dimension index. The idx should be checked beforehand, or double_value() should be marked unsafe. It would help that we can identify errors by message rather than helpless segfaults.

It seems current tch-rs lacks distribution primitives like torch.distributions and tf.distributions. Though it already have probability generators, there's still rooms for probability arithmetic and deductions.

I see a promising crate rv that may fit my needs. However, its interface is not general enough. We cannot pass a mean/std tensor to rv's Gaussion new() function.

Porting torch.distributions may solve the problem. It requires some patience to make it work, or I could go ahead to improve rv. I'd like to know if author has any plan to port the module, or leaves it to other crates.

If you're asking which place needs this feature, I'm implementing ELBO loss for GQN (paper).

Hi,

I'm not super familiar with the design of the library, but I've been trying to use it in a project of mine. In doing so I found that Tensor::f_copy mutates a Tensor that's passed as a immutable reference, which is UB in Rust. I think one of the arguments should be changed to &mut to fix this. Sorry if I've missed some context.

Thanks!

Hi

Do you mind explaining how tch-rs/gen work? and when does it kick in?

Why not binding against c-api in $LIBTORCH/include/torch/csrc/api/include/torch/ directly?

Thanks

Could example the significance of using the following pattern over ffi in your code vs. empty struct despite having the same size and alignment?

#[repr(C)]
pub struct MyStruct {
    _private: [u8; 0],
}

vs.

#[repr(C)]
pub struct MyStruct;

Hi Laurent

First of all, I wanted to thank you again for making this happen. Given the pace of the developments and I would love to see an amazing NN crate for Rust, below are my suggestions for v0.1 release.

• Improve error handling:
  • Use failure crate for error handle.
  • Less panic and use unsafe_torch_err! more often.
  • Handling device errors #16
• Various idiomatic Rust improvements:
  • Customizable optimizers #18
• More unit test coverage.
• Improve overall documentations.
  • For module level docs use //!
  • Add doc examples more important methods/functions.
  • Cross-reference modules.
• Decouple implementations from codegen.
• Complete tutorials at least as much as the ocaml-torch equivalent.
• Integration with Rust ndarray.
• GPU build and testing:
  • Local
  • CI (no free option)
• Cover as much as PyTorch API as possible. (see how it goes?)
  • Linalg ops for dense and sparse tensors.
  • Add as much nn ops as possible in nn.
  • Initializers.
  • Data loading and augmentations.
  • Multiprocessing with rayon.
  • Distributed (though it's harder).
• Pytorch extensions C++ <--> C <--> Rust
• Subcrate core, vision, model_zoo, ffi inside tch through vitual workspace manifest.

Since you've put a lot of efforts so far and I guess functionality-wise you want to make this crate mimic your other similar projects, please let us know of any other plans to be on the same page.

Add some automated tests for tensor operations (including cat).
Also add some tests for gradient computation.

It seems that the C wrappers get recompiled on pretty much any change. Using ccache makes it a bit better but we should try to fix this as it makes the development loop a bit slow.
A possibility could be to have a subcrate torch-sys which contains the C wrappers so that the C code only gets recompiled when parts of this new crate are modified.

Hi I'm new to tch, and the Rust implementation is amazing. I took some time to write tch code, and found the example in "Writing a Simple Neural Network" section cannot compile. Simply the argument list of nn::linear is wrong. It expects additional LinearConfig type parameter, and first argument should have nn::Path type instead of nn::VarStore.

Another mild suggestion is that please make the example complete. At least include the extern crate tch; and a very simple main function. That would be a great help for newbies.

When going through the code found that imagenet has a handy function for loading images from a directory provided they are in a good structure.

use tch::vision::imagenet::load_from_dir;

I have a code in the below format.

let image_dataset = load_from_dir(DATASET_FOLDER).unwrap();

where the dataset folder is in this format

dataset
├── train
│   ├── accordion
│   │   ├── image_0001.jpg
│   │   ├── image_0002.jpg
│   ├── airplanes
│   │   ├── image_0001.jpg
│       └── image_0060.jpg
	...
└── val
    ├── accordion
    │   └── image_0036.jpg
    ├── airplanes
    │   └── image_0685.jpg
	...

204 directories, 9060 files

Running this code gets me the below error

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: TorchError { c_error: "invalid argument 0: Sizes of tensors must match except in dimension 0. Got 225 and 224 in dimension 2 at /pytorch/aten/src/TH/generic/THTensor.cpp:711" }

Not sure if this is a mistake from my end or a bug.

I raised this question for multi-GPU. In my scenario, I would run forward step on multiple GPU devices. Then, after computing gradient on one device, I would like to copy the model parameters over.

I seems there's no easy way to copy the model parameters. VarStore::trainable_variables() returns vec of tensors does not help much. Also, I did not see demanding method of tch::nn::Path to iterate over known variables.

So far the only solution I came up with is to save the VarStore to a file, and load them back into multiple devices. However, it implies overhead. Does anyone have suggestion for this?

It seems that changing something like Device implementation that is used in c_wrapper_generated.rs cannot be made automatic due to none existent code generation setup in the crate, so it's a greater barrier for further developments and improvements.

Could you remove the generated codes from the crate and make them available only when the crate is built as artifacts? (This is possible with torch-sys though)

While trying to build this from a manually downloaded torch library, I got the below error.

cargo:warning=In file included from libtch/torch_api.cpp:6:0:
cargo:warning=libtch/torch_api.cpp: In function ‘at::Tensor* at_tensor_of_data(void*, int64_t*, int, int, int)’:
cargo:warning=libtch/torch_api.cpp:48:48: error: ‘class at::DeprecatedTypeProperties’ has no member named ‘elementSizeInBytes’
cargo:warning=     if (element_size_in_bytes != tensor.type().elementSizeInBytes())
cargo:warning=                                                ^

Now I can see that this method element_size_in_bytes has been removed recently. pytorch/pytorch#17785, which might be the reason for this failure in the build process

Continue discussion from #51.

Interoperability with the ndarray crate would be very nice. As a first step I think we should be able to convert to/from ndarray using std::convert::Into. I'm thinking something along these lines:

use ndarray::{Array, ArrayD, Dimension, IxDyn};

pub fn to_tensor<D>(arr: Array<f64, D>) -> tch::Tensor
where
    D: Dimension,
{
    let tn = tch::Tensor::of_slice(arr.as_slice().unwrap());
    let shape: Vec<i64> = arr.shape().iter().map(|s| *s as i64).collect();
    tn.reshape(&shape)
}

pub fn from_tensor(tensor: &tch::Tensor) -> ArrayD<f64> {
    let v: Vec<f64> = tensor.into();
    let shape: Vec<usize> = tensor.size().iter().map(|s| *s as usize).collect();
    ArrayD::from_shape_vec(IxDyn(&shape), v).unwrap()
}

What do you think about supporting CUDA, either via an environment variable TORCH_USE_CUDA, TORCH_CUDA_VERSION, TORCH_DEVICE or similar or having it automatically detect and use CUDA when available in torch-sys's build.rs ?

I would be happy to open a PR for it...

The generated code for functions that use a list of tensors, e.g. cat, use the ptr_list function which creates a vec and converts it to a pointer via as_ptr. However the vec is deallocated at the end of ptr_list resulting in a dangling pointer.
The corresponding rust type should also be &[&Tensor] rather than &[Tensor].

I am having some trouble saving and loading VarStores of Sequential models. I have adapted the mnist_nn example in the following way:

extern crate tch;
use tch::{nn, nn::Module, nn::OptimizerConfig, Device};

const IMAGE_DIM: i64 = 784;
const HIDDEN_NODES: i64 = 128;
const LABELS: i64 = 10;

fn net(vs: &nn::Path) -> impl Module {
    nn::seq()
        .add(nn::linear(
            vs / "layer1",
            IMAGE_DIM,
            HIDDEN_NODES,
            Default::default(),
        ))
        .add_fn(|xs| xs.relu())
        .add(nn::linear(vs, HIDDEN_NODES, LABELS, Default::default()))
}

pub fn train(vs: &mut nn::VarStore) -> failure::Fallible<()> {
    let m = tch::vision::mnist::load_dir("data")?;
    let net = net(&vs.root());
    let opt = nn::Adam::default().build(&vs, 1e-3)?;
    for epoch in 1..200 {
        let loss = net
            .forward(&m.train_images)
            .cross_entropy_for_logits(&m.train_labels);
        opt.backward_step(&loss);
        let test_accuracy = net
            .forward(&m.test_images)
            .accuracy_for_logits(&m.test_labels);
        println!(
            "epoch: {:4} train loss: {:8.5} test acc: {:5.2}%",
            epoch,
            f64::from(&loss),
            100. * f64::from(&test_accuracy),
        );
    }
    Ok(())
}

fn main() -> failure::Fallible<()> {
    let args: Vec<String> = std::env::args().collect();
    let mut vs = nn::VarStore::new(Device::Cpu);
    if args.len() < 2 {
      train(&mut vs)?;
      vs.save("weights.pt")?;
    } else {
      vs.load(args[1].as_str())?;
    }

    println!("{:#?}", vs.root());
    Ok(())
}

For cargo run the output is:

[...]
epoch:  197 train loss:  0.19554 test acc: 94.28%
epoch:  198 train loss:  0.19488 test acc: 94.32%
epoch:  199 train loss:  0.19422 test acc: 94.32%
Path {
    path: [],
    var_store: VarStore {
        variables: Mutex {
            data: {
                "weight": Variable {
                    tensor: Tensor[[10, 128], Float],
                    trainable: true,
                },
                "bias": Variable {
                    tensor: Tensor[[10], Float],
                    trainable: true,
                },
                "layer1|bias": Variable {
                    tensor: Tensor[[128], Float],
                    trainable: true,
                },
                "layer1|weight": Variable {
                    tensor: Tensor[[128, 784], Float],
                    trainable: true,
                },
            },
        },
        device: Cpu,
    },
}

However, for a subsequentcargo run weights.pt the output is just:

Path {
    path: [],
    var_store: VarStore {
        variables: Mutex {
            data: {},
        },
        device: Cpu,
    },
}

Shouldn't the output of both be the same?

Maybe this comes from generating the bindings automatically, but is there a use case for expecting &[i64] in Tensor::reshape and similar methods? Maybe it would make sense to change this to &[usize]? This change would also bring Tensor more in line with ndarray, which uses usize to represent shapes.

Hi,

I have tried running the basic examples with: cargo run --example basics on MacOSX and here is the error message that I am getting:

dyld: Library not loaded: @rpath/libmklml.dylib
Referenced from: /Users/vegapit/dev/libtorch/lib/libcaffe2.dylib
Reason: image not found
Abort trap: 6

It seems to be looking for a library that is not available. I downloaded the libtorch 1.1.0 zip file at the following link:

libtorch 1.1.0 for MacOSX

Have I misunderstood the installation process?

At https://github.com/LaurentMazare/tch-rs/blob/master/examples/cifar/main.rs#L45 we have a line of code where we take a Tensor, then multiply every element by 0.125 .

What is the purpose of this line and why do we do this? (Scaling everything by 1/8th seems pointless). Is this some type of magical constant that causes the algorithm to converge faster? If so, where can I read up more on this?