wenwei202 / terngrad Goto Github PK

when I run Imagenet， Error in `/home/anaconda2/envs/tensorflow/bin/python': : double free or corruption (!prev): 0x0000000000c6c780

I do such step :
i). Download and generate ImageNet TFRecord
I do it like you say and get TFrecord 1024 train and 128 validation
ii). Build and run ImageNet
I do it like you say:
cd ${TERNGRAD_ROOT}/terngrad
bazel build inception/imagenet_train
bazel build inception/imagenet_eval

bazel-bin/inception/imagenet_train
--optimizer momentum
--net alexnet
--image_size 224
--num_gpus 2
--batch_size 256
--train_dir /tmp/imagenet_train
--data_dir ~/dataset/imagenet-data/

but at this step ,
Error in `/home/anaconda2/envs/tensorflow/bin/python': : double free or corruption (!prev): 0x0000000000c6c780.

I would appreciate for your answer. thank you

When I ran cifar10_alexnet on cifar10 dataset, I used terngrad and set FLAGS.floating_grad_epoch=0 (leave other arguments default), achieved top_1=86.04 after 300000 steps.
And then I want to see original accuracy without terngrad and set FLAGS.floating_grad_epoch=1,but I got top_1=85.64 after 300000 steps, which is not consistent with results in paper. I want to make sure that I used in a correct way.

By the way, if I want to test it on other models like densenet, what arguments should I modified? (weight decay? base lr?)

when eval
bazel-bin/inception/imagenet_eval
--data_dir ~/dataset/imagenet-data/
--net alexnet
--image_size 224
--batch_size 50
--checkpoint_dir /tmp/imagenet_train
--restore_avg_var True
--eval_dir /tmp/imagenet_eval

error : ExponentialMovingAverage not found in checkpoint

I'm interested in terngrad and want to try it in my own project. I want to make sure the meaning of ternarize in your paper, but I can't find it in code.

Can u specific it for me? Which part of code is for formula (1) of your Terngrad paper?

On this code:

I try in caffe.
Alexnet ,single worker. I do like this:
1). ternarize all gradients in convolution layers and fc layers
2). ternarizing method: a). m=diff/diff_max(among one layer), b) generate random num : rand_blob_ ∈（0，1） c).compare m and rand_blob_ : if( m >rand_blob_[i] ) diff_[i] = diff_max; else diff_[i]=0

when training , first 500 epoch,it behaves well ,but than,it keep from ~6.1 to ~6.7 in the later 5000 iters, then I stop training.

Because I want to train in resnet or vgg or other nets , I want to know whether the code I change is right.
Thanks！

If i want to run terngrad on CIFAR10 dataset for distributed system (for example: 2 worker machine, 1 parameter server, i.e, 3 machines), can you please describe the steps to execute? In readme section, it shows workers under the same localhost, but I need to run them on different machines with different IP addresses.
Thank you.

Since -1,0,1 are still represented using float32, how you guys achieved speedup as shown in the experiment of the paper?

@wenwei202
Could help me to understand your code?
The steps to ternarize gradient is as below:

1. Find the scaler for a layer, ex. 0.77370787
2. Turn the gradient into (0, -scaler, scaler) by formula(1), ex.
   array([[ 0. , -0.77370787, 0.77370787, 0.77370787, 0.77370787],
   [ 0. , 0. , 0. , 0. , 0.77370787],
   [ 0. , 0. , -0.77370787, 0.77370787, 0.77370787],
   [ 0. , 0.77370787, 0. , 0.77370787, 0. ],
   [ 0.77370787, 0. , 0. , 0.77370787, 0. ]], dtype=float32)
3. encode gradient into scaler and 3-level array
4. Send scaler and 3-level array to parameter server

Is that right? Thanks a lot!!

Can we install the official TensorFlow binary version and run your code based on it? Does that mean you changed the source code of TensorFlow? I find that your algorithm just inserts some tensor operation in the computation graph, but have not changed the source code of TensorFlow.

"Currently, distributed-node mode only supports 32bit gradients. It will take a while to hack the highly-encapsulated SyncReplicasOptimizer to integrate TernGrad. Keep updating"
-----mentioned in the doc

hi,wenwei, thanks for the amazing job. any progress or plan about this job?
thanks!

my pc is mac ， can terngrad run with cpu only?

$ ./build_all.sh
bazel: /home/wew57/anaconda3/lib/libstdc++.so.6: version GLIBCXX_3.4.21' not found (required by bazel) bazel: /home/wew57/anaconda3/lib/libstdc++.so.6: version GLIBCXX_3.4.20' not found (required by bazel)
bazel: /home/wew57/anaconda3/lib/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by bazel)