hikettei / cl-waffe Goto Github PK

View Code? Open in Web Editor NEW

48.0 5.0 2.0 5.18 MB

[Deprecated] Common Lisp Differentiable Matrix Operations for Mathematical Optimization.

Home Page: https://hikettei.github.io/cl-waffe-docs/

License: MIT License

Common Lisp 71.72% Jupyter Notebook 25.13% Python 1.87% Shell 0.06% Swift 1.12% Metal 0.10%

commonlisp deep-learning blas

cl-waffe's Issues

!argmax/!argmin requires tensor to be copied.

In the function trainer:

(defun valid (trainer dataset batch-size)
  "Valid trainer"
  (let ((count 0)
	(correct 0))
    (loop for index fixnum upfrom 0 below (get-dataset-length dataset) by batch-size
	  do (let* ((ds (get-dataset dataset index))
		    (x (car ds))
		    (y (second ds))
		    (out (const (value (call (slot-value trainer 'model) x))))
		    (out-labels (!argmax out))
		    (y-labels   (!argmax (const (copy-mat (data y)))))) ; <--- Here
	       (with-facets ((out-labels ((data out-labels) 'backing-array :direction :input))
			     (y-labels   ((data y-labels) 'backing-array :direction :input)))
		 (loop for i below (length out-labels)
		       do (progn
			    (incf count 1)
			    (if (= (aref out-labels i) (aref y-labels i))
				(incf correct 1)
				(incf correct 0)))))))
	     (format t "Accuracy:~a~C" (coerce (/ correct count) 'float) #\newline)))

This creates y's copy every time otherwise y-labels' value is not updated.

I guess this is due to accessing the wrong facets or something like that.
hmm

Gradient explosion of Seq2Seq (possibly related to linearlayer/embedding)

(defmodel Encoder (vocab-size embedding-dim hidden-size)
  :parameters ((embedding (Embedding vocab-size embedding-dim :pad-idx 0))
               (layer     (RNN embedding-dim hidden-size :num-layers 1)))
  :forward ((x)
            (with-calling-layers x
                (embedding x)
                (layer x))))

(defmodel Decoder (vocab-size embedding-dim hidden-size)
  :parameters ((embedding (Embedding vocab-size embedding-dim :pad-idx 0))
               (layer     (RNN embedding-dim hidden-size :num-layers 1))
               (h2l       (linearlayer hidden-size vocab-size)))
  :forward ((encoder-state y)
            (let* ((ye (call (self embedding) y))
                   (hs (call (self layer) ye encoder-state))
                   (h-output (call (self h2l) hs)))
                  h-output)))

(defmodel Seq2Seq (vocab-size-lang1 vocab-size-lang2 embedding-dim input-size)
  :parameters ((encoder (Encoder vocab-size-lang1 embedding-dim input-size))
               (decoder (Decoder vocab-size-lang2 embedding-dim input-size)))
  :forward ((x y)
            (let ((x-state (call (self encoder) x)))
                 (call (self decoder)
                       x-state
                       (!aref y t '(0 -1))))))```

こんな感じのSeq2Seqがあって、

```lisp
(setq model (Seq2Seq 1000 1000 100 100))

vocab_sizeを1000以上で初期化して、学習を進めると2~3回で勾配爆発を起こす

LinearLayer/Embeddingの重みの初期値は適切だと思うんだけど、どこが悪いんだぁ〜ぐあぁ〜〜

Split large files into small

The total lines of files like ‘tensor. lisp’, ‘operators. lisp’ is too large and for example algorithms like sampling beta shouldn't write in that file.

Move them into /distributions, activations.lisp, copy.lisp

The backpropagation of !split is undefined.

To Do: define it with using lazy-evaluation

Support GPU

Almost features are based on mgl-mat so it is not going to be a rocky road.
If possible, I want backends to support FP16

TODO

~~Optimize the overhead of generic functions call-forward/call-backward (Less Important)~~ It's done.
~~make print-object rich~~(Done)

Save and restore the weights of model

Allowing other options to use matrices in cl-waffe

As of this writing, all matrices in cl-waffe is a type of mgl-mat, even if I enter simple-array, cl-waffe coerces it to mgl-mat. However, The only part that depends on mgl-mat is the standard implementation. Considering with-backend allows to use other matrix operation libraries, it would be more reasonable not to restricted the datastructure of matrix.

To add: Dispatch nodes based on a given tensor’s backend

sorry for being written in Japanese...

defnode-backendみたいなマクロを作る。←このマクロはdefnodeで定義されたマクロのバックエンドを拡張する。

バックエンドはシンボルで管理されていて、(display-backend)関数で定義されたバックエンド一覧とノードの数を表示する。

define-node-methodのcall-forward/call-backwardでカーネルを割り当てる。

カーネルが存在しない場合はcl-waffe標準の:mglカーネルで計算。

そのためテンソルの型などは変更しない、他のライブラリで使うときはfacetを作ってもらう。

→numericals等他のライブラリに拡張できるようにする。
→他のバックエンドのバージョンを:cl-waffe-openclみたいに別のライブラリとして書けるようにする

重みの初期化手法について

(setf !aref) is ugly

To update the computation nodes, (setf !aref) needed to be called like:

(setq tensor (setf (!aref tensor ~) x))

This should be rewritten with macro, being easy to cause mistakes.

call-forward, call-backward is reducible

(defgeneric call (self &rest inputs))

みたいに書き換えればいいのでは・・・

Add: (with-config ~) macro

Tensorデータ型やBackendは全てdefparameterを参照しているから

(with-config (:dtype :float)
             (:backends (:simd :numcl :mgl)) ; Backendの優先順位
   ...)```

みたいなマクロでまとめて設定できると便利そう。FP16対応する方法思いついたら作ろっかな〜

Sampling beta distributions is failed with alpha=1.0, beta=1.0

(!beta `(10 10) 1.0 1.0)
;arithmetic error FLOATING-POINT-INVALID-OPERATION signalled
;Operation was (/ 0.0 0.0).

後でなおす

hikettei / cl-waffe Goto Github PK

cl-waffe's Issues

Recommend Projects

Recommend Topics

Recommend Org