intel-analytics / ipex-llm Goto Github PK

1, rm -r $DIST_DIR/* (if nothing under this folder)
2, cp $BASEDIR/scripts/bigdlvars.sh $BIN_DIR/ (shell file doesn't exist)

Implement a a noop CompressedTensor (that can be used in place of FP16CompressedTensor & FP16SplitsCompressedTensor) that does no compression at all.

We should make Module:backward final, and overload the updategradInput and accGradParameters methods in its subclasses instead.

We should explicitly manage the preferred locations of DataSet, so that they are evenly and deterministically distributed across each node in the cluster

Intel MKL release 2017 version and it contains a DNN API, which provide DNN operation optimized for IA architecture. We will add new layers which leverage these new APIs to get a better performance on CPU.

Test code:

    val x = Tensor[Float](2, 1).fill(2f)
    val y = Tensor(Storage(Array(1f, 2, 3, 4, 5, 6)), 1, Array(2, 3))
    x.expandAs(y)
    val z = Tensor[Float](2, 3).zero()
    z.cmul(x, y)  //will call apply3

Exception:

java.lang.ArrayIndexOutOfBoundsException: 2
    at scala.runtime.ScalaRunTime$.array_apply(ScalaRunTime.scala:76)
    at com.intel.analytics.sparkdl.tensor.DenseTensorMath$$anon$32.apply(DenseTensorMath.scala:60)
    at com.intel.analytics.sparkdl.tensor.DenseTensorApply$.apply3(DenseTensorApply.scala:177)
    at com.intel.analytics.sparkdl.tensor.DenseTensorMath$.cmul(DenseTensorMath.scala:63)
    at com.intel.analytics.sparkdl.tensor.DenseTensor.cmul(DenseTensor.scala:841)

We should add ConcatTable layer into spark-dl to enable ResNet. We can reference torch implementation at https://github.com/torch/nn/blob/master/ConcatTable.lua

I find that when save googlenet model, the model size is 5.7G, while its parameter size should be only Megabytes level. The root cause is that the buffer of the model is also saved to the file. We should mark these buffer field as transient, so it won't be a part of the model persistent file.

Make sure BigDL works for Spark local mode

Why the loop of val set be false while it's true for the training set? what if the remaining val samples less then the batch_size?

val dataIter = validationDataSet.get.asInstanceOf[LocalDataSet[Batch[T]]].data(looped = false)

https://github.com/intel-analytics/BigDL/blob/master/dl/src/main/scala/com/intel/analytics/bigdl/optim/LocalOptimizer.scala#L196

We need to support MulConstant layer to enable ResNet. We can reference torch implementation at https://github.com/torch/nn/blob/master/MulConstant.lua

Run googlenet_v2 with MKL 2017 DNN API. The performance should achieve 80% of intel-caffe with MKL 2017 DNN API. It also should converge to top-1 error(68%) and top-5 error(88%)

Test Inception with DistriOptimizer2 with drop != 0 on performance and convergence

val outputWidth = (inputWidth + 2 * padW - kernelW) / strideW + 1
val outputHeight = (inputHeight + 2 * padH - kernelH) / strideH + 1

Adding checking for these two fields which might be negative if user incorrectly input the parameters.

Exception in thread "main" java.lang.NullPointerException
at com.intel.analytics.bigdl.tensor.DenseTensor.fill(DenseTensor.scala:226)
at com.intel.analytics.bigdl.nn.SpatialConvolution.updateOutput(SpatialConvolution.scala:123)
at com.intel.analytics.bigdl.nn.SpatialConvolution.updateOutput(SpatialConvolution.scala:30)
at com.intel.analytics.bigdl.nn.abstractnn.AbstractModule.forward(AbstractModule.scala:129)
at com.intel.analytics.bigdl.nn.Sequential.updateOutput(Sequential.scala:32)
at com.intel.analytics.bigdl.nn.abstractnn.AbstractModule.forward(AbstractModule.scala:129)
at com.intel.analytics.bigdl.optim.LocalOptimizer$$anonfun$5$$anonfun$apply$1.apply$mcD$sp(LocalOptimizer.scala:117)
at com.intel.analytics.bigdl.optim.LocalOptimizer$$anonfun$5$$anonfun$apply$1.apply(LocalOptimizer.scala:111)
at com.intel.analytics.bigdl.optim.LocalOptimizer$$anonfun$5$$anonfun$apply$1.apply(LocalOptimizer.scala:111)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

We find that when learning rate is large or drop too many module gradients. The loss and activities will become NA.

When defining a model, instead of directly calling the constructors of individual modules, we should use factory methods (e.g., through Scala companion object’s apply methods). This allows one to define the same model that can run using different underlying engine (e.g., MKL-DNN).

Googlenet_V2 Model reference this paper.

A caffe model is here

We should achieve 80% intel-caffe single node performance, and achieve same top-1(68%) and top-5(88%) error compare to GPU version on local mode and cluster mode.

The Storage.size will cause GC and be confused with Storage.length. The performance is also terrible.

1. Engine
   We should implement an Engine object to represent the environment for the training, including:

   • Type of the underlying execution engine: MKL-BLAS vs. MKL-DNN. We can then provide factory functions to automatically create appropriate version of modules (that is, using MKL-DNN or not) based on the specified type; for now, we can throw an error in the factory functions if MKL-DNN is specified but there is no such version of the module.

   • Configurations for distributed training: partition#, worker#, core#, batch size, batchPerPartition, batchPerWorker, batchPerCore, OMP parameters, etc. We can also perform various check on these configurations – e.g., batch size should be a multiple of worker# X core# when using MKL-BLAS

   • Pool of threads for running multiple tasks in a worker: we shouldn’t expose the multi-threading code in the application logic; instead, we can encapsulate the multi-threading code in the Engine object, and its users (such as the Optimizer) can simply call something like

     Engine.parallelInvoke(0 until core#) { i =>
        …
     }

2. Parameters
   We should rename the ps package to parameters, and implement a Parameter class which represents the shared variables (containing both weights and gradients) for both local and distributed modes (somewhat similar to the Broadcast variable). It should provide the following support:

   • User-specified serializer object (e.g., FP16) and update method

   • getWeights/getGradients methods: these should be non-blocking methods that return a FutureResult; the user can then fetch the value through something like FutureResult.getValue(). In this way, the sync weight operations can be overlapped with other operations as follows:

     val W = parameter.getWeights()
     Engine.parallelInvoke(0 until core#) { i =>
         models(i).zeroGradient()
         ...
         W.getValue(localWeights(i))
         …
     }

   • putWeights/putGradients methods for updating the Parameter

3. Optimizer and DataSet
   We should implement the Optimizer which only expose DL related concepts to the users; it can be constructed using DataSet, Module, Criterion, OptimMethod, etc.; on top of it, we can implement LocalOptimizer and DistributedOptimizer that accept LocalDataSet and DistributedDataSet respectively. Inside the Optimizer, it can create Parameter objects (LocalParameter or DistribtuedParameter) to manage the local or distributed training respectively.

Run googlenet_v2 in local mode on spark-dl. It should achieve 80% intel-caffe performance and convergence to top1-acurracy(68%) and top-5-accuracy(88%).

We should not restrict BigDL to only accept Image as input

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: input image smaller than kernel size
at scala.Predef$.require(Predef.scala:233)
at com.intel.analytics.bigdl.nn.SpatialMaxPooling.updateOutput(SpatialMaxPooling.scala:55)
at com.intel.analytics.bigdl.nn.SpatialMaxPooling.updateOutput(SpatialMaxPooling.scala:30)

Change GoogleNet to Inception

We can use Conv2D for now, but it might be more straight forward and optimized with 1D implementation.

We need to support resnet in spark-dl, including single node performance test and tunning. Test convergence on single node and multi-node.

We can start with ResNet50. The model topology should reference facebook torch model and caffe torch model. The only difference between these two models are some convolution stride parameter.

The single node performance goal is to achieve 80% of intel-caffe.

The convergency goal is to achieve the same error rate(top-1 and top-5) on the refrence model.

Change all "looped" parameters to "train"

We need to support Identity layer in Spark-DL to enable ResNet. We can reference torch implementation at https://github.com/torch/nn/blob/master/Identity.lua

Currently we only provide Reshape layer which require user to manually calc the size from the previous layer.

It would be better if we can have a similar layer like Keras:
x = Flatten()(x)

nelements_pre_layer/batch is the size user need to give if we rely on the current Reshape layer.

1. change the folder name to ImageClassification
2. change googlenet to inception
3. change utils.File to utils.TorchFile
4. support the case that the input images have no lables
5. converge the local mode and spark mode implementations

Converge the implementations of example code for local and Spark modes

Instead of calling dataset.data(train = true) in each iteration, call dataset.data(train = true) for each epoch in DistriOptimizer instead

Run googlenet_v2 in spark cluster mode on spark-dl. It should achieve 80% intel-caffe performance and convergence to top1-acurracy(68%) and top-5-accuracy(88%).

Batching is quite a common logic, it might be good if user don't need to be bother by that.
We might be able to hide that from user or just give a general Batch transformer.

Ideally What the user need to provide is just a simple Iterator[Sample], we take care of the batching shuffling internally(or this kind of logic can be a out-of-box component which can be picked up easily).
i.e
Iterator[Sample] -----batching---shuffling--> training/validation

Related code:
https://github.com/intel-analytics/BigDL/blob/0b095036ef3e3b45913d0209ee617e7836e40974/dl/src/main/scala/com/intel/analytics/bigdl/dataset/image/RGBImgToBatch.scala#L33

In SpatialConvolution, weightMM is the view of weight and won't be updated until it is empty.

When calling getParameter fuc, Module.flatten will change the memory address of weight .

So if we call forward/backward and get weight by getParameter, weight's change won't affect nonempty weightMM.

There is a need to change weightMM updated rule.

There are lots of require statement here but without enough info for debugging.
i.e require(1 == this.nDimension, "invalid size")
A better version:
require(1 == this.nDimension, s"nDimension size: ${this.nDimension} should be 1")

Don't specify Float or Double for our examples/models - using import of implicits instead, so that scala can automatically figure out the right type.

Test Inception with DistriOptimizer2 with drop = 0 on performance and convergence

https://github.com/intel-analytics/BigDL/blob/master/dl/src/main/scala/com/intel/analytics/bigdl/dataset/image/RGBImgToBatch.scala#L46

We support use different dataset(imagenet, cifar10 and mnist) to train and test the model. But the code is hard to maintain. We should do some code refactor.

The target is unify the dataload, transform function of different dataset in local or spark cluster mode.

We need to support CrossEntropyCriterion layer to enable resnet. We can reference the torch implementation at https://github.com/torch/nn/blob/master/CrossEntropyCriterion.lua.

We should add CAddTable layer into spark-dl to enable ResNet. We can reference torch implementation at https://github.com/torch/nn/blob/master/CAddTable.lua

The default Apache Spark build doesn't include netlib-java because of license problem. We use netlib-java for some blas operation. So the lib perf will be very bad on the default Apache Spark build. Should we migrate all of the blas operation to a custom blas wrapper?

1. Instead of use DataSet in the driver side, we should use RDD or DataFrame only in driver
2. In DLClassifier.process, we may transform the Iterator[Row] to LocalDataSet in mapPartition
3. In DLClassifier.process, we cannot share models between different partitions; we need to clone a new version of localModel in mapPartition
4. Move spark.ml.DLClassifier to the utils folder

With the test of GoogLeNet v1 on sparkdl, there may be a performance issue on the Concat layer.

The time distribution of an iteration is,

• total, 1471.642277ms
• forward, 333.476ms
• backward, 475.903ms

The other time (not forward and backward) is 662.351. That almost is the costs of Concat layer.

In the WebscaleML, I implement the concat copy with some repeated and dirty code, which is one special case for apply2 for DenseTensorApply.
When the tesor1stride and tensor2stride are all 1, it will use System.arrayCopy instead continuous one byte copy.