Benefit of the gradients on the 2-dimensional Rosenbrock function about smt HOT 9 CLOSED

Hello Paul,

Thank you for your answer.

I totally understand your point about dimension reduction, but I was more interested in observing the contribution of the derivatives.

I chose a relatively small size of the training sample on purpose: I wanted to measure the contribution of the derivatives when the data is scarce. I hoped that the derivatives could compensate the data scarcity to some extent.
I tried to double the size of the training sample for the 2-dimensional Rosenbrock function: KPLS becomes more accurate than GEKPLS.

It's counterintuitive to me that the derivatives worsen the accuracy. (Maybe it comes from ill-conditioning.)

Thank you for the 5-dimensional example. I observe that we gain only 5% of accuracy at the cost of computing 50 gradients.

I take note that GEKPLS requires an effective dimension reduction.

Cheers,
Benoît

from smt.

Hi Benoit. Maybe rosenbrock is not the best function to make GEK shine. The surface changes smoothly and plain kriging fits it basically pretty well and derivatives does not bring much more information (at least it does not worsen the prediction 😅 ). I would say that benefits should be seen with a function with more strong changes between training points.

from smt.

Hi Rémi,

Thank you for your answer.

On the figure attached to my previous message we can see that the median of the relative L2-errors of both KPLS and GEKPLS is about 50%. So neither KPLS nor GEKPLS seem to fit the Rosenbrock function pretty well.

You will find below a graph of the KPLS model that achieves median accuracy and the GEKPLS model trained at the same inputs (in orange, they look almost the same) and the Rosenbrock function (in blue).

As you can see the KPLS model does not fit the Rosenbrock function very well, there is clearly room for improvement. Adding the derivatives to the training data with GEKPLS does not really improve the accuracy.

Isn't it surprising? I would expect the derivatives to bring a lot of information, especially for such a smooth function.

Thank you for your time,
Benoît

from smt.

We have detected some problems in GEKPLS in SMT GitHub master, not present in SMT 1.3. Do you use SMT 1.3?

from smt.

Yes, I use SMT 1.3.

from smt.

Hello Benoit,

• KPLS and GEKPS are meant to reduce the dimension of the problem. Here, you chose n_comp=2 but Rosenbrock(ndim=2). Therefore, the PLS matrices are just the identity matrix with numerical errors for both KPLS and GEKPLS.

• You chose to build a model with only 10 points (sample_size=10). But the Rosenbrock function grows quick on the sides. Therefore, if you have no points in that zones, the model just can't predict it.

=> What you are comparing is therefore a prediction error related to the sample size and some noise from an ill-conditioned PLS matrix.

If you chose a reduction of the model size from Rosenbrock(ndim=5) to n_comp=2 and if you chose a sample size of 50 points you would obtain the result below.
PredictionAccuracy(Rosenbrock(ndim=5), validation_size=1000).plot(
sample_size=50,
number_of_trainings=20,
n_comp=2,
n_start=20,

So GEKPLS helps but need a certain amount of points and an effective reduction of the dimension for the model.

from smt.

Hi,

I totally understand your point. What you want to compare is Kriging vs Gradient-Enhanced Kriging (GEK). In this case, we should obtain better performances with GEK, you are totally right !

Unfortunately, GEK is not implemented in SMT (it has a lot of limitations for a large number of points or an high-dimensional problem).

You are also right for the bad accuracy, as we add more badly approximate direction with GEKPLS, the numerical errors should add up and the accuracy decrease.

In fact, GEKPLS is based on the PLS-reduced principal components derivatives. In this case, not only the PLS directions are ill-conditioned but it do not corresponds to GEK as GEK do not use dimension reduction at the price of being more expensive.

https://arxiv.org/pdf/1708.02663.pdf

from smt.

Thank you Paul for the explanation.
Cheers,
Benoît

from smt.

Thanks Paul!

from smt.