Fenzo holds all resource offers of slaves that have tasks assigned about fenzo HOT 8 CLOSED

Hello liulong0118, what quantity of resources are unused on the host after assigning some tasks? I will try to reproduce that and get back to you.

from fenzo.

Oh, also, can you confirm that your framework is calling Fenzo's TaskScheduler.scheduleOnce(...) routinely, even if there are no tasks to assign? See my explanation in issue #51

from fenzo.

Hi spodila. Thanks for your reply.

Yes, my framework is calling scheduleOnce(...) routinely even if there are no tasks to assign, just like what the sample framework does.

My slave has 8 CPUs and 14.4GB RAM, and my task takes 1 CPU and 1GB RAM. After one task is assigned to the slave, Fenzo holds all CPUs and RAM (1CPU + 1GB used and 7CPU + 13.4GB offered) until the task finishes.

Actually, I've found the piece of code that makes this happen. In doSchedule(..) method of TaskScheduler, Line 604, it has the following code:

if(!avm.hasPreviouslyAssignedTasks())
    idleResourcesList.add(avm.getCurrTotalLease());

So, if the slave has previously assigned tasks, then the leases of this slave will not be added to the idleResourcesList. As a result, these leases will not participate in removeLimitedLeases on Line 611. Therefore, these offers will not be rejected.

from fenzo.

That's a great catch. The list of leases was prepared correctly for AutoScaler, but it is not totally right for limited offer rejections.

I created a fix for it and checked it in. fenzo-core 0.7.12 should be available soon in maven central.

Earlier today, I had created 0.7.11 that gives you control on how many offers to reject within a period of leaseOfferExpirySecs time.

Let me know if how your framework does with 0.7.12 version.

from fenzo.

Just tried fenzo-core 0.7.12 with my framework. The fix works well. Offers are rejected as expected. Thanks!

I'll close the issue then.

from fenzo.

Glad to hear it works for you.
BTW, now there is a Google Group for questions and discussions as well. Join at https://groups.google.com/forum/#!forum/fenzo

from fenzo.

In com.netflix.fenzo.AssignableVMs#removeLimitedLeases there is this comment:
// randomize the list so we don't always reject leases of the same VM before hitting the reject limit
Don't we want to free up large chunks of contiguous space on a single VM, rather than fragmenting space across lots of machines?
Also, that mailing list appears to be empty. Is there another?

from fenzo.

Good observation.
The randomization itself only impacts the list of hosts form which offers are selected to reject. An improvement to address the issue you bring up can either take the form of

• rejecting all offers from a host selected (and therefore the limit on leases to reject effectively becomes the limit on #hosts to reject from), or,
• sorting the offers to reject by highest percentage of the total host resources they represent and rejecting the "top N" of them.

The latter seems a bit excessive in terms of computation cost, besides being not fully achievable since entire resource sizing may be unavailable if a host's resources are offered to multiple frameworks. I would be tempted to pick the former. Additional thought may be needed before committing on the former.

Is this what you in mind?

Feel free to open a new (improvement) issue for this.

from fenzo.