Workloads (Redis, Curl, R) failing with Out of memory PAL error after new manifest syntax to define lists of SGX trusted files. about graphene HOT 7 CLOSED

@jinengandhi-intel Could you attach a redis-server.manifest.sgx final manifest file? I wonder what is so special about RHEL/CentOS.

I assume that the difference is in how this final manifest file is generated on Ubuntu vs on RHEL/CentOS.

from graphene.

@jinengandhi-intel Also could you attach redis-server.manifest.sgx file in Ubuntu (where it doesn't fail)?

I want to look at two versions of this redis-server.manifest.sgx file: one in Ubuntu, one in RHEL. Comparing them side by side, we may spot the difference, which will be the root cause of this failure.

from graphene.

Manifest SGX file for Ubuntu are attached here. For RHEL, I am awaiting the manifest files from my colleague
R.manifest.sgx.txt

curl.manifest.sgx.txt

from graphene.

Please find the manifest.sgx files for RHEL attached.
curl.manifest.sgx.txt
R.manifest.sgx.txt
redis-server.manifest.sgx.txt

from graphene.

RHEL manifest files are ~10MB in size... This feels like way too much for the initial 64MB pre-allocated by Graphene.

@aniket-intelx @jinengandhi-intel Can any of you run the failing workload (e.g., redis-server) on RHEL under GDB and find the exact place where the out of PAL memory error happens? I would assume that it happens here: https://github.com/oscarlab/graphene/blob/33a68bc4302891e8c591570942bc39394acefd23/Pal/src/host/Linux-SGX/db_main.c#L683

from graphene.

@dimakuv
Before the mentioned commit the RHEL manifest files for curl is ~16MB in size and it did worked.

@aniket-intelx will be sharing the rest of the details soon

from graphene.

We debugged and the problem is in Graphene's pre-allocated internal PAL memory pool of 64MB.

We fail on toml_parse():

But we read loader.pal_internal_mem_size (which increases the internal PAL memory) only after parsing:

So we get a chicken-and-egg problem.

Easy solution: if we detect that the manifest size is greater than some threshold (I recommend 1MB), then we immediately increase internal PAL memory by additional 64MB.

from graphene.