Performance of innerOuterPersistent scheduler is very sensitive to register usage and spills about fuser HOT 3 OPEN

attached code from main branch and main branch without promote hoisting of vectorize predicates.
high_perf.txt
low_perf.txt

from fuser.

Generated with:

NVFUSER_PROF=print NVFUSER_DUMP=cuda_to_file,ptxas_verbose,scheduler_params,launch_param pytest /opt/pytorch/nvfuser/benchmarks/python/test_layernorm_bwd.py -k "float16 and 16384_16384" -v -s  --disable-benchmarking 2>&1 |tee 1.log

from fuser.

The possible reason of regression may because of compiler optimization from cuda to ptx code.
The slow version has 2500 more lines of PTX code.
In the fast version , #2742 ,b47 is used in both if and else branches. Compiler can optimize out the else branch, a very short PTX code generated for else branch.

In the slow version (current main), b47 is only used in the else branch. The compiler generates PTX code for the else branch. That’s why we see 2500 more lines of PTX code.

PTX compare:

slow.ptx.txt
fast_revert.ptx.txt

from fuser.