This is because the way Lua sets these values is from the largest index to smallest index - presumably to optimise performance. But we need to do the reverse when setting ravi arrays as we only grow the array if the len+1 element is set.

When auto compiling all Lua functions, the memory usage increases greatly during the running of Lua tests. The problem has become apparent when running tests with -port=true; previously I was running tests with -U=true.

The issue seems to be because in JIT mode each compiled function has associated with it the LLVM module, execution engine plus the generated code. But Lua's garbage collector is unaware of the size of the function objects - hence the rate at which these objects are collected / finalized is not fast enough.

Putting a call to lua_gc(L, LUA_GCSTEP, 200) seems to help but then causes tests in gc.lua to fail.

For now I have put in a call to lua_gc(L, LUA_GCCOLLECT, 0) after every n JIT compilations - n can be configured. This is crude but appears to limit the amount of memory being used. A better solution is required - one that doesn't cause a full GC.

On Windows there is occasionally a crash when Lua attempts a Longjmp because an error has been raised. Windows complains about misaligned or invalid stack.

The error does not occur on Linux or MAC OSX (Yosemite).

This is because the check on function type is not differentiating between C and Lua functions.

To remind myself

In Lua test events.lua line 395 there is an expression such as:

(10)[3]

This is incorrectly identified as of type integer - as the sub-expression's type is propagated.

Thanks to the fact that via debug api even primitive types can have a metatable assigned - above snippet is actually valid Lua code.

local x: int[] = { 1 }
local y: int = x[1]

Slices will allow efficient access of subset of arrays, for example, each matrix column of a matrix could be accessed via slice.

Initial work checked in - outstanding issues:
a) Need to move the API to Ravi module
b) After a slice has been created if the original table array is reallocated the slice will be invalid. Simple solution is to disallow size change after slices are created

This will be an alternative to the existing LLVM JIT compiler - and both implementations will be kept in sync.

At present following fails:

local arry : int[] = { 1, 2 }

Following works:

local arry : int[] = {}

It would be nice to see the final machine code produced by LLVM after JITting a given function.

Performance can be improved by generating different opcodes for the common case of integer for loop where the index is a constant and therefore known to be <0 or >0.

The following opcodes are misbehaving in JITed code:

OP_RAVI_TOARRAYI
OP_RAVI_TOARRAYF
OP_RAVI_MOVEAI
OP_RAVI_MOVEAF

There are two bugs - first wrong variable being used in comparison, secondly the type of a table is LUA_TTABLE | BIT_ISCOLLECTABLE = 69. So the type check always fails.

Currently we hold on to the module and EE even after generating code for a function. It is possible to release these as described in referenced thread below:

http://lists.cs.uiuc.edu/pipermail/llvmdev/2015-February/082051.html

Compiled code should be treated as C functions and skipped over

Reminder to myself

When values are assigned to arrays the type check is missing so any garbage can be assigned

Need to ensure that C code does not break arrays.

At the moment there are some issues.

remove() fails as it attempts to insert a nil value.

If small constant values we can embed the value in the instruction when the constant is being added to or multiplied by a register

A Lua binding for LLVM will make the power of LLVM accessible to Lua programmers. This binding needs to work for Lua 5.3 hence we cannot use Ravi specific features.

Initially the binding will be part of Ravi but needs to be built so that the binding can be loaded in Lua programs without having to use Ravi.

Work has started on this - please see:
https://github.com/dibyendumajumdar/ravi/tree/master/llvmbinding

Current implementation represents values as unions similar to Lua, however this may be causing issues in generating optimized code. Need to try out the struct based approach used for LLVM

Lua 5.3 uses a separate type code field in values to identify types. This means that when performing arithmetic operations on numbers (doubles) there is a need to update two fields, the value and the type.

The overhead of updating the type field can be reduced for floating point operations by using the NaN tagging approach. Basically we can split the value into two parts.

The first part will be a double - holding either doubles or a type code encoded in a NaN.
The second part will be the actual value.

I misunderstood the tbaa metadata usage - it seems that when loading or storing the primitive types the metadata should be set to path node. e.g.

store double %3, double* %5, align 8, !tbaa !11

!11 = metadata !{metadata !12, metadata !12, i64 0}
!12 = metadata !{metadata !"double", metadata !3, i64 0}

You are implementing opcodes and changing the code. Something can be accidentally broken. Since Lua itself and Ravi have some tests, it worth using a continuous integration platform for this project.

You can use lua-travis-example as an example.

Possible solution is to use Mike Pall's Coco package.

We don't really need Register+Konstant and Konstant+Register as one of them is enough.

local aa: int[] = print()

Failed to compile but should have compiled and given a runtime error.

The main issue is that the 'savedpc' in the call frame is not updated in JIT code. Updating this on every bytecode instruction would inhibit optimization. However one potential solution is to update it when one of following happens:
a) OP_CALL or OP_TAILCALL is invoked
b) A metamethod is invoked
c) Apart from above we also need to handle hooks. Since this is a large overhead (function call) we need to only enable this conditionally

At present the parser does not support typed parameters in functions.

To fix the issue with upvalues subverting static types new opcodes have been introduced - these are type specialized versions of OP_SETUPVAL. These opcodes are not yet JIT compiled

You currently seem to hardcode the LVVM optimization level:

llvm::PassManagerBuilder pmb;
pmb.OptLevel = 1;
pmb.SizeLevel = 0;

May be it is a good idea to provide a command-line parameter or some other means of configuring it?
Have you tried with OptLevel=2 or OptLevel=3? I can imagine that require more time for JITting, but may be they produce a much better code?

This will enable us to use type specific operators in more scenarios - right this degenerates to ANY type

function x() local i: integer; return function(j) i = j; end; end
f=x()
f(5)
f('hello')

Currently all comparison ops call a function which means that comparisons are inefficient and I think they particularly affect the performance of benchmarks such as fannkuchen.

When types are known and both types are the same (common case) we should be able to generate more specialised comparison opcodes that can be inlined

The type of a logical operators and/or is determined based on the arguments - so this poses a problem when determining the return type statically.

Possible solutions:
We could specialise for situations where both operands of the boolean operator are of known type - because in this case the resulting type is going to be the same as that of the operands. However, when either operand is of a different type - only if the comparison is between constants can the resulting type be determined at compile time

data = {}
local n: integer = #data

Above should work

See localvar_explist() and localstat()

When I look at the dumped LLVM IR, it looks pretty inefficient.

Let me try to explain what I mean:

One of the main reasons for it seems to be the fact that Ravi currently very closely models how LuaVM works with a stack. As a result, Ravi JIT always loads/stores operands of any Opcode to from/to memory, which prohibits many LLVM optimizations. While this is required upon entering a JITted function or leaving it (e.g. calling into a non-JITted code or returning), I'm not sure it is required inside the function being JITted.

Overall, I think it is important to closely model LuaVM stack only when there is a chance it is being "observed" from outside of a JITted function and the stack is expected to be exactly as if it was prepared by LuaVM.

But if you have a temporary value or a local variable which is not "observable" from outside, you are free to work with it without mapping it to a LuaVM stack. You can most likely simply model it as an "alloca" (of a TValue?) in LLVM IR and perform operations directly on it.

Eventually, you may even try to map all incoming LuaVM values to local LLVM "alloca" upon entering a function and store them back to the LuaVM stack upon leaving a function.

IMHO, keeping values in LLVM IR %-values would allow LLVM to perform common-subexpression-elimination, hoisting of values out of loops, load-store optimizations, copy forwarding and much more.

May be my view is rather naive and I miss an obvious reason why this cannot be done and one should always closely model the LuaVM stack inside the JITted function.

LLVM version 3.6.0
Windows 8.1 64-bit
VS 2013

The crash occurs in following code in calls.lua:
function err_on_n (n)
if n==0 then error(); exit(1);
else err_on_n (n-1); exit(1);
end
end

do
function dummy (n)
if n > 0 then
assert(not pcall(err_on_n, n))
dummy(n-1)
end
end
end

-- FIXME - this causes a fault
dummy(10)

To remind myself