mjanicek / rembulan Goto Github PK

I get this exception:

net.sandius.rembulan.exec.CallException: net.sandius.rembulan.lib.BadArgumentException: bad argument #1 to 'ipairs' (table expected, got light userdata)

	at net.sandius.rembulan.exec.DirectCallExecutor$Result.get(DirectCallExecutor.java:184)
	at net.sandius.rembulan.exec.DirectCallExecutor.execute(DirectCallExecutor.java:310)
	at net.sandius.rembulan.exec.DirectCallExecutor.resume(DirectCallExecutor.java:252)
	at net.sandius.rembulan.exec.DirectCallExecutor.call(DirectCallExecutor.java:228)
	at com.jukusoft.mmo.engine.applayer.script.lua.LuaScriptEngine.execFunc(LuaScriptEngine.java:77)
	at com.jukusoft.mmo.engine.applayer.script.lua.LuaScriptEngineTest.testExecGlobalFunc(LuaScriptEngineTest.java:102)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: net.sandius.rembulan.lib.BadArgumentException: bad argument #1 to 'ipairs' (table expected, got light userdata)
	at net.sandius.rembulan.lib.impl.ArgumentIterator.badArgument(ArgumentIterator.java:117)
	at net.sandius.rembulan.lib.impl.ArgumentIterator.nextStrict(ArgumentIterator.java:181)
	at net.sandius.rembulan.lib.impl.ArgumentIterator.nextTable(ArgumentIterator.java:354)
	at net.sandius.rembulan.lib.impl.DefaultBasicLib$IPairs.invoke(DefaultBasicLib.java:380)
	at net.sandius.rembulan.lib.impl.AbstractLibFunction.invoke(AbstractLibFunction.java:36)
	at net.sandius.rembulan.runtime.AbstractFunctionAnyArg.invoke(AbstractFunctionAnyArg.java:31)
	at net.sandius.rembulan.runtime.Dispatch.mt_invoke(Dispatch.java:64)
	at net.sandius.rembulan.runtime.Dispatch.call(Dispatch.java:199)
	at engine0$0.run(script_add:3)
	at engine0$0.invoke(script_add)
	at net.sandius.rembulan.runtime.AbstractFunction1.invoke(AbstractFunction1.java:57)
	at net.sandius.rembulan.runtime.Dispatch.mt_invoke(Dispatch.java:95)
	at net.sandius.rembulan.runtime.Dispatch.call(Dispatch.java:402)
	at net.sandius.rembulan.runtime.Coroutine$BootstrapResumable.resume(Coroutine.java:89)
	at net.sandius.rembulan.runtime.ResumeInfo.resume(ResumeInfo.java:35)
	at net.sandius.rembulan.runtime.Call$Resumer.continueCurrentCoroutine(Call.java:513)
	at net.sandius.rembulan.runtime.Call$Resumer.resume(Call.java:537)
	at net.sandius.rembulan.runtime.Call.resume(Call.java:232)
	at net.sandius.rembulan.runtime.Call.access$000(Call.java:37)
	at net.sandius.rembulan.runtime.Call$CallContinuation.resume(Call.java:181)
	at net.sandius.rembulan.exec.DirectCallExecutor.execute(DirectCallExecutor.java:290)
	... 28 more

What does this exception means?

Source code:

public class LuaScriptEngine implements IScriptEngine {

    protected StateContext state = null;
    protected Table env = null;

    //map with all registered global lua functions
    protected ObjectObjectMap<String, LuaFunction> luaFunctions = new ObjectObjectHashMap<>();

    public LuaScriptEngine () {
        //initialize state
        this.state = StateContexts.newDefaultInstance();
        this.env = StandardLibrary.in(RuntimeEnvironments.system()).installInto(state);

        //register global functions
        this.env.rawset("logd", new LogDebugFunc());
        this.env.rawset("logi", new LogInfoFunc());

        this.env.rawset("now", new NowFunc());
    }

    /**
     * compile lua script as string to function
     *
     * //@param rootClassPrefix  the class name prefix for compiled classes, must not be {@code null}
     */
    @Override
    public void compile (String scriptName, String programStr) throws ScriptLoadException {
        Log.d("Scripts", "compile script '" + scriptName + "'");

        scriptName = "script_" + scriptName;

        //compile
        ChunkLoader loader = CompilerChunkLoader.of("engine");
        LuaFunction func = null;
        try {
            func = loader.loadTextChunk(new Variable(env), scriptName, programStr);
        } catch (LoaderException e) {
            Log.e("Scripts", "Couldn't compile lua script: " + programStr + ", error: " + e.getLuaStyleErrorMessage(), e);
            throw new ScriptLoadException("Coulnd't compile script: " + programStr + ", error: " + e.getLuaStyleErrorMessage());
        }

        this.luaFunctions.put(scriptName, func);
    }

    @Override
    public Object execFunc(String funcName, Object... args) throws CallException {
        // get a reference to the function
        LuaFunction func = (LuaFunction) env.rawget(funcName);

        if (func == null) {
            throw new NullPointerException("lua function cannot be null.");
        }

        try {
            Object[] objs = DirectCallExecutor.newExecutor().call(state, func, args);

            if (objs.length > 0) {
                return objs[0];
            } else {
                //it's a void method
                return null;
            }
        } catch (CallException e) {
            Log.w("Scripts", "CallException: ", e);
            throw e;
            //throw new ScriptExecutionException("CallException: " + e.getLocalizedMessage());
        } catch (CallPausedException e) {
            Log.w("Scripts", "CallPausedException: ", e);
        } catch (InterruptedException e) {
            Log.w("Scripts", "InterruptedException: ", e);
        }

        return null;
    }

    @Override
    public Object execFunc(String funcName) throws CallException {
        return this.execFunc(funcName, new Object[0]);
    }

    @Override
    public Object execScript(String scriptName, Object... args) {
        scriptName = "script_" + scriptName;

        LuaFunction func = this.luaFunctions.get(scriptName);

        if (func == null) {
            throw new IllegalStateException("lua script '" + scriptName + "' doesn't exists in cache, you have to compile it first!");
        }

        Log.d("Scripts", "execute lua script '" + scriptName + "'");

        try {
            Object[] objs = DirectCallExecutor.newExecutor().call(state, func);

            if (objs.length > 0) {
                return objs[0];
            } else {
                //it's a void method
                return null;
            }
        } catch (CallException e) {
            Log.w("Scripts", "CallException: ", e);
        } catch (CallPausedException e) {
            Log.w("Scripts", "CallPausedException: ", e);
        } catch (InterruptedException e) {
            Log.w("Scripts", "InterruptedException: ", e);
        }

        return null;
    }

    protected void printEnvDebug () {
        for (long i = 0; i < env.rawlen(); i++) {
            Log.d("Scripts", "rawget[" + i + "]: " + env.rawget(i));
        }
    }

}

Test method which throws this exception:

@Test
    public void testExecGlobalFunc () throws ScriptLoadException, CallException {
        GameTime time = GameTime.getInstance();
        time.setTime(System.currentTimeMillis());

        LuaScriptEngine engine = new LuaScriptEngine();
        engine.compile("add", "function add (a)\n" +
                "      local sum = 0\n" +
                "      for i,v in ipairs(a) do\n" +
                "        sum = sum + v\n" +
                "      end\n" +
                "      return sum\n" +
                "    end");
        engine.execScript("add");

        engine.printEnvDebug();

        assertEquals(6, engine.execFunc("add", new int[] {1, 2, 3}));
    }

Lua strings in Rembulan are currently mapped to java.lang.String. While this is convenient for Java interoperability and is in line with the way Lua booleans and numbers are handled, it is not consistent with the definition on strings as per §2.1 of the Lua Reference Manual:

The type string represents immutable sequences of bytes. Lua is 8-bit clean: strings can contain any 8-bit value, including embedded zeros ('\0'). Lua is also encoding-agnostic; it makes no assumptions about the contents of a string.

java.lang.Strings are essentially immutable sequences of Java char, i.e., 16-bit signed integers. In order to treat java.lang.String as a Lua string, there ought to be a value mapping from 16-bit values to 8 bits (e.g. taking the least significant byte).

Problems

This is a (probably incomplete) list of reasons why using java.lang.String for Lua strings is broken:

• Currently, string equality is determined using the equals() method. This means that regardless of how the 16-bit-to-8-bit value mapping is chosen, there will always be strings a and b such that !a.equals(b), but in Lua, a == b in Lua. Additionally, it would be possible that a.hashCode() != b.hashCode() in Java, but a == b in Lua.
• This means that, unless strings are "normalised", (i.e. converted using the 16-bit-to-8-bit value map to a new string wrapping the mapped values), using strings as keys in a HashMap may lead to a situation where the map contains multiple keys that are equal in Lua.
• In order to be able to use HashMap in a way compatible with Lua equality, normalisation would need to be done before insertion and retrieval, which may seriously degrade performance. Furthermore, there is no way of telling whether a given instance of java.lang.String is normalised or not, so it would need to be done every time.
• java.lang.String does not carry around information about its character encoding, so normalisation must be charset-independent. This could lead to very confusing behaviour. Custom normalisation methods could be provided, but they would not bring any benefits performance-wise (see the previous point).
• There is a mismatch in the spirit of Lua strings and java.lang.String. While Lua strings do not care about their contents, Java strings do: the char type is explicitly meant to represent 16-bit Unicode characters.
• 16-bit strings require twice as much space as 8-bit strings to be represented correctly (though the actual memory usage is JVM-dependent).

Solutions

Keep java.lang.String

Some of the problems above may be avoided:

• Lua string comparison can take the value mapping into account.
• Lua table implementations can be implemented from scratch, without using HashMap or other collections that use equals() to determine the equality of keys.

However, there are still outstanding issues:

• Strings equal in Lua may still be unequal in Java, and there would be no way of distinguishing them. In Lua, this is no problem, but it opens up an entire class of hard-to-detect bugs in the Java code that interoperates with Lua:

-- in Lua, it does not matter whether we return a or b... but it might in Java!
if a == b then return a end

• The spirit problem still remains, and turns into a smell.

Use a custom immutable byte string class

The only correct solution appears to be to use another class (a "byte string") for representing Lua strings. This of course has its downsides:

• The impact on the code base would be non-trivial.
• There is no immutable byte string class in the JDK.
• java.lang.String is ubiquitous in the Java world: introducing extra conversion step to and from the new class would make Java interoperability more cumbersome.
• If the rest of the value mapping rules remain, java.lang.String would become light userdata, which may lead to confusion, especially if booleans are still mapped to java.lang.Boolean and numbers to java.lang.Number.
• java.lang.String can be interned; it might be difficult to get such functionality for byte strings.
• Similarly for string deduplication (since Java 8 Update 20).

The following code throws an exceotion on strmatch

local function updatedecpoint ()
    decpoint = strmatch(tostring(0.5), "([^05+])")
    -- build a filter that can be used to remove group separators
    numfilter = "[^0-9%-%+eE" .. gsub(decpoint, "[%^%$%(%)%%%.%[%]%*%+%-%?]", "%%%0") .. "]+"
end

This code is working fine with luac and luaj. Its the dkjson luarocks library.

The exception is:

net.sandius.rembulan.exec.CallException: java.lang.IllegalArgumentException: error at character 6: unexpected character '+'

	at net.sandius.rembulan.exec.DirectCallExecutor$Result.get(DirectCallExecutor.java:184)
	at net.sandius.rembulan.exec.DirectCallExecutor.execute(DirectCallExecutor.java:310)
	at net.sandius.rembulan.exec.DirectCallExecutor.resume(DirectCallExecutor.java:252)
	at net.sandius.rembulan.exec.DirectCallExecutor.call(DirectCallExecutor.java:228)
	at hu.blackbelt.judo.scripting.lua.TestScriptRun.testRembulanScriptLoading(TestScriptRun.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at com.intellij.junit4.JUnit4TestRunnerUtil$IgnoreIgnoredTestJUnit4ClassRunner.runChild(JUnit4TestRunnerUtil.java:365)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.lang.IllegalArgumentException: error at character 6: unexpected character '+'
	at net.sandius.rembulan.lib.StringPattern$PatternBuilder.parseError(StringPattern.java:941)
	at net.sandius.rembulan.lib.StringPattern$PatternBuilder.CC_lit(StringPattern.java:1087)
	at net.sandius.rembulan.lib.StringPattern$PatternBuilder.characterSetElement(StringPattern.java:1017)
	at net.sandius.rembulan.lib.StringPattern$PatternBuilder.characterSetBody(StringPattern.java:1025)
	at net.sandius.rembulan.lib.StringPattern$PatternBuilder.cclass(StringPattern.java:1133)
	at net.sandius.rembulan.lib.StringPattern$PatternBuilder.PI_cc(StringPattern.java:1163)
	at net.sandius.rembulan.lib.StringPattern$PatternBuilder.PI(StringPattern.java:1186)
	at net.sandius.rembulan.lib.StringPattern$PatternBuilder.PI_capture(StringPattern.java:1196)
	at net.sandius.rembulan.lib.StringPattern$PatternBuilder.PI(StringPattern.java:1170)
	at net.sandius.rembulan.lib.StringPattern$PatternBuilder.parse(StringPattern.java:1218)
	at net.sandius.rembulan.lib.StringPattern$PatternBuilder.access$900(StringPattern.java:913)
	at net.sandius.rembulan.lib.StringPattern.fromString(StringPattern.java:1228)
	at net.sandius.rembulan.lib.StringPattern.fromString(StringPattern.java:1232)
	at net.sandius.rembulan.lib.StringLib$Match.invoke(StringLib.java:1699)
	at net.sandius.rembulan.lib.AbstractLibFunction.invoke(AbstractLibFunction.java:39)
	at net.sandius.rembulan.runtime.AbstractFunctionAnyArg.invoke(AbstractFunctionAnyArg.java:36)
	at net.sandius.rembulan.runtime.Dispatch.mt_invoke(Dispatch.java:70)
	at net.sandius.rembulan.runtime.Dispatch.call(Dispatch.java:236)

Hello developer, first of all, thank you for opening your code. We developed a tool to scan whether there is runtime exception in the source code of the project. Then we scanned your code and found a method that may need to add try catch module! It is the putto method of rembulan-runtime \src\main\java\net\sandius\rembulan\stringbytestring.java.

Error reason: there is not enough space for ByteBuffer to store byte array parsed by string. Here is my test code,Thank you for your reply.

import sun.nio.cs.UTF_32;
import java.nio.ByteBuffer;
import java.nio.charset.Charset;
import java.util.Arrays;
import java.util.Objects;

public class StringByteString {
    private final String string;
    private final Charset charset;

    private int byteHashCode;
    private int byteLength;

    StringByteString(String s, Charset charset) {
        this.string = Objects.requireNonNull(s);
        this.charset = Objects.requireNonNull(charset);
        if (!charset.canEncode()) {
            throw new IllegalArgumentException("Charset cannot encode: " + charset.name());
        }
        this.byteHashCode = 0;
        this.byteLength = string.isEmpty() ? 0 : -1;
    }

    private byte[] toBytes() {
        // TODO: cache the result
        return string.getBytes(charset);
    }

    public byte[] getBytes() {
        byte[] bytes = toBytes();

        // must make a defensive copy
        return Arrays.copyOf(bytes, bytes.length);
    }

    public void putTo(ByteBuffer buffer) {
        // ByteBuffer cannot be directly extended: it's safe to use a possibly cached array
        buffer.put(toBytes());
    }

    public static void main(String[] args) {
        StringByteString stringByteString = new StringByteString("I am Test",new UTF_32());
        ByteBuffer byteBuffer = ByteBuffer.allocate(2);
        stringByteString.putTo(byteBuffer);
    }
}

It appears that the source has to be loaded as a string into memory right now. Some sort of load method that takes an InputStream and has the possibility of future optimization via evicting unused source chunks would be nice to have.

Hi there, can you attach a performance graph similar to LuaJ?
e.g https://github.com/darmie/LuaJ#performance

I'd like to be able to run commands like the following

time java -cp lib/luaj-jse-3.0.1.jar lua test/lua/perf/binarytrees.lua 10

When used on very large strings, the standard library functions string.find, string.gmatch, string.gsub and string.match may take a very long time to execute. In order to support execution time limiting, these functions should periodically check with the scheduler whether they should be paused.

This requires adjusting the pattern matching code to make it resumable.

The following Lua code

main.lua

t = {}
function f()
return t
end
f().a=true
print(t.a)

is supposed to print true (Confirmed with https://www.lua.org/cgi-bin/demo).

Instead it fails with:

Exception in thread "main" net.sandius.rembulan.exec.CallException: net.sandius.rembulan.runtime.IllegalOperationAttemptException: attempt to index a boolean value
	at net.sandius.rembulan.exec.DirectCallExecutor$Result.get(DirectCallExecutor.java:184)
	at net.sandius.rembulan.exec.DirectCallExecutor.execute(DirectCallExecutor.java:310)
	at net.sandius.rembulan.exec.DirectCallExecutor.resume(DirectCallExecutor.java:252)
	at net.sandius.rembulan.exec.DirectCallExecutor.call(DirectCallExecutor.java:228)
	at net.wizardsoflua.rembulan.bug.RembulanBugMain.main(RembulanBugMain.java:30)
Caused by: net.sandius.rembulan.runtime.IllegalOperationAttemptException: attempt to index a boolean value
	at net.sandius.rembulan.runtime.Errors.illegalIndexAttempt(Errors.java:70)
	at net.sandius.rembulan.runtime.Dispatch.setindex(Dispatch.java:1533)
	at ByteCode0.run(main.lua:5)
	at ByteCode0.invoke(main.lua)
	at net.sandius.rembulan.runtime.AbstractFunction0.invoke(AbstractFunction0.java:51)
	at net.sandius.rembulan.runtime.Dispatch.mt_invoke(Dispatch.java:95)
	at net.sandius.rembulan.runtime.Dispatch.call(Dispatch.java:402)
	at net.sandius.rembulan.runtime.Coroutine$BootstrapResumable.resume(Coroutine.java:90)
	at net.sandius.rembulan.runtime.ResumeInfo.resume(ResumeInfo.java:36)
	at net.sandius.rembulan.runtime.Call$Resumer.continueCurrentCoroutine(Call.java:513)
	at net.sandius.rembulan.runtime.Call$Resumer.resume(Call.java:537)
	at net.sandius.rembulan.runtime.Call.resume(Call.java:232)
	at net.sandius.rembulan.runtime.Call.access$000(Call.java:37)
	at net.sandius.rembulan.runtime.Call$CallContinuation.resume(Call.java:181)
	at net.sandius.rembulan.exec.DirectCallExecutor.execute(DirectCallExecutor.java:290)
	... 3 more

It looks like the boolean value and table t get swapped when passing them to net.sandius.rembulan.runtime.Dispatch.setindex(ExecutionContext, Object, Object, Object) from the generate byte code. With a breakboint in Dispatch.setindex you can clearly see this.

Complete example:

package net.wizardsoflua.rembulan.bug;

import net.sandius.rembulan.StateContext;
import net.sandius.rembulan.Table;
import net.sandius.rembulan.Variable;
import net.sandius.rembulan.compiler.CompilerChunkLoader;
import net.sandius.rembulan.env.RuntimeEnvironment;
import net.sandius.rembulan.env.RuntimeEnvironments;
import net.sandius.rembulan.exec.DirectCallExecutor;
import net.sandius.rembulan.impl.StateContexts;
import net.sandius.rembulan.lib.BasicLib;
import net.sandius.rembulan.runtime.LuaFunction;

public class RembulanBugMain {
  public static void main(String[] args) throws Exception {
    StateContext stateContext = StateContexts.newDefaultInstance();
    Table env = stateContext.newTable();
    CompilerChunkLoader loader = CompilerChunkLoader.of("ByteCode");
    RuntimeEnvironment runtimeEnvironment = RuntimeEnvironments.system();
    BasicLib.installInto(stateContext, env, runtimeEnvironment, loader);
    Variable envVariable = new Variable(env);
    LuaFunction mainFunction = loader.loadTextChunk(envVariable, "main.lua", "t = {}\n" + //
        "function f()\n" + //
        "return t\n" + //
        "end\n" + //
        "f().a=true\n" + //
        "print(t.a)\n" //
    );
    DirectCallExecutor newExecutor = DirectCallExecutor.newExecutor();
    newExecutor.call(stateContext, mainFunction);
  }
}

In Lua, tables and userdata may be marked for finalisation using the __gc metamethod. (For details, see §2.5.1 of the Lua Reference Manual). This is currently not possible in Rembulan.

When I require a string that contains an illegal path character an InvalidPathException is thrown. This is a problem because I want to require an URL wich contains ':'. This require would be processed by a custom searcher that has been installed into the environment at the end of package.searchers similar to here. However this code throws an InvalidPathException before my searcher is even called. In my opinion this exception should be caught and handled just like when the file is not readable.

The compiler currently routes all non-constructor table accesses via Dispatch, in order to correctly handle possible metamethod calls. When it is known that the table cannot have a metatable, its accesses may be raw.

Raw table accesses are preferable to non-raw accesses, since the former does not require the insertion of resumption points, making the generated bytecode smaller and simpler (fewer entry points), giving the JVM more opportunities to perform JIT optimisations.

Consider the following Lua snippet (the first 80 numbers in the Fibonacci series):

local t = {1, 1}
for i = 3, 80 do
  t[i] = t[i - 2] + t[i - 1]
end

Here, all accesses of t can be raw, since the freshly-allocated t does not have a metatable, and t does not participate in any operation that may modify its metatable.

If the snippet above was followed by

for k, v in ipairs(t) do
  print(k, v)
end
print(#t)  -- non-raw, t escaped

then the #t operation would not be raw, since t has been passed to ipairs as an argument: and its contents including the metatable may have been modified.

How to go about this

This feature could be implemented by a simple escape analysis.

The compiler should keep track of the "status" of table locals and temporaries. Tables are constructed fresh; table operations (t[x], t[x] = y and #t) on fresh tables are raw and do not change the table status. Whenever a table participates in a call (as an argument, e.g., f(t)) or in operation that may involve a metamethod call (e.g., t + 1) their status changes to dirty. Table operations on dirty tables are invoked via Dispatch, i.e., with checking for metamethods.

Rembulan's standard library lacks the utf8 module. The module can be implemented now since string handling conforms to Lua 5.3 (see #2).

Not seen any commits for over a year, and the author doesn't appear to be on GitHub anymore...

Would love to see a blog or doc page on why you thought Lua on the JVM was a good idea.

• What use cases does this make easier?
• What problems does this solve?
• Are you using it for any real world solutions?

capsule
very handy

It would be desirable to have the capability to serialise (and deserialise and restore) the execution state of a paused call.

Is it not possible to use the original implementation from luajava?
Or it is because it is not compatible with intermediate representation?

Feature request: implement javax.script.ScriptEngineFactory to satisfy JSR 223 standard. Any interest?

Rembulan is currently only able to load text chunks. It might be a good idea to consider adding support for binary chunks.

Java bytecode (as generated by Rembulan), even packed in a JAR file, is probably too low-level for this purpose. Classes in Java bytecode are named, whereas function prototypes in the binary chunk should be (mostly) anonymous. Additionally, there is no way of controlling the origin of the JAR file (and hence its contents): most users would not want to load arbitrary bytecode if they expect a pre-compiled Lua function.

Therefore, these binary chunks should probably be based on the compiler's already-existing IR (intermediate representation).

Advantages

• Loading binary chunks would eliminate the time spent in parsing, performing static analysis, and optimising the code.
• string.dump could be implemented.
• Function prototypes could be transmitted over the wire.
• This feature would help in serialisation (#4). (Upvalues need to be serialised as well, and at least in PUC-Lua, these are not included in a binary chunk -- an expectation that should probably not be broken in Rembulan.)

Disadvantages

• This is strictly speaking not necessary to implement the Lua programming language. (The Lua bytecode is an implementation detail of PUC-Lua.)
• The implementation would introduce additional complexity and source of bugs into the compiler.
• The IR should not be considered stable at this point: there are still outstanding optimisations that may require changing the IR. Fixing the IR would slow down progress on that front.
• Rembulan's IR is very different from the PUC-Lua bytecode, and its binary format would be too. (In other words, it would still not be possible to load a chunk compiled by PUC-Lua in Rembulan.)
• Security and safety implications: a malformed binary chunk may crash the compiler. To do this right, this feature would probably require a verifier. (On the other hand, the IR does have some of the necessary checks in place already, and is immutable. The compiler could be made resilient to malformed IR input.)
• Binary chunk loading is typically not permitted in sandboxed environments, so the usefulness in the niche Rembulan is explicitly targeting is limited.

How could this be done

The following steps are required in order to implement this feature such that it is usable/useful:

• Define a binary format for the IR, and implement its writer and reader.
• (Optionally, implement a verifier for security.)
• Add an additional entry point into the compiler pipeline that accepts loaded IR.
• Attach the binary representation to compiled functions in a way that is accessible at runtime. (The best way to go about this is probably using Java annotations.)
• Implement string.dump.

The compiler currently requires all local and temporary variables to be boxed. This introduces a significant performance penalty.

For instance, in the following snippet:

local x = 0
for i = 1, 10 do
  x = x + i
end

every arithmetic operation with x or i involves unboxing it, performing the operation, and boxing the result again. Boxing here is not necessary: it can be determined statically that x is and i are integers, so they can be replaced with primitives while maintaining semantics.

How to go about this

The required static type analysis is already in place. The only remaining thing to do is to change the layout of the generated code to support unboxed local variables and temporaries, and wire the operations up accordingly.

PUC-Lua's standard module library should be implemented in Rembulan in order to allow the execution of non-trivial programs.

This has two facets:

• it should be easy enough to use from the 'host' Java code to bootstrap the Lua environment;
• the require function should expose module loading capabilities to the 'client' Lua programs.

This also requires the implementations of (now missing) functions dofile and loadfile from the basic library.

Tables in Lua may be weak, i.e., their contents may be weak references rather than strong references. (See §2.5.2 of the Lua Reference Manual.) However, weak tables are currently not supported in Rembulan.

Exceptions thrown from Lua programs do not include the filename and line information as they do in PUC-Lua. E.g.:

$ rembulan
Rembulan 0.1-SNAPSHOT (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_60)
> nil + 1
attempt to perform arithmetic on a nil value
stack traceback:
   (omitted)

$ lua
Lua 5.3.3  Copyright (C) 1994-2016 Lua.org, PUC-Rio
> nil + 1
stdin:1: attempt to perform arithmetic on a nil value
stack traceback:
  (omitted)

(It's about the stdin:1 prefix of the error message.)

Debugging Lua programs is difficult without this information.

How can I call a function (with params) which is defined in lua? Ie some algorithm is written in lua, I just want to call it from java, and interpret it.

There are many projects out there that use Lua 5.1 or 5.2, mainly because they are based on LuaJIT rather than PUC-Lua. In order to run such Lua scripts on the JVM, currently the best way to do this involves using LuaJ. In order to LuaJ users a viable alternative, it would be good to have a compatibility layer for running Lua 5.1 and 5.2 on Rembulan.

The simplest way to go about this would be adding alternative implementations of the standard library. This would not provide the exact semantics as Lua 5.1 or 5.2, but might be sufficient in most cases.