A script that parses XML takes only 0.00718 seconds to execute through Python, but it takes 17 seconds to execute through Python4J. Why is there such a difference in performance? about deeplearning4j HOT 6 OPEN

@aizhimin thanks for posting! Can you post the script so I can look? Python version would help as well.

from deeplearning4j.

@agibsonccc Python version is 3.10.2
the script is like this :
`
xml_data_str = xml_data_bytes.decode(encoding='utf-8')

root = etree.fromstring(xml_data_str)
lotRunJson =generate_lotrunjson(root)
machineRcpJson =generate_machinercpjson(root)
imageJson =generate_imagejson(root, lotRunJson)
waferRunJson =generate_waferrunjson(root)
targetJson =generate_targetjson(root, lotRunJson)
measurementJson =generate_measurementjson(root)
measurementResultJson =generate_measurementresultjson(root)
`

this script takes only 0.00718 seconds to execute through Python. But it takes 17 seconds to execute through Python4J.

Is it slower to pass the binary data of a file as input in Java? Do I need to pass the file path in and let Python read the file?

from deeplearning4j.

@aizhimin can you give me something I can run standalone? If I'm going to benchmark something you and I need a common baseline to work with.

from deeplearning4j.

@agibsonccc Sorry, my file is confidential. My question is that the code for xml parsing executed directly in the python environment runs very fast. However, the xml parsing executed through python4j calls is very slow. Is this due to the need to load the python parser? Or is it because the input parameters cannot pass file data streams?

from deeplearning4j.

@aizhimin I don't care about your secrets. A vague description I can't directly run isn't something I'm inclined to spend time on. I believe you but you putting up barriers to me reproducing the issue isn't going to help get this fixed. Meet me half way and setup a trivial example you can show me and I'll be more likely to take a look at this when I get time. The goal is to have a common "language" we can speak here (in this case code) that allows us both to run the same environment and baseline so we can both agree the issue is resolved.

from deeplearning4j.

@agibsonccc

The xml like this:

The python script like this:

from lxml import etree  
  
def dumps_json(obj):
    if obj:
        return json.dumps(obj)
    return json.dumps("")

xml_data = xml_data_bytes.decode(encoding='utf-8')
root = etree.fromstring(xml_data)  
result = {}
for child in root:  
    result[child.tag] = child.text

resultJson = dumps_json(result)

The java code like this:

try(PythonGIL gil = PythonGIL.lock()){
            try(PythonGC gc = PythonGC.watch()){
                List<PythonVariable> inputs = new ArrayList<>();
                byte[] xml_data_bytes = FileUtils.readFileToByteArray(new File("/Users/aizhimin/Documents/test.xml"));
                inputs.add(new PythonVariable<>("xml_data_bytes", PythonTypes.BYTES, xml_data_bytes));

                List<PythonVariable> outputs = new ArrayList<>();
                outputs.add(new PythonVariable<>("resultJson", PythonTypes.STR));

                String code = FileUtils.readFileToString(new File("/Users/aizhimin/Documents/testxml.py"));
                long startTime = System.currentTimeMillis();
                PythonExecutioner.exec(code, inputs, outputs);
                long endTime = System.currentTimeMillis();
                System.out.println("Cost time:" + (endTime - startTime) / 1000.0);
                for(PythonVariable out : outputs){
                    System.out.println(out.getName()+":"+out.getValue());
                }
            }
        }catch (Throwable e){
            e.printStackTrace();
        }

Java takes 2.193s

If only python ，it takes only 0.0119s

from deeplearning4j.