This is recursion. <a href="https://github.com/m3dev/gokart/blob/mas

Sample that takes a long time to start. <div class="highlight highlight-source-pyt

<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="10

There are two solutions. luigi.Task.

Recursively large parameters about gokart HOT 6 CLOSED

vaaaaanquish commented on June 24, 2024

Recursively large parameters

from gokart.

Comments (6)

vaaaaanquish commented on June 24, 2024 1

Sample that takes a long time to start.

import luigi

import gokart


class Zero(gokart.TaskOnKart):
    def run(self):
        self.dump(0)


class Add(gokart.TaskOnKart):
    x = gokart.TaskInstanceParameter()
    y = luigi.IntParameter()

    def run(self):
        self.dump(self.load() + self.y)


x = Zero()
for i in range(100):
    x = Add(x=x, y=i)

gokart.build(x)

from gokart.

vaaaaanquish commented on June 24, 2024 1

#257 will solve the problem of bloated memory.

from gokart.

vaaaaanquish commented on June 24, 2024 1

[future] Caching TaskInstanceParameter.serialize input can speed up the process.

from gokart.

vaaaaanquish commented on June 24, 2024

DictParameter recursively json serializes the parameters.
https://github.com/spotify/luigi/blob/master/luigi/parameter.py#L1003

Same goes for TaskInstanceParameter. And TaskInstanceParameter is using DictParameter.
https://github.com/m3dev/gokart/blob/master/gokart/parameter.py

These are what's causing this hell.

from gokart.

vaaaaanquish commented on June 24, 2024

There are two solutions.

luigi.Task.to_str_params is overridden by gokart
TaskInstanceParameter.serialize has its own serialize

from gokart.

vaaaaanquish commented on June 24, 2024

TaskInstanceParameter.serialize is executed 25249 times in the above sample code.

And values what's inside serialize is following

{'type': 'Zero', 'params': {}}
{'type': 'Add', 'params': {'x': '{"type": "Zero", "params": {}}', 'y': '0'}}
{'type': 'Zero', 'params': {}}
{'type': 'Zero', 'params': {}}
{'type': 'Add', 'params': {'x': '{"type": "Zero", "params": {}}', 'y': '0'}}
{'type': 'Add', 'params': {'x': '{"type": "Add", "params": {"x": "{\\"type\\": \\"Zero\\", \\"params\\": {}}", "y": "0"}}', 'y': '1'}}
{'type': 'Add', 'params': {'x': '{"type": "Add", "params": {"x": "{\\"type\\": \\"Add\\", \\"params\\": {\\"x\\": \\"{\\\\\\"type\\\\\\": \\\\\\"Zero\\\\\\", \\\\\\"params\\\\\\": {}}\\", \\"y\\": \\"0\\"}}",
 "y": "1"}}', 'y': '2'}}
...