The processing-text-data's discuss from carted

processing-text-data's Issues

Spurious error when the pipeline is run on Dataflow

Updated code is here: https://github.com/carted/processing-text-data/tree/dev.

The following is the error Dataflow runs into:

INFO:apache_beam.runners.dataflow.dataflow_runner:2022-02-17T11:10:13.462Z: JOB_MESSAGE_ERROR: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/dataflow_worker/batchworker.py", line 773, in run
    self._load_main_session(self.local_staging_directory)
  File "/usr/local/lib/python3.8/site-packages/dataflow_worker/batchworker.py", line 514, in _load_main_session
    pickler.load_session(session_file)
  File "/usr/local/lib/python3.8/site-packages/apache_beam/internal/pickler.py", line 311, in load_session
    return dill.load_session(file_path)
  File "/usr/local/lib/python3.8/site-packages/dill/_dill.py", line 368, in load_session
    module = unpickler.load()
  File "/usr/local/lib/python3.8/site-packages/dill/_dill.py", line 472, in load
    obj = StockUnpickler.load(self)
  File "/usr/local/lib/python3.8/site-packages/dill/_dill.py", line 462, in find_class
    return StockUnpickler.find_class(self, module, name)
ModuleNotFoundError: No module named 'configs'

@Nilabhra do you see anything immediately pesky in main.py?

Recommend Projects