Why does my Dask application fail to start a worker process?
While running a Dask application, it fails with the following message in the stack trace: ERROR - Failed while trying to start worker process: Set changed size during iteration.
For more details, check the log for the worker that is failing. A common cause of this error is
that the worker requires more memory. To increase the worker memory for your application, complete
one of the following actions:
- Modify your instance group to increase the dask_worker_memory_limit parameter for the Dask component.
- Re-initialize your ConductorCluster using a higher memory limit. For
example, to increase to 10 GB,
set:
cluster = ConductorCluster(memlimit='10GB')