[pyar] Celery chains con muchas subtareas

[Francisco Roldán]

Buen día lista.

Estoy teniendo un problema usando Celery chains con celery==3.2.1, Django==1.11.21 y python 2.7.

La cadena se genera desde una tarea que la arma dependiendo una lista de datos a procesar, se paginan esos datos según un valor por página y arma la cadena con n cantidad de subtasks.
Venía funcionando bien hasta que empezamos a tener un error al generarse la cadena: EncodeError: maximum recursion depth exceeded while calling a Python object


Traceback (most recent call last):
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/sentry_sdk/integrations/celery.py", line 120, in _inner
    reraise(*exc_info)
  File "/app/.heroku/python/lib/python2.7/site-packages/sentry_sdk/integrations/celery.py", line 115, in _inner
    return f(*args, **kwargs)
  File "/app/core/tasks_decorators.py", line 27, in _decorator
    return function(*args, **kwargs)
  File "/app/core/tasks_decorators.py", line 14, in _decorator
    return function(*args, **kwargs)
  File "/app/lmsplatform/tasks.py", line 659, in send_phishing_campaign_task
    chain(subtasks).apply_async()
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/canvas.py", line 251, in apply_async
    return _apply(args, kwargs, **options)
  File "/app/.heroku/python/lib/python2.7/site-packages/sentry_sdk/integrations/celery.py", line 74, in apply_async
    return f(*args, **kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/builtins.py", line 293, in apply_async
    tasks[0].apply_async(**options)
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/canvas.py", line 251, in apply_async
    return _apply(args, kwargs, **options)
  File "/app/.heroku/python/lib/python2.7/site-packages/sentry_sdk/integrations/celery.py", line 74, in apply_async
    return f(*args, **kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/tenant_schemas_celery/task.py", line 27, in apply_async
    return super(TenantTask, self).apply_async(args, kwargs, *arg, **kw)
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/task.py", line 573, in apply_async
    **dict(self._get_exec_options(), **options)
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/base.py", line 354, in send_task
    reply_to=reply_to or self.oid, **options
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/amqp.py", line 310, in publish_task
    **kwargs
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 165, in publish
    compression, headers)
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 241, in _prepare
    body) = dumps(body, serializer=serializer)
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/serialization.py", line 164, in dumps
    payload = encoder(data)
  File "/app/.heroku/python/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/serialization.py", line 59, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/serialization.py", line 55, in _reraise_errors
    yield
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/serialization.py", line 164, in dumps
    payload = encoder(data)
  File "/app/.heroku/python/lib/python2.7/site-packages/anyjson/__init__.py", line 141, in dumps
    return implementation.dumps(value)
  File "/app/.heroku/python/lib/python2.7/site-packages/anyjson/__init__.py", line 87, in dumps
    return self._encode(data)
  File "/app/.heroku/python/lib/python2.7/json/__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "/app/.heroku/python/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/app/.heroku/python/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
EncodeError: maximum recursion depth exceeded while calling a Python object

El error es muy similar al reportado en el siguiente issue [1], probé con el workaround propuesto en el mismo pero no me dió resultado.

¿Alguien tuvo el mismo problema, alguna idea para dónde puedo encarar?


Gracias!

Francisco



[1] https://github.com/celery/celery/issues/1078 <https://github.com/celery/celery/issues/1078>
------------ próxima parte ------------
Se ha borrado un adjunto en formato HTML...
URL: <http://listas.python.org.ar/pipermail/pyar/attachments/20200304/8a27b94f/attachment.html>