celery问题排查命令

命令

# 查看节点状态
celery -A tasks status
celery -A tasks inspect stats
# 获取任务结果
celery -A tasks result 021ce9a0-f006-49f5-b1e2-3a5ef60a5ba4
redis-cli -n 1 get celery-task-meta-021ce9a0-f006-49f5-b1e2-3a5ef60a5ba4
# 查看节点任务信息
celery -A tasks inspect active|scheduled|revoked|registered

# 查看redis中key的数量
redis-cli -n 1 dbsize
# 查看redis中的kes
redis-cli -n 1 --scan
redis-cli -n 1 keys \* (不推荐）

# 获取所有的keys值
redis-cli -n 1 --scan | sed 's/celery/get celery/' | redis-cli -n 1 -x

学习笔记

• Getting Started

  • Introduction to Celery
  • First Step with Celery
    • Choosing a Broker: RabbitMQ or Rdis
    • Install Celery: pip install celery
    • Application
    • Running the Celery worker server: celery -A tasks worker --loglevel=INFO
    • Calling the task: delay() apply_async()
    • Keeping Results:
      • Backend: SQLAlchemy/Django ORM, MongoDB, Memcached, Redis, RPC (RabbitMQ/AMQP), and – or you can define your own
      • result.ready() result.get(timeout=1) result.get(propagate=False) result.traceback
      • Warning ：To ensure that resources are released, you must eventually call get() or forget() on EVERY AsyncResult instance returned after calling a task. 刚一delay就马上forget是不起作用的，要根据任务的状态调用forget
    • Configuration
      • app.conf.task_serializer = 'json'
      • app.config_from_object('celeryconfig')
  • Next Steps
    • Using Celery in your Application
      • Results can also be disabled for individual tasks by setting the @task(ignore_result=True) option.
    • Starting the worker
      • Run in the background: celery multi start\stop\stopwait
      • The default concurrency number is the number of CPU’s on that machine (including cores).
    • Calling Tasks
      • Also note that result backends aren’t used for monitoring tasks and workers: for that Celery uses dedicated event messages
      • The started state is a special state that’s only recorded if the task_track_started setting is enabled, or if the @task(track_started=True) option is set for the task.
    • Canvas: Designing Work-flows
      • signatures: add.signature((2, 2), countdown=10), add.s(2, 2)
      • The Primitives: group, chain, chord, map, starmap, chunks
    • Routing
    • Remote Control
    • Timezone: app.conf.timezone = 'Europe/London'
    • Optimization
  • Backends and Brokers
    • Using Redis
      • Installation: pip install -U "celery[redis]"
      • Configuration: app.conf.broker_url = 'redis://:password@hostname:port/db_number'
        • Redis Sentinel
      • Visibility Timeout: app.conf.broker_transport_options = {'visibility_timeout': 3600}
      • Results: app.conf.result_backend = 'redis://localhost:6379/0'
        • Connection timeouts: app.conf.result_backend_transport_options = {'retry_policy': {'timeout': 5.0}}
      • Caveats
        • Visibility timeout: Periodic tasks won’t be affected by the visibility timeout
        • Key eviction: configure the redis-server to not evict keys by setting in the redis configuration file: maxmemory and maxmemory-policy. 当Broker对应的redis触发OOM时，celery会崩溃；当Backend对应的redis触发OOM时，celery只是报错。
        • Group result ordering

• User Guide

  • Application
    • The application is thread-safe so that multiple Celery applications with different configurations, components, and tasks can co-exist in the same process space.
    • Main Name
    • Configuration
      • config_from_object: 1)Using the name of a module; 2)Passing an actual module object; 3)Using a configuration class/object
      • config_from_envvar
      • Censored configuration: app.conf.humanize(); app.conf.table()
    • Laziness
    • Breaking the chain
      • the best practice is to always pass the app instance around to anything that needs it.
      • uses the celery.app.app_or_default() function
      • In development you can set the CELERY_TRACE_APP environment variable to raise an exception if the app chain breaks
    • Abstract Tasks
  • Tasks
    • Basics
      • If you’re using Django, or you’re the author of a library then you probably want to use the shared_task() decorator
      • When using multiple decorators in combination with the task decorator you must make sure that the task decorator must be first in the list
    • Bound tasks: 任何函数的第一个参数self是app.Task.request类型
    • Task inheritance
    • Names
      • Changing the automatic naming behavior
    • Task Request
    • Logging: get_task_logger(); worker_redirect_stdouts
    • Argument checking
    • Hiding sensitive information in arguments
    • Retrying
      • Using a custom retry delay
      • Automatic retry for known exceptions
        • exponential backoff
    • List of Options
    • States
      • Result Backends
        • RPC Result Backend (RabbitMQ/QPid): It doesn’t actually store the states, but rather sends them as messages, a result can only be retrieved once, and only by the client that initiated the task. Two different processes can’t wait for the same result.
        • Database Result Backend: In MySQL changing the default transaction isolation level REPEATABLE-READ to the READ-COMMITTED is recommended.
      • Built-in States: PENDING, STARTED, SUCCESS, FAILURE, RETRY, REVOKED
      • Custom states
    • Creating pickleable exceptions
    • Semipredicates: Ignore, Reject, Retry
    • Custom task classes
      • Instantiation: _init_ constructor will only be called once per process. This can also be useful to cache resources, For example, a base Task class that caches a database connection
      • Per task usage
      • App-wide usage
      • Handlers
      • Requests and custom requests
    • How it works
    • Tips and Best Practices
      • Ignore results you don’t want
      • More optimization tips: Optimizing Guide
      • Avoid launching synchronous subtasks
    • Performance and Strategies
      • Granularity
      • Data locality
      • State: Django model objects shouldn’t be passed on as arguments to tasks
      • Database transactions
  • Calling Tasks
    • Basics
    • Linking (callbacks/errbacks): add.apply_async((2, 2), link=add.s(16))
    • On message
    • ETA and Countdown
    • Expiration
    • Message Sending Retry
      • Retry Policy: max_retries, interval_start, interval_step, interval_max
    • Connection Error Handling
    • Serializers: The default serializer is JSON, but you can change this using the task_serializer setting, or for each individual task, or even per message.
    • Compression
    • Connections
    • Routing options
    • Results options
    • Advanced Options
  • Canvas: Designing Work-flows
    • Signatures
    • Partials
    • Immutability
    • Callbacks
    • The Primitives
  • Workers Guide
    • Starting the worker: Note for supervisor users:The % sign must be escaped by adding a second one: %%h.
    • Stopping the worker
    • Restarting the worker
    • Process Signals
    • Variables in file paths
      • Node name replacements
      • Prefork pool process index
    • Concurrency
    • Remote control
      • The broadcast() function
      • revoke
        • Revoking tasks
        • Revoking multiple tasks
        • Persistent revokes
    • Time Limits
      • Changing time limits at run-time: app.control.time_limit()
    • Rate Limits
      • Changing rate-limits at run-time: app.control.rate_limit()
    • Max tasks per child setting
    • Max memory per child setting
    • Autoscaling
    • Queues
      • Adding consumers: celery -A proj control add_consumer foo, app.control.add_consumer()
      • Canceling consumers
      • List of active queues
    • Inspecting workers
    • Additional Commands: Remote shutdown, Ping, Enable/disable events
    • Writing your own remote control commands
  • Daemonization（略）