grab.spider.queue_backend.redis

Spider task queue backend powered by redis.

Module Contents

Classes

CustomPriorityQueue	Priority queue
RedisTaskQueue

Attributes

system_random

grab.spider.queue_backend.redis.system_random

class grab.spider.queue_backend.redis.CustomPriorityQueue(key: str, **kwargs: Any)

Bases: fastrq.priorityqueue.PriorityQueue

Priority queue

The lower the score, the higher the priority. Usage:

pq = PriorityQueue(“hello-pq”) pq.push({“google”: 100, “alibaba”: 101}) pq.pop()

connect() → redis.Redis[Any]

clear() → None

class grab.spider.queue_backend.redis.RedisTaskQueue(queue_name: None | str = None, connection_args: None | dict[str, Any] = None)

Bases: grab.spider.queue_backend.base.BaseTaskQueue

put(task: grab.spider.task.Task, priority: int, schedule_time: None | datetime = None) → None

get() → grab.spider.task.Task

size() → int

clear() → None

close() → None