grab.spider

Subpackages

• grab.spider.queue_backend
  • grab.spider.queue_backend.base
  • grab.spider.queue_backend.memory
  • grab.spider.queue_backend.mongodb
  • grab.spider.queue_backend.redis
• grab.spider.service
  • grab.spider.service.base
  • grab.spider.service.network
  • grab.spider.service.parser
  • grab.spider.service.task_dispatcher
  • grab.spider.service.task_generator

Submodules

• grab.spider.base
• grab.spider.errors
• grab.spider.interface
• grab.spider.task

Package Contents

Classes

Spider	Asynchronous scraping framework.
Task	Task for spider.

class grab.spider.Spider(task_queue: None | BaseTaskQueue = None, thread_number: None | int = None, network_try_limit: None | int = None, task_try_limit: None | int = None, priority_mode: str = 'random', meta: None | dict[str, Any] = None, config: None | dict[str, Any] = None, parser_requests_per_process: int = 10000, parser_pool_size: int = 1, network_service: None | BaseNetworkService = None, grab_transport: None | BaseTransport | type[BaseTransport] = None)

Asynchronous scraping framework.

spider_name

initial_urls :list[str] = []

collect_runtime_event(name: str, value: None | str) → None

setup_queue(*_args: Any, **_kwargs: Any) → None

Set up queue.

add_task(task: grab.spider.task.Task, queue: None | BaseTaskQueue = None, raise_error: bool = False) → bool

Add task to the task queue.

stop() → None

Instruct spider to stop processing new tasks and start shutting down.

load_proxylist(source: str | BaseProxySource, source_type: None | str = None, proxy_type: str = 'http', auto_init: bool = True, auto_change: bool = True) → None

Load proxy list.

Parameters

• source – Proxy source. Accepts string (file path, url) or BaseProxySource instance.

• source_type – The type of the specified source. Should be one of the following: ‘text_file’ or ‘url’.

• proxy_type – Should be one of the following: ‘socks4’, ‘socks5’ or’http’.

• auto_change – If set to True then automatically random proxy rotation will be used.

Proxy source format should be one of the following (for each line): - ip:port - ip:port:login:password

render_stats() → str

prepare() → None

Do additional spider customization here.

This method runs before spider has started working.

shutdown() → None

Override this method to do some final actions after parsing has been done.

update_grab_instance(grab: grab.base.Grab) → None

Update config of any Grab instance created by the spider.

WTF it means?

create_grab_instance(**kwargs: Any) → grab.base.Grab

task_generator() → collections.abc.Iterator[grab.spider.task.Task]

You can override this method to load new tasks.

It will be used each time as number of tasks in task queue is less then number of threads multiplied on 2 This allows you to not overload all free memory if total number of tasks is big.

check_task_limits(task: grab.spider.task.Task) → tuple[bool, str]

Check that task’s network & try counters do not exceed limits.

Returns: * if success: (True, None) * if error: (False, reason)

generate_task_priority() → int

process_initial_urls() → None

get_task_from_queue() → None | Literal[True] | Task

setup_grab_for_task(task: grab.spider.task.Task) → grab.base.Grab

is_valid_network_response_code(code: int, task: grab.spider.task.Task) → bool

Test if response is valid.

Valid response is handled with associated task handler. Failed respoosne is processed with error handler.

process_parser_error(func_name: str, task: grab.spider.task.Task, exc_info: tuple[type[Exception], Exception, types.TracebackType]) → None

find_task_handler(task: grab.spider.task.Task) → collections.abc.Callable[Ellipsis, Any]

log_network_result_stats(res: grab.spider.service.network.NetworkResult, task: grab.spider.task.Task) → None

process_grab_proxy(task: grab.spider.task.Task, grab: grab.base.Grab) → None

Assign new proxy from proxylist to the task.

change_active_proxy(task: grab.spider.task.Task, grab: grab.base.Grab) → None

get_task_queue() → grab.spider.queue_backend.base.BaseTaskQueue

is_idle_estimated() → bool

is_idle_confirmed(services: list[grab.spider.service.base.BaseService]) → bool

Test if spider is fully idle.

WARNING: As side effect it stops all services to get state of queues anaffected by sercies.

Spider is full idle when all conditions are met: * all services are paused i.e. the do not change queues * all queues are empty * task generator is completed

run() → None

shutdown_services(services: list[grab.spider.service.base.BaseService]) → None

log_failed_network_result(res: grab.spider.service.network.NetworkResult) → None

log_rejected_task(task: grab.spider.task.Task, reason: str) → None

get_fallback_handler(task: grab.spider.task.Task) → None | Callable[..., Any]

srv_process_service_result(result: Task | None | Exception | dict[str, Any], task: grab.spider.task.Task, meta: None | dict[str, Any] = None) → None

Process result submitted from any service to task dispatcher service.

Result could be: * Task * None * Task instance * ResponseNotValid-based exception * Arbitrary exception * Network response:

{ok, ecode, emsg, exc, grab, grab_config_backup}

Exception can come only from parser_service and it always has meta {“from”: “parser”, “exc_info”: <…>}

srv_process_network_result(result: grab.spider.service.network.NetworkResult, task: grab.spider.task.Task) → None

srv_process_task(task: grab.spider.task.Task) → None

exception grab.spider.SpiderError

Bases: grab.errors.GrabError

Base class for Spider exceptions.

exception grab.spider.SpiderMisuseError

Bases: SpiderError

Improper usage of Spider framework.

exception grab.spider.FatalError

Bases: SpiderError

Fatal error which should stop parsing process.

exception grab.spider.SpiderInternalError

Bases: SpiderError

Raises when error throwned by internal spider logic.

Like spider class discovering, CLI error.

exception grab.spider.NoTaskHandler

Bases: SpiderError

Raise when no handler found to process network response.

exception grab.spider.NoDataHandler

Bases: SpiderError

Raise when no handler found to process Data object.

class grab.spider.Task(name: None | str = None, url: None | str = None, grab: None | Grab = None, grab_config: None | GrabConfig = None, priority: None | int = None, priority_set_explicitly: bool = True, network_try_count: int = 0, task_try_count: int = 1, valid_status: None | list[int] = None, use_proxylist: bool = True, delay: None | int = None, raw: bool = False, callback: None | Callable[..., None] = None, fallback_name: None | str = None, store: None | dict[str, Any] = None, disable_cache: bool = False, refresh_cache: bool = False, cache_timeout: None | int = None, **kwargs: Any)

Bases: BaseTask

Task for spider.

process_init_url_grab_options(url: None | str, grab: None | Grab, grab_config: None | GrabConfig) → None

get(key: str, default: Any = None) → Any

Return value of attribute or None if such attribute does not exist.

process_delay_option(delay: None | float) → None

setup_grab_config(grab_config: grab.types.GrabConfig) → None

test_clone_options_integrity(url: None | str, grab: None | Grab, grab_config: None | GrabConfig) → None

clone(**kwargs: Any) → Task

Clone Task instance.

Reset network_try_count, increase task_try_count. Reset priority attribute if it was not set explicitly.

__repr__() → str

Return repr(self).

__lt__(other: Task) → bool

Return self<value.

__eq__(other: object) → bool

Return self==value.