Metadata-Version: 2.1
Name: crawl-bot
Version: 0.1.3
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
Requires-Dist: scrapy

# CrawlBot

CrawlBot is a Scrapy-based project designed to crawl specified domains and extract various webpage components such as titles, headings, images, and links. This project supports dynamic configuration and can be used to run different spiders with specified start URLs.

## Table of Contents

- [Installation](#installation)
- [Usage](#usage)
  - [Spiders](#spiders)
  - [Command-Line Usage](#command-line-usage)
  - [Programmatic Usage](#programmatic-usage)
- [Project Structure](#project-structure)
- [Contributing](#contributing)
- [License](#license)

## Installation

To install the CrawlBot package, use pip:

\```bash
pip install crawl_bot
\```

## Usage

### Spiders

This project includes the following spiders:

- `BasicSpider`: A basic spider that extracts titles, headings, images, links, etc.

### Command-Line Usage

You can run the spiders from the command line using the `run_spider` command. Replace `<spider_name>` with the name of the spider you want to run and provide the start URLs:

\```bash
run_spider <spider_name> <url1> <url2> ... <urlN>
\```

Example:

\```bash
run_spider basic_spider http://example.com http://another-example.com
\```

### Programmatic Usage

You can also run the spiders programmatically from another Python script:

\```python
from crawl_bot.run_spider import run_spider

spider_name = 'basic_spider'
start_urls = ['http://example.com', 'http://another-example.com']
run_spider(spider_name, start_urls)

\```

## Project Structure

Here is an overview of the project structure:

- **scrapy.cfg**: Scrapy configuration file.
- **my_scrapy_project/**: Directory containing the Scrapy project.
  - **items.py**: Defines the items that will be scraped.
  - **middlewares.py**: Custom middlewares for the Scrapy project.
  - **pipelines.py**: Pipelines for processing scraped data.
  - **settings.py**: Configuration settings for the Scrapy project.
  - **spiders/**: Directory containing the spiders.
    - **basic_spider.py**: Basic spider implementation.
    - **another_spider.py**: Another example spider.
- **run_spider.py**: Script to run the spiders.
- **setup.py**: Setup script for installing the package.
- **MANIFEST.in**: Configuration for including additional files in the package.
- **README.md**: Project documentation.

## Contributing

We welcome contributions to CrawlBot! If you have an idea for a new feature or have found a bug, please open an issue or submit a pull request. Here's how you can contribute:

1. Fork the repository.
2. Create a new branch: `git checkout -b my-feature-branch`
3. Make your changes and commit them: `git commit -m 'Add some feature'`
4. Push to the branch: `git push origin my-feature-branch`
5. Open a pull request.

Please ensure your code adheres to the project's coding standards and includes appropriate tests.

## License

This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
