Wayback Image Scraper Base

A modular, object-oriented web scraping system for downloading images from the Wayback Machine. This refactored version transforms the original monolithic script into a well-structured, extensible package with proper separation of concerns.

• Modular Architecture: Clean separation between crawling, downloading, database operations, and utilities
• Abstract Base Classes: Proper inheritance hierarchy for easy extension and customization
• Robust Error Handling: Comprehensive retry mechanisms and error recovery
• Blocklist Support: Flexible URL filtering with file-based and programmatic blocklists
• Database Persistence: SQLite-based storage with transaction safety
• Batch Processing: Efficient batch downloading with configurable batch sizes
• Analytics Support: Built-in statistics and reporting capabilities

wayback-image-scraper/
├── main.py                 # Main entry point
├── scraper/               # Core package
│   ├── __init__.py        # Package initialization
│   ├── config.py          # Configuration constants
│   ├── database.py        # Database operations
│   ├── blocklist.py       # URL filtering
│   ├── crawler.py         # Web crawling logic
│   ├── downloader.py      # Image downloading
│   └── utils.py           # Utility functions
├── advanced_example.py    # Advanced usage examples
├── setup.py              # Package setup
├── requirements.txt      # Dependencies
└── README.md            # This file

git clone <repository-url>
cd wayback-image-scraper
pip install -r requirements.txt

pip install -e .

# Crawl a specific domain
python main.py --domain example.com --image-dir ./images

# Crawl from a specific Wayback URL
python main.py --start-url "https://web.archive.org/web/20200101000000/https://example.com" --image-dir ./images

# Only crawl (don't download)
python main.py --domain example.com --crawl-only

# Only download pending images
python main.py --download-only --image-dir ./images

from scraper import DatabaseManager, BlocklistManager, WaybackCrawler, ImageDownloader

# Initialize components
db = DatabaseManager()
db.setup()

blocklist = BlocklistManager()
crawler = WaybackCrawler(db, blocklist)
downloader = ImageDownloader(db, blocklist)

# Crawl and download
crawler.crawl_domain_snapshots("example.com")
downloader.download_pending_images("./images")

• DatabaseInterface: Abstract base class defining database operations
• DatabaseManager: Concrete SQLite implementation with image URL tracking, status management, and statistics

• BlocklistInterface: Abstract base class for URL filtering
• BlocklistManager: File-based blocklist with pattern matching

• CrawlerInterface: Abstract base class for web crawlers
• BaseCrawler: Common crawling functionality with URL extraction and filtering
• WaybackCrawler: Wayback Machine specific implementation with CDX API integration

• DownloaderInterface: Abstract base class for downloaders
• BaseDownloader: Common download functionality with retry logic
• ImageDownloader: Standard image downloader with HTML wrapper handling
• BatchDownloader: Enhanced version with batch processing capabilities

• URLUtils: URL manipulation and validation
• FileUtils: File system operations and path handling
• ContentUtils: Content-type analysis and validation

Centralized configuration management with:

• Crawling limits and delays
• File paths and database settings
• HTTP headers and session management
• Image file extensions

You can extend any of the base classes to create custom behavior:

from scraper import BaseCrawler, DatabaseManager, BlocklistManager

class CustomCrawler(BaseCrawler):
    def _should_crawl_url(self, url):
        # Add custom URL filtering logic
        return "important" in url.lower()

# Use custom implementation
db = DatabaseManager()
blocklist = BlocklistManager()
crawler = CustomCrawler(db, blocklist)

from scraper import DatabaseManager

db = DatabaseManager()
stats = db.get_stats()

print(f"Total images found: {stats['total']}")
print(f"Downloaded: {stats.get('downloaded', 0)}")
print(f"Failed: {stats.get('failed', 0)}")

from scraper import BlocklistManager
import re

class PatternBlocklist(BlocklistManager):
    def __init__(self, patterns=None):
        super().__init__()
        self.patterns = patterns or []
    
    def is_blocked(self, url):
        if super().is_blocked(url):
            return True
        return any(pattern.search(url) for pattern in self.patterns)

# Usage with regex patterns
patterns = [re.compile(r'.*ads.*', re.IGNORECASE)]
blocklist = PatternBlocklist(patterns)

• requests: HTTP client for web requests
• beautifulsoup4: HTML parsing and extraction
• lxml: Fast XML/HTML parser (optional but recommended)

• SCRAPER_DB_PATH: Custom database path
• SCRAPER_LOG_LEVEL: Logging level (DEBUG, INFO, WARNING, ERROR)

• blocklist.txt: Line-separated list of blocked URL patterns
• image_archive.db: SQLite database for image tracking

The system includes comprehensive error handling:

• Network timeouts: Automatic retry with exponential backoff
• Database errors: Transaction rollback and logging
• File system errors: Directory creation and permission handling
• Content parsing errors: Graceful degradation with logging

• Rate Limiting: Built-in delays to respect server resources
• Batch Processing: Configurable batch sizes for efficient downloading
• Database Optimization: Indexed columns and prepared statements
• Memory Management: Streaming downloads for large files

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request

This project is licensed under the MIT License. See LICENSE file for details.

For issues, feature requests, or questions:

• Open an issue on GitHub
• Check the advanced_example.py for usage patterns
• Review the inline documentation in each module