icrawler is an MIT-licensed Python framework for building multi-thread crawlers, shipping ready-made crawlers for Google, Bing and Baidu image search plus Flickr, and is aimed at developers who need to harvest image or media datasets without writing their own threading, retry and queueing code.

What it is

icrawler describes itself as a mini framework of web crawlers, built around a modularization design that makes it easy to use and extend. It is a Python package (Python 3.5 or later recommended) published on PyPI and on Anaconda, and it targets media data such as images and videos especially well, while still being applicable to texts and other types of files. Internally a crawler consists of three components — Feeder, Parser and Downloader — connected with each other by FIFO queues. The url_queue stores the urls of pages that may contain images, and the task_queue stores the image url plus any metadata, each element being a dictionary that must contain the field img_url. Feeder puts page urls into url_queue, Parser requests and parses the page and extracts image urls into task_queue, and Downloader takes tasks from task_queue, requests the images and saves them to the given path. All three are thread pools, so the number of threads each uses is configurable.

The concrete problem it solves is the boilerplate that surrounds any threaded scraper. Writing a multiple-thread crawler by hand means dealing with exception handling, thread scheduling and thread communication; icrawler keeps those away so the author focuses only on the contents to crawl. It also removes the need to write a separate site-specific scraper for each image source, because crawlers for Flickr and for the Google, Bing and Baidu search engines are already provided.

Key capabilities

• Built-in crawlers for Google, Bing and Baidu image search and for Flickr, imported from icrawler.builtin, for example GoogleImageCrawler.
• A one-call crawl API: google_crawler.crawl(keyword='cat', max_num=100).
• Per-stage thread control through feeder_threads, parser_threads and downloader_threads.
• Output location set through storage={'root_dir': 'your_image_dir'}, with file_idx_offset to continue numbering from an earlier run.
• Advanced search filters including size, color, license='commercial,modify' and a date tuple range.
• Extensibility for custom sources, since the framework documents writing your own crawlers as well as using the built-in ones.
• Support for media beyond images, covering videos and other file types.

Who uses it and how

• Developers assembling image datasets, who point GoogleImageCrawler at a keyword such as cat with a max_num of 1000 and let the thread pools fill a local directory.
• Users who need licence-constrained material, filtering on license='commercial,modify' and on size, colour and date windows.
• Teams that need a crawler for their own target site and reuse the Feeder, Parser and Downloader structure instead of building queues and thread pools from scratch.
• Anyone who finds Scrapy too heavy for a focused media-harvesting job, since icrawler positions itself as the tiny and flexible alternative.

Getting started

Install with pip install icrawler or conda install -c hellock icrawler. The minimal example imports GoogleImageCrawler from icrawler.builtin, constructs it with a storage root directory, and calls crawl(keyword='cat', max_num=100).

How it compares

The README contrasts it directly with Scrapy: Scrapy is heavy and powerful, while icrawler is tiny and flexible. Both appear in the same topic space, so the choice comes down to whether a full-scale scraping framework is needed or a small threaded media crawler is enough. No paid commercial alternatives are named in the supplied facts, so no licence, hosting or cost comparison is made here.

When to use it — and when not to

Choose it when the job is pulling images by keyword or from Flickr and the value lies in thread management and per-site parsing being handled already. Avoid it if the project needs Scrapy's breadth of scraping features, or if the built-in crawlers' dependence on Google, Bing, Baidu and Flickr serving the pages their parsers expect is a problem. Note also that the README defers the more advanced usage of the built-in crawlers and the tutorials for writing custom ones to the ReadTheDocs site, so the repository overview alone is thin on detail.

from icrawler.builtin import GoogleImageCrawler

google_crawler = GoogleImageCrawler(storage={'root_dir': 'your_image_dir'})
google_crawler.crawl(keyword='cat', max_num=100)

from icrawler.builtin import GoogleImageCrawler

google_crawler = GoogleImageCrawler(
    feeder_threads=1,
    parser_threads=2,
    downloader_threads=4,
    storage={'root_dir': 'your_image_dir'})
filters = dict(
    size='large',
    color='orange',
    license='commercial,modify',
    date=((2017, 1, 1), (2017, 11, 30)))
google_crawler.crawl(keyword='cat', filters=filters, max_num=1000, file_idx_offset=0)