DotnetSpider is an MIT-licensed .NET Standard web crawling and scraping framework written in C#, aimed at .NET developers who need to build high-level crawlers that run either on a single machine or across a distributed cluster.

What it is

DotnetSpider is a .NET Standard library and framework for web crawling and scraping, distributed as the DotnetSpider NuGet package and maintained as a member project of the .NET Core Community (NCC). It is written in C# and lives in the .NET Core ecosystem, with development documented against Visual Studio 2017 (15.3 or later) or JetBrains Rider on .NET Core 2.2 or later. The registry topics describe it as a crawler, cross-platform, distributed, csharp, and dotnetcore project. The repository carries an Azure DevOps build pipeline for dotnetcore.DotnetSpider on master, and a companion wiki holds the fuller documentation.

The concrete problem it addresses is the plumbing between "fetch this URL" and "store this record". Instead of hand-writing download, scheduling, link-following, and parsing code per site, a developer declares an entity class inheriting EntityBase and annotates it with selectors such as [EntitySelector], [ValueSelector], [GlobalValueSelector], and [FollowRequestSelector], using SelectorType.XPath for expression evaluation. The EntitySpider sample shows this shape directly: a CnblogsEntry entity carrying [Schema("cnblogs", "news")], index configuration through HasIndex, and validation attributes including [Required] and [StringLength(200)]. What it replaces is bespoke, per-target extraction and persistence scaffolding.

Key capabilities

• Attribute-driven entity extraction: [EntitySelector], [ValueSelector], [GlobalValueSelector], and [FollowRequestSelector] declare what to scrape and which links to follow, with SelectorType.XPath evaluation.
• Configurable entity spider: the EntitySpider sample maps a page into a strongly typed class, including [ValueSelector(Expression = "类别", Type = SelectorType.Environment)] for values taken from request environment metadata.
• Storage schema and indexing declared in code: [Schema("cnblogs", "news")], HasIndex(x => x.Title), and composite indexes such as HasIndex(x => new { x.WebSite, x.Guid }, true).
• Fluent builder entry point: Builder.CreateDefaultBuilder, options.Speed, UseSerilog(), IgnoreServerCertificateError(), and Build().RunAsync().
• Lifecycle hooks: InitializeAsync, AddDataFlow, AddRequestsAsync, and Request objects carrying a metadata dictionary such as { "网站", "博客园" }.
• Distributed operation across message and coordination services: RabbitMQ, Redis (optional), HBase, and the Docker remote API, with socat exposing the Docker socket on port 2375 for macOS.
• Pluggable storage backends: MySql, SqlServer, PostgreSQL (optional), and MongoDb (optional) are all provisioned in the documented development environment.

Who uses it and how

• .NET teams working in Visual Studio 2017 (15.3 or later) or JetBrains Rider on .NET Core 2.2 or later, integrating the library through the DotnetSpider NuGet package.
• Distributed crawl deployments that use RabbitMQ as the message broker and the Docker remote API to run downloader or agent containers, with Redis optionally handling coordination.
• Data collection pipelines that write crawled entities into relational stores such as MySql, SqlServer, or PostgreSQL, or into MongoDb and HBase for larger volumes.
• Developers who need pre-release builds can add the MyGet feed to obtain the latest beta packages rather than waiting on the stable NuGet release.
• Single-machine development and evaluation follows the samples in the DotnetSpider.Sample project, beginning with BaseUsageSpider.cs.

Getting started

Install the DotnetSpider NuGet package, adding the MyGet feed only when beta packages are required, and run the samples under src/DotnetSpider.Sample/samples/ such as BaseUsageSpider.cs and EntitySpider.cs. The documented development environment is provisioned with docker run commands for MySql, Redis, SqlServer, PostgreSQL, MongoDb, RabbitMQ, and HBase.

How it compares

No comparable crawling or scraping tools are named in the facts provided, and no list of paid products it displaces is supplied, so it stands alone in this registry. Contrasts against commercial scraping services on licence, self-hosting, data ownership, or cost model cannot be drawn from the available material.

When to use it — and when not to

The operational burden is real: a working environment assumes MySql plus RabbitMQ, and potentially Redis, HBase, SqlServer, PostgreSQL, MongoDb, and a Docker remote API endpoint alongside the crawler itself, all of which a self-hoster must run and maintain. Teams outside the .NET ecosystem, or those wanting a managed service rather than a library, should look elsewhere. The README is thin as documentation — much of it is a Chinese-language disclaimer about lawful use plus a list of docker run commands — with real documentation pushed to the GitHub wiki, and the NuGet badge points at vpre, indicating pre-release packaging.

Please see the Project DotnetSpider.Sample in the solution.

[DisplayName("博客园爬虫")]
public class EntitySpider(
    IOptions options,
    DependenceServices services,
    ILogger logger)
    : Spider(options, services, logger)
{
    public static async Task RunAsync()
    {
        var builder = Builder.CreateDefaultBuilder(options =>
        {
            options.Speed = 1;
        });
        builder.UseSerilog();
        builder.IgnoreServerCertificateError();
        await builder.Build().RunAsync();
    }

    protected override async Task InitializeAsync(CancellationToken stoppingToken = default)
    {
        AddDataFlow>();
        AddDataFlow(GetDefaultStorage);
        await AddRequestsAsync(
            new Request(
                "https://news.cnblogs.com/n/page/1", new Dictionary { { "网站", "博客园" } }));
    }

    [Schema("cnblogs", "news")]
    [EntitySelector(Expression = ".//div[@]", Type = SelectorType.XPath)]
    [GlobalValueSelector(Expression = ".//a[@]", Name = "类别", Type = SelectorType.XPath)]
    [GlobalValueSelector(Expression = "//title", Name = "Title", Type = SelectorType.XPath)]
    [FollowRequestSelector(Expressions = ["//div[@]"])]
    public class CnblogsEntry : EntityBase
    {
        protected override void Configure()
        {
            HasIndex(x => x.Title);
            HasIndex(x => new { x.WebSite, x.Guid }, true);
        }

        public int Id { get; set; }

        [Required]
        [StringLength(200)]
        [ValueSelector(Expression = "类别", Type = SelectorType.Environment)]
        public string Category { get; set; }

        [Required]
        [StringLength(200)]
        [ValueSelector(Expression = "网站", Type = SelectorType.Environment)]
        public string WebSite { get; set; }

        [StringLength(200)]
        [ValueSelector(Expression = "Title", Type = SelectorType.Environment)]
        [ReplaceFormatter(NewValue = "", OldValue = " - 博客园")]
        public string Title { get; set; }

        [StringLength(40)]
        [ValueSelector(Expression = "GUID", Type = SelectorType.Environment)]
        public string Guid { get; set; }

        [ValueSelector(Expression = ".//h2[@]/a")]
        public string News { get; set; }

        [ValueSelector(Expression = ".//h2[@]/a/@href")]
        public string Url { get; set; }

        [ValueSelector(Expression = ".//div[@]")]
        [TrimFormatter]
        public string PlainText { get; set; }

        [ValueSelector(Expression = "DATETIME", Type = SelectorType.Environment)]
        public DateTime CreationTime { get; set; }
    }
}

timeout 0
tcp-keepalive 60