BUbiNG

DOI Web Site 被引用文献1件
  • Paolo Boldi
    Dipartimento di Informatica, Università degli Studi di Milano, Milano, MI, Italy
  • Andrea Marino
    Dipartimento di Informatica, Università degli Studi di Milano, Milano, MI, Italy
  • Massimo Santini
    Dipartimento di Informatica, Università degli Studi di Milano, Milano, MI, Italy
  • Sebastiano Vigna
    Dipartimento di Informatica, Università degli Studi di Milano, Milano, MI, Italy

書誌事項

タイトル別名
  • Massive Crawling for the Masses

説明

<jats:p>Although web crawlers have been around for twenty years by now, there is virtually no freely available, open-source crawling software that guarantees high throughput, overcomes the limits of single-machine systems, and, at the same time, scales linearly with the amount of resources available. This article aims at filling this gap, through the description of BUbiNG, our next-generation web crawler built upon the authors’ experience with UbiCrawler [9] and on the last ten years of research on the topic. BUbiNG is an open-source Java fully distributed crawler; a single BUbiNG agent, using sizeable hardware, can crawl several thousand pages per second respecting strict politeness constraints, both host- and IP-based. Unlike existing open-source distributed crawlers that rely on batch techniques (like MapReduce), BUbiNG job distribution is based on modern high-speed protocols to achieve very high throughput.</jats:p>

収録刊行物

被引用文献 (1)*注記

もっと見る

詳細情報 詳細情報について

問題の指摘

ページトップへ