عکس erfannoury
A pyLucene-based search module for searching books from goodreads.comJupyter Notebook
موضوع‌ها
۱
فورک‌ها
۲
ستاره‌ها
۲۵
تاریخ ایجاد
۱۱ دی ۱۳۹۳
آخرین بروزرسانی
نزدیک ۵ سال قبل
لایسنس
MIT License

GoodSearcher

A pyLucene-based crawl and search module for searching books from goodreads.com

Image of GoodSearcher homepage

Module Files Description
Crawling crawler.py, scraper.py, SetQueue.py These files contain the implementation of a multi-threaded crawler. Specifically, scraper.py is the implementation for scraping webpages from Goodreads website. crawler.py implements the crawler threads, thread-safe url queues and related stuff. A thread-safe queue with a built-in set for url deduplication is implemented in SetQueue.py
Indexing booktype.py, indexer.py, PageRank.py Indexing using two libraries has been implemented. You can choose between Elasticsearch and Lucene for indexing. For document boosting, PageRank scoring is also implemented.
Searching searcher.py It is a web.py-based minimal web server for searching indexes and displaying results in a web page.

Libraries used

Elasticsearch

Lucene

  • PyLucene: PyLucene is a Python extension for accessing Java Lucene.
  • PyLucene: PyLucene clone on Github

Web

  • Web.py: web.py is a web framework for Python that is as simple as it is powerful.
  • Beautifulsoup4: For web scraping

Scipy/Numpy was also used for calculating PageRank scores.