Home

C

• https://curl.haxx.se/libcurl/c/crawler.html
• https://www.monkey.org/~provos/crawl/
• http://www.aosabook.org/en/500L/a-web-crawler-with-asyncio-coroutines.html
• swish-e http://www.esa.org/tiee/search/html/spider.html
• sphinx http://sphinxsearch.com/docs/current.html

Go

• https://github.com/gocolly/colly
• https://github.com/jivesearch/jivesearch

Web-crawlers

• swish-e http://www.esa.org/tiee/search/html/spider.html
• Apache Solr + Nutch
• Xapian - Аксенов пишет что этот движок уже умер и есть только два opensource движка: sphinx и lucene
• harvest http://harvest.sourceforge.net/harvest/doc/index.html
• https://wiby.me/about.html
• https://github.com/Stazer/Crawler
• https://curl.haxx.se/libcurl/c/crawler.html
• Solr 5+ DOES in fact now do web crawling! http://lucene.apache.org/solr/
• Nutch - http://lucene.apache.org/nutch/
• Websphinx - http://www.cs.cmu.edu/~rcm/websphinx/
• JSpider - http://j-spider.sourceforge.net/
• Heritrix - http://crawler.archive.org/
• Web-archive https://webarchive.jira.com/wiki/spaces/Heritrix/overview
• http://www.crawl-anywhere.com/
• Simple web crawler
• Manticore
• wget as a crawler and sphinx as search engine
• Go https://github.com/PuerkitoBio/gocrawl
• Go: https://git.autistici.org/ale/crawl/
• Go: https://github.com/temoto/heroshi
• Sphinx
  • http://sphinxsearch.com/docs/current.html
  • https://pypi.python.org/pypi/sphinxsearch/0.1
  • python client https://github.com/grangier/sphinxapi/blob/master/sphinxapi.py

$ pkg_add sphinx--pgsql
/usr/local/bin/searchd -h

Технологии:

• http://www.yacyweb.de/

• https://censys.io/about

• Lucene, Solr, Sphinx, Xapian, Indri

• https://github.com/blevesearch/bleve

• FamilySearch https://familysearch.org/developers/

• Javascript https://github.com/FamilySearch/fs-js-lite

• как искать - http://www.genealogyintime.com/GenealogyResources/Articles/a_guide_to_performing_online_genealogy_searches_page01.html

• https://web.archive.org/web/20160324121529/http://codavr.ru/

• http://ljsearch.org/?hl=ru

• http://jeka.by/post/1014/google-search-engine

• http://tutorialzine.com/2010/09/google-powered-site-search-ajax-jquery/

• https://developers.google.com/custom-search/docs/tutorial/implementingsearchbox?hl=ru

Можно ли узнать свою родословную за три минуты?