xfnw/searpl
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
tiny php sqlite search engine
21 commits 1 branch 0 tags 44 KiB
PHP 72.1%
CSS 22.1%
Shell 5.8%
Find a file
xfnw 826e3c2b7c delete page before downloading new one, so dead pages do not sit in the database
2021-01-27 20:33:25 -05:00
assets dont use a cdn for one little search icon 2020-12-16 21:42:08 -05:00
.gitignore uniquely sort urls and put the db the gitignore 2020-12-31 21:05:50 -05:00
crawl.php delete page before downloading new one, so dead pages do not sit in the database 2021-01-27 20:33:25 -05:00
create.sql some instructions 2020-12-15 10:59:23 -05:00
index.php re-crawl sites 2020-12-22 10:13:45 -05:00
LICENCE some instructions 2020-12-15 10:59:23 -05:00
README.md no more cloudflare 2020-12-29 16:40:36 -05:00
urls.sh crawl slower so sites wont ban it lol 2021-01-16 13:39:46 -05:00

README.md

searpl

searpl is a small php search engine with the following features:

  • robot.txt compliant
  • sqlite, so theres no need to run some fancy database daemon
  • javascript-free
  • no cdns!
  • read-only database, nothing is written except with the shell

licensing

searpl is licenced under an MIT licence, see LICENSE for more information

setup

this guide assumes you have shell access and are comfortable using command line tools like git.

  • make sure you have php, php-pdo, wget, sqlite3 and git installed
  • go in your htdocs, public_html or whatever and git clone this repo
  • touch db.sqlite to create the database
  • copy the contents of create.sql and paste it into the prompt on sqlite3 db.sqlite to create the table

crawling

to crawl a site, do ./urls.sh https://example.com

to recursively crawl, change the recursion limit with -l

./urls.sh -l5 https://example.com