xfnw/searpl
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

uniquely sort urls and put the db the gitignore

Browse source
This commit is contained in:
xfnw 2020-12-31 21:05:50 -05:00
parent 036e3addb2
commit d4d5e67da8
2 changed files with 3 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.gitignore vendored Normal file
View file

@ -0,0 +1,2 @@
db.sqlite

1
urls.sh
View file

@ -2,6 +2,7 @@ wget --spider --force-html --tries 1 --timeout 1 -r -l1 -H -U 'searplbot/1.0' $@
grep '^--' wg | awk '{ print $3 }' \ grep '^--' wg | awk '{ print $3 }' \
| grep -v '\.\(css\|js\|png\|gif\|jpg\|txt\|ico\|ttf\|svg\)$' \ | grep -v '\.\(css\|js\|png\|gif\|jpg\|txt\|ico\|ttf\|svg\)$' \
| sort | uniq \
| tee -a ur | tee -a ur
rm wg rm wg