xfnw
|
c6b476c4a3
|
crawl slower so sites wont ban it lol
|
2021-01-16 13:39:46 -05:00 |
|
xfnw
|
d4d5e67da8
|
uniquely sort urls and put the db the gitignore
|
2020-12-31 21:05:50 -05:00 |
|
xfnw
|
be5b250319
|
go through down sites faster
|
2020-12-15 12:33:00 -05:00 |
|
xfnw
|
5eb94a6834
|
some instructions
|
2020-12-15 10:59:23 -05:00 |
|
xfnw
|
2b46e8899f
|
set a useragent
|
2020-12-15 10:17:28 -05:00 |
|
xfnw
|
8c4421108b
|
dont track svgs and drop the / from the end of urls so they wont be duplicated
|
2020-12-15 09:59:19 -05:00 |
|
xfnw
|
c62b48ae1f
|
unique urls
|
2020-12-14 17:22:28 -05:00 |
|
xfnw
|
69de9f49dc
|
working
|
2020-12-14 16:59:16 -05:00 |
|