6 commits

Author	SHA1	Message	Date
xfnw	be5b250319	go through down sites faster	2020-12-15 12:33:00 -05:00
xfnw	5eb94a6834	some instructions	2020-12-15 10:59:23 -05:00
xfnw	2b46e8899f	set a useragent	2020-12-15 10:17:28 -05:00
xfnw	8c4421108b	dont track svgs and drop the / from the end of urls so they wont be duplicated	2020-12-15 09:59:19 -05:00
xfnw	c62b48ae1f	unique urls	2020-12-14 17:22:28 -05:00
xfnw	69de9f49dc	working	2020-12-14 16:59:16 -05:00