Jeff Atwood
|
ceef06e771
|
add support for "Save Page Now" archive.org/web
|
2015-01-06 01:05:45 -08:00 |
|
riking
|
37dbc4b5e6
|
Add archive.org to crawler list to serve no-js to
|
2014-11-02 16:51:23 -08:00 |
|
Vikhyat Korrapati
|
e3702ecb30
|
Improved crawler detection: add Twitterbot, Facebook, curl, Bing, Baidu.
|
2014-03-16 19:30:20 +05:30 |
|
Robin Ward
|
c4b5455c21
|
REFACTOR: Rename GooglebotDetection to CrawlerDetection because we
will likely whitelist more crawlers in the future.
|
2014-02-20 16:07:02 -05:00 |
|