About Anansi
Anansi is a research project that uses Internet-connected computers to explore web resources around the world. Basically we want to do an evaluation regarding accuracy and performance on a distributed web crawler, and BOINC is our final choice after consideration. Measurements in such a system including accuracy, robustness, flexibility and performance will be evaluated.
In Anansi, clients returned only URIs been crawled associate with URI's http status code that indcating availability of it. Only URIs with scheme http itself that can be reached by the public will be crawled. No E-mail address, words content or user, password will be collected. It is an non-cpu-intensive project, which is trying to reduce CPU loads on the client. Associative information such as robots exclusion and some page contents are being collected and used by BOINC Volunteers during crawling, but none of them will be returned to Anansi server.
The data(URIs) collected by Anansi will be used by a Map-reduce engine that calculates priorities for each URI. The priority is established upon In-degree, out-degree and timestamp created by Anansi Server. Anansi server take it into consideration for revisit plans, which maintains a continuely working system.
Anansi is currently maintained by Operating Systems Research (OSR) Lab @ North Carolina State University
- More on Active Projects in OSR
- The project is currently maintained by Kunsheng Chen@LinkedIn
- Contact us through kchen3 AT ncsu.edu
About Stats
Stats info could be found at http://canis.csc.ncsu.edu:8005/anansi/stats and is provided for any third-party website on statistics purpose only.
Join Anansi
- Read our rules and policies
- This project uses BOINC. If you're already running BOINC, select Attach to Project. If not, download BOINC.
- When prompted, enter
http://canis.csc.ncsu.edu:8005/anansi/ - If you're running a command-line or pre-5.0 version of BOINC, create an account first.
- If you have any problems, get help here.
Returning participants
- Your account - view stats, modify preferences
- Teams - create or join a team
- Certificate
- Applications
.