HUMAN Protocol partnered with the world-renowned SALK Institute to explore the comparative benefits of crowd-sourced annotation over professional annotation. This groundbreaking collaboration aimed to enhance the efficiency and accuracy of dataset annotation for Alzheimer's research, showcasing the potential of decentralized workforces in scientific endeavors.
Project Overview
Comparative Study
SALK Institute compared datasets annotated by HUMAN Protocol's workforce via CVAT integration against those annotated by a professional team.
Annotation Tasks
Annotators labeled still frames of mice using Single Point, Bounding Box, and Skeleton Annotation techniques to detect various features.
Complexity Management
Complex annotation tasks were broken down into multiple simple tasks to ensure accurate image annotation.
Annotation Techniques
Single Point Annotation
Identifies key objects within an image using a single point marker.
Bounding Box Annotation
Identifies the area of the image that the key object occupies using a rectangular box.
Skeleton Annotation
Identifies specific points of an object, such as head, ears, and tail, often split into multiple sub-tasks.
Impressive Outcomes
Crowd Accuracy
The second round of crowdsourced annotation achieved an impressive 85.1% accuracy, surpassing the 80% threshold.
Professional Accuracy
The professional in-house annotation team achieved a 74.7% accuracy rate, demonstrating the competitiveness of crowdsourced results.
Cost Savings
SALK reported that traditional annotation platforms would have cost 2 to 5 times more than crowdsourced annotation via HUMAN Protocol.
These results highlight the potential of crowdsourced annotation to deliver high-quality results at a fraction of the cost of traditional methods.
Key Findings
Superior Accuracy
Crowdsourced annotations matched or surpassed in-house annotations in quality.
Cost-Effectiveness
Significantly lower costs compared to traditional annotation methods.
Reduced Bias
Greater diversity of annotators led to more comprehensive and unbiased datasets.
Scalability
Ability to handle large-scale annotation tasks efficiently.
Technological Innovation
Successful integration of blockchain and AI for scientific research.
These findings demonstrate the transformative potential of HUMAN Protocol's crowdsourced annotation approach for scientific research and beyond.
Advantages of Crowdsourced Annotation
Diverse Perspectives
Thousands of annotators contribute, reducing individual biases and creating more comprehensive datasets applicable to various scenarios.
Stronger Consensus
More participants lead to a stronger consensus, helping to create technologies that accommodate diverse backgrounds and viewpoints.
Scalability
The ability to quickly mobilize a large workforce allows for rapid annotation of massive datasets, accelerating research timelines.
Future Directions
Expanded Experiments
Conduct larger-scale experiments to further validate the efficacy of crowdsourced annotation across various scientific domains.
Conference Presentations
Showcase HUMAN Protocol's capabilities at scientific conferences to demonstrate its potential for complex dataset annotation.
Semantic Segmentation
Implement semantic segmentation via CVAT integration to enable fine-tuning of AI models like GPT.
Cross-Blockchain Expansion
Explore annotation of massive datasets across different blockchains to enhance interoperability and reach.
HUMAN Protocol is actively seeking partners to participate in these expanded experiments and explore the full potential of decentralized annotation.
Join the HUMAN Protocol Ecosystem
Partnerships
Collaborate on groundbreaking research and data annotation projects.
Community
Join our Discord and social media channels to stay updated and contribute to discussions.
Innovation
Explore our open-source framework and contribute to the future of decentralized work.
For more information on partnering or joining our ecosystem, visit our website at humanprotocol.org or contact us at partnerships@hmt.ai.
Crowdsourced Annotation: HUMAN Protocol's Collaboration with SALK Institute