Jan 2, 2019, 12:40 AM ETInterview

a16z Podcast | The Taxonomy of Collective Knowledge

Data Ontologies are structured frameworks that add order to massive datasets, defining how entities relate and classify relative to one another.
Historical Origin: The term originates from philosophy, specifically the study of the nature of being, as articulated by Ludwig Wittgenstein's theory on the limits of language defining the limits of the world.
Human-Computer Interface: Ontologies enable scalable coordination between humans and machines by providing a shared, mutually agreed-upon structure for representing reality.
Implicit Human Ontologies: Humans possess inherent ontologies that often conflict; establishing a common ontology is essential to prevent chaos and enable flexible, large-scale coordination.
Machine Limitations: Computers lack innate world knowledge (e.g., knowing a "cat" is an "animal") and rely on ontologies to process unstructured data effectively.

reCAPTCHA Mechanism: Leveraged 100 million daily users to digitize books by requiring users to type distorted text, effectively outsourcing optical character recognition (OCR) to humans.
Ground Truth Validation: Initial ground truth was established by requiring consensus among ten different users typing the same word to filter out errors.
Duolingo Complexity: Translation proved harder than text recognition due to the existence of multiple valid translations for a single sentence, preventing simple agreement-based validation.
Duolingo Solution: Implemented a voting system where users translate, others vote/correct, and a weighted scoring system accounts for user accuracy and reliability.
Scalability Challenge: The system must optimize between human effort and machine capability, as the computer acts as a pre-filter to minimize the need for human intervention on easy cases.
Efficiency Gains: By weighting users based on performance with known-answer "test" items, the system reduces the number of humans required per data point, improving both accuracy and throughput.
Terminology Distinction:
- Crowdsourcing: Using multiple people to reach an answer.
- Human-based Computation: Outsourcing tasks that machines find difficult to humans.
- Collective Intelligence: Coalescing the capabilities of humans, machines, and organizations to solve complex problems better than any single entity could.

Collective Diagnosis: An open platform where physicians independently solve clinical cases, allowing for the comparison of individual versus collective diagnostic performance.
Performance Metrics: Collectives of physicians have been observed to outperform 90% of individual physicians when solving standardized reference cases.
Clinical Quotient: The system measures a physician's specific expertise (e.g., a radiologist vs. an endocrinologist) to weight their opinions appropriately within a collective answer.
Ontology Standard: Uses the World Health Organization's International Classification of Diseases (ICD) to structure diagnoses, enabling direct one-to-one comparison of cases across different users.
Hybrid Knowledge Integration: Final decisions combine collective human judgment with clinical guidelines, research data, and electronic medical records.
Truth Definition: Truth is viewed as an approximation that asymptotically approaches reality as the number of information sources and interacting agents increases.
Augmentation Goal: The primary aim is to extend the capacity of existing physicians to serve underserved populations without replacing doctors.
Scalability: The project seeks to encode physician decision-making processes to allow knowledge transfer at zero marginal cost to patients lacking access to specialists.

Monetary Incentives: Direct payment (e.g., Amazon Mechanical Turk) is effective but requires significant overhead to prevent fraud and ensure quality.
Utility Incentives: Users contribute when the task solves an immediate personal need, such as completing a reCAPTCHA to access a ticket purchase or a website.
Reciprocity: In Duolingo, users learn a language in exchange for translating content, creating a mutual benefit loop.
Epistemic Meaning: Contributors are motivated by the "epic meaning" of creating open-source tools or shared knowledge resources (e.g., Wikipedia, Linux).
Social Impact: In projects like Folding@Home, users are driven by the desire to contribute to scientific research against diseases like Alzheimer's.
Gamification: Scoreboards, badges, and competitive mechanics effectively engage users in micro-interactions.
Impact-Based Weighting: HumanDx uses a market-like mechanism to give higher "impact" scores to contributions that fill critical knowledge gaps in the system.
Decentralized Tokens: Emerging blockchain models propose using application-specific tokens to compensate participants in distributed knowledge networks (e.g., Augur).

Evolving Ontologies: Unlike static frameworks (e.g., Encyclopedia Britannica), modern ontologies must dynamically evolve via machine suggestions and human validation (e.g., splitting or merging medical conditions).
Human vs. Machine Roles: While AI excels at pattern recognition in narrow domains, humans remain superior in tasks requiring synthesis of information across multiple scales (e.g., subatomic biology to childhood social environment).
Interpretability Gap: Machine learning outputs often lack human interpretability; ontologies provide the necessary bridge to make AI decisions intelligible and actionable for humans.
Governance Applications: The concept of weighting opinions by expertise (epistocracy) suggests potential applications in legal system inconsistency detection, patent analysis, and democratic voting structures.
Healthcare Data Complexity: Integrating unstructured data (genomics, images) with human-interpretable data (symptoms, social history) remains a unique and significant challenge in medical ontologies.
Inverting the Pyramid: The long-term vision is to use scalable knowledge creation to empower billions of people currently unserved by the global healthcare system with direct access to diagnostic and treatment logic.