--- title: Warbler CDA RAG System emoji: 🦜 colorFrom: blue colorTo: purple sdk: gradio sdk_version: 5.49.1 app_file: app.py pinned: false license: mit tags: - rag - retrieval - semantic-search - stat7 - embeddings - nlp --- ## Warbler CDA - Cognitive Development Architecture RAG System [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/) [![HuggingFace](https://img.shields.io/badge/%F0%9F%A4%97-HuggingFace-orange)](https://huggingface.co/) A production-ready RAG (Retrieval-Augmented Generation) system with **STAT7 multi-dimensional addressing** for intelligent document retrieval and semantic memory. ## 🌟 Features ### Core RAG System - **Semantic Anchors**: Persistent memory with provenance tracking - **Hierarchical Summarization**: Micro/macro distillation for efficient compression - **Conflict Detection**: Automatic detection and resolution of contradictory information - **Memory Pooling**: Performance-optimized object pooling for high-throughput scenarios ### STAT7 Multi-Dimensional Addressing - **7-Dimensional Coordinates**: Realm, Lineage, Adjacency, Horizon, Luminosity, Polarity, Dimensionality - **Hybrid Scoring**: Combines semantic similarity with STAT7 resonance for superior retrieval - **Entanglement Detection**: Identifies relationships across dimensional space - **Validated System**: Comprehensive experiments (EXP-01 through EXP-10) validate uniqueness, efficiency, and narrative preservation ### Production-Ready API - **FastAPI Service**: High-performance async API with concurrent query support - **CLI Tools**: Command-line interface for queries, ingestion, and management - **HuggingFace Integration**: Direct ingestion from HF datasets - **Docker Support**: Containerized deployment ready ## 📚 Data Sources The Warbler system is trained on carefully curated, MIT-licensed datasets from HuggingFace: ### Primary Datasets - **arXiv Papers** (`nick007x/arxiv-papers`) - 2.5M+ scholarly papers covering scientific domains - **Prompt Engineering Report** (`PromptSystematicReview/ThePromptReport`) - 83 comprehensive prompt documentation entries - **Generated Novels** (`GOAT-AI/generated-novels`) - 20 narrative-rich novels for storytelling patterns - **Technical Manuals** (`nlasso/anac-manuals-23`) - 52 procedural and operational documents - **ChatEnv Enterprise** (`SustcZhangYX/ChatEnv`) - 112K+ software development conversations - **Portuguese Education** (`Solshine/Portuguese_Language_Education_Texts`) - 21 multilingual educational texts - **Educational Stories** (`MU-NLPC/Edustories-en`) - 1.5K+ case studies and learning narratives ### Original Warbler Packs - `warbler-pack-core` - Core narrative and reasoning patterns - `warbler-pack-wisdom-scrolls` - Philosophical and wisdom-based content - `warbler-pack-faction-politics` - Political and faction dynamics All datasets are provided under MIT or compatible licenses. For complete attribution, see the HuggingFace Hub pages listed above. ## 📦 Installation ### From PyPI (when published) ```bash pip install warbler-cda ``` ### From Source ```bash git clone https://github.com/tiny-walnut-games/the-seed.git cd the-seed/warbler-cda-package pip install -e . ``` ### With Optional Dependencies ```bash # OpenAI embeddings pip install warbler-cda[openai] # Performance optimizations pip install warbler-cda[performance] # Development tools pip install warbler-cda[dev] ``` ## 🚀 Quick Start ### Basic Usage ```python from warbler_cda import RetrievalAPI, SemanticAnchorGraph, EmbeddingProviderFactory # Initialize components embedding_provider = EmbeddingProviderFactory.get_default_provider() semantic_anchors = SemanticAnchorGraph(embedding_provider=embedding_provider) # Create retrieval API api = RetrievalAPI( semantic_anchors=semantic_anchors, embedding_provider=embedding_provider ) # Add documents api.add_document( doc_id="doc1", content="The Warbler CDA system provides intelligent retrieval.", metadata={"realm_type": "documentation", "realm_label": "system_docs"} ) # Query results = api.query_semantic_anchors("How does Warbler CDA work?", max_results=5) for result in results: print(f"Score: {result.relevance_score:.3f} - {result.content}") ``` ### STAT7 Hybrid Scoring ```python from warbler_cda import STAT7RAGBridge # Enable STAT7 hybrid scoring stat7_bridge = STAT7RAGBridge() api = RetrievalAPI( semantic_anchors=semantic_anchors, embedding_provider=embedding_provider, stat7_bridge=stat7_bridge, config={"enable_stat7_hybrid": True} ) # Query with hybrid scoring from warbler_cda import RetrievalQuery, RetrievalMode query = RetrievalQuery( query_id="hybrid_query_1", mode=RetrievalMode.SEMANTIC_SIMILARITY, semantic_query="Find wisdom about resilience", stat7_hybrid=True, weight_semantic=0.6, weight_stat7=0.4 ) assembly = api.retrieve_context(query) print(f"Found {len(assembly.results)} results with quality {assembly.assembly_quality:.3f}") ``` ### Running the API Service ```bash # Start the FastAPI service uvicorn warbler_cda.api.service:app --host 0.0.0.0 --port 8000 # Or use the CLI warbler-api --port 8000 ``` ### Using the CLI ```bash # Query the API warbler-cli query --query-id q1 --semantic "wisdom about courage" --max-results 10 # Enable hybrid scoring warbler-cli query --query-id q2 --semantic "narrative patterns" --hybrid # Bulk concurrent queries warbler-cli bulk --num-queries 10 --concurrency 5 --hybrid # Check metrics warbler-cli metrics ``` ## 📊 STAT7 Experiments The system includes validated experiments demonstrating: - **EXP-01**: Address uniqueness (0% collision rate across 10K+ entities) - **EXP-02**: Retrieval efficiency (sub-millisecond at 100K scale) - **EXP-03**: Dimension necessity (all 7 dimensions required) - **EXP-10**: Narrative preservation under concurrent load ```python from warbler_cda import run_all_experiments # Run validation experiments results = run_all_experiments( exp01_samples=1000, exp01_iterations=10, exp02_queries=1000, exp03_samples=1000 ) print(f"EXP-01 Success: {results['EXP-01']['success']}") print(f"EXP-02 Success: {results['EXP-02']['success']}") print(f"EXP-03 Success: {results['EXP-03']['success']}") ``` ## 🎯 Use Cases ### 1. Intelligent Document Retrieval ```python # Add documents from various sources for doc in documents: api.add_document( doc_id=doc["id"], content=doc["text"], metadata={ "realm_type": "knowledge", "realm_label": "technical_docs", "lifecycle_stage": "emergence" } ) # Retrieve with context awareness results = api.query_semantic_anchors("How to optimize performance?") ``` ### 2. Narrative Coherence Analysis ```python from warbler_cda import ConflictDetector conflict_detector = ConflictDetector(embedding_provider=embedding_provider) # Process statements statements = [ {"id": "s1", "text": "The system is fast"}, {"id": "s2", "text": "The system is slow"} ] report = conflict_detector.process_statements(statements) print(f"Conflicts detected: {report['conflict_summary']}") ``` ### 3. HuggingFace Dataset Ingestion ```python from warbler_cda.utils import HFWarblerIngestor ingestor = HFWarblerIngestor() # Transform HF dataset to Warbler format docs = ingestor.transform_npc_dialogue("amaydle/npc-dialogue") # Create pack pack_path = ingestor.create_warbler_pack(docs, "warbler-pack-npc-dialogue") ``` ## 🏗️ Architecture ```none warbler_cda/ ├── retrieval_api.py # Main RAG API ├── semantic_anchors.py # Semantic memory system ├── anchor_data_classes.py # Core data structures ├── anchor_memory_pool.py # Performance optimization ├── summarization_ladder.py # Hierarchical compression ├── conflict_detector.py # Conflict detection ├── castle_graph.py # Concept extraction ├── melt_layer.py # Memory consolidation ├── evaporation.py # Content distillation ├── stat7_rag_bridge.py # STAT7 hybrid scoring ├── stat7_entity.py # STAT7 entity system ├── stat7_experiments.py # Validation experiments ├── embeddings/ # Embedding providers │ ├── base_provider.py │ ├── local_provider.py │ ├── openai_provider.py │ └── factory.py ├── api/ # Production API │ ├── service.py # FastAPI service │ └── cli.py # CLI interface └── utils/ # Utilities ├── load_warbler_packs.py └── hf_warbler_ingest.py ``` ## 🔬 Technical Details ### STAT7 Dimensions 1. **Realm**: Domain classification (type + label) 2. **Lineage**: Generation/version number 3. **Adjacency**: Graph connectivity (0.0-1.0) 4. **Horizon**: Lifecycle stage (logline, outline, scene, panel) 5. **Luminosity**: Clarity/activity level (0.0-1.0) 6. **Polarity**: Resonance/tension (0.0-1.0) 7. **Dimensionality**: Complexity/thread count (1-7) ### Hybrid Scoring Formula ```math hybrid_score = (weight_semantic × semantic_similarity) + (weight_stat7 × stat7_resonance) ``` Where: - `semantic_similarity`: Cosine similarity of embeddings - `stat7_resonance`: Multi-dimensional alignment score - Default weights: 60% semantic, 40% STAT7 ## 📚 Documentation - [API Reference](docs/api.md) - [STAT7 Guide](docs/stat7.md) - [Experiments](docs/experiments.md) - [Deployment](docs/deployment.md) ## 🤝 Contributing Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. ## 📄 License MIT License - see [LICENSE](LICENSE) for details. ## 🙏 Acknowledgments - Built on research from The Seed project - STAT7 addressing system inspired by multi-dimensional data structures - Semantic anchoring based on cognitive architecture principles ## 📞 Contact - **Project**: [The Seed](https://github.com/tiny-walnut-games/the-seed) - **Issues**: [GitHub Issues](https://github.com/tiny-walnut-games/the-seed/issues) - **Discussions**: [GitHub Discussions](https://github.com/tiny-walnut-games/the-seed/discussions) --- ## **Made with ❤️ by Tiny Walnut Games** Check out the configuration reference at