archive/docs/QUICK_REFERENCE_GUIDE.md

Quick Reference Guide: Crypto-DT-Source Implementation

Quick lookup for common tasks during implementation

---

🚀 Start Here

Launch Development Server

cd /home/user/crypto-dt-source
python api_server_extended.py
# Opens on http://localhost:8000

Access Documentation

Swagger UI:  http://localhost:8000/docs
ReDoc:       http://localhost:8000/redoc

Test Endpoints

# Health check
curl http://localhost:8000/api/health | jq

# Real-time prices
curl 'http://localhost:8000/api/prices?symbols=BTC,ETH' | jq

# Trending coins
curl http://localhost:8000/api/trending | jq

# Sentiment analysis
curl -X POST http://localhost:8000/api/sentiment/analyze \
  -H "Content-Type: application/json" \
  -d '{"text":"Bitcoin price is going up"}'

---

📁 Key Files to Modify

API Endpoints

• api_server_extended.py - Main FastAPI application + all endpoints
• api/endpoints.py - Additional endpoint definitions
• api/websocket.py - WebSocket handling

Data Collection

• collectors/market_data_extended.py - Price data fetching
• collectors/sentiment_extended.py - Sentiment analysis
• collectors/news.py - News aggregation

Database

• database/db_manager.py - Database connection
• database/models.py - Table definitions
• database/migrations.py - Schema setup

AI Models

• ai_models.py - HuggingFace model loading

Utilities

• utils/auth.py - Authentication
• utils/rate_limiter_enhanced.py - Rate limiting
• log_manager.py - Logging

---

💻 Common Commands

Database Operations

# Initialize database
python -c "from database.db_manager import DBManager; \
  import asyncio; \
  asyncio.run(DBManager().initialize())"

# Check database
sqlite3 data/crypto_aggregator.db ".tables"

# Backup database
cp data/crypto_aggregator.db data/crypto_aggregator.db.backup

# Clear old data (keep 90 days)
sqlite3 data/crypto_aggregator.db \
  "DELETE FROM prices WHERE timestamp < datetime('now', '-90 days')"

Environment Setup

# Copy environment template
cp .env.example .env

# Edit configuration
nano .env

# Essential variables:
PORT=8000
JWT_SECRET_KEY=your-secret-key
ENABLE_AUTO_DISCOVERY=true
ENABLE_SENTIMENT_ANALYSIS=true
ENABLE_BACKGROUND_TASKS=true
DATABASE_URL=sqlite:///data/crypto_aggregator.db

Testing

# Run all tests
pytest

# Run specific test file
pytest tests/test_api.py -v

# Run with coverage
pytest --cov=.

# Test specific endpoint
pytest tests/ -k "test_prices"

Deployment

# Build Docker image
docker build -t crypto-dt-source .

# Run Docker container
docker run -p 8000:8000 crypto-dt-source

# Push to HuggingFace Spaces
git remote add spaces https://huggingface.co/spaces/your-username/crypto-dt-source
git push spaces main

---

🔑 Configuration Quick Reference

Rate Limiting Tiers

# In .env or code
FREE_TIER_LIMIT = "30/minute"      # 30 req/min, 1000/day
PRO_TIER_LIMIT = "300/minute"      # 300 req/min, 50000/day
ADMIN_TIER = None                  # Unlimited

Cache TTLs

CACHE_TTL = {
    "prices": 300,          # 5 minutes
    "ohlcv": 900,           # 15 minutes
    "defi": 3600,           # 1 hour
    "trending": 600,        # 10 minutes
    "news": 1800,           # 30 minutes
}

Background Task Schedules

SCHEDULE = {
    "collect_prices": "*/5 * * * *",      # Every 5 min
    "fetch_news": "*/30 * * * *",         # Every 30 min
    "analyze_sentiment": "0 * * * *",     # Hourly
    "collect_defi": "0 */4 * * *",        # Every 4 hours
    "health_check": "*/5 * * * *",        # Every 5 min
    "backup_database": "0 2 * * *",       # Daily 2 AM
    "cleanup_logs": "0 3 * * *",          # Daily 3 AM
}

---

🐛 Debugging

View Logs

# Recent logs
tail -f logs/app.log

# Specific log level
grep "ERROR" logs/app.log

# By provider
grep "coingecko" logs/app.log

# Tail real-time
tail -f logs/*.log | grep ERROR

API Diagnostics

# Full diagnostic report
curl http://localhost:8000/api/diagnostics/run?auto_fix=false | jq

# Provider status
curl http://localhost:8000/api/providers | jq '.[] | {name, status}'

# System metrics
curl http://localhost:8000/api/metrics | jq

# WebSocket sessions
curl http://localhost:8000/api/sessions | jq

Database Inspection

# Open database
sqlite3 data/crypto_aggregator.db

# View schema
.schema

# Check table sizes
SELECT name, COUNT(*) as rows FROM sqlite_master
WHERE type='table' GROUP BY name;

# View recent prices
SELECT * FROM prices ORDER BY timestamp DESC LIMIT 10;

# View sentiment scores
SELECT * FROM news ORDER BY timestamp DESC LIMIT 5;

---

✅ Pre-Deployment Checklist

FUNCTIONALITY
[ ] All 50+ endpoints return real data (not mock)
[ ] Database storing all collected data
[ ] Sentiment analysis works with HuggingFace models
[ ] WebSocket streaming real-time updates
[ ] Background tasks running continuously

CONFIGURATION
[ ] .env file configured correctly
[ ] JWT_SECRET_KEY set to secure value
[ ] Rate limits configured for tiers
[ ] Cache TTLs optimized
[ ] Database path valid

SECURITY
[ ] Authentication required on protected endpoints
[ ] Rate limiting enforced
[ ] API key validation working
[ ] No hardcoded secrets in code
[ ] HTTPS configured (for production)

PERFORMANCE
[ ] API response time < 500ms
[ ] Sentiment analysis < 2s
[ ] WebSocket latency < 1s
[ ] Database queries < 100ms
[ ] Memory usage < 1GB
[ ] CPU usage < 50%

TESTING
[ ] All unit tests pass
[ ] Integration tests pass
[ ] Manual endpoint testing successful
[ ] Load testing acceptable (100+ req/s)
[ ] WebSocket tested with multiple clients

DEPLOYMENT
[ ] Docker builds successfully
[ ] Container runs without errors
[ ] Health check endpoint returns OK
[ ] All endpoints accessible
[ ] Logs writing correctly
[ ] Database initializing
[ ] Background tasks started

---

🆘 Common Issues & Solutions

Issue: Models not loading

Error: RuntimeError: Unable to load model
Solution:
  1. Check torch/transformers installed: pip list | grep torch
  2. Check disk space: df -h
  3. Try smaller model: distilbert instead of bert
  4. Check HF token: huggingface-cli login

Issue: Database locked

Error: sqlite3.OperationalError: database is locked
Solution:
  1. Restart server
  2. Close other connections
  3. Use WAL mode: PRAGMA journal_mode=WAL;

Issue: Rate limit too strict

Error: 429 Too Many Requests
Solution:
  1. Check tier: GET /api/sessions/stats
  2. Increase limit: FREE_TIER_LIMIT=50/minute
  3. Get API key: POST /api/auth/token

Issue: WebSocket not updating

Error: No new messages in WebSocket
Solution:
  1. Check background tasks: curl /api/health
  2. Verify WebSocket connection: ws://localhost:8000/ws
  3. Check logs: tail -f logs/app.log

Issue: Sentiment analysis fails

Error: Model not found or OutOfMemory
Solution:
  1. Check available memory: free -h
  2. Reduce batch size in sentiment pipeline
  3. Use quantized model (smaller)
  4. Disable sentiment: ENABLE_SENTIMENT_ANALYSIS=false

Issue: High memory usage

Error: Server crashes with OutOfMemory
Solution:
  1. Reduce cache size
  2. Implement memory limits
  3. Reduce WebSocket connection limit
  4. Archive old database entries
  5. Use lighter models

---

📊 Monitoring Commands

System Health

# One-liner health check
curl http://localhost:8000/api/health | jq '.components | to_entries[] | "\(.key): \(.value.status)"'

# Monitor in real-time
watch -n 5 'curl -s http://localhost:8000/api/metrics | jq "."'

# Provider status
watch -n 10 'curl -s http://localhost:8000/api/providers | jq ".[] | {name, status, health_score}"'

Performance Monitoring

# Response times
curl -w "\nTime: %{time_total}s\n" http://localhost:8000/api/prices

# Concurrent connections
curl http://localhost:8000/api/sessions/stats | jq '.total_connections'

# Database size
du -h data/crypto_aggregator.db

# Log file size
du -h logs/

Continuous Monitoring

# Dashboard-style monitoring
while true; do
    echo "=== System Status ==="
    curl -s http://localhost:8000/api/health | jq .status
    echo "=== Active Connections ==="
    curl -s http://localhost:8000/api/sessions/stats | jq .total_connections
    echo "=== Provider Status ==="
    curl -s http://localhost:8000/api/stats | jq '{online, offline, degraded}'
    sleep 5
done

---

📚 Reference Links

Code References

• API Endpoints: api_server_extended.py:1-100
• Provider Manager: provider_manager.py (global instance)
• Database Setup: database/db_manager.py:50-150
• Sentiment Analysis: ai_models.py:200-300
• WebSocket Handling: api_server_extended.py:WebSocket endpoint

Documentation

• Deployment Guide: HUGGINGFACE_DEPLOYMENT_PROMPT.md
• Implementation Timeline: IMPLEMENTATION_ROADMAP.md
• Audit Report: See audit in conversation history

External Resources

• CoinGecko API: https://docs.coingecko.com/
• Binance API: https://binance-docs.github.io/
• HuggingFace Models: https://huggingface.co/models
• FastAPI Documentation: https://fastapi.tiangolo.com/

---

🎯 Next Steps

1. Review HUGGINGFACE_DEPLOYMENT_PROMPT.md (comprehensive guide)
2. Follow IMPLEMENTATION_ROADMAP.md (step-by-step timeline)
3. Use this guide for quick lookup during implementation
4. Track progress with provided checklist
5. Test each phase before moving to next
6. Deploy to HF Spaces when all items completed

---

Last Updated: 2025-11-15 Version: 1.0 Quick Questions? Check this guide first!