archive/docs/QUICK_REFERENCE_GUIDE.md

# Quick Reference Guide: Crypto-DT-Source Implementation

**Quick lookup for common tasks during implementation**

---

## 🚀 Start Here

### Launch Development Server
```bash
cd /home/user/crypto-dt-source
python api_server_extended.py
# Opens on http://localhost:8000
```

### Access Documentation
```
Swagger UI:  http://localhost:8000/docs
ReDoc:       http://localhost:8000/redoc
```

### Test Endpoints
```bash
# Health check
curl http://localhost:8000/api/health | jq

# Real-time prices
curl 'http://localhost:8000/api/prices?symbols=BTC,ETH' | jq

# Trending coins
curl http://localhost:8000/api/trending | jq

# Sentiment analysis
curl -X POST http://localhost:8000/api/sentiment/analyze \
  -H "Content-Type: application/json" \
  -d '{"text":"Bitcoin price is going up"}'
```

---

## 📁 Key Files to Modify

### API Endpoints
- **`api_server_extended.py`** - Main FastAPI application + all endpoints
- **`api/endpoints.py`** - Additional endpoint definitions
- **`api/websocket.py`** - WebSocket handling

### Data Collection
- **`collectors/market_data_extended.py`** - Price data fetching
- **`collectors/sentiment_extended.py`** - Sentiment analysis
- **`collectors/news.py`** - News aggregation

### Database
- **`database/db_manager.py`** - Database connection
- **`database/models.py`** - Table definitions
- **`database/migrations.py`** - Schema setup

### AI Models
- **`ai_models.py`** - HuggingFace model loading

### Utilities
- **`utils/auth.py`** - Authentication
- **`utils/rate_limiter_enhanced.py`** - Rate limiting
- **`log_manager.py`** - Logging

---

## 💻 Common Commands

### Database Operations
```bash
# Initialize database
python -c "from database.db_manager import DBManager; \
  import asyncio; \
  asyncio.run(DBManager().initialize())"

# Check database
sqlite3 data/crypto_aggregator.db ".tables"

# Backup database
cp data/crypto_aggregator.db data/crypto_aggregator.db.backup

# Clear old data (keep 90 days)
sqlite3 data/crypto_aggregator.db \
  "DELETE FROM prices WHERE timestamp < datetime('now', '-90 days')"
```

### Environment Setup
```bash
# Copy environment template
cp .env.example .env

# Edit configuration
nano .env

# Essential variables:
PORT=8000
JWT_SECRET_KEY=your-secret-key
ENABLE_AUTO_DISCOVERY=true
ENABLE_SENTIMENT_ANALYSIS=true
ENABLE_BACKGROUND_TASKS=true
DATABASE_URL=sqlite:///data/crypto_aggregator.db
```

### Testing
```bash
# Run all tests
pytest

# Run specific test file
pytest tests/test_api.py -v

# Run with coverage
pytest --cov=.

# Test specific endpoint
pytest tests/ -k "test_prices"
```

### Deployment
```bash
# Build Docker image
docker build -t crypto-dt-source .

# Run Docker container
docker run -p 8000:8000 crypto-dt-source

# Push to HuggingFace Spaces
git remote add spaces https://huggingface.co/spaces/your-username/crypto-dt-source
git push spaces main
```

---

## 🔑 Configuration Quick Reference

### Rate Limiting Tiers
```python
# In .env or code
FREE_TIER_LIMIT = "30/minute"      # 30 req/min, 1000/day
PRO_TIER_LIMIT = "300/minute"      # 300 req/min, 50000/day
ADMIN_TIER = None                  # Unlimited
```

### Cache TTLs
```python
CACHE_TTL = {
    "prices": 300,          # 5 minutes
    "ohlcv": 900,           # 15 minutes
    "defi": 3600,           # 1 hour
    "trending": 600,        # 10 minutes
    "news": 1800,           # 30 minutes
}
```

### Background Task Schedules
```python
SCHEDULE = {
    "collect_prices": "*/5 * * * *",      # Every 5 min
    "fetch_news": "*/30 * * * *",         # Every 30 min
    "analyze_sentiment": "0 * * * *",     # Hourly
    "collect_defi": "0 */4 * * *",        # Every 4 hours
    "health_check": "*/5 * * * *",        # Every 5 min
    "backup_database": "0 2 * * *",       # Daily 2 AM
    "cleanup_logs": "0 3 * * *",          # Daily 3 AM
}
```

---

## 🐛 Debugging

### View Logs
```bash
# Recent logs
tail -f logs/app.log

# Specific log level
grep "ERROR" logs/app.log

# By provider
grep "coingecko" logs/app.log

# Tail real-time
tail -f logs/*.log | grep ERROR
```

### API Diagnostics
```bash
# Full diagnostic report
curl http://localhost:8000/api/diagnostics/run?auto_fix=false | jq

# Provider status
curl http://localhost:8000/api/providers | jq '.[] | {name, status}'

# System metrics
curl http://localhost:8000/api/metrics | jq

# WebSocket sessions
curl http://localhost:8000/api/sessions | jq
```

### Database Inspection
```bash
# Open database
sqlite3 data/crypto_aggregator.db

# View schema
.schema

# Check table sizes
SELECT name, COUNT(*) as rows FROM sqlite_master
WHERE type='table' GROUP BY name;

# View recent prices
SELECT * FROM prices ORDER BY timestamp DESC LIMIT 10;

# View sentiment scores
SELECT * FROM news ORDER BY timestamp DESC LIMIT 5;
```

---

## ✅ Pre-Deployment Checklist

```
FUNCTIONALITY
[ ] All 50+ endpoints return real data (not mock)
[ ] Database storing all collected data
[ ] Sentiment analysis works with HuggingFace models
[ ] WebSocket streaming real-time updates
[ ] Background tasks running continuously

CONFIGURATION
[ ] .env file configured correctly
[ ] JWT_SECRET_KEY set to secure value
[ ] Rate limits configured for tiers
[ ] Cache TTLs optimized
[ ] Database path valid

SECURITY
[ ] Authentication required on protected endpoints
[ ] Rate limiting enforced
[ ] API key validation working
[ ] No hardcoded secrets in code
[ ] HTTPS configured (for production)

PERFORMANCE
[ ] API response time < 500ms
[ ] Sentiment analysis < 2s
[ ] WebSocket latency < 1s
[ ] Database queries < 100ms
[ ] Memory usage < 1GB
[ ] CPU usage < 50%

TESTING
[ ] All unit tests pass
[ ] Integration tests pass
[ ] Manual endpoint testing successful
[ ] Load testing acceptable (100+ req/s)
[ ] WebSocket tested with multiple clients

DEPLOYMENT
[ ] Docker builds successfully
[ ] Container runs without errors
[ ] Health check endpoint returns OK
[ ] All endpoints accessible
[ ] Logs writing correctly
[ ] Database initializing
[ ] Background tasks started
```

---

## 🆘 Common Issues & Solutions

### Issue: Models not loading
```
Error: RuntimeError: Unable to load model
Solution:
  1. Check torch/transformers installed: pip list | grep torch
  2. Check disk space: df -h
  3. Try smaller model: distilbert instead of bert
  4. Check HF token: huggingface-cli login
```

### Issue: Database locked
```
Error: sqlite3.OperationalError: database is locked
Solution:
  1. Restart server
  2. Close other connections
  3. Use WAL mode: PRAGMA journal_mode=WAL;
```

### Issue: Rate limit too strict
```
Error: 429 Too Many Requests
Solution:
  1. Check tier: GET /api/sessions/stats
  2. Increase limit: FREE_TIER_LIMIT=50/minute
  3. Get API key: POST /api/auth/token
```

### Issue: WebSocket not updating
```
Error: No new messages in WebSocket
Solution:
  1. Check background tasks: curl /api/health
  2. Verify WebSocket connection: ws://localhost:8000/ws
  3. Check logs: tail -f logs/app.log
```

### Issue: Sentiment analysis fails
```
Error: Model not found or OutOfMemory
Solution:
  1. Check available memory: free -h
  2. Reduce batch size in sentiment pipeline
  3. Use quantized model (smaller)
  4. Disable sentiment: ENABLE_SENTIMENT_ANALYSIS=false
```

### Issue: High memory usage
```
Error: Server crashes with OutOfMemory
Solution:
  1. Reduce cache size
  2. Implement memory limits
  3. Reduce WebSocket connection limit
  4. Archive old database entries
  5. Use lighter models
```

---

## 📊 Monitoring Commands

### System Health
```bash
# One-liner health check
curl http://localhost:8000/api/health | jq '.components | to_entries[] | "\(.key): \(.value.status)"'

# Monitor in real-time
watch -n 5 'curl -s http://localhost:8000/api/metrics | jq "."'

# Provider status
watch -n 10 'curl -s http://localhost:8000/api/providers | jq ".[] | {name, status, health_score}"'
```

### Performance Monitoring
```bash
# Response times
curl -w "\nTime: %{time_total}s\n" http://localhost:8000/api/prices

# Concurrent connections
curl http://localhost:8000/api/sessions/stats | jq '.total_connections'

# Database size
du -h data/crypto_aggregator.db

# Log file size
du -h logs/
```

### Continuous Monitoring
```bash
# Dashboard-style monitoring
while true; do
    echo "=== System Status ==="
    curl -s http://localhost:8000/api/health | jq .status
    echo "=== Active Connections ==="
    curl -s http://localhost:8000/api/sessions/stats | jq .total_connections
    echo "=== Provider Status ==="
    curl -s http://localhost:8000/api/stats | jq '{online, offline, degraded}'
    sleep 5
done
```

---

## 📚 Reference Links

### Code References
- **API Endpoints:** `api_server_extended.py:1-100`
- **Provider Manager:** `provider_manager.py` (global instance)
- **Database Setup:** `database/db_manager.py:50-150`
- **Sentiment Analysis:** `ai_models.py:200-300`
- **WebSocket Handling:** `api_server_extended.py:WebSocket endpoint`

### Documentation
- Deployment Guide: `HUGGINGFACE_DEPLOYMENT_PROMPT.md`
- Implementation Timeline: `IMPLEMENTATION_ROADMAP.md`
- Audit Report: See audit in conversation history

### External Resources
- CoinGecko API: https://docs.coingecko.com/
- Binance API: https://binance-docs.github.io/
- HuggingFace Models: https://huggingface.co/models
- FastAPI Documentation: https://fastapi.tiangolo.com/

---

## 🎯 Next Steps

1. **Review** `HUGGINGFACE_DEPLOYMENT_PROMPT.md` (comprehensive guide)
2. **Follow** `IMPLEMENTATION_ROADMAP.md` (step-by-step timeline)
3. **Use** this guide for quick lookup during implementation
4. **Track** progress with provided checklist
5. **Test** each phase before moving to next
6. **Deploy** to HF Spaces when all items completed

---

**Last Updated:** 2025-11-15
**Version:** 1.0
**Quick Questions?** Check this guide first!