hf-data-engine/docs/components/README_BACKEND.md

metadata

title: Crypto API Monitor Backend
emoji: 📊
colorFrom: blue
colorTo: purple
sdk: docker
app_port: 7860

Crypto API Monitor Backend

Real-time cryptocurrency API monitoring backend service built with FastAPI.

Features

• Real-time Health Monitoring: Automatically monitors 11+ cryptocurrency API providers every 5 minutes
• WebSocket Support: Live updates for frontend dashboard integration
• REST API: Comprehensive endpoints for status, logs, categories, and analytics
• SQLite Database: Persistent storage for connection logs, metrics, and configuration
• Rate Limit Tracking: Monitor API usage and rate limits per provider
• Connection Logging: Track all API requests with response times and error details
• Authentication: Token-based authentication and IP whitelist support

API Providers Monitored

Market Data

• CoinGecko (free)
• CoinMarketCap (requires API key)
• CryptoCompare (requires API key)
• Binance (free)

Blockchain Explorers

• Etherscan (requires API key)
• BscScan (requires API key)
• TronScan (requires API key)

News & Sentiment

• CryptoPanic (free)
• NewsAPI (requires API key)
• Alternative.me Fear & Greed (free)

On-chain Analytics

• The Graph (free)
• Blockchair (free)

API Documentation

Visit /docs for interactive API documentation (Swagger UI). Visit /redoc for alternative API documentation (ReDoc).

Main Endpoints

Status & Monitoring

• GET /api/status - Overall system status
• GET /api/categories - Category statistics
• GET /api/providers - List all providers with filters
• GET /api/logs - Connection logs with pagination
• GET /api/failures - Failure analysis
• GET /api/rate-limits - Rate limit status

Configuration

• GET /api/config/keys - API key configuration
• GET /api/schedule - Schedule configuration
• POST /api/schedule/trigger - Manually trigger scheduled task

Analytics

• GET /api/charts/health-history - Health history for charts
• GET /api/charts/compliance - Compliance chart data
• GET /api/freshness - Data freshness status

WebSocket

• WS /ws/live - Real-time updates

Environment Variables

Create a .env file or set environment variables:

# Optional: API authentication tokens (comma-separated)
API_TOKENS=token1,token2

# Optional: IP whitelist (comma-separated)
ALLOWED_IPS=192.168.1.1,10.0.0.1

# Optional: Database URL (default: sqlite:///./crypto_monitor.db)
DATABASE_URL=sqlite:///./crypto_monitor.db

# Optional: Server port (default: 7860)
PORT=7860

Deployment to Hugging Face Spaces

Option 1: Docker SDK

1. Create a new Hugging Face Space
2. Select Docker SDK
3. Push this repository to GitHub
4. Connect the GitHub repository to your Space
5. Add environment variables in Space settings:
   • API_TOKENS=your_secret_token_here
   • ALLOWED_IPS= (optional, leave empty for no restriction)
6. The Space will automatically build and deploy

Option 2: Local Docker

# Build Docker image
docker build -t crypto-api-monitor .

# Run container
docker run -p 7860:7860 \
  -e API_TOKENS=your_token_here \
  crypto-api-monitor

Local Development

# Install dependencies
pip install -r requirements.txt

# Run the application
python app.py

# Or with uvicorn
uvicorn app:app --host 0.0.0.0 --port 7860 --reload

Visit http://localhost:7860 to access the API. Visit http://localhost:7860/docs for interactive documentation.

Database Schema

The application uses SQLite with the following tables:

• providers: API provider configurations
• connection_attempts: Log of all API connection attempts
• data_collections: Data collection records
• rate_limit_usage: Rate limit tracking
• schedule_config: Scheduled task configuration

WebSocket Protocol

Connect to ws://localhost:7860/ws/live for real-time updates.

Message Types

Status Update

{
  "type": "status_update",
  "data": {
    "total_apis": 11,
    "online": 10,
    "degraded": 1,
    "offline": 0
  }
}

New Log Entry

{
  "type": "new_log_entry",
  "data": {
    "timestamp": "2025-11-11T00:00:00",
    "provider": "CoinGecko",
    "status": "success",
    "response_time_ms": 120
  }
}

Rate Limit Alert

{
  "type": "rate_limit_alert",
  "data": {
    "provider": "CoinMarketCap",
    "usage_percentage": 85
  }
}

Frontend Integration

Update your frontend dashboard configuration:

// config.js
const config = {
  apiBaseUrl: 'https://YOUR_USERNAME-crypto-api-monitor.hf.space',
  wsUrl: 'wss://YOUR_USERNAME-crypto-api-monitor.hf.space/ws/live',
  authToken: 'your_token_here' // Optional
};

Architecture

app.py                          # FastAPI application entry point
config.py                       # Configuration & API registry loader
database/
  ├── db.py                     # Database initialization
  └── models.py                 # SQLAlchemy models
monitoring/
  └── health_monitor.py         # Background health monitoring
api/
  ├── endpoints.py              # REST API endpoints
  ├── websocket.py              # WebSocket handler
  └── auth.py                   # Authentication
utils/
  ├── http_client.py            # Async HTTP client with retry
  ├── logger.py                 # Structured logging
  └── validators.py             # Input validation

API Keys

API keys are loaded from all_apis_merged_2025.json in the discovered_keys section:

{
  "discovered_keys": {
    "etherscan": ["key1", "key2"],
    "bscscan": ["key1"],
    "coinmarketcap": ["key1", "key2"],
    ...
  }
}

Performance

• Health checks run every 5 minutes
• Response time tracking for all providers
• Automatic retry with exponential backoff
• Connection timeout: 10 seconds
• Database queries optimized with indexes

Security

• Optional token-based authentication
• IP whitelist support
• API keys masked in logs and responses
• CORS enabled for frontend access
• SQL injection protection via SQLAlchemy ORM

License

MIT License

Author

Nima Zasinich

• GitHub: @nimazasinich
• Project: Crypto API Monitor Backend

---

Built for the crypto dev community