hf-data-engine/README.md

🚀 HuggingFace Cryptocurrency Data Engine

A production-ready cryptocurrency data aggregator that consolidates multiple data sources into unified APIs. Designed to serve as a reliable data provider for the Dreammaker Crypto Signal & Trader application.

HuggingFace Space: Really-amin/Datasourceforcryptocurrency Local URL: http://localhost:8000

🎯 Features

Core Functionality

• ✅ Multi-Provider OHLCV Data - Binance, Kraken with automatic fallback
• ✅ Real-Time Prices - Aggregated from CoinGecko, CoinCap, Binance
• ✅ Market Sentiment - Fear & Greed Index from Alternative.me
• ✅ Market Overview - Global market statistics from CoinGecko
• ✅ Provider Health Monitoring - Real-time status of all data sources

Technical Features

• 🔄 Automatic Fallback - Seamless provider switching on failure
• ⚡ In-Memory Caching - Configurable TTL for optimal performance
• 🛡️ Circuit Breaker - Prevents repeated requests to failed services
• 📊 Rate Limiting - IP-based throttling for API protection
• 🔍 Comprehensive Logging - Detailed request and error tracking
• 📖 OpenAPI Documentation - Interactive API docs at /docs

📊 Supported Data

Cryptocurrencies (14+)

BTC, ETH, SOL, XRP, BNB, ADA, DOT, LINK, LTC, BCH, MATIC, AVAX, XLM, TRX

Timeframes

• 1m - 1 minute
• 5m - 5 minutes
• 15m - 15 minutes
• 1h - 1 hour
• 4h - 4 hours
• 1d - 1 day
• 1w - 1 week

🚀 Quick Start

Docker (Recommended)

# Build and run
docker build -t hf-crypto-engine .
docker run -p 8000:8000 hf-crypto-engine

# Access the API
curl http://localhost:8000/api/health

Local Development

# Install dependencies
pip install -r requirements.txt

# Copy environment configuration
cp .env.example .env

# Run the server
python main.py

# Or with uvicorn
uvicorn main:app --reload --host 0.0.0.0 --port 8000

Access Points

• API Root: http://localhost:8000/
• Health Check: http://localhost:8000/api/health
• Interactive Docs: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

📡 API Endpoints

1️⃣ Health Check

Get service status and provider health.

GET /api/health

Response:

{
  "status": "healthy",
  "uptime": 3600,
  "version": "1.0.0",
  "providers": [
    {
      "name": "binance",
      "status": "online",
      "latency": 120,
      "lastCheck": "2024-01-15T10:30:00Z"
    }
  ],
  "cache": {
    "size": 1250,
    "hitRate": 0.78
  }
}

2️⃣ OHLCV Data

Get candlestick (OHLCV) data with automatic provider fallback.

GET /api/ohlcv?symbol=BTCUSDT&interval=1h&limit=100

Parameters:

• symbol (required): Symbol (e.g., BTC, BTCUSDT, BTC/USDT)
• interval (optional): Timeframe - 1m, 5m, 15m, 1h, 4h, 1d, 1w (default: 1h)
• limit (optional): Number of candles 1-1000 (default: 100)

Response:

{
  "success": true,
  "data": [
    {
      "timestamp": 1699920000000,
      "open": 43250.50,
      "high": 43500.00,
      "low": 43100.25,
      "close": 43420.75,
      "volume": 125.45
    }
  ],
  "symbol": "BTCUSDT",
  "interval": "1h",
  "count": 100,
  "source": "binance",
  "timestamp": 1699920000000
}

Fallback Order: Binance → Kraken

Cache TTL: 5 minutes (configurable)

3️⃣ Real-Time Prices

Get current prices for multiple cryptocurrencies with multi-provider aggregation.

GET /api/prices?symbols=BTC,ETH,SOL

Parameters:

• symbols (optional): Comma-separated symbols (default: all supported)
• convert (optional): Currency conversion - USD, USDT (default: USDT)

Response:

{
  "success": true,
  "data": [
    {
      "symbol": "BTC",
      "name": "Bitcoin",
      "price": 43420.75,
      "priceUsd": 43420.75,
      "change1h": 0.25,
      "change24h": 2.15,
      "change7d": -1.50,
      "volume24h": 28500000000,
      "marketCap": 850000000000,
      "rank": 1,
      "lastUpdate": "2024-01-15T10:30:00Z"
    }
  ],
  "timestamp": 1699920000000,
  "source": "coingecko+coincap+binance"
}

Data Sources: CoinGecko, CoinCap, Binance (aggregated)

Cache TTL: 30 seconds (configurable)

4️⃣ Market Sentiment

Get market sentiment indicators including Fear & Greed Index.

GET /api/sentiment

Response:

{
  "success": true,
  "data": {
    "fearGreed": {
      "value": 65,
      "classification": "Greed",
      "timestamp": "2024-01-15T10:00:00Z"
    },
    "news": {
      "bullish": 0,
      "bearish": 0,
      "neutral": 0,
      "total": 0
    },
    "overall": {
      "sentiment": "bullish",
      "score": 65,
      "confidence": 0.8
    }
  },
  "timestamp": 1699920000000
}

Data Source: Alternative.me Fear & Greed Index

Cache TTL: 10 minutes (configurable)

5️⃣ Market Overview

Get global market statistics and metrics.

GET /api/market/overview

Response:

{
  "success": true,
  "data": {
    "totalMarketCap": 1650000000000,
    "totalVolume24h": 95000000000,
    "btcDominance": 51.5,
    "ethDominance": 17.2,
    "activeCoins": 12500,
    "topGainers": [],
    "topLosers": [],
    "trending": []
  },
  "timestamp": 1699920000000
}

Data Source: CoinGecko Global API

Cache TTL: 5 minutes (configurable)

6️⃣ Cache Management

Clear cached data and view statistics.

POST /api/cache/clear
GET /api/cache/stats

⚙️ Configuration

Environment Variables

Create a .env file based on .env.example:

# Server
PORT=8000
HOST=0.0.0.0
ENV=production

# Cache TTL (seconds)
CACHE_TTL_PRICES=30
CACHE_TTL_OHLCV=300
CACHE_TTL_SENTIMENT=600

# Rate Limits (requests per minute)
RATE_LIMIT_PRICES=120
RATE_LIMIT_OHLCV=60
RATE_LIMIT_SENTIMENT=30

# Optional API Keys
COINGECKO_API_KEY=your_key_here

Supported Symbols

Edit SUPPORTED_SYMBOLS in .env:

SUPPORTED_SYMBOLS=BTC,ETH,SOL,XRP,BNB,ADA,DOT,LINK,LTC,BCH,MATIC,AVAX,XLM,TRX

🐳 HuggingFace Spaces Deployment

1. Create README.md for HF Space

---
title: Crypto Data Engine
emoji: 📊
colorFrom: blue
colorTo: green
sdk: docker
app_port: 8000
---

2. Deploy Files

Upload these files to your HuggingFace Space:

• Dockerfile
• requirements.txt
• main.py
• All core/ and providers/ directories
• .env.example (rename to .env if setting variables)

3. Configure Secrets (Optional)

In Space settings, add:

• COINGECKO_API_KEY - For higher rate limits
• Other API keys as needed

4. Access Your Space

https://huggingface.co/spaces/YOUR_USERNAME/YOUR_SPACE_NAME

📊 Performance

Response Times

Endpoint	Target	Maximum	Cache
/api/prices	<1s	3s	30s
/api/ohlcv (50 bars)	<2s	5s	5min
/api/ohlcv (200 bars)	<5s	15s	5min
/api/sentiment	<3s	10s	10min
/api/health	<100ms	500ms	None

Rate Limits

Endpoint	Limit
/api/prices	120 req/min
/api/ohlcv	60 req/min
/api/sentiment	30 req/min
/api/health	Unlimited

🔧 Integration with Dreammaker

Backend Configuration (.env)

HF_ENGINE_BASE_URL=http://localhost:8000
# or
HF_ENGINE_BASE_URL=https://really-amin-datasourceforcryptocurrency.hf.space

HF_ENGINE_ENABLED=true
HF_ENGINE_TIMEOUT=30000
PRIMARY_DATA_SOURCE=huggingface

TypeScript Client Example

import axios from 'axios';

const hfClient = axios.create({
  baseURL: process.env.HF_ENGINE_BASE_URL,
  timeout: 30000,
});

// Fetch OHLCV
const ohlcv = await hfClient.get('/api/ohlcv', {
  params: { symbol: 'BTCUSDT', interval: '1h', limit: 200 }
});

// Fetch Prices
const prices = await hfClient.get('/api/prices', {
  params: { symbols: 'BTC,ETH,SOL' }
});

// Fetch Sentiment
const sentiment = await hfClient.get('/api/sentiment');

🛡️ Error Handling

Error Response Format

{
  "success": false,
  "error": {
    "code": "PROVIDER_ERROR",
    "message": "All data providers are currently unavailable",
    "details": {
      "binance": "HTTP 403",
      "kraken": "Timeout"
    },
    "retryAfter": 60
  },
  "timestamp": 1699920000000
}

Error Codes

• INVALID_SYMBOL - Unknown cryptocurrency symbol
• INVALID_INTERVAL - Unsupported timeframe
• PROVIDER_ERROR - All providers failed
• RATE_LIMITED - Too many requests
• INTERNAL_ERROR - Server error

📈 Monitoring

Logs

All requests and errors are logged:

2024-01-15 10:30:00 - INFO - Trying binance for OHLCV data: BTCUSDT 1h
2024-01-15 10:30:00 - INFO - Successfully fetched 100 candles from binance

Health Monitoring

Monitor provider status via /api/health:

• online - Provider working normally
• degraded - Recent errors but still functional
• offline - Circuit breaker open, provider unavailable

🧪 Testing

Manual Testing

# Health check
curl http://localhost:8000/api/health

# OHLCV data
curl "http://localhost:8000/api/ohlcv?symbol=BTC&interval=1h&limit=10"

# Prices
curl "http://localhost:8000/api/prices?symbols=BTC,ETH"

# Sentiment
curl http://localhost:8000/api/sentiment

# Market overview
curl http://localhost:8000/api/market/overview

Load Testing

# Using Apache Bench
ab -n 1000 -c 10 http://localhost:8000/api/prices?symbols=BTC

# Using k6
k6 run loadtest.js

📝 Architecture

┌─────────────────────────────────────────┐
│         FastAPI Application             │
│  ┌────────────────────────────────────┐ │
│  │  Rate Limiter (SlowAPI)            │ │
│  └────────────────────────────────────┘ │
│  ┌────────────────────────────────────┐ │
│  │  Cache Layer (In-Memory)           │ │
│  └────────────────────────────────────┘ │
│  ┌────────────────────────────────────┐ │
│  │  Data Aggregator                   │ │
│  │  ┌──────────┬──────────┬─────────┐ │ │
│  │  │ Binance  │ Kraken   │CoinGecko│ │ │
│  │  └──────────┴──────────┴─────────┘ │ │
│  │  ┌──────────────────────────────┐  │ │
│  │  │  Circuit Breaker             │  │ │
│  │  └──────────────────────────────┘  │ │
│  └────────────────────────────────────┘ │
└─────────────────────────────────────────┘

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

📄 License

MIT License - See LICENSE file for details

🙏 Acknowledgments

• Binance - Primary OHLCV data source
• CoinGecko - Price and market data
• Alternative.me - Fear & Greed Index
• CoinCap - Real-time price data
• Kraken - Backup OHLCV provider

---

Made with ❤️ for the Crypto Community

Version: 1.0.0 Last Updated: 2024-01-15