collectors/QUICK_START.md

Collectors Quick Start Guide

Files Created

/home/user/crypto-dt-source/collectors/
├── __init__.py              # Package exports
├── market_data.py           # Market data collectors (16 KB)
├── explorers.py             # Blockchain explorer collectors (17 KB)
├── news.py                  # News aggregation collectors (13 KB)
├── sentiment.py             # Sentiment data collectors (7.8 KB)
├── onchain.py               # On-chain analytics (placeholder, 13 KB)
├── demo_collectors.py       # Comprehensive demo script (6.6 KB)
├── README.md                # Full documentation
└── QUICK_START.md          # This file

Quick Test

Test All Collectors

cd /home/user/crypto-dt-source
python collectors/demo_collectors.py

Test Individual Modules

# Market Data (CoinGecko, CoinMarketCap, Binance)
python -m collectors.market_data

# Blockchain Explorers (Etherscan, BscScan, TronScan)
python -m collectors.explorers

# News (CryptoPanic, NewsAPI)
python -m collectors.news

# Sentiment (Alternative.me Fear & Greed)
python -m collectors.sentiment

# On-chain Analytics (Placeholder)
python -m collectors.onchain

Import and Use

Collect All Market Data

import asyncio
from collectors import collect_market_data

results = asyncio.run(collect_market_data())

for result in results:
    print(f"{result['provider']}: {result['success']}")

Collect All Data from All Categories

import asyncio
from collectors import (
    collect_market_data,
    collect_explorer_data,
    collect_news_data,
    collect_sentiment_data,
    collect_onchain_data
)

async def main():
    # Run all collectors concurrently
    results = await asyncio.gather(
        collect_market_data(),
        collect_explorer_data(),
        collect_news_data(),
        collect_sentiment_data(),
        collect_onchain_data()
    )

    market, explorers, news, sentiment, onchain = results

    print(f"Market data: {len(market)} sources")
    print(f"Explorers: {len(explorers)} sources")
    print(f"News: {len(news)} sources")
    print(f"Sentiment: {len(sentiment)} sources")
    print(f"On-chain: {len(onchain)} sources (placeholder)")

asyncio.run(main())

Individual Collector Example

import asyncio
from collectors.market_data import get_coingecko_simple_price

async def get_prices():
    result = await get_coingecko_simple_price()

    if result['success']:
        data = result['data']
        print(f"BTC: ${data['bitcoin']['usd']:,.2f}")
        print(f"ETH: ${data['ethereum']['usd']:,.2f}")
        print(f"BNB: ${data['binancecoin']['usd']:,.2f}")
        print(f"Data age: {result['staleness_minutes']:.2f} minutes")
    else:
        print(f"Error: {result['error']}")

asyncio.run(get_prices())

Collectors Summary

1. Market Data (market_data.py)

Function	Provider	API Key Required	Description
get_coingecko_simple_price()	CoinGecko	No	BTC, ETH, BNB prices with market data
get_coinmarketcap_quotes()	CoinMarketCap	Yes	Professional market data
get_binance_ticker()	Binance	No	Real-time 24hr ticker
collect_market_data()	All above	-	Collects from all sources

2. Blockchain Explorers (explorers.py)

Function	Provider	API Key Required	Description
get_etherscan_gas_price()	Etherscan	Yes	Current Ethereum gas prices
get_bscscan_bnb_price()	BscScan	Yes	BNB price and BSC stats
get_tronscan_stats()	TronScan	Optional	TRON network statistics
collect_explorer_data()	All above	-	Collects from all sources

3. News Aggregation (news.py)

Function	Provider	API Key Required	Description
get_cryptopanic_posts()	CryptoPanic	No	Latest crypto news posts
get_newsapi_headlines()	NewsAPI	Yes	Crypto-related headlines
collect_news_data()	All above	-	Collects from all sources

4. Sentiment Analysis (sentiment.py)

Function	Provider	API Key Required	Description
get_fear_greed_index()	Alternative.me	No	Market Fear & Greed Index
collect_sentiment_data()	All above	-	Collects from all sources

5. On-Chain Analytics (onchain.py)

Function	Provider	Status	Description
get_the_graph_data()	The Graph	Placeholder	GraphQL blockchain data
get_blockchair_data()	Blockchair	Placeholder	Blockchain statistics
get_glassnode_metrics()	Glassnode	Placeholder	Advanced on-chain metrics
collect_onchain_data()	All above	-	Collects from all sources

API Keys Setup

Create a .env file or set environment variables:

# Market Data
export COINMARKETCAP_KEY_1="your_key_here"

# Blockchain Explorers
export ETHERSCAN_KEY_1="your_key_here"
export BSCSCAN_KEY="your_key_here"
export TRONSCAN_KEY="your_key_here"

# News
export NEWSAPI_KEY="your_key_here"

Output Format

All collectors return standardized format:

{
    "provider": "CoinGecko",                    # Provider name
    "category": "market_data",                  # Category
    "data": {...},                              # Raw API response
    "timestamp": "2025-11-11T00:20:00Z",       # Collection time
    "data_timestamp": "2025-11-11T00:19:30Z",  # Data timestamp
    "staleness_minutes": 0.5,                   # Data age
    "success": True,                            # Success flag
    "error": None,                              # Error message
    "error_type": None,                         # Error type
    "response_time_ms": 342.5                   # Response time
}

Key Features

✓ Async/Concurrent - All collectors run asynchronously ✓ Error Handling - Comprehensive error handling and logging ✓ Staleness Tracking - Calculates data age in minutes ✓ Rate Limiting - Respects API rate limits ✓ Retry Logic - Automatic retries with exponential backoff ✓ Structured Logging - JSON-formatted logs ✓ API Key Management - Secure key handling from environment ✓ Standardized Output - Consistent response format ✓ Production Ready - Ready for production deployment

Common Issues

1. Missing API Keys

Error: API key required but not configured for CoinMarketCap

Solution: Set the required environment variable:

export COINMARKETCAP_KEY_1="your_api_key"

2. Rate Limit Exceeded

Error Type: rate_limit

Solution: Collectors automatically retry with backoff. Check rate limits in provider documentation.

3. Network Timeout

Error Type: timeout

Solution: Collectors automatically increase timeout and retry. Check network connectivity.

Next Steps

1. Run the demo: python collectors/demo_collectors.py
2. Configure API keys for providers requiring authentication
3. Integrate collectors into your monitoring system
4. Implement on-chain collectors (currently placeholders)
5. Add custom collectors following the existing patterns

Support

• Full documentation: collectors/README.md
• Demo script: collectors/demo_collectors.py
• Configuration: config.py
• API Client: utils/api_client.py
• Logger: utils/logger.py

---

Total Collectors: 14 functions across 5 modules Total Code: ~75 KB of production-ready Python code Status: Ready for production use (except on-chain placeholders)