hf-data-engine/docs/components/GRADIO_DASHBOARD_README.md

🚀 Gradio Dashboard for Crypto Data Sources

Overview

Comprehensive Gradio-based monitoring dashboard that provides real-time health checking, force testing, and auto-healing capabilities for all crypto data sources in the project.

✨ Features

1. System Overview Dashboard

• Real-time status of FastAPI backend
• HF Data Engine health monitoring
• Loaded resources statistics
• System uptime tracking

2. Force Testing

• Test ALL 200+ data sources with retries
• Detailed latency measurements
• Success/failure tracking
• Automatic retry on failures

3. Resource Explorer

• Browse all API resources
• Detailed configuration view
• Force test results per resource
• JSON configuration display

4. FastAPI Endpoint Monitor

• Test all backend endpoints
• Response time tracking
• Status code monitoring
• Automatic health checks

5. HF Data Engine Monitor

• Test OHLCV endpoints
• Price feed monitoring
• Sentiment analysis checks
• Cache statistics

6. Custom API Testing

• Test any URL with custom headers
• Configurable retry attempts
• Multiple HTTP methods (GET, POST, PUT, DELETE)
• Detailed response analysis

7. Analytics Dashboard

• Resource statistics by category
• Source file breakdowns
• Performance metrics
• Success rate tracking

8. Auto-Healing

• Automatic retry with different strategies
• Header modification attempts
• Timeout adjustments
• Redirect following

🚀 Quick Start

Option 1: Using Startup Script

# Make script executable (first time only)
chmod +x start_gradio_dashboard.sh

# Start dashboard
./start_gradio_dashboard.sh

Option 2: Manual Start

# Install requirements
pip install -r requirements_gradio.txt

# Start dashboard
python gradio_ultimate_dashboard.py

Option 3: Direct Python

python3 gradio_ultimate_dashboard.py

🌐 Access

Once started, the dashboard is available at:

URL: http://localhost:7861

You can also access it from other devices on your network using your machine's IP address:

Network URL: http://YOUR_IP:7861

📊 Dashboard Tabs

🏠 Dashboard

• System overview
• Core systems status (FastAPI, HF Engine)
• Resource statistics
• Quick health summary

🧪 Force Test

• Comprehensive testing of ALL sources
• Multiple retry attempts per source
• Detailed success/failure tracking
• Performance metrics

How to use:

1. Click "⚡ START FORCE TEST" button
2. Wait for completion (may take 2-5 minutes for all sources)
3. Review results table
4. Check individual resource details

🔍 Resource Explorer

• Search and explore all API resources
• View complete configuration
• See force test results
• Analyze individual sources

How to use:

1. Select resource from dropdown
2. View detailed configuration
3. Check test results
4. Copy configuration if needed

⚡ FastAPI Status

• Monitor main backend server
• Test all API endpoints
• Check response times
• Verify functionality

Tested Endpoints:

• /health - Health check
• /api/status - System status
• /api/providers - Provider list
• /api/pools - Pool management
• /api/hf/health - HuggingFace health
• /api/feature-flags - Feature flags
• /api/data/market - Market data
• /api/data/news - News data

🤗 HF Data Engine

• Monitor HuggingFace Data Engine
• Test all data endpoints
• Check provider status
• Verify cache performance

Tested Endpoints:

• /api/health - Engine health
• /api/prices - Price data
• /api/ohlcv - Candlestick data
• /api/sentiment - Market sentiment
• /api/market/overview - Market overview
• /api/cache/stats - Cache statistics

🎯 Custom Test

• Test any API endpoint
• Custom headers support
• Configurable retries
• All HTTP methods

Features:

• URL input
• Method selection (GET, POST, PUT, DELETE)
• Custom headers (JSON format)
• Retry attempts (1-5)
• Detailed response display

📊 Analytics

• Comprehensive resource statistics
• Category breakdowns
• Source file analysis
• Performance metrics

🔧 Configuration

Enable Auto-Heal

Toggle the "🔧 Enable Auto-Heal" checkbox at the top of the dashboard to enable automatic retry with different strategies when a source fails.

Auto-Heal Strategies:

1. Add custom headers (User-Agent, etc.)
2. Increase timeout duration
3. Follow redirects automatically

Enable Real-Time Monitoring

Toggle "📡 Enable Real-Time Monitoring" to activate continuous background monitoring (coming in future update).

📁 Files

Main Dashboard Files

• gradio_ultimate_dashboard.py - Advanced dashboard with all features
• gradio_dashboard.py - Basic dashboard (simpler version)

Configuration

• requirements_gradio.txt - Python dependencies
• start_gradio_dashboard.sh - Startup script

Data Sources

• api-resources/crypto_resources_unified_2025-11-11.json - Unified resources (200+ sources)
• api-resources/ultimate_crypto_pipeline_2025_NZasinich.json - Pipeline resources (162 sources)
• all_apis_merged_2025.json - Merged APIs
• providers_config_extended.json - Extended provider configs
• providers_config_ultimate.json - Ultimate provider configs

🧪 Testing Workflow

Complete System Test

1. Start All Services:

   # Terminal 1: Main FastAPI backend
   python app.py
   
   # Terminal 2: HF Data Engine
   cd hf-data-engine && python main.py
   
   # Terminal 3: Gradio Dashboard
   ./start_gradio_dashboard.sh
   

2. Verify Systems:

   • Open dashboard: http://localhost:7861
   • Check Dashboard tab for system status
   • Verify both FastAPI and HF Engine show "✅ ONLINE"

3. Run Force Test:

   • Go to "🧪 Force Test" tab
   • Click "⚡ START FORCE TEST"
   • Wait for completion
   • Review results

4. Test Individual Endpoints:

   • Go to "⚡ FastAPI Status" tab
   • Click "🧪 Test All Endpoints"
   • Check all endpoints are working

5. Test HF Engine:

   • Go to "🤗 HF Data Engine" tab
   • Click "🧪 Test All Endpoints"
   • Verify data is returned

6. Explore Resources:

   • Go to "🔍 Resource Explorer" tab
   • Browse different data sources
   • View configurations

7. Check Analytics:

   • Go to "📊 Analytics" tab
   • Review statistics
   • Check resource distribution

🚨 Troubleshooting

Dashboard won't start

Problem: Import errors

Solution:

pip install -r requirements_gradio.txt

Can't connect to FastAPI/HF Engine

Problem: Services not running

Solution:

# Check if services are running
curl http://localhost:7860/health
curl http://localhost:8000/api/health

# Start if needed
python app.py  # FastAPI
cd hf-data-engine && python main.py  # HF Engine

Force test shows all offline

Problem: Network/firewall issues or services not started

Solution:

1. Verify services are running (see above)
2. Check if you're behind a restrictive firewall
3. Try testing individual endpoints first
4. Enable auto-heal for retry attempts

Slow performance

Problem: Testing too many sources

Solution:

• Test only specific categories instead of all
• Increase timeout values
• Test during off-peak hours
• Use caching for repeated tests

💡 Tips & Best Practices

1. Test Incrementally

Don't run force test on all sources at once during development. Start with:

• FastAPI endpoints only
• HF Engine endpoints only
• Small subset of resources

2. Use Auto-Heal Wisely

Enable auto-heal when testing external APIs that might have temporary issues. Disable for internal services.

3. Monitor Regularly

Schedule regular health checks:

• Every hour: FastAPI and HF Engine
• Every 6 hours: All external sources
• Daily: Full force test

4. Export Results

After force testing, export results for:

• Historical tracking
• Performance analysis
• Downtime investigation

5. Custom Testing

Use the custom test tab to:

• Debug specific endpoints
• Test new APIs before adding to system
• Verify authentication
• Test with different headers

📊 Metrics & KPIs

The dashboard tracks:

• Uptime: Percentage of time services are available
• Response Time: Average latency for requests
• Success Rate: Percentage of successful requests
• Error Rate: Percentage of failed requests
• Resource Coverage: Number of working vs total resources

🔄 Integration

With Existing Systems

The dashboard integrates with:

1. FastAPI Backend (app.py)

   • Monitors all endpoints
   • Tests provider health
   • Checks feature flags

2. HF Data Engine (hf-data-engine/)

   • Tests all data endpoints
   • Monitors provider status
   • Checks cache performance

3. API Resources (api-resources/)

   • Loads all resource configurations
   • Tests each resource
   • Tracks availability

API Endpoints Used

The dashboard calls these endpoints:

FastAPI:

• GET /health
• GET /api/status
• GET /api/providers
• GET /api/hf/health

HF Engine:

• GET /api/health
• GET /api/prices
• GET /api/ohlcv
• GET /api/sentiment

📈 Future Enhancements

Planned features:

• Real-time monitoring with auto-refresh
• Alert system for downtimes
• Historical data tracking
• Performance graphs and charts
• Email notifications
• Slack/Discord integration
• Automated daily reports
• Resource availability heatmap
• Comparative analytics
• Export to multiple formats (PDF, Excel)

🤝 Contributing

To add new features:

1. Fork the dashboard code
2. Add new tab or functionality
3. Test thoroughly
4. Submit pull request

📝 License

Same as main project

🙏 Acknowledgments

Built using:

• Gradio - UI framework
• httpx - HTTP client
• pandas - Data analysis
• FastAPI - Backend server

---

Version: 2.0 Last Updated: 2024-11-14 Status: ✅ Production Ready