APL_USAGE_GUIDE.md

# Auto Provider Loader (APL) - Usage Guide

**Version:** 1.0  
**Last Updated:** 2025-11-16  
**Status:** PRODUCTION READY ✅

---

## Overview

The Auto Provider Loader (APL) is a **real-data-only** system that automatically discovers, validates, and integrates cryptocurrency data providers (both HTTP APIs and Hugging Face models) into your application.

### Key Features

- 🔍 **Automatic Discovery** - Scans JSON resources for provider definitions
- ✅ **Real Validation** - Tests each provider with actual API calls (NO MOCKS)
- 🔧 **Smart Integration** - Automatically adds valid providers to config
- 📊 **Comprehensive Reports** - Generates detailed validation reports
- ⚡ **Performance Optimized** - Parallel validation with configurable timeouts
- 🛡️ **Auth Handling** - Detects and handles API key requirements

---

## Architecture

### Components

1. **provider_validator.py** - Core validation engine

   - Validates HTTP JSON APIs

   - Validates HTTP RPC endpoints

   - Validates Hugging Face models

   - Handles authentication requirements



2. **auto_provider_loader.py** - Discovery and orchestration

   - Scans resource files

   - Coordinates validation

   - Integrates valid providers

   - Generates reports



### Provider Types Supported



| Type | Description | Example |

|------|-------------|---------|

| `HTTP_JSON` | REST APIs returning JSON | CoinGecko, CoinPaprika |

| `HTTP_RPC` | JSON-RPC endpoints | Ethereum nodes, BSC RPC |

| `WEBSOCKET` | WebSocket connections | Alchemy WS, real-time feeds |

| `HF_MODEL` | Hugging Face models | Sentiment analysis models |



---



## Quick Start



### 1. Basic Usage



Run the APL to discover and validate all providers:



```bash

cd /workspace

python3 auto_provider_loader.py

```



This will:

- Scan `api-resources/*.json` for provider definitions

- Scan `providers_config*.json` for existing providers

- Discover HF models from `backend/services/`

- Validate each provider with real API calls

- Generate comprehensive reports

- Update `providers_config_extended.json` with valid providers



### 2. Understanding Output



```

================================================================================

🚀 AUTO PROVIDER LOADER (APL) - REAL DATA ONLY

================================================================================



📡 PHASE 1: DISCOVERY

  Found 339 HTTP provider candidates

  Found 4 HF model candidates



🔬 PHASE 2: VALIDATION

  ✅ Valid providers

  ❌ Invalid providers

  ⚠️  Conditionally available (requires auth)



📊 PHASE 3: COMPUTING STATISTICS

🔧 PHASE 4: INTEGRATION

📝 PHASE 5: GENERATING REPORTS

```



### 3. Generated Files



After running APL, you'll find:



- `PROVIDER_AUTO_DISCOVERY_REPORT.md` - Human-readable report

- `PROVIDER_AUTO_DISCOVERY_REPORT.json` - Machine-readable detailed results

- `providers_config_extended.backup.{timestamp}.json` - Config backup

- `providers_config_extended.json` - Updated with new valid providers



---



## Validation Logic



### HTTP Providers



For each HTTP provider, APL:



1. **Checks URL structure**
   - Detects placeholder variables (`{API_KEY}`, `{PROJECT_ID}`)
   - Identifies WebSocket endpoints (`ws://`, `wss://`)

2. **Determines endpoint type**
   - JSON REST API → GET request to test endpoint
   - JSON-RPC → POST request with `eth_blockNumber` method

3. **Makes real test call**
   - 8-second timeout
   - Handles redirects
   - Validates response format

4. **Classifies result**
   - ✅ `VALID` - Responds with 200 OK and valid data
   - ❌ `INVALID` - Connection fails, timeout, or error response
   - ⚠️ `CONDITIONALLY_AVAILABLE` - Requires API key (401/403)
   - ⏭️ `SKIPPED` - WebSocket (requires separate validation)

### Hugging Face Models

For each HF model, APL:

1. **Queries HF Hub API**
   - Checks if model exists: `GET https://huggingface.co/api/models/{model_id}`
   - Does NOT download or load the full model (saves time/resources)

2. **Validates accessibility**
   - ✅ `VALID` - Model found and publicly accessible
   - ⚠️ `CONDITIONALLY_AVAILABLE` - Requires HF_TOKEN

   - ❌ `INVALID` - Model not found (404) or other error



---



## Configuration



### Environment Variables



APL respects these environment variables:



| Variable | Purpose | Default |

|----------|---------|---------|

| `HF_TOKEN` | Hugging Face API token | None |
| `ETHERSCAN_API_KEY` | Etherscan API key | None |
| `BSCSCAN_API_KEY` | BSCScan API key | None |
| `INFURA_PROJECT_ID` | Infura project ID | None |
| `ALCHEMY_API_KEY` | Alchemy API key | None |

### Validation Timeout

Default timeout is 8 seconds. To customize:

```python

from auto_provider_loader import AutoProviderLoader



apl = AutoProviderLoader()

apl.validator.timeout = 15.0  # 15 seconds

await apl.run()

```

---

## Adding New Provider Sources

### 1. Add to JSON Resources

Create or update a JSON file in `api-resources/`:

```json

{

  "registry": {

    "my_providers": [

      {

        "id": "my_api",

        "name": "My API",

        "category": "market_data",

        "base_url": "https://api.example.com/v1",

        "endpoints": {

          "prices": "/prices"

        },

        "auth": {

          "type": "none"

        }

      }

    ]

  }

}

```

### 2. Re-run APL

```bash

python3 auto_provider_loader.py

```

APL will automatically discover and validate your new provider.

---

## Integration with Existing Code

### Using Validated Providers

After APL runs, valid providers are in `providers_config_extended.json`:

```python

import json



# Load validated providers

with open('providers_config_extended.json', 'r') as f:

    config = json.load(f)



# Get all valid providers

valid_providers = config['providers']



# Use a specific provider

coingecko = valid_providers['coingecko']

print(f"Provider: {coingecko['name']}")

print(f"Category: {coingecko['category']}")

print(f"Response time: {coingecko['response_time_ms']}ms")

```

### Filtering by Category

```python

# Get all market data providers

market_providers = {

    pid: data for pid, data in valid_providers.items()

    if data.get('category') == 'market_data'

}

```

---

## Conditional Providers

Providers marked as `CONDITIONALLY_AVAILABLE` require API keys:

### 1. Check Requirements

See `PROVIDER_AUTO_DISCOVERY_REPORT.md` for required env vars:

```markdown

### Conditionally Available Providers (90)



- **Etherscan** (`etherscan_primary`)

  - Required: `ETHERSCAN_PRIMARY_API_KEY` environment variable

  - Reason: HTTP 401 - Requires authentication

```

### 2. Set Environment Variables

```bash

export ETHERSCAN_API_KEY="your_key_here"

export BSCSCAN_API_KEY="your_key_here"

```

### 3. Re-run Validation

```bash

python3 auto_provider_loader.py

```

Previously conditional providers will now validate as VALID if keys are correct.

---

## Performance Tuning

### Parallel Validation

HTTP providers are validated in batches of 10 to balance speed and resource usage:

```python

# In auto_provider_loader.py

batch_size = 10  # Adjust based on your needs

```

Larger batches = faster but more network load  
Smaller batches = slower but more conservative

### Timeout Adjustment

For slow or distant APIs:

```python

validator = ProviderValidator(timeout=15.0)  # 15 seconds

```

---

## Troubleshooting

### Issue: Many providers marked INVALID

**Possible causes:**
- Network connectivity issues
- Rate limiting (try again later)
- Providers genuinely down

**Solution:** Check individual error reasons in report

### Issue: All providers CONDITIONALLY_AVAILABLE



**Cause:** Most providers require API keys



**Solution:** Set required environment variables



### Issue: HF models all INVALID



**Causes:**

- No internet connection to HuggingFace

- Models moved or renamed

- Rate limiting from HF Hub



**Solution:** Check HF Hub status, verify model IDs



### Issue: Validation takes too long



**Solutions:**

- Reduce batch size

- Decrease timeout

- Filter providers before validation



---



## Advanced Usage



### Validating Specific Providers



```python

from provider_validator import ProviderValidator
import asyncio

async def validate_one():

    validator = ProviderValidator()

    

    result = await validator.validate_http_provider(

        "coingecko",

        {

            "name": "CoinGecko",

            "category": "market_data",
            "base_url": "https://api.coingecko.com/api/v3",

            "endpoints": {"ping": "/ping"}

        }

    )

    

    print(f"Status: {result.status}")

    print(f"Response time: {result.response_time_ms}ms")


asyncio.run(validate_one())

```



### Custom Discovery Logic



```python

from auto_provider_loader import AutoProviderLoader



class CustomAPL(AutoProviderLoader):

    def discover_http_providers(self):

        # Your custom logic

        providers = super().discover_http_providers()

        # Filter or augment

        return [p for p in providers if p['data'].get('free') == True]



apl = CustomAPL()

await apl.run()

```



---



## API Reference



### ProviderValidator



```python

class ProviderValidator:

    def __init__(self, timeout: float = 10.0)

    

    async def validate_http_provider(

        provider_id: str,
        provider_data: Dict[str, Any]

    ) -> ValidationResult

    

    async def validate_hf_model(

        model_id: str,

        model_name: str,

        pipeline_tag: str = "sentiment-analysis"

    ) -> ValidationResult

    

    def get_summary() -> Dict[str, Any]

```


### AutoProviderLoader

```python

class AutoProviderLoader:

    def __init__(self, workspace_root: str = "/workspace")

    

    def discover_http_providers() -> List[Dict[str, Any]]

    def discover_hf_models() -> List[Dict[str, Any]]

    

    async def validate_all_http_providers(providers: List)

    async def validate_all_hf_models(models: List)

    

    def integrate_valid_providers() -> Dict[str, Any]

    def generate_reports()

    

    async def run()  # Main entry point

```

---

## Best Practices

1. **Regular Re-validation**
   - Run APL weekly to catch provider changes
   - Providers can go offline or change endpoints

2. **Monitor Conditional Providers**
   - Set up API keys for high-value providers
   - Track which providers need auth

3. **Review Reports**
   - Check invalid providers for patterns
   - Update configs based on error reasons

4. **Backup Configs**
   - APL creates automatic backups
   - Keep manual backups before major changes

5. **Test Integration**
   - After APL runs, test your application
   - Verify new providers work in your context

---

## Zero Mock/Fake Data Guarantee

**APL NEVER uses mock or fake data.**

- All validations are REAL API calls
- All response times are ACTUAL measurements
- All status classifications based on REAL responses
- Invalid providers are GENUINELY unreachable
- Valid providers are GENUINELY functional

This guarantee ensures:
- Production-ready validation results
- Accurate performance metrics
- Trustworthy provider recommendations
- No surprises in production

---

## Support

### Documentation

- `PROVIDER_AUTO_DISCOVERY_REPORT.md` - Latest validation results
- `APL_FINAL_SUMMARY.md` - Implementation summary
- This guide - Usage instructions

### Common Questions

**Q: Can I use APL in CI/CD?**  
A: Yes! Run `python3 auto_provider_loader.py` in your pipeline.

**Q: How often should I run APL?**  
A: Weekly for production, daily for development.

**Q: Can I add custom provider types?**  
A: Yes, extend `ProviderValidator` class with new validation methods.

**Q: Does APL support GraphQL APIs?**  
A: Not yet, but you can extend it by adding GraphQL validation logic.

---

## Version History

### v1.0 (2025-11-16)
- Initial release
- HTTP JSON validation
- HTTP RPC validation
- HF model validation (API-based, lightweight)
- Automatic discovery from JSON resources
- Comprehensive reporting
- Zero mock data guarantee

---

*Auto Provider Loader - Real Data Only, Always.*