TLE Processor Service

Overview

The TLE (Two-Line Element) Processor Service is a containerized microservice that automatically fetches, processes, and stores satellite orbital data from Celestrak. It monitors satellite orbital data for specified countries/operators and integrates seamlessly with the PostgreSQL database.

What This Service Does

• Automated TLE Data Fetching: Downloads latest TLE data from Celestrak API
• Country-Based Filtering: Processes only satellites from specified countries/operators
• Smart Scheduling: Runs randomly every 3-6 hours to avoid predictable patterns
• Duplicate Prevention: Handles data conflicts gracefully with PostgreSQL UPSERT
• Slack Notifications: Sends detailed processing statistics after each run
• Health Monitoring: Built-in Docker health checks and monitoring

How It Works

1. Startup: Service starts and runs immediately
2. Data Fetching: Downloads TLE data from Celestrak API
3. Filtering: Processes only satellites matching configured countries
4. Database Operations:
   • Fetches relevant NORAD IDs from maneuver_app_norad_db
   • Inserts/updates TLE data in maneuver_app_tle_data
5. Notifications: Sends processing statistics to Slack
6. Scheduling: Schedules next run randomly between 3-6 hours

Configuration

Environment Variables

Variable	Description	Default	Required
COUNTRY_OPERATORS	Comma-separated list of countries/operators	xxxx,xxx	Yes
SLACK_WEBHOOK_URL	Slack webhook URL for notifications	None	No
DB_HOST	PostgreSQL host	localhost	No
DB_PORT	PostgreSQL port	5432	No
POSTGRES_USER	Database user	maneuver_db	No
POSTGRES_PASSWORD	Database password	secret@796	No
POSTGRES_DB	Database name	maneuver	No

Database Schema

Required Tables:

-- NORAD IDs with country operators
CREATE TABLE maneuver_app_norad_db (
    id SERIAL PRIMARY KEY,
    norad_id VARCHAR(10) NOT NULL,
    country_operator VARCHAR(50) NOT NULL,
    -- other fields...
);

-- TLE data storage
CREATE TABLE maneuver_app_tle_data (
    id SERIAL PRIMARY KEY,
    tle_date DATE NOT NULL,
    norad_id VARCHAR(10) NOT NULL,
    tle TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(norad_id, tle_date)
);

Usage

Start the Service

# Build and start all services
docker-compose up -d

# Start only TLE processor
docker-compose up -d tle-processor

Monitor Logs

# Follow live logs
docker-compose logs -f tle-processor

# View specific log file
tail -f tle-processor/logs/tle_log_$(date +%Y-%m-%d).log

Check Service Status

# Check if service is running
docker-compose ps tle-processor

# Check health status
docker inspect tle-processor --format='{{.State.Health.Status}}'

Data Flow

1. Initialization: Service connects to PostgreSQL and validates required tables
2. Country Filtering: Retrieves NORAD IDs for configured countries from maneuver_app_norad_db
3. TLE Fetching: Downloads latest TLE data from Celestrak API
4. Data Processing: Filters TLEs for satellites matching country operators
5. Database Storage: Inserts/updates TLE data in maneuver_app_tle_data with conflict resolution
6. Statistics: Calculates processing statistics (total TLEs, new entries, updates)
7. Notifications: Sends statistics to Slack if webhook configured
8. Scheduling: Schedules next run randomly between 3-6 hours

Monitoring and Health Checks

Health Check Endpoint

The service includes built-in health monitoring:

# Check container health
docker inspect sss_tle_processor --format='{{.State.Health.Status}}'

# View health check logs
docker inspect sss_tle_processor --format='{{range .State.Health.Log}}{{.Output}}{{end}}'

Logging

The service provides comprehensive logging:

• Info Level: Processing steps, statistics, scheduling
• Warning Level: Data conflicts, retry attempts
• Error Level: Connection failures, API errors, database issues

Metrics

Key metrics tracked:

• Total TLEs processed per run
• New TLE entries added
• Updated TLE entries
• Processing duration
• Error counts
• Success/failure rates

Error Handling

Common Issues

1. Database Connection: Service retries connection with exponential backoff
2. API Failures: Celestrak API failures trigger retry mechanism
3. Data Conflicts: Duplicate TLEs are handled gracefully with UPSERT
4. Memory Issues: Large datasets are processed in chunks

Troubleshooting

# Check service logs
docker-compose logs tle-processor

# Restart service
docker-compose restart tle-processor

# Check database connectivity
docker exec -it sss_tle_processor python -c "
import psycopg2
try:
    conn = psycopg2.connect(
        host='postgres',
        database='maneuver',
        user='maneuver_db',
        password='secret@796'
    )
    print('Database connection successful')
    conn.close()
except Exception as e:
    print(f'Database connection failed: {e}')
"

Integration

With FastAPI Service

The TLE processor feeds data to the FastAPI service:

• Satellite Operations: TLE data used for position calculations
• Pass Analyzer: TLE data required for pass analysis
• Maneuver Detection: TLE history used for maneuver detection

With Other Services

• News Processor: Independent service, no direct integration
• Monitoring: Prometheus metrics available for monitoring
• Logging: Centralized logging through Docker volumes

Performance Considerations

Optimization

• Batch Processing: TLEs processed in batches for efficiency
• Connection Pooling: Database connections are pooled
• Memory Management: Large datasets processed incrementally
• Caching: Frequently accessed data cached in memory

Resource Usage

• CPU: Low to moderate usage during processing
• Memory: Minimal memory footprint (~50-100MB)
• Storage: Log files and temporary data
• Network: Periodic API calls to Celestrak

Security

Data Protection

• Encryption: Database connections use SSL/TLS
• Authentication: Database credentials stored as environment variables
• Access Control: Service runs with minimal required permissions

Network Security

• Internal Communication: Service communicates only with PostgreSQL
• External APIs: HTTPS connections to Celestrak API
• Firewall: Service doesn't expose external ports

Deployment

Docker Configuration

tle-processor:
  build: ./tle-processor
  container_name: sss_tle_processor
  restart: unless-stopped
  volumes:
    - ./logs:/app/logs
  environment:
    - TZ=${TZ}
    - DB_HOST=${DB_HOST}
    - DB_PORT=${DB_PORT}
    - DB_USER=${DB_USER}
    - DB_PASSWORD=${DB_PASSWORD}
    - DB_NAME=${DB_NAME}
    - COUNTRY_OPERATORS=${COUNTRY_OPERATORS}
    - SLACK_WEBHOOK_URL=${SLACK_WEBHOOK_URL}
  depends_on:
    postgres:
      condition: service_healthy

Production Considerations

1. High Availability: Service automatically restarts on failure
2. Data Backup: Regular database backups recommended
3. Monitoring: Set up alerts for processing failures
4. Scaling: Service can be scaled horizontally if needed
5. Updates: Regular updates for security and features