Ingestion Control Guide

This guide explains how to manage data ingestion, including pausing/resuming data collection, adjusting rate limiting, and monitoring ingestion status.

Overview

The platform automatically collects data from multiple external sources. Admins can control this process to:

• Pause ingestion during maintenance
• Resume ingestion after troubleshooting
• Adjust rate limiting to prevent API throttling
• Monitor ingestion health and status
• Enable/disable individual feed types

Data Sources & Schedules

Pause/Resume Ingestion

Using the UI

1. Navigate to the ingestion page (/ingestion)
2. Look for the "Pause Ingestion" button in the top right
3. Click to pause ingestion immediately
4. The button changes to "Resume Ingestion" - click to resume
5. The button is color-coded:
   • Amber when ingestion is active (shows "Pause Ingestion")
   • Green when ingestion is paused (shows "Resume Ingestion")

Using the API

Get ingestion status:

GET /api/ingestion/control

Response:

{
  "paused": false,
  "delayMs": 60000
}

Pause ingestion:

POST /api/ingestion/control
Body: {"action": "pause"}

Resume ingestion:

POST /api/ingestion/control
Body: {"action": "resume"}

Individual Feed Control

Enabling/Disabling Feeds

Each feed type can be individually enabled or disabled via environment variables:

Triggering Manual Runs

From the ingestion feeds page (/ingestion/feeds):

1. Find the feed you want to run
2. Click the "Run Now" button
3. The feed will execute immediately regardless of its cron schedule
4. View results in the run history

Rate Limiting

Rate limiting controls the delay between processing each email during newsletter ingestion.

Default Settings

• Default delay: 1000ms (1 second between emails)
• Valid range: 100ms to 300000ms (5 minutes)

Adjusting Rate Limiting

Via API:

POST /api/ingestion/control
Body: {"delayMs": 10000}

Via Environment Variable:

NEWSLETTER_RATE_LIMIT_MS=1000

Feed Concurrency Controls

Safety Limits

Monitoring Ingestion Status

Ingestion Feeds Page (/ingestion/feeds)

View all feed types with:

• Status indicators (running, success, error, locked)
• Last run time and duration
• Registration status and scheduled cadence
• Error details for failed runs
• Lock status (prevents duplicate runs)

Check Recent Runs

1. Navigate to Campaign → Admin → Ingestion
2. View "Recent Runs" table:
   • Green checkmark: Successful run
   • Red X: Failed run
   • Clock icon: Currently running
3. Click on a run to see details:
   • Records found
   • Records added vs duplicates
   • Errors encountered

Health Monitoring Workers

These background jobs automatically monitor ingestion health:

When to Use Pause/Resume

Pause ingestion when:

• Troubleshooting data issues
• Performing database maintenance
• Experiencing rate limiting errors from external APIs
• Need to reduce system load during peak traffic
• Running manual data corrections

Resume ingestion when:

• Troubleshooting is complete
• Maintenance is finished
• Rate limits have reset
• Ready to process new data

Troubleshooting

Issue: Ingestion is paused but I didn't pause it

• Solution: Check if another admin paused it via API or UI

Issue: Feed shows "Locked" but isn't running

• Solution: Feed locks expire after 45 minutes. If a feed crashed mid-run, the lock will auto-expire. You can also check Redis for stale locks.

Issue: Getting rate limited by external API

• Solution: Reduce concurrency settings or increase the feed's cron interval

Issue: Gmail polling not starting

• Solution: Check that REP_LABEL_ID and NEWSLETTER_LABEL_ID are set to valid Gmail Label IDs. Use npx tsx src/workers/mailbox/label-utils.ts to list available labels.

Issue: Pause state resets unexpectedly

• Note: The pause state is in-memory and resets on server restart. This is by design to prevent ingestion from staying paused indefinitely.

Issue: Feed retry queue backing up

• Solution: Check the dead letter queue (/admin/dlq) for persistent failures. Resolve root causes before retrying.

Environment Variables Summary

Core Ingestion

Gmail Polling

Feed Retry

Best Practices

1. Monitor regularly: Check ingestion status daily via /ingestion/feeds
2. Use appropriate concurrency: Start with defaults and increase if the database handles the load
3. Review exceptions: Address data quality issues promptly to prevent backlog
4. Check DLQ weekly: Review /admin/dlq for persistent failures
5. Watch feed health: Stale watermark monitor will alert on feeds that stop running
6. Test config changes: Verify schedule and concurrency changes in development first

Last Updated: March 2026