# Operations Runbook

## Queue Worker

1. Start worker:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan queue:work --queue=default --tries=3 --sleep=3 --timeout=120
```

2. Recommended cPanel cron (every minute):

```bash
cd /home/hometech/laravel && /opt/cpanel/ea-php83/root/usr/bin/php artisan schedule:run >> /dev/null 2>&1
```

## Health Checks

1. Queue summary:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan queue:health-check
```

2. Strict threshold check:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan queue:health-check --warn-pending=50 --warn-failed=5
```

This command now reports:
- pending retry jobs
- pending non-retry jobs
- failed retry jobs

3. Send warning alert to n8n when thresholds are exceeded:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan queue:health-notify --warn-pending=50 --warn-failed=5
```

Force a live alert test even if thresholds are not exceeded:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan queue:health-notify --force
```

Preview the exact alert payload without posting to n8n:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan queue:health-notify --force --dry-run
```

4. Required env for queue alerts:

```dotenv
N8N_HEALTH_WEBHOOK_URL=https://your-n8n/webhook/queue-health
```

5. Max retry policy (recommended):

```dotenv
N8N_MAX_RETRY_ATTEMPTS=5
N8N_HEALTH_ALERT_CHANNEL=ops
```

When the threshold is reached, webhook status is set to `abandoned` and retry actions are blocked.

## Webhook Recovery Commands

1. Retry failed webhooks immediately:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan webhooks:retry-failed --limit=50
```

2. Queue retry flow:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan webhooks:retry-failed --limit=50 --queue
```

3. Preview queued retry job cleanup:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan webhooks:clear-retry-queue --dry-run
```

4. Remove retry jobs from queue tables only:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan webhooks:clear-retry-queue
```

Retry timeline metadata is recorded in webhook log responses as structured `RetryMeta` entries.

Webhook log triage helpers in admin:
- Preset tabs: All, Needs Attention, Inbound, Outbound
- For `abandoned` items, use the `Force Retry` action with required override reason (audit tracked).

## Readiness and Reporting

1. Check integration configuration readiness:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan integrations:check-readiness
```

2. Check readiness and verify health webhook reachability:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan integrations:check-readiness --ping
```

3. Generate abandoned webhook trend report:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan webhooks:report-abandoned --hours=24 --limit=10
```

4. Generate and send abandoned webhook trend report to n8n:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan webhooks:report-abandoned --hours=24 --limit=10 --notify
```

5. Scheduler automation:
- Queue health notify: every 5 minutes
- Abandoned webhook report notify: daily at 08:00

## Static Content Utilities

Normalize imported static page bodies for section-based rendering:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan cms:normalize-static-page-bodies
```

## Integration Smoke Tests

1. WHMCS API connectivity test:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan integrations:whmcs-test
```

2. Structured seed data bootstrap:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan cms:seed-structured-content
```

## Production Activation Checklist

Follow these steps in order for live n8n + WHMCS activation.

1. Set production values in `.env`:

```dotenv
APP_ENV=production
APP_DEBUG=false
APP_URL=https://hometech.one

N8N_ENABLED=true
N8N_CONTACT_WEBHOOK_URL=https://<your-n8n-host>/webhook/contact
N8N_HEALTH_WEBHOOK_URL=https://<your-n8n-host>/webhook/queue-health
N8N_WEBHOOK_TOKEN=<your-n8n-bearer-token>
N8N_SIGNING_SECRET=<your-n8n-signing-secret>
N8N_TIMEOUT=8
N8N_MAX_RETRY_ATTEMPTS=5
N8N_HEALTH_ALERT_CHANNEL=ops

WHMCS_URL=https://cp.hometech.one/includes/api.php
WHMCS_IDENTIFIER=<your-whmcs-identifier>
WHMCS_SECRET=<your-whmcs-secret>
WHMCS_ACCESS_KEY=<your-whmcs-access-key>
WHMCS_WEBHOOK_SECRET=<your-whmcs-webhook-secret>
WHMCS_TIMEOUT=8
```

2. Reload cached configuration:

```bash
cd /home/hometech/laravel
/opt/cpanel/ea-php83/root/usr/bin/php artisan optimize:clear
/opt/cpanel/ea-php83/root/usr/bin/php artisan config:cache
/opt/cpanel/ea-php83/root/usr/bin/php artisan route:cache
/opt/cpanel/ea-php83/root/usr/bin/php artisan view:cache
```

3. Validate integration keys are present:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan integrations:check-readiness
```

Expected result:
- Integration readiness: OK

4. Verify health webhook reachability:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan integrations:check-readiness --ping
```

Expected result:
- Health webhook ping succeeded.

5. Trigger a forced live queue warning alert:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan queue:health-notify --force
```

Expected result:
- Queue warning alert sent successfully.

6. Confirm dry-run payload shape (safe diagnostic):

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan queue:health-notify --force --dry-run
```

Validate these fields are present:
- event
- forced
- pending_retry_jobs
- pending_non_retry_jobs
- failed_retry_jobs
- top_failing_webhook_events
- host
- domain
- brand_name
- alert_channel
- timestamp

7. Validate abandoned webhook reporting:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan webhooks:report-abandoned --hours=24 --limit=10
/opt/cpanel/ea-php83/root/usr/bin/php artisan webhooks:report-abandoned --hours=24 --limit=10 --notify
```

Expected result:
- command prints totals and top abandoned events
- notify mode sends report successfully to n8n

8. Validate WHMCS API connectivity:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan integrations:whmcs-test
```

Expected result:
- WHMCS API request succeeded

9. Scheduler and worker confirmation:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan schedule:list
```

Confirm scheduled entries:
- queue:health-notify every five minutes
- webhooks:report-abandoned --hours=24 --limit=10 --notify daily at 08:00

Worker command:

```bash
/opt/cpanel/ea-php83/root/usr/bin/php artisan queue:work --queue=default --tries=3 --sleep=3 --timeout=120
```

10. Post-activation checks in admin:
- Review Webhook Logs tabs: All, Needs Attention, Inbound, Outbound
- Confirm Force Retry prompts for override reason
- Confirm abandoned items can be manually recovered when approved