Jobs & schedulers
Async work and reconciliation
Mivama uses three asynchronous mechanisms: Frappe background jobs, external pull workers and scheduled reconciliation. They solve different failure modes.
Job types
Root workers pull work instead of receiving arbitrary commands from web requests. A claimed job has owner, timestamps and heartbeat. Stale claims can be recovered according to the worker policy; duplicate concurrent execution is prevented with service/workspace locks.
Frappe schedule
Configured in mivama_core/hooks.py:
| Cadence | Function | Why |
|---|---|---|
| Daily | reconcile_portal_stripe_subscriptions | Repair webhook gaps against Stripe truth |
| Daily | clear_old_portal_notifications | Bound notification storage |
| Daily | reconcile_hosting_subscriptions | Repair lifecycle drift |
| Hourly | retry_pending_portal_stripe_events | Reprocess failed/deferred events |
| Hourly | expire_stale_migrations | Revoke stale transfer capabilities |
| Hourly | process_hosting_cancellations | Advance backup, suspend, retention and delete |
Immediate hooks still run; schedulers are the safety net, not the primary user feedback loop.
Idempotency
External IDs deduplicate Stripe events/invoices/subscriptions. Desired-state generation and hash prevent stale host apply. Migration commands require legal state transitions. Studio job/state checks reject incompatible actions. Retried work must either converge safely or return the existing completed result.
Monitoring signals
- queue depth and oldest queued age;
- active claim heartbeat age;
- repeated job retries for one service;
- Stripe events not Processed;
- migrations in importing/final sync beyond expected duration;
- Studio workspaces stuck in provisioning, deploying or rollback;
- cancellation records past their next-action timestamp;
- failed email queue entries for essential messages.
Alerts should include stable service/job/migration/workspace IDs, never credentials or raw customer data.