Design preview · Architecture proposal. No generation service has been deployed.View the plan ↗
Waving
Knowledge
← Customer experience plan

System design / One GPU / 26 October 2026

Many requests.
One GPU.
A reliable queue.

Accept avatar requests independently of GPU availability. Generate a reviewed preview first, show the customer a quote afterwards, and unlock the full package only when they choose to pay.

Open SVG diagram ↗Download SVG ↓
VERIFIED GPURTX 5070 Ti

One card · approximately 16 GB VRAM

VERIFIED DATABASEPostgreSQL 17.9

Dedicated wavingknowledge database exists

PROPOSED GPU CONCURRENCYOne job

All other accepted requests wait durably

01 / The whole journey

Generation comes
before the purchase.

Architecture flow from customer upload through a PostgreSQL queue and single GPU worker to preview, quote, optional Stripe payment and private downloads.
Open or download the SVG to inspect every component. This is a proposed architecture, not a running deployment.

02 / What is already on the server

Build around
the existing machine.

Read-only inspection confirmed the GPU, PostgreSQL and existing character pipeline on the same server. The vLLM service is active and was using approximately 15 GB of VRAM at inspection time. Generation cannot run alongside it in the current configuration.

Reuse

  • Existing Node website/API, migrated from its current SQLite order model to PostgreSQL accounts.
  • Existing Python environment, head-generation scripts and read-only model weights.
  • The clothed-bust extension, after creating or selecting a verified trained head.
  • Conversation Kit rendering and the customer-area prototype.

Add

  • A durable generation queue and immutable avatar versions, independent of paid orders.
  • A supervised Python worker, a shared GPU controller and private job directories.
  • Quality review, preview publishing, quote binding and transactional notifications.
  • Operator controls for pause, retry, cancellation and recovery.

No GPU process was stopped, no queue tables were created and no generation worker was installed for this design. The existing database foundation remains empty.

03 / Deployment boundaries

Three services.
One source of truth.

Component Responsibility Privilege boundary
wk-api.service
Node application		 Verified accounts, upload validation, request admission, status, previews, quotes, Stripe and downloads. Restricted PostgreSQL role and private file access. No CUDA or permission to stop services.
wk-avatar-worker.service
Python supervisor 		Claim work, run allowlisted stages, heartbeat, validate outputs and publish manifests. Unprivileged service user; access only to job storage, required weights and the assigned GPU.
wk-gpu-controller.service
Local resource controller 		Own the host-wide GPU slot; enforce approved windows; drain and restore vLLM. Narrow local interface. Only this controller can manage the specifically allowlisted GPU service.
PostgreSQL + private files Durable state, ownership, retries, outbox and manifests; files stored outside the public web directory. Loopback database access; encrypted backups and a tested restore procedure before launch.

Start with PostgreSQL as the queue, avoiding a separate message broker for this single-worker deployment. Job rows are durable; optional notifications only wake the worker. Polling remains the recovery path if a notification is missed.

04 / The invariant

At most one workload
owns the GPU.

  1. Admit independently. The API persists a validated request and returns its ID. Closing the browser does not cancel work.
  2. Acquire the resource before a job. The controller obtains the shared host lock for the physical GPU. Every avatar worker and manual GPU tool must use the same controller/lock; unmanaged GPU processes block dispatch.
  3. Claim atomically. In a short transaction, select the oldest eligible job with FOR UPDATE SKIP LOCKED, set its lease owner, expiry and increasing attempt token, then commit. Do not hold a database transaction open throughout training.
  4. Verify exclusivity. A row claim alone does not limit GPU concurrency. The controller’s lock and supervised process group enforce the single physical slot. An advisory lock may coordinate controllers, but losing a database session must never imply that an old GPU subprocess has stopped.
  5. Run and heartbeat. Proposed heartbeat every 15 seconds; lease expiry after 90 seconds without renewal. Publish results only while the current attempt token still owns the job.
  6. Release safely. On completion, failure or cancellation, terminate the full job process group, verify GPU release and restore vLLM if the controller stopped it. Hold the lock through restoration and health checking.

A lease expiry moves a request to recovery required. It is not permission to immediately start a second process. Reconcile or terminate the previous attempt first. On an uncertain controller/host state, pause dispatch and notify the operator.

05 / The pipeline adapter

Keep each customer’s
work separate.

Stage Resource Checkpoint and acceptance
Validate photograph & consent CPU Decode, strip metadata, check size/face/framing and retain an immutable source hash. Current checkout minimums are too low for the documented training workflow; define and test a higher input-quality gate.
Create attempt workspace CPU /var/lib/wavingknowledge/generation/<job-id>/<attempt>/. Pin pipeline version, model hashes, seed, settings and source assets. Shared weights are read-only.
Generate head views GPU Existing six-video head pipeline, with each completed and validated video recorded separately. Review consistency before fitting.
Track, segment & fit CPU Save validated masks, tracked landmarks, fitted geometry and camera reports.
Train the head GPU Validated checkpoint and new head asset. Resume only if the adapter supports the exact checkpoint format; otherwise rerun this stage, not the whole job.
Optional clothed bust GPU + CPU Use the verified head and shape, generate the two clothed views, fit fixed identity and train garment splats. A clothing revision may reuse the same validated head.
Export, render & review CPU; GPU only if required by renderer Asset hashes, geometry checks, neutral/turned/speaking renders and human likeness review. Render jobs use the same GPU slot if they require the GPU.
Publish preview CPU Approved, watermarked video; immutable version ID; account-authorized access. Free the GPU before human review or customer decisions.

The existing research wrapper uses a fixed working directory and can stop vLLM. It must be adapted into controlled stage commands before unattended use. Never execute customer-supplied shell commands or overwrite an existing character’s outputs.

New identities run the full required pipeline. Revisions reference an immutable base version and a change request; the adapter decides which stages can safely be reused. Existing purchased files remain available while a revision is being prepared.

06 / Queue and recovery policy

A clear place in line.
A recoverable job.

DraftValidatingQueuedWaiting for GPURunning stageQuality reviewPreview ready

Admission and fairness

  • Proposed pilot: one queued/running generation per verified customer and a configurable daily limit. Duplicate clicks reuse the same request ID.
  • FIFO by accepted time among eligible jobs; retries use backoff and do not occupy the GPU while waiting. Operator reprioritization is explicit and audited.
  • Limit total queued GPU minutes to the scheduled capacity available before the admission deadline. When full, show a waitlist or ask the customer to return.
  • Rate-limit free preview generation. The business incurs GPU cost before the customer chooses to buy.

Failure and cancellation

  • Invalid images require customer action. Transient stage failures may retry twice with backoff; exhausted attempts require operator review.
  • Out-of-memory errors trigger resource checks, not repeated identical retries. Save sanitized errors and bounded logs.
  • Queued jobs cancel immediately. Running jobs use a cancellation flag; stop the process group, retain valid checkpoints and release the GPU safely.
  • On restart or lost lease, reconcile workspaces, processes and manifests. Never run two attempts for the same job.

Proposed status API: authenticated polling every 5–10 seconds, with backoff. Show queue state, current stage and a measured estimate range—not a fabricated percentage or other customers’ details.

07 / Preview, quote and purchase

Show the result.
Then the value.

  1. Approve a version. Operator QA publishes the preview; the customer reviews it. No price is shown on marketing, account or library screens.
  2. Create the quote. The ordering screen requests a server-issued quote tied to customer, request type, approved version/hash, licence scope, price, tax configuration and expiry. The initial and additional-avatar rules remain server-side; revision pricing is still to be decided.
  3. Accept explicitly. The customer may leave or request changes without paying. Changes produce a new version and invalidate any unaccepted quote for the previous version. A paid quote always retains its original version.
  4. Verify payment. Create Stripe Checkout from the frozen quote. Signed payment events are reconciled with the saved session, customer, amount, currency and version. Deduplicate and fulfil once.
  5. Grant the right entitlement. Initial/additional avatars issue a new licence; a revision adds an approved release to the existing owned licence. Record each revision purchase separately.
  6. Package and deliver. Package the already approved artifacts with the player and guide on CPU. Payment does not queue another generation. Downloads require account ownership and an active entitlement; refund/dispute handling follows the affected purchase.

Keep the full model out of the unpaid browser.

A WebGL preview that sends the full avatar model to the customer can be copied, even without a download button. For the first release, show watermarked turn-and-speech video before purchase; deliver the complete interactive model after payment. Preview videos can still be captured. The current HTML demonstration uses the public Kenji sample only.

08 / Data model changes

Generation is a request.
Payment is a later decision.

Record Purpose and constraints
generation_requests Customer, avatar, initial/additional/revision kind, base version, consent, normalized input, change request and customer-scoped idempotency key. Exists before an order.
generation_jobs Request, stage, state, queue time, available-at, attempt count, lease/heartbeat, cancellation and current attempt token. One live job per request; configured account admission cap.
generation_attempts / stage_artifacts Immutable attempt history, pinned versions, checksums, file keys, checkpoints, timings and error codes. Publish by atomic manifest commit.
gpu_resources Resource identity, controller ownership, mode, desired service restoration state and heartbeat. Database state is reconciled with the host process lock.
avatar_versions / preview_assets Version provenance, QA result, private full model and separate watermarked preview. Every reference carries customer ownership.
quotes / orders Reviewed version, immutable commercial snapshot, expiry, acceptance and Stripe identifiers. Order uniqueness prevents duplicate checkout for one accepted quote.
licences / release_entitlements Purchased identity plus version/release access. Revision orders reference an existing licence instead of creating a duplicate active licence.
outbox_events / audit_events Reliable notifications and an operator trail. Insert outbox work with the state change, then send/retry independently.

The earlier draft’s payment-linked fulfillment_jobs is insufficient for this flow. Generation must be independent; keep a separate post-payment packaging task. The existing SQL draft is marked superseded and has not been applied. Review this revised model before writing migrations.

09 / Service contracts

Small endpoints.
Explicit transitions.

Proposed endpoint Contract
POST /api/generation-requests Verified account, photo, request type and idempotency key → 202 with request ID. Persist normalized input, request and queue entry before acknowledging.
GET /api/generation-requests/:id Owner-only status, current stage, cancellation availability, estimated range and preview readiness.
POST /api/generation-requests/:id/cancel Owner-only state transition. Cancellation is not deletion of an already purchased version.
GET /api/avatar-versions/:id/preview Owner-only QA-approved preview, never the full unpaid model. Short-lived access; private cache policy.
POST /api/quotes Explicitly reviewed approved version → frozen server quote. Revalidate identity, ownership, revision base and price eligibility.
POST /api/quotes/:id/checkout Unexpired accepted quote → idempotent hosted checkout. A redirect cannot issue a licence.
POST /api/stripe/webhook Signature verification, payment reconciliation and idempotent licence/release grant.
GET /api/releases/:id/download Authenticated customer + active entitlement + released private artifact.

10 / Capacity and operating decisions

The GPU schedule
is part of the product.

The documented head-only baseline is approximately 100 GPU minutes per character. That is a theoretical ceiling of about 14 head-only jobs per day if the GPU ran generation continuously. It excludes clothing, retries, CPU work, review and vLLM time; it is not a delivery promise. Measure the complete production recipe before publishing estimates.

11 / Implementation after design review

Prove recovery
before accepting customers.

  1. Agree the GPU/vLLM operating policy and unpaid generation limits. Finalize model provenance, retention and revision pricing.
  2. Write PostgreSQL migrations and account authorization; add private file storage and idempotent request admission.
  3. Wrap the existing pipeline into deterministic stage commands and isolated job directories. Use a dummy worker first to test FIFO, claims, restarts and cancellations without occupying the GPU.
  4. Implement the controller and supervised worker. Test two worker instances, database interruption, killed subprocesses, full disk and vLLM restoration. No second GPU attempt may start until the old process is stopped.
  5. Run one actual portrait end to end in an agreed GPU window. Validate head and clothing outputs, QA, preview publication and notification.
  6. Bind approved versions to quotes and Stripe sandbox payments. Prove exactly-once entitlement effects under duplicate/out-of-order events, cross-account isolation and revision/refund behavior.
  7. Open a bounded pilot, measure actual throughput and support effort, then increase admission capacity.

Design references: PostgreSQL supports queue consumers using row locking with SKIP LOCKED; its advisory-lock lifecycle explains why a database lock alone cannot supervise GPU subprocesses. Payment fulfilment follows Stripe’s webhook-driven fulfilment guidance. Pipeline timings and stage boundaries come from the local Conversation Kit character-creation and clothed-bust documentation, plus the read-only server inspection.

Review the customer flow →Download architecture SVG ↓