Technical Due-Diligence Report¶
HumanityAI'd Museum AR Smart Glass Guide · Backend (musem_API) + Glasses App (musem_ANDROID) — v1.0 · July 2026
Confidential — prepared for prospective partners & acquirers
Download
1. Purpose & scope¶
This report is an honest, buyer-grade technical assessment of the two codebases that make up the Museum AR Smart Glass Guide:
- musem_API — the on-premises AI backend (~6,300 lines Python): FastAPI, PostgreSQL 15, Redis 7, CLIP ViT-L/14 + FAISS recognition, Chatterbox multilingual TTS, WebRTC ingest, nginx, Prometheus/Grafana.
- musem_ANDROID — the RayNeo X3 Pro glasses app (~1,600 lines Kotlin/Java): Camera2 → WebRTC H.264 streaming, DataChannel result overlay, ExoPlayer narration.
It is written to survive scrutiny by a buyer's own engineers. It states strengths plainly and flaws plainly, and it distinguishes what is product IP from what is POC-grade and must be hardened before a production sale. A conservative hardening estimate is included.
2. Verdict in one paragraph¶
The product contains genuine, differentiated engineering — a dwell-based "look and listen" recognition UX, a multilingual content pipeline with an Arabic human-approval workflow, and a low-latency latest-frame-wins streaming design — and it is demonstrably working end-to-end today (10 exhibits, 8 languages, 80 approved narrations, recognition verified at 0.995 similarity). It is a hardened proof-of-concept, not yet a hardened product: the largest gaps are authentication, transport security, automated tests, and single-node capacity. None of these are architectural dead-ends; the realistic hardening effort is 8–14 engineer-weeks across both repos, and five significant backend defects were already professionally resolved in the most recent hardening pass — a positive signal about code trajectory.
3. System architecture¶
RayNeo X3 Pro glasses
│ (Camera2 → HW H.264 → WebRTC, DTLS-SRTP media; DataChannel results)
│ (or snapshot JPEG POST /recognize)
▼
nginx (TLS, rate-limit, static cache)
▼
FastAPI "museum_api" (single uvicorn worker)
├─ webrtc_stream_service (aiortc, latest-frame-wins buffer=1)
├─ recognition_service (dwell state machine: scan→detect→confirm→narrate→cooldown)
│ ├─ CLIP ViT-L/14 (open-clip) → 768-d embedding
│ └─ FAISS IndexFlatIP + position_mapping.npy
├─ tts_service → Chatterbox Multilingual → opus (ffmpeg loudnorm)
├─ PostgreSQL 15 (museums→galleries→exhibits→{images,translations,embeddings,narrations}; sessions, events, tours, api_keys, admin_users)
└─ Redis 7 (tag-based cache)
Admin panel (React) · Prometheus :9091 · Grafana :3002
Data model is well normalized with UUID keys, sensible composite indexes, timezone-aware timestamps, and a narration status workflow (pending → generating → pending_review → approved/failed) with reviewer attribution and an Arabic-requires-approval gate. Alembic manages migrations.
4. Strengths (sellable engineering)¶
Backend
- Clean layered structure (routers → services → models), Pydantic v2 schemas, async SQLAlchemy 2.0 with
selectinloadto avoid lazy-load traps. - Real-time design done right: size-1 latest-frame buffer with drop counting, FPS-throttled recognition, a versioned DataChannel result schema (
museum.exhibit_results.v1), in-band language/gallery switching. - The dwell-state machine ("look at an exhibit, hear narration") is genuine product IP with tunable trigger/cooldown knobs.
- Human-in-the-loop content workflow: statuses, review/approve/reject, AI-generated flag, Arabic approval gating — a strong story for cultural institutions.
- i18n depth: 8 languages including Arabic/Chinese/Hindi/Russian,
name_arRTL-aware columns, per-language scripts and narrations, engine support for 23 languages. - Museum-tuned audio: fade in/out, peak normalization, ffmpeg loudnorm, 80 kbps Opus for quiet galleries.
- Ops posture: multi-stage CUDA Docker image, non-root user, healthchecks, startup-order gating, nginx TLS + rate limits, Prometheus/Grafana,
/healthvs/health/ready, privacy purge endpoint, no-frame-persistence option.
Glasses app
- Small, readable, sensibly packaged (streaming / recognition / overlay / audio / session).
- Correct architecture: WebRTC with hardware H.264 + DataChannel; media is DTLS-SRTP encrypted.
ExhibitSmoothertemporal hysteresis (3-frame confirm / 5-frame lost) is a real UX quality touch.- Modern toolchain (Gradle 9.4.1, AGP 9.2.1, compileSdk 36), scoped network-security config, minimal permissions, 8-language string resources incl. RTL.
5. Findings (prioritized, both repos)¶
Severity: C Critical · H High · M Medium · L Low.
| # | Sev | Area | Finding | Fix |
|---|---|---|---|---|
| B1 | C | Backend auth | Authorization: Bearer <anything> passes the API-key middleware; verify_api_key accepts any signed JWT of any role; admin role not enforced |
Validate JWT in middleware; per-route RBAC; remove blanket Bearer pass-through |
| B2 | C | Backend secrets | Live .env, TLS private key, and DB creds in the tree; ADMIN_SESSION_SECRET defaults to empty → forgeable admin JWTs |
Secrets manager / install-time generation; fail-fast on empty secret; scrub tree |
| A1 | C | App secrets | A hardcoded API_KEY ships in an unobfuscated APK (R8 off). (The value is not reproduced in this report, which is distributed; it is cited by file and line in our internal record and must be treated as compromised.) |
Per-device provisioned token in Keystore; rotate key; enable R8 |
| A2 | C | App signing | Release keystore passwords committed in build.gradle.kts; keystore ships in tree |
New keystore; passwords via gitignored keystore.properties |
| A3 | C | App transport | All signaling/REST is cleartext HTTP incl. API key; 8 dev IPs whitelisted; user CA trust enabled | TLS + pinning; drop user anchors; strip dev IPs from release |
| B3 | H | Backend streaming | WebRTC video frames likely never consumed (no recv() pump) — headline stream path needs on-hardware verification |
Add track-consumption task; load-test |
| B4 | H | Backend multi-tenancy | museum_id ignored in FAISS search — cross-museum matches possible |
Filter FAISS by museum_id / per-museum indexes |
| B5 | H | Backend privacy | Unapproved audio + all images publicly reachable under /static |
Signed URLs or auth on static |
| B6 | H | Backend perf | CLIP inference & FAISS writes block the single event loop (esp. CPU mode) | asyncio.to_thread / inference worker |
| B7 | H | Backend perf | TTS generation runs synchronously in the request → 504 past nginx 30 s | Background job + status polling |
| A4 | H | App robustness | No reconnect/recovery on ICE failure — a Wi-Fi roam kills the session until app restart | Backoff restart state machine; wire NetworkMonitor |
| A5 | H | App analytics | Museum/gallery selection unimplemented → sessions/analytics never run on a fresh install | Fetch museums in onboarding / infer server-side |
| A6 | H | App resources | Camera/encoder teardown races init; VideoSource/Track never disposed | Per-resource cleanup on worker thread; CameraEventsHandler |
| A7 | H | App deps | Unofficial WebRTC M119 (Oct 2023) repackage, ~2.5 yrs stale | Move to maintained build; pin upgrade cadence |
| A8 | H | App build | Build not reproducible — AARs, launcher art, keystore untracked; a clean clone fails | Commit/host binaries with licenses; gate signing on file existence |
| B8/A9 | H | Both | No automated tests, no CI in either repo | pytest+httpx (backend), JUnit (pure-logic classes), CI pipelines |
| — | M/L | Both | FAISS↔DB drift durability, N+1 hot-path queries, dead code, thermal-hostile 1440p/20 Mbps capture on glasses, login lockout, dev-compose mount mismatch, Prometheus metrics declared-but-unset | See roadmap §8 |
6. Already resolved during hardening (credit as fixed)¶
These were found and professionally fixed in the most recent pass and are verified present in the code:
- Exhibit-create 500 — relations now eager-loaded (
db.refresh(..., attribute_names=["images","audio_narrations"])) so Pydantic serialization no longer triggers a lazy load outside the async context. - TTS never initialized — lazy, lock-serialized provider init on first use.
- Multilingual TTS + watermarker crash — switched to
ChatterboxMultilingualTTS;perth.DummyWatermarkerfallback for the brokenPerthImplicitWatermarkerin the base image. - Event-loop freeze during model load/synthesis —
asyncio.to_threadwraps bothfrom_pretrainedandgenerate, so the healthcheck no longer kills the container mid-load. - Persistent-storage data loss — volume mounts corrected from
/backend/...to/backend/app/...(matching the app's path derivation), so images, FAISS index, and audio survive restarts.
7. Dependencies & licensing¶
- Clean for commercial use: Chatterbox TTS (MIT), open-clip + CLIP ViT-L/14 weights (MIT), FAISS (MIT), aiortc (BSD), OkHttp/Media3 (Apache-2.0), loguru (MIT). No insightface in this product — none of its non-commercial model-license baggage applies here (worth stating to buyers).
- Attention items:
faiss-gpu-cu12bundles NVIDIA CUDA runtime (EULA redistribution terms when shipping the image); distro ffmpeg may include GPL components (use an LGPL-only build to be safe — only libopus/BSD paths are actually used);pynvvideocodecis NVIDIA-proprietary and currently unused (remove from requirements to simplify the story); the TTS watermark is currently disabled via the Dummy fallback (a conscious decision to document). - Security hygiene before sale:
python-jose 3.3.0,passlib 1.7.4,python-multipart 0.0.9,starlette 0.38.6all predate known CVEs/fixes — runpip-auditand upgrade. RayNeo AARs (MercurySDK, RayNeoIPCSDK) are vendor binaries with no license files in the tree — clear redistribution rights in any acquisition.
8. Hardening roadmap & effort¶
| Phase | Work | Effort |
|---|---|---|
| 0 — Demo blockers | Rotate all credentials; new app keystore + secrets out of source; enable R8; commit/host AARs + launcher art with licenses | ~1 week |
| 1 — Security | Backend JWT validation + RBAC + fail-fast secrets; TLS + pinning on app signaling/REST; drop dev IPs & user CA trust; per-device tokens | 2–3 weeks |
| 2 — Correctness/robustness | Verify & repair WebRTC frame path; museum_id FAISS filter; gate /static audio; thread-pool CLIP + background TTS jobs; app reconnect state machine, camera events, teardown fix |
3–4 weeks |
| 3 — Maturity | pytest + JUnit suites, CI, pip-audit, dependency upgrades; FAISS rebuild-from-DB durability; DB-backed hashed API keys/RBAC; museum selection in app; thermal-adaptive capture ladder |
3–5 weeks |
| 4 — Ops polish | Wire declared Prometheus metrics / real GPU exporter; remove dead modules; gate /docs in prod; README + protocol docs |
ongoing |
Total to production-grade: ~8–14 engineer-weeks across both repos.
9. What a buyer is actually acquiring¶
- A working, differentiated product with proven multilingual hands-free museum UX and an on-prem, data-sovereign architecture that fits GCC procurement.
- A clean, readable codebase a competent team can own within days, on a modern toolchain.
- Defined, bounded technical debt with a costed remediation plan — not open-ended risk.
- Genuine IP in the dwell-recognition UX and the Arabic-aware content/approval pipeline.