Challenge
LuxiEdge is a deterministic compute engine with strong theoretical performance claims but no independent validation to support them. The product had not undergone structured performance, load, or determinism testing by the time we came on board, and there was no verified data demonstrating how it behaved under sustained GPU load or concurrent access.
The client needed an objective, third-party assessment that could confirm throughput, stability, and result reproducibility within a limited cooperation period. With no access to internal implementation details and a short, one-week testing run, the challenge was to design meaningful tests, define measurable benchmarks, and produce publishable evidence that accurately reflected real system behavior under load.
Key challenges included:
- Short engagement window. All testing activities needed to be completed and documented within a one-week project timeframe.
- No independent benchmarks. The product lacked third-party performance and stability validation to support technical or commercial discussions.
- Unproven determinism. Result reproducibility across repeated runs and execution modes had not been formally verified.
- Unknown load behavior. System performance under sustained concurrency had not been measured or documented.
- Limited observability. Testing had to be performed as a black-box API without access to internal implementation details.
Solutions
TestFort designed and executed a focused testing strategy to independently assess LuxiEdge’s performance, stability, and determinism within a one-week engagement. Given the early stage of the product and the lack of prior testing, the approach prioritized measurable, reproducible benchmarks that could demonstrate real system behavior under sustained GPU load rather than synthetic or short-run checks.
All testing was performed as black-box API validation, without access to internal implementation details. The team worked closely with the client to define meaningful metrics, select representative mathematical workloads, and validate both raw GPU kernel performance and API-level behavior. Special attention was given to deterministic execution, ensuring that identical inputs consistently produced bit-exact outputs across repeated runs and execution modes.
Key solutions included:
- Publishable reporting. Delivered a structured test report with transparent methodology and verifiable results suitable for external use.
- Sustained GPU load testing. Executed limited-window endurance tests with 200 concurrent users to observe long-running stability and throughput behavior.
- Raw kernel performance benchmarking. Measured throughput and latency of core mathematical functions using large vector workloads.
- API-level performance validation. Assessed request handling, concurrency, and latency distribution under continuous load.
- Determinism verification. Validated bit-exact output reproducibility using SHA-256 hashing across repeated GPU and CPU executions.
- Cross-mode parity testing. Confirmed identical results between GPU-accelerated and CPU execution modes.
- Resource and efficiency monitoring. Collected GPU power and utilization metrics to quantify performance-per-watt characteristics.
Technologies
The technologies used on this project were chosen for precise measurement of the API, GPU performance, deterministic behavior, and system stability under sustained load. The selected tools supported reliable benchmarking, low-level metric collection, and transparent result validation, allowing QA engineers to evaluate both raw computing efficiency and API-level behavior with confidence.
- JMeter
- K6
- AWS Infrastructure
- Python
