A cross-platform end-to-end test framework on real devices

An ad-mediation SDK only earns trust if it behaves correctly on real hardware, across platforms, ad formats, and a shifting set of network adapters. Manual QA for that matrix doesn't scale and isn't repeatable. I designed and built a cross-platform end-to-end automation framework that runs the real SDK on real devices and produces a signal I'm willing to gate a release on.

What makes it interesting is that it's bespoke by necessity. In a Unity title, the ad SDK and its native plugins take over Unity's focus and control — fullscreen ads, banners, and consent flows render on native surfaces that live outside the engine. Unity's own automation can't see or drive them, so off-the-shelf in-engine testing can't validate the integration at all. This harness had to operate at the device / OS level, exercising the real app exactly as players — and those native plugins — actually behave.

The problem

• The Unity ↔ native-plugin gap — ad surfaces seize Unity's focus and input, so in-engine test hooks are blind to the very behavior under test; automation has to drive the device, not the engine.
• A large behavior matrix: platforms × ad formats (fullscreen, banner, rewarded) × adapters × callbacks.
• Manual runs were slow, non-repeatable, and flaky — failures were hard to attribute.
• No single place to see "is the SDK healthy right now?" before shipping.

The pipeline

Tests are authored once as Maestro flows — device-level UI automation that drives the real OS, not Unity internals, which is exactly what lets it reach the native plugin surfaces — and driven through a generation pipeline that expands them across platforms and configurations. They execute on BrowserStack real devices in CI, and results flow into a single reporting surface: TestRail as the source of truth for cases and Allure for a unified, trend-aware report, with screenshots archived to S3 and a distilled summary pinged to Slack.

flowchart TB
  DEF[Test definitions
single source] --> GEN[Generate-flows
pipeline]
  GEN --> MAE[Maestro flows]
  MAE --> BS[BrowserStack
real devices]
  BS --> RES[Results + screenshots]
  RES --> TR[(TestRail
case status)]
  RES --> AL[Allure unified
report + trends]
  RES --> S3[(S3 screenshot
archive)]
  AL --> SLK[Slack summary]
  TR --> SLK
    

Making the signal trustworthy

• Single-source authoring. Cases are written once and synced to TestRail, so the report and the test definitions never drift.
• Assertion depth. Flows assert real behavior — click-through callbacks, rewarded rewards, banner visibility and placement, session/observability markers — not just "did the app launch".
• Flake control. A quarantine lane isolates unstable tests so they inform without blocking; stabilized tests graduate back into the gate.
• Right-sized CI. PR-smoke runs just the changed tests; nightly runs the full matrix and publishes the unified report.

Wired into the release gate

The framework isn't a side project — it sits in the same CI as the team's AI-assisted review tooling, so correctness and code-quality signal arrive together, before merge and before release.