Testing alerting in Google Cloud Monitoring requires a proactive approach, especially with uncommon scenarios. For assertions that almost always pass—such as DNS record verification or health checks—you can simulate failures using synthetic log injection.

The Alerting Signal: assertion_passed: false

At the heart of this strategy is a convention: services perform their internal checks and log a JSON payload with an assertion_passed boolean. When a check fails—be it a DNS mismatch or a stale cache—the service logs {"assertion_passed": false}.

Instrumentation is a critical part of any application. Along with system counters like cpu, heap, free disk, etc– it’s important to create application-level metrics to make sure health is measured closer to your customer’s experience.

Example metrics could be user-registration, password-change, profile-change, etc. If you see a major spike or dip in these metrics, a wider problem could be indicated.

For this example a custom metric was needed, and no infrastructure was in place for harvesting it (e.g. collectd). Golang is handy for creating an easy-to-install daemon which performs the measurement and periodically harvests the data into stackdriver.