Intermittent Test Failures

"X test fails intermittently on Solano. This isn't a problem locally, so what's going on?"

Unfortunately, unexpected failures sometimes occur within Solano CI. There are two main reasons: Solano CI’s parallelism, and your test’s interaction with asynchronous behavior; below are some frequent offenders related to each reason. (For more fleshed out descriptions, please see our documentation.) 

Parallelism-related Failures:

  • Solano CI Environment Variable
  • Hardcoded Ports
  • Hardcoded Paths
  • File Upload Tools
  • FileStore Caching
  • OpenID (e.g., with Omniauth)
  • Analytics Agents (e.g. Kissmetrics)

Other Causes of Failures:

  • Timeouts
  • Xvfb
  • Selenium and Asynchronous Behavior
  • Disabling Garbage Collection

To properly identify the cause of the failures, we recommend three main troubleshooting steps. 

  1. Isolate the test
  2. Check for dependency issues
  3. Check for concurrency issues

You can learn more about these steps and more from our troubleshooting guide.

Have more questions? Submit a request


Please sign in to leave a comment.
Powered by Zendesk