Post-commit tests policies
Post-commit tests validate that Unomi works correctly in a live environment. The tests also catch errors that are hard to predict in the design and implementation stages
Even though post-commit tests run after the code is merged into the repository, it is important that the tests pass reliably. Jenkins executes post-commit tests against the HEAD of the master branch. If post-commit tests fail, there is a problem with the HEAD build. In addition, post-commit tests are time consuming to run, and it is often hard to triage test failures.
Policies
To ensure that Unomi's post-commit tests are reliable and healthy, the Unomi community follows these post-commit test policies:
- Rollback first
- A failing test is a critical bug
- A flaky test is a critical bug
- Flaky tests must either be fixed or removed
- Fixes for post-commit failures should include a corresponding new pre-commit test
Post-commit test failure scenarios
When a post-commit test fails, follow the provided steps for your situation.
I found a test failure
- Create a JIRA issue and assign it to yourself.
- Do high level triage of the failure.
- Assign the JIRA issue to a relevant person.
I was assigned a JIRA issue for a test failure
- Rollback the culprit change.
- If you determine that rollback will take longer than 8 hours, disable the test temporarily while you rollback or create a fix.
Note: Rollback is always the first course of action. If a fix is trivial, open a pull request with the proposed fix while doing rollback.
My change was rolled back due to a test failure
- Look at the JIRA issue to find the reason for the rollback.
- Fix your code and re-run the post-commit tests.
- Implement new pre-commit tests that will catch similar bugs before future code is merged into the repository.
- Open a new PR that contains your fix and the new pre-commit tests.