Migration Tests and Data Tests: Safe State Changes
- date
- category
- Testing
- also in
- Databases · Reliability Engineering
- reading
- 2 min / 445 words
Code is usually easier to roll back than data.
You can switch a container image.
You can turn a feature flag off.
You can return to the previous commit.
But if the new version wrote data in a format the old version cannot understand, rollback stops being simple.
That is where migration tests and data tests belong.
What kind of test this is
Migration test checks a schema or data migration.
It can cover:
adding a column
changing a type
new index
backfill
table split
constraint change
Data compatibility test checks whether different code versions understand the data they may meet in the system.
The key questions are:
can new code read old data?
can old code read new data?
can the migration run twice?
does rollback leave data in a readable state?
Expand and contract
A safe data change often moves in several steps.
add the new field
write old and new field
move reads to the new field
clean up the old field
remove the old field
This approach is often called expand and contract.
Migration tests should protect each step.
If we remove the old field immediately, the old application version may break before the new version is everywhere.
If we start writing only the new format, an older worker may stop reading the queue.
What happens when they are missing
Without migration tests, the pipeline can allow a change that cannot be rolled back.
Examples:
migration works only on an empty database
backfill takes an hour instead of a minute
NOT NULL arrives before data is filled
rollback removes data that cannot be rebuilt
new version writes an event old consumers cannot read
These are not only SQL problems.
They are problems of operating a system during change.
Test on real state
A migration test is most valuable when it starts from realistic state.
It does not need to copy production.
It should contain cases production really has:
missing values
old records
large tables
duplicates
inactive accounts
rare statuses
A migration tested only on perfect fixtures can pass locally and fail on the first old record.
Data as a contract
Data is a contract between code versions.
Not only between services.
Also between:
old and new application version
application and worker
backend and reports
event and consumer
backup and restore
When we change data format, we change a contract.
The test should say which code version can read which format.
Without that, rollback is hope, not a plan.
Tool from the stack
For migration tests I would choose Testcontainers with real PostgreSQL.
The test should run migrations through the same mechanism the application uses, then check data through normal code or SQL.
The minimal setup:
PostgreSQL in a container
project migrations
fixture with old state
run migration
assertions on new state
On a Node backend this can run through Vitest, because the runner is not the important part.
The important part is that the test uses a real database and the real migration.
When the test is too late
A migration test run only after deployment is too late.
It can confirm the environment is alive, but it should not be the first place where we check the migration.
First, the pipeline needs a test.
Then a post-deployment smoke test can check that the application sees the new schema.
Those are two different questions.
The first is: is the migration correct.
The second is: is the deployed environment usable after migration.
The next layer concerns a system that is functionally correct but cannot handle traffic.