Learn · When apply fails

Half an apply, and what to do about it

Some resources created, one failed, the rest never attempted. The most instructive situation in the subject, and the one every tutorial assumes away.

Three kinds of thing, afterwards

When an apply dies part-way you are left with three categories, and telling them apart is most of the work. There are resources that *finished* — they are in the cloud and in state, and they are fine. There is the one that *failed*, which is in an unknown condition: the tool does not know how far the provider got. And there is everything that was *never attempted*, which is untouched and still describes the old world.

The state file records all three. The one that failed is marked, and the next plan proposes replacing it — not because replacing is right, but because nobody knows what is there.

The tail is the confusing part. Four things changed, one broke, and eight are exactly as they were — so the configuration, the state file and the cloud now disagree in three different ways at once.

A plan is not a script

The instinct is that the apply has to be resumed — that there is a queue somewhere with the rest of the work in it. There is not. The next plan is *recomputed from what is actually there*, which means it is usually smaller than the first one: the resources that succeeded are already correct and do not appear in it.

That is why re-running is so often the right answer, and why it is safe in a way that re-running a migration script is not. It is also why reading the second plan matters as much as the first: it will not be the same plan.

Try this

-     a = 1
+     a = 11

Watch: The subnet, and then the workers sitting in it. That second group is what a stopped apply leaves behind.

The taint-and-reapply reflex, and when it is wrong

taint marks a resource for replacement with no configuration change. It is the usual response to something being in a strange state, and it is right when the resource is cheap and stateless — a machine that will not converge, a node that is half-registered.

It is wrong when the resource holds something. Tainting a database replaces it, and a replacement destroys the data unless a snapshot happens to exist. The half-applied marker looks like a problem to clear, and clearing it that way costs more than the problem did.

The alternative is always the same and always less satisfying: read the state file, work out which half happened, and decide.

The reflex is fast and usually harmless, which is exactly what makes it dangerous on the one resource where it is not.

The plan

The same service every claim above was made about. Nothing here is graded — load whatever you like, or change nothing and read what the configuration already produces.

No change loaded — the configuration as it stands.

Plan: 0 to add, 0 to change, 0 to destroy.

Nothing in this plan is destroyed.

Read a plan