snapshot: Ensure all snapshot files are durable

[kim]

When creating or compressing a snapshot, fsync all files and directories, so as to ensure that the snapshot is durable on the local disk.

This obviously amounts to a large number of fsync calls, which may negatively impact performance of taking a snapshot -- since we hold a transaction lock while taking a snapshot, this is not to be taken lightly.

Expected complexity level and risk

3 -- performance impact

Testing

I haven't quantified the performance impact.

[gefjon]

Is it necessary for replication to fsync before dropping the transaction lock? We were originally treating snapshots as an unreliable optimization, but I gather that may no longer be the case. The reason I ask is to suggest that we might write the files with the transaction lock held, then release it, and only then call fsync.

[kim]

to suggest that we might write the files with the transaction lock held, then release it, and only then call fsync.

Yeah, could try that.

[gefjon]

Looks good. If you'd like to try the version that fsyncs after releasing the lock, it might be worth trying, but if you just wanna merge this version, that's fine by me.

[kim]

I've changed this to defer syncing until after the datastore lock is released.
This is slightly unpleasant, because it implies that we also defer writing the snapshot file until the object trie is durable.
Added a metric to track how long the sync step takes.

Let me know what you think @gefjon

[gefjon]

Thanks for doing that. The code is slightly more complicated, but we can now be much more confident this won't negatively affect performance.