Proper fsync on OS X

[lazka]

I think this is missing

if hasattr(fcntl, "F_FULLFSYNC"):
    fcntl.fcntl(f.fileno(), fcntl.F_FULLFSYNC)
else:
    os.fsync(f.fileno())

on OSX.

As mentioned here [0] and here [1] fsync doesn't wait for the changes to hit the disk there.

[0] https://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html
[1] https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fsync.2.html#//apple_ref/doc/man/2/fsync

[untitaker]

While safer, I'm not sure this is desirable behavior on e.g. SSDs (where small
writes are damaging).

Overall fsync provides the amount of "safety" I've come to expect. I haven't
seen any other impl do anything special for OS X and therefore assumed that my
level of "safety" is sufficient.

That said: We probably want this, but with a flag to turn it off in
performance-critical situations.

On Wed, Dec 02, 2015 at 04:51:37AM -0800, Christoph Reiter wrote:

I think this is missing

if hasattr(fcntl, "F_FULLFSYNC"):
    fcntl.fcntl(f.fileno(), fcntl.F_FULLFSYNC)
else:
    os.fsync(f.fileno())

on OSX.

As mentioned here [0] and here [1] fsync doesn't wait for the changes to hit the disk there.

[0] https://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html
[1] https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fsync.2.html#//apple_ref/doc/man/2/fsync

---

Reply to this email directly or view it on GitHub:
#6

[untitaker]

@Unrud Do you have any opinion on this?

[untitaker]

After re-reading this I'm not sure if it's always expected that we block for the data to hit the disk. As long as it doesn't linger in the cache for too long (> 1sec, for example), it should be fine, no?

[Unrud]

I would use F_FULLFSYNC on OS X. If fsync doesn't ensure that data is written into permanent storage (and the on-disk cache doesn't have battery backup), the operation is neither atomic nor durable.

As long as it doesn't linger in the cache for too long (> 1sec, for example)

But there are no guarantees for this. The data can linger in the cache for an arbitrary amount of time.

The same problem seems to exist on Windows.

[untitaker]

It seems my reading comprehension is getting worse. Will implement as-is.