Error handling when writing to a file in Go

2023-03-03

Writing content to a file on disk can be a common operation in Go programs. There are many different ways to do it, but one often used is to open a file in write mode and write to it. This sounds trivial, but there are a few ways that Go programmers (beginner and experienced alike) can get this wrong.

An inexperienced Go programmer may initially attempt something like this:

func writeFile(filename string) error {
    f, err := os.Create(filename)
    if err != nil {
        return err
    }
    defer f.Close()

    buf, err := getBytes()
    if err != nil {
        return err
    }

    if _, err := f.Write(buf); err != nil {
        return err
    }
    return nil
}

At first glance, this seems sensible. One of the first things Go programmers learn is that it’s a good idea to defer close operations on resources after creating them so that they’re automatically closed when the function exits.

The problem is that f.Close() can return an error. Data that has been written via the Write method but has not yet persisted to disk may fail to be persisted during the close operation and cause an error to be returned. If this happens, it will go unnoticed.

This is an insidious bug to reproduce because disk write failures are rare!

A pattern that more intermediate Go programmers may attempt is to check the error from f.Close() by making the call after all of the f.Write(buf) calls rather than deferring it. This looks like:

 func writeFile(filename string) error {
     f, err := os.Create(filename)
     if err != nil {
         return err
     }
-    defer f.Close()
 
     buf, err := getBytes()
     if err != nil {
         return err
     }
 
     if _, err := f.Write(buf); err != nil {
         return err
     }
+    return f.Close()
-    return nil
 }

This has a problem as well though! If getBytes() or f.Write fails, the OS-level file descriptor will leak.

You can only have so many file descriptions open simultaneously. If no free file descriptors are available, then os.Create will fail. Depending on how the program is deployed, this may affect just the program itself or other unrelated processes as well.

The solution is to add back the deferred close after file creation and also keep the f.Close() call at the end of the function.

 func writeFile(filename string) error {
     f, err := os.Create(filename)
     if err != nil {
         return err
     }
+    defer f.Close()
 
     buf, err := getBytes()
     if err != nil {
         return err
     }
 
     if _, err := f.Write(buf); err != nil {
         return err
     }
     return f.Close()
 }

If the file is successfully closed by the final return f.Close(), then the deferred f.Close() operation does nothing. The deferred f.Close() will return an error because the file is already closed, but this has no impact on the program since the error is not checked.

If getBytes() or f.Write(buf) fail, then the file will be cleaned up by the deferred f.Close() call.

The file descriptor leak is gone, and any error occurring while persisting the file to disk will be detected.

This is enough for most use-cases. That’s right, you can stop here most of the time.

For some niche use cases though, it’s still not watertight enough. When the os.File is closed, its contents may not flush to disk immediately. If the physical machine unexpectedly loses power, the file’s contents may not be there when the machine restarts. This matters when you’re doing things like building your own database because data persistence behaviour under crash scenarios is critical. It can be remedied with the Sync method:

 func writeFile(filename string) error {
     f, err := os.Create(filename)
     if err != nil {
         return err
     }
     defer f.Close()
 
     buf, err := getBytes()
     if err != nil {
         return err
     }
 
     if _, err := f.Write(buf); err != nil {
         return err
     }
+    if err := f.Sync(); err != nil {
+        return err
+    }
     return f.Close()
 }

You might think this is enough to ensure the file contents are now persisted to disk. Most of the time it is, but it may ultimately depend on the behaviour of the physical disk.