File reading is one of the most common operations performed in any programming language. In this tutorial we will learn about how files can be read using Go.

This tutorial has the following sections.

• Reading an entire file into memory
  • Using absolute file path
  • Passing the file path as a command line flag
  • Bundling the file inside the binary
• Reading a file in small chunks
• Reading a file line by line

Reading an entire file into memory

One of the most basic file operations is reading an entire file into memory. This is done with the help of the ReadFile function of the ioutil package.

Let's read a file from the directory where our go program is located. I have created a folder filehandling

src  
    filehandling
                filehandling.go
                test.txt
 

Let's get to the code right away.

package main

import (  
    "fmt"
    "io/ioutil"
)

func main() {  
    data, err := ioutil.ReadFile("test.txt")
    if err != nil {
        fmt.Println("File reading error", err)
        return
    }
    fmt.Println("Contents of file:", string(data))
}

Please run this program from your local environment as it's not possible to read files in the playground.

Line no. 9 of the program above reads the file and returns a byte slice which is stored in data. In line no. 14 we convert data to a string and display the contents of the file.

Please run this program from the location where test.txt is present.

For example in the case of linux/mac, if test.txt is located at home/naveen/go/src/filehandling, then run this program using the following steps,

$]cd /home/naveen/go/src/filehandling/
$]go install filehandling
$]workspacepath/bin/filehandling

In the case of windows, if test.txt is located at C:\Users\naveen.r\go\src\filehandling, then run this program using the following steps,

> cd C:\Users\naveen.r\go\src\filehandling
> go install filehandling
> workspacepath\bin\filehandling.exe 

This program will output,

Contents of file: Hello World. Welcome to file handling in Go.  

If this program is run from any other location, for instance try running the program from /home/userdirectory, it will print the following error.

File reading error open test.txt: The system cannot find the file specified.  

The reason is Go is a compiled language. What go install does is, it creates a binary from the source code. The binary is independent of the source code and it can be run from any location. Since test.txt is not found in the location from which the binary is run, the program complains that it cannot find the file specified.

There are three ways to approach this problem,

1. Using absolute file path
2. Passing the file path as a command line flag
3. Bundling the text file along with the binary

Let's discuss them one by one.

1. Using absolute file path

The simplest way to solve this problem is to pass the absolute file path. I have modified the program and changed the path to an absolute one.

package main

import (  
    "fmt"
    "io/ioutil"
)

func main() {  
    data, err := ioutil.ReadFile("/home/naveen/go/src/filehandling/test.txt")
    if err != nil {
        fmt.Println("File reading error", err)
        return
    }
    fmt.Println("Contents of file:", string(data))
}

Now the program can be run from any location and it will print the contents of test.txt.

For example, it will work even when I run it from my home directory

$]cd $HOME
$]go install filehandling
$]workspacepath/bin/filehandling

The program will print the contents of test.txt

This seems to be an easy way but comes with the pitfall that the file should be located in the path specified in the program else this method will fail.

2. Passing the file path as a command line flag

Another way to solve this problem is to pass the file path as a command line flag. Using the flag package, we can get the file path as input from the command line and then read its contents.

Let's first understand how the flag package works. The flag package has a String function. This function accepts 3 arguments. The first is the name of the flag, second is the default value and the third is a short description of the flag.

Let's write a small program to read the file name from the command line. Replace the contents of filehandling.go with the following,

package main  
import (  
    "flag"
    "fmt"
)

func main() {  
    fptr := flag.String("fpath", "test.txt", "file path to read from")
    flag.Parse()
    fmt.Println("value of fpath is", *fptr)
}

Line no. 8 of the program above, creates a string flag named fpath with default value test.txt and description file path to read from using the String function. This function returns the address of the string variable that stores the value of the flag.

flag.Parse() should be called before any flag is accessed by the program.

We print the value of the flag in line no. 10

When this program is run using using the command

wrkspacepath/bin/filehandling -fpath=/path-of-file/test.txt  

we pass /path-of-file/test.txt as the value of the flag fpath.

This program outputs

value of fpath is /path-of-file/test.txt  

If the program is run using just filehandling without passing any fpath, it will print

value of fpath is test.txt  

since test.txt is the default value of fpath.

Now that we know how to read the file path from the command line, let's go ahead and finish our file reading program.

package main  
import (  
    "flag"
    "fmt"
    "io/ioutil"
)

func main() {  
    fptr := flag.String("fpath", "test.txt", "file path to read from")
    flag.Parse()
    data, err := ioutil.ReadFile(*fptr)
    if err != nil {
        fmt.Println("File reading error", err)
        return
    }
    fmt.Println("Contents of file:", string(data))
}

The program above reads the content of the file path passed from the command line. Run this program using the command

wrkspacepath/bin/filehandling -fpath=/path-of-file/test.txt  

Please replace /path-of-file/ with the actual path of test.txt. The program will print

Contents of file: Hello World. Welcome to file handling in Go.  

3. Bundling the text file along with the binary

The above option of getting the file path from command line is good but there is an even better way to solve this problem. Wouldn't it be awesome if we are able to bundle the text file along with our binary. This is what we are going to do next.

There are various packages that help us achieve this. We will be using packr because it's quite simple and I have been using it for my projects without any problems.

The first step is to install the packr package.

Type the following command in the command prompt to install the package

go get -u github.com/gobuffalo/packr/...  

packr converts static files such as .txt to .go files which are then embedded directly into the binary. Packer is intelligent enough to fetch the static files from disk rather than from the binary during development. This prevents the need for re-compilation during development when only static files change.

A program will make us understand things better. Replace the contents of filehandling.go with the following,

package main

import (  
    "fmt"

    "github.com/gobuffalo/packr"
)

func main() {  
    box := packr.NewBox("../filehandling")
    data := box.String("test.txt")
    fmt.Println("Contents of file:", data)
}

In line no. 10 of the program above, we are creating a New Box. A box represents a folder whose contents will be embedded to the binary. In this case I am specifying the filehandling folder which contains test.txt. In the next line we read the contents of the file and print it.

When we are in development phase, we can use the go install command to run this program. It will work as expected. packr is intelligent enough to load the file from disk during development phase.

Run the program using the following commands.

go install filehandling  
workspacepath/bin/filehandling  

These commands can be run from any location. Packr is intelligent enough to get the absolute path of the directory passed to the NewBox command.

This program will print

Contents of file: Hello World. Welcome to file handling in Go.  

Try changing the contents of test.txt and run filehandling again. You can see that the program prints the updated contents of test.txt without the need for any recompilation. Perfect :).

Now let's move to the step and bundle test.txt to our binary. We use the packr command to do this.

Run the following command

packr install -v filehandling  

This will print

building box ../filehandling  
packing file filehandling.go  
packed file filehandling.go  
packing file test.txt  
packed file test.txt  
built box ../filehandling with ["filehandling.go" "test.txt"]  
filehandling  

This command bundles the static file along with the binary.

After running the above command, run the program using the command workspacepath/bin/filehandling. The program will print the contents of test.txt. Now test.txt is being read from the binary.

If you doubt whether the file is served from within the binary or from disk, I suggest that you delete test.txt and run the command filehandling again. You can see that test.txt's contents are printed. Awesome :D We have successfully embedded static files to our binary.

Reading a file in small chunks

In the last section, we learnt how to load an entire file into memory. When the size of the file is extremely large it doesn't make sense to read the entire file into memory especially if you are running low on RAM. A more optimal way is to read the file in small chunks. This can be done with the help of the bufio package.

Let's write a program that reads our test.txt file in chunks of 3 bytes. Replace the contents of filehandling.go with the following,

package main

import (  
    "bufio"
    "flag"
    "fmt"
    "log"
    "os"
)

func main() {  
    fptr := flag.String("fpath", "test.txt", "file path to read from")
    flag.Parse()

    f, err := os.Open(*fptr)
    if err != nil {
        log.Fatal(err)
    }
    defer func() {
        if err = f.Close(); err != nil {
            log.Fatal(err)
        }
    }()
    r := bufio.NewReader(f)
    b := make([]byte, 3)
    for {
        _, err := r.Read(b)
        if err != nil {
            fmt.Println("Error reading file:", err)
            break
        }
        fmt.Println(string(b))
    }
}

In line no. 15 of the program above, we open the file using the path passed from the command line flag.

In line no. 19, we defer the file closing.

Line no. 24 of the program above creates a new buffered reader. In the next line, we create a byte slice of length and capacity 3 into which the bytes of the file will be read.

The Read method in line no. 27 reads up to len(b) bytes i.e up to 3 bytes and returns the number of bytes read. Once the end of file is reached, it will return a EOF error. The rest of the program is straight forward.

If we run the program above using the commands,

$]go install filehandling
$]wrkspacepath/bin/filehandling -fpath=/path-of-file/test.txt

the following will be output

Hel  
lo  
Wor  
ld.  
 We
lco  
me  
to  
fil  
e h  
and  
lin  
g i  
n G  
o.  
Error reading file: EOF  

Reading a file line by line

In the section we will discuss how to read a file line by line using Go. This can done using the bufio package.

Please replace the contents in test.txt with the following

Hello World. Welcome to file handling in Go.  
This is the second line of the file.  
We have reached the end of the file.  

The following are the steps involved in reading a file line by line.

1. Open the file
2. Create a new scanner from the file
3. Scan the file and read it line by line.

Replace the contents of filehandling.go with the following

package main

import (  
    "bufio"
    "flag"
    "fmt"
    "log"
    "os"
)

func main() {  
    fptr := flag.String("fpath", "test.txt", "file path to read from")
    flag.Parse()

    f, err := os.Open(*fptr)
    if err != nil {
        log.Fatal(err)
    }
    defer func() {
        if err = f.Close(); err != nil {
        log.Fatal(err)
    }
    }()
    s := bufio.NewScanner(f)
    for s.Scan() {
        fmt.Println(s.Text())
    }
    err = s.Err()
    if err != nil {
        log.Fatal(err)
    }
}

In line no. 15 of the program above, we open the file using the path passed from the command line flag. In line no. 24, we create a new scanner using the file. The scan() method in line no. 25 reads the next line of the file which will be available through the text() method.

After Scan returns false, the Err() method will return any error that occurred during scanning, except that if it was End of File, Err() will return nil.

If we run the program above using the commands,

$] go install filehandling
$] workspacepath/bin/filehandling -fpath=/path-of-file/test.txt

It will output

Hello World. Welcome to file handling in Go.  
This is the second line of the file.  
We have reached the end of the file.  

This brings us to an end of this tutorial. Hope you enjoyed it. Have a good day.

Like my tutorials? Please show your support by donating. Your donations will help me create more awesome tutorials.