File Input and Output in Go

File input and file output are primary tasks of any operating system.

The main purpose of this topic is to teach how the Go standard library permits us to open files, read their contents, process them if we like, create new files, and put the desired data into them.

two main ways:

• read and write files.
• using the io package and using the functions of the bufio package.

This topic will tell you about the following:

• Opening files for writing and reading
• Using the io package for file input and output.
• Using the bufio package for buffered input and output
• Copying files in Go
• Implementing a version of the wc(1) utility in Go.
• Developing a version of the dd(1) command in Go
• Create sparse files.
• byte slices
• Storing structured data in files and reading them afterwards
• Converting tab into space characters and vice versa.

About file input and output

File input and output includes everything that has to do with reading the data of a file and writing the desired data to a file.

The first thing that you will learn in this chapter is byte slices, which are very important in applications that are concerned with file input and output.

Byte slices

Byte slices are a kind of slices used for file reading and writing.

A byte slices is used for writing to a file and reading from a file.

[maxwell@MaxwellDBA fileinputandoutput]$ go run byteSlice.go usingByteSlices
Read 19 bytes: Mihalis Tsoukalos!
[maxwell@MaxwellDBA fileinputandoutput]$ wc usingByteSlices
 1  2 19 usingByteSlices
[maxwell@MaxwellDBA fileinputandoutput]$ ls -ltr
total 12
-rw-rw-r-- 1 maxwell maxwell 524 Dec 16 11:19 byteSlice.go
-rw-rw-r-- 1 maxwell maxwell  19 Dec 16 11:21 test
-rw-r--r-- 1 maxwell maxwell  19 Dec 16 13:41 usingByteSlices
[maxwell@MaxwellDBA fileinputandoutput]$ cat byteSlice.go
package main

import (
   "fmt"
   "io/ioutil"
   "os"
)

func main(){
   if len(os.Args) != 2 {
         fmt.Println("Please provide a filename")
         os.Exit(1)
   }
   filename := os.Args[1]

   aByteSlice := []byte("Mihalis Tsoukalos!\n")
   ioutil.WriteFile(filename, aByteSlice, 0644)
   f, err := os.Open(filename)
   if err != nil {
        fmt.Println(err)
        os.Exit(1)
   }
   defer f.Close()

   anotherByteSlice := make([]byte, 100)
   n, err := f.Read(anotherByteSlice)
   fmt.Printf("Read %d bytes: %s", n, anotherByteSlice)
}

[maxwell@MaxwellDBA fileinputandoutput]$ cat usingByteSlices
Mihalis Tsoukalos!
[maxwell@MaxwellDBA fileinputandoutput]$ 

About binary files

Go offers a package named binary that allows you to make translations between different encodings such as little endian and big endian.

The readBinary.go file breifly illustrates how to convert an integer number to a little endian number and to a big endian number, which might be useful when the files you want to process contain certain kinds of data.

[maxwell@MaxwellDBA fileinputandoutput]$ vim readBinary.go
[maxwell@MaxwellDBA fileinputandoutput]$ 
[maxwell@MaxwellDBA fileinputandoutput]$ 
[maxwell@MaxwellDBA fileinputandoutput]$ go run readBinary.go 1
1 is 0100000000000000 in Little Endian
And 0000000000000001 in Big Endian
[maxwell@MaxwellDBA fileinputandoutput]$ cat readBinary.go
package main

import (
   "bytes"
   "encoding/binary"
   "fmt"
   "os"
   "strconv"
)

func main() {
   if len(os.Args) != 2 {
         fmt.Println("Please provide an integer")
         os.Exit(1)
   }
   aNumber,_ := strconv.ParseInt(os.Args[1], 10, 64)

   buf := new(bytes.Buffer)
   err := binary.Write(buf, binary.LittleEndian, aNumber)
   if err != nil {
        fmt.Println("Little Endian:", err)
   }

   fmt.Printf("%d is %x in Little Endian\n", aNumber, buf)
   buf.Reset()
   err = binary.Write(buf, binary.BigEndian, aNumber)
   if err != nil {
        fmt.Println("Big Endian:", err)
   }
   fmt.Printf("And %x in Big Endian\n", buf)
}
[maxwell@MaxwellDBA fileinputandoutput]$ 

Useful I/O packages in Go

The io package is for performing primitive file I/O operations , whereas the bufio package is for executing buffered I/O.

In buffered I/O, the operating system uses an intermediate buffer during file read and write operations in order to reduce the number of filesystem calls. As a result, buffered input and output is faster and more efficient.

The io package

The io package offers functions that allow you to write to or read from files.

What the program does is read 8 bytes from a file and write them in a standard output.

[maxwell@MaxwellDBA fileinputandoutput]$ vim usingIO.go
[maxwell@MaxwellDBA fileinputandoutput]$ go run usingIO.go usingByteSlices
Mihalis 
[maxwell@MaxwellDBA fileinputandoutput]$ cat usingIO.go
package main

import (
   "fmt"
   "io"
   "os"
)

func main() {
   if len(os.Args) != 2 {
        fmt.Println("Please provide a filename")
        os.Exit(1)
   }

   filename := os.Args[1]
   f, err := os.Open(filename)
   if err != nil {
         fmt.Printf("error opening %s", filename, err)
         os.Exit(1)
   }
   defer f.Close()
   buf := make([]byte, 8)
   if _, err := io.ReadFull(f, buf); err != nil {
         if err == io.EOF {
              err = io.ErrUnexpectedEOF
        }
  }
  io.WriteString(os.Stdout, string(buf))
  fmt.Println()
}

[maxwell@MaxwellDBA fileinputandoutput]$ 

The bufio package

The functions of the bufio package allow you to perform buffered file operations, which means that although its operations looks similar to the ones found in io, they work in a slightly different way.

[maxwell@MaxwellDBA fileinputandoutput]$ go run bufIO.go inputFile | wc
      0       0       0
[maxwell@MaxwellDBA fileinputandoutput]$ wc inputFile/
wc: inputFile/: Is a directory
      0       0       0 inputFile/
[maxwell@MaxwellDBA fileinputandoutput]$ cat bufIO.go
package main

import (
   "bufio"
   "fmt"
   "os"
)


func main() {
   if len(os.Args) != 2 {
         fmt.Println("Please provide a filename")
         os.Exit(1)
    }

    filename := os.Args[1]

    f, err := os.Open(filename)
    if err != nil {
          fmt.Printf("error operating %s: %s", filename, err)
          os.Exit(1)
    }
    defer f.Close()

    scanner := bufio.NewScanner(f)

    for scanner.Scan() {
         line := scanner.Text()

         if scanner.Err() != nil {
               fmt.Printf("error reading file %s", err)
               os.Exit(1)
         }
         fmt.Println(line)
    }

}
[maxwell@MaxwellDBA fileinputandoutput]$ 

File I/O operations

Writing to files using fmt.Fprintf()

The fmt.Fprintf() function allows you to write formatted text to files in a way that is similar to the way the fmt.Printf() function works.

Note that fmt.Fprintf() can write to any io.Writer interface and that our files will satisfy the io.Writer interface.

[maxwell@MaxwellDBA fileinputandoutput]$ vim fmtF.go
[maxwell@MaxwellDBA fileinputandoutput]$ go run fmtF.go test
[maxwell@MaxwellDBA fileinputandoutput]$ cat test
[test]: Using fmt.Fprintf in test
[maxwell@MaxwellDBA fileinputandoutput]$ cat fmtF.go
package main

import (
   "fmt"
   "os"
)

func main(){
   if len(os.Args) != 2 {
        fmt.Println("Please provide a filename")
        os.Exit(1)
   }

   filename := os.Args[1]
   destination, err := os.Create(filename)
   if err != nil {
         fmt.Println("os.Create:", err)
         os.Exit(1)
   }
   defer destination.Close()
   fmt.Fprintf(destination, "[%s]: ", filename)
   fmt.Fprintf(destination,"Using fmt.Fprintf in %s\n", filename)
}
[maxwell@MaxwellDBA fileinputandoutput]$ 

About io.Writer and io.Reader

Both io.Writer and io.Reader are interfaces that embed the io.Write() and io.Read() methods, respectively.The use of io.Writer and io.Reader will be illustrated in readerWriter.go. The program computes to another file: if you are dealing with Unicode characters that take more than one byte per character, you might consider that the program is reading bytes.The output filename has the name of the original file plus the .count extension.

[maxwell@MaxwellDBA goproject]$ vim readerWriter.go                                     
[maxwell@MaxwellDBA goproject]$ go run readerWriter.go /home/maxwell/goproject/swtag.log
[maxwell@MaxwellDBA goproject]$ wc /home/maxwell/goproject/swtag.log
   6  190 1101 /home/maxwell/goproject/swtag.log
[maxwell@MaxwellDBA goproject]$
[maxwell@MaxwellDBA goproject]$ cat /home/maxwell/goproject/swtag.log.Count
1101
[maxwell@MaxwellDBA goproject]$ 
[maxwell@MaxwellDBA goproject]$ cat readerWriter.go
package main
import (
   "fmt"
   "os"
   "io"
)

func countChars(r io.Reader) int {
    buf := make([]byte, 16)
    total := 0
    for {
          n, err := r.Read(buf)
          if err != nil && err != io.EOF {
                return 0
           }
           if err == io.EOF {
                 break
           }
           total = total + n
    }
    return total
}

func writeNumberOfChars(w io.Writer, x int){
   fmt.Fprintf(w, "%d\n", x)
}



func main() {
    if len(os.Args) != 2 {
          fmt.Println("Please provide a filename")
          os.Exit(1)
    }

    filename := os.Args[1]
    _, err := os.Stat(filename)
    if err != nil {
           fmt.Printf("Error on file %s: %s\n", filename, err)
           os.Exit(1)
    }

    f, err := os.Open(filename)
    if err != nil {
           fmt.Println("Cannot open file:", err)
           os.Exit(-1)
    }
    defer f.Close()

    chars := countChars(f)
    filename = filename + ".Count"
    f, err = os.Create(filename)
    if err != nil {
          fmt.Println("os.Create:", err)
          os.Exit(1)
    }
    defer f.Close()
    writeNumberOfChars(f, chars)
}
[maxwell@MaxwellDBA goproject]$ 

Finding out the third column of a line

maxwell@MaxwellDBA goproject]$ go run readColumn.go -COL=3 pF.data isThereAFile up.data
                 pF.data
case,
code
                 isThereAFile
error opening file open isThereAFile: no such file or directory          up.data
error opening file open up.data: no such file or directory[maxwell@MaxwellDBA goproject]$ 
[maxwell@MaxwellDBA goproject]$ cat pF.data
In this case, there is no file named isThereAFile and the pF.data file does not have a third column. However, the program did its best and printed what it could!

The preceding code does not do anything that you have not seen before.The for loop is used for processing all command-line arguments. However,if a file fails to open for some reason, the program will not stop its execution, but it will continue processing the rest of the files if they exist.However, the program expects that its input files end in a newline and you might see strange results if an input file ends differently.
[maxwell@MaxwellDBA goproject]$ 
[maxwell@MaxwellDBA goproject]$ cat readColumn.go
package main

import (
  "bufio"
  "flag"
  "fmt"
  "io"
  "os"
  "strings"
)

func main() {
   minusCOL := flag.Int("COL", 1, "Column")
   flag.Parse()
   flags := flag.Args()

   if len(flags) == 0 {
          fmt.Printf("usage: readColumn <file1> [<file2> [... <fileN>]]\n")
          os.Exit(1)
   }
   column := *minusCOL

   if column < 0 {
         fmt.Println("Invalid Column number!")
         os.Exit(1)
   }

   for _, filename := range flags {
         fmt.Println("\t\t", filename)
         f, err := os.Open(filename)
         if err != nil {
                fmt.Printf("error opening file %s", err)
                continue
         }
         defer f.Close()

         r := bufio.NewReader(f)
         for {
              line, err := r.ReadString('\n')
              if err == io.EOF {
                    break
              } else if err != nil {
                    fmt.Printf("error reading file %s", err)
              }
              data := strings.Fields(line)
              if len(data) >= column {
                    fmt.Println((data[column-1]))
              }
        }

   }
}
[maxwell@MaxwellDBA goproject]$ 

Copying files in Go

This section will show you how to copy files in Go.

There is more than one way to copy a file!

The there is more than one way to do it rule applies to almost everything implemented in this book, but file copying is the most characteristic example of this rule because you can copy a file by reading it line by line, byte by byte, or all at once! However, this rule does not apply to the way Go likes to format its code!

 Using io.Copy

[maxwell@MaxwellDBA goproject]$ vim testFile
[maxwell@MaxwellDBA goproject]$ vim notGoodCP.go
[maxwell@MaxwellDBA goproject]$ go run notGoodCP.go testFile aCopy
Copied 6576 bytes!
[maxwell@MaxwellDBA goproject]$ diff aCopy testFile
[maxwell@MaxwellDBA goproject]$ wc testFile aCopy
   17   997  6576 testFile
   17   997  6576 aCopy
   34  1994 13152 total
[maxwell@MaxwellDBA goproject]$ cat notGoodCP.go
package main

import (
   "fmt"
   "io"
   "os"
)

func Copy(src, dst string)(int64, error){
   sourceFileStat, err := os.Stat(src)
   if err != nil {
         return 0, err
   }
   if !sourceFileStat.Mode().IsRegular() {
         return 0, fmt.Errorf("%s is not a regular file", src)
   }

   source, err := os.Open(src)
   if err != nil {
        return 0, err 
   }
   defer source.Close()

   destination, err := os.Create(dst)
   if err != nil {
         return 0, err
   }

   defer destination.Close()
   nBytes, err := io.Copy(destination, source)
   return nBytes, err
   }

func main() {
   if len(os.Args)!= 3 {
         fmt.Println("Please provide two command line arguments!")
         os.Exit(1)
   }

   sourceFile := os.Args[1]
   destinationFile := os.Args[2]
   nBytes, err := Copy(sourceFile, destinationFile)

   if err != nil {
      fmt.Printf("The copy operation failed %q\n", err)
   } else {
        fmt.Printf("Copied %d bytes!\n",nBytes)
   }
}
[maxwell@MaxwellDBA goproject]$ 

The best tool for testing whether a file is an exact copy of another file is the diff(1) utility, which also works with binary files. You can learn more about diff(1) by reading its main page.

Reading a file all at once!

we will use the ioutil.WriteFile() and ioutil.ReadFile() functions.Note that ioutil.ReadFile() does not implement the io.Reader interface and therefore is a little restrictive.

[maxwell@MaxwellDBA goproject]$ cat readAll.go 
package main

import (
   "fmt"
   "io/ioutil"
   "os"
)

func main() {
   if len(os.Args) != 3 {
          fmt.Println("Please provide two command line arguments!")
          os.Exit(1)
   }

   sourceFile := os.Args[1]
   destinationFile := os.Args[2]

   input, err := ioutil.ReadFile(sourceFile)

   if err != nil {
          fmt.Println(err)
          os.Exit(1)
   }

   err = ioutil.WriteFile(destinationFile, input, 0644)
   if err != nil {
         fmt.Println("Error creating the new file", destinationFile)
         fmt.Println(err)
         os.Exit(1)
   }
}
[maxwell@MaxwellDBA goproject]$ go run readAll.go testFile bCopy
[maxwell@MaxwellDBA goproject]$ diff bCopy testFile
[maxwell@MaxwellDBA goproject]$ ls -l testFile bCopy
-rw-r--r-- 1 maxwell maxwell 6576 Dec 18 13:07 bCopy
-rw-rw-r-- 1 maxwell maxwell 6576 Dec 18 11:35 testFile
[maxwell@MaxwellDBA goproject]$ go run readAll.go doesNotExist bCopy
open doesNotExist: no such file or directory
exit status 1
[maxwell@MaxwellDBA goproject]$ 

An even better file copy program

Although traditional Unix command-line utilities are silent when there are no errors, it is not bad to print some kind of information, such as the number of bytes read, in your own tools. However, the right thing to do is to follow the Unix way.

There exist two main reasons that make cp.go better than notGoodCP.go.

• more contorl over the process in exchange for having to write more Go code.
• allow you to define the size of the buffer, which is the most important parameter in the copy operation.

[maxwell@MaxwellDBA goproject]$ go run cp.go inputFile cCopy 2048 
Copying inputFile to cCopy
[maxwell@MaxwellDBA goproject]$ diff inputFile cCopy
[maxwell@MaxwellDBA goproject]$ go run cp.go A /tmp/myCP 1024
Copying A to /tmp/myCP
File copying failed: "stat A: no such file or directory"
[maxwell@MaxwellDBA goproject]$ go run cp.go inputFile /tmp/myCP 1024
Copying inputFile to /tmp/myCP
[maxwell@MaxwellDBA goproject]$ go run cp.go inputFile outputfile 1024            
Copying inputFile to outputfile
[maxwell@MaxwellDBA goproject]$ go run cp.go inputFile outputfile 1024
Copying inputFile to outputfile
File copying failed: "File outputfile already exists."
[maxwell@MaxwellDBA goproject]$ cat cp.go
package main

import (
   "fmt"
   "io"
   "os"
   "path/filepath"
   "strconv"
)

var BUFFERSIZE int64

func Copy(src, dst string, BUFFERSIZE int64) error {
    sourceFileStat, err := os.Stat(src)
    if err != nil {
          return err
    }

    if !sourceFileStat.Mode().IsRegular() {
          return fmt.Errorf("%s is not a regular file.", src)
    }

    source, err := os.Open(src)
    if err != nil {
         return err
     }

     defer source.Close()
     _, err = os.Stat(dst)
     if err == nil {
           return fmt.Errorf("File %s already exists.", dst)
     }

     destination, err := os.Create(dst)
     if err != nil {
           return err
     }

     buf := make([]byte, BUFFERSIZE)
     for {
          n, err := source.Read(buf)
          if err != nil && err != io.EOF {
                return err
         }
         if n == 0 {
               break
         }

         if _, err := destination.Write(buf[:n]); err != nil {
                return err
         }

      }
      return err
}

func main() {
   if len(os.Args) != 4 {
         fmt.Printf("usage: %s source destination BUFFERSIZE\n",filepath.Base(os.Args[0]))
         os.Exit(1)
   }
   source := os.Args[1]
   destination := os.Args[2]
   BUFFERSIZE, _ = strconv.ParseInt(os.Args[3], 10, 64)
   fmt.Printf("Copying %s to %s\n", source, destination)
   err := Copy(source, destination, BUFFERSIZE)
   if err != nil {
         fmt.Printf("File copying failed: %q\n", err)
   }
}
[maxwell@MaxwellDBA goproject]$ 

Benchmarking file copying operations

The size of the buffer you use in file operations is really important and affect the performance of your system tools, especially when you are dealing with very big files.

Although developing reliable software should be your main concern, you should not forget to make your systems software fast and efficient!

Developing wc(1) in Go

The principal idea behind the code of the wc.go program is that you can read a text file line by line until there is nothing left to read.

Counting words

r := regexp.MustCompile("[^\\s]+") 
for range r.FindAllString(line, -1) { 
numberOfWords++ 
} 

The wc.go code

[maxwell@MaxwellDBA goproject]$ vim wc.go
[maxwell@MaxwellDBA goproject]$ go build wc.go
[maxwell@MaxwellDBA goproject]$ ls -l wc
-rwxrwxr-x 1 maxwell maxwell 60448 Dec 18 20:55 wc
[maxwell@MaxwellDBA goproject]$ ./wc wc.go notGoodCP.go
117     383     2458    wc.go
49      166     923     notGoodCP.go
166     549     3381    total
[maxwell@MaxwellDBA goproject]$
[maxwell@MaxwellDBA goproject]$ ./wc -l wc.go
117     wc.go
[maxwell@MaxwellDBA goproject]$ cat wc.go
package main

import (
   "bufio"
   "flag"
   "fmt"
   "io"
   "os"
   "regexp"
)

func countLines(filename string)(int, int, int) {
    var err error
    var numberOfLines int
    var numberOfCharacters int
    var numberOfWords int
    numberOfLines = 0
    numberOfCharacters = 0
    numberOfWords = 0

    f, err := os.Open(filename)
    if err != nil {
          fmt.Printf("error opening file %s", err)
          os.Exit(1)
    }
    defer f.Close()

    r := bufio.NewReader(f)
    for {
         line, err := r.ReadString('\n')
         if err == io.EOF {
               break
         } else if err != nil {
                      fmt.Printf("error reading file %s", err)
                      break
        }

        numberOfLines++
        r := regexp.MustCompile("[^\\S]+")
        for range r.FindAllString(line, -1){
             numberOfWords++
        }
        numberOfCharacters += len(line)
    }

    return numberOfLines, numberOfWords, numberOfCharacters
}

func main() {
   minusC := flag.Bool("c", false, "Characters")
   minusW := flag.Bool("w", false, "Words")
   minusL := flag.Bool("l", false, "Lines")

   flag.Parse()
   flags := flag.Args()

   if len(flags) == 0 {
         fmt.Printf("usage: wc <file1> [<file2>[...<fileN]]\n")
         os.Exit(1)
   }

   totalLines := 0
   totalWords := 0
   totalCharacters := 0
   printAll := false


   for _, filename := range flag.Args() {numberOfLines, numberOfWords, numberOfCharacters := countLines(filename)
           totalLines = totalLines + numberOfLines
           totalWords = totalWords + numberOfWords
           totalCharacters = totalCharacters + numberOfCharacters

           if (*minusC && *minusW && *minusL)||(!*minusC && !*minusW && !*minusL){
                 fmt.Printf("%d", numberOfLines)
                 fmt.Printf("\t%d", numberOfWords)
                 fmt.Printf("\t%d", numberOfCharacters)
                 fmt.Printf("\t%s\n", filename)
                 printAll = true
                 continue
           }

           if *minusL {
                 fmt.Printf("%d", numberOfLines)
           }

           if *minusW {
                 fmt.Printf("%t%d", numberOfWords)
           }
           if *minusC {
                 fmt.Printf("\t%d", numberOfCharacters)
           }

           fmt.Printf("\t%s\n", filename)
   }

   if (len(flags) != 1) && printAll {
         fmt.Printf("%d", totalLines)
         fmt.Printf("\t%d", totalWords)
         fmt.Printf("\t%d", totalCharacters)
         fmt.Println("\ttotal")
         return
   }

   if (len(flags) != 1) && *minusL {
         fmt.Printf("%d", totalLines)
   }

   if (len(flags) != 1) && *minusW {
          fmt.Printf("\t%d", totalWords)
   }
   if (len(flags) != 1) && *minusC {
           fmt.Printf("\t%d", totalCharacters)
   }
   if len(flags) != 1 {
        fmt.Printf("\ttotal\n")
   }
}
[maxwell@MaxwellDBA goproject]$ 

Comparing the performance of wc.go and wc(1)

[maxwell@MaxwellDBA goproject]$ 
[maxwell@MaxwellDBA goproject]$ file wc
wc: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, with debug_info, not stripped
[maxwell@MaxwellDBA goproject]$ time ./wc *.data
3       105     591     pF.data

real    0m0.030s
user    0m0.020s
sys     0m0.009s
[maxwell@MaxwellDBA goproject]$ file `which wc`
/usr/bin/wc: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=95e5c876eefa8d7f0b9ded2eabf39955768e7e4f, stripped
[maxwell@MaxwellDBA goproject]$ time wc *.data
  3 104 591 pF.data

real    0m0.002s
user    0m0.000s
sys     0m0.002s
[maxwell@MaxwellDBA goproject]$ 

 The general idea when developing software of any kind, on any platform,using any programming language, is that you should try to have a working version of it, which does not contain any bugs before trying to optimize it and not the other way round!

Reading a text file character by character

[maxwell@MaxwellDBA goproject]$ 
[maxwell@MaxwellDBA goproject]$ vim charByChar.go
[maxwell@MaxwellDBA goproject]$ go run charByChar.go test | wc
exit status 1
      1       7      37
[maxwell@MaxwellDBA goproject]$ wc test
wc: test: No such file or directory
[maxwell@MaxwellDBA goproject]$ wc testFile
  17  997 6576 testFile
[maxwell@MaxwellDBA goproject]$ cat charByChar.go
package main

import (
   "bufio"
   "fmt"
   "io/ioutil"
   "os"
   "strings"
)

func main() {
   arguments := os.Args
   if len(arguments) == 1 {
         fmt.Println("Not enough arguments!")
         os.Exit(1)
   }
   input := arguments[1]
   buf, err := ioutil.ReadFile(input)
   if err != nil {
         fmt.Println(err)
         os.Exit(1)
   }

   in := string(buf)
   s := bufio.NewScanner(strings.NewReader(in))
   s.Split(bufio.ScanRunes)

   for s.Scan(){
         fmt.Print(s.Text())
   }
}
[maxwell@MaxwellDBA goproject]$ 

Doing some file editing!

The go program that converts tab characters to space characters in files and vice versa!

Note that tabSpace.go reads text files line by line, but you can also develop a version that reads text file character by character.

[maxwell@MaxwellDBA goproject]$ 
[maxwell@MaxwellDBA goproject]$ vim tabSpace.go
[maxwell@MaxwellDBA goproject]$ go run tabSpace.go -t cp.go > convert
[maxwell@MaxwellDBA goproject]$ wc convert cp.go 
  70  182 1428 convert
  70  182 1410 cp.go
 140  364 2838 total
[maxwell@MaxwellDBA goproject]$ go run tabSpace.go -s convert | wc
     70     182    1274
[maxwell@MaxwellDBA goproject]$ cat tabSpace.go
package main

import (
   "bufio"
   "fmt"
   "io"
   "os"
   "path/filepath"
   "strings"
)

func main() {
   if len(os.Args) != 3 {
         fmt.Printf("Usage: %s [-t|-s] filename!\n", filepath.Base(os.Args[0]))
         os.Exit(1)
   }
   convertTabs := false
   convertSpaces := false
   newLine := ""

   option := os.Args[1]
   filename := os.Args[2]
   if option == "-t" {
        convertTabs = true
   } else if option == "-s" {
        convertSpaces = true
   } else {
        fmt.Println("Unknown option!")
        os.Exit(1)
   }

   f, err := os.Open(filename)
   if err != nil {
         fmt.Printf("error opening %s: %s", filename, err)
         os.Exit(1)
   }

   defer f.Close()

   r := bufio.NewReader(f)
   for {
        line, err := r.ReadString('\n')

        if err == io.EOF {
              break
        } else if err != nil {
             fmt.Printf("error reading file %s", err)
             os.Exit(1)
        }
        if convertTabs == true {
              newLine = strings.Replace(line, "\t", "   ", -1)
         } else if convertSpaces == true {
               newLine = strings.Replace(line, "   ","\t",-1)
        }

        fmt.Print(newLine)
   }
}
[maxwell@MaxwellDBA goproject]$ 

Interprocess communication

Interprocess communication (IPC), putting it simply,is allowing Unix processes to talk to each other. Various techniques exist that allow processes and programs to talk to each other.

Sparse files in Go

[maxwell@MaxwellDBA goproject]$ vim sparse.go             
[maxwell@MaxwellDBA goproject]$ go run sparse.go 1000 test
[maxwell@MaxwellDBA goproject]$ go run sparse.go 1000 test
File test already exists. 
exit status 1
[maxwell@MaxwellDBA goproject]$ go run sparse.go 100000 testSparse$ dd if=/dev/urandom bs=1 count=100000 of=noSparseDD
usage: sparse SIZE filename
exit status 1
[maxwell@MaxwellDBA goproject]$ go run sparse.go 100000 testSparse$ dd if=/dev/urandom bs=1 count=100000 of=noSparseDD
usage: sparse SIZE filename
exit status 1
[maxwell@MaxwellDBA goproject]$ dd if=/dev/urandom seek=100000 bs=1 count=0 of=noSparseDD     
0+0 records in
0+0 records out
0 bytes copied, 6.6192e-05 s, 0.0 kB/s

[maxwell@MaxwellDBA goproject]$ cat sparse.go
package main

import (
   "fmt"
   "log"
   "os"
   "path/filepath"
   "strconv"
)

func main() {
   if len(os.Args) != 3 {
          fmt.Printf("usage: %s SIZE filename\n", filepath.Base(os.Args[0]))
          os.Exit(1)
   }

   SIZE, _ := strconv.ParseInt(os.Args[1], 10, 64)
   filename := os.Args[2]

   _, err := os.Stat(filename)
   if err == nil {
          fmt.Printf("File %s already exists. \n", filename)
          os.Exit(1)
   }

   fd, err := os.Create(filename)
   if err != nil {
          log.Fatal("Failed to create output")
   }

   _, err = fd.Seek(SIZE-1, 0)
   if err != nil {
         fmt.Println(err)
         log.Fatal("Failed to seek")
   }

   _, err = fd.Write([]byte{0})
   if err != nil {
         fmt.Println(err)
         log.Fatal("Write operation failed")
   }

   err = fd.Close()
   if err != nil {
         fmt.Println(err)
         log.Fatal("Failed to close file")
   }
}
[maxwell@MaxwellDBA goproject]$ 

Reading and writing data records

The sections will teach you how to deal with writing and reading data records.

The Go code of records.go will save data in the CSV format and will be presented as below :

[maxwell@MaxwellDBA goproject]$ vim records.go
[maxwell@MaxwellDBA goproject]$ go run records.go recordsDataFile
M:T:I.
D:T:I.
M:T:D
V:T:D
A:T:D
[maxwell@MaxwellDBA goproject]$ ls -l recordsDataFile
-rw-rw-r-- 1 maxwell maxwell 32 Dec 19 09:41 recordsDataFile
[maxwell@MaxwellDBA goproject]$ cat recordsDataFile
M,T,I.
D,T,I.
M,T,D
V,T,D
A,T,D
[maxwell@MaxwellDBA goproject]$ cat records.go
package main

import (
   "encoding/csv"
   "fmt"
   "os"
)

func main(){
    if len(os.Args) != 2 {
          fmt.Println("Need just one filename!")
          os.Exit(-1)
    }

    filename := os.Args[1]
    _, err := os.Stat(filename)
    if err == nil {
          fmt.Printf("File %s already exists.\n", filename)
          os.Exit(1)
    }
    output, err := os.Create(filename)
    if err != nil {
          fmt.Println(err)
          os.Exit(-1)
    }
    defer output.Close()

    inputData := [][]string{{"M", "T","I."},{"D", "T","I."},{"M","T","D"},{"V","T","D"},{"A","T","D"}}
    writer := csv.NewWriter(output)
    for _, record := range inputData {
         err := writer.Write(record)
         if err != nil {
               fmt.Println(err)
               os.Exit(-1)
         }
    }
    writer.Flush()
    f, err := os.Open(filename)
    if err != nil {
          fmt.Println(err)
          return
    }
    defer f.Close()

    reader := csv.NewReader(f)
    reader.FieldsPerRecord = -1
    allRecords, err := reader.ReadAll()
    if err != nil {
          fmt.Println(err)
          os.Exit(1)
    }

    for _, rec := range allRecords {
          fmt.Printf("%s:%s:%s\n", rec[0], rec[1], rec[2])
    }
}
[maxwell@MaxwellDBA goproject]$ 

File locking in Go

Various techniques exist for file locking. One of them is by creating an additional file that signifies that another program or process is using a given resource. The presented technique is more suitable for programs that use multiple go routines.

[maxwell@MaxwellDBA goproject]$ go run fileLocking.go 123
Wrote from 2
Wrote from 0
Wrote from 1
[maxwell@MaxwellDBA goproject]$ cat fileLocking.go
package main

import (
   "fmt"
   "math/rand"
   "os"
   "sync"
   "time"
)

var mu sync.Mutex

func random(min, max int) int {
   return rand.Intn(max-min) + min
}

func writeDataToFile(i int, file *os.File, w *sync.WaitGroup){
   mu.Lock()
   time.Sleep(time.Duration(random(10, 1000)) * time.Millisecond)
   fmt.Fprintf(file, "From %d, writing %d\n", i, 2*i)
   fmt.Printf("Wrote from %d\n", i)
   w.Done()
mu.Unlock()
}

func main() {
    if len(os.Args) != 2 {
          fmt.Println("Please provide one command line argument!")
          os.Exit(-1)
    }

    filename := os.Args[1]
    number := 3

    file, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY|os.O_TRUNC,0644)
    if err != nil {
          fmt.Println(err)
          os.Exit(1)
    }

    var w *sync.WaitGroup = new(sync.WaitGroup)
    w.Add(number)

    for r := 0; r < number; r++ {
          go writeDataToFile(r, file, w)
    }

    w.Wait()
}
[maxwell@MaxwellDBA goproject]$ 

 

A simplified Go version of the dd utility

The dd(1) tool can do many things, currently implement a small part of its functionality. dd(1) will include support for two command-line flag:

• one for specifying the block size in bytes(-bs)
• specifying the total number of blocks that will be written(-count)

The Go code is saved as ddGo.go  as below :

[maxwell@MaxwellDBA goproject]$ vim ddGo.go
[maxwell@MaxwellDBA goproject]$ time go run ddGo.go -bs=10000 -count=5000 test3

real    0m3.256s
user    0m3.151s
sys     0m0.257s
[maxwell@MaxwellDBA goproject]$ ls -ltr test3
-rw-rw-r-- 1 maxwell maxwell 50000000 Dec 19 11:24 test3
[maxwell@MaxwellDBA goproject]$ go run ddGo.go -bs=100 -count=50 test1
[maxwell@MaxwellDBA goproject]$ go run ddGo.go -bs=100 -count=50 test2
[maxwell@MaxwellDBA goproject]$ ls -l test1 test2
-rw-rw-r-- 1 maxwell maxwell 5000 Dec 19 11:25 test1
-rw-rw-r-- 1 maxwell maxwell 5000 Dec 19 11:25 test2
[maxwell@MaxwellDBA goproject]$ diff test1 test2
Binary files test1 and test2 differ
[maxwell@MaxwellDBA goproject]$ cat ddGo.go
package main

import (
  "flag"
  "fmt"
  "math/rand"
  "os"
  "time"
)

func random(min, max int) int {
    return rand.Intn(max-min) + min
}


func createBytes(buf *[]byte, count int) {
    if count == 0 {
          return
    }
    for i:= 0; i < count; i++ {
         intByte := byte(random(0, 9))
         *buf = append(*buf, intByte)
    }
}

func main() {
    minusBS := flag.Int("bs", 0, "Block Size")
    minusCOUNT := flag.Int("count", 0, "Counter")
    flag.Parse()
    flags := flag.Args()

    if len(flags) == 0 {
          fmt.Println("Not enough arguments!")
          os.Exit(-1)
     }

     if *minusBS < 0 || *minusCOUNT < 0 {
           fmt.Println("Count or/and Byte Size < 0!")
           os.Exit(-1)
      }

      filename := flags[0]
      rand.Seed(time.Now().Unix())

      _, err := os.Stat(filename)
      if err == nil {
            fmt.Printf("File %s already exists.\n", filename)
            os.Exit(1)
      }

      destination, err := os.Create(filename)
      if err != nil {
            fmt.Println("os.Create:", err)
            os.Exit(1)
      }

      buf := make([]byte, *minusBS)
      buf = nil
      for i := 0; i < *minusCOUNT; i++ {
            createBytes(&buf, *minusBS)
            if _, err := destination.Write(buf); err != nil {
                  fmt.Println(err)
                  os.Exit(-1)
            }
            buf = nil
      }
}
[maxwell@MaxwellDBA goproject]$