Working with System Files

• Appending data to an existing file
• Reading a file adn altering each one of its lines
• Regular expressions and pattern matching in Go
• Sending information to Unix log files
• Working with dates and times in Go
• Working with user IDs and group IDs
• Learning more information about files and directiories
• Processing log files and extracting useful information from them
• Generating difficult to guess passwords using random numbers

Putting data at the end of a file

We will talk about opening a file for writing without destroying its existing data.

maxwell@MaxwellDBA goproject]$ cat test
[test]: test
: test
[maxwell@MaxwellDBA goproject]$ go run appendData.go test test
[maxwell@MaxwellDBA goproject]$ cat test
[test]: test
: test
test
[maxwell@MaxwellDBA goproject]$ cat appendData.go
package main

import (
   "fmt"
   "os"
   "path/filepath"
)

func main() {
   arguments := os.Args
   if len(arguments) != 3 {
          fmt.Printf("usage: %s message filename\n", filepath.Base(arguments[0]))
          os.Exit(1)
   }
   message := arguments[1]
   filename := arguments[2]

   f, err := os.OpenFile(filename,os.O_RDWR|os.O_APPEND|os.O_CREATE,0660)
   if err != nil {
         fmt.Println(err)
         os.Exit(-1)
   }
   defer f.Close()

   fmt.Fprintf(f, "%s\n", message)
}
[maxwell@MaxwellDBA goproject]$ 

 Altering existing data

how to modify the contents of a file.

[maxwell@MaxwellDBA goproject]$ vim insertLineNumber.go
[maxwell@MaxwellDBA goproject]$ cat test
a



b
[maxwell@MaxwellDBA goproject]$ go run insertLineNumber.go -init=10 test
Processing: test
Processed 6 lines!
[maxwell@MaxwellDBA goproject]$ cat test
10: a 
11:  
12:  
13:  
14: b 
[maxwell@MaxwellDBA goproject]$ 

[maxwell@MaxwellDBA goproject]$ cat test
a


  
b 
[maxwell@MaxwellDBA goproject]$ go run insertLineNumber.go -init=10 test test test
Processing: test
Processing: test
Processing: test
Processed 18 lines!
[maxwell@MaxwellDBA goproject]$ cat test
22: 16: 10: a   
23: 17: 11:    
24: 18: 12:    
25: 19: 13:      
26: 20: 14: b    
[maxwell@MaxwellDBA goproject]$ 
[maxwell@MaxwellDBA goproject]$ cat insertLineNumber.go
package main

import (
   "flag"
   "fmt"
   "io/ioutil"
   "os"
   "strings"
)

func main(){
   minusINIT := flag.Int("init", 1, "Initial Value")
   flag.Parse()
   flags := flag.Args()

   if len(flags) == 0 {
         fmt.Printf("usage: insertLineNumber <files>\n")
         os.Exit(1)
   }

   lineNumber := *minusINIT
   for _, filename := range flags {
      fmt.Println("Processing:", filename)
      input, err := ioutil.ReadFile(filename)
      if err != nil {
            fmt.Println(err)
            os.Exit(-1)
      }

      lines := strings.Split(string(input),"\n")

      for i, line := range lines {
            lines[i] = fmt.Sprintf("%d: %s ", lineNumber, line)
            lineNumber = lineNumber + 1
      }

      lines[len(lines)-1] = ""
      output := strings.Join(lines,"\n")
      err = ioutil.WriteFile(filename, []byte(output),0644)
      if err != nil {
            fmt.Println(err)
            os.Exit(-1)
      }
   }
   fmt.Println("Processed", lineNumber-*minusINIT, "lines!")
}
[maxwell@MaxwellDBA goproject]$ 

About log files

Log files are important and you should not underestimate the value of the information stored in them. Log files should be the first place to look for help when strange things start happening on a Unix machine.

•  first, the output does not get lost, as it is stored on a file
• second , you can search and process log files using Unix tools, such as grep(1), awk(1), and sed(1), which cannot be done when messages are printed on a Terminal window.

About logging

the name of the process is rsyslogd(8), which is an improved and more reliable version of syslogd(8), which was the original Unix system utility for message logging.

The best way to watch one or more log files is with the help of the tail(1)utility, followed by the -f flag and the name of the log file you want to watch. The -f flag tells tail(1) to wait for additional data. You will need to terminate such a tail(1) command by pressing Ctrl + C.

Logging facilities

A logging facility is like a category used for logging information. The value of the logging facility part can be any one of auth, authpriv, cron,daemon, kern,lpr, mail, mark, news, syslog, user, UUCP,local0, local1, local2, local3,local4, local5, local6, and local7.this is defined inside /etc/syslog.conf , /etc/rsyslog.conf.

Logging levels

A logging level or priority is a value that specifies the severity of the log entry. There exist various logging levels including debug, info, notice, warning, err, crit, alert, and emerg, in reverse order of severity.

[maxwell@MaxwellDBA goproject]$ cat useSyslog.go
package main

import (
  "fmt"
  "log"
  "log/syslog"
  "os"
  "path/filepath"
)

func main(){
   programName := filepath.Base(os.Args[0])
   sysLog, e := syslog.New(syslog.LOG_INFO|syslog.LOG_LOCAL7, programName)
   if e != nil {
        log.Fatal(e)
   }
   sysLog.Crit("Crit: Logging in Go!")
   sysLog, e = syslog.New(syslog.LOG_ALERT|syslog.LOG_LOCAL7, "some program!")
   if e != nil {
        log.Fatal(sysLog)
   }
sysLog.Emerg("Emerg: Logging in Go!")
   fmt.Fprintf(sysLog, "log.Print: Logging in Go!")
}
[maxwell@MaxwellDBA goproject]$ go run useSyslog.go

Broadcast message from systemd-journald@MaxwellDBA (Mon 2022-12-19 17:18:19 CST):

[2374196]: some program![2374196]: Emerg: Logging in Go!

[maxwell@MaxwellDBA goproject]$ 
Message from syslogd@MaxwellDBA at Dec 19 17:18:19 ...
 journal[2374196]:some program![2374196]: Emerg: Logging in Go!

[maxwell@MaxwellDBA goproject]$ grep "Logging in Go" /var/log/* 2>/dev/null
[maxwell@MaxwellDBA goproject]$ tail -5 /var/log/syslog
tail: cannot open '/var/log/syslog' for reading: No such file or directory
[maxwell@MaxwellDBA goproject]$ 

Processing log files

[maxwell@MaxwellDBA goproject]$ 
[maxwell@MaxwellDBA goproject]$ go run countIP.go /tmp/log.1 /tmp/log.2
                 /tmp/log.1
error opening file open /tmp/log.1: no such file or directory
                 /tmp/log.2
error opening file open /tmp/log.2: no such file or directory
[maxwell@MaxwellDBA goproject]$ mkdir /tmp/log.1
[maxwell@MaxwellDBA goproject]$ rmdir /tmp/log.1
[maxwell@MaxwellDBA goproject]$ touch /tmp/log.1
[maxwell@MaxwellDBA goproject]$ touch /tmp/log.2
[maxwell@MaxwellDBA goproject]$ vim /tmp/log.1
[maxwell@MaxwellDBA goproject]$ vim /tmp/log.2
[maxwell@MaxwellDBA goproject]$ 
[maxwell@MaxwellDBA goproject]$ 
[maxwell@MaxwellDBA goproject]$ go run countIP.go /tmp/log.1 /tmp/log.2
                 /tmp/log.1
                 /tmp/log.2
64.183.178.218 31
164.132.161.85 1
66.102.8.135 1
5.248.196.10 1
180.76.15.10 1
66.249.69.40 1
51.255.65.35 31
95.158.53.56 31
[maxwell@MaxwellDBA goproject]$ go run countIP.go /tmp/log.1 /tmp/log.2 | wc
     10      18     155
[maxwell@MaxwellDBA goproject]$ go run countIP.go /tmp/log.1 /tmp/log.2 | sort -rn -k2
95.158.53.56 31
64.183.178.218 31
51.255.65.35 31
66.249.69.40 1
66.102.8.135 1
5.248.196.10 1
180.76.15.10 1
164.132.161.85 1
                 /tmp/log.2
                 /tmp/log.1
[maxwell@MaxwellDBA goproject]$ go run countIP.go /tmp/log.1 /tmp/log.2 | sort -rn -k2 | head
95.158.53.56 31
64.183.178.218 31
51.255.65.35 31
66.249.69.40 1
66.102.8.135 1
5.248.196.10 1
180.76.15.10 1
164.132.161.85 1
                 /tmp/log.2
                 /tmp/log.1
[maxwell@MaxwellDBA goproject]$ cat countIP.go
package main

import (
   "bufio"
   "flag"
   "fmt"
   "io"
   "net"
   "os"
   "path/filepath"
   "strings"
)

func main(){
   minusCOL := flag.Int("COL", 1, "Column")
   flag.Parse()
   flags := flag.Args()
   if len(flags) == 0 {
         fmt.Printf("usage: %s <file1> [<file2> [...<fileN]]\n",filepath.Base(os.Args[0]))
         os.Exit(1)
   }

   column := *minusCOL
   if column < 0 {
         fmt.Println("Invalid Column number!")
         os.Exit(1)
   }
   myIPs := make(map[string]int)
   for _, filename := range flags {
         fmt.Println("\t\t", filename)
         f, err := os.Open(filename)
         if err != nil {
              fmt.Printf("error opening file %s\n", err)
              continue
         }
         defer f.Close()

         r := bufio.NewReader(f)
         for {
              line, err := r.ReadString('\n')

              if err == io.EOF {
                   break
              } else if err != nil {
                    fmt.Printf("error reading file %s", err)
                    continue
             }
             data := strings.Fields(line)
             ip := data[column-1]
             trial := net.ParseIP(ip)
             if trial.To4() == nil {
                  continue
             }
             _, ok := myIPs[ip]
             if ok {
                  myIPs[ip] = myIPs[ip] + 1
             } else {
                   myIPs[ip] = 1
            }
         }
   }
   for key, _ := range myIPs {
         fmt.Printf("%s %d\n", key, myIPs[key])
   }
}

[maxwell@MaxwellDBA goproject]$ 

File permissions revisited

The filePerm.go Go utility will teach you how to read the Unix file permissions of a file or a directory and print them as a binary number, a decimal number, and a string.

[maxwell@MaxwellDBA goproject]$ vim filePerm.go
[maxwell@MaxwellDBA goproject]$ go run filePerm.go .
. mode is drwxrwxr-x
As string is rwxrwxr-x
As binary is 111111101
[maxwell@MaxwellDBA goproject]$
[maxwell@MaxwellDBA goproject]$ go run filePerm.go /tmp/swtag.log
/tmp/swtag.log mode is -rw-rw-r--
As string is rw-rw-r--
As binary is 110110100
[maxwell@MaxwellDBA goproject]$ 
[maxwell@MaxwellDBA goproject]$ cat filePerm.go
package main

import (
   "fmt"
   "os"
   "path/filepath"
)

func tripletToBinary(triplet string) string {
   if triplet == "rwx" {
         return "111"
    }
    if triplet == "-wx" {
          return "011"
    }
    if triplet == "--x" {
          return "001"
    }
    if triplet == "---" {
          return "000"
    }
    if triplet == "r-x"{
          return  "101"
    }
    if triplet == "r--" {
          return "100"
    }
    if triplet == "--x" {
          return "001"
    }
    if triplet == "rw-" {
          return "110"
    }
    if triplet == "-w-" {
          return "010"
    }
    return "unknown"
}

func convertToBinary(permissions string) string {
   binaryPermissions := permissions[1:]
   p1 := binaryPermissions[0:3]
   p2 := binaryPermissions[3:6]
   p3 := binaryPermissions[6:9]
   return tripletToBinary(p1) + tripletToBinary(p2) + tripletToBinary(p3)
}

func main() {
   arguments := os.Args
   if len(arguments) == 1 {
           fmt.Printf("usage: %s filename\n", filepath.Base(arguments[0]))
           os.Exit(1)
   }

   filename := arguments[1]
   info, _ := os.Stat(filename)
   mode := info.Mode()

   fmt.Println(filename, "mode is", mode)
   fmt.Println("As string is", mode.String()[1:10])
   fmt.Println("As binary is", convertToBinary(mode.String()))

}
[maxwell@MaxwellDBA goproject]$

 Changing file permissions

how to change the Unix permissions of a file or a directory to the desired value.

[maxwell@MaxwellDBA goproject]$ vim setFilePerm.go
[maxwell@MaxwellDBA goproject]$ go run setFilePerm.go /tmp/swtag.log rwxrwxrwx
[maxwell@MaxwellDBA goproject]$ ls -l /tmp/swtag.log 
-rwxrwxrwx 1 maxwell maxwell 1101 Dec 17 12:53 /tmp/swtag.log
[maxwell@MaxwellDBA goproject]$ go run setFilePerm.go /tmp/swtag.log rwxrwx---
[maxwell@MaxwellDBA goproject]$ ls -l /tmp/swtag.log                          
-rwxrwx--- 1 maxwell maxwell 1101 Dec 17 12:53 /tmp/swtag.log
[maxwell@MaxwellDBA goproject]$ cat setFilePerm.go
package main

import (
   "fmt"
   "os"
   "path/filepath"
   "strconv"
)

func tripletToBinary(triplet string) string {
   if triplet == "rwx" {
         return "111"
    }
    if triplet == "-wx" {
         return "011"
    }
    if triplet == "--x" {
         return "001"
    }
    if triplet == "---" {
         return "000"
    }
    if triplet == "r-x" {
         return "101"
    }
    if triplet == "r--" {
         return "100"
    }
    if triplet == "--x" {
         return "001"
    }
    if triplet == "rw-" {
         return "110"
    }
    if triplet == "-w-" {
          return "010"
    }
    return "unknown"
}

func convertToBinary(permissions string) string {
   p1 := permissions[0:3]
   p2 := permissions[3:6]
   p3 := permissions[6:9]

   p1 = tripletToBinary(p1)
   p2 = tripletToBinary(p2)
   p3 = tripletToBinary(p3)

   p1Int, _ := strconv.ParseInt(p1, 2, 64)
   p2Int, _ := strconv.ParseInt(p2, 2, 64)
   p3Int, _ := strconv.ParseInt(p3, 2, 64)

   returnValue := p1Int*100 + p2Int*10 + p3Int
   tempReturnValue := int(returnValue)
   returnString := "0" + strconv.Itoa(tempReturnValue)
   return returnString
}

func main() {
   arguments := os.Args
   if len(arguments) != 3 {
         fmt.Printf("usage: %s filename permissions\n", filepath.Base(arguments[0]))
         os.Exit(1)
   }

   filename, _ := filepath.EvalSymlinks(arguments[1])
   permissions := arguments[2]
   if len(permissions) != 9 {
         fmt.Println("Permissions should be 9 characters (rwxrwxrwx):", permissions)
         os.Exit(-1)
   }
   
   bin := convertToBinary(permissions)
   newPerms, _ := strconv.ParseUint(bin, 0, 32)
   newMode := os.FileMode(newPerms)
   os.Chmod(filename,newMode)
}
[maxwell@MaxwellDBA goproject]$ 

 A simple pattern matching example

The findIP.go program that automatically detects the field with the IP address. it will not require the user to define the field of each log entry that contains the IP address.

[maxwell@MaxwellDBA goproject]$ wc /var/log/auth.log              
 38  76 657 /var/log/auth.log
[maxwell@MaxwellDBA goproject]$ go run findIP.go /var/log/auth.log
111.224.233.41 2
131.224.233.41 1
193.115.116.50 1
191.60.251.143 1
245.16.210.120 1
5.248.196.10 1
66.249.69.40 1
199.115.116.50 2
171.60.251.143 2
51.141.131.102 1
51.255.65.35 1
64.183.178.218 1
139.160.113.181 2
33.115.101.107 1
129.41.147.179 1
58.31.112.181 1
164.132.161.85 1
66.102.8.135 1
218.237.65.48 2
238.237.65.48 1
133.160.113.181 1
95.158.53.56 1
189.41.147.179 2
24.16.210.120 2
39.114.101.107 2
55.31.112.181 2
180.76.15.10 1
5.141.131.102 2
[maxwell@MaxwellDBA goproject]$ go run findIP.go /var/log/auth.log | sort -nr -k2 | head
55.31.112.181 2
5.141.131.102 2
39.114.101.107 2
24.16.210.120 2
218.237.65.48 2
199.115.116.50 2
189.41.147.179 2
171.60.251.143 2
139.160.113.181 2
111.224.233.41 2
[maxwell@MaxwellDBA goproject]$ cat findIP.go
package main

import (
   "bufio"
   "fmt"
   "io"
   "net"
   "os"
   "path/filepath"
   "regexp"
)

func findIP(input string) string {
    partIP := "(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])"
    grammar := partIP + "\\." + partIP + "\\." + partIP + "\\." + partIP
    matchMe := regexp.MustCompile(grammar)
    return matchMe.FindString(input)
}

func main() {
    if len(os.Args) != 2 {
           fmt.Printf("usage: %s logFile\n", filepath.Base(os.Args[0]))
           os.Exit(1)
     }
     filename := os.Args[1]

     f, err := os.Open(filename)
     if err != nil {
            fmt.Printf("error opening file %s\n", err)
            os.Exit(-1)
     }
     defer f.Close()


     myIPs := make(map[string]int)
     r := bufio.NewReader(f)
     for {
          line, err := r.ReadString('\n')
          if err == io.EOF {
                break
          } else if err != nil {
                fmt.Printf("error reading file %s", err)
                break
         }
         ip := findIP(line)
         trial := net.ParseIP(ip)
         if trial.To4() == nil {
              continue
        } else {
              _, ok := myIPs[ip]
              if ok {
                   myIPs[ip] = myIPs[ip] + 1
              } else {
                    myIPs[ip] = 1
              }
       }
     }
     for key, _ := range myIPs {
           fmt.Printf("%s %d\n", key, myIPs[key])
     }
}
[maxwell@MaxwellDBA goproject]$ 

Consider how many incredible and useful utilities you can develop with a small amount of Go code: it is really amazing!

 An advanced example of pattern matching

how to swap the values of two fields of each line of a text file, provided they are in the correct format. this mainly happens in log files or other text files where you want to scan a line for certain types of data.

As you will see, using regular expressions in Go, can be perplexing sometimes, mainly because regular expressions are relatively difficult to build, in general.

[maxwell@MaxwellDBA goproject]$ vim swapRE.go                 
[maxwell@MaxwellDBA goproject]$ go run swapRE.go /tmp/log.log 
127.0.0.1 - -  3  "GET /contact HTTP/1.1" 200 6048 "http://www.mtsoukalos.eu/" "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko" 13295 [24/May/2017:06:41:11 +0300]
Line processed: 1
Line matched: 1
[maxwell@MaxwellDBA goproject]$ cat swapRE.go
package main

import (
   "bufio"
   "flag"
   "fmt"
   "io"
   "os"
   "regexp"
)

func main() {
   flag.Parse()
   if flag.NArg() != 1 {
         fmt.Println("Please provide one log file to process!")
         os.Exit(-1)
   }
   numberOfLines := 0
   numberOfLinesMatched := 0

   filename := flag.Arg(0)
   f, err := os.Open(filename)
   if err != nil {
         fmt.Printf("error opening file %s", err)
         os.Exit(1)
   }
   defer f.Close()
   r := bufio.NewReader(f)
   for {
         line, err := r.ReadString('\n')
         if err == io.EOF {
               break
         } else if err != nil {
               fmt.Printf("error reading file %s", err)
        }
        numberOfLines++
        r := regexp.MustCompile(`(.*)(\[\d\d\/(\w+)/\d\d\d\d:\d\d:\d\d:\d\d(.*)\])(.*)(\d+)`)
        if r.MatchString(line) {
              numberOfLinesMatched++
              match :=r.FindStringSubmatch(line)
              fmt.Println(match[1], match[6], match[5], match[2])
        }
   }
   fmt.Println("Line processed:", numberOfLines)
   fmt.Println("Line matched:", numberOfLinesMatched)
}
[maxwell@MaxwellDBA goproject]$ 

Note that there might exist alternative ways to write the same regular expression. The general advice here is to write it in a way that is clear and that you can understand.

 Renaming multiple files using regular expressions

When dealing with files, you should be extra careful because you might accidentally destroy things! Putting it simply, do not test such utilities on a production server.

[maxwell@MaxwellDBA goproject]$ ls -l /tmp/swtag.log
-rw-rw-r-- 1 maxwell maxwell 1101 Dec 17 12:53 /tmp/swtag.log
[maxwell@MaxwellDBA goproject]$ chmod 666 /tmp/swtag.log
[maxwell@MaxwellDBA goproject]$ ls -l /tmp/swtag.log    
-rw-rw-rw- 1 maxwell maxwell 1101 Dec 17 12:53 /tmp/swtag.log
[maxwell@MaxwellDBA goproject]$ go run multipleMV.go 'log$' new /tmp
[maxwell@MaxwellDBA goproject]$ ls -l /tmp/new_swtag.log            
-rw-rw-rw- 1 maxwell maxwell 1101 Dec 17 12:53 /tmp/new_swtag.log
[maxwell@MaxwellDBA goproject]$ go run multipleMV.go 'log$' new /tmp
[maxwell@MaxwellDBA goproject]$ ls -l /tmp/new_new_swtag.log        
-rw-rw-rw- 1 maxwell maxwell 1101 Dec 17 12:53 /tmp/new_new_swtag.log
[maxwell@MaxwellDBA goproject]$ go run multipleMV.go 'log$' new /tmp
[maxwell@MaxwellDBA goproject]$ ls -l /tmp/new_new_new_swtag.log    
-rw-rw-rw- 1 maxwell maxwell 1101 Dec 17 12:53 /tmp/new_new_new_swtag.log
[maxwell@MaxwellDBA goproject]$ cat multipleMV.go
package main

import (
  "flag"
  "fmt"
  "os"
  "path/filepath"
  "regexp"
)
var RE string
var renameString string

func walk(path string, f os.FileInfo, err error) error {
    regex, err := regexp.Compile(RE)
    if err != nil {
           fmt.Printf("Error in RE: %s\n", RE)
           return err
    }

    if path == "." {
          return nil
    }
    nameOfFile := filepath.Base(path)
    if regex.MatchString(nameOfFile) {
          newName := filepath.Dir(path) + "/" + renameString + "_" + nameOfFile
          os.Rename(path, newName)
    }
    return nil
}

func main() {
    flag.Parse()
    if flag.NArg() != 3 {
           fmt.Printf("Usage: %s REGEXP RENAME Path", filepath.Base(os.Args[0]))
           os.Exit(-1)
    }

    RE = flag.Arg(0)
    renameString = flag.Arg(1)
    Path := flag.Arg(2)
    Path, _ = filepath.EvalSymlinks(Path)
    filepath.Walk(Path, walk)
}


[maxwell@MaxwellDBA goproject]$