Using mmap from `golang.org/x/exp/mmap`

mmap.go

package main

import (
	"bufio"
	"fmt"
	"io"
	"log"
	"os"

	"golang.org/x/exp/mmap"
)

// This program demonstrates how to use the mmap package from the
// golang.org/x/exp/mmap module to memory-map a file and read lines
// from it using a bufio.Scanner. By memory-mapping the file, we avoid
// the need to copy its contents into a new buffer, which can be
// inefficient and time-consuming for large files. Instead, we create
// a "view" into the file's contents by mapping it into memory, allowing
// us to read from it just as if it were a regular byte slice.
//
// We use the mmap.Open function to open the file and create a memory-mapped
// region. We then use io.NewSectionReader to create an io.Reader that reads
// from a specific portion of the memory-mapped region. This allows us to create
// a bufio.Scanner that reads lines from the memory-mapped region without
// having to copy the entire file to a new buffer. Finally, we loop over the
// scanner with scanner.Scan() and process each line using scanner.Text().
// If there is an error while scanning the file, scanner.Err() will return
// the error.
//
// Memory-mapping a file can be a useful technique when dealing with large
// files that need to be read or written quickly. It can also be useful for
// reading and writing data in a way that is more memory-efficient than using
// a traditional file I/O API. However, memory-mapping a file can also have
// some drawbacks, such as increased memory usage and the potential for data
// corruption if the file is modified while it is mapped. Therefore, it is
// important to use memory-mapping judiciously and be aware of its limitations.

func main() {
	mmapped, err := mmap.Open("/etc/hosts")
	if err != nil {
		log.Fatalln("couldn't mmap")
	}
	defer mmapped.Close()

	// Wrap the mapped region in an io.Reader
	r := io.NewSectionReader(mmapped, 0, int64(mmapped.Len()))

	// Create a scanner from the io.Reader
	scanner := bufio.NewScanner(r)
	for scanner.Scan() {
		fmt.Println(scanner.Text()) // Println will add back the final '\n'
	}
	if err := scanner.Err(); err != nil {
		fmt.Fprintln(os.Stderr, "reading standard input:", err)
	}
}