使用go語言繞過page cache讀寫文件

阿新 • • 發佈：2017-08-27

tdi port 否則測試程序 ++ cal creation slice def

有時候，我們希望我們的數據不通過page cache的緩沖直接落盤。go語言裏，用參數DIRECT打開文件可以實現這一點要求。

但這樣做有一個硬性的要求，就是在讀寫的時候，對應的數據在內存中的地址一定要滿足512對齊，即首地址的2進制形式中後面至少要有9個0結尾，且數據長度為512字節的整數倍，否則讀寫會失敗。

我們用go語言中的切片slice來驗證這件事。

首先我們建立一個go語言的切片並隨便賦一些值：

buf := make([]byte, 8192)
for i := 0; i < 20; i++ {
	buf[i] = byte(i)
}

我們首先嘗試一下正常的讀寫文件過程，先不使用DIRECT參數繞開page cache。

func writeWithoutAlignmentWithoutDIRECT(buf []byte) {
	// open file
	file, err := os.OpenFile("/dev/sdb",
		os.O_WRONLY|os.O_CREATE, 0666)
	if err != nil {
		fmt.Printf("An error occurred with file opening or creation\n")
		return
	}
	defer file.Close()

	// write file
	fmt.Println("buffer ", unsafe.Pointer(&buf))
	fmt.Println("buffer[0] ", unsafe.Pointer(&buf[0]))
	buf2 := buf[4:516]
	fmt.Println("write with buffer ", unsafe.Pointer(&buf2[0]))

	_, err = file.WriteAt(buf2, 512)
	if err != nil {
		fmt.Println("write error ", err)
	} else {
		fmt.Println("write succeed")
	}
}

這段代碼的運行結果如下：

buffer  0xc42000a2a0
buffer[0]  0xc42005a000
write with buffer  0xc42005a004
write succeed

可以看出，這個切片的地址是0xc42000a2a0，這個切片內數據的首地址是0xc42005a000，這是一個與c語言不同的地方，c語言的數據首地址即為其中數據的首地址，而go語言中，切片的地址和切片內數據首地址是不同的。

我們要寫入的數據的首地址是從切片的第5個元素開始，其首地址為0xc42005a004，雖然並沒有512對齊，但是由於我們沒有嘗試繞過page cache，所以根據WriteAt函數的返回值err可以看到，我們的寫入操作是成功的。

下面我們嘗試一下使用DIRECT參數打開文件，繞過page cache進行寫數據操作，其他參數不變。

func writeWithoutAlignmentWithDIRECT(buf []byte) {
	// open file
	file, err := os.OpenFile("/dev/sdc",
		os.O_WRONLY|os.O_CREATE|syscall.O_DIRECT, 0666)
	if err != nil {
		fmt.Printf("An error occurred with file opening or creation\n")
		return
	}
	defer file.Close()

	// write file
	fmt.Println("buffer ", unsafe.Pointer(&buf))
	fmt.Println("buffer[0] ", unsafe.Pointer(&buf[0]))
	buf2 := buf[4:516]
	fmt.Println("write with buffer ", unsafe.Pointer(&buf2[0]))

	_, err = file.WriteAt(buf2, 512)
	if err != nil {
		fmt.Println("write error ", err)
	} else {
		fmt.Println("write succeed")
	}
}

這段代碼運行後，我們可以看到如下結果。

buffer  0xc42000a2e0
buffer[0]  0xc42005a000
write with buffer  0xc42005a004
write error  write /dev/sdc: invalid argument

看到了write error，WriteAt函數這次的返回值給出的並不是nil了，我們的寫入操作失敗了，其返回值返回了一個不可理解的invalid argument（非法參數）。

but我們的參數毫無問題啊！下面我們嘗試一下把要寫入的數據改為512對齊。

func writeWithAlignmentWithDIRECT(buf []byte) {
	// open file
	file, err := os.OpenFile("/dev/sdd",
		os.O_WRONLY|os.O_CREATE|syscall.O_DIRECT, 0666)
	if err != nil {
		fmt.Printf("An error occurred with file opening or creation\n")
		return
	}
	defer file.Close()

	// write file
	fmt.Println("buffer ", unsafe.Pointer(&buf))
	fmt.Println("buffer[0] ", unsafe.Pointer(&buf[0]))
	buf2 := buf[512 : 512+512]
	fmt.Println("write with buffer ", unsafe.Pointer(&buf2[0]))

	_, err = file.WriteAt(buf2, 512)
	if err != nil {
		fmt.Println("write error ", err)
	} else {
		fmt.Println("write succeed")
	}
}

這段代碼運行後，結果如下。

white with alignment and DIRECT:
buffer  0xc42000a340
buffer[0]  0xc42005a000
write with buffer  0xc42005a200
write succeed

我們的寫操作成功了！而這段代碼與上次未成功的不同之處只有一個，那就是將要寫入數據的首地址改成了512對齊。

通過這三段go程序，我們很清晰的驗證了繞過page cache寫文件的條件。

類似的，下面給出驗證繞過page cache讀文件也需要512對齊條件的代碼。

func readWithoutAlignmentWithoutDIRECT(buf []byte) {
	// read file
	file, err := os.OpenFile("/dev/sdb", os.O_RDONLY, 0666)
	if err != nil {
		fmt.Printf("An error occurred whit file ipening.\n")
		return
	}
	defer file.Close()

	buf = buf[2:514]
	fmt.Println("read with buffer ", unsafe.Pointer(&buf[0]))

	_, err = file.ReadAt(buf, 512)
	if err != nil {
		fmt.Println("read error ", err)
	} else {
		fmt.Println("read succeed", buf)
	}
}
func readWithoutAlignmentWithDIRECT(buf []byte) {
	// read file
	file, err := os.OpenFile("/dev/sdc", os.O_RDONLY|syscall.O_DIRECT, 0666)
	if err != nil {
		fmt.Printf("An error occurred whit file ipening.\n")
		return
	}
	defer file.Close()

	buf = buf[2:514]
	fmt.Println("read with buffer ", unsafe.Pointer(&buf[0]))

	_, err = file.ReadAt(buf, 512)
	if err != nil {
		fmt.Println("read error ", err)
	} else {
		fmt.Println("read succeed", buf)
	}
}
func readWithAlignmentWithDIRECT(buf []byte) {
	// read file
	file, err := os.OpenFile("/dev/sdd", os.O_RDONLY|syscall.O_DIRECT, 0666)
	if err != nil {
		fmt.Printf("An error occurred whit file ipening.\n")
		return
	}
	defer file.Close()

	buf = buf[512 : 512+512]
	fmt.Println("read with buffer ", unsafe.Pointer(&buf[0]))

	_, err = file.ReadAt(buf, 512)
	if err != nil {
		fmt.Println("read error ", err)
	} else {
		fmt.Println("read succeed", buf)
	}
}

這三個函數的運行結果分如下。

read with buffer  0xc42005a002
read succeed [4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

read with buffer  0xc42005a002
read error  read /dev/sdc: invalid argument

read with buffer  0xc42005a200
read succeed [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

可以看出，由於最初我們將切片的前20位分別賦值為0-20，其他位賦值默認值為0，第一個寫入函數將buf[4:516]寫入到/dev/sdb中，第二個寫入函數寫入失敗，第三個寫入函數將buf[512 : 512+512]寫入到/dev/sdd，所以根據讀取結果可以看出，我們的讀取函數也是ok的。

最後，給出整段測試程序的完整代碼。

package main

import (
	"fmt"
	"os"
	"syscall"
	"unsafe"
)

func main() {
	buf := make([]byte, 8192)
	for i := 0; i < 20; i++ {
		buf[i] = byte(i)
	}
	fmt.Println("----------------------------------------")
	fmt.Println("white without alignment and DIRECT:")
	writeWithoutAlignmentWithoutDIRECT(buf)
	fmt.Println("----------------------------------------")
	fmt.Println("white without alignment but with DIRECT:")
	writeWithoutAlignmentWithDIRECT(buf)
	fmt.Println("----------------------------------------")
	fmt.Println("white with alignment and DIRECT:")
	writeWithAlignmentWithDIRECT(buf)
	fmt.Println("----------------------------------------")
	fmt.Println("read without alignment and DIRECT:")
	readWithoutAlignmentWithoutDIRECT(buf)
	fmt.Println("----------------------------------------")
	fmt.Println("read without alignment but with DIRECT:")
	readWithoutAlignmentWithDIRECT(buf)
	fmt.Println("----------------------------------------")
	fmt.Println("read with alignment and DIRECT:")
	readWithAlignmentWithDIRECT(buf)
}

func writeWithoutAlignmentWithoutDIRECT(buf []byte) {
	// open file
	file, err := os.OpenFile("/dev/sdb",
		os.O_WRONLY|os.O_CREATE, 0666)
	if err != nil {
		fmt.Printf("An error occurred with file opening or creation\n")
		return
	}
	defer file.Close()

	// write file
	fmt.Println("buffer ", unsafe.Pointer(&buf))
	fmt.Println("buffer[0] ", unsafe.Pointer(&buf[0]))
	buf2 := buf[4:516]
	fmt.Println("write with buffer ", unsafe.Pointer(&buf2[0]))

	_, err = file.WriteAt(buf2, 512)
	if err != nil {
		fmt.Println("write error ", err)
	} else {
		fmt.Println("write succeed")
	}
}
func writeWithoutAlignmentWithDIRECT(buf []byte) {
	// open file
	file, err := os.OpenFile("/dev/sdc",
		os.O_WRONLY|os.O_CREATE|syscall.O_DIRECT, 0666)
	if err != nil {
		fmt.Printf("An error occurred with file opening or creation\n")
		return
	}
	defer file.Close()

	// write file
	fmt.Println("buffer ", unsafe.Pointer(&buf))
	fmt.Println("buffer[0] ", unsafe.Pointer(&buf[0]))
	buf2 := buf[4:516]
	fmt.Println("write with buffer ", unsafe.Pointer(&buf2[0]))

	_, err = file.WriteAt(buf2, 512)
	if err != nil {
		fmt.Println("write error ", err)
	} else {
		fmt.Println("write succeed")
	}
}
func writeWithAlignmentWithDIRECT(buf []byte) {
	// open file
	file, err := os.OpenFile("/dev/sdd",
		os.O_WRONLY|os.O_CREATE|syscall.O_DIRECT, 0666)
	if err != nil {
		fmt.Printf("An error occurred with file opening or creation\n")
		return
	}
	defer file.Close()

	// write file
	fmt.Println("buffer ", unsafe.Pointer(&buf))
	fmt.Println("buffer[0] ", unsafe.Pointer(&buf[0]))
	buf2 := buf[512 : 512+512]
	fmt.Println("write with buffer ", unsafe.Pointer(&buf2[0]))

	_, err = file.WriteAt(buf2, 512)
	if err != nil {
		fmt.Println("write error ", err)
	} else {
		fmt.Println("write succeed")
	}
}

func readWithoutAlignmentWithoutDIRECT(buf []byte) {
	// read file
	file, err := os.OpenFile("/dev/sdb", os.O_RDONLY, 0666)
	if err != nil {
		fmt.Printf("An error occurred whit file ipening.\n")
		return
	}
	defer file.Close()

	buf = buf[2:514]
	fmt.Println("read with buffer ", unsafe.Pointer(&buf[0]))

	_, err = file.ReadAt(buf, 512)
	if err != nil {
		fmt.Println("read error ", err)
	} else {
		fmt.Println("read succeed", buf)
	}
}
func readWithoutAlignmentWithDIRECT(buf []byte) {
	// read file
	file, err := os.OpenFile("/dev/sdc", os.O_RDONLY|syscall.O_DIRECT, 0666)
	if err != nil {
		fmt.Printf("An error occurred whit file ipening.\n")
		return
	}
	defer file.Close()

	buf = buf[2:514]
	fmt.Println("read with buffer ", unsafe.Pointer(&buf[0]))

	_, err = file.ReadAt(buf, 512)
	if err != nil {
		fmt.Println("read error ", err)
	} else {
		fmt.Println("read succeed", buf)
	}
}
func readWithAlignmentWithDIRECT(buf []byte) {
	// read file
	file, err := os.OpenFile("/dev/sdd", os.O_RDONLY|syscall.O_DIRECT, 0666)
	if err != nil {
		fmt.Printf("An error occurred whit file ipening.\n")
		return
	}
	defer file.Close()

	buf = buf[512 : 512+512]
	fmt.Println("read with buffer ", unsafe.Pointer(&buf[0]))

	_, err = file.ReadAt(buf, 512)
	if err != nil {
		fmt.Println("read error ", err)
	} else {
		fmt.Println("read succeed", buf)
	}
}

使用go語言繞過page cache讀寫文件

tdi port 否則測試程序 ++ cal creation slice def 有時候，我們希望我們的數據不通過page cache的緩沖直接落盤。go語言裏，用參數DIRECT打開文件可以實現這一點要求。但這樣做有一個硬性的要求，就是在讀寫的時候，對應的數據在內存

使用go語言繞過page cache讀寫文件

使用go語言繞過page cache讀寫文件

C語言讀寫文件

Go(day7 [終端讀寫| 文件操作 | 命令行參數 | Json序列化])

002-Go通過ioutil 讀寫文件

第28月第3天 c語言讀寫文件

Go語言執行緒簡單讀寫

FileStream讀寫文件

python 讀寫文件

Python讀寫文件

java 讀寫文件常用方法

java.io幾種讀寫文件的方式

[Python 3系列]讀寫文件

c讀寫文件

Go語言學習筆記（五）文件操作

aNDROID讀寫文件的N種寫法

在android中讀寫文件

用numpy計算成交量加權平均價格（VWAP），並實現讀寫文件

讀寫文件

標準I/O讀寫文件

簡單的文件讀寫---文件簡單的加密、解密

使用go語言繞過page cache讀寫文件

相關推薦