内部/字节算法:valgrind报告C.GoString的无效读取

mzmfm0qo  于 3个月前  发布在  Go
关注(0)|答案(8)|浏览(41)

你使用的Go版本是什么( go version )?

go version go1.11 linux/amd64

这个问题在最新版本中是否会重现?

是的。

你正在使用什么操作系统和处理器架构( go env )?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/kivikakk/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/kivikakk/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build515123667=/tmp/go-build -gno-record-gcc-switches"

你做了什么?

package main

/*
#include <string.h>
#include <stdlib.h>
char* s() {
return strdup("hello");
}
*/
import "C"
import "unsafe"

func main() {
        s := C.s()
        C.GoString(s)
        C.free(unsafe.Pointer(s))
}
$ go build
$ valgrind ./sscce
==11241== Memcheck, a memory error detector
==11241== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==11241== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==11241== Command: ./sscce
==11241==
==11241== Warning: ignored attempt to set SIGRT32 handler in sigaction();
==11241==          the SIGRT32 signal is used internally by Valgrind
==11241== Warning: ignored attempt to set SIGRT32 handler in sigaction();
==11241==          the SIGRT32 signal is used internally by Valgrind
==11241== Warning: client switching stacks?  SP change: 0xfff0001b0 --> 0xc0000367d8
==11241==          to suppress, use: --max-stackframe=755931244072 or greater
==11241== Warning: client switching stacks?  SP change: 0xc000036790 --> 0xfff000260
==11241==          to suppress, use: --max-stackframe=755931243824 or greater
==11241== Warning: client switching stacks?  SP change: 0xfff000260 --> 0xc000036790
==11241==          to suppress, use: --max-stackframe=755931243824 or greater
==11241==          further instances of this message will not be shown.
==11241== Conditional jump or move depends on uninitialised value(s)
==11241==    at 0x40265B: indexbytebody (/usr/local/go/src/internal/bytealg/indexbyte_amd64.s:151)
==11241==
==11241==
==11241== HEAP SUMMARY:
==11241==     in use at exit: 1,200 bytes in 6 blocks
==11241==   total heap usage: 10 allocs, 4 frees, 1,310 bytes allocated
==11241==
==11241== LEAK SUMMARY:
==11241==    definitely lost: 0 bytes in 0 blocks
==11241==    indirectly lost: 0 bytes in 0 blocks
==11241==      possibly lost: 1,152 bytes in 4 blocks
==11241==    still reachable: 48 bytes in 2 blocks
==11241==         suppressed: 0 bytes in 0 blocks
==11241== Rerun with --leak-check=full to see details of leaked memory
==11241==
==11241== For counts of detected and suppressed errors, rerun with: -v
==11241== Use --track-origins=yes to see where uninitialised values come from
==11241== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

(忽略"可能丢失"的块;它们是由Go运行时启动的pthreads。)

你期望看到什么?

没有基于未初始化的值的条件跳转/移动。

你看到了什么?

一个基于未初始化的值的条件跳转/移动。
如果你用 --partial-loads-ok=no 运行Valgrind,问题的本质会更加明显:

$ valgrind --partial-loads-ok=no ./sscce
==11376== Memcheck, a memory error detector
==11376== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==11376== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==11376== Command: ./sscce
==11376==
==11376== Warning: ignored attempt to set SIGRT32 handler in sigaction();
==11376==          the SIGRT32 signal is used internally by Valgrind
==11376== Warning: ignored attempt to set SIGRT32 handler in sigaction();
==11376==          the SIGRT32 signal is used internally by Valgrind
==11376== Warning: client switching stacks?  SP change: 0xfff0001b0 --> 0xc0000367d8
==11376==          to suppress, use: --max-stackframe=755931244072 or greater
==11376== Warning: client switching stacks?  SP change: 0xc000036790 --> 0xfff000260
==11376==          to suppress, use: --max-stackframe=755931243824 or greater
==11376== Warning: client switching stacks?  SP change: 0xfff000260 --> 0xc000036790
==11376==          to suppress, use: --max-stackframe=755931243824 or greater
==11376==          further instances of this message will not be shown.
==11376== Invalid read of size 32
==11376==    at 0x40264E: indexbytebody (/usr/local/go/src/internal/bytealg/indexbyte_amd64.s:148)
==11376==  Address 0x53f47c0 is 0 bytes inside a block of size 12 alloc'd
==11376==    at 0x4C2BBAF: malloc (vg_replace_malloc.c:299)
==11376==    by 0x45165D: s (main.go:7)
==11376==    by 0x4516A5: _cgo_a004886745c9_Cfunc_s (cgo-gcc-prolog:54)
==11376==    by 0x44A0DF: runtime.asmcgocall (/usr/local/go/src/runtime/asm_amd64.s:637)
==11376==    by 0x7: ???
==11376==    by 0x6C287F: ??? (in /home/kivikakk/sscce/sscce)
==11376==    by 0xFFF00024F: ???
==11376==    by 0x4462B1: runtime.(*mcache).nextFree.func1 (/usr/local/go/src/runtime/malloc.go:749)
==11376==    by 0x448905: runtime.systemstack (/usr/local/go/src/runtime/asm_amd64.s:351)
==11376==    by 0x4283BF: ??? (/usr/local/go/src/runtime/proc.go:1146)
==11376==    by 0x448798: runtime.rt0_go (/usr/local/go/src/runtime/asm_amd64.s:201)
==11376==    by 0x451DEF: ??? (in /home/kivikakk/sscce/sscce)
==11376==
==11376==
==11376== HEAP SUMMARY:
==11376==     in use at exit: 1,200 bytes in 6 blocks
==11376==   total heap usage: 10 allocs, 4 frees, 1,316 bytes allocated
==11376==
==11376== LEAK SUMMARY:
==11376==    definitely lost: 0 bytes in 0 blocks
==11376==    indirectly lost: 0 bytes in 0 blocks
==11376==      possibly lost: 1,152 bytes in 4 blocks
==11376==    still reachable: 48 bytes in 2 blocks
==11376==         suppressed: 0 bytes in 0 blocks
==11376== Rerun with --leak-check=full to see details of leaked memory
==11376==
==11376== For counts of detected and suppressed errors, rerun with: -v
==11376== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

我知道已经做了一些工作,以确保 IndexByte 不会跨越页面边界( #24206 ),因此这不太可能产生负面影响。我应该只是添加一个抑制并就此了事吗?

{
   indexbytebody_loves_to_read
   Memcheck:Addr32
   fun:indexbytebody
}
ua4mk5z4

ua4mk5z41#

感谢您提交了一个问题。
cc @TocarIP

zbwhf8kr

zbwhf8kr2#

IndexByte中存在一段代码,用于避免跨页边界加载,这可能会导致运行时检测器(如valgrind)出错。但您看到的错误并不是可能发生的那种情况。它发生在32字节以上的部分,我们从未在传入的范围内读取过外部数据。

产生越界读取的地方是src/runtime/string.go:findnull。我们这样做:

// pageSize is the unit we scan at a time looking for NULL.
// It must be the minimum page size for any architecture Go
// runs on. It's okay (just a minor performance loss) if the
// actual system page size is larger than this value.
const pageSize = 4096

offset := 0
ptr := unsafe.Pointer(s)
// IndexByteString uses wide reads, so we need to be careful
// with page boundaries. Call IndexByteString on
// [ptr, endOfPage) interval.
safeLen := int(pageSize - uintptr(ptr)%pageSize)

for {
	t := *(*string)(unsafe.Pointer(&stringStruct{ptr, safeLen}))
	// Check one page at a time.
	if i := bytealg.IndexByteString(t, 0); i != -1 {
		return offset + i
	}
	// Move to next page
	ptr = unsafe.Pointer(uintptr(ptr) + uintptr(safeLen))
	offset += safeLen
	safeLen = pageSize
}

基本上,我们一次查找一个页面的空值。我们将范围传递给IndexByte,该范围可能超出一个小分配的范围。从虚拟内存的Angular 来看它是安全的,因为我们没有偏离页面,但我肯定能理解它会如何使Valgrind感到困惑。

ki0zmccv

ki0zmccv3#

确实,它在不跑出页面的情况下是安全的,但它确实在读取未初始化的内存(即该部分页面尚未被malloc或类似工具分配给用户空间),而Valgrind正在捕获这一点。我理解它是安全的,我们在这里没有问题;只是遗憾这种事情会出现并玷污Valgrind的结果。
话虽如此,上述提到的pthread“可能丢失”的分配也是Valgrind的噪音。也许我们可以记录抑制/包含一个样本抑制文件?我怀疑还有许多我没有遇到的,尽管如此。

utugiqy6

utugiqy64#

是的,如果我们有一个干净的Valgrind运行,或者有一种方法可以通过配置文件或其他方式诱导一个,那就太好了。我们在Go运行时的一些地方进行了过度读取。通常,过度读取发生在Go堆中,而Valgrind可能并不关心这一点。我认为findnull可能是在C分配的内存上发生的最常见操作。但是,如果有人创建了一个指向C分配内存的字符串或切片,那么在涉及这些字符串/切片的操作中也可能发生这种情况。

C是如何处理这个问题的呢?我想C库函数,如strlen,做的和我们一样的事情。

njthzxwz

njthzxwz5#

是的,这很有道理。我还没有深入研究过这个问题,但我确实认为C库也做了同样的事情,而且Valgrind有自己的strlen,它不会像这样陷入困境。(这看起来像是Valgrind在了解这些优化之前发布的报告:https://stackoverflow.com/a/3246124/499609)

ukdjmx9f

ukdjmx9f6#

我认为Go标准库中的所有案例都会被Valgrind选项--partial-loads-ok=yes处理,而且我认为自2015年以来一直是这样。
你能检查一下看看是否有区别吗?你的Valgrind看起来像是2015年的版本。

xuo3flqw

xuo3flqw7#

我使用的是3.12.0版本(2016年10月),刚刚尝试了最新的稳定版本3.13.0,结果仍然是一样的:

$ valgrind --partial-loads-ok=yes ./sscce
==15735== Memcheck, a memory error detector
==15735== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==15735== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==15735== Command: ./sscce
==15735==
==15735== Warning: ignored attempt to set SIGRT32 handler in sigaction();
==15735==          the SIGRT32 signal is used internally by Valgrind
==15735== Warning: ignored attempt to set SIGRT32 handler in sigaction();
==15735==          the SIGRT32 signal is used internally by Valgrind
==15735== Warning: client switching stacks?  SP change: 0x1fff0001f0 --> 0xc0000347d8
==15735==          to suppress, use: --max-stackframe=687211759080 or greater
==15735== Warning: client switching stacks?  SP change: 0xc000034790 --> 0x1fff0002a0
==15735==          to suppress, use: --max-stackframe=687211758832 or greater
==15735== Warning: client switching stacks?  SP change: 0x1fff0002a0 --> 0xc000034790
==15735==          to suppress, use: --max-stackframe=687211758832 or greater
==15735==          further instances of this message will not be shown.
==15735== Conditional jump or move depends on uninitialised value(s)
==15735==    at 0x40265B: indexbytebody (/usr/local/go/src/internal/bytealg/indexbyte_amd64.s:151)
==15735==
==15735==
==15735== HEAP SUMMARY:
==15735==     in use at exit: 1,176 bytes in 5 blocks
==15735==   total heap usage: 10 allocs, 5 frees, 1,316 bytes allocated
==15735==
==15735== LEAK SUMMARY:
==15735==    definitely lost: 0 bytes in 0 blocks
==15735==    indirectly lost: 0 bytes in 0 blocks
==15735==      possibly lost: 1,152 bytes in 4 blocks
==15735==    still reachable: 24 bytes in 1 blocks
==15735==         suppressed: 0 bytes in 0 blocks
==15735== Rerun with --leak-check=full to see details of leaked memory
==15735==
==15735== For counts of detected and suppressed errors, rerun with: -v
==15735== Use --track-origins=yes to see where uninitialised values come from
==15735== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
hfwmuf9z

hfwmuf9z8#

看起来我们不应该仅仅为了让valgrind满意而改变Go库的行为。我们需要一种方法来告诉valgrind,这个特定的未初始化的内存读取是OK的。有没有办法做到这一点?

相关问题