[release-branch.go1.20] internal/testpty: fix error handling

When calling a c library function, you discover that an error has
occurred, typically by looking at the return value of the function. Only
after that can you use errno to figure out the cause of the error.

Nothing about cgo changes that story -- you still have to look at the
result before checking the error that represents errno. If not you can
get false errors if the function happens to leak a non-zero errno.

Fix testpty to check errors correctly.

Fixes #58942.

Change-Id: Idb95f8dd6a8ed63f653190c2e722e742cf50542b
Reviewed-on: https://go-review.googlesource.com/c/go/+/463397
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
(cherry picked from commit f85c282a18bbe7197ba645fff58ba5e0065962ca)
Reviewed-on: https://go-review.googlesource.com/c/go/+/474616
Reviewed-by: Carlos Amedee <carlos@golang.org>

1 files changed, 3 insertions, 3 deletions

diff --git a/src/internal/testpty/pty_cgo.go b/src/internal/testpty/pty_cgo.go
index 1db6a925af..442fbcf618 100644
--- a/src/internal/testpty/pty_cgo.go
+++ b/src/internal/testpty/pty_cgo.go
@@ -18,14 +18,14 @@ import "os"
 
 func open() (pty *os.File, processTTY string, err error) {
 	m, err := C.posix_openpt(C.O_RDWR)
-	if err != nil {
+	if m < 0 {
 		return nil, "", ptyError("posix_openpt", err)
 	}
-	if _, err := C.grantpt(m); err != nil {
+	if res, err := C.grantpt(m); res < 0 {
 		C.close(m)
 		return nil, "", ptyError("grantpt", err)
 	}
-	if _, err := C.unlockpt(m); err != nil {
+	if res, err := C.unlockpt(m); res < 0 {
 		C.close(m)
 		return nil, "", ptyError("unlockpt", err)
 	}