tool_cb_wrt: Silence function cast compiler warning

Commit 5bfaa86ceb3c2a9ac474a928e748c4a86a703b33 introduced a new
compiler warning on Windows cross compilation with GCC. See below
for an example of the warning from the autobuild logs (whitespace
edited to fit):

/src/tool_cb_wrt.c:175:9: warning: cast from function call of type
    'intptr_t {aka long long int}' to non-matching type 'void *'
    [-Wbad-function-cast]
(HANDLE) _get_osfhandle(fileno(outs->stream)),
^

Store the return value from _get_osfhandle() in an intermediate
variable and cast the variable in WriteConsoleW() rather than the
function call directly to avoid a compiler warning.

In passing, also add inspection of the MultiByteToWideChar() return
value and return failure in case an error is reported.

Closes #3263
Reviewed-by: Marcel Raad <Marcel.Raad@teamviewer.com>
Reviewed-by: Viktor Szakats <commit@vszakats.net>

1 files changed, 8 insertions, 1 deletions

diff --git a/src/tool_cb_wrt.c b/src/tool_cb_wrt.c
index 0e74392ac..2cb5e1b41 100644
--- a/src/tool_cb_wrt.c
+++ b/src/tool_cb_wrt.c
@@ -161,6 +161,7 @@ size_t tool_write_cb(char *buffer, size_t sz, size_t nmemb, void *userdata)
     DWORD in_len = (DWORD)(sz * nmemb);
     wchar_t* wc_buf;
     DWORD wc_len;
+    intptr_t fhnd;
 
     /* calculate buffer size for wide characters */
     wc_len = MultiByteToWideChar(CP_UTF8, 0, buffer, in_len,  NULL, 0);
@@ -170,9 +171,15 @@ size_t tool_write_cb(char *buffer, size_t sz, size_t nmemb, void *userdata)
 
     /* calculate buffer size for multi-byte characters */
     wc_len = MultiByteToWideChar(CP_UTF8, 0, buffer, in_len, wc_buf, wc_len);
+    if(!wc_len) {
+      free(wc_buf);
+      return failure;
+    }
+
+    fhnd = _get_osfhandle(fileno(outs->stream));
 
     if(!WriteConsoleW(
-        (HANDLE) _get_osfhandle(fileno(outs->stream)),
+        (HANDLE) fhnd,
         wc_buf,
         wc_len,
         &wc_len,