Nginx compile error with OpenSSL 3 on Msys2 (mingw64) Windows

[pavlusha23@…]

Nginx compile error with OpenSSL 3 on Msys2 (mingw64) Windows. With OpenSSL 1.1.1 compilation was successful.

Environment:

Windows 10
Msys2 (mingw64)
Nginx 1.23.1
OpenSSL 3.0.5

Error:

-o objs/src/http/ngx_http_special_response.o \
        src/http/ngx_http_special_response.c
src/event/ngx_event_openssl.c: In function 'ngx_ssl_sendfile':
src/event/ngx_event_openssl.c:2972:52: error: passing argument 2 of 'SSL_sendfile' makes integer from pointer without a cast [-Werror=int-conversion]
 2972 |     n = SSL_sendfile(c->ssl->connection, file->file->fd, file->file_pos,
      |                                          ~~~~~~~~~~^~~~
      |                                                    |
      |                                                    ngx_fd_t {aka void *}
In file included from src/event/ngx_event_openssl.h:17,
                 from src/core/ngx_core.h:84,
                 from src/event/ngx_event_openssl.c:9:
openssl-3.0.5/.openssl/include/openssl/ssl.h:1970:46: note: expected 'int' but argument is of type 'ngx_fd_t' {aka 'void *'}
 1970 | __owur ossl_ssize_t SSL_sendfile(SSL *s, int fd, off_t offset, size_t size,
      |                                          ~~~~^~

With this fix, the compilation was successful. Change this code

    n = SSL_sendfile(c->ssl->connection, file->file->fd, file->file_pos,

to

    n = SSL_sendfile(c->ssl->connection, (int)(uintptr_t) file->file->fd, file->file_pos,

in file ".\nginx\src\event\ngx_event_openssl.c"

[Maxim Dounin]

For the record, building OpenSSL 3.0.5 on Windows as part of nginx release process currently fails with the following error:

        "c:\Strawberry\perl\bin\perl.exe" "util\mkdef.pl" --type dso --ordinals util\providers.num --name providers\legacy --OS windows > providers\legacy.def
        IF EXIST providers\legacy.dll.manifest DEL /F /Q providers\legacy.dll.manifest
        cmd /C ""link" /nologo /debug /dll  /nologo /debug @C:/Users/mdounin/AppData/Local/Temp\nm823D.tmp || (DEL /Q legacy.* providers\legacy.* & EXIT 1)"
providers\legacy.def(5) : warning LNK4093: Drive/Directory component ignored in 'LIBRARY' statement
   Creating library providers\legacy.lib and object providers\legacy.exp
libcrypto.lib(libcrypto-lib-threads_win.obj) : error LNK2019: unresolved external symbol _InterlockedOr64 referenced in function _CRYPTO_atomic_or
providers\legacy.dll : fatal error LNK1120: 1 unresolved externals
Could Not Find c:\nginx\nginx\objs.msvc8\lib\openssl-3.0.5\legacy.*
NMAKE : fatal error U1077: 'cmd' : return code '0x1'

This seems to be ​this issue, though currently committed fix seems to be incomplete and fails to address build with MSVC more recent than 2008 but with _WIN32_WINNT explicitly set to support Windows XP (see 7da71a7b141a).

[Maxim Dounin]

Thanks for the report.

Indeed, nginx uses OS file handles to work with files on Windows, and the SSL_sendfile() interface is not compatible with this, since it expects C integer file descriptor instead.

Casting the file handle to int does not look like a good solution to me though: while it should work as long as SSL_sendfile() is not used, OS file handles and C file descriptors are different things and casting one into another is not possible (see ​_open_oshandle()). A better solution would be to explicitly disable the code on Windows, patch below.

# HG changeset patch
# User Maxim Dounin <mdounin@mdounin.ru>
# Date 1661827232 -10800
#      Tue Aug 30 05:40:32 2022 +0300
# Node ID 315c665136d5161b7feebee685363cd2c49fbfab
# Parent  c62d55979bbdbd6ac03a4883bd850c7e47739bac
SSL: fixed build on Windows with OpenSSL 3.0.x (ticket #2379).

SSL_sendfile() expects integer file descriptor as an argument, but nginx
uses OS file handles (HANDLE) to work with files on Windows, and passing
HANDLE instead of an integer correctly results in build failure.  Since
SSL_sendfile() is not expected to work on Windows anyway, the code is now
disabled on Windows with appropriate compile-time checks.

diff -r c62d55979bbd -r 315c665136d5 src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c	Tue Aug 30 05:32:56 2022 +0300
+++ b/src/event/ngx_event_openssl.c	Tue Aug 30 05:40:32 2022 +0300
@@ -1770,7 +1770,7 @@ ngx_ssl_handshake(ngx_connection_t *c)
 #endif
 #endif
 
-#ifdef BIO_get_ktls_send
+#if (defined BIO_get_ktls_send && !NGX_WIN32)
 
         if (BIO_get_ktls_send(SSL_get_wbio(c->ssl->connection)) == 1) {
             ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
@@ -1915,7 +1915,7 @@ ngx_ssl_try_early_data(ngx_connection_t 
         c->read->ready = 1;
         c->write->ready = 1;
 
-#ifdef BIO_get_ktls_send
+#if (defined BIO_get_ktls_send && !NGX_WIN32)
 
         if (BIO_get_ktls_send(SSL_get_wbio(c->ssl->connection)) == 1) {
             ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
@@ -2944,7 +2944,7 @@ ngx_ssl_write_early(ngx_connection_t *c,
 static ssize_t
 ngx_ssl_sendfile(ngx_connection_t *c, ngx_buf_t *file, size_t size)
 {
-#ifdef BIO_get_ktls_send
+#if (defined BIO_get_ktls_send && !NGX_WIN32)
 
     int        sslerr, flags;
     ssize_t    n;

[Maxim Dounin <mdounin@…>]

In 8068:0546ab9351c8/nginx:

Win32: fixed build on Windows with OpenSSL 3.0.x (ticket #2379).

SSL_sendfile() expects integer file descriptor as an argument, but nginx
uses OS file handles (HANDLE) to work with files on Windows, and passing
HANDLE instead of an integer correctly results in build failure. Since
SSL_sendfile() is not expected to work on Windows anyway, the code is now
disabled on Windows with appropriate compile-time checks.

[Sergey Kandaurov]

Fix committed, thanks.