nginx does not compile with LibreSSL 2.8.0

[Leon Klingele]

LibreSSL (2.8.0) doesn't compile with nginx 1.15.2 (static build). Compilation worked fine with 2.7.4.

cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g -g -O2 -fdebug-prefix-map=/packaging/nginx/upstream=. -fstack-protector-strong -Wformat -Werror=format-security -I src/core -I src/event -I src/event/modules -I src/os/unix -I submodules/pcre -I submodules/libressl/.openssl/include -I objs \
	-o objs/src/event/ngx_event_openssl.o \
	src/event/ngx_event_openssl.c
src/event/ngx_event_openssl.c: In function ‘ngx_ssl_session_cache’:
src/event/ngx_event_openssl.c:2340:43: error: passing argument 2 of ‘SSL_CTX_sess_set_get_cb’ from incompatible pointer type [-Werror=incompatible-pointer-types]
         SSL_CTX_sess_set_get_cb(ssl->ctx, ngx_ssl_get_cached_session);
                                           ^~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from src/event/ngx_event_openssl.h:15:0,
                 from src/core/ngx_core.h:83,
                 from src/event/ngx_event_openssl.c:9:
submodules/libressl/.openssl/include/openssl/ssl.h:729:6: note: expected ‘SSL_SESSION * (*)(struct ssl_st *, const unsigned char *, int,  int *) {aka struct ssl_session_st * (*)(struct ssl_st *, const unsigned char *, int,  int *)}’ but argument is of type ‘SSL_SESSION * (*)(SSL *, u_char *, int,  int *) {aka struct ssl_session_st * (*)(struct ssl_st *, unsigned char *, int,  int *)}’
 void SSL_CTX_sess_set_get_cb(SSL_CTX *ctx,
      ^~~~~~~~~~~~~~~~~~~~~~~

See ​https://github.com/libressl-portable/portable/issues/435#issuecomment-410455246 for an explanation of the issue.

Bug report on the LibreSSL repo: ​https://github.com/libressl-portable/portable/issues/435

[Maxim Dounin]

Thank you for reporting this. Please try the following patch:

# HG changeset patch
# User Maxim Dounin <mdounin@mdounin.ru>
# Date 1533571015 -10800
#      Mon Aug 06 18:56:55 2018 +0300
# Node ID 635a911ea3de7cdbc918d18591eaa1f32c77ec4c
# Parent  b43ff9b0dfb601bc1f59384fb28ba1e3a15d210b
SSL: fixed build with LibreSSL 2.8.0 (ticket #1605).

LibreSSL 2.8.0 "added const annotations to many existing APIs from OpenSSL,
making interoperability easier for downstream applications".  This includes
the const change in the SSL_CTX_sess_set_get_cb() callback function (see
9dd43f4ef67e), which breaks compilation.

To fix this, added a condition on how we redefine OPENSSL_VERSION_NUMBER
when using working with LibreSSL (see 382fc7069e3a).  With LibreSSL 2.8.0,
we now set OPENSSL_VERSION_NUMBER to 0x1010000fL (OpenSSL 1.1.0), so the
appropriate conditions in the code will use "const" as it happens with
OpenSSL 1.1.0 and later versions.

diff --git a/src/event/ngx_event_openssl.h b/src/event/ngx_event_openssl.h
--- a/src/event/ngx_event_openssl.h
+++ b/src/event/ngx_event_openssl.h
@@ -36,8 +36,12 @@
 
 #if (defined LIBRESSL_VERSION_NUMBER && OPENSSL_VERSION_NUMBER == 0x20000000L)
 #undef OPENSSL_VERSION_NUMBER
+#if (LIBRESSL_VERSION_NUMBER >= 0x2080000fL)
+#define OPENSSL_VERSION_NUMBER  0x1010000fL
+#else
 #define OPENSSL_VERSION_NUMBER  0x1000107fL
 #endif
+#endif
 
 
 #if (OPENSSL_VERSION_NUMBER >= 0x10100001L)

[Leon Klingele]

Works fine, land it!

[Maxim Dounin <mdounin@…>]

In 7337:cab37803ebb3/nginx:

SSL: fixed build with LibreSSL 2.8.0 (ticket #1605).

LibreSSL 2.8.0 "added const annotations to many existing APIs from OpenSSL,
making interoperability easier for downstream applications". This includes
the const change in the SSL_CTX_sess_set_get_cb() callback function (see
9dd43f4ef67e), which breaks compilation.

To fix this, added a condition on how we redefine OPENSSL_VERSION_NUMBER
when working with LibreSSL (see 382fc7069e3a). With LibreSSL 2.8.0,
we now set OPENSSL_VERSION_NUMBER to 0x1010000fL (OpenSSL 1.1.0), so the
appropriate conditions in the code will use "const" as it happens with
OpenSSL 1.1.0 and later versions.

[Maxim Dounin]

Committed, thanks.

[Maxim Dounin <mdounin@…>]

In 7415:2cf1d945bbb3/nginx:

SSL: fixed build with LibreSSL 2.8.0 (ticket #1605).

LibreSSL 2.8.0 "added const annotations to many existing APIs from OpenSSL,
making interoperability easier for downstream applications". This includes
the const change in the SSL_CTX_sess_set_get_cb() callback function (see
9dd43f4ef67e), which breaks compilation.

To fix this, added a condition on how we redefine OPENSSL_VERSION_NUMBER
when working with LibreSSL (see 382fc7069e3a). With LibreSSL 2.8.0,
we now set OPENSSL_VERSION_NUMBER to 0x1010000fL (OpenSSL 1.1.0), so the
appropriate conditions in the code will use "const" as it happens with
OpenSSL 1.1.0 and later versions.