Issue compiling with openSSL1.1 on Mac:

In file included from ./tls.c:84:
./dh_params.h:33:4: error: incomplete definition of type 'struct dh_st'
        dh->p = BN_bin2bn(dh2048_p, sizeof(dh2048_p), NULL);
        ~~^
/usr/local/opt/[email protected]/include/openssl/ossl_typ.h:104:16: note: forward declaration of 'struct dh_st'
typedef struct dh_st DH;
               ^
In file included from ./tls.c:84:
./dh_params.h:34:4: error: incomplete definition of type 'struct dh_st'
        dh->g = BN_bin2bn(dh2048_g, sizeof(dh2048_g), NULL);
        ~~^
/usr/local/opt/[email protected]/include/openssl/ossl_typ.h:104:16: note: forward declaration of 'struct dh_st'
typedef struct dh_st DH;
               ^
In file included from ./tls.c:84:
./dh_params.h:35:9: error: incomplete definition of type 'struct dh_st'
        if ((dh->p == NULL) || (dh->g == NULL))
             ~~^
/usr/local/opt/[email protected]/include/openssl/ossl_typ.h:104:16: note: forward declaration of 'struct dh_st'
typedef struct dh_st DH;
               ^
In file included from ./tls.c:84:
./dh_params.h:35:28: error: incomplete definition of type 'struct dh_st'
        if ((dh->p == NULL) || (dh->g == NULL))
                                ~~^
/usr/local/opt/[email protected]/include/openssl/ossl_typ.h:104:16: note: forward declaration of 'struct dh_st'
typedef struct dh_st DH;


Fixed by updating the dh_params.h to use the getter/setter methods (used DH_set0_pqg instead of dh->p and dh->g):

	if ((dh = DH_new()) == NULL) return(NULL);
	DH_set0_pqg(dh, BN_bin2bn(dh2048_p, sizeof(dh2048_p), NULL), NULL, BN_bin2bn(dh2048_g, sizeof(dh2048_g), NULL));
	if ((DH_get0_p(dh) == NULL) || (DH_get0_g(dh) == NULL))
		{ DH_free(dh); return(NULL); }

The actual issue here is that the `openssl` program used for generating dh_param.h is the one found in the search path (PATH), and not one found via pkg-config.

That is, by default macOS ships an old OpenSSL 0.9.8 (actually LibreSSL, IIRC).  When one provides a newer one (e.g. installed through brew) with PKG_CONFIG_PATH, the gen_dh_params script will use the first `openssl` executable it finds on PATH, which generates code incompatible with the new API.

A possible solution would be to search an openssl executable in the exec_prefix found by pkg-config, and use it in gen_dh_params.

The comment by medranocalvo saved me here!

Considering the very high degree of redundancy in the gen_dh_params script:
 1. call openssl executable,
 2. download parameters from the internet,
 3. use precomputed parameter values embedded in the script;
it seems ridiculous that it does not check whether the openssl executable 
option 1 picks has a suitable version. Option 3 *does* account for the API 
differences (has more bitsizes, and is likely way faster, so why is that 
not preferred?).

This issue is now obsolete with the change to use the built-in DH support in OpenSSL per commit [d3d16ea77fa810d8].