The actual issue here is that the `openssl` program used for generating dh_param.h is the one found in the search path (PATH), and not one found via pkg-config.

That is, by default macOS ships an old OpenSSL 0.9.8 (actually LibreSSL, IIRC).  When one provides a newer one (e.g. installed through brew) with PKG_CONFIG_PATH, the gen_dh_params script will use the first `openssl` executable it finds on PATH, which generates code incompatible with the new API.

A possible solution would be to search an openssl executable in the exec_prefix found by pkg-config, and use it in gen_dh_params.