From patchwork Wed Feb 22 23:15:45 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: sfan5 X-Patchwork-Id: 2653 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.65.149 with SMTP id x21csp1107894vsf; Wed, 22 Feb 2017 15:16:00 -0800 (PST) X-Received: by 10.28.155.5 with SMTP id d5mr447687wme.85.1487805360137; Wed, 22 Feb 2017 15:16:00 -0800 (PST) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 49si3499996wrx.326.2017.02.22.15.15.59; Wed, 22 Feb 2017 15:16:00 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 232B8688337; Thu, 23 Feb 2017 01:15:49 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from BAY004-OMC3S18.hotmail.com (bay004-omc3s18.hotmail.com [65.54.190.156]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C4EC1688309 for ; Thu, 23 Feb 2017 01:15:41 +0200 (EET) Received: from EUR02-AM5-obe.outbound.protection.outlook.com ([65.54.190.187]) by BAY004-OMC3S18.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008); Wed, 22 Feb 2017 15:15:48 -0800 Received: from HE1EUR02FT030.eop-EUR02.prod.protection.outlook.com (10.152.10.60) by HE1EUR02HT028.eop-EUR02.prod.protection.outlook.com (10.152.11.44) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.919.10; Wed, 22 Feb 2017 23:15:46 +0000 Received: from VI1PR0201MB2414.eurprd02.prod.outlook.com (10.152.10.60) by HE1EUR02FT030.mail.protection.outlook.com (10.152.10.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.919.10 via Frontend Transport; Wed, 22 Feb 2017 23:15:45 +0000 Received: from VI1PR0201MB2414.eurprd02.prod.outlook.com ([10.168.64.10]) by VI1PR0201MB2414.eurprd02.prod.outlook.com ([10.168.64.10]) with mapi id 15.01.0919.018; Wed, 22 Feb 2017 23:15:45 +0000 From: Stefan _ To: FFmpeg development discussions and patches Thread-Topic: [PATCH] configure/rtmpdh: Fix OpenSSL 1.1.0 support Thread-Index: AQHSjWGYx7qxbH/COUerXcobwDpQMQ== Date: Wed, 22 Feb 2017 23:15:45 +0000 Message-ID: References: <6ad03f60-ff4f-2490-5f60-4ef91b9900fc@live.de> In-Reply-To: <6ad03f60-ff4f-2490-5f60-4ef91b9900fc@live.de> Accept-Language: de-DE, en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: authentication-results: ffmpeg.org; dkim=none (message not signed) header.d=none; ffmpeg.org; dmarc=none action=none header.from=live.de; x-incomingtopheadermarker: OriginalChecksum:641DE045272A1968C4DC8025C0C8C883257E7BA126C5D1A5535C4FDA135E0A7A; UpperCasedChecksum:F2FAD12F208E62E339A863936C0849038A003788450E91213DB3A57BA31B45D1; SizeAsReceived:7582; Count:36 x-ms-exchange-messagesentrepresentingtype: 1 x-incomingheadercount: 36 x-eopattributedmessage: 0 x-microsoft-exchange-diagnostics: 1; HE1EUR02HT028; 5:0lZO2cPdhUWf7lT4KubJ8kan6mwggrWgD/0paXz4Sk8vaIwq3eYhfCJQHIVDnNfrpITVkdNPdlL2bh9hWyTJMnm3kFuRO7qDiIqg6F8W3HnYJs2aEdMqS+DYrJuTLttouYByWl8SyIIepXir0zimYQ==; 24:eufkz0pK8wRl5pBF9+sqf90wkCpwxSW2ei+azYIao2vu6MR7KYjBbb3ckueTTR6BbONS7QEK5HXWMAwrXI76tcC7dUJRw4jEAhbzsyDc/oY=; 7:FRNWUXoxpYKIdCX5mu3rUeyT/sBgyU6pi5jxFN1bM8jezfJGPoe3viXGFH3nTz1zflx4b3pk35bdvsjpRp0/UCbXjTxYAVUrBAz8mM9Zzxn9vrDT9B8WhN518sNuJ5qXyBK9Lk1AFK5qHyhxK/hPXNZNMCIoppACT49x33Z5dG9aGjKEakJOqk+Queq6U0R5UtL73TIvLglaPOxk+rwqR2bGV4zEbDwNXk9IbjZqJuwsyH3FwPviINMAtSJ1308DPVb/oOXLGPRqvUfNRE3CAnhYpGZI1RBtI3oAvtfBWx1Lu123IX744hQJtglRuXES x-forefront-antispam-report: EFV:NLI; SFV:NSPM; SFS:(10019020)(98900012); DIR:OUT; SFP:1102; SCL:1; SRVR:HE1EUR02HT028; H:VI1PR0201MB2414.eurprd02.prod.outlook.com; FPR:; SPF:None; MLV:ovrnspm; LANG:en; x-ms-office365-filtering-correlation-id: a4dd375b-e6f1-4fa4-5d42-08d45b78ba63 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(201702061074)(5061506569)(5061507331)(1603103135)(1601125250)(1701031045); SRVR:HE1EUR02HT028; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(432015087)(444000031); SRVR:HE1EUR02HT028; BCL:0; PCL:0; RULEID:; SRVR:HE1EUR02HT028; x-forefront-prvs: 022649CC2C spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Feb 2017 23:15:45.6467 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1EUR02HT028 X-OriginalArrivalTime: 22 Feb 2017 23:15:48.0364 (UTC) FILETIME=[9A15F4C0:01D28D61] Subject: Re: [FFmpeg-devel] [PATCH] configure/rtmpdh: Fix OpenSSL 1.1.0 support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" The rtmpdh patch was kinda (completly) broken, fixed patch is attached. Concerning LibreSSL: They use OPENSSL_VERSION_NUMBER = 0x2050200fL which breaks the other openssl code in tls_openssl.c anyway, so LibreSSL support should probably be worried about in a different patch. From 0edf1d179824fa6400031c6d05b3f464f01abf36 Mon Sep 17 00:00:00 2001 From: sfan5 Date: Wed, 22 Feb 2017 15:38:16 +0100 Subject: [PATCH 2/2] rtmpdh: Stop using OpenSSL-provided DH functions to support 1.1.0 DH (struct dh_st) was made private in the 1.1 series, instead DH is now done the same way as with gcrypt / libgmp. --- libavformat/rtmpdh.c | 94 ++++++++++++++++++++++++---------------------------- libavformat/rtmpdh.h | 13 +++----- 2 files changed, 49 insertions(+), 58 deletions(-) diff --git a/libavformat/rtmpdh.c b/libavformat/rtmpdh.c index 1876fd44f9..1ec1286d23 100644 --- a/libavformat/rtmpdh.c +++ b/libavformat/rtmpdh.c @@ -54,7 +54,6 @@ "F71C35FDAD44CFD2D74F9208BE258FF324943328F67329C0" \ "FFFFFFFFFFFFFFFF" -#if CONFIG_GMP || CONFIG_GCRYPT #if CONFIG_GMP #define bn_new(bn) \ do { \ @@ -93,7 +92,11 @@ else \ ret = 1; \ } while (0) -#define bn_modexp(bn, y, q, p) mpz_powm(bn, y, q, p) +#define bn_modexp(bn, y, q, p, ret) \ + do { \ + mpz_powm(bn, y, q, p); \ + ret = 1; \ + } while(0) #define bn_random(bn, num_bits) \ do { \ int bits = num_bits; \ @@ -125,8 +128,34 @@ #define bn_bn2bin(bn, buf, len) gcry_mpi_print(GCRYMPI_FMT_USG, buf, len, NULL, bn) #define bn_bin2bn(bn, buf, len) gcry_mpi_scan(&bn, GCRYMPI_FMT_USG, buf, len, NULL) #define bn_hex2bn(bn, buf, ret) ret = (gcry_mpi_scan(&bn, GCRYMPI_FMT_HEX, buf, 0, 0) == 0) -#define bn_modexp(bn, y, q, p) gcry_mpi_powm(bn, y, q, p) +#define bn_modexp(bn, y, q, p, ret) \ + do { \ + cry_mpi_powm(bn, y, q, p); \ + ret = 1; \ + } while (0) #define bn_random(bn, num_bits) gcry_mpi_randomize(bn, num_bits, GCRY_WEAK_RANDOM) +#elif CONFIG_OPENSSL +#define bn_new(bn) bn = BN_new() +#define bn_free(bn) BN_free(bn) +#define bn_set_word(bn, w) BN_set_word(bn, w) +#define bn_cmp(a, b) BN_cmp(a, b) +#define bn_copy(to, from) BN_copy(to, from) +#define bn_sub_word(bn, w) BN_sub_word(bn, w) +#define bn_cmp_1(bn) BN_cmp(bn, BN_value_one()) +#define bn_num_bytes(bn) BN_num_bytes(bn) +#define bn_bn2bin(bn, buf, len) BN_bn2bin(bn, buf) +#define bn_bin2bn(bn, buf, len) bn = BN_bin2bn(buf, len, 0) +#define bn_hex2bn(bn, buf, ret) ret = BN_hex2bn(&bn, buf) +#define bn_modexp(bn, y, q, p, ret) \ + do { \ + BN_CTX *ctx = BN_CTX_new(); \ + if (!ctx) \ + ret = 0; \ + else \ + ret = BN_mod_exp(bn, y, q, p, ctx); \ + BN_CTX_free(ctx); \ + } while (0) +#define bn_random(bn, num_bits) BN_rand(bn, num_bits, 0, 0) #endif #define MAX_BYTES 18000 @@ -135,7 +164,7 @@ static FFBigNum dh_generate_key(FF_DH *dh) { - int num_bytes; + int num_bytes, ret; num_bytes = bn_num_bytes(dh->p) - 1; if (num_bytes <= 0 || num_bytes > MAX_BYTES) @@ -152,7 +181,9 @@ static FFBigNum dh_generate_key(FF_DH *dh) return NULL; } - bn_modexp(dh->pub_key, dh->g, dh->priv_key, dh->p); + bn_modexp(dh->pub_key, dh->g, dh->priv_key, dh->p, ret); + if (!ret) + return NULL; return dh->pub_key; } @@ -161,12 +192,15 @@ static int dh_compute_key(FF_DH *dh, FFBigNum pub_key_bn, uint32_t secret_key_len, uint8_t *secret_key) { FFBigNum k; + int ret; bn_new(k); if (!k) return -1; - bn_modexp(k, pub_key_bn, dh->priv_key, dh->p); + bn_modexp(k, pub_key_bn, dh->priv_key, dh->p, ret); + if (!ret) + return -1; bn_bn2bin(k, secret_key, secret_key_len); bn_free(k); @@ -184,53 +218,11 @@ void ff_dh_free(FF_DH *dh) bn_free(dh->priv_key); av_free(dh); } -#elif CONFIG_OPENSSL -#define bn_new(bn) bn = BN_new() -#define bn_free(bn) BN_free(bn) -#define bn_set_word(bn, w) BN_set_word(bn, w) -#define bn_cmp(a, b) BN_cmp(a, b) -#define bn_copy(to, from) BN_copy(to, from) -#define bn_sub_word(bn, w) BN_sub_word(bn, w) -#define bn_cmp_1(bn) BN_cmp(bn, BN_value_one()) -#define bn_num_bytes(bn) BN_num_bytes(bn) -#define bn_bn2bin(bn, buf, len) BN_bn2bin(bn, buf) -#define bn_bin2bn(bn, buf, len) bn = BN_bin2bn(buf, len, 0) -#define bn_hex2bn(bn, buf, ret) ret = BN_hex2bn(&bn, buf) -#define bn_modexp(bn, y, q, p) \ - do { \ - BN_CTX *ctx = BN_CTX_new(); \ - if (!ctx) \ - return AVERROR(ENOMEM); \ - if (!BN_mod_exp(bn, y, q, p, ctx)) { \ - BN_CTX_free(ctx); \ - return AVERROR(EINVAL); \ - } \ - BN_CTX_free(ctx); \ - } while (0) - -#define dh_new() DH_new() -#define dh_generate_key(dh) DH_generate_key(dh) - -static int dh_compute_key(FF_DH *dh, FFBigNum pub_key_bn, - uint32_t secret_key_len, uint8_t *secret_key) -{ - if (secret_key_len < DH_size(dh)) - return AVERROR(EINVAL); - return DH_compute_key(secret_key, pub_key_bn, dh); -} - -void ff_dh_free(FF_DH *dh) -{ - if (!dh) - return; - DH_free(dh); -} -#endif static int dh_is_valid_public_key(FFBigNum y, FFBigNum p, FFBigNum q) { FFBigNum bn = NULL; - int ret = AVERROR(EINVAL); + int ret = AVERROR(EINVAL), ret2; bn_new(bn); if (!bn) @@ -254,7 +246,9 @@ static int dh_is_valid_public_key(FFBigNum y, FFBigNum p, FFBigNum q) * random data. */ /* y must fulfill y^q mod p = 1 */ - bn_modexp(bn, y, q, p); + bn_modexp(bn, y, q, p, ret2); + if (!ret2) + goto fail; if (bn_cmp_1(bn)) goto fail; diff --git a/libavformat/rtmpdh.h b/libavformat/rtmpdh.h index 2b250f595d..3f01e6b17b 100644 --- a/libavformat/rtmpdh.h +++ b/libavformat/rtmpdh.h @@ -26,7 +26,6 @@ #include "config.h" -#if CONFIG_GMP || CONFIG_GCRYPT #if CONFIG_GMP #include @@ -35,6 +34,11 @@ typedef mpz_ptr FFBigNum; #include typedef gcry_mpi_t FFBigNum; + +#elif CONFIG_OPENSSL +#include + +typedef BIGNUM *FFBigNum; #endif typedef struct FF_DH { @@ -45,13 +49,6 @@ typedef struct FF_DH { long length; } FF_DH; -#elif CONFIG_OPENSSL -#include -#include - -typedef BIGNUM *FFBigNum; -typedef DH FF_DH; -#endif /** * Initialize a Diffie-Hellmann context. -- 2.11.1