From patchwork Thu Apr 25 14:23:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derek Buitenhuis X-Patchwork-Id: 48259 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:2d06:b0:1a9:af23:56c1 with SMTP id tw6csp1418462pzb; Thu, 25 Apr 2024 07:24:12 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUrCfudUhbhMq1jQN4dwSpWR2caKz3SRgjSbR+VQsng3u8QGonuMhYY0WrTghJtnDeTTxtZ8w5xKdxTXQQhlhBZXbIfgeChwHfKYQ== X-Google-Smtp-Source: AGHT+IH8WfGXIOM5ErcsXKjPfZV5mh4dmXd4O22h4lkK+GpDrXoLSm4m3LPU2NynNJN9LWLil+Xn X-Received: by 2002:a05:6512:6b:b0:51c:7a01:6ea2 with SMTP id i11-20020a056512006b00b0051c7a016ea2mr598305lfo.6.1714055052541; Thu, 25 Apr 2024 07:24:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714055052; cv=none; d=google.com; s=arc-20160816; b=mSzil1sWEGFJzQp+iy34KPvw/TJS+CGwuOsB8PwQwkRmPHtPLBtemUcJFuZu9VCc/O oL1lZh3QgiSinAbpHFOitaDL393O9M5khDSLmvVOWCteU67R+wdDEFDRwluZUMO3iIe8 cjnCDMHO8ASMrBexc12irsjrQ6zp0QJtpMYmLYzkF0RLvhSMweiGbzJwTn3WCJZGDR50 i6ySmuFs4qsPzvth6b90h1NGda2msyGWnZPL72J4qXlrSb0TJVYYGPK/+HYqtmwQw3lZ FtE+cSOkwaUawd78wi/dFjN72C4qaVPIQZRW6r+3DYoDPwIJX7/2yKZ4oPMrqk0mZL1P LRHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=rQIgFwhAOjZw3mHHe7SrQ+d1IBE896wv9A2wRPy9dws=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=ancSbl38bVF1Mzf2+XFoPZvd9KeNYriJ8vVBqiM9/0z4qm/jXroFzCa2SlbRE1fC14 3AfdQ0yFyOkBf9ateIDHn7dA2auoZURKIV+zgFNOmSSQay995wP/vlLSAgkcgprO7RIb 2iRiusNDp1+kosvYNC21Qh6BUnsCBQ4KAX2rOw9hEZIUGUsVN2PlPuzHW/FDx4EFb6f5 5ocwsHshQWqQNS8zrE+YZCp+OePppMfEfqRBwnVvEd4rzQTpPXQpmxuZzo5VcjVgQuz+ pB9sYu5tCYuyRoMxW+TvsSQl0s7IgoFTTrZezLe7ERZ0JDGqOQfApHQDQGKoS7VPcgDj CmCw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=mcHrt5n4; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d15-20020a05651233cf00b00516ce08e5bfsi5781506lfg.21.2024.04.25.07.24.12; Thu, 25 Apr 2024 07:24:12 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=mcHrt5n4; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 060C368D3EE; Thu, 25 Apr 2024 17:23:52 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6A3CE68D366 for ; Thu, 25 Apr 2024 17:23:44 +0300 (EEST) Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-41b5e74f7d1so3103935e9.3 for ; Thu, 25 Apr 2024 07:23:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714055023; x=1714659823; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=Sq+g+lqKiHDbd3m1Vw8FaI3/y6hUf8a+XoJQl6FkjB4=; b=mcHrt5n4vibOSp54rkpOlYHqLfhFN7MsoCajm2KHF47KCiiP0fKuroRFxOES856HR7 HHlIZx2dykAI26OXH+aX6Gx6jgz65fX/G7iFoRW4ywlGzttRQEMSA5ZwLue/s5nqHx+k fFTEmsa/XcPOMaTAbYCR/y5J97FsZhU0ShH4C731fkX+WC2rwARrTXYpqUM/5tykjSiZ L+uBAZCC9Mpy+WTEzTqi9iDRiqtwOlE+P+mJ6jKPvQFE1ckyNIvlPelAC4wYdQmxqREp cb1qLt7GLZ4p4qBQanAbY1hL4A1mPMB2xgF23LUQMZCbso5GKM0050dz80ZDwKbzZVdz shKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714055023; x=1714659823; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Sq+g+lqKiHDbd3m1Vw8FaI3/y6hUf8a+XoJQl6FkjB4=; b=UkD801jLqSdI2qnBJbd6leNhBVlyZE+kvCd+jtOyNul8Kz9olseBlPGDpKcNJ+3o3a n000vXwh+Wvgh4HFWJgPzNzrv4l5Em1gCY5YiIx8B8AREA8n5gREknT615RjGDvQaOuG UpwfcGRlxnaiSHeBLI/o+4bI+SADevKuOUxNWA9OOCVAkqIy8kOrdkuh37d6oNg1Srcw MF3oYcDGcVpqx5Od4eFrjG15HVdSReHCCxHL+oonma46PVMpY0EX3iNMGxFBqNGccP2+ +Ytyluk6CECH9L47JYBgRAIaDR6JgRt+I0/OSaxkbuKaleRaUswQ6u7Z2FBR8dzP0w/+ F3Og== X-Gm-Message-State: AOJu0YyEO4eYErhucImdFpRe4V0iA7gfJWyipnqV9br20/G3Xfcuhz/P tARHuHVTct3oHqFmBzhtM51kt3kMOdM1UopsD2SBFUMfMdwXi6fDdRdOXQ== X-Received: by 2002:a05:600c:1c16:b0:416:2a95:6e8b with SMTP id j22-20020a05600c1c1600b004162a956e8bmr4583283wms.26.1714055023158; Thu, 25 Apr 2024 07:23:43 -0700 (PDT) Received: from localhost.localdomain (33bf0029.skybroadband.com. [51.191.0.41]) by smtp.gmail.com with ESMTPSA id o12-20020a5d684c000000b0034aa1e534c2sm14582018wrw.96.2024.04.25.07.23.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Apr 2024 07:23:42 -0700 (PDT) From: Derek Buitenhuis To: ffmpeg-devel@ffmpeg.org Date: Thu, 25 Apr 2024 15:23:30 +0100 Message-ID: <20240425142330.288815-3-derek.buitenhuis@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240425142330.288815-1-derek.buitenhuis@gmail.com> References: <20240425142330.288815-1-derek.buitenhuis@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 2/2] avformat/http: Add support for Retry-After header X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: YTHL5w1Id7zR 429 and 503 codes can, and often do (e.g. all Google Cloud Storage URLs can), return a Retry-After header with the error, indicating how long to wait, asd either a date, or in seconds, before retrying again. If it is not respected by, for example, using our default backoff stratetgy instead, chances of success are very unlikely. Some references: * https://datatracker.ietf.org/doc/html/rfc6585 * https://datatracker.ietf.org/doc/html/rfc7231#section-7.1.3 This adds an AVOption to respect that header. Signed-off-by: Derek Buitenhuis --- doc/protocols.texi | 5 +++++ libavformat/http.c | 24 ++++++++++++++++++++++++ libavformat/version.h | 2 +- 3 files changed, 30 insertions(+), 1 deletion(-) diff --git a/doc/protocols.texi b/doc/protocols.texi index 571d78ec43..ed70af4b33 100644 --- a/doc/protocols.texi +++ b/doc/protocols.texi @@ -569,6 +569,11 @@ Set the maximum number of times to retry a connection. Default unset. @item reconnect_delay_total_max Set the maximum total delay in seconds after which to give up reconnecting. +@item respect_retry_after +If enabled, and a Retry-After header is encountered, its requested reconnection +delay will be honored, rather than using exponential backoff. Useful for 429 and +503 errors. Default enabled. + @item listen If set to 1 enables experimental HTTP server. This can be used to send data when used as an output option, or read data from a client with HTTP POST when used as diff --git a/libavformat/http.c b/libavformat/http.c index d324674e97..1a67068a44 100644 --- a/libavformat/http.c +++ b/libavformat/http.c @@ -31,6 +31,7 @@ #include "libavutil/avstring.h" #include "libavutil/bprint.h" #include "libavutil/getenv_utf8.h" +#include "libavutil/macros.h" #include "libavutil/mem.h" #include "libavutil/opt.h" #include "libavutil/time.h" @@ -138,6 +139,8 @@ typedef struct HTTPContext { char *new_location; AVDictionary *redirect_cache; uint64_t filesize_from_content_range; + int respect_retry_after; + unsigned int retry_after; int reconnect_max_retries; int reconnect_delay_total_max; } HTTPContext; @@ -180,6 +183,7 @@ static const AVOption options[] = { { "reconnect_delay_max", "max reconnect delay in seconds after which to give up", OFFSET(reconnect_delay_max), AV_OPT_TYPE_INT, { .i64 = 120 }, 0, UINT_MAX/1000/1000, D }, { "reconnect_max_retries", "the max number of times to retry a connection", OFFSET(reconnect_max_retries), AV_OPT_TYPE_INT, { .i64 = -1 }, -1, INT_MAX, D }, { "reconnect_delay_total_max", "max total reconnect delay in seconds after which to give up", OFFSET(reconnect_delay_total_max), AV_OPT_TYPE_INT, { .i64 = 256 }, 0, UINT_MAX/1000/1000, D }, + { "respect_retry_after", "respect the Retry-After header when retrying connections", OFFSET(respect_retry_after), AV_OPT_TYPE_BOOL, { .i64 = 1 }, 0, 1, D }, { "listen", "listen on HTTP", OFFSET(listen), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 2, D | E }, { "resource", "The resource requested by a client", OFFSET(resource), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, E }, { "reply_code", "The http status code to return to a client", OFFSET(reply_code), AV_OPT_TYPE_INT, { .i64 = 200}, INT_MIN, 599, E}, @@ -393,6 +397,14 @@ redo: reconnect_delay_total > s->reconnect_delay_total_max) goto fail; + /* Both fields here are in seconds. */ + if (s->respect_retry_after && s->retry_after > 0) { + reconnect_delay = s->retry_after; + if (reconnect_delay > s->reconnect_delay_max) + goto fail; + s->retry_after = 0; + } + av_log(h, AV_LOG_WARNING, "Will reconnect at %"PRIu64" in %d second(s).\n", off, reconnect_delay); ret = ff_network_sleep_interruptible(1000U * 1000 * reconnect_delay, &h->interrupt_callback); if (ret != AVERROR(ETIMEDOUT)) @@ -1241,6 +1253,18 @@ static int process_line(URLContext *h, char *line, int line_count, int *parsed_h parse_expires(s, p); } else if (!av_strcasecmp(tag, "Cache-Control")) { parse_cache_control(s, p); + } else if (!av_strcasecmp(tag, "Retry-After")) { + /* The header can be either an integer that represents seconds, or a date. */ + struct tm tm; + int date_ret = parse_http_date(p, &tm); + if (!date_ret) { + time_t retry = av_timegm(&tm); + int64_t now = av_gettime() / 1000000; + int64_t diff = ((int64_t) retry) - now; + s->retry_after = (unsigned int) FFMAX(0, diff); + } else { + s->retry_after = strtoul(p, NULL, 10); + } } } return 1; diff --git a/libavformat/version.h b/libavformat/version.h index 41dbd4ad01..5310326bda 100644 --- a/libavformat/version.h +++ b/libavformat/version.h @@ -32,7 +32,7 @@ #include "version_major.h" #define LIBAVFORMAT_VERSION_MINOR 3 -#define LIBAVFORMAT_VERSION_MICRO 102 +#define LIBAVFORMAT_VERSION_MICRO 103 #define LIBAVFORMAT_VERSION_INT AV_VERSION_INT(LIBAVFORMAT_VERSION_MAJOR, \ LIBAVFORMAT_VERSION_MINOR, \