From patchwork Thu Feb 10 01:13:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Gaiser X-Patchwork-Id: 34211 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2c4e:0:0:0:0 with SMTP id x14csp2228492iov; Wed, 9 Feb 2022 17:15:34 -0800 (PST) X-Google-Smtp-Source: ABdhPJzv1hDr6TdUL9q9Fkk/JgaT8IjV4e/mGgNUFkLG4mV5PXb/YySUrUWt3/ET9Iz5Yn5+qg5X X-Received: by 2002:a17:907:62a0:: with SMTP id nd32mr4483932ejc.96.1644455734117; Wed, 09 Feb 2022 17:15:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1644455734; cv=none; d=google.com; s=arc-20160816; b=f/QS9pr0GWNz4KGkPsbybqGRw0D7x8z4aHrdkOpfpMZWApLDrcME3bK/dRb4kVslXw KGFwukAvvZtWvk70bPVmDvA3fpsUgAeMchUd3FKXw+aH0glSx/X6nN3A83fLM3zx5Nfk i6r5TGjrT8S86u2Mm9Q1Yxh7HTE7qIQQGFbRPkbtrr/eS0Mw2VxHZj9dq+a+nZSyZJD4 +k3NRYvXUPDzj14JZZ6oVZ2mLnix8qPHJD9tCC0odl5/8o5LKIoHzjdB35tet/KY9i2q Ql6if9PwGU7JywQgMzivFiBzlITb+C5W3ihBZnPsropHM7h6qS3egwWKnaU0fMVv81+D pD+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=OoM4h/NWEPNpdRRHzxKkRzrUl+LA/ir4wgW0QGxcZj0=; b=lXWwkJcO/1ZFy4nFaQkBX8jYfKQsz3xQDVo9Uq7zQmxCE2MnTglpMsFdFVpHeSpgZH DfLzW6/zxE6esYTUs1SzTVcMH5MUZNO9rEtdodJ9wKYkokeo0NkLqCN01M8KOga83mWo JPlUKbgsN33BCqaeHJom/xnubJpDFaec1DLWq0/tvfH77m8EW23HAW5HRuNoDlR5noY8 qFjeIAIAcFLCsidsaFqPlxLc4TbIpj6BICps3mVrUmj+ck105lTuQSxCi707bNmLJRc5 Q3BPZu3Eqwe8809id1COvqEmk2qRHcCe4YDGvjvflDep1LvKpk4aUDcxXh8yyKUnlN5T jguA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=RF3YrF0Y; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id nd36si12626415ejc.457.2022.02.09.17.15.30; Wed, 09 Feb 2022 17:15:34 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=RF3YrF0Y; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8E36168B149; Thu, 10 Feb 2022 03:15:25 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8278C68ADDB for ; Thu, 10 Feb 2022 03:15:19 +0200 (EET) Received: by mail-ej1-f52.google.com with SMTP id u20so7321378ejx.3 for ; Wed, 09 Feb 2022 17:15:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=RCkdB2EoL/XVRPmWX4LCj7IfzTqAUHEJlE1qIqg7rIQ=; b=RF3YrF0Y/BvAChjrcCskfzZ4RvF0p9cg3K2JpUGWjAVABSi85//36M44RD6mMhtP0v HBlkvp5EwejhW6ETR4QYagFDTahRCs/pStcIqcT7DP6DqMH+Qq+riDKwKJXq2cRz9Jha jaG2xzfJW859kHw2Xgl36K51STMSWhiGxRwLhbcOj8rP4S6LzpKwnVZOCf8kHvhp3m8q IkUqqqA4W2yN28+KP+Vj4b3QtpsRKKkz/FDIi8/z7dx0M4A8zQrHE6sbfS7ZEj5Wxi25 cy70yGFRVk3YhPk8GyayGbHKhkJqWPI+CiC2odBdhYnHuvhbngLMdBBKw9CxpneXRf5Q 8B+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=RCkdB2EoL/XVRPmWX4LCj7IfzTqAUHEJlE1qIqg7rIQ=; b=iGDnaBXsgEsZv6fCZYvyrv03KaQpaez/hLGhMmhDpXyZM0O/JDHUOH1fzFACNiVxv6 hjNEHDWSCoN5hqHZQeVghopuu/O5Z68C+ANWcd2y+nWHNErwb33CbTmrR8orz93JpK4I 5zIFc/WXXL+5n0CSFcDQGYz/jisOyFDwwBWiekIvWtzu5Yygjqkd3AS/GJtcs2rya8MM WvQYzX8Ht9jQ5p59YZJPVTc1UuQhwox0gGPZl2xOJ9pDA6vE4Rq+oXEFQ28aILP7LqyN 6rj+QH5/SKK2k1LWOs2oZiERvMC2wqS5J7PM3TC74tWPAdWa3cIuoAqUjNS2cU0RMoMN 8i+g== X-Gm-Message-State: AOAM530EKHPm/RE/t3kQn6HBx9sbZ4Rb43R6auYY3qImtp8+iEOfxyOy SBcH49CbvZrpvYwSm2EC6SqY19C0hT0pHA== X-Received: by 2002:a17:906:564c:: with SMTP id v12mr4400530ejr.255.1644455718120; Wed, 09 Feb 2022 17:15:18 -0800 (PST) Received: from localhost.localdomain (ip60-253-211-87.adsl2.static.versatel.nl. [87.211.253.60]) by smtp.gmail.com with ESMTPSA id y16sm3675404ejd.72.2022.02.09.17.15.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Feb 2022 17:15:17 -0800 (PST) From: Mark Gaiser To: ffmpeg-devel@ffmpeg.org Date: Thu, 10 Feb 2022 02:13:51 +0100 Message-Id: <20220210011352.11582-1-markg85@gmail.com> X-Mailer: git-send-email 2.35.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v5 0/1] Add IPFS protocol support. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Mark Gaiser Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: tWsRa8U9bRkp Hi, This patch series adds support for IPFS. V5: - "c->gateway" is now not modified anymore - Moved most variables to the stack - Even more strict checks with the auto detection logic - Errors are now AVERROR :) - Added more logging and changed some debug ones to info ones as they are valuable to aid debugging as a user when something goes wrong. V3 (V4): - V4: title issue from V3.. - A lot of style changes - Made url checks a lot more strict - av_asprintf leak fixes - So many changes that a diff to v2 is again not sensible. V2: - Squashed and changed so much that a diff to v1 was not sensible. The following is a short summary. In the IPFS ecosystem you access it's content by a "Content IDentifier" (CID). This CID is, in simplified terms, a hash of the content. IPFS itself is a distributed network where any user can run a node to be part of the network and access files by their CID. If any reachable node within that network has the CID, you can get it. IPFS (as a technology) has two protocols, ipfs and ipns. The ipfs protocol is the immutable way to access content. The ipns protocol is a mutable layer on top of it. It's essentially a new CID that points to a ipfs CID. This "pointer" if you will can be changed to point to something else. Much more information on how this technology works can be found here [1]. This patch series allows to interact natively with IPFS. That means being able to access files like: - ffplay ipfs:// - ffplay ipns:// There are multiple ways to access files on the IPFS network. This patch series uses the gateway driven way. An IPFS node - by default - exposes a local gateway (say http://localhost:8080) which is then used to get content from IPFS. The gateway functionality on the IPFS side contains optimizations to be as ideal to streaming data as it can be. Optimizations that the http protocol in ffmpeg also has and are thus reused for free in this approach. A note on other "more appropiate" ways, as I received some feedback on that. For ffmpeg purposes the gateway approach is ideal! There is a "libipfs" but that would spin up an ipfs node with the overhead of: - bootstrapping - connecting to nodes - finding other nodes to connect too - finally finding your file This alternative approach could take minutes before a file is played. The gateway approach immediately connects to an already running node thus gives the file the fastest. Much of the logic in this patch series is to find that gateway and essentially rewrite: "ipfs://" to: "http://localhost:8080/ipfs/" Once that's found it's forwared to the protocol handler where eventually the http protocol is going to handle it. Note that it could also be https. There's enough flexibility in the implementation to allow the user to provide a gateway. There are also public https gateways which can be used just as well. After this patch is accepted, I'll work on getting IPFS supported in: - mpv (requires this ffmpeg patch) - vlc (prefers this patch but can be made to work without this patch) - kodi (requires this ffmpeg patch) Best regards, Mark Gaiser [1] https://docs.ipfs.io/concepts/ Mark Gaiser (1): avformat: Add IPFS protocol support. configure | 2 + doc/protocols.texi | 30 ++++ libavformat/Makefile | 2 + libavformat/ipfsgateway.c | 326 ++++++++++++++++++++++++++++++++++++++ libavformat/protocols.c | 2 + 5 files changed, 362 insertions(+) create mode 100644 libavformat/ipfsgateway.c