From patchwork Sun Feb 13 18:18:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Gaiser X-Patchwork-Id: 34264 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6838:14aa:0:0:0:0 with SMTP id bz10csp4916355nkb; Sun, 13 Feb 2022 10:20:08 -0800 (PST) X-Google-Smtp-Source: ABdhPJxwccedvS4wiBHs3CCWgB6g5UX8rp6eEtduE0+qXPDME9/9/MkJXFXHyhr+nelWUz0CmUGT X-Received: by 2002:a17:907:3e24:: with SMTP id hp36mr8783884ejc.530.1644776408301; Sun, 13 Feb 2022 10:20:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1644776408; cv=none; d=google.com; s=arc-20160816; b=bI1QytFpgcX+YpPcjKnlX+JLCo75B0Gs5QZxI0dYxkvephrb8rOB5DH3zrEzwOR8RY 3JKl6aKyFbi9zy59i3bEAG3p/TciVeTEmaLjRk+CKjhfIVdMm4fkE94kQ3x3Oz1rUFSr QajfDPTr6vO6nU4gVz+N1S+JCd1P2AdoYUFsWSmY39Sby1uTQttcdkK4Tmm+Zq+Vtjd6 baLKjCt1A/HM8rhosjTK1VEQYEzZcsZHZWt7Vaiz7MD3OCJKUfdwxaUEb7zpAdyYWxDn sYvlICI5ctflHPu8IBXMqbUGsyJBA6CF7zkmhqvKDgKooxMnz8lyxmxoFHL1DI8NS/Lr fkYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=uPg+/uU6rDwEcOvx6zm8LoU62IPs5oRVfl4LhXrdzmo=; b=vkBG+rLoSTo0TPasOOnSbsXRLrO/7DuJ/k/yDVcyUOPH8njZDpj0hGX5naAv1sxJhq 3t+a60jgqLTTdxleOuZOn7eijrblmqlxs3Tu8HZ0p3MIyp4BCUM5MEvFJb3flO4yeJJZ gaUlWuLq/9P3HWNpp8b4N3opfGwcDdgW3KOhNE9l8wZZt3fZQBWlqnj+jbNaYCDCQIVj n5jqlC0H26xsuRuIzNYt8pwjZz04N6XBIizTFN6gvAAq3ajgflKEL9kThYUvvLPiYaM3 GvDElULjDXLWflnZ6mtxx9k2LxhhYtE98HNST2KzKQTuoxh5HDwdXKOuuB/bq8aWjuS4 rInQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=pIoKnxpe; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z22si9273029edm.288.2022.02.13.10.20.07; Sun, 13 Feb 2022 10:20:08 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=pIoKnxpe; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 557B668B279; Sun, 13 Feb 2022 20:20:03 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 668DB68B0C3 for ; Sun, 13 Feb 2022 20:19:56 +0200 (EET) Received: by mail-ej1-f52.google.com with SMTP id h8so6644465ejy.4 for ; Sun, 13 Feb 2022 10:19:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=OBNM2i01ntfVPPlRcAp2c5nrhjWtoRWdQlwHd90NE3A=; b=pIoKnxpevRcF06YWqVPZ+ugfBiMzCUEL9NEmnTk862PEB+ZrFk9L8zCmVLnEHxZ1K1 dAWg613V2EEYNYVhZsZig0eCWHdT5hsNUVS4CM1WCUuui7Fd1FKeG/riaVd8O/o/wc2H rc2ZiqTXOeoAWQqOqHR8nPXFsqKTVxq73og0DGI6Bw2k8MX/vRDn0RiidDG0dnd71AHt U5+YAMDtkM3ZiR3f1gQZqp0B622KALnMAEyUqLTqz7Dm5rJPzGNo83zFu0aPJgZf5mgF futwOMNjFiuIfqoQKnc+GE/vYaHZJ/M2XRqb25LK/K9MzY9bbIhljqIZ/KAOmQZbU5QR vT2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=OBNM2i01ntfVPPlRcAp2c5nrhjWtoRWdQlwHd90NE3A=; b=CKx7bzs+xoTGfDaGDTLOSFBOJK/eeVwQ7MUoW855ear50MzqBtmWqmcXM4sOGtlq9I /OUpiv7I1PyH1kPHnwKgHkqei4ki4C9B3aTMNx6at04mpptkR8VDjb3NDzFzgQXLtpfP OJOdFuMN6XGBhgAqpjNB33bM6HvZTnkjaoAcde/WWre4n8RnZJj/046OSbeF8pvD1sZo nesqO4By7ZwYA7n5//CE0iwB/f3GgisP828O2etc/ZlNmcpDwNGgdng+R9WbHlfOvcte 7D7xCytvById/CMkC4X8fGy6CxLqfG3thluE5uIpPaeRPv8dmxQGBmtxCUQSFnGDLmrn zfXA== X-Gm-Message-State: AOAM533guv6HfOSRa7IAt2Xusdhw5xIcP0RJk67wdmF4iNcn63lfjAwl ANtNq/TdMnkK5nin6OtYbmrAQ1De964wukaC X-Received: by 2002:a17:906:2991:: with SMTP id x17mr6175524eje.140.1644776395389; Sun, 13 Feb 2022 10:19:55 -0800 (PST) Received: from localhost.localdomain (ip60-253-211-87.adsl2.static.versatel.nl. [87.211.253.60]) by smtp.gmail.com with ESMTPSA id g2sm14007902edt.44.2022.02.13.10.19.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Feb 2022 10:19:55 -0800 (PST) From: Mark Gaiser To: ffmpeg-devel@ffmpeg.org Date: Sun, 13 Feb 2022 19:18:25 +0100 Message-Id: <20220213181826.251039-1-markg85@gmail.com> X-Mailer: git-send-email 2.35.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v6 0/1] Add IPFS protocol support. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Mark Gaiser Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Y3OeFY6trIkD Hi, This patch series adds support for IPFS. V6: - Moved the gateway buffer (now called gateway_buffer) to IPFSGatewayContext - Changed nearly all PATH_MAX uses to sizeof(...) uses for future flexibility - The rest is relatively minor feedback changes V5: - "c->gateway" is now not modified anymore - Moved most variables to the stack - Even more strict checks with the auto detection logic - Errors are now AVERROR :) - Added more logging and changed some debug ones to info ones as they are valuable to aid debugging as a user when something goes wrong. V3 (V4): - V4: title issue from V3.. - A lot of style changes - Made url checks a lot more strict - av_asprintf leak fixes - So many changes that a diff to v2 is again not sensible. V2: - Squashed and changed so much that a diff to v1 was not sensible. The following is a short summary. In the IPFS ecosystem you access it's content by a "Content IDentifier" (CID). This CID is, in simplified terms, a hash of the content. IPFS itself is a distributed network where any user can run a node to be part of the network and access files by their CID. If any reachable node within that network has the CID, you can get it. IPFS (as a technology) has two protocols, ipfs and ipns. The ipfs protocol is the immutable way to access content. The ipns protocol is a mutable layer on top of it. It's essentially a new CID that points to a ipfs CID. This "pointer" if you will can be changed to point to something else. Much more information on how this technology works can be found here [1]. This patch series allows to interact natively with IPFS. That means being able to access files like: - ffplay ipfs:// - ffplay ipns:// There are multiple ways to access files on the IPFS network. This patch series uses the gateway driven way. An IPFS node - by default - exposes a local gateway (say http://localhost:8080) which is then used to get content from IPFS. The gateway functionality on the IPFS side contains optimizations to be as ideal to streaming data as it can be. Optimizations that the http protocol in ffmpeg also has and are thus reused for free in this approach. A note on other "more appropiate" ways, as I received some feedback on that. For ffmpeg purposes the gateway approach is ideal! There is a "libipfs" but that would spin up an ipfs node with the overhead of: - bootstrapping - connecting to nodes - finding other nodes to connect too - finally finding your file This alternative approach could take minutes before a file is played. The gateway approach immediately connects to an already running node thus gives the file the fastest. Much of the logic in this patch series is to find that gateway and essentially rewrite: "ipfs://" to: "http://localhost:8080/ipfs/" Once that's found it's forwared to the protocol handler where eventually the http protocol is going to handle it. Note that it could also be https. There's enough flexibility in the implementation to allow the user to provide a gateway. There are also public https gateways which can be used just as well. After this patch is accepted, I'll work on getting IPFS supported in: - mpv (requires this ffmpeg patch) - vlc (prefers this patch but can be made to work without this patch) - kodi (requires this ffmpeg patch) Best regards, Mark Gaiser [1] https://docs.ipfs.io/concepts/ Mark Gaiser (1): avformat: Add IPFS protocol support. configure | 2 + doc/protocols.texi | 30 ++++ libavformat/Makefile | 2 + libavformat/ipfsgateway.c | 324 ++++++++++++++++++++++++++++++++++++++ libavformat/protocols.c | 2 + 5 files changed, 360 insertions(+) create mode 100644 libavformat/ipfsgateway.c