From patchwork Thu Feb 17 14:49:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Gaiser X-Patchwork-Id: 34366 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6838:d078:0:0:0:0 with SMTP id x24csp33962nkx; Thu, 17 Feb 2022 06:50:54 -0800 (PST) X-Google-Smtp-Source: ABdhPJzylLZIqU0Xr8x6WXW/gnSwRPsllR4oqAPFp04LoQi+cUzBCIkJSY6QBFHT/WaYEJo38irj X-Received: by 2002:aa7:c582:0:b0:40a:92b2:1bc4 with SMTP id g2-20020aa7c582000000b0040a92b21bc4mr3020100edq.202.1645109454445; Thu, 17 Feb 2022 06:50:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1645109454; cv=none; d=google.com; s=arc-20160816; b=ChUD/5DWv3zuTh6VXmdWQ/UVUkqXiv50acJja5oMELIOqvrr+33+m1N7KdW5oQ9+cM ysQU2KshZ9XRCBD+U8c5UXBtM4TwIK5vf8HU6aERYg73KWz0TlK5No6ketUThAVtEsqb zDn7LXi1iZVII40vNOYM89DBZKCJg3V8T9zkUlhDI+8Vdzuii9fiGQ3qsW8AIw1Kj8h5 k0rZLTw7DA+LmlTmR45MDR2SLRUfJb61vDWyFjs0C9qjJSLeE2So/yBls8MrJlLip1Kq AfQdc4Q37piCo7/tA1CAcp8S5DUzyVL3kMgHSc1t6aXbLh09+e9zNRcip32xA5UpCy/A 4NOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=HWmfEZYfKObjV8ufmq5Lqd37LTX1JrEH1QDv5k6ojlk=; b=AGjJSq1Iz+KdC3tuFrSctwSG5UF9Li3iIxbQ4jWyP9BblZpzXnBGDwiOOm74Fx5w7E lebhh8PH9T94pgesBCI0Xgfbi3UcOEO738tgET/idxPGzdJlRffRdosfZS9DMwigfbdr h1L8kAi4md0fAXuC0NIZzQQlNJTOcsLcv5PMPS6PhPR/Dz9rkHGOwfXrk+GRB/nogwUg DUiw8vH3w7j7pqmshhNlXDb+jlXxYE6VoWkBmN143JzDqQqGnhsf/Wue6vd19zqUaEbi gFeXX0YzLUr1kYdY6ZQQnkjYIP1k/yd4BIUbL09P+C1tDMfSM0GT7l1UZqO2C3m+SXjA cXIA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=WQnpDAIn; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id w25si3819589edr.67.2022.02.17.06.50.47; Thu, 17 Feb 2022 06:50:54 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=WQnpDAIn; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E98C868B2F8; Thu, 17 Feb 2022 16:50:44 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B909E68B13C for ; Thu, 17 Feb 2022 16:50:37 +0200 (EET) Received: by mail-ej1-f43.google.com with SMTP id d10so7828176eje.10 for ; Thu, 17 Feb 2022 06:50:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=oLSnySBTbKByyDWQMcIoidTmsdUxdcqaBVaOeQVCuIU=; b=WQnpDAIn6mlbQsenLDMFeeemqtbKX28O+S8XoKWVkXMTrbiD8FcxR47TfntmKckoMH QhbByieXMEr/ShmZLd6krltrckIk7nKqEdn0dHxKyIbCq0MVOEnAv0tgcLRMkqnNzqsu sJBtTgB83wTCWt41ZGHx3DCXjNQixnG2IAkJuSYzPHLeyMI3xivUeg63xwiVceQC7ixx gwFnBaLWKb71vkF1GPYIoHexAKEzA4LbdJ3eblNF+g99YRNc00OTLPE5JvSXNd7ETRAO 1qz8FJayyoUkJIDxcUduzHHNBmD0yeS7teRiOvgfogJohgmd7bAr/RaMGUU7oRnxrMan QFdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=oLSnySBTbKByyDWQMcIoidTmsdUxdcqaBVaOeQVCuIU=; b=it/JF9YUmUDLu9XmzExSMh29ylHI+/xlQcpN2shD/8e79cj6XoqggfmT49LWeTR+EH unaoHTDABeCQ8JaFrKHeEhnU48J9/pFtGDlGdKAyGEpxkdZuMdI5rOlF90uMBsAlyrhz +qlhGnLwYBbuS/TrABbmL+pigz0hnUHhErI4QN/eQDrbbQP8jp+oDSK/BR9osw0seqnP dTx/IafUh5tkAI+e854ntUEJVu2PIjPrBHceARON2DpClTiVJYQtW+ebmlR2myCv8/o7 MWgz2pZEhBjMntK6Rmd4nrZCnGNqC+b4DtLyZt0TSzXIoueFC7MftUEwvziIT8rm2UyT JMSw== X-Gm-Message-State: AOAM531kj2kOzy2XrVMSPlwJjSEwFtwQ6rqvvkUEDbZtOCcwMMW4uFum cMday1OxojoW6+C+daAOiQfapvq5+nrvNA== X-Received: by 2002:a17:907:6006:b0:6cd:efc0:3d67 with SMTP id fs6-20020a170907600600b006cdefc03d67mr2679273ejc.32.1645109436660; Thu, 17 Feb 2022 06:50:36 -0800 (PST) Received: from localhost.localdomain (ip60-253-211-87.adsl2.static.versatel.nl. [87.211.253.60]) by smtp.gmail.com with ESMTPSA id z1sm1254600eji.136.2022.02.17.06.50.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Feb 2022 06:50:36 -0800 (PST) From: Mark Gaiser To: ffmpeg-devel@ffmpeg.org Date: Thu, 17 Feb 2022 15:49:03 +0100 Message-Id: <20220217144904.158764-1-markg85@gmail.com> X-Mailer: git-send-email 2.35.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v8 0/1] Add IPFS protocol support. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Mark Gaiser Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 7TimvGa0JO0U Hi, This patch series adds support for IPFS. V8: - Removed unnecessary change to set the first gateway_buffer character to 0. It made no sense as the buffer is always overwritten in the function context. - Change %li to %zu (it's intended to print the sizeof in all cases) V7: - Removed sanitize_ipfs_gateway. Only the http/https check stayed and that's now in translate_ipfs_to_http. - Added a check for ipfs_cid. It's only to show an error is someone happens to profide `ffplay ipfs://` without a cid. - All snprintf usages are now checked. - Adding a / to a gateway if it didn't end with it is now done in the same line that composes the full resulting url. - And a couple more minor things. V6: - Moved the gateway buffer (now called gateway_buffer) to IPFSGatewayContext - Changed nearly all PATH_MAX uses to sizeof(...) uses for future flexibility - The rest is relatively minor feedback changes V5: - "c->gateway" is now not modified anymore - Moved most variables to the stack - Even more strict checks with the auto detection logic - Errors are now AVERROR :) - Added more logging and changed some debug ones to info ones as they are valuable to aid debugging as a user when something goes wrong. V3 (V4): - V4: title issue from V3.. - A lot of style changes - Made url checks a lot more strict - av_asprintf leak fixes - So many changes that a diff to v2 is again not sensible. V2: - Squashed and changed so much that a diff to v1 was not sensible. The following is a short summary. In the IPFS ecosystem you access it's content by a "Content IDentifier" (CID). This CID is, in simplified terms, a hash of the content. IPFS itself is a distributed network where any user can run a node to be part of the network and access files by their CID. If any reachable node within that network has the CID, you can get it. IPFS (as a technology) has two protocols, ipfs and ipns. The ipfs protocol is the immutable way to access content. The ipns protocol is a mutable layer on top of it. It's essentially a new CID that points to a ipfs CID. This "pointer" if you will can be changed to point to something else. Much more information on how this technology works can be found here [1]. This patch series allows to interact natively with IPFS. That means being able to access files like: - ffplay ipfs:// - ffplay ipns:// There are multiple ways to access files on the IPFS network. This patch series uses the gateway driven way. An IPFS node - by default - exposes a local gateway (say http://localhost:8080) which is then used to get content from IPFS. The gateway functionality on the IPFS side contains optimizations to be as ideal to streaming data as it can be. Optimizations that the http protocol in ffmpeg also has and are thus reused for free in this approach. A note on other "more appropiate" ways, as I received some feedback on that. For ffmpeg purposes the gateway approach is ideal! There is a "libipfs" but that would spin up an ipfs node with the overhead of: - bootstrapping - connecting to nodes - finding other nodes to connect too - finally finding your file This alternative approach could take minutes before a file is played. The gateway approach immediately connects to an already running node thus gives the file the fastest. Much of the logic in this patch series is to find that gateway and essentially rewrite: "ipfs://" to: "http://localhost:8080/ipfs/" Once that's found it's forwared to the protocol handler where eventually the http protocol is going to handle it. Note that it could also be https. There's enough flexibility in the implementation to allow the user to provide a gateway. There are also public https gateways which can be used just as well. After this patch is accepted, I'll work on getting IPFS supported in: - mpv (requires this ffmpeg patch) - vlc (prefers this patch but can be made to work without this patch) - kodi (requires this ffmpeg patch) Best regards, Mark Gaiser [1] https://docs.ipfs.io/concepts/ Mark Gaiser (1): avformat: Add IPFS protocol support. configure | 2 + doc/protocols.texi | 30 ++++ libavformat/Makefile | 2 + libavformat/ipfsgateway.c | 309 ++++++++++++++++++++++++++++++++++++++ libavformat/protocols.c | 2 + 5 files changed, 345 insertions(+) create mode 100644 libavformat/ipfsgateway.c