From patchwork Tue May 24 13:58:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 34728 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp554441pzj; Tue, 24 May 2022 06:58:44 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxaiYN4h4gVjMCip0Cs5XmjwFLIXUKqqCN16uKyZjcWSIB5YAh6AwqTb+KFLYkfknb7Ycen X-Received: by 2002:a17:907:7ea0:b0:6fe:f024:d006 with SMTP id qb32-20020a1709077ea000b006fef024d006mr7592675ejc.248.1653400724124; Tue, 24 May 2022 06:58:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653400724; cv=none; d=google.com; s=arc-20160816; b=HUaaalkGq7wca6htm5mcGNk8egbGVfaw6Zf3QKh2LDFDB7FzagIdkMbXcuDZqxVZ5d 0qkyvUFT+GrDUer77pYvT9kO/rb2pfkaaFiwxna84cxrol+tTFXCGbkpJ1rXUAvWhsI6 U9lv5mGCa///bGVYTi8E8mzUV+hiSGXcG6SWlptIBm0eFFNhPP24VcnGCeoh8vCgTmLy UvnvhYrZLvlS+TCb2lVLOcKttWf1s78VL2txjQixEgqw8M79Fsr7b5VthukDLoxEsoPR +z2k8/LLB2vh8/GpIe92mzXzXcICJbByE+3qjE+R8NM/2mZvhEdYdSTPpF8tL3g1I9Dd 1Qvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:from:references :in-reply-to:message-id:dkim-signature:delivered-to; bh=cR3AcvrwEAynQNjc6AgBtfEJrwPoOHTyGMGUQ5Sgksg=; b=ZWzJ9QadAbC2JBX4LW9u6gOHGAj1jptaIaH6TcVET0ln47bg0DYSeAPdWQ1fJ/Th/P sA4lHHQ5mLbdbDagZ+GvorWcaseuIgZMuMbJadUcaOb9vX+Wpj7EZTvv+6mSBn6ahyaI PfOK/jHTDNdju4dWft3MjH2egPe7PQ4WN2h24qu5Ec5GofSoCW5qf26k8Xu+CcLJKLyV 7c2bzoqS2yjPEGC2SHbj6wHoRvMG5xL1aL6bO814Jde1n/lkqCEOS6ntuMtcK0S3K7Sb 0z0imLSnVa+ixJSCD2p/0Uxw+2YEwsOq4rO5Rc0s0DMkdoOHXe0BHLa5dtai37BuKl5j tSUg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=nEc+FtdL; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id ht12-20020a170907608c00b006f4fe0c838bsi13389852ejc.270.2022.05.24.06.58.43; Tue, 24 May 2022 06:58:44 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=nEc+FtdL; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0276868B502; Tue, 24 May 2022 16:58:40 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D697B68B4BA for ; Tue, 24 May 2022 16:58:33 +0300 (EEST) Received: by mail-pj1-f48.google.com with SMTP id cs3-20020a17090af50300b001e0808b5838so1289846pjb.1 for ; Tue, 24 May 2022 06:58:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Mu1dNCsJjHSGwJY3vfQ55f1URWb8Z+WtI+vbaYT5WrE=; b=nEc+FtdLsc0GOdIw3NQC8HcWzs3aw0c22MdFsI/Gr6ZDRuOX3AunnJdBWZrK6FOBrQ e5VWvSbmUFAIbpBOjSdUDDRpYgdBcATd+IFM+iZJ9+pnc839gt7mPNlzdX7wrXRQ5w7J Io6DJNnIFBsnOBdfa2Iu/tM4C8c9L6YBbJj1ViqwroYw3U3E2E+ei/zXmCgydFyNQKT5 AKFV76oLHetsEsx5v57RK8KiSDLHM1sj6PcBKTZdssbSJvId/QxKvikZaO7QttvROVSJ VM+fnKeAcDBAx3WI/Ho4WMMeR1GUSQ/NdKxiwMQAHEeSTt+TAG4MxSDd7m1Yq29sAL+M udqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Mu1dNCsJjHSGwJY3vfQ55f1URWb8Z+WtI+vbaYT5WrE=; b=MUQ0TbZopk/LYihSd0qy3n/SrCs9KVC+I81UTcsk97hklOQ4OIwQ9xGzeUHIYZxPGY 5k4Jk5hhUdiIynA1U0+uUYIM8fStBM8ZrqY2ALjheEIt9xF4MeszuUOqpWpKSXBiPxO7 6KBIpbAzBmGJmj14qqVOyORc5dL1afqoXGW4UM3e1GIDMFQs8EXl3lFUrEJN2BZUDR7Z 63xANNfYm6WOsPvE9K9tzoRSOi83Sj5wsT8V/xFXcUpyWEJe0h1h8FtHTlprXkpb+Ysf 2/dYYbzI1W2Md1PBP6X+0bhYq9u3bDqYNlSBkxciBhcA6DaBwMH5+c2Ra9vkjGGYqHE9 k5Aw== X-Gm-Message-State: AOAM531JooT+4LVpMrKofxw29dNj95Y8Afl6mYRVPgwnqBxJw/EnPpVC LKnDTyO+3adlnJ4Eb35EBBOs7j2gBCwWlw== X-Received: by 2002:a17:903:1211:b0:15e:8208:8cc0 with SMTP id l17-20020a170903121100b0015e82088cc0mr28082333plh.52.1653400711466; Tue, 24 May 2022 06:58:31 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id w2-20020a170902d70200b0015e8d4eb2aesm7274683ply.248.2022.05.24.06.58.30 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 May 2022 06:58:30 -0700 (PDT) Message-Id: In-Reply-To: References: From: ffmpegagent Date: Tue, 24 May 2022 13:58:27 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v6 0/2] Support long file names on Windows X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Martin =?utf-8?q?Storsj=C3=B6?= , softworkz , Hendrik Leppkes Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: IkD7HTK5t7KE This patchset adds support for long file and directory paths on Windows. The implementation follows the same logic that .NET is using internally, with the only exception that it doesn't expand short path components in 8.3 format. .NET does this as the same function is also used for other purposes, but in our case, that's not required. Short (8.3) paths are working as well with the extended path prefix, even when longer than 260. Successfully tested: * Regular paths wth drive letter * Regular UNC paths * Long paths wth drive letter * Long paths wth drive letter and forward slashes * Long UNC paths * Prefixed paths wth drive letter * Prefixed UNC paths I have kept the individual functions separate on purpose, to make it easy to compare with the .NET impl. (compilers should inlinie those anyway) v2 * wchar_filename: Improve comments and function documentation * os_support: adjust defines to use win32_stat v3 * removed length check in path_is_extended() * added path_is_device_path() check in add_extended_prefix() * add_extended_prefix(): clarified doc and add checks * clarified string allocation length calculation * replaced 260 with MAX_PATH * removed redundant checks after normalization v4 * rebased. no changes v5 * resolved the ugly struct duplication * compatible with _USE_32BIT_TIME_T v6 * wchar_filename.h: added links to .NET source code * wchar_filename.h: free allocations on error * os_support.hs: use clean and safe way to redirect stat() calls softworkz (2): avutil/wchar_filename,file_open: Support long file names on Windows avformat/os_support: Support long file names on Windows libavformat/os_support.h | 87 +++++++++++++----- libavutil/file_open.c | 2 +- libavutil/wchar_filename.h | 180 +++++++++++++++++++++++++++++++++++++ 3 files changed, 244 insertions(+), 25 deletions(-) base-commit: 6076dbcb55d0c9b6693d1acad12a63f7268301aa Published-As: https://github.com/ffstaging/FFmpeg/releases/tag/pr-ffstaging-28%2Fsoftworkz%2Fsubmit_long_filenames-v6 Fetch-It-Via: git fetch https://github.com/ffstaging/FFmpeg pr-ffstaging-28/softworkz/submit_long_filenames-v6 Pull-Request: https://github.com/ffstaging/FFmpeg/pull/28 Range-diff vs v5: 1: 13118dc1fa ! 1: 960aa795ff avutil/wchar_filename,file_open: Support long file names on Windows @@ libavutil/wchar_filename.h: static inline int utf8towchar(const char *filename_u +/** + * Checks for extended path prefixes for which normalization needs to be skipped. + * see .NET6: PathInternal.IsExtended() ++ * https://github.com/dotnet/runtime/blob/9260c249140ef90b4299d0fe1aa3037e25228518/src/libraries/Common/src/System/IO/PathInternal.Windows.cs#L165 + */ +static inline int path_is_extended(const wchar_t *path) +{ @@ libavutil/wchar_filename.h: static inline int utf8towchar(const char *filename_u + +/** + * Checks for a device path prefix. -+ * see .NET6: PathInternal.IsDevicePath() ++ * see .NET6: PathInternal.IsDevice() ++ * we don't check forward slashes and extended paths (as already done) ++ * https://github.com/dotnet/runtime/blob/9260c249140ef90b4299d0fe1aa3037e25228518/src/libraries/Common/src/System/IO/PathInternal.Windows.cs#L132 + */ +static inline int path_is_device_path(const wchar_t *path) +{ @@ libavutil/wchar_filename.h: static inline int utf8towchar(const char *filename_u +/** + * Performs path normalization by calling GetFullPathNameW(). + * see .NET6: PathHelper.GetFullPathName() ++ * https://github.com/dotnet/runtime/blob/2a99e18eedabcf1add064c099da59d9301ce45e0/src/libraries/System.Private.CoreLib/src/System/IO/PathHelper.Windows.cs#L70 + */ +static inline int get_full_path_name(wchar_t **ppath_w) +{ @@ libavutil/wchar_filename.h: static inline int utf8towchar(const char *filename_u + + num_chars = GetFullPathNameW(*ppath_w, num_chars, temp_w, NULL); + if (num_chars <= 0) { ++ av_free(temp_w); + errno = EINVAL; + return -1; + } @@ libavutil/wchar_filename.h: static inline int utf8towchar(const char *filename_u + * Normalizes a Windows file or folder path. + * Expansion of short paths (with 8.3 path components) is currently omitted + * as it is not required for accessing long paths. -+ * see .NET6: PathHelper.Normalize(). ++ * see .NET6: PathHelper.Normalize() ++ * https://github.com/dotnet/runtime/blob/2a99e18eedabcf1add064c099da59d9301ce45e0/src/libraries/System.Private.CoreLib/src/System/IO/PathHelper.Windows.cs#L25 + */ +static inline int path_normalize(wchar_t **ppath_w) +{ @@ libavutil/wchar_filename.h: static inline int utf8towchar(const char *filename_u + * This function expects that the path has been normalized before by + * calling path_normalize() and it doesn't check whether the path is + * actually long (> MAX_PATH). -+ * see .NET6: PathInternal.EnsureExtendedPrefix() * ++ * see .NET6: PathInternal.EnsureExtendedPrefix() ++ * https://github.com/dotnet/runtime/blob/9260c249140ef90b4299d0fe1aa3037e25228518/src/libraries/Common/src/System/IO/PathInternal.Windows.cs#L107 + */ +static inline int add_extended_prefix(wchar_t **ppath_w) +{ @@ libavutil/wchar_filename.h: static inline int utf8towchar(const char *filename_u + * Longs paths (>= MAX_PATH) are prefixed with the extended path or extended + * UNC path prefix. + * see .NET6: Path.GetFullPath() and Path.GetFullPathInternal() ++ * https://github.com/dotnet/runtime/blob/2a99e18eedabcf1add064c099da59d9301ce45e0/src/libraries/System.Private.CoreLib/src/System/IO/Path.Windows.cs#L126 + */ +static inline int get_extended_win32_path(const char *path, wchar_t **ppath_w) +{ @@ libavutil/wchar_filename.h: static inline int utf8towchar(const char *filename_u + return 0; + } + -+ if ((ret = path_normalize(ppath_w)) < 0) ++ if ((ret = path_normalize(ppath_w)) < 0) { ++ av_freep(ppath_w); + return ret; ++ } + -+ // see .NET6: PathInternal.EnsureExtendedPrefixIfNeeded() ++ /* see .NET6: PathInternal.EnsureExtendedPrefixIfNeeded() ++ * https://github.com/dotnet/runtime/blob/9260c249140ef90b4299d0fe1aa3037e25228518/src/libraries/Common/src/System/IO/PathInternal.Windows.cs#L92 ++ */ + len = wcslen(*ppath_w); + if (len >= MAX_PATH) { -+ if ((ret = add_extended_prefix(ppath_w)) < 0) ++ if ((ret = add_extended_prefix(ppath_w)) < 0) { ++ av_freep(ppath_w); + return ret; ++ } + } + + return 0; 2: 5313aeec0e ! 2: 6f8d400db7 avformat/os_support: Support long file names on Windows @@ libavformat/os_support.h # undef stat # endif -# define stat _stati64 ++ +# define stat win32_stat + + struct win32_stat + { -+ struct _stati64; ++ _dev_t st_dev; /* ID of device containing file */ ++ _ino_t st_ino; /* inode number */ ++ unsigned short st_mode; /* protection */ ++ short st_nlink; /* number of hard links */ ++ short st_uid; /* user ID of owner */ ++ short st_gid; /* group ID of owner */ ++ _dev_t st_rdev; /* device ID (if special file) */ ++ long st_size; /* total size, in bytes */ ++ time_t st_atime; /* time of last access */ ++ time_t st_mtime; /* time of last modification */ ++ time_t st_ctime; /* time of last status change */ + }; + # ifdef fstat @@ libavformat/os_support.h: static inline int win32_##name(const char *filename_ut return -1; \ if (!filename_w) \ goto fallback; \ -@@ libavformat/os_support.h: static inline int win32_##name(const char *filename_utf8, partype par) \ - wchar_t *filename_w; \ - int ret; \ - \ +@@ libavformat/os_support.h: DEF_FS_FUNCTION(unlink, _wunlink, _unlink) + DEF_FS_FUNCTION(mkdir, _wmkdir, _mkdir) + DEF_FS_FUNCTION(rmdir, _wrmdir , _rmdir) + +-#define DEF_FS_FUNCTION2(name, wfunc, afunc, partype) \ +-static inline int win32_##name(const char *filename_utf8, partype par) \ +-{ \ +- wchar_t *filename_w; \ +- int ret; \ +- \ - if (utf8towchar(filename_utf8, &filename_w)) \ -+ if (get_extended_win32_path(filename_utf8, &filename_w)) \ - return -1; \ - if (!filename_w) \ - goto fallback; \ -@@ libavformat/os_support.h: static inline int win32_rename(const char *src_utf8, const char *dest_utf8) +- return -1; \ +- if (!filename_w) \ +- goto fallback; \ +- \ +- ret = wfunc(filename_w, par); \ +- av_free(filename_w); \ +- return ret; \ +- \ +-fallback: \ +- /* filename may be be in CP_ACP */ \ +- return afunc(filename_utf8, par); \ ++static inline int win32_access(const char *filename_utf8, int par) ++{ ++ wchar_t *filename_w; ++ int ret; ++ if (get_extended_win32_path(filename_utf8, &filename_w)) ++ return -1; ++ if (!filename_w) ++ goto fallback; ++ ret = _waccess(filename_w, par); ++ av_free(filename_w); ++ return ret; ++fallback: ++ return _access(filename_utf8, par); + } + +-DEF_FS_FUNCTION2(access, _waccess, _access, int) +-DEF_FS_FUNCTION2(stat, _wstati64, _stati64, struct stat*) ++static inline int win32_stat(const char *filename_utf8, struct stat *par) ++{ ++ wchar_t *filename_w; ++ int ret; ++ struct _stati64 winstat = { 0 }; ++ ++ if (get_extended_win32_path(filename_utf8, &filename_w)) ++ return -1; ++ ++ if (filename_w) { ++ ret = _wstat64(filename_w, &winstat); ++ av_free(filename_w); ++ } else ++ ret = _stat64(filename_utf8, &winstat); ++ ++ par->st_dev = winstat.st_dev; ++ par->st_ino = winstat.st_ino; ++ par->st_mode = winstat.st_mode; ++ par->st_nlink = winstat.st_nlink; ++ par->st_uid = winstat.st_uid; ++ par->st_gid = winstat.st_gid; ++ par->st_rdev = winstat.st_rdev; ++ par->st_size = winstat.st_size; ++ par->st_atime = winstat.st_atime; ++ par->st_mtime = winstat.st_mtime; ++ par->st_ctime = winstat.st_ctime; ++ ++ return ret; ++} + + static inline int win32_rename(const char *src_utf8, const char *dest_utf8) + { wchar_t *src_w, *dest_w; int ret;