| Message ID | pull.28.v8.ffstaging.FFmpeg.1653557330.ffmpegagent@gmail.com |
|---|---|
| Headers | show |
| Series | Support long file names on Windows | expand |
On Thu, 26 May 2022, ffmpegagent wrote: > This patchset adds support for long file and directory paths on Windows. The > implementation follows the same logic that .NET is using internally, with > the only exception that it doesn't expand short path components in 8.3 > format. .NET does this as the same function is also used for other purposes, > but in our case, that's not required. Short (8.3) paths are working as well > with the extended path prefix, even when longer than 260. > > Successfully tested: > > * Regular paths wth drive letter > * Regular UNC paths > * Long paths wth drive letter > * Long paths wth drive letter and forward slashes > * Long UNC paths > * Prefixed paths wth drive letter > * Prefixed UNC paths > > I have kept the individual functions separate on purpose, to make it easy to > compare with the .NET impl. (compilers should inlinie those anyway) > > v2 > > * wchar_filename: Improve comments and function documentation > * os_support: adjust defines to use win32_stat > > v3 > > * removed length check in path_is_extended() > * added path_is_device_path() check in add_extended_prefix() > * add_extended_prefix(): clarified doc and add checks > * clarified string allocation length calculation > * replaced 260 with MAX_PATH > * removed redundant checks after normalization > > v4 > > * rebased. no changes > > v5 > > * resolved the ugly struct duplication > * compatible with _USE_32BIT_TIME_T > > v6 > > * wchar_filename.h: added links to .NET source code > * wchar_filename.h: free allocations on error > * os_support.hs: use clean and safe way to redirect stat() calls > > v7 > > * os_support.h: remapped fstat with win32_stat structure > * os_support.h: use int64_t for some members > * avformat/file: remove _WIN32 condition > > v8 > > * os_support.h: documented win32_stat structure > * os_support.h: renamed function parameters > > softworkz (3): > avutil/wchar_filename,file_open: Support long file names on Windows > avformat/os_support: Support long file names on Windows > avformat/file: remove _WIN32 condition > > libavformat/file.c | 4 - > libavformat/os_support.h | 116 ++++++++++++++++++------ > libavutil/file_open.c | 2 +- > libavutil/wchar_filename.h | 180 +++++++++++++++++++++++++++++++++++++ > 4 files changed, 272 insertions(+), 30 deletions(-) This is still ok with me, fwiw. // Martin
On Thu, 26 May 2022, ffmpegagent wrote: > This patchset adds support for long file and directory paths on Windows. The > implementation follows the same logic that .NET is using internally, with > the only exception that it doesn't expand short path components in 8.3 > format. .NET does this as the same function is also used for other purposes, > but in our case, that's not required. Short (8.3) paths are working as well > with the extended path prefix, even when longer than 260. > > Successfully tested: > > * Regular paths wth drive letter > * Regular UNC paths > * Long paths wth drive letter > * Long paths wth drive letter and forward slashes > * Long UNC paths > * Prefixed paths wth drive letter > * Prefixed UNC paths > > I have kept the individual functions separate on purpose, to make it easy to > compare with the .NET impl. (compilers should inlinie those anyway) > > v2 > > * wchar_filename: Improve comments and function documentation > * os_support: adjust defines to use win32_stat > > v3 > > * removed length check in path_is_extended() > * added path_is_device_path() check in add_extended_prefix() > * add_extended_prefix(): clarified doc and add checks > * clarified string allocation length calculation > * replaced 260 with MAX_PATH > * removed redundant checks after normalization > > v4 > > * rebased. no changes > > v5 > > * resolved the ugly struct duplication > * compatible with _USE_32BIT_TIME_T > > v6 > > * wchar_filename.h: added links to .NET source code > * wchar_filename.h: free allocations on error > * os_support.hs: use clean and safe way to redirect stat() calls > > v7 > > * os_support.h: remapped fstat with win32_stat structure > * os_support.h: use int64_t for some members > * avformat/file: remove _WIN32 condition > > v8 > > * os_support.h: documented win32_stat structure > * os_support.h: renamed function parameters > > softworkz (3): > avutil/wchar_filename,file_open: Support long file names on Windows > avformat/os_support: Support long file names on Windows > avformat/file: remove _WIN32 condition > > libavformat/file.c | 4 - > libavformat/os_support.h | 116 ++++++++++++++++++------ > libavutil/file_open.c | 2 +- > libavutil/wchar_filename.h | 180 +++++++++++++++++++++++++++++++++++++ > 4 files changed, 272 insertions(+), 30 deletions(-) This looks fine to me, and the discussion seems to have converged, so I'll go ahead and push this. // Martin
> This looks fine to me, and the discussion seems to have converged, so I'll > go ahead and push this. Build is now failing: https://github.com/yt-dlp/FFmpeg-Builds/runs/6819319105?check_suite_focus=true. In file included from /opt/ct-ng/i686-w64-mingw32/sysroot/mingw/include/dshow.h:33, from libavcodec/mf_utils.h:32, from libavcodec/mfenc.c:26: ./libavutil/wchar_filename.h: In function 'add_extended_prefix': ./libavutil/wchar_filename.h:211:9: error: 'wcscpy_instead_use_StringCbCopyW_or_StringCchCopyW' undeclared (first use in this function) 211 | wcscpy(temp_w, unc_prefix); | ^~~~~~ ./libavutil/wchar_filename.h:211:9: note: each undeclared identifier is reported only once for each function it appears in wchar.h is indeed not included in wchar_filename.h. Incompatible pointer types warning is still there as well: In file included from ./libavformat/internal.h:30, from libavdevice/alldevices.c:21: ./libavformat/os_support.h: In function 'win32_stat': ./libavformat/os_support.h:241:36: warning: passing argument 2 of '_wstat64' from incompatible pointer type [-Wincompatible-pointer-types] 241 | ret = _wstat64(filename_w, &crtstat); | ^~~~~~~~ | | | struct _stati64 * In file included from ./libavformat/os_support.h:32, from ./libavformat/internal.h:30, from libavdevice/alldevices.c:21: /opt/ct-ng/i686-w64-mingw32/sysroot/mingw/include/sys/stat.h:129:69: note: expected 'struct _stat64 *' but argument is of type 'struct _stati64 *' 129 | _CRTIMP int __cdecl _wstat64(const wchar_t *_Name,struct _stat64 *_Stat); | ~~~~~~~~~~~~~~~~^~~~~
On Thu, 9 Jun 2022, nil-admirari@mailo.com wrote: >> This looks fine to me, and the discussion seems to have converged, so I'll >> go ahead and push this. > > Build is now failing: https://github.com/yt-dlp/FFmpeg-Builds/runs/6819319105?check_suite_focus=true. > > In file included from /opt/ct-ng/i686-w64-mingw32/sysroot/mingw/include/dshow.h:33, > from libavcodec/mf_utils.h:32, > from libavcodec/mfenc.c:26: > ./libavutil/wchar_filename.h: In function 'add_extended_prefix': > ./libavutil/wchar_filename.h:211:9: error: 'wcscpy_instead_use_StringCbCopyW_or_StringCchCopyW' undeclared (first use in this function) > 211 | wcscpy(temp_w, unc_prefix); > | ^~~~~~ > ./libavutil/wchar_filename.h:211:9: note: each undeclared identifier is reported only once for each function it appears in This error isn't reproducible in git master - it's triggered by your yet-unmerged patches (that include wchar_filename.h in w32dlfcn.h). > Incompatible pointer types warning is still there as well: > > In file included from ./libavformat/internal.h:30, > from libavdevice/alldevices.c:21: > ./libavformat/os_support.h: In function 'win32_stat': > ./libavformat/os_support.h:241:36: warning: passing argument 2 of '_wstat64' from incompatible pointer type [-Wincompatible-pointer-types] > 241 | ret = _wstat64(filename_w, &crtstat); > | ^~~~~~~~ > | | > | struct _stati64 * > In file included from ./libavformat/os_support.h:32, > from ./libavformat/internal.h:30, > from libavdevice/alldevices.c:21: > /opt/ct-ng/i686-w64-mingw32/sysroot/mingw/include/sys/stat.h:129:69: note: expected 'struct _stat64 *' but argument is of type 'struct _stati64 *' > 129 | _CRTIMP int __cdecl _wstat64(const wchar_t *_Name,struct _stat64 *_Stat); > | ~~~~~~~~~~~~~~~~^~~~~ This I can indeed reproduce now. I did build these patches in a couple other environments (both mingw and msvc) where those warnings didn't appear, but now I managed to find one that shows those warnings. // Martin
> This error isn't reproducible in git master - it's triggered by your > yet-unmerged patches (that include wchar_filename.h in w32dlfcn.h). Ok. It can be fixed by either - defining NO_DSHOW_STRSAFE in libavcodec/mf_utils.h - or by migrating os_support.h to StrSafe.h functions. Which way is preferable?
On Thu, 9 Jun 2022, nil-admirari@mailo.com wrote: >> This error isn't reproducible in git master - it's triggered by your >> yet-unmerged patches (that include wchar_filename.h in w32dlfcn.h). > > Ok. It can be fixed by either > - defining NO_DSHOW_STRSAFE in libavcodec/mf_utils.h > - or by migrating os_support.h to StrSafe.h functions. > > Which way is preferable? I think avoiding wcscat (with whatever usable alternative, not necessarily from strsafe.h) is the more robust solution, instead of having to play whack-a-mole with silencing such warnings. The 10 year old trac you referenced mentioned that the strsafe.h alternative functions weren't available in all toolchains used at that time though. Or if we'd add the define projectwide in e.g. configure it probably wouldn't be that bad? Kinda like how we already add "-D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_WARNINGS" in MSVC builds. Then we wouldn't need to worry about missing it somewhere accidentally. // Martin
> -----Original Message----- > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Soft Works > Sent: Thursday, June 9, 2022 10:16 PM > To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org> > Subject: Re: [FFmpeg-devel] [PATCH v8 0/3] Support long file names on Windows > > > > > -----Original Message----- > > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of nil- > > admirari@mailo.com > > Sent: Thursday, June 9, 2022 9:37 PM > > To: ffmpeg-devel@ffmpeg.org > > Subject: Re: [FFmpeg-devel] [PATCH v8 0/3] Support long file names on > Windows > > > > > This looks fine to me, and the discussion seems to have converged, so > I'll > > > go ahead and push this. > > > > Build is now failing: https://github.com/yt-dlp/FFmpeg- > > Builds/runs/6819319105?check_suite_focus=true. > > > > In file included from /opt/ct-ng/i686-w64- > > mingw32/sysroot/mingw/include/dshow.h:33, > > from libavcodec/mf_utils.h:32, > > from libavcodec/mfenc.c:26: > > ./libavutil/wchar_filename.h: In function 'add_extended_prefix': > > ./libavutil/wchar_filename.h:211:9: error: > > 'wcscpy_instead_use_StringCbCopyW_or_StringCchCopyW' undeclared (first use > in > > this function) > > 211 | wcscpy(temp_w, unc_prefix); > > | ^~~~~~ > > ./libavutil/wchar_filename.h:211:9: note: each undeclared identifier is > > reported only once for each function it appears in > > > > wchar.h is indeed not included in wchar_filename.h. > > wcscpy is defined in corecrt_wstring.h, included in string.h, > included in winnt.h included in windef.h, included in windows.h > > > > Incompatible pointer types warning is still there as well: > > > > In file included from ./libavformat/internal.h:30, > > from libavdevice/alldevices.c:21: > > ./libavformat/os_support.h: In function 'win32_stat': > > ./libavformat/os_support.h:241:36: warning: passing argument 2 of > '_wstat64' > > from incompatible pointer type [-Wincompatible-pointer-types] > > 241 | ret = _wstat64(filename_w, &crtstat); > > | ^~~~~~~~ > > | | > > | struct _stati64 * > > In file included from ./libavformat/os_support.h:32, > > from ./libavformat/internal.h:30, > > from libavdevice/alldevices.c:21: > > /opt/ct-ng/i686-w64-mingw32/sysroot/mingw/include/sys/stat.h:129:69: note: > > expected 'struct _stat64 *' but argument is of type 'struct _stati64 *' > > 129 | _CRTIMP int __cdecl _wstat64(const wchar_t *_Name,struct _stat64 > > *_Stat); > > | > > ~~~~~~~~~~~~~~~~^~~~~ > > Yea, right. We need to get rid of the 'i' in the struct > types for getting fully independent of USE_32BIT_TIME_T. > > Will send an update in a minute. Here's a patch for this part: https://github.com/ffstaging/FFmpeg/pull/34 https://github.com/ffstaging/FFmpeg/pull/34.patch Can't send it right now because Google decided to no longer username/password login (which affects ffmpegagent). It's said to work with an "app password" which in turn can only be used when 2step verification is set up, and that again requires either a phone or a security key :-( Does anybody know a "security key" emulator. softworkz
> ... > > - defining NO_DSHOW_STRSAFE in libavcodec/mf_utils.h > ... > Or if we'd add the define projectwide in e.g. configure it probably > wouldn't be that bad? Kinda like how we already add > "-D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_WARNINGS" in MSVC builds. > Then we wouldn't need to worry about missing it somewhere accidentally. Ended up defining NO_DSHOW_STRSAFE in mf_utils.h, just like dshow_capture.h does (these are the only two uses). -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_WARNINGS are defined only for MVSC: elif test_${pfx}cpp_condition crtversion.h "defined _VC_CRT_MAJOR_VERSION"; then NO_DSHOW_STRSAFE should cover MinGW as well, and probably others. If you still want global NO_DSHOW_STRSAFE, please point where to add it exactly.
This patchset adds support for long file and directory paths on Windows. The implementation follows the same logic that .NET is using internally, with the only exception that it doesn't expand short path components in 8.3 format. .NET does this as the same function is also used for other purposes, but in our case, that's not required. Short (8.3) paths are working as well with the extended path prefix, even when longer than 260. Successfully tested: * Regular paths wth drive letter * Regular UNC paths * Long paths wth drive letter * Long paths wth drive letter and forward slashes * Long UNC paths * Prefixed paths wth drive letter * Prefixed UNC paths I have kept the individual functions separate on purpose, to make it easy to compare with the .NET impl. (compilers should inlinie those anyway) v2 * wchar_filename: Improve comments and function documentation * os_support: adjust defines to use win32_stat v3 * removed length check in path_is_extended() * added path_is_device_path() check in add_extended_prefix() * add_extended_prefix(): clarified doc and add checks * clarified string allocation length calculation * replaced 260 with MAX_PATH * removed redundant checks after normalization v4 * rebased. no changes v5 * resolved the ugly struct duplication * compatible with _USE_32BIT_TIME_T v6 * wchar_filename.h: added links to .NET source code * wchar_filename.h: free allocations on error * os_support.hs: use clean and safe way to redirect stat() calls v7 * os_support.h: remapped fstat with win32_stat structure * os_support.h: use int64_t for some members * avformat/file: remove _WIN32 condition v8 * os_support.h: documented win32_stat structure * os_support.h: renamed function parameters softworkz (3): avutil/wchar_filename,file_open: Support long file names on Windows avformat/os_support: Support long file names on Windows avformat/file: remove _WIN32 condition libavformat/file.c | 4 - libavformat/os_support.h | 116 ++++++++++++++++++------ libavutil/file_open.c | 2 +- libavutil/wchar_filename.h | 180 +++++++++++++++++++++++++++++++++++++ 4 files changed, 272 insertions(+), 30 deletions(-) base-commit: 6076dbcb55d0c9b6693d1acad12a63f7268301aa Published-As: https://github.com/ffstaging/FFmpeg/releases/tag/pr-ffstaging-28%2Fsoftworkz%2Fsubmit_long_filenames-v8 Fetch-It-Via: git fetch https://github.com/ffstaging/FFmpeg pr-ffstaging-28/softworkz/submit_long_filenames-v8 Pull-Request: https://github.com/ffstaging/FFmpeg/pull/28 Range-diff vs v7: 1: 960aa795ff = 1: 960aa795ff avutil/wchar_filename,file_open: Support long file names on Windows 2: 7751335906 ! 2: d0cc40a0d4 avformat/os_support: Support long file names on Windows @@ libavformat/os_support.h + +# define stat win32_stat + ++ /* ++ * The POSIX definition for the stat() function uses a struct of the ++ * same name (struct stat), that why it takes this extra effort for ++ * redirecting/replacing the stat() function with our own one which ++ * is capable to handle long path names on Windows. ++ * The struct below roughly follows the POSIX definition. Time values ++ * are 64bit, but in cases when _USE_32BIT_TIME_T is defined, they ++ * will be set to values no larger than INT32_MAX which corresponds ++ * to file times up to the year 2038. ++ */ + struct win32_stat + { + _dev_t st_dev; /* ID of device containing file */ @@ libavformat/os_support.h: DEF_FS_FUNCTION(unlink, _wunlink, _unlink) -fallback: \ - /* filename may be be in CP_ACP */ \ - return afunc(filename_utf8, par); \ -+static inline int win32_access(const char *filename_utf8, int par) ++static inline int win32_access(const char *filename_utf8, int mode) +{ + wchar_t *filename_w; + int ret; @@ libavformat/os_support.h: DEF_FS_FUNCTION(unlink, _wunlink, _unlink) + return -1; + if (!filename_w) + goto fallback; -+ ret = _waccess(filename_w, par); ++ ret = _waccess(filename_w, mode); + av_free(filename_w); + return ret; +fallback: -+ return _access(filename_utf8, par); ++ return _access(filename_utf8, mode); +} + -+static inline void copy_stat(struct _stati64 *winstat, struct win32_stat *par) ++static inline void copy_stat(struct _stati64 *crtstat, struct win32_stat *buf) +{ -+ par->st_dev = winstat->st_dev; -+ par->st_ino = winstat->st_ino; -+ par->st_mode = winstat->st_mode; -+ par->st_nlink = winstat->st_nlink; -+ par->st_uid = winstat->st_uid; -+ par->st_gid = winstat->st_gid; -+ par->st_rdev = winstat->st_rdev; -+ par->st_size = winstat->st_size; -+ par->st_atime = winstat->st_atime; -+ par->st_mtime = winstat->st_mtime; -+ par->st_ctime = winstat->st_ctime; ++ buf->st_dev = crtstat->st_dev; ++ buf->st_ino = crtstat->st_ino; ++ buf->st_mode = crtstat->st_mode; ++ buf->st_nlink = crtstat->st_nlink; ++ buf->st_uid = crtstat->st_uid; ++ buf->st_gid = crtstat->st_gid; ++ buf->st_rdev = crtstat->st_rdev; ++ buf->st_size = crtstat->st_size; ++ buf->st_atime = crtstat->st_atime; ++ buf->st_mtime = crtstat->st_mtime; ++ buf->st_ctime = crtstat->st_ctime; } -DEF_FS_FUNCTION2(access, _waccess, _access, int) -DEF_FS_FUNCTION2(stat, _wstati64, _stati64, struct stat*) -+static inline int win32_stat(const char *filename_utf8, struct win32_stat *par) ++static inline int win32_stat(const char *filename_utf8, struct win32_stat *buf) +{ -+ struct _stati64 winstat = { 0 }; ++ struct _stati64 crtstat = { 0 }; + wchar_t *filename_w; + int ret; + @@ libavformat/os_support.h: DEF_FS_FUNCTION(unlink, _wunlink, _unlink) + return -1; + + if (filename_w) { -+ ret = _wstat64(filename_w, &winstat); ++ ret = _wstat64(filename_w, &crtstat); + av_free(filename_w); + } else -+ ret = _stat64(filename_utf8, &winstat); ++ ret = _stat64(filename_utf8, &crtstat); + -+ copy_stat(&winstat, par); ++ copy_stat(&crtstat, buf); + + return ret; +} + -+static inline int win32_fstat(int fd, struct win32_stat *par) ++static inline int win32_fstat(int fd, struct win32_stat *buf) +{ -+ struct _stati64 winstat = { 0 }; ++ struct _stati64 crtstat = { 0 }; + int ret; + -+ ret = _fstat64(fd, &winstat); ++ ret = _fstat64(fd, &crtstat); + -+ copy_stat(&winstat, par); ++ copy_stat(&crtstat, buf); + + return ret; +} 3: 0522fc2315 = 3: e13c6b0aaa avformat/file: remove _WIN32 condition