[FFmpeg-devel,v14,1/5] libavutil: Add wchartoutf8(), wchartoansi(), utf8toansi() and getenv_utf8()

wchartoutf8() converts strings returned by WinAPI into UTF-8,
which is FFmpeg's preffered encoding.

Some external dependencies, such as AviSynth, are still
not Unicode-enabled. utf8toansi() converts UTF-8 strings
into ANSI in two steps: UTF-8 -> wchar_t -> ANSI.
wchartoansi() is responsible for the second step of the conversion.
Conversion in just one step is not supported by WinAPI.

Since these character converting functions allocate the buffer
of necessary size, they also facilitate the removal of MAX_PATH limit
in places where fixed-size ANSI/WCHAR strings were used
as filename buffers.

getenv_utf8() wraps _wgetenv() converting its input from
and its output to UTF-8. Compared to plain getenv(),
getenv_utf8() requires a cleanup.

Because of that, in places that only test the existence of
an environment variable or compare its value with a string
consisting entirely of ASCII characters, the use of plain getenv()
is still preferred. (libavutil/log.c check_color_terminal()
is an example of such a place.)

Plain getenv() is also preffered in UNIX-only code,
such as bktr.c, fbdev_common.c, oss.c in libavdevice
or af_ladspa.c in libavfilter.
---
 libavutil/getenv_utf8.h    | 63 ++++++++++++++++++++++++++++++++++++++
 libavutil/wchar_filename.h | 51 ++++++++++++++++++++++++++++++
 2 files changed, 114 insertions(+)
 create mode 100644 libavutil/getenv_utf8.h

On Mon, 13 Jun 2022, Nil Admirari wrote:

> wchartoutf8() converts strings returned by WinAPI into UTF-8,
> which is FFmpeg's preffered encoding.
>
> Some external dependencies, such as AviSynth, are still
> not Unicode-enabled. utf8toansi() converts UTF-8 strings
> into ANSI in two steps: UTF-8 -> wchar_t -> ANSI.
> wchartoansi() is responsible for the second step of the conversion.
> Conversion in just one step is not supported by WinAPI.
>
> Since these character converting functions allocate the buffer
> of necessary size, they also facilitate the removal of MAX_PATH limit
> in places where fixed-size ANSI/WCHAR strings were used
> as filename buffers.
>
> getenv_utf8() wraps _wgetenv() converting its input from
> and its output to UTF-8. Compared to plain getenv(),
> getenv_utf8() requires a cleanup.
>
> Because of that, in places that only test the existence of
> an environment variable or compare its value with a string
> consisting entirely of ASCII characters, the use of plain getenv()
> is still preferred. (libavutil/log.c check_color_terminal()
> is an example of such a place.)
>
> Plain getenv() is also preffered in UNIX-only code,
> such as bktr.c, fbdev_common.c, oss.c in libavdevice
> or af_ladspa.c in libavfilter.
> ---
> libavutil/getenv_utf8.h    | 63 ++++++++++++++++++++++++++++++++++++++
> libavutil/wchar_filename.h | 51 ++++++++++++++++++++++++++++++
> 2 files changed, 114 insertions(+)
> create mode 100644 libavutil/getenv_utf8.h
>
> diff --git a/libavutil/getenv_utf8.h b/libavutil/getenv_utf8.h
> new file mode 100644
> index 0000000000..2c48a36355
> --- /dev/null
> +++ b/libavutil/getenv_utf8.h
> @@ -0,0 +1,63 @@
> +/*
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#ifndef AVUTIL_GETENV_UTF8_H
> +#define AVUTIL_GETENV_UTF8_H
> +
> +#include <stdlib.h>
> +
> +#include "mem.h"
> +
> +#ifdef _WIN32
> +
> +#include "libavutil/wchar_filename.h"
> +
> +static inline char *getenv_utf8(const char *varname)
> +{
> +    wchar_t *varname_w, *var_w;
> +    char *var;
> +
> +    if (utf8towchar(varname, &varname_w))
> +        return NULL;
> +    if (!varname_w)
> +        return NULL;
> +
> +    var_w = _wgetenv(varname_w);
> +    av_free(varname_w);
> +
> +    if (!var_w)
> +        return NULL;
> +    if (wchartoutf8(var_w, &var))
> +        return NULL;
> +
> +    return var;
> +
> +    // No CP_ACP fallback compared to other *_utf8() functions:
> +    // non UTF-8 strings must not be returned.
> +}

Note, there are Windows configurations that entirely lack getenv() (and 
presumably _wgetenv() too), namely the Windows Store App / Windows Phone 
App API subsets for Windows 8.x. I think those functions were allowed 
again in Windows 10 though...

In configure, we have this:

check_func_headers stdlib.h getenv

enabled getenv || echo "#define getenv(x) NULL" >> $TMPH

I guess we'd might have to add getenv to e.g. the SYSTEM_FUNCS list, so 
we'd get a HAVE_GETENV in config.h - then we could make getenv_utf8 a 
no-op if HAVE_GETENV is 0.

(I'm not in a situation where I can test and investigate properly right 
now, I can maybe look closer into it later.)

// Martin

> I guess we'd might have to add getenv to e.g. the SYSTEM_FUNCS list, so 
> we'd get a HAVE_GETENV in config.h - then we could make getenv_utf8 a 
> no-op if HAVE_GETENV is 0.

Done: https://ffmpeg.org/pipermail/ffmpeg-devel/2022-June/297596.html