| Message ID | 20171215070244.34419-1-wbsecg1@gmail.com |
|---|---|
| State | New |
| Headers | show |
On Fri, 15 Dec 2017 15:02:44 +0800 wbsecg1@gmail.com wrote: > From: wang-bin <wbsecg1@gmail.com> > > 1. a cvpixelbuffer backed by iosurface can always be converted to an opengl texture, using CGLTexImageIOSurface2D for macOS, and undocumented api texImageIOSurface(which is internally used by public api CVOpenGLESTextureCacheCreateTextureFromImage) for iOS4.0+. > 2. enabling the attribute can slow down decoding speed a lot. I tested many video clips on my macbook air. for example: ffmpeg -ss 00:00:00 -t 00:03:00 -hwaccel videotoolbox -an -i big_buck_bunny_1080p_h264.mov -f null ->/dev/null, result with the attribute > enabled: frame= 2082 fps= 85 q=-0.0 Lsize=N/A time=00:03:00.00 bitrate=N/A speed=7.34x > disabled: frame= 2031 fps=104 q=-0.0 Lsize=N/A time=00:03:00.00 bitrate=N/A speed=9.22x > --- > libavcodec/videotoolbox.c | 5 ----- > 1 file changed, 5 deletions(-) > > diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c > index 9d2f0afa20..24631684d7 100644 > --- a/libavcodec/videotoolbox.c > +++ b/libavcodec/videotoolbox.c > @@ -664,11 +664,6 @@ static CFDictionaryRef videotoolbox_buffer_attributes_create(int width, > CFDictionarySetValue(buffer_attributes, kCVPixelBufferIOSurfacePropertiesKey, io_surface_properties); > CFDictionarySetValue(buffer_attributes, kCVPixelBufferWidthKey, w); > CFDictionarySetValue(buffer_attributes, kCVPixelBufferHeightKey, h); > -#if TARGET_OS_IPHONE > - CFDictionarySetValue(buffer_attributes, kCVPixelBufferOpenGLESCompatibilityKey, kCFBooleanTrue); > -#else > - CFDictionarySetValue(buffer_attributes, kCVPixelBufferIOSurfaceOpenGLTextureCompatibilityKey, kCFBooleanTrue); > -#endif > > CFRelease(io_surface_properties); > CFRelease(cv_pix_fmt); Does this have a negative effect on compatibility or performance? (In both cases I'm asking about the case when actually using GL rendering.)
2017-12-16 2:52 GMT+08:00 wm4 <nfxjfg@googlemail.com>: > On Fri, 15 Dec 2017 15:02:44 +0800 > wbsecg1@gmail.com wrote: > >> From: wang-bin <wbsecg1@gmail.com> >> >> 1. a cvpixelbuffer backed by iosurface can always be converted to an opengl texture, using CGLTexImageIOSurface2D for macOS, and undocumented api texImageIOSurface(which is internally used by public api CVOpenGLESTextureCacheCreateTextureFromImage) for iOS4.0+. >> 2. enabling the attribute can slow down decoding speed a lot. I tested many video clips on my macbook air. for example: ffmpeg -ss 00:00:00 -t 00:03:00 -hwaccel videotoolbox -an -i big_buck_bunny_1080p_h264.mov -f null ->/dev/null, result with the attribute >> enabled: frame= 2082 fps= 85 q=-0.0 Lsize=N/A time=00:03:00.00 bitrate=N/A speed=7.34x >> disabled: frame= 2031 fps=104 q=-0.0 Lsize=N/A time=00:03:00.00 bitrate=N/A speed=9.22x >> --- >> libavcodec/videotoolbox.c | 5 ----- >> 1 file changed, 5 deletions(-) >> >> diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c >> index 9d2f0afa20..24631684d7 100644 >> --- a/libavcodec/videotoolbox.c >> +++ b/libavcodec/videotoolbox.c >> @@ -664,11 +664,6 @@ static CFDictionaryRef videotoolbox_buffer_attributes_create(int width, >> CFDictionarySetValue(buffer_attributes, kCVPixelBufferIOSurfacePropertiesKey, io_surface_properties); >> CFDictionarySetValue(buffer_attributes, kCVPixelBufferWidthKey, w); >> CFDictionarySetValue(buffer_attributes, kCVPixelBufferHeightKey, h); >> -#if TARGET_OS_IPHONE >> - CFDictionarySetValue(buffer_attributes, kCVPixelBufferOpenGLESCompatibilityKey, kCFBooleanTrue); >> -#else >> - CFDictionarySetValue(buffer_attributes, kCVPixelBufferIOSurfaceOpenGLTextureCompatibilityKey, kCFBooleanTrue); >> -#endif >> >> CFRelease(io_surface_properties); >> CFRelease(cv_pix_fmt); > > Does this have a negative effect on compatibility or performance? (In > both cases I'm asking about the case when actually using GL rendering.) Disabling the attribute improves performance in my tests. I can not find any document about these keys. What i know is the decoded cvpixelbuffer is backed by iosurface, and the api to create texture from iosurface is available since macOS10.6
On 12/16/2017 02:07 PM, Wang Bin wrote: > 2017-12-16 2:52 GMT+08:00 wm4 <nfxjfg@googlemail.com>: >> On Fri, 15 Dec 2017 15:02:44 +0800 >> wbsecg1@gmail.com wrote: >> >>> From: wang-bin <wbsecg1@gmail.com> >>> >>> 1. a cvpixelbuffer backed by iosurface can always be converted to an opengl texture, using CGLTexImageIOSurface2D for macOS, and undocumented api texImageIOSurface(which is internally used by public api CVOpenGLESTextureCacheCreateTextureFromImage) for iOS4.0+. >>> 2. enabling the attribute can slow down decoding speed a lot. I tested many video clips on my macbook air. for example: ffmpeg -ss 00:00:00 -t 00:03:00 -hwaccel videotoolbox -an -i big_buck_bunny_1080p_h264.mov -f null ->/dev/null, result with the attribute >>> enabled: frame= 2082 fps= 85 q=-0.0 Lsize=N/A time=00:03:00.00 bitrate=N/A speed=7.34x >>> disabled: frame= 2031 fps=104 q=-0.0 Lsize=N/A time=00:03:00.00 bitrate=N/A speed=9.22x >>> --- >>> libavcodec/videotoolbox.c | 5 ----- >>> 1 file changed, 5 deletions(-) >>> >>> diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c >>> index 9d2f0afa20..24631684d7 100644 >>> --- a/libavcodec/videotoolbox.c >>> +++ b/libavcodec/videotoolbox.c >>> @@ -664,11 +664,6 @@ static CFDictionaryRef videotoolbox_buffer_attributes_create(int width, >>> CFDictionarySetValue(buffer_attributes, kCVPixelBufferIOSurfacePropertiesKey, io_surface_properties); >>> CFDictionarySetValue(buffer_attributes, kCVPixelBufferWidthKey, w); >>> CFDictionarySetValue(buffer_attributes, kCVPixelBufferHeightKey, h); >>> -#if TARGET_OS_IPHONE >>> - CFDictionarySetValue(buffer_attributes, kCVPixelBufferOpenGLESCompatibilityKey, kCFBooleanTrue); >>> -#else >>> - CFDictionarySetValue(buffer_attributes, kCVPixelBufferIOSurfaceOpenGLTextureCompatibilityKey, kCFBooleanTrue); >>> -#endif >>> >>> CFRelease(io_surface_properties); >>> CFRelease(cv_pix_fmt); >> >> Does this have a negative effect on compatibility or performance? (In >> both cases I'm asking about the case when actually using GL rendering.) > > Disabling the attribute improves performance in my tests. I can not > find any document about these keys. What i know is the decoded > cvpixelbuffer is backed by iosurface, and the api to create texture > from iosurface is available since macOS10.6 Maybe you can add a flag to make them optional? They are documented at: https://developer.apple.com/documentation/corevideo/kcvpixelbufferopenglescompatibilitykey https://developer.apple.com/documentation/corevideo/kcvpixelbufferiosurfaceopengltexturecompatibilitykey Things may still work without them but I would like to follow the documentation whenever possible. And I think iOS DOES require the flag for those buffers to be consumed through GLES. Pure speculation here. GPU texture units usually require special memory layouts that may be suboptimal for the decoder. When these constraints are not met either the driver has to perform the conversion or you pay the penalty at access time. > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel >
2017-12-16 15:33 GMT+08:00 Xiaolei Yu <dreifachstein@gmail.com>: > On 12/16/2017 02:07 PM, Wang Bin wrote: >> 2017-12-16 2:52 GMT+08:00 wm4 <nfxjfg@googlemail.com>: >>> On Fri, 15 Dec 2017 15:02:44 +0800 >>> wbsecg1@gmail.com wrote: >>> >>>> From: wang-bin <wbsecg1@gmail.com> >>>> >>>> 1. a cvpixelbuffer backed by iosurface can always be converted to an opengl texture, using CGLTexImageIOSurface2D for macOS, and undocumented api texImageIOSurface(which is internally used by public api CVOpenGLESTextureCacheCreateTextureFromImage) for iOS4.0+. >>>> 2. enabling the attribute can slow down decoding speed a lot. I tested many video clips on my macbook air. for example: ffmpeg -ss 00:00:00 -t 00:03:00 -hwaccel videotoolbox -an -i big_buck_bunny_1080p_h264.mov -f null ->/dev/null, result with the attribute >>>> enabled: frame= 2082 fps= 85 q=-0.0 Lsize=N/A time=00:03:00.00 bitrate=N/A speed=7.34x >>>> disabled: frame= 2031 fps=104 q=-0.0 Lsize=N/A time=00:03:00.00 bitrate=N/A speed=9.22x >>>> --- >>>> libavcodec/videotoolbox.c | 5 ----- >>>> 1 file changed, 5 deletions(-) >>>> >>>> diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c >>>> index 9d2f0afa20..24631684d7 100644 >>>> --- a/libavcodec/videotoolbox.c >>>> +++ b/libavcodec/videotoolbox.c >>>> @@ -664,11 +664,6 @@ static CFDictionaryRef videotoolbox_buffer_attributes_create(int width, >>>> CFDictionarySetValue(buffer_attributes, kCVPixelBufferIOSurfacePropertiesKey, io_surface_properties); >>>> CFDictionarySetValue(buffer_attributes, kCVPixelBufferWidthKey, w); >>>> CFDictionarySetValue(buffer_attributes, kCVPixelBufferHeightKey, h); >>>> -#if TARGET_OS_IPHONE >>>> - CFDictionarySetValue(buffer_attributes, kCVPixelBufferOpenGLESCompatibilityKey, kCFBooleanTrue); >>>> -#else >>>> - CFDictionarySetValue(buffer_attributes, kCVPixelBufferIOSurfaceOpenGLTextureCompatibilityKey, kCFBooleanTrue); >>>> -#endif >>>> >>>> CFRelease(io_surface_properties); >>>> CFRelease(cv_pix_fmt); >>> >>> Does this have a negative effect on compatibility or performance? (In >>> both cases I'm asking about the case when actually using GL rendering.) >> >> Disabling the attribute improves performance in my tests. I can not >> find any document about these keys. What i know is the decoded >> cvpixelbuffer is backed by iosurface, and the api to create texture >> from iosurface is available since macOS10.6 > > Maybe you can add a flag to make them optional? > They are documented at: > https://developer.apple.com/documentation/corevideo/kcvpixelbufferopenglescompatibilitykey > https://developer.apple.com/documentation/corevideo/kcvpixelbufferiosurfaceopengltexturecompatibilitykey > > Things may still work without them but I would like to follow the documentation whenever possible. And I think iOS DOES require the flag for those buffers to be consumed through GLES. > Your links explain nothing. iOS does not need it as mentioned in the patch. I tested on iphone4s+iOS9 and some new devices. > Pure speculation here. GPU texture units usually require special memory layouts that may be suboptimal for the decoder. When these constraints are not met either the driver has to perform the conversion or you pay the penalty at access time. Not sure about memory layout. The fact is on mac the performance is much better without it. No difference on ios.
On 12/16/2017 04:19 PM, Wang Bin wrote: > 2017-12-16 15:33 GMT+08:00 Xiaolei Yu <dreifachstein@gmail.com>: >> On 12/16/2017 02:07 PM, Wang Bin wrote: >>> 2017-12-16 2:52 GMT+08:00 wm4 <nfxjfg@googlemail.com>: >>>> On Fri, 15 Dec 2017 15:02:44 +0800 >>>> wbsecg1@gmail.com wrote: >>>> >>>>> From: wang-bin <wbsecg1@gmail.com> >>>>> >>>>> 1. a cvpixelbuffer backed by iosurface can always be converted to an opengl texture, using CGLTexImageIOSurface2D for macOS, and undocumented api texImageIOSurface(which is internally used by public api CVOpenGLESTextureCacheCreateTextureFromImage) for iOS4.0+. >>>>> 2. enabling the attribute can slow down decoding speed a lot. I tested many video clips on my macbook air. for example: ffmpeg -ss 00:00:00 -t 00:03:00 -hwaccel videotoolbox -an -i big_buck_bunny_1080p_h264.mov -f null ->/dev/null, result with the attribute >>>>> enabled: frame= 2082 fps= 85 q=-0.0 Lsize=N/A time=00:03:00.00 bitrate=N/A speed=7.34x >>>>> disabled: frame= 2031 fps=104 q=-0.0 Lsize=N/A time=00:03:00.00 bitrate=N/A speed=9.22x >>>>> --- >>>>> libavcodec/videotoolbox.c | 5 ----- >>>>> 1 file changed, 5 deletions(-) >>>>> >>>>> diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c >>>>> index 9d2f0afa20..24631684d7 100644 >>>>> --- a/libavcodec/videotoolbox.c >>>>> +++ b/libavcodec/videotoolbox.c >>>>> @@ -664,11 +664,6 @@ static CFDictionaryRef videotoolbox_buffer_attributes_create(int width, >>>>> CFDictionarySetValue(buffer_attributes, kCVPixelBufferIOSurfacePropertiesKey, io_surface_properties); >>>>> CFDictionarySetValue(buffer_attributes, kCVPixelBufferWidthKey, w); >>>>> CFDictionarySetValue(buffer_attributes, kCVPixelBufferHeightKey, h); >>>>> -#if TARGET_OS_IPHONE >>>>> - CFDictionarySetValue(buffer_attributes, kCVPixelBufferOpenGLESCompatibilityKey, kCFBooleanTrue); >>>>> -#else >>>>> - CFDictionarySetValue(buffer_attributes, kCVPixelBufferIOSurfaceOpenGLTextureCompatibilityKey, kCFBooleanTrue); >>>>> -#endif >>>>> >>>>> CFRelease(io_surface_properties); >>>>> CFRelease(cv_pix_fmt); >>>> >>>> Does this have a negative effect on compatibility or performance? (In >>>> both cases I'm asking about the case when actually using GL rendering.) >>> >>> Disabling the attribute improves performance in my tests. I can not >>> find any document about these keys. What i know is the decoded >>> cvpixelbuffer is backed by iosurface, and the api to create texture >>> from iosurface is available since macOS10.6 >> >> Maybe you can add a flag to make them optional? >> They are documented at: >> https://developer.apple.com/documentation/corevideo/kcvpixelbufferopenglescompatibilitykey >> https://developer.apple.com/documentation/corevideo/kcvpixelbufferiosurfaceopengltexturecompatibilitykey >> >> Things may still work without them but I would like to follow the documentation whenever possible. And I think iOS DOES require the flag for those buffers to be consumed through GLES. >> > > Your links explain nothing. iOS does not need it as mentioned in the > patch. I tested on iphone4s+iOS9 and some new devices. Will find time to test this week. VTDecompressionOutputCallback actually returns a CVImageBuffer and blindly casting it to a CVPixelBuffer is dangerous. I will test if the type ids of the output are affected with different key combinations. > >> Pure speculation here. GPU texture units usually require special memory layouts that may be suboptimal for the decoder. When these constraints are not met either the driver has to perform the conversion or you pay the penalty at access time. > > Not sure about memory layout. The fact is on mac the performance is > much better without it. No difference on ios. > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel >
diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c index 9d2f0afa20..24631684d7 100644 --- a/libavcodec/videotoolbox.c +++ b/libavcodec/videotoolbox.c @@ -664,11 +664,6 @@ static CFDictionaryRef videotoolbox_buffer_attributes_create(int width, CFDictionarySetValue(buffer_attributes, kCVPixelBufferIOSurfacePropertiesKey, io_surface_properties); CFDictionarySetValue(buffer_attributes, kCVPixelBufferWidthKey, w); CFDictionarySetValue(buffer_attributes, kCVPixelBufferHeightKey, h); -#if TARGET_OS_IPHONE - CFDictionarySetValue(buffer_attributes, kCVPixelBufferOpenGLESCompatibilityKey, kCFBooleanTrue); -#else - CFDictionarySetValue(buffer_attributes, kCVPixelBufferIOSurfaceOpenGLTextureCompatibilityKey, kCFBooleanTrue); -#endif CFRelease(io_surface_properties); CFRelease(cv_pix_fmt);
From: wang-bin <wbsecg1@gmail.com> 1. a cvpixelbuffer backed by iosurface can always be converted to an opengl texture, using CGLTexImageIOSurface2D for macOS, and undocumented api texImageIOSurface(which is internally used by public api CVOpenGLESTextureCacheCreateTextureFromImage) for iOS4.0+. 2. enabling the attribute can slow down decoding speed a lot. I tested many video clips on my macbook air. for example: ffmpeg -ss 00:00:00 -t 00:03:00 -hwaccel videotoolbox -an -i big_buck_bunny_1080p_h264.mov -f null ->/dev/null, result with the attribute enabled: frame= 2082 fps= 85 q=-0.0 Lsize=N/A time=00:03:00.00 bitrate=N/A speed=7.34x disabled: frame= 2031 fps=104 q=-0.0 Lsize=N/A time=00:03:00.00 bitrate=N/A speed=9.22x --- libavcodec/videotoolbox.c | 5 ----- 1 file changed, 5 deletions(-)