| Message ID | 20171006165615.39580-1-dheitmueller@ltnglobal.com |
|---|---|
| State | Superseded |
| Headers | show |
2017-10-06 18:56 GMT+02:00 Devin Heitmueller <dheitmueller@ltnglobal.com>: > From: Devin Heitmueller <dheitmueller@kernellabs.com> > > Hook in libklvanc and use it for output of EIA-708 captions over > SDI. The bulk of this patch is just general support for ancillary > data for the Decklink SDI module - the real work for construction > of the EIA-708 CDP and VANC line construction is done by libklvanc. Nothing except the decklink device could use VANC? Carl Eugen
Hello Carl, > On Oct 6, 2017, at 5:07 PM, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote: > > 2017-10-06 18:56 GMT+02:00 Devin Heitmueller <dheitmueller@ltnglobal.com>: >> From: Devin Heitmueller <dheitmueller@kernellabs.com> >> >> Hook in libklvanc and use it for output of EIA-708 captions over >> SDI. The bulk of this patch is just general support for ancillary >> data for the Decklink SDI module - the real work for construction >> of the EIA-708 CDP and VANC line construction is done by libklvanc. > > Nothing except the decklink device could use VANC? You could absolutely have other SDI device types which can contain VANC. This was a key reason that we put all the business logic for VANC processing in a separate library. The goal behind developing libklvanc was to separate out VANC processing from device and application specific business logic. This allows us to have a single parser and implementation of popular VANC protocols in a single library that can be reused by VLC, OBE, and ffmpeg. It also allows the VANC processing to be shared across different device types, although admittedly there are not many vendors other than BlackMagic that are very popular. The point of the remark in the commit message though was to observe that most of the code in the patch implements the decklink specific glue for accessing VANC lines. The libklvanc library provides all the functions for parsing/generation of the VANC packets. Devin
2017-10-06 23:21 GMT+02:00 Devin Heitmueller <dheitmueller@ltnglobal.com>: > Hello Carl, > >> On Oct 6, 2017, at 5:07 PM, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote: >> >> 2017-10-06 18:56 GMT+02:00 Devin Heitmueller <dheitmueller@ltnglobal.com>: >>> From: Devin Heitmueller <dheitmueller@kernellabs.com> >>> >>> Hook in libklvanc and use it for output of EIA-708 captions over >>> SDI. The bulk of this patch is just general support for ancillary >>> data for the Decklink SDI module - the real work for construction >>> of the EIA-708 CDP and VANC line construction is done by libklvanc. >> >> Nothing except the decklink device could use VANC? > > You could absolutely have other SDI device types which can contain VANC. Sorry, what I meant was: Nothing inside FFmpeg except the decklink device could use VANC? Thank you, Carl Eugen
> > Sorry, what I meant was: > Nothing inside FFmpeg except the decklink device could use > VANC? Ah, I understand now. Yes, the decklink device is currently the only SDI device which is supported by libavdevice. I’ve got a whole pile of patches coming which add support for a variety of protocols for both capture and output (e.g. EIA-708, SCTE-104, AFD, SMPTE 2038, etc). But today the decklink module is the only device supported. Would love to see more SDI devices supported and potentially interested in adding such support myself if we can find good candidates. The DVEO/linsys cards are largely obsolete and the AJA boards are significantly more expensive than any of BlackMagic’s cards. If anyone has good experiences with other vendors I would like to hear about it (and there may be an opportunity to extend libavdevice to support another SDI vendor). Devin
On Fri, Oct 6, 2017 at 4:31 PM, Devin Heitmueller <dheitmueller@ltnglobal.com> wrote: > > Ah, I understand now. > > Yes, the decklink device is currently the only SDI device which is supported by libavdevice. I’ve got a whole pile of patches coming which add support for a variety of protocols for both capture and output (e.g. EIA-708, SCTE-104, AFD, SMPTE 2038, etc). But today the decklink module is the only device supported. > > Would love to see more SDI devices supported and potentially interested in adding such support myself if we can find good candidates. The DVEO/linsys cards are largely obsolete and the AJA boards are significantly more expensive than any of BlackMagic’s cards. If anyone has good experiences with other vendors I would like to hear about it (and there may be an opportunity to extend libavdevice to support another SDI vendor). Bluefish has some very nice ($$$) SDI cards with Linux support. I think their SDK also supports v4l2. Have never been able to get my hands on one though.
> On Oct 6, 2017, at 12:56 PM, Devin Heitmueller <dheitmueller@ltnglobal.com> wrote: > > From: Devin Heitmueller <dheitmueller@kernellabs.com> > > Hook in libklvanc and use it for output of EIA-708 captions over > SDI. The bulk of this patch is just general support for ancillary > data for the Decklink SDI module - the real work for construction > of the EIA-708 CDP and VANC line construction is done by libklvanc. > > Libklvanc can be found at: https://github.com/stoth68000/libklvanc > > Signed-off-by: Devin Heitmueller <dheitmueller@ltnglobal.com> Any additional technical comments related to this patch, or should I resubmit for merge? Devin
On 10/18/17 1:31 PM, Devin Heitmueller wrote: >> On Oct 6, 2017, at 12:56 PM, Devin Heitmueller <dheitmueller@ltnglobal.com> wrote: >> >> From: Devin Heitmueller <dheitmueller@kernellabs.com> >> >> Hook in libklvanc and use it for output of EIA-708 captions over >> SDI. The bulk of this patch is just general support for ancillary >> data for the Decklink SDI module - the real work for construction >> of the EIA-708 CDP and VANC line construction is done by libklvanc. >> >> Libklvanc can be found at: https://github.com/stoth68000/libklvanc >> >> Signed-off-by: Devin Heitmueller <dheitmueller@ltnglobal.com> > Any additional technical comments related to this patch, or should I resubmit for merge? > > Devin I was going to actually test this with some old broadcast equipment I have just dying for a purpose, but I don't see how to generate AV_PKT_DATA_A53_CC side packet data except using the Decklink capture. I have A53 documentation, but it just refers to CEA-708 (or SMPTE 334, or ... what an unraveling ball of yarn it is. Looks like I could spend a months income on standards just trying to learn how this is encoded). On a side note, can AV_PKT_DATA_A53_CC be used for something besides CEA-708? Not sure I understand the line between A53 CC encoding (which is at least in part what this generates, right?) and CEA-708 (which is what this takes, right?) and why this side data is called A53_CC? I know these questions are outside the scope that you were asking... Deron
Hi Deron, > I was going to actually test this with some old broadcast equipment I have just dying for a purpose, but I don't see how to generate AV_PKT_DATA_A53_CC side packet data except using the Decklink capture. I have A53 documentation, but it just refers to CEA-708 (or SMPTE 334, or ... what an unraveling ball of yarn it is. Looks like I could spend a months income on standards just trying to learn how this is encoded). Yeah. You could certainly spend a good bit of cash if you had to buy the individual specs. Worth noting that the ATSC specs are freely available though on their website, and the CEA-708 is largely described in the FCC specification (not a substitute for the real thing, but good enough for the casual reader). SMPTE has a “digital library” where you can get access to *all* their specs with a subscription of around $600/year. It’s not ideal for a non-professional, but for people who *need* the specs it’s way cheaper than buying them piecemeal for $120/spec. > > On a side note, can AV_PKT_DATA_A53_CC be used for something besides CEA-708? Not sure I understand the line between A53 CC encoding (which is at least in part what this generates, right?) and CEA-708 (which is what this takes, right?) and why this side data is called A53_CC? > > I know these questions are outside the scope that you were asking… > No problem. I should really write a primer on this stuff since there are a whole bunch of specs which are inter-related. Briefly…. CEA-708 is what non-technical people typically consider to be “digital closed captions”. They represent the standard that replaces old fashioned NTSC closed captions, which were described in EIA/CEA-608. The spec describes what could be characterized as a protocol stack of functionality, including transport through presentation layers (i.e. how the captions are constructed, rules for how to render them on-screen, etc). CEA-708 also includes a construct for tunneling old CEA-608 packets. In fact, most CEA-708 streams are really just up-converted from CEA-608, since the FCC requires both to be supported and 608 is a subset in functionality of 708. On the other hand, you can’t typically down convert 708 to 608 since there are a bunch of formatting codes in 708 which have no corresponding capability in 608. If you’re using VLC or most other applications, they will claim to render 708 captions, but they’re really just rendering the 608 captions contained in 708. One component of the CEA-708 spec describes a “CDP”, which is “Caption Distribution Packet”. This is a low-level packet format which includes not just multiple caption streams but also timecodes and service data (e.g. caption languages, etc). CDP packets can be sent over a number of different physical transports, including old-fashioned serial ports. SMPTE 334M describes how to transport CEA-708 CDP packets over an SDI link in the VANC area of the frame. A53 refers to the ATSC A/53 specification, which basically refers to how digital TV is transmitted over-the-air. One part of that spec includes how to embed CEA-708 captions into an MPEG2 transport stream. The A/53 spec basically says how to embed the CEA-708 caption bytes into an MPEG-2 stream, and then refers you to CEA-708 for the details of what to do with those bytes. Both the CEA-708 CDP format and A/53 come down to a series of three byte packets which contain the actual captioning data. This corresponds to what is being serialized in AV_PKT_DATA_A53_CC. In order to encode an SDI feed into an MPEG-2 stream, you would need to deconstruct the CDP, extract the captioning bytes, and load them into the side data packet. Once that’s done, the avpacket is handed off to an H.264/MPEG-2 video encoder, which knows how to take those captioning bytes and embed them into the compressed video (using the MPEG-2 user_data field if it’s MPEG-2 video, or the SEI field if it’s H.264). That series of three-byte packets is essentially the “lowest common denominator” representing the captioning data (assuming you only care about closed captions and not timecodes or service info). I have use cases where this stuff should really be preserved, and am weighing the merits of introducing a new side data format for the CDP which preserves all the info, and then encoders can extract what they need. There are plusses/minuses to this approach and it’s still under consideration. I hope that gives you a bit more background. Cheers, Devin
> On Oct 6, 2017, at 5:31 PM, Devin Heitmueller <dheitmueller@ltnglobal.com> wrote: > >> Sorry, what I meant was: >> Nothing inside FFmpeg except the decklink device could use >> VANC? > > Ah, I understand now. > > Yes, the decklink device is currently the only SDI device which is supported by libavdevice. I’ve got a whole pile of patches coming which add support for a variety of protocols for both capture and output (e.g. EIA-708, SCTE-104, AFD, SMPTE 2038, etc). But today the decklink module is the only device supported. > > Would love to see more SDI devices supported and potentially interested in adding such support myself if we can find good candidates. The DVEO/linsys cards are largely obsolete and the AJA boards are significantly more expensive than any of BlackMagic’s cards. If anyone has good experiences with other vendors I would like to hear about it (and there may be an opportunity to extend libavdevice to support another SDI vendor). The President of AJA has publicly stated an intent to add an open license to their SDK, https://twitter.com/ajaprez/status/910100436224499713 <https://twitter.com/ajaprez/status/910100436224499713>. I’m glad to hear that handling other VANC data is in the works, I’m particularly interested in VITC and EIA-608 with inputs. Dave Rice
Hi Dave, > > The President of AJA has publicly stated an intent to add an open license to their SDK, https://twitter.com/ajaprez/status/910100436224499713 <https://twitter.com/ajaprez/status/910100436224499713>. This is certainly good news. Looking at AJA’s offering is on my TODO list but I just haven’t found the time to pick up a card and dig into their SDK. > I’m glad to hear that handling other VANC data is in the works, I’m particularly interested in VITC and EIA-608 with inputs. I am certainly interested in supporting VITC, assuming you’re talking about SMPTE 12M/RP-188 time codes. EIA-608 wouldn’t be very hard to do, although there isn’t much equipment out there which does CEA-608 but not CEA-708. Maybe some old legacy pre-HD equipment. It would actually be pretty easy to do EIA-608 and just shoe-horn it in as AV_FRAME_DATA_A53_CC side data. You would just need to compute the correct value for cc_count based on the frame rate, create an array of bytes of size [cc_count][3], and then fill the first two entries with the EIA-608 byte pairs. Creating a new side-data type would seem like the reasonable approach at first glance, but then you have to duplicate all the logic on the insertion side for the various encoder codecs. That said, supporting both 608 and 708 creates an unfortunate side effect - you have to write logic to decide which takes precedence if both are in an SDI frame (or expose configuration options to let the user specify). Any SDI capture routine would have to choose one or the other, since downstream codecs general don’t have the capacity to insert both into a transport stream. My inclination would probably be to simply ignore EIA-608 if there are also 708 VANC packets present, but I can imagine you would also want a config option to allow the user to override that behavior. Devin
On Fri, 6 Oct 2017, Devin Heitmueller wrote: > From: Devin Heitmueller <dheitmueller@kernellabs.com> > > Hook in libklvanc and use it for output of EIA-708 captions over > SDI. The bulk of this patch is just general support for ancillary > data for the Decklink SDI module - the real work for construction > of the EIA-708 CDP and VANC line construction is done by libklvanc. > > Libklvanc can be found at: https://github.com/stoth68000/libklvanc Sorry for the delay, I had little time lately. In general I think it is OK to put VANC functionality into a library, but libklvanc does not seem like a very mature one, it has some pretty generic function names without namespacing, e.g. "generate_vanc_line". Or it is using simple printf for the dumper functions. You plan to work on these kind of issues to make it more like a "stable" generic library? > > Signed-off-by: Devin Heitmueller <dheitmueller@ltnglobal.com> > --- > configure | 3 ++ > libavcodec/v210enc.c | 8 +++ > libavdevice/decklink_common.h | 1 + > libavdevice/decklink_enc.cpp | 113 +++++++++++++++++++++++++++++++++++++++--- > 4 files changed, 119 insertions(+), 6 deletions(-) > > diff --git a/configure b/configure > index 391c141e7a..18647896b1 100755 > --- a/configure > +++ b/configure > @@ -238,6 +238,7 @@ External library support: > --enable-libgsm enable GSM de/encoding via libgsm [no] > --enable-libiec61883 enable iec61883 via libiec61883 [no] > --enable-libilbc enable iLBC de/encoding via libilbc [no] > + --enable-libklvanc enable Kernel Labs VANC processing [no] > --enable-libkvazaar enable HEVC encoding via libkvazaar [no] > --enable-libmodplug enable ModPlug via libmodplug [no] > --enable-libmp3lame enable MP3 encoding via libmp3lame [no] > @@ -1603,6 +1604,7 @@ EXTERNAL_LIBRARY_LIST=" > libgsm > libiec61883 > libilbc > + libklvanc > libkvazaar > libmodplug > libmp3lame > @@ -6027,6 +6029,7 @@ enabled libx264 && { use_pkg_config libx264 x264 "stdint.h x264.h" x26 > enabled libx265 && require_pkg_config libx265 x265 x265.h x265_api_get && > require_cpp_condition x265.h "X265_BUILD >= 68" > enabled libxavs && require libxavs "stdint.h xavs.h" xavs_encoder_encode -lxavs > +enabled libklvanc && require libklvanc libklvanc/vanc.h vanc_context_create -lklvanc > enabled libxvid && require libxvid xvid.h xvid_global -lxvidcore > enabled libzimg && require_pkg_config libzimg "zimg >= 2.3.0" zimg.h zimg_get_api_version > enabled libzmq && require_pkg_config libzmq libzmq zmq.h zmq_ctx_new > diff --git a/libavcodec/v210enc.c b/libavcodec/v210enc.c > index a6afbbfc41..44cc3c5c81 100644 > --- a/libavcodec/v210enc.c > +++ b/libavcodec/v210enc.c > @@ -123,6 +123,7 @@ static int encode_frame(AVCodecContext *avctx, AVPacket *pkt, > int aligned_width = ((avctx->width + 47) / 48) * 48; > int stride = aligned_width * 8 / 3; > int line_padding = stride - ((avctx->width * 8 + 11) / 12) * 4; > + AVFrameSideData *side_data = NULL; initializer seems unnecesarry. > int h, w, ret; > uint8_t *dst; > > @@ -233,6 +234,13 @@ static int encode_frame(AVCodecContext *avctx, AVPacket *pkt, > } > } > > + side_data = av_frame_get_side_data(pic, AV_FRAME_DATA_A53_CC); > + if (side_data && side_data->size) { > + uint8_t* buf = av_packet_new_side_data(pkt, AV_PKT_DATA_A53_CC, side_data->size); > + if (buf) > + memcpy(buf, side_data->data, side_data->size); else return AVERROR(ENOMEM)? > + } > + > pkt->flags |= AV_PKT_FLAG_KEY; > *got_packet = 1; > return 0; > diff --git a/libavdevice/decklink_common.h b/libavdevice/decklink_common.h > index 6b2525fb53..285a244000 100644 > --- a/libavdevice/decklink_common.h > +++ b/libavdevice/decklink_common.h > @@ -78,6 +78,7 @@ struct decklink_ctx { > AVStream *audio_st; > AVStream *video_st; > AVStream *teletext_st; > + uint16_t cdp_sequence_num; > > /* Options */ > int list_devices; > diff --git a/libavdevice/decklink_enc.cpp b/libavdevice/decklink_enc.cpp > index 81df563b3b..3049e936a9 100644 > --- a/libavdevice/decklink_enc.cpp > +++ b/libavdevice/decklink_enc.cpp > @@ -38,16 +38,20 @@ extern "C" { > > #include "decklink_common.h" > #include "decklink_enc.h" > - > +#if CONFIG_LIBKLVANC > +#include "libklvanc/vanc.h" > +#include "libklvanc/vanc-lines.h" > +#include "libklvanc/pixels.h" > +#endif > > /* DeckLink callback class declaration */ > class decklink_frame : public IDeckLinkVideoFrame > { > public: > decklink_frame(struct decklink_ctx *ctx, AVFrame *avframe, AVCodecID codec_id, int height, int width) : > - _ctx(ctx), _avframe(avframe), _avpacket(NULL), _codec_id(codec_id), _height(height), _width(width), _refs(1) { } > + _ctx(ctx), _avframe(avframe), _avpacket(NULL), _codec_id(codec_id), _ancillary(NULL), _height(height), _width(width), _refs(1) { } > decklink_frame(struct decklink_ctx *ctx, AVPacket *avpacket, AVCodecID codec_id, int height, int width) : > - _ctx(ctx), _avframe(NULL), _avpacket(avpacket), _codec_id(codec_id), _height(height), _width(width), _refs(1) { } > + _ctx(ctx), _avframe(NULL), _avpacket(avpacket), _codec_id(codec_id), _ancillary(NULL), _height(height), _width(width), _refs(1) { } > > virtual long STDMETHODCALLTYPE GetWidth (void) { return _width; } > virtual long STDMETHODCALLTYPE GetHeight (void) { return _height; } > @@ -87,8 +91,13 @@ public: > } > > virtual HRESULT STDMETHODCALLTYPE GetTimecode (BMDTimecodeFormat format, IDeckLinkTimecode **timecode) { return S_FALSE; } > - virtual HRESULT STDMETHODCALLTYPE GetAncillaryData(IDeckLinkVideoFrameAncillary **ancillary) { return S_FALSE; } > - > + virtual HRESULT STDMETHODCALLTYPE GetAncillaryData(IDeckLinkVideoFrameAncillary **ancillary) > + { > + *ancillary = _ancillary; > + return _ancillary ? S_OK : S_FALSE; > + } > + virtual HRESULT STDMETHODCALLTYPE SetAncillaryData(IDeckLinkVideoFrameAncillary > + *ancillary) { _ancillary = ancillary; return S_OK; } > virtual HRESULT STDMETHODCALLTYPE QueryInterface(REFIID iid, LPVOID *ppv) { return E_NOINTERFACE; } > virtual ULONG STDMETHODCALLTYPE AddRef(void) { return ++_refs; } > virtual ULONG STDMETHODCALLTYPE Release(void) > @@ -106,6 +115,7 @@ public: > AVFrame *_avframe; > AVPacket *_avpacket; > AVCodecID _codec_id; > + IDeckLinkVideoFrameAncillary *_ancillary; > int _height; > int _width; > > @@ -169,7 +179,7 @@ static int decklink_setup_video(AVFormatContext *avctx, AVStream *st) > return -1; > } > if (ctx->dlo->EnableVideoOutput(ctx->bmd_mode, > - bmdVideoOutputFlagDefault) != S_OK) { > + bmdVideoOutputVANC) != S_OK) { Are you sure this does not fail of the hardware has no support for Ancillary data? > av_log(avctx, AV_LOG_ERROR, "Could not enable video output!\n"); > return -1; > } > @@ -265,6 +275,93 @@ av_cold int ff_decklink_write_trailer(AVFormatContext *avctx) > return 0; > } > > +#if CONFIG_LIBKLVANC > +static int decklink_construct_vanc(struct decklink_ctx *ctx, AVPacket *pkt, > + decklink_frame *frame) > +{ > + struct vanc_line_set_s vanc_lines; > + memset(&vanc_lines, 0, sizeof(vanc_lines)); maybe struct xx = {0}; instead? > + > + int size; > + const uint8_t *data = av_packet_get_side_data(pkt, AV_PKT_DATA_A53_CC, &size); > + if (data) { > + struct packet_eia_708b_s *pkt; > + uint16_t *cdp; > + uint16_t len; > + uint8_t cc_count = size / 3; I'd put an upper limit to cc_count to be sure I won't overwrite something... > + > + klvanc_create_eia708_cdp(&pkt); Missing error check > + klvanc_set_framerate_EIA_708B(pkt, ctx->bmd_tb_num, ctx->bmd_tb_den); Missing error check. I have not checked, but it is possible that you have to av_reduce() on the timebase before calling set_frame_rate to get rid of "1000/25000"-like time bases. > + > + /* CC data */ > + pkt->header.ccdata_present = 1; > + pkt->ccdata.cc_count = cc_count; > + for (size_t i = 0; i < cc_count; i++) { > + if (data [3*i] & 0x40) > + pkt->ccdata.cc[i].cc_valid = 1; > + pkt->ccdata.cc[i].cc_type = data[3*i] & 0x03; > + pkt->ccdata.cc[i].cc_data[0] = data[3*i+1]; > + pkt->ccdata.cc[i].cc_data[1] = data[3*i+2]; > + } > + > + klvanc_finalize_EIA_708B(pkt, ctx->cdp_sequence_num++); > + convert_EIA_708B_to_words(pkt, &cdp, &len); > + klvanc_destroy_eia708_cdp(pkt); > + > + vanc_line_insert(&vanc_lines, cdp, len, 11, 0); Maybe some error checks here as well? > + } > + > + IDeckLinkVideoFrameAncillary *vanc; > + int result = ctx->dlo->CreateAncillaryData(bmdFormat10BitYUV, &vanc); I guess this might fail if the card or the output has no 10 bit anicllary data support. So instead of outputting ancillary data always, maybe it is better to introduce a new option to be able to enable it explicitly but disable it by default? > + if (result != S_OK) { > + fprintf(stderr, "Failed to create vanc\n"); av_log > + return -1; > + } > + > + /* Now that we've got all the VANC lines in a nice orderly manner, generate the > + final VANC sections for the Decklink output */ > + for (int i = 0; i < vanc_lines.num_lines; i++) { > + struct vanc_line_s *line = vanc_lines.lines[i]; > + uint16_t *out_line; > + int real_line; > + int out_len; > + void *buf; > + > + if (line == NULL) > + break; > + > + real_line = line->line_number; > +#if 0 > + /* FIXME: include hack for certain Decklink cards which mis-represent > + line numbers for pSF frames */ > + if (decklink_sys->b_psf_interlaced) > + real_line = Calculate1080psfVancLine(line->line_number); > +#endif > + result = vanc->GetBufferForVerticalBlankingLine(real_line, &buf); > + if (result != S_OK) { > + fprintf(stderr, "Failed to get VANC line %d: %d", real_line, result); av_log > + vanc_line_free(line); > + continue; > + } > + > + /* Generate the full line taking into account all VANC packets on that line */ > + generate_vanc_line(line, &out_line, &out_len, ctx->bmd_width); Error checks... > + > + /* Repack the 16-bit ints into 10-bit, and push into final buffer */ > + klvanc_y10_to_v210(out_line, (uint8_t *) buf, out_len); > + free(out_line); > + vanc_line_free(line); > + } > + > + result = frame->SetAncillaryData(vanc); > + if (result != S_OK) { > + fprintf(stderr, "Failed to set vanc: %d", result); > + return AVERROR(EIO); > + } > + return 0; > +} > +#endif > + > static int decklink_write_video_packet(AVFormatContext *avctx, AVPacket *pkt) > { > struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data; > @@ -299,6 +396,10 @@ static int decklink_write_video_packet(AVFormatContext *avctx, AVPacket *pkt) > } > > frame = new decklink_frame(ctx, avpacket, st->codecpar->codec_id, ctx->bmd_height, ctx->bmd_width); > + > +#if CONFIG_LIBKLVANC > + decklink_construct_vanc(ctx, pkt, frame); Fail on error here as well? > +#endif Missing docs update. Regards, Marton > } > > if (!frame) { > -- > 2.13.2 > > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> On Oct 25, 2017, at 2:23 PM, Marton Balint <cus@passwd.hu> wrote: > > > On Fri, 6 Oct 2017, Devin Heitmueller wrote: > >> From: Devin Heitmueller <dheitmueller@kernellabs.com <mailto:dheitmueller@kernellabs.com>> >> >> Hook in libklvanc and use it for output of EIA-708 captions over >> SDI. The bulk of this patch is just general support for ancillary >> data for the Decklink SDI module - the real work for construction >> of the EIA-708 CDP and VANC line construction is done by libklvanc. >> >> Libklvanc can be found at: https://github.com/stoth68000/libklvanc <https://github.com/stoth68000/libklvanc> > > Sorry for the delay, I had little time lately. In general I think it is OK to put VANC functionality into a library, but libklvanc does not seem like a very mature one, it has some pretty generic function names without namespacing, e.g. "generate_vanc_line". Or it is using simple printf for the dumper functions. You plan to work on these kind of issues to make it more like a "stable" generic library? Yeah, the name spacing and logging are known issues and have been on my todo list for a while. The focus has been on the core functionality for VANC management and protocol support, and there clearly needs a bit more polish on some of the peripheral areas. Thanks for providing feedback. I will incorporate your suggestions and submit a revised patch this week. Regards, Devin
diff --git a/configure b/configure index 391c141e7a..18647896b1 100755 --- a/configure +++ b/configure @@ -238,6 +238,7 @@ External library support: --enable-libgsm enable GSM de/encoding via libgsm [no] --enable-libiec61883 enable iec61883 via libiec61883 [no] --enable-libilbc enable iLBC de/encoding via libilbc [no] + --enable-libklvanc enable Kernel Labs VANC processing [no] --enable-libkvazaar enable HEVC encoding via libkvazaar [no] --enable-libmodplug enable ModPlug via libmodplug [no] --enable-libmp3lame enable MP3 encoding via libmp3lame [no] @@ -1603,6 +1604,7 @@ EXTERNAL_LIBRARY_LIST=" libgsm libiec61883 libilbc + libklvanc libkvazaar libmodplug libmp3lame @@ -6027,6 +6029,7 @@ enabled libx264 && { use_pkg_config libx264 x264 "stdint.h x264.h" x26 enabled libx265 && require_pkg_config libx265 x265 x265.h x265_api_get && require_cpp_condition x265.h "X265_BUILD >= 68" enabled libxavs && require libxavs "stdint.h xavs.h" xavs_encoder_encode -lxavs +enabled libklvanc && require libklvanc libklvanc/vanc.h vanc_context_create -lklvanc enabled libxvid && require libxvid xvid.h xvid_global -lxvidcore enabled libzimg && require_pkg_config libzimg "zimg >= 2.3.0" zimg.h zimg_get_api_version enabled libzmq && require_pkg_config libzmq libzmq zmq.h zmq_ctx_new diff --git a/libavcodec/v210enc.c b/libavcodec/v210enc.c index a6afbbfc41..44cc3c5c81 100644 --- a/libavcodec/v210enc.c +++ b/libavcodec/v210enc.c @@ -123,6 +123,7 @@ static int encode_frame(AVCodecContext *avctx, AVPacket *pkt, int aligned_width = ((avctx->width + 47) / 48) * 48; int stride = aligned_width * 8 / 3; int line_padding = stride - ((avctx->width * 8 + 11) / 12) * 4; + AVFrameSideData *side_data = NULL; int h, w, ret; uint8_t *dst; @@ -233,6 +234,13 @@ static int encode_frame(AVCodecContext *avctx, AVPacket *pkt, } } + side_data = av_frame_get_side_data(pic, AV_FRAME_DATA_A53_CC); + if (side_data && side_data->size) { + uint8_t* buf = av_packet_new_side_data(pkt, AV_PKT_DATA_A53_CC, side_data->size); + if (buf) + memcpy(buf, side_data->data, side_data->size); + } + pkt->flags |= AV_PKT_FLAG_KEY; *got_packet = 1; return 0; diff --git a/libavdevice/decklink_common.h b/libavdevice/decklink_common.h index 6b2525fb53..285a244000 100644 --- a/libavdevice/decklink_common.h +++ b/libavdevice/decklink_common.h @@ -78,6 +78,7 @@ struct decklink_ctx { AVStream *audio_st; AVStream *video_st; AVStream *teletext_st; + uint16_t cdp_sequence_num; /* Options */ int list_devices; diff --git a/libavdevice/decklink_enc.cpp b/libavdevice/decklink_enc.cpp index 81df563b3b..3049e936a9 100644 --- a/libavdevice/decklink_enc.cpp +++ b/libavdevice/decklink_enc.cpp @@ -38,16 +38,20 @@ extern "C" { #include "decklink_common.h" #include "decklink_enc.h" - +#if CONFIG_LIBKLVANC +#include "libklvanc/vanc.h" +#include "libklvanc/vanc-lines.h" +#include "libklvanc/pixels.h" +#endif /* DeckLink callback class declaration */ class decklink_frame : public IDeckLinkVideoFrame { public: decklink_frame(struct decklink_ctx *ctx, AVFrame *avframe, AVCodecID codec_id, int height, int width) : - _ctx(ctx), _avframe(avframe), _avpacket(NULL), _codec_id(codec_id), _height(height), _width(width), _refs(1) { } + _ctx(ctx), _avframe(avframe), _avpacket(NULL), _codec_id(codec_id), _ancillary(NULL), _height(height), _width(width), _refs(1) { } decklink_frame(struct decklink_ctx *ctx, AVPacket *avpacket, AVCodecID codec_id, int height, int width) : - _ctx(ctx), _avframe(NULL), _avpacket(avpacket), _codec_id(codec_id), _height(height), _width(width), _refs(1) { } + _ctx(ctx), _avframe(NULL), _avpacket(avpacket), _codec_id(codec_id), _ancillary(NULL), _height(height), _width(width), _refs(1) { } virtual long STDMETHODCALLTYPE GetWidth (void) { return _width; } virtual long STDMETHODCALLTYPE GetHeight (void) { return _height; } @@ -87,8 +91,13 @@ public: } virtual HRESULT STDMETHODCALLTYPE GetTimecode (BMDTimecodeFormat format, IDeckLinkTimecode **timecode) { return S_FALSE; } - virtual HRESULT STDMETHODCALLTYPE GetAncillaryData(IDeckLinkVideoFrameAncillary **ancillary) { return S_FALSE; } - + virtual HRESULT STDMETHODCALLTYPE GetAncillaryData(IDeckLinkVideoFrameAncillary **ancillary) + { + *ancillary = _ancillary; + return _ancillary ? S_OK : S_FALSE; + } + virtual HRESULT STDMETHODCALLTYPE SetAncillaryData(IDeckLinkVideoFrameAncillary + *ancillary) { _ancillary = ancillary; return S_OK; } virtual HRESULT STDMETHODCALLTYPE QueryInterface(REFIID iid, LPVOID *ppv) { return E_NOINTERFACE; } virtual ULONG STDMETHODCALLTYPE AddRef(void) { return ++_refs; } virtual ULONG STDMETHODCALLTYPE Release(void) @@ -106,6 +115,7 @@ public: AVFrame *_avframe; AVPacket *_avpacket; AVCodecID _codec_id; + IDeckLinkVideoFrameAncillary *_ancillary; int _height; int _width; @@ -169,7 +179,7 @@ static int decklink_setup_video(AVFormatContext *avctx, AVStream *st) return -1; } if (ctx->dlo->EnableVideoOutput(ctx->bmd_mode, - bmdVideoOutputFlagDefault) != S_OK) { + bmdVideoOutputVANC) != S_OK) { av_log(avctx, AV_LOG_ERROR, "Could not enable video output!\n"); return -1; } @@ -265,6 +275,93 @@ av_cold int ff_decklink_write_trailer(AVFormatContext *avctx) return 0; } +#if CONFIG_LIBKLVANC +static int decklink_construct_vanc(struct decklink_ctx *ctx, AVPacket *pkt, + decklink_frame *frame) +{ + struct vanc_line_set_s vanc_lines; + memset(&vanc_lines, 0, sizeof(vanc_lines)); + + int size; + const uint8_t *data = av_packet_get_side_data(pkt, AV_PKT_DATA_A53_CC, &size); + if (data) { + struct packet_eia_708b_s *pkt; + uint16_t *cdp; + uint16_t len; + uint8_t cc_count = size / 3; + + klvanc_create_eia708_cdp(&pkt); + klvanc_set_framerate_EIA_708B(pkt, ctx->bmd_tb_num, ctx->bmd_tb_den); + + /* CC data */ + pkt->header.ccdata_present = 1; + pkt->ccdata.cc_count = cc_count; + for (size_t i = 0; i < cc_count; i++) { + if (data [3*i] & 0x40) + pkt->ccdata.cc[i].cc_valid = 1; + pkt->ccdata.cc[i].cc_type = data[3*i] & 0x03; + pkt->ccdata.cc[i].cc_data[0] = data[3*i+1]; + pkt->ccdata.cc[i].cc_data[1] = data[3*i+2]; + } + + klvanc_finalize_EIA_708B(pkt, ctx->cdp_sequence_num++); + convert_EIA_708B_to_words(pkt, &cdp, &len); + klvanc_destroy_eia708_cdp(pkt); + + vanc_line_insert(&vanc_lines, cdp, len, 11, 0); + } + + IDeckLinkVideoFrameAncillary *vanc; + int result = ctx->dlo->CreateAncillaryData(bmdFormat10BitYUV, &vanc); + if (result != S_OK) { + fprintf(stderr, "Failed to create vanc\n"); + return -1; + } + + /* Now that we've got all the VANC lines in a nice orderly manner, generate the + final VANC sections for the Decklink output */ + for (int i = 0; i < vanc_lines.num_lines; i++) { + struct vanc_line_s *line = vanc_lines.lines[i]; + uint16_t *out_line; + int real_line; + int out_len; + void *buf; + + if (line == NULL) + break; + + real_line = line->line_number; +#if 0 + /* FIXME: include hack for certain Decklink cards which mis-represent + line numbers for pSF frames */ + if (decklink_sys->b_psf_interlaced) + real_line = Calculate1080psfVancLine(line->line_number); +#endif + result = vanc->GetBufferForVerticalBlankingLine(real_line, &buf); + if (result != S_OK) { + fprintf(stderr, "Failed to get VANC line %d: %d", real_line, result); + vanc_line_free(line); + continue; + } + + /* Generate the full line taking into account all VANC packets on that line */ + generate_vanc_line(line, &out_line, &out_len, ctx->bmd_width); + + /* Repack the 16-bit ints into 10-bit, and push into final buffer */ + klvanc_y10_to_v210(out_line, (uint8_t *) buf, out_len); + free(out_line); + vanc_line_free(line); + } + + result = frame->SetAncillaryData(vanc); + if (result != S_OK) { + fprintf(stderr, "Failed to set vanc: %d", result); + return AVERROR(EIO); + } + return 0; +} +#endif + static int decklink_write_video_packet(AVFormatContext *avctx, AVPacket *pkt) { struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data; @@ -299,6 +396,10 @@ static int decklink_write_video_packet(AVFormatContext *avctx, AVPacket *pkt) } frame = new decklink_frame(ctx, avpacket, st->codecpar->codec_id, ctx->bmd_height, ctx->bmd_width); + +#if CONFIG_LIBKLVANC + decklink_construct_vanc(ctx, pkt, frame); +#endif } if (!frame) {