[FFmpeg-devel,RFC] libavdevice/decklink: Add support for EIA-708 output over SDI

Submitted by Devin Heitmueller on Oct. 6, 2017, 4:56 p.m.

Details

Message ID 20171006165615.39580-1-dheitmueller@ltnglobal.com
State New
Headers show

Commit Message

Devin Heitmueller Oct. 6, 2017, 4:56 p.m.
From: Devin Heitmueller <dheitmueller@kernellabs.com>

Hook in libklvanc and use it for output of EIA-708 captions over
SDI.  The bulk of this patch is just general support for ancillary
data for the Decklink SDI module - the real work for construction
of the EIA-708 CDP and VANC line construction is done by libklvanc.

Libklvanc can be found at: https://github.com/stoth68000/libklvanc

Signed-off-by: Devin Heitmueller <dheitmueller@ltnglobal.com>
---
 configure                     |   3 ++
 libavcodec/v210enc.c          |   8 +++
 libavdevice/decklink_common.h |   1 +
 libavdevice/decklink_enc.cpp  | 113 +++++++++++++++++++++++++++++++++++++++---
 4 files changed, 119 insertions(+), 6 deletions(-)

Comments

Carl Eugen Hoyos Oct. 6, 2017, 9:07 p.m.
2017-10-06 18:56 GMT+02:00 Devin Heitmueller <dheitmueller@ltnglobal.com>:
> From: Devin Heitmueller <dheitmueller@kernellabs.com>
>
> Hook in libklvanc and use it for output of EIA-708 captions over
> SDI.  The bulk of this patch is just general support for ancillary
> data for the Decklink SDI module - the real work for construction
> of the EIA-708 CDP and VANC line construction is done by libklvanc.

Nothing except the decklink device could use VANC?

Carl Eugen
Devin Heitmueller Oct. 6, 2017, 9:21 p.m.
Hello Carl,

> On Oct 6, 2017, at 5:07 PM, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
> 
> 2017-10-06 18:56 GMT+02:00 Devin Heitmueller <dheitmueller@ltnglobal.com>:
>> From: Devin Heitmueller <dheitmueller@kernellabs.com>
>> 
>> Hook in libklvanc and use it for output of EIA-708 captions over
>> SDI.  The bulk of this patch is just general support for ancillary
>> data for the Decklink SDI module - the real work for construction
>> of the EIA-708 CDP and VANC line construction is done by libklvanc.
> 
> Nothing except the decklink device could use VANC?

You could absolutely have other SDI device types which can contain VANC.  This was a key reason that we put all the business logic for VANC processing in a separate library.

The goal behind developing libklvanc was to separate out VANC processing from device and application specific business logic.  This allows us to have a single parser and implementation of popular VANC protocols in a single library that can be reused by VLC, OBE, and ffmpeg.  It also allows the VANC processing to be shared across different device types, although admittedly there are not many vendors other than BlackMagic that are very popular.

The point of the remark in the commit message though was to observe that most of the code in the patch implements the decklink specific glue for accessing VANC lines.  The libklvanc library provides all the functions for parsing/generation of the VANC packets.

Devin
Carl Eugen Hoyos Oct. 6, 2017, 9:25 p.m.
2017-10-06 23:21 GMT+02:00 Devin Heitmueller <dheitmueller@ltnglobal.com>:
> Hello Carl,
>
>> On Oct 6, 2017, at 5:07 PM, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
>>
>> 2017-10-06 18:56 GMT+02:00 Devin Heitmueller <dheitmueller@ltnglobal.com>:
>>> From: Devin Heitmueller <dheitmueller@kernellabs.com>
>>>
>>> Hook in libklvanc and use it for output of EIA-708 captions over
>>> SDI.  The bulk of this patch is just general support for ancillary
>>> data for the Decklink SDI module - the real work for construction
>>> of the EIA-708 CDP and VANC line construction is done by libklvanc.
>>
>> Nothing except the decklink device could use VANC?
>
> You could absolutely have other SDI device types which can contain VANC.

Sorry, what I meant was:
Nothing inside FFmpeg except the decklink device could use
VANC?

Thank you, Carl Eugen
Devin Heitmueller Oct. 6, 2017, 9:31 p.m.
> 
> Sorry, what I meant was:
> Nothing inside FFmpeg except the decklink device could use
> VANC?

Ah, I understand now.

Yes, the decklink device is currently the only SDI device which is supported by libavdevice.  I’ve got a whole pile of patches coming which add support for a variety of protocols for both capture and output (e.g. EIA-708, SCTE-104, AFD, SMPTE 2038, etc).  But today the decklink module is the only device supported.

Would love to see more SDI devices supported and potentially interested in adding such support myself if we can find good candidates.  The DVEO/linsys cards are largely obsolete and the AJA boards are significantly more expensive than any of BlackMagic’s cards.  If anyone has good experiences with other vendors I would like to hear about it (and there may be an opportunity to extend libavdevice to support another SDI vendor).

Devin
Reuben Martin Oct. 7, 2017, 2:58 a.m.
On Fri, Oct 6, 2017 at 4:31 PM, Devin Heitmueller
<dheitmueller@ltnglobal.com> wrote:
>
> Ah, I understand now.
>
> Yes, the decklink device is currently the only SDI device which is supported by libavdevice.  I’ve got a whole pile of patches coming which add support for a variety of protocols for both capture and output (e.g. EIA-708, SCTE-104, AFD, SMPTE 2038, etc).  But today the decklink module is the only device supported.
>
> Would love to see more SDI devices supported and potentially interested in adding such support myself if we can find good candidates.  The DVEO/linsys cards are largely obsolete and the AJA boards are significantly more expensive than any of BlackMagic’s cards.  If anyone has good experiences with other vendors I would like to hear about it (and there may be an opportunity to extend libavdevice to support another SDI vendor).

Bluefish has some very nice ($$$) SDI cards with Linux support. I
think their SDK also supports v4l2. Have never been able to get my
hands on one though.

Patch hide | download patch | download mbox

diff --git a/configure b/configure
index 391c141e7a..18647896b1 100755
--- a/configure
+++ b/configure
@@ -238,6 +238,7 @@  External library support:
   --enable-libgsm          enable GSM de/encoding via libgsm [no]
   --enable-libiec61883     enable iec61883 via libiec61883 [no]
   --enable-libilbc         enable iLBC de/encoding via libilbc [no]
+  --enable-libklvanc       enable Kernel Labs VANC processing [no]
   --enable-libkvazaar      enable HEVC encoding via libkvazaar [no]
   --enable-libmodplug      enable ModPlug via libmodplug [no]
   --enable-libmp3lame      enable MP3 encoding via libmp3lame [no]
@@ -1603,6 +1604,7 @@  EXTERNAL_LIBRARY_LIST="
     libgsm
     libiec61883
     libilbc
+    libklvanc
     libkvazaar
     libmodplug
     libmp3lame
@@ -6027,6 +6029,7 @@  enabled libx264           && { use_pkg_config libx264 x264 "stdint.h x264.h" x26
 enabled libx265           && require_pkg_config libx265 x265 x265.h x265_api_get &&
                              require_cpp_condition x265.h "X265_BUILD >= 68"
 enabled libxavs           && require libxavs "stdint.h xavs.h" xavs_encoder_encode -lxavs
+enabled libklvanc         && require libklvanc libklvanc/vanc.h vanc_context_create -lklvanc
 enabled libxvid           && require libxvid xvid.h xvid_global -lxvidcore
 enabled libzimg           && require_pkg_config libzimg "zimg >= 2.3.0" zimg.h zimg_get_api_version
 enabled libzmq            && require_pkg_config libzmq libzmq zmq.h zmq_ctx_new
diff --git a/libavcodec/v210enc.c b/libavcodec/v210enc.c
index a6afbbfc41..44cc3c5c81 100644
--- a/libavcodec/v210enc.c
+++ b/libavcodec/v210enc.c
@@ -123,6 +123,7 @@  static int encode_frame(AVCodecContext *avctx, AVPacket *pkt,
     int aligned_width = ((avctx->width + 47) / 48) * 48;
     int stride = aligned_width * 8 / 3;
     int line_padding = stride - ((avctx->width * 8 + 11) / 12) * 4;
+    AVFrameSideData *side_data = NULL;
     int h, w, ret;
     uint8_t *dst;
 
@@ -233,6 +234,13 @@  static int encode_frame(AVCodecContext *avctx, AVPacket *pkt,
         }
     }
 
+    side_data = av_frame_get_side_data(pic, AV_FRAME_DATA_A53_CC);
+    if (side_data && side_data->size) {
+        uint8_t* buf = av_packet_new_side_data(pkt, AV_PKT_DATA_A53_CC, side_data->size);
+        if (buf)
+            memcpy(buf, side_data->data, side_data->size);
+    }
+
     pkt->flags |= AV_PKT_FLAG_KEY;
     *got_packet = 1;
     return 0;
diff --git a/libavdevice/decklink_common.h b/libavdevice/decklink_common.h
index 6b2525fb53..285a244000 100644
--- a/libavdevice/decklink_common.h
+++ b/libavdevice/decklink_common.h
@@ -78,6 +78,7 @@  struct decklink_ctx {
     AVStream *audio_st;
     AVStream *video_st;
     AVStream *teletext_st;
+    uint16_t cdp_sequence_num;
 
     /* Options */
     int list_devices;
diff --git a/libavdevice/decklink_enc.cpp b/libavdevice/decklink_enc.cpp
index 81df563b3b..3049e936a9 100644
--- a/libavdevice/decklink_enc.cpp
+++ b/libavdevice/decklink_enc.cpp
@@ -38,16 +38,20 @@  extern "C" {
 
 #include "decklink_common.h"
 #include "decklink_enc.h"
-
+#if CONFIG_LIBKLVANC
+#include "libklvanc/vanc.h"
+#include "libklvanc/vanc-lines.h"
+#include "libklvanc/pixels.h"
+#endif
 
 /* DeckLink callback class declaration */
 class decklink_frame : public IDeckLinkVideoFrame
 {
 public:
     decklink_frame(struct decklink_ctx *ctx, AVFrame *avframe, AVCodecID codec_id, int height, int width) :
-        _ctx(ctx), _avframe(avframe), _avpacket(NULL), _codec_id(codec_id), _height(height), _width(width),  _refs(1) { }
+        _ctx(ctx), _avframe(avframe), _avpacket(NULL), _codec_id(codec_id), _ancillary(NULL), _height(height), _width(width),  _refs(1) { }
     decklink_frame(struct decklink_ctx *ctx, AVPacket *avpacket, AVCodecID codec_id, int height, int width) :
-        _ctx(ctx), _avframe(NULL), _avpacket(avpacket), _codec_id(codec_id), _height(height), _width(width), _refs(1) { }
+        _ctx(ctx), _avframe(NULL), _avpacket(avpacket), _codec_id(codec_id), _ancillary(NULL), _height(height), _width(width), _refs(1) { }
 
     virtual long           STDMETHODCALLTYPE GetWidth      (void)          { return _width; }
     virtual long           STDMETHODCALLTYPE GetHeight     (void)          { return _height; }
@@ -87,8 +91,13 @@  public:
     }
 
     virtual HRESULT STDMETHODCALLTYPE GetTimecode     (BMDTimecodeFormat format, IDeckLinkTimecode **timecode) { return S_FALSE; }
-    virtual HRESULT STDMETHODCALLTYPE GetAncillaryData(IDeckLinkVideoFrameAncillary **ancillary)               { return S_FALSE; }
-
+    virtual HRESULT STDMETHODCALLTYPE GetAncillaryData(IDeckLinkVideoFrameAncillary **ancillary)
+    {
+        *ancillary = _ancillary;
+        return _ancillary ? S_OK : S_FALSE;
+    }
+    virtual HRESULT STDMETHODCALLTYPE SetAncillaryData(IDeckLinkVideoFrameAncillary
+                                                       *ancillary) { _ancillary = ancillary; return S_OK; }
     virtual HRESULT STDMETHODCALLTYPE QueryInterface(REFIID iid, LPVOID *ppv) { return E_NOINTERFACE; }
     virtual ULONG   STDMETHODCALLTYPE AddRef(void)                            { return ++_refs; }
     virtual ULONG   STDMETHODCALLTYPE Release(void)
@@ -106,6 +115,7 @@  public:
     AVFrame *_avframe;
     AVPacket *_avpacket;
     AVCodecID _codec_id;
+    IDeckLinkVideoFrameAncillary *_ancillary;
     int _height;
     int _width;
 
@@ -169,7 +179,7 @@  static int decklink_setup_video(AVFormatContext *avctx, AVStream *st)
         return -1;
     }
     if (ctx->dlo->EnableVideoOutput(ctx->bmd_mode,
-                                    bmdVideoOutputFlagDefault) != S_OK) {
+                                    bmdVideoOutputVANC) != S_OK) {
         av_log(avctx, AV_LOG_ERROR, "Could not enable video output!\n");
         return -1;
     }
@@ -265,6 +275,93 @@  av_cold int ff_decklink_write_trailer(AVFormatContext *avctx)
     return 0;
 }
 
+#if CONFIG_LIBKLVANC
+static int decklink_construct_vanc(struct decklink_ctx *ctx, AVPacket *pkt,
+                                   decklink_frame *frame)
+{
+    struct vanc_line_set_s vanc_lines;
+    memset(&vanc_lines, 0, sizeof(vanc_lines));
+
+    int size;
+    const uint8_t *data = av_packet_get_side_data(pkt, AV_PKT_DATA_A53_CC, &size);
+    if (data) {
+        struct packet_eia_708b_s *pkt;
+        uint16_t *cdp;
+        uint16_t len;
+        uint8_t cc_count = size / 3;
+
+        klvanc_create_eia708_cdp(&pkt);
+        klvanc_set_framerate_EIA_708B(pkt, ctx->bmd_tb_num, ctx->bmd_tb_den);
+
+        /* CC data */
+        pkt->header.ccdata_present = 1;
+        pkt->ccdata.cc_count = cc_count;
+        for (size_t i = 0; i < cc_count; i++) {
+            if (data [3*i] & 0x40)
+                pkt->ccdata.cc[i].cc_valid = 1;
+            pkt->ccdata.cc[i].cc_type = data[3*i] & 0x03;
+            pkt->ccdata.cc[i].cc_data[0] = data[3*i+1];
+            pkt->ccdata.cc[i].cc_data[1] = data[3*i+2];
+        }
+
+        klvanc_finalize_EIA_708B(pkt, ctx->cdp_sequence_num++);
+        convert_EIA_708B_to_words(pkt, &cdp, &len);
+        klvanc_destroy_eia708_cdp(pkt);
+
+        vanc_line_insert(&vanc_lines, cdp, len, 11, 0);
+    }
+
+    IDeckLinkVideoFrameAncillary *vanc;
+    int result = ctx->dlo->CreateAncillaryData(bmdFormat10BitYUV, &vanc);
+    if (result != S_OK) {
+        fprintf(stderr, "Failed to create vanc\n");
+        return -1;
+    }
+
+    /* Now that we've got all the VANC lines in a nice orderly manner, generate the
+       final VANC sections for the Decklink output */
+    for (int i = 0; i < vanc_lines.num_lines; i++) {
+        struct vanc_line_s *line = vanc_lines.lines[i];
+        uint16_t *out_line;
+        int real_line;
+        int out_len;
+        void *buf;
+
+        if (line == NULL)
+            break;
+
+        real_line = line->line_number;
+#if 0
+        /* FIXME: include hack for certain Decklink cards which mis-represent
+           line numbers for pSF frames */
+        if (decklink_sys->b_psf_interlaced)
+            real_line = Calculate1080psfVancLine(line->line_number);
+#endif
+        result = vanc->GetBufferForVerticalBlankingLine(real_line, &buf);
+        if (result != S_OK) {
+            fprintf(stderr, "Failed to get VANC line %d: %d", real_line, result);
+            vanc_line_free(line);
+            continue;
+        }
+
+        /* Generate the full line taking into account all VANC packets on that line */
+        generate_vanc_line(line, &out_line, &out_len, ctx->bmd_width);
+
+        /* Repack the 16-bit ints into 10-bit, and push into final buffer */
+        klvanc_y10_to_v210(out_line, (uint8_t *) buf, out_len);
+        free(out_line);
+        vanc_line_free(line);
+    }
+
+    result = frame->SetAncillaryData(vanc);
+    if (result != S_OK) {
+        fprintf(stderr, "Failed to set vanc: %d", result);
+        return AVERROR(EIO);
+    }
+    return 0;
+}
+#endif
+
 static int decklink_write_video_packet(AVFormatContext *avctx, AVPacket *pkt)
 {
     struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data;
@@ -299,6 +396,10 @@  static int decklink_write_video_packet(AVFormatContext *avctx, AVPacket *pkt)
         }
 
         frame = new decklink_frame(ctx, avpacket, st->codecpar->codec_id, ctx->bmd_height, ctx->bmd_width);
+
+#if CONFIG_LIBKLVANC
+        decklink_construct_vanc(ctx, pkt, frame);
+#endif
     }
 
     if (!frame) {