IJKPLAYER源码分析-iOS端显示-编程知识

1 简介

1.1 EAGL(Embedded Apple Graphics Library)

与Android系统使用EGL连接OpenGL ES与原生窗口进行surface输出类似，iOS则用EAGL将CAEAGLLayer作为OpenGL ES输出目标。

与 Android EGL 不同的是，iOS EAGL 不会让应用直接向 BackendFrameBuffer 和 FrontFrameBuffer 进行绘制，也不会让app直接控制双缓冲区的交换，系统自己保留了操作权，以便可以随时使用 Core Animation 合成器来控制显示的最终外观。

OpenGL ES 通过 CAEAGLLayer 与 Core Animation 连接，CAEAGLLayer 是一种特殊类型的 Core Animation 图层，它的内容来自 OpenGL ES 的 RenderBuffer，Core Animation 将 RenderBuffer 的内容与其他图层合成，并在屏幕上显示生成的图像。所以同一时刻可以有任意数量的层。Core Animation 合成器会联合这些层并在后帧缓存中产生最终的像素颜色，然后切换缓存。

一句话就是，OpenGL ES可与CAEAGLLayer共享RenderBuffer，用EAGLContext将CAEGLLayer与RenderBuffer进行绑定，可实现OpenGL ES最终输出到CAEGLLayer上并显示出

来。

2 显示

2.1 软解

FFmpeg软解后的输出是AVFrame，其像素数据在uint8_t* data[AV_NUM_DATA_POINTERS]所指向的内存中；
在软解码线程里解码后可以播放时，调用SDL_Vout中SDL_VoutOverlay中func_fill_frame方法，对AVFrame增加引用计数，将overlay的pixels[]也指向这块儿内存；
然后将AVFrame放到FrameQueue队列尾部，待video_refresh_thread线程消费render；
此后在video_refresh_thread线程里把AVFrame取出来，将pixels[]所指像素数据喂给OpenGL ES进行render；
循环以上步骤；

2.1.1 像素源

因此，FFmpeg软解后的像素数据来源了然了，来自AVFrame的uint8_t* data[AV_NUM_DATA_POINTERS]。

此处举例说明，不需要转换像素格式的填充：

static void overlay_fill(SDL_VoutOverlay *overlay, AVFrame *frame, int planes)
{overlay->planes = planes;for (int i = 0; i < AV_NUM_DATA_POINTERS; ++i) {overlay->pixels[i] = frame->data[i];overlay->pitches[i] = frame->linesize[i];}
}

需要转换像素格式的，调用libyuv或sws_scale处理，再入FrameQueue队列。

2.2 硬解

videotoolbox解码后输出是CVPixelBuffer，为了与FFmpeg统一接口，硬解后将CVPixelBuffer由AVFrame的opaque指向；
然后，在硬解码线程入队FrameQueue尾部，待video_refresh_thread线程消费render；
随后的逻辑基本与FFmpeg软解一致，在video_refresh_thread线程里把AVFrame取出来，将pixels[]所指向的像素喂给OpenGL ES进行render；
具体差异在yuv420sp_vtb_uploadTexture方法里；
循环以上步骤；

2.2.1 像素源

所以，videotoolbox硬解后的像素数据来源也清楚了，来自AVFrame的opaque，实际是CVPixelBuffer。

static int func_fill_frame(SDL_VoutOverlay *overlay, const AVFrame *frame)
{assert(frame->format == IJK_AV_PIX_FMT__VIDEO_TOOLBOX);CVBufferRef pixel_buffer = CVBufferRetain(frame->opaque);SDL_VoutOverlay_Opaque *opaque = overlay->opaque;if (opaque->pixel_buffer != NULL) {CVBufferRelease(opaque->pixel_buffer);}opaque->pixel_buffer = pixel_buffer;overlay->format = SDL_FCC__VTB;overlay->planes = 2;if (CVPixelBufferLockBaseAddress(pixel_buffer, 0) != kCVReturnSuccess) {overlay->pixels[0]  = NULL;overlay->pixels[1]  = NULL;overlay->pitches[0] = 0;overlay->pitches[1] = 0;overlay->w = 0;overlay->h = 0;CVBufferRelease(pixel_buffer);opaque->pixel_buffer = NULL;return -1;}overlay->pixels[0]  = NULL;overlay->pixels[1]  = NULL;overlay->pitches[0] = CVPixelBufferGetWidthOfPlane(pixel_buffer, 0);overlay->pitches[1] = CVPixelBufferGetWidthOfPlane(pixel_buffer, 1);CVPixelBufferUnlockBaseAddress(pixel_buffer, 0);overlay->is_private = 1;overlay->w = (int)frame->width;overlay->h = (int)frame->height;return 0;
}

2.3 render

首先让UIView的子类视频窗口的layerClass返回CAEAGLLayer；
设置CAEAGLLayer的属性，创建EAGLContext，并设置render的上下文；
创建FBO和RBO并绑定，再用所创建的EAGLContext调用renderbufferStorage，将GL_RENDERBUFFER与CAEAGLLayer进行绑定；
将RBO作为GL_COLOR_ATTACHMENT0附加到FBO上；
将EAGLContext上下文切为当前所创建的Context，并用OpenGL ES开始1帧的render工作；
然后，调用EAGLContext的presentRenderbuffer方法将RenderBuffer输出到CAEAGLLayer再显示；
最后，将把Context切为render之前的Context；
重复4~6步骤，继续render下一帧；

让UIView子类返回CAEAGLLayer class：

+ (Class) layerClass
{return [CAEAGLLayer class];
}

设置CAEAGLLayer属性为不透明，并创建EAGLContext，设置render的上下文：

- (BOOL)setupGL
{if (_didSetupGL)return YES;CAEAGLLayer *eaglLayer = (CAEAGLLayer*) self.layer;// 设置为不透明eaglLayer.opaque = YES;eaglLayer.drawableProperties = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithBool:NO], kEAGLDrawablePropertyRetainedBacking,kEAGLColorFormatRGBA8, kEAGLDrawablePropertyColorFormat,nil];_scaleFactor = [[UIScreen mainScreen] scale];if (_scaleFactor < 0.1f)_scaleFactor = 1.0f;[eaglLayer setContentsScale:_scaleFactor];_context = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2];if (_context == nil) {NSLog(@"failed to setup EAGLContext\n");return NO;}EAGLContext *prevContext = [EAGLContext currentContext];[EAGLContext setCurrentContext:_context];_didSetupGL = NO;if ([self setupEAGLContext:_context]) {NSLog(@"OK setup GL\n");_didSetupGL = YES;}[EAGLContext setCurrentContext:prevContext];return _didSetupGL;
}

创建FBO和RBO并绑定，并将RBO以GL_COLOR_ATTACHMENT0附加到FBO上：

- (BOOL)setupEAGLContext:(EAGLContext *)context
{glGenFramebuffers(1, &_framebuffer);glGenRenderbuffers(1, &_renderbuffer);glBindFramebuffer(GL_FRAMEBUFFER, _framebuffer);glBindRenderbuffer(GL_RENDERBUFFER, _renderbuffer);[_context renderbufferStorage:GL_RENDERBUFFER fromDrawable:(CAEAGLLayer*)self.layer];glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_WIDTH, &_backingWidth);glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_HEIGHT, &_backingHeight);glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, _renderbuffer);GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);if (status != GL_FRAMEBUFFER_COMPLETE) {NSLog(@"failed to make complete framebuffer object %x\n", status);return NO;}GLenum glError = glGetError();if (GL_NO_ERROR != glError) {NSLog(@"failed to setup GL %x\n", glError);return NO;}return YES;
}

iOS端显示的主函数，是在iOS端SDL_Vout接口display_overlay中回调的：

- (void)display: (SDL_VoutOverlay *) overlay
{if (_didSetupGL == NO)return;if ([self isApplicationActive] == NO)return;if (![self tryLockGLActive]) {if (0 == (_tryLockErrorCount % 100)) {NSLog(@"IJKSDLGLView:display: unable to tryLock GL active: %d\n", _tryLockErrorCount);}_tryLockErrorCount++;return;}_tryLockErrorCount = 0;if (_context && !_didStopGL) {EAGLContext *prevContext = [EAGLContext currentContext];[EAGLContext setCurrentContext:_context];[self displayInternal:overlay];[EAGLContext setCurrentContext:prevContext];}[self unlockGLActive];
}

具体执行render显示工作：

// NOTE: overlay could be NULl
- (void)displayInternal: (SDL_VoutOverlay *) overlay
{if (![self setupRenderer:overlay]) {if (!overlay && !_renderer) {NSLog(@"IJKSDLGLView: setupDisplay not ready\n");} else {NSLog(@"IJKSDLGLView: setupDisplay failed\n");}return;}[[self eaglLayer] setContentsScale:_scaleFactor];if (_isRenderBufferInvalidated) {NSLog(@"IJKSDLGLView: renderbufferStorage fromDrawable\n");_isRenderBufferInvalidated = NO;glBindRenderbuffer(GL_RENDERBUFFER, _renderbuffer);[_context renderbufferStorage:GL_RENDERBUFFER fromDrawable:(CAEAGLLayer*)self.layer];glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_WIDTH, &_backingWidth);glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_HEIGHT, &_backingHeight);IJK_GLES2_Renderer_setGravity(_renderer, _rendererGravity, _backingWidth, _backingHeight);}glBindFramebuffer(GL_FRAMEBUFFER, _framebuffer);glViewport(0, 0, _backingWidth, _backingHeight);if (!IJK_GLES2_Renderer_renderOverlay(_renderer, overlay))ALOGE("[EGL] IJK_GLES2_render failed\n");glBindRenderbuffer(GL_RENDERBUFFER, _renderbuffer);[_context presentRenderbuffer:GL_RENDERBUFFER];int64_t current = (int64_t)SDL_GetTickHR();int64_t delta   = (current > _lastFrameTime) ? current - _lastFrameTime : 0;if (delta <= 0) {_lastFrameTime = current;} else if (delta >= 1000) {_fps = ((CGFloat)_frameCount) * 1000 / delta;_frameCount = 0;_lastFrameTime = current;} else {_frameCount++;}
}

首次或overlay的format发生变更，则创建 IJK_GLES2_Renderer实例：

- (BOOL)setupRenderer: (SDL_VoutOverlay *) overlay
{if (overlay == nil)return _renderer != nil;if (!IJK_GLES2_Renderer_isValid(_renderer) ||!IJK_GLES2_Renderer_isFormat(_renderer, overlay->format)) {IJK_GLES2_Renderer_reset(_renderer);IJK_GLES2_Renderer_freeP(&_renderer);_renderer = IJK_GLES2_Renderer_create(overlay);if (!IJK_GLES2_Renderer_isValid(_renderer))return NO;if (!IJK_GLES2_Renderer_use(_renderer))return NO;IJK_GLES2_Renderer_setGravity(_renderer, _rendererGravity, _backingWidth, _backingHeight);}return YES;
}