1. 目标
到目前为止,所有的GStreamer都是靠video sink来创建一个窗口并显示视频内容的(除了Basic教程里面的第5讲)。但在iOS系统里面video sink不能创建自己的窗口,所以必须提供一个绘制层。本教程会讲述如何在Xcode的IB工具里面创建一个绘制层并传给GStreamer。
2. 介绍
由于iOS没有提供窗口系统,GStreamer的video sink不能象桌面平台一样创建弹出窗口。好在XOverlay接口可以给video sink提供已经建好的窗口,video sink可以在这个传入的窗口内绘制。
本教程内一个UIView(或者它的子类)会放在main storyboard里面,在ViewController的viewDidLoad方法里面,我们会把UIView的指针传递给GStreamerBackend的实例,这样GStreamer就知道往哪里绘制了。
3. UI
我们把上个教程的storyboard扩展一下:在工具条和消息栏的上面增加一个UIView,这个UIView占据所有的空闲的区域(video_container_view),在这个UIView的里面,还有增加一个UIView(video_view),这个增加的UIView才是真正显示视频的地方,它的区域大小会根据视频的尺寸变化。
VideoController.h
#import <UIKit/UIKit.h> #import "GStreamerBackendDelegate.h" @interface ViewController : UIViewController <GStreamerBackendDelegate> { IBOutlet UILabel *message_label; IBOutlet UIBarButtonItem *play_button; IBOutlet UIBarButtonItem *pause_button; IBOutlet UIView *video_view; IBOutlet UIView *video_container_view; IBOutlet NSLayoutConstraint *video_width_constraint; IBOutlet NSLayoutConstraint *video_height_constraint; } -(IBAction) play:(id)sender; -(IBAction) pause:(id)sender; /* From GStreamerBackendDelegate */ -(void) gstreamerInitialized; -(void) gstreamerSetUIMessage:(NSString *)message; @end4. ViewControllerViewController类会管理UI,GStreamerBackend也会执行一些UI相关的任务。
ViewController.m
#import "ViewController.h" #import "GStreamerBackend.h" #import <UIKit/UIKit.h> @interface ViewController () { GStreamerBackend *gst_backend; int media_width; int media_height; } @end @implementation ViewController /* * Methods from UIViewController */ - (void)viewDidLoad { [super viewDidLoad]; play_button.enabled = FALSE; pause_button.enabled = FALSE; /* Make these constant for now, later tutorials will change them */ media_width = 320; media_height = 240; gst_backend = [[GStreamerBackend alloc] init:self videoView:video_view]; } - (void)didReceiveMemoryWarning { [super didReceiveMemoryWarning]; // Dispose of any resources that can be recreated. } /* Called when the Play button is pressed */ -(IBAction) play:(id)sender { [gst_backend play]; } /* Called when the Pause button is pressed */ -(IBAction) pause:(id)sender { [gst_backend pause]; } - (void)viewDidLayoutSubviews { CGFloat view_width = video_container_view.bounds.size.width; CGFloat view_height = video_container_view.bounds.size.height; CGFloat correct_height = view_width * media_height / media_width; CGFloat correct_width = view_height * media_width / media_height; if (correct_height < view_height) { video_height_constraint.constant = correct_height; video_width_constraint.constant = view_width; } else { video_width_constraint.constant = correct_width; video_height_constraint.constant = view_height; } } /* * Methods from GstreamerBackendDelegate */ -(void) gstreamerInitialized { dispatch_async(dispatch_get_main_queue(), ^{ play_button.enabled = TRUE; pause_button.enabled = TRUE; message_label.text = @"Ready"; }); } -(void) gstreamerSetUIMessage:(NSString *)message { dispatch_async(dispatch_get_main_queue(), ^{ message_label.text = message; }); } @end这里我们扩充了一下ViewController类,让它记录下当前播放的视频的尺寸。@interface ViewController () { GStreamerBackend *gst_backend; int media_width; int media_height; }在后面的教程中,这个数据可以从GStreamer的pipeline中获得。为了简单起见,在本教程中,这个数据是常数并在viewDidLoad中置值了。
- (void)viewDidLoad { [super viewDidLoad]; play_button.enabled = FALSE; pause_button.enabled = FALSE; /* Make these constant for now, later tutorials will change them */ media_width = 320; media_height = 240; gst_backend = [[GStreamerBackend alloc] init:self videoView:video_view]; }GStreamerBackend的构造函数同样也扩充了——增加了UIView*的参数,video sink可以往这个地方绘制。ViewController里面除了适配视频尺寸的代码外其他的代码和上一个教程一模一样。
- (void)viewDidLayoutSubviews { CGFloat view_width = video_container_view.bounds.size.width; CGFloat view_height = video_container_view.bounds.size.height; CGFloat correct_height = view_width * media_height / media_width; CGFloat correct_width = view_height * media_width / media_height; if (correct_height < view_height) { video_height_constraint.constant = correct_height; video_width_constraint.constant = view_width; } else { video_width_constraint.constant = correct_width; video_height_constraint.constant = view_height; } }每次屏幕尺寸变化的时候(比如旋转方向)viewDidLayoutSubviews都会被调用,整个的布局也会重新计算。我们可以获得video_container_view的bounds参数,然后计算出新的尺寸并修改video_view的尺寸。
当视频尺寸超过video_view的宽/高的最大值时,我们会保留当前的宽高比,修改另一坐标方向上的值。因为目的仅仅是提供给GStreamer的video sink一个绘制层,所以并不没有增加黑边。
最后的尺寸会给到系统的布局引擎,修改video_view的宽高的约束条件里的constant项。这些约束条件是在storyboard里面建立的,和其他的widget没啥不同。
5. GStreamer Backend
GStreamerBackend实现了所有的和GStreamer相关的内容并提供了一个和应用交互的简单接口。当需要执行一些UI动作的时候,会通过GStreamerBackendDelegate的协议来完成。
GStreamerBackend.m
#import "GStreamerBackend.h" #include <gst/gst.h> #include <gst/interfaces/xoverlay.h> GST_DEBUG_CATEGORY_STATIC (debug_category); #define GST_CAT_DEFAULT debug_category @interface GStreamerBackend() -(void)setUIMessage:(gchar*) message; -(void)app_function; -(void)check_initialization_complete; @end @implementation GStreamerBackend { id ui_delegate; /* Class that we use to interact with the user interface */ GstElement *pipeline; /* The running pipeline */ GstElement *video_sink;/* The video sink element which receives XOverlay commands */ GMainContext *context; /* GLib context used to run the main loop */ GMainLoop *main_loop; /* GLib main loop */ gboolean initialized; /* To avoid informing the UI multiple times about the initialization */ UIView *ui_video_view; /* UIView that holds the video */ } /* * Interface methods */ -(id) init:(id) uiDelegate videoView:(UIView *)video_view { if (self = [super init]) { self->ui_delegate = uiDelegate; self->ui_video_view = video_view; GST_DEBUG_CATEGORY_INIT (debug_category, "tutorial-3", 0, "iOS tutorial 3"); gst_debug_set_threshold_for_name("tutorial-3", GST_LEVEL_DEBUG); /* Start the bus monitoring task */ dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{ [self app_function]; }); } return self; } -(void) dealloc { if (pipeline) { GST_DEBUG("Setting the pipeline to NULL"); gst_element_set_state(pipeline, GST_STATE_NULL); gst_object_unref(pipeline); pipeline = NULL; } } -(void) play { if(gst_element_set_state(pipeline, GST_STATE_PLAYING) == GST_STATE_CHANGE_FAILURE) { [self setUIMessage:"Failed to set pipeline to playing"]; } } -(void) pause { if(gst_element_set_state(pipeline, GST_STATE_PAUSED) == GST_STATE_CHANGE_FAILURE) { [self setUIMessage:"Failed to set pipeline to paused"]; } } /* * Private methods */ /* Change the message on the UI through the UI delegate */ -(void)setUIMessage:(gchar*) message { NSString *string = [NSString stringWithUTF8String:message]; if(ui_delegate && [ui_delegate respondsToSelector:@selector(gstreamerSetUIMessage:)]) { [ui_delegate gstreamerSetUIMessage:string]; } } /* Retrieve errors from the bus and show them on the UI */ static void error_cb (GstBus *bus, GstMessage *msg, GStreamerBackend *self) { GError *err; gchar *debug_info; gchar *message_string; gst_message_parse_error (msg, &err, &debug_info); message_string = g_strdup_printf ("Error received from element %s: %s", GST_OBJECT_NAME (msg->src), err->message); g_clear_error (&err); g_free (debug_info); [self setUIMessage:message_string]; g_free (message_string); gst_element_set_state (self->pipeline, GST_STATE_NULL); } /* Notify UI about pipeline state changes */ static void state_changed_cb (GstBus *bus, GstMessage *msg, GStreamerBackend *self) { GstState old_state, new_state, pending_state; gst_message_parse_state_changed (msg, &old_state, &new_state, &pending_state); /* Only pay attention to messages coming from the pipeline, not its children */ if (GST_MESSAGE_SRC (msg) == GST_OBJECT (self->pipeline)) { gchar *message = g_strdup_printf("State changed to %s", gst_element_state_get_name(new_state)); [self setUIMessage:message]; g_free (message); } } /* Check if all conditions are met to report GStreamer as initialized. * These conditions will change depending on the application */ -(void) check_initialization_complete { if (!initialized && main_loop) { GST_DEBUG ("Initialization complete, notifying application."); if (ui_delegate && [ui_delegate respondsToSelector:@selector(gstreamerInitialized)]) { [ui_delegate gstreamerInitialized]; } initialized = TRUE; } } /* Main method for the bus monitoring code */ -(void) app_function { GstBus *bus; GSource *bus_source; GError *error = NULL; GST_DEBUG ("Creating pipeline"); /* Create our own GLib Main Context and make it the default one */ context = g_main_context_new (); g_main_context_push_thread_default(context); /* Build pipeline */ pipeline = gst_parse_launch("videotestsrc ! warptv ! ffmpegcolorspace ! autovideosink", &error); if (error) { gchar *message = g_strdup_printf("Unable to build pipeline: %s", error->message); g_clear_error (&error); [self setUIMessage:message]; g_free (message); return; } /* Set the pipeline to READY, so it can already accept a window handle */ gst_element_set_state(pipeline, GST_STATE_READY); video_sink = gst_bin_get_by_interface(GST_BIN(pipeline), GST_TYPE_X_OVERLAY); if (!video_sink) { GST_ERROR ("Could not retrieve video sink"); return; } gst_x_overlay_set_window_handle(GST_X_OVERLAY(video_sink), (guintptr) (id) ui_video_view); /* Instruct the bus to emit signals for each received message, and connect to the interesting signals */ bus = gst_element_get_bus (pipeline); bus_source = gst_bus_create_watch (bus); g_source_set_callback (bus_source, (GSourceFunc) gst_bus_async_signal_func, NULL, NULL); g_source_attach (bus_source, context); g_source_unref (bus_source); g_signal_connect (G_OBJECT (bus), "message::error", (GCallback)error_cb, (__bridge void *)self); g_signal_connect (G_OBJECT (bus), "message::state-changed", (GCallback)state_changed_cb, (__bridge void *)self); gst_object_unref (bus); /* Create a GLib Main Loop and set it to run */ GST_DEBUG ("Entering main loop..."); main_loop = g_main_loop_new (context, FALSE); [self check_initialization_complete]; g_main_loop_run (main_loop); GST_DEBUG ("Exited main loop"); g_main_loop_unref (main_loop); main_loop = NULL; /* Free resources */ g_main_context_pop_thread_default(context); g_main_context_unref (context); gst_element_set_state (pipeline, GST_STATE_NULL); gst_object_unref (pipeline); return; } @end和上篇教程主要的不同在于XOverlay接口的处理。
@implementation GStreamerBackend { id ui_delegate; /* Class that we use to interact with the user interface */ GstElement *pipeline; /* The running pipeline */ GstElement *video_sink;/* The video sink element which receives XOverlay commands */ GMainContext *context; /* GLib context used to run the main loop */ GMainLoop *main_loop; /* GLib main loop */ gboolean initialized; /* To avoid informing the UI multiple times about the initialization */ UIView *ui_video_view; /* UIView that holds the video */ }这个类也增加了video sink element参数和显示需要的UIView*的参数。-(id) init:(id) uiDelegate videoView:(UIView *)video_view { if (self = [super init]) { self->ui_delegate = uiDelegate; self->ui_video_view = video_view; GST_DEBUG_CATEGORY_INIT (debug_category, "tutorial-3", 0, "iOS tutorial 3"); gst_debug_set_threshold_for_name("tutorial-3", GST_LEVEL_DEBUG); /* Start the bus monitoring task */ dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{ [self app_function]; }); } return self; }构造函数接受UIView*做为参数,这里简单的把它存储到了ui_video_view里面。/* Build pipeline */ pipeline = gst_parse_launch("videotestsrc ! warptv ! ffmpegcolorspace ! autovideosink", &error);接下来,在app_function中创建pipeline。这次在pipeline中会包含videotestsrc element,然后用warptv进行一次封装,video sink使用autovideosink,这个sink会选择合适的sink来使用(eglglesssink是iOS)/* Set the pipeline to READY, so it can already accept a window handle */ gst_element_set_state(pipeline, GST_STATE_READY); video_sink = gst_bin_get_by_interface(GST_BIN(pipeline), GST_TYPE_X_OVERLAY); if (!video_sink) { GST_ERROR ("Could not retrieve video sink"); return; } gst_x_overlay_set_window_handle(GST_X_OVERLAY(video_sink), (guintptr) (id) ui_video_view);pipeline一旦建立,我们就把它置成READY状态。在这个时候,数据流还没有开始流动,但所有的cap都已经确认是兼容的并且互相的pad已经连接了。而且autovideosink已经实例化了真正的video sink。gst_bin_get_by_interface()方法会检查整个pipeline并返回一个有所需的接口的element的指针。我们在这个例子中请求的是XOverlay接口(Basic教程5中有描述),这个控件会实现在非GStreamer系统的窗口绘制。本例中,在pipeline里只有实例化的video sink(由autovideosink生成)符合条件,所以是返回这个sink。
我们一旦有了video sink,我们就能通过gst_x_overlay_set_window_handle()方法在UIView上进行绘制。
6. EaglUIView
最后一个细节了。为了能让eglglessink能在UIView上绘制,这个类的Layer必须是CAEAGLLayer类。所以我们创建EaglUIView类继承在UIView并实现了layerClass方法。
EaglUIView.m
#import "EaglUIVIew.h" #import <QuartzCore/QuartzCore.h> @implementation EaglUIView + (Class) layerClass { return [CAEAGLLayer class]; } @end
在创建storyboard时,需要牢牢记住包含视频的UIView必须是EaglUIView类,这个在Xcode中的IB里面实现起来一点也不难。然后。。。就没有然后了,在iOS应用里让GStreamer输出视频就这么简单。
7.结论
本教程讲述了在iOS上如何使用UIView和XOverlay接口来显示视频以及如何根据多媒体宽度/高度来设置布局。
下一个教程会增加一些控件来实现一个简单的播放器,然后正式播放一个多媒体片段。