Video Manipulation in iOS : Resizing,Merging and Overlapping Videos in iOS

Video Manipulation in iOS : Resizing,Merging and Overlapping Videos in iOS

April 2, 201215 Comments

We all have seen video merging applications on Appstore where you can add multiple videos one after one other or inserting videos between other videos.Like the Video Editor App on Appstore.

But what if we want to resize the Video and Merge Multiple videos so that they overlap Each other and both Videos are playing at the same time.Much like the Google Video and Skype Video where you can see yourself in a small video screen overlapping the video of the other participant.

Surprisingly there is very little support on how to implement this and so far i have not found a decent tutorial on how to achieve this,Thus writing this tutorial for any one else who wants to add same features to his application.

It might look hard but actually it is a very simple feature.You just have to dig a little deeper.Apple has provided a very clean and efficient way to manipulate resize and overlap your Videos and thiese features where also explained in WWDC2010.So by digging a little deeper you have to watch the WWDC2010 “Editing Media with AV Foundation” videos and start from there.Lucky for you you will find everything you need write here in this tutorial.

So lets get started.

Our Aim : We have 2 input .mp4 video files and we want to merge them into one single video such that one video overlaps the other and are playign simultaneously.

So here is how our final Video will look like.

See how the smaller video is overlapping the bigger video and YES both are playing simultaneously as well.

How to Do this:

AvFoundation is really power full framework if you want to work and edit media in iOS.

Starting from iOS4.0 we can use AVComposition object combines media data from multiple file-based sources in a custom temporal arrangement, in order to present or process media data from multiple sources together. All file-based audiovisual assets are eligible to be combined, regardless of container type. The tracks in anAVComposition object are fixed; to change the tracks, you use an instance of its subclass, AVMutableComposition.

At its top-level, AVComposition is a collection of tracks, each presenting media of a specific media type, e.g. audio or video, according to a timeline. Each track is represented by an instance of AVCompositionTrack.
A higher-level interface for constructing compositions is also presented by AVMutableComposition and AVMutableCompositionTrack, offering insertion, removal, and scaling operations without direct manipulation of the trackSegment arrays of composition tracks. This interface makes use of higher-level constructs such as AVAsset and AVAssetTrack, allowing the client to make use of the same references to candidate sources that it would have created in order to inspect or preview them prior to inclusion in a composition.

So in short u have a AVComposition and you can add multiple AVMutableCompositionTrack to it. Each AVMutableCompositionTrack will have a separate video Asset.

Ok enough with the theory lets get started with some code.

//Here where load our movie Assets using AVURLAsset

AVURLAsset* firstAsset = [AVURLAsset URLAssetWithURL:[NSURL fileURLWithPath:[[NSBundle mainBundle] pathForResource: @”gizmo” ofType: @”mp4″]] options:nil];

AVURLAsset * secondAsset = [AVURLAsset URLAssetWithURL:[NSURL fileURLWithPath:[[NSBundle mainBundle] pathForResource: @”gizmo” ofType: @”mp4″]] options:nil];

 

//Create AVMutableComposition Object.This object will hold our multiple AVMutableCompositionTrack.

AVMutableComposition* mixComposition = [[AVMutableComposition alloc] init];

 

//Here we are creating the first AVMutableCompositionTrack.See how we are adding a new track to our AVMutableComposition.

AVMutableCompositionTrack *firstTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];

//Now we set the length of the firstTrack equal to the length of the firstAsset and add the firstAsset to out newly created track at kCMTimeZero so video plays from the start of the track.

[firstTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, firstAsset.duration) ofTrack:[[firstAsset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0] atTime:kCMTimeZero error:nil];

 

//Now we repeat the same process for the 2nd track as we did above for the first track.Note that the new track also starts at kCMTimeZero meaning both tracks will play simultaneously.

AVMutableCompositionTrack *secondTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];

[secondTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, secondAsset.duration) ofTrack:[[secondAsset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0] atTime:kCMTimeZero error:nil];

As you can see that we have added our 2 AVMutableCompositionTrack to our AVMutableComposition,but this is not the end.If you play the newly created composition at this stage you will only see the video of the first Asset.Thsi is because we have not added any instructions to our AVMutableCompositionTrack.Instructions such as video size position and transition effects.

To add these instructions AVFoundations provides us with AVVideoComposition Class.

AVVideoComposition contains an array of AVVideoCompositionInstructions .AVVideoCompositionInstructions object represents an operation to be performed by a compositor and AVVideoComposition object maintains an array of instructions to perform its composition.AVVideoCompositionInstructions itself contains an Array of AVVideoCompositionLayerInstruction defined for each layer.

■ Each AVVideoCompositionInstructions instruction describes the output video in terms of input layers (AVVideoCompositionLayerInstruction)

■ Each AVVideoCompositionLayerInstruction layer has an opacity and an affine transform and we will be using this affine transform to scale and position our videos.

Ok so i know this is a lot to digest so lets get into the code and see how it actually works.

//See how we are creating AVMutableVideoCompositionInstruction object.This object will contain the array of our AVMutableVideoCompositionLayerInstruction objects.You set the duration of the layer.You should add the lenght equal to the lingth of the longer asset in terms of duration.

AVMutableVideoCompositionInstruction * MainInstruction = [AVMutableVideoCompositionInstruction videoCompositionInstruction];

[MainInstruction retain];

MainInstruction.timeRange = CMTimeRangeMake(kCMTimeZero, firstAsset.duration);

 

//We will be creating 2 AVMutableVideoCompositionLayerInstruction objects.Each for our 2 AVMutableCompositionTrack.here we are creating AVMutableVideoCompositionLayerInstruction for out first track.see how we make use of Affinetransform to move and scale our First Track.so it is displayed at the bottom of the screen in smaller size.(First track in the one that remains on top).

AVMutableVideoCompositionLayerInstruction *FirstlayerInstruction = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:firstTrack];

CGAffineTransform Scale = CGAffineTransformMakeScale(0.7f,0.7f);

CGAffineTransform Move = CGAffineTransformMakeTranslation(230,230);

[FirstlayerInstruction setTransform:CGAffineTransformConcat(Scale,Move) atTime:kCMTimeZero];

 

//Here we are creating AVMutableVideoCompositionLayerInstruction for out second track.see how we make use of Affinetransform to move and scale our second Track.

AVMutableVideoCompositionLayerInstruction *SecondlayerInstruction = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:secondTrack];

CGAffineTransform SecondScale = CGAffineTransformMakeScale(1.2f,1.5f);

CGAffineTransform SecondMove = CGAffineTransformMakeTranslation(0,0);

[SecondlayerInstruction setTransform:CGAffineTransformConcat(SecondScale,SecondMove) atTime:kCMTimeZero];

 

//Now we add our 2 created AVMutableVideoCompositionLayerInstruction objects to our AVMutableVideoCompositionInstruction in form of an array.

MainInstruction.layerInstructions = [NSArray arrayWithObjects:FirstlayerInstruction,SecondlayerInstruction,nil];;

 

//Now we create AVMutableVideoComposition object.We can add mutiple AVMutableVideoCompositionInstruction to this object.We have only one AVMutableVideoCompositionInstruction object in our example.You can use multiple AVMutableVideoCompositionInstruction objects to add multiple layers of effects such as fade and transition but make sure that time ranges of the AVMutableVideoCompositionInstruction objects dont overlap.

AVMutableVideoComposition *MainCompositionInst = [AVMutableVideoComposition videoComposition];

[MainCompositionInst retain];

MainCompositionInst.instructions = [NSArray arrayWithObject:MainInstruction];

MainCompositionInst.frameDuration = CMTimeMake(1, 30);

MainCompositionInst.renderSize = CGSizeMake(640, 480);

 

//Finally just add the newly created AVMutableComposition with multiple tracks to an AVPlayerItem and play it using AVPlayer.

AVPlayerItem * newPlayerItem = [AVPlayerItem playerItemWithAsset:mixComposition];

[newPlayerItem retain];

newPlayerItem.videoComposition = MainCompositionInst;

self.mPlayer = [AVPlayer playerWithPlayerItem:newPlayerItem];

[mPlayer addObserver:self forKeyPath:@"status" options:0 context:AVPlayerDemoPlaybackViewControllerStatusObservationContext];

And thats it.Run the code and you should see 2 videos running side by side overlapping each other.We can multiple videos and resize and rotate and transform them in any way we like thus adding extremely cool effects.

These concepts are hard to understand at first but once you get hold of the concept its real fun to play around with this.

For more details i would recommend see the “Editing Media with AV Foundation” presentation at the WWDC2010.It also come with AVEditDemo which guides you how to add cool transition and fade effects in merged video.

Full code for the above example is available here for download.

If you want any more help just send me an email or leave a comments.

You can follow me at twitter @Abdul_Azeem

 

About these ads