In this gist, I will discuss how I used the Multipeer Connectivity framework to create Open Source Selfie Stick. Open Source Selfie Stick is a free open-source iOS app that allows users to sync two devices over WiFi or Bluetooth and allows one to act as a remote control for the other's camera.
This tutorial assumes some knowledge of the Swift programming language and iOS development with Xcode.
Feel free to comment and point out any errors or improvements. If you'd like to help improve the app itself, make a fork from dev branch of the git repo. I plan on updating this document and explaining any newly added features or refactoring. As this gist will be a tutorial on how to build the current app that is available in the App Store, some parts of this gist may be edited or removed entirely depending upon the future development of the app.
When the app is loaded, the user will see the following initial screen:
- a UIButton to indicate the device will be used as the camera
- a UIButton to indicate the device will be designated as the remote control
Once the user chooses whether the device will be the remote control or the camera, the device immediately starts looking for peers over WiFi and/or Bluetooth. While this search is initiated, the user is asked if they want photos from the upcoming session to be saved to the device. Photos can be saved to the camera, the remote control, or both.
After choosing if photos will be saved to the device, the camera device will display the following UI:
- a live preview of what the camera sees
- UIButton to enable/disable flash
- UIButton to change your photo saving preference
- a UIButton in the top center which allows the user to choose whether they want to save photos to the camera device.
- an advanced settings UIButton which brings up options to adjust ISO and color settings.
The device acting as the remote control will see the following UI:
- The big rectangular button that sends a command from the remote control device to the camera device, instructing it to take a photo.
- The timer button; tapping this will bring up an interface to allow the user to set a countdown timer. For example, if the user sets the timer to 3 seconds, after hitting the button to take a photo, the remote control device will wait 3 seconds before sending the command instructing the camera device to take a photo.
- UIButton to enable/disable the flash on the camera device.
- a UIButton in the top center which allows the user to choose whether they want to save photos to the remote control device.
That's it. There's not too much going on with the storyboard. When the app is loaded, they see the initial screen. If they choose for the device to be the camera, a segue is performed which brings up the camera scene. If the user chooses for the device to be the remote control, a segue is performed, bringing up the remote control scene. The UIButton sizes and placement were created using AutoLayout constraints, so the UI scales proportionally on larger devices such as an iPad. This is probably not the best use of screen real estate on iPads, and one of the goals on the To-Do list is to improve the UI overall.
Testing this app will require two iOS devices or one iOS device and a Mac.
If you have only one iOS device but also have a Mac, you can use the Simulator that comes with Xcode as one of the devices. Running the app on your iOS device and in a simulated device will work very similarly to using the app with two actual iOS devices. The one limitation is that the simulated device must act as the remote control. The Simulator is limited in that it does not have a functioning camera. Testing this software on two simulated devices will not work. You will need at least one real iOS device with a functioning camera to use this software in conjunction with the Simulator. That said, the Simulator will allow you to save photos to the simulated device, access photos from the simulated device's camera roll, and communicate with peers using the multipeer connectivity framework.
To allow realtime communication of data between two iOS devices, we need to open a peer-to-peer connection between them. Potentially, we could use some of the functionality in the GameKit framework to achieve this. Unfortunately, in doing so we don't have a great amount of control over connectivity and architecture. Using a WebSocket library is an option, but then we sacrifice Bluetooth connectivity and it doesn't come with built-in device advertising/discovery.
Luckily, starting with iOS 7, Apple provided developers with the Multipeer Connectivity framework which hits all the rights notes for what we need to do in this project.
Multipeer connectivity (abbreviated as "MPC" from here on) allows iOS devices to discover, advertise, and transmit data between other iOS devices over the course of a session (an MCSession, to be exact).
There are several different ways to start a session and find peers. Apple provides the MCBrowserViewController UIViewController to allow the user to view nearby iOS devices and invite them to a session. This is probably more useful for something like a game, where you may be connecting with multiple peers. For the purposes of Open Source Selfie Stick, we know we only need two peers (one camera device and one remote control device) to find each other, and therefore can create a more seamless experience using a dedicated advertiser/browser approach using the camera device as the MCNearbyServiceAdvertiser and the remote control device as the MCNearbyServiceBrowser. The process of finding and inviting peers to a session is known as the "discovery" phase.
To connect all these pieces together, our app uses the delegation design pattern. In Apple's own words, "Delegation is a simple and powerful pattern in which one object in a program acts on behalf of, or in coordination with, another object." You may have used this design pattern before if you've implemented a UITableView. When you create a new UITableView, for the UITableView to function properly, you will need to implement certain methods, such as the numberOfSections method, which dictates how many sections there will be, and the numberOfRowsInSection(_:) method to tell the UITableView instance how many rows there will be in each section of the table view. The methods needed are declared in the protocol of the UITableViewDelegate. If you do not include all the methods of a protocol (not including any optional methods) your app will crash when trying to load the view because the delegate does not "conform" to the protocol. Often, developers will use the ViewController containing the UITableView as the delegate which contains these methods. This is why you will often see something similar to tableView.delegate = self; in the viewDidLoad of a ViewController containing a UITableView. That line of code tells the UITableView object referenced as the variable tableView to look in this class for the methods it needs to operate.
In this project, we create our own NSObject called CameraServiceManager. This class will act as the delegate for the MCSession protocol. It will also implement the CameraServiceManagerDelegate protocol which will require certain methods from the CameraViewController (the UIViewController for the camera storyboard scene) and the CameraControllerViewController (the UIViewController for the remote control scene).
The CameraServiceManager class has a number of variables which conform the class to the MPC protocols (which include MCNearbyServiceAdvertiserDelegate, MCNearbyServiceBrowserDelegate, and MCSessionDelegate):
private let myPeerId = MCPeerID(displayName: UIDevice.currentDevice().name)
let serviceAdvertiser : MCNearbyServiceAdvertiser
let serviceBrowser : MCNearbyServiceBrowser
var delegate : CameraServiceManagerDelegate?
myPeerId is required by the MCNearbyServiceAdvertiser and represents a value with which we will identify peers and (later on in future development) use to display a confirmation UIAlertController asking if you want a device named myPeerId to connect with your device. Upon initialization of an instance of the CameraServiceManager class, we set the above variables as so:
override init() {
super.init()
self.serviceAdvertiser = MCNearbyServiceAdvertiser(peer: myPeerId, discoveryInfo: nil, serviceType: ServiceType)
self.serviceBrowser = MCNearbyServiceBrowser(peer: myPeerId, serviceType: ServiceType)
self.serviceAdvertiser.delegate = self
self.serviceBrowser.delegate = self
}
With this code, we are setting the MCNearbyServiceAdvertiser and MCNearbyServiceBrowser variables and indicating that this class will conform to their individual protocols. Note that even though both the camera device and remote control device will inherit the ability to advertise and browse for other peers, we won't have both devices doing both at the same time.
In both the CameraViewController and CameraControllerViewController classes, we create a new instance of CameraServiceManager by saying var cameraService = CameraServiceManager(). In the viewDidLoad() of both classes, we also say cameraService.delegate = self which tells the cameraService instance of the CameraServiceManager class that the methods necessary to conform to the CameraServiceManagerDelegate protocol are located within the class itself.
On the camera side, we have the following line of code in viewWillAppear(): self.cameraService.serviceAdvertiser.startAdvertisingPeer(), while on the remote control side, we have the following line in ViewWillAppear(): self.cameraService.serviceBrowser.startBrowsingForPeers(). This tells the camera device to start advertising an available session for other peers to join, while the remote control device is instructed to browse for available sessions.
If the remote control is browsing while the camera device is advertising, the two devices should find one another. This will invoke the appropriate method of the CameraServiceManager class extension which inherits from and conforms to the MCNearbyServiceBrowserDelegate and looks like this:
extension CameraServiceManager : MCNearbyServiceBrowserDelegate {
func browser(browser: MCNearbyServiceBrowser, didNotStartBrowsingForPeers error: NSError) {
NSLog("%@", "didNotStartBrowsingForPeers: \(error)")
}
func browser(browser: MCNearbyServiceBrowser, foundPeer peerID: MCPeerID, withDiscoveryInfo info: [String : String]?) {
NSLog("%@", "foundPeer: \(peerID)")
NSLog("%@", "invitePeer: \(peerID)")
browser.invitePeer(peerID, toSession: self.session, withContext: nil, timeout: 10)
}
func browser(browser: MCNearbyServiceBrowser, lostPeer peerID: MCPeerID) {
NSLog("%@", "lostPeer: \(peerID)")
}
}
These methods are called when certain MPC events occur, such as foundPeer and lostPeer. You can see some logs to the debug console where the device name is printed along with whether it was found, invited to a session, or lost (dropped from a session). In the current implementation, we short-circuit the handshake between the two devices and the advertiser (the camera) will automatically invite the browser (the remote control). This is far from optimal and will be addressed in the next app update; a malicious user could join an existing session and send or retrieve photos unwanted by the two original and legitimate users.
When the browsing device (the remote control) connects with the advertising device (the camera), the two devices will now share the same MCSession object and are now in the 'session' phase.
Now we have our camera and remote control connected to the same MCSession, we need for them to be able to send data to one another. Let's follow the example of sending a message from the remote control to the camera, instructing it to take a photo.
First, the button with which to send the command is not always enabled--it is only active when there is an active MCSession. The way we watch for this is in the remote control controller's implementation of the connectedDevicesChanged protocol method of CameraServiceManagerDelegate:
extension CameraControllerViewController : CameraServiceManagerDelegate {
func connectedDevicesChanged(manager: CameraServiceManager, state: MCSessionState, connectedDevices: [String]) {
NSOperationQueue.mainQueue().addOperationWithBlock({
switch (state) {
case .Connected:
self.enableButton()
case .Connecting:
break
case .NotConnected:
self.disableButton()
}
})
}
...
We make sure to execute this code on the app's main thread using NSOperationQueue.mainQueue().addOperationWithBlock(), as is necessary for the servicing of events.
Once the UIButton is enabled, when the user taps the button the tellCameraToTakePhoto method is called:
func tellCameraToTakePhoto() {
// SEND MESSAGE TO CAMERA TAKE PHOTO, ALONG WITH A BOOLEAN REPRESENTING
// WHETHER THE CAMERA SHOULD ATTEMPT SENDING THE PHOTO BACK TO THE CONTROLLER
cameraService.takePhoto(self.savePhoto!)
}
Let's take a look at the takePhoto function on the CameraServiceManager:
func takePhoto(sendPhoto: Bool) {
do {
var boolString = ""
if (sendPhoto) {
boolString = "true"
} else {
boolString = "false"
}
// ATTEMPT TO SEND DATA TO CAMERA
try self.session.sendData((boolString.dataUsingEncoding(NSUTF8StringEncoding))!, toPeers: self.session.connectedPeers, withMode: MCSessionSendDataMode.Reliable)
}
catch {
print("SOMETHING WENT WRONG IN CameraServiceManager.takePhoto()")
}
}
The MCSession's sendData method only supports sending UTF-8 strings as data between devices, so we convert the sendPhoto boolean into a string representing whether or not the camera, after snapping a photo, should send the photo back to the remote control device. When either peer receives data, the following method of the CameraServiceManager is called, which is a protocol method of MCSessionDelegate:
func session(session: MCSession, didReceiveData data: NSData, fromPeer peerID: MCPeerID) {
let dataString = NSString(data: data, encoding: NSUTF8StringEncoding)
// CHECK DATA STRING AND ACT ACCORDINGLY
if (dataString == "toggleFlash") {
self.delegate?.toggleFlash(self)
} else {
// CREATE VARIABLE REPRESENTING WHETHER OR NOT TO SEND PHOTO BACK TO CONTROLLER
if (dataString == "true" || dataString == "false") {
let sendPhoto : Bool?
if (dataString == "true") {
sendPhoto = true
} else {
sendPhoto = false
}
self.delegate?.shutterButtonTapped(self, sendPhoto!)
}
}
}
The above function is called whenever either device receives raw data. The camera device, after receiving the data sent from takePhoto(), will process the encoded string back into the NSString dataString which we can work with. We check the value of the dataString, and you can see by the first equality test, we use a similar method for toggling the flash of the camera device.
At this time our app is very simple so this code style may be acceptable, but if you're dealing with more complex data models you should create a Dictionary object, which can be encoded and decoded in UTF-8 and passed as an object between peers.
If the dataString does not equal toggleFlash, we know a command is being sent to take a photo so we test the dataString for "true" or "false", which represents whether the remote control device wants the photo sent back to it. The boolean that is derived is passed as a parameter to shutterButtonTapped, which is a protocol method of CameraServiceManagerDelegate. Let's see what this method looks like in the CameraViewController class:
func shutterButtonTapped(manager: CameraServiceManager, _ sendPhoto: Bool) {
self.sendPhoto = sendPhoto
dispatch_async(dispatch_get_main_queue(), {
self.takePhoto()
})
}
First the boolean instance variable sendPhoto for the CameraViewController is set to the corresponding boolean, and then the takePhoto method is called, which contains the code for saving image data using the captureStillImageAsynchronouslyFromConnection method of the AVCaptureStillImageOutput class:
func takePhoto() {
dispatch_async(self.sessionQueue, {
if let videoConnection = self.photoFileOutput!.connectionWithMediaType(AVMediaTypeVideo) {
if UIDevice.currentDevice().multitaskingSupported {
self.backgroundRecordId = UIApplication.sharedApplication().beginBackgroundTaskWithExpirationHandler({})
}
self.photoFileOutput!.captureStillImageAsynchronouslyFromConnection(videoConnection) {
(imageDataSampleBuffer, error) -> Void in
let imageData = AVCaptureStillImageOutput.jpegStillImageNSDataRepresentation(imageDataSampleBuffer)
let cgImage = UIImage(data: imageData)?.CGImage
// GRAB AN Int REPRESENTING THE CURRENT ORIENTATION OF THE iOS UI
let orientation = UIApplication.sharedApplication().statusBarOrientation.rawValue
var newImageOrientation : UIImageOrientation?
// SAVE IMAGE ORIENTATION TO newImageOrientation DEPENDING UPON DEVICE ROTATION
if (orientation == 1) {
newImageOrientation = UIImageOrientation.Right
} else if (orientation == 3) {
newImageOrientation = UIImageOrientation.Up
} else if (orientation == 4) {
newImageOrientation = UIImageOrientation.Down
}
// CREATE NEW IMAGE AND PRESERVE CORRECT ORIENTATION
self.lastImage = UIImage(CGImage: cgImage!, scale: 1.0, orientation: newImageOrientation!)
// IF THE REMOTE CONTROL DEVICE OPTED TO SAVE PHOTOS, TRANSFER THE PHOTO FILE
if (self.sendPhoto!) {
let outputFilePath = NSURL(fileURLWithPath: NSTemporaryDirectory()).URLByAppendingPathComponent("photo.jpg")
UIImageJPEGRepresentation(self.lastImage!, 100)?.writeToURL(outputFilePath, atomically: true)
self.cameraService.transferFile(outputFilePath)
}
// IF THE CAMERA OPTED TO SAVE PHOTOS, SAVE THE PHOTO TO CAMERA ROLL
if (self.savePhoto!) {
ALAssetsLibrary().writeImageToSavedPhotosAlbum(self.lastImage!.CGImage, orientation: ALAssetOrientation(rawValue: self.lastImage!.imageOrientation.rawValue)!, completionBlock: nil)
}
}
}
})
}
We convert the image data to a CGImage and create a new UIImage using the proper orientation and save it as photo.jpg in the NSTemporaryDirectory(). If the remote control device chose to save the photos from this session, we call the transferFile method in the CameraServiceManager instance cameraService. If the camera device chose to save photos from this session, this method saves those photos to the device's photo album. The transferFile method takes advantage of the sendResourceAtURL method of MCSession to send the file named photo.jpg from the user's temporary directory to the remote control device:
func transferFile(file: NSURL) {
if session.connectedPeers.count > 0 {
for id in session.connectedPeers {
self.session.sendResourceAtURL(file, withName: "photo.jpg", toPeer: id, withCompletionHandler: nil)
}
}
}
First we check that we have a connect peer to which we will send the file. Then, we use our instanced MCSession variable session's sendResourceAtURL method to send the file located at the NSURL passed in from the CameraViewController and give the file the name photo.jpg.(Note: This is one of the security risks; if a malicious user was able to join a session as a remote control, the above code would send photos to the legitimate user as well as the malicious user. As programmed above, the CameraServiceManager sends the photos to all connected peers.)
When the CameraControllerViewController starts receiving data, the CameraServiceManagerDelegate's didStartReceivingData method is called:
func didStartReceivingData(manager: CameraServiceManager, withName resourceName: String, withProgress progress: NSProgress) {
NSOperationQueue.mainQueue().addOperationWithBlock({
self.fileTransferLabel.text = "Receiving photo..."
self.fileTransferLabel.hidden = false
self.progressView.hidden = false
self.fileTransferProgress = progress
self.addObserver(self,
forKeyPath: "fileTransferProgress.fractionCompleted",
options: [.Old, .New],
context: &FileTransferProgressContext)
})
}
We don't do anything with this data in the method, but we do set up an observer to display a UIProgressView to the user so they can see the status of the file transfer. Most of the heavy lifting for the progressView functionality is done here:
override func observeValueForKeyPath(keyPath: String?, ofObject object: AnyObject?, change: [String : AnyObject]?, context: UnsafeMutablePointer<Void>) {
if (context == &FileTransferProgressContext) {
dispatch_async(dispatch_get_main_queue(), {
let fractionCompleted : Float = Float(self.fileTransferProgress.fractionCompleted)
self.progressView.setProgress(fractionCompleted, animated: true)
if fractionCompleted == 1.0 {
// DO THIS WHEN FILE TRANSFER IS COMPLETE
self.fileTransferLabel.text = "Saving photo to camera roll..."
}
})
} else {
return super.observeValueForKeyPath(keyPath, ofObject: object, change: change, context: context)
}
}
This process will take less than a few seconds over WiFi but may take as many as 10-20 seconds over Bluetooth depending on connection quality. Assuming the file transfer successfully completes, the CameraServiceManagerDelegate's didFinishReceivingData protocol method is called, which is implemented in the CameraControllerViewController as follows:
func didFinishReceivingData(manager: CameraServiceManager, url: NSURL) {
NSOperationQueue.mainQueue().addOperationWithBlock({
self.removeObserver(self,
forKeyPath: "fileTransferProgress.fractionCompleted",
context: &FileTransferProgressContext)
let fileSaveClosure : ALAssetsLibraryWriteImageCompletionBlock = {_,_ in
self.progressView.hidden = true
self.progressView.progress = 0
self.fileTransferLabel.text = "Photo saved to camera roll"
NSTimer.scheduledTimerWithTimeInterval(2, target: self, selector: "hideFileTransferLabel", userInfo: nil, repeats: false)
}
if (self.savePhoto!) {
// SAVE PHOTO TO PHOTOS APP
let data = NSData(contentsOfFile: url.absoluteString)
ALAssetsLibrary().writeImageDataToSavedPhotosAlbum(data, metadata: nil, completionBlock: fileSaveClosure)
}
})
}
First we remove the observer that was monitoring the NSProgress variable named fileTransferProgress. Then we create a closure of type ALAssetsLibraryWriteImageCompletionBlock to be called upon the successful saving of the photo to the remote control device's photo library.
##Conclusion
This is how the Multipeer Connectivity framework was used in the creation of Open Source Selfie Stick. In the future, I hope to refactor the code for this app and add new features such as increased security, video support, and a UIImagePickerController for browsing and sharing photos taken during a session. If you have any suggestions for improving this tutorial or the app itself, please feel free to comment.
- Create a graphic modeling the interaction between the
CameraControllerViewController,CameraServiceManager, andCameraViewController - Explain the need for
dispatch_asyncandNSOperationQueue.mainQueue() - Create a section about the camera functionality (using
AVCaptureDeviceInput,AVCaptureStillImageOutput,AVCamPreviewView, etc.) - Add documentation detailing the advanced photo features (white balance, ISO, etc.)
Very useful article! - Thanks @RF-Nelson this has really helped me to solve the video / multi-peer problem I'm having!