• Make sure that you are using the OpenTok on WebRTC JS library. You can find the library here, and find the reference documentation here. If you are using the v1.1 JS library, you will need to update your application to the v2.0 library.

• Use a browser with WebRTC support – Chrome 26, FireFox Beta, or IE9 with the Google Chrome Frame plugin.

• When you generate a session, make sure that the p2p.preference flag is set to disabled. If you’re generating your sessions from the Developer Dashboard, then you will need to download one of our server-side SDKs and generate sessions yourself.

• If you haven’t already asked to participate in the Mantis beta, please contact us at mantis@tokbox.com. Then make sure that you are using the correct API key for the Mantis beta. If you are not sure which API key you sent us, then please email us, and we will let you know. Mantis requires that your API key be enabled to access the infrastructure.

It really is that quick, and if you’re finding that you need some more help, then let us know. To make sure that your question gets answered as quickly as possible, please send an email to support@tokbox.com using the following template:

Email subject: [Mantis bug report] [Description of issue]
Email body:

API key:
Session ID:
Browser version:
Time of issue:

and then feel free to describe the issue further after filling out those fields.

We’re excited to see what our partners can build with Mantis, and we’re also ready and available whenever you need help. Happy coding!

This is another big step forward for the TokBox team as we continue to pursue our goal of providing application developers with simple yet powerful APIs. APIs that not only leverage the latest standards to deliver the best possible experience, but that are backed by a scalable, smart cloud which supports interoperability across a variety of end-points.

It was just over six months ago that we launched the OpenTok on WebRTC platform. Since that time we’ve been hard at work, constantly pushing the boundaries of OpenTok on WebRTC’s functionality and performance. We launched the first iOS SDK for WebRTC, introduced cross-platform and device support, improved connectivity with cross-platform TURN support and more.

Mantis for OpenTok on WebRTC acts as a central switching station for all the WebRTC streams in the OpenTok cloud.  Mantis enables:

• Reduced upload bandwidth consumption, with the ability to scale out a single WebRTC stream to many endpoints
• High-quality multi-party video calls
• Cross-browser compatibility for Chrome, Firefox and Internet Explorer (through Google Chrome Frame)
• Cross-device compatibility for iOS native apps and Chrome on Android
• SIP interop: the ability to dial in to a WebRTC video session from any cell or landline phone

Why should you care about Mantis?

1) It’s scalable…

WebRTC inherently uses a peer-to-peer based communication model.

In reality, while this is useful, several real-world applications naturally lend themselves to a multi-party use-case:  online classrooms, collaboration tools, the list goes on. The only way to enable a group video call using off-the-shelf WebRTC would be to create a mesh of PeerConnections between all participants in a call.

This mechanism doesn’t scale the number of connections well (it scales quadratically with the number of participants), and it is bandwidth inefficient. Every publisher needs to pipe out one video stream per call participant, leading to a linear increase in the amount of upload bandwidth required.

Mantis is a cloud-based media routing server, which enables us to get around this mesh problem by providing a centralized relay authority.

Every browser/device end-point thinks it’s connecting to a peer, but instead connects to one of our Mantis servers. Mantis can mediate multiple connections and then route streams from each publisher to any relevant end-point subscribing to that stream. Now OpenTok on WebRTC can scale subscribers effectively without constraints on publisher upload bandwidth, all while still leveraging standards-compliant WebRTC implementations under the covers.

2) It’s smart…

Scale is just one piece of the puzzle.

Mantis also provides a call-control/signaling mechanism that works in lockstep with media routing. This means OpenTok on WebRTC uses some pretty sophisticated algorithms to dynamically adapt to various network conditions that the client endpoints may experience.

Let’s say you were on your iPhone in a coffee shop with Wi-Fi, and then you step out into the street and switch to a much slower cellular connection. Mantis is aware of the change in network conditions and can adapt the bitrate of the encoded stream in order to adapt.

3) It interoperates…

Supporting different devices endpoints is one of the core tenets we strive towards at TokBox.  Mantis helps us work towards interoperability between WebRTC and other communications ecosystems.

One such use-case we are demonstrating with Mantis is interoperability with telephony endpoints. Leveraging Telefónica Digital’s telecom infrastructure, OpenTok on WebRTC can now connect PSTN or VOIP endpoints with WebRTC-enabled group video calls, using SIP to interoperate. This enables a user who is not video-enabled to dial-in to a multi-party OpenTok video session, and have the audio streams appropriately mixed and piped in each direction.

We believe Mantis lays the foundation for not only a powerful routing technology, but also for other functionality that can be layered on top of it: stream mixing, recording of media, signaling/event-notification APIs, and more.

We’re excited to show off our initial implementation of Mantis at the The Next Web ’13 in Amsterdam.  If you are going to be around, swing by our demo booth to see bandwidth-efficient multi-party WebRTC video in action, as well as a demo of dialing in to a WebRTC video call from your everyday mobile phone.

Mantis is now in external beta.  If you are interested in taking it for a spin, please contact us at mantis@tokbox.com.

Mantis: Cloud-scaling infrastructure for OpenTok on WebRTC from TokBox on Vimeo.

Firepad provides true collaborative editing, complete with intelligent OT-based merging and conflict resolution. It’s full-featured and has support for both rich text and code editing. Some of its features include cursor position synchronization, undo / redo, text highlighting, user attribution, presence detection, and version checkpointing.

For the first time, developers can now add text editing and document collaboration to their own apps without too much extra work.

This project is open-sourced, which is exciting because it means that we can build plugins for it. Since collaboration usually involves communication, we decided to build a video chat plugin right into Firepad, so you can not only collaborate, but talk to your companions as well. Here’s how you can add video chat to your firepad project:

First, include all the necessary Firepad Javascript and CSS libraries

Second, include TokBox library and Tokbox plugin for firepad:

Third, add a ‘div’ element for video to appear:

Lastly, put videos in that div element!

And you will now be able to video chat with your collaborators while you work together. For a live demo, check out our demo page.

Happy Coding!

*Disclaimer*: WebRTC is currently only supported in Chrome, Firefox Nightly, Internet Explorer (with Chrome Frame Plugin), and Android Chrome Beta (with webrtc flag enabled)

Highlights

• A lot of audio bugs in WebRTC were fixed dealing with crashes and non-standard audio bitrates
• Chrome on Android can now be WebRTC-enabled by enabling a flag
• Improvements to the connectivity stack in WebRTC
• Ability to set media constraints for audio

Full list

• Avoids crash in WebRTC audio clients for unsupported capture sample rates.
• Avoids crash in WebRTC audio clients for 96kHz render rate on Mac OSX.
• Enable webrtc build on android.
• Set WebMediaPlayerMS network state to loading instead of loaded
  • This indirectly fixes the problem where WebRTC audio is muted upon refresh. The HTMLMediaElement will try to cache fully Loaded videos when the element is destructed. This will signal to the HTMLMediaElement that the player was destroyed when loading, so it needs to recreate WebMediaPlayerMS upon destruction of the media tag.
• Allowing multiple MediaPlayers to connect to WebRtcAudioDeviceImpl by sharing one WebRtcAudioRenderer.
  •  The audio is gone when new PeerConnection is connecting to a media stream. What is happening is that the stream will pause the existing MediaPlayer and create new MediaPlayers to associated to it. But since we only allow one WebRtcAudioRenderer to connect to WebRtcAudioDeviceImpl, the new MediaPlayers audio won’t be able to associate to stream.

    This patch fixes the problem by allowing multiple MediaPlayers to connect to WebRtcAudioDeviceImpl by sharing one WebRtcAudioRenderer

•  Add webrtc audio-mirroring flag for automated testing
• Set MediaConstraints for audio to WebRTC
• When both stun and turn server are provided, use the turn server as the stun server.
•  This patch fixes two audio issues for webrtc hangout.
  • #1 Audio is muted when switching between the local stream to remote stream.
  • #2 It has glitches when switching between views.
    The first issue is fixed by storing the volume for local stream before muting, and recover the volume when it goes away. The second issue is fixed by re-using the |audio_renderer_| between different webmediaplayer.

Aviary provides an API that lets you put instagram-esque filters and all sorts of image editing capabilities to your web or mobile app. As an amateur instagrammer, I’ve always appreciated the ability to make an ordinary picture look good so this promise was actually one that I was excited to fulfill.

Inspired by Sendgrid’s Photo Booth at their SXSW party, I decided to build a virtual Photo Booth online.

Game Plan:

• Use OpenTok’s getting started tutorial to build a simple group video chat app.
• When user clicks on a button, snap a picture of each video by calling getImgData
• Use HTML5 canvas to stitch all the images together
• Pass the stitched image into Aviary and let it do its magic!

OpenTok provides a live video chat API. I started with a base nodejs project that uses each url as a video chat room. publisher is a video object of myself, and subscriber is the video object of other people’s streams that I’m watching or listening to.

To snap a picture of publisher or subscriber objects, call their ‘getImgData’ method to return a base64 encoded image string. Here is a code snippet iterating through an array of subscribers, getting image data, and displaying them in image tags.

To stitch the images together, draw each image onto a canvas and then retrieve a base64 string representation of the canvas. Since Aviary’s API requires an image element or the image element’s ID, I take the base64 image string from the canvas and put it into an image element.

Using Aviary’s API is the simplest part of this process. First you initialize Aviary’s Feather editor. The required fields here are your apiKey and apiVersion. Everything else is included by default. I also defined an onSave function handler so I can display the processed image to the user and automatically close the editor.

After initializing the Feather editor, simply launch it whenever some clicks on a button

And just like that, OpenTok+Aviary Photo Booth was born. Try it out! The code is open sourced and available on Github, feel free to fork it and do cool things.

If you are available on the weekend of April 6th, 2013, we are proud sponsors of Photo Hack Day 4 and we would love to see you there at Facebook HQ in Menlo Park for a full weekend of Photo-Centric Awesomeness! We will be giving away Bit Coins for the best use of OpenTok API.

If you can’t join us at the hackathon, no worries. Fork our Photo Booth project and add something cool (or find a bug in our code and send a pull request) and we will mail you our very own TokBox Wristband so you can go fist-bumping!

This year we are giving out TokBox WristBands. They are motion activated and light up with a brilliant flare whenever you shake hands or fist bump someone. Make a visual connection! Here’s how it works:

TokBox WristBands

Ankur Oberoi and Song Zheng will be roaming the city. To get your TokBox wristband simply find us and ask for one!

To find out where we are, tweet at @songz or @aoberoi with hashtag #sxsw

See you at SXSW!

We’ve also been working hard on making OpenTok on WebRTC work with both Firefox and Chrome so you too can enjoy all this cross-browser goodness!

Off to the races

The first thing that you need is version 21 or higher of Firefox, currently available through the Aurora FTP site and Nightly site.

If you’re a Firefox user already you may want to set up a separate profile to test with. This will keep your default profile clean, and your test profile will likely also start up a bit faster than your default one. You can find instructions on how to do this here and here.

If you’re testing with Chrome, you will want to make sure that you have updated to the latest version (version 25 at the time of this article), which has all of the fixes required for interoperability with Firefox. All of the relevant WebRTC bits are enabled by default in Firefox 21+ and in Chrome 25, so you should be ready to roll. Now go have a play with TokBox’s OpenTok on WebRTC demo. Then come back here afterwards.

When can I have this in the production release of Firefox?

Parts of WebRTC are working in the current production release version of Firefox (Firefox 19). However, Firefox 21 is the version that interoperates successfully with Chrome, and it also has all of the WebRTC preferences turned on by default. The gang at Mozilla is also pushing hard to fix lots of critical WebRTC bugs before production release of Firefox 21 and hopefully turning on WebRTC for everyone. According to the Mozilla release calendar, Firefox 21 is targeted to be released the week of May 13th.

If this isn’t in the production release of Firefox, why is this news?

What we’re doing today is giving you a head start.

The great thing about OpenTok on WebRTC is that we make face-to-face video work in your users’ browsers when those browsers are ready to work correctly.  So our announcement today lets you know that we are ready for Firefox WebRTC support to go production.  Right now, you can install Firefox Aurora and use it to test your app using OpenTok on WebRTC.  And you can watch as Firefox 21 makes it from Aurora to Beta and then to release and know when a big new chunk of your users can get access to WebRTC through OpenTok on WebRTC, all without your having to write an extra line of code.

What doesn’t work yet?

Mozilla isn’t quite done yet, so there are still some holes. The biggest one involves anything to do with disabling audio and video tracks. Namely:

• Muting audio
• subscribeToAudio and subscribeToVideo on Subscribers
• publishAudio and publishVideo on Publishers

You may also see some connectivity issues in particularly restrictive networks. We’re working on these issues now, and we’ll update you as we have good news to share in the not too distant future.

What were some of the challenges of getting Firefox and Chrome to play nicely?

Some of things we had to do were pretty straight-forward – normalizing the cosmetic differences between the APIs, for example. This just involved making sure that we were using the appropriate vendor prefixed object and function names, say webkitGetUserMedia in Chrome and mozGetUserMedia in Mozilla.

The more complex bits involved working around browser specific issues and incompatibilities in the messaging layer (this is the part that uses text blobs of SDP). One notable example was that earlier implementations in Chrome and Firefox didn’t support compatible implementations of cryptography, with Firefox supporting DTLS and Chrome SDES. This was eventually resolved, but a new related issue popped up in Chrome as a result.

Simply, DTLS adds an extra “a=fingerprint” line to the SDP payload which makes Chrome 25 incorrectly assume that the payload is SDES when it sees it. If SDES was being used there should also be an “a=crypto” line, but in this case there won’t be. So Chrome complains and won’t accept the SDP message.

As Chrome doesn’t actually care about the value of the crypto line, only that it’s present and well formed, the solution is to add a dummy crypto line when sending offers from Firefox to Chrome. Google is landing a fix for this in Chrome 26, but leaving the extra line in causes no harm and allows compatibility with Chrome 25. The dummy line we add in OpenTok looks like this:

a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:FakeFakeFakeFakeFakeFakeFakeFakeFakeFake\\r\\n

On the Mozilla side, Firefox doesn’t support FQDNs when specifying ICE servers for the PeerConnection. There’s a Bug open for this on Bugzilla, but in the meantime, ICE servers must to be specified as raw IPs for Firefox.

Another issue is caused by Firefox sending Data Channel information as part of its JSEP Offers – something that Firefox is doing temporarily until its Data Channels implementation is completed. Unfortunately this upsets Chrome and requires another workaround.

The good news is that we are doing all this hard work for you – making OpenTok on WebRTC interoperate smoothly – so you can spend your time building great applications, not worrying about browser compatibility between different implementations of the WebRTC standard.

Wrapping it up

At TokBox we’re super excited to see another great browser jump on board with the WebRTC standard, and we’re really impressed with how quickly the Firefox and Chrome teams have come together to make this happen. We really believe in the power of WebRTC – in particular, in the quality of the video and audio it delivers while maintaining low latency. We’re constantly keeping an eye out for all of the new developments in WebRTC, making sure we’re staying ahead of the curve to keep our API supporting the latest and greatest. Now we’re off to take a hard look at Microsoft’s first implementation of CU-RTC-Web.

Last week I wrote a post that called for more hackathons to be purpose driven—Music Hack Day is not one of those of events. Instead, Music Hack Day is an event driven by a desire to learn and a shared passion for music. These types of events are, without question, very good for the hacker ecosystem, and Music Hack Day is a shining example of how they should be run.

What makes Music Hack Day great is that expectations are clear. There is no question—we’re there to mingle, build, and learn—not to make money. There’s only a few small prizes and no PowerPoints—it’s a true hacker’s hackathon. And over the years this event has blossomed into a passionate community filled with incredibly smart developers and musicians.

Here’s a couple of my favorite hacks from the event:

Leap Orchestra
An orchestra controlled by the new Leap Motion device.

TokBox Ukulele
Play an imaginary instrument via webcam using OpenTok.

Live Crowdsourced Concert Visuals with LSD
A collaborative VJing webapp that uses OpenTok to crowd source visuals from a concert audience.

Mugatu
A playable piano neck tie.

Inspiration
Provides example chord progressions for a given genre or song.

We’re thankful to the wonderful folks at Echo Nest and the attendees for making it a pleasure for us to host. Cya next year!

In today’s struggling economy, the founders of LiveNinja saw a new work trend emerging. Friends, family and acquaintances were setting up consultancy shops online, using siloed video chat services like Skype to connect face-to-face with clients. While this worked for some consultants, they also saw many people struggling to bring in to new clients. Just google “Spanish lessons” and imagine how hard it would be to stand out amongst the 59 million + results. ¡Ay caramba!

On why his team decided to develop LiveNinja, CEO and Co-founder Will Weinraub said:

The number of skillful individuals across the world that are unemployed and unable to produce income is exorbitant. However, we feel if we can modernize the traditional concepts of employment by providing the proper tools, we can eliminate the barriers to both job and income creation. LiveNinja effectively allows people to create and accelerate these jobs by empowering any individual with all the tools they need for an online service-based business.

In response, LiveNinja created a full service marketplace that enables experts to set up their own web page and customize the look and feel. Certified Ninjas can set their own rates and schedules, and collect income in the LiveNinja one stop shop.

What is the experience like for a student? Let’s say you’re dying to improve the height of your round-house kick (who isn’t?) Simply head over to LiveNinja and search for “martial arts”. If they can’t find exactly what you’re looking for, LiveNinja’s intelligent search will make recommendations for other areas you may be interested in. Once you find the expert you’re looking for, simply book an appointment and let them welcome you to their virtual dojo.

Have a skill that you’d like to share with the world? Become a Certified Ninja today. Prefer to learn something new? Start browsing expert profiles, reading user reviews, and booking sessions. Welcome to the dojo.

Walking away from the panel, I found myself quite convinced of a few things that beforehand I had only held as loose opinions, and I wanted to share those with the TokBox blog audience.

1) Ultimately a common language set will prevail… but not for the reasons you’re thinking
Today, as we all know, it takes a developer a long time, and real effort, to be available on all platforms (namely web, iOS, and Android) from day one. The reason is simple – HTML5 based applications don’t perform as well as their native counterparts. If anyone learned this lesson the hard way, it was Facebook, but other app developers have similar horror stories.

Further, it takes first-world incomes to be able to buy the top of the line smartphones from the likes of Apple, Samsung, and Motorola. Most of these devices are extreme luxury items outside of the US and Western Europe. As a result, an already saturated app store for existing platforms is a daunting marketplace for new and emerging apps.

Given all of that, we asked ourselves… but what if the native API was HTML5?

The upcoming launch of Firefox OS is exciting exactly because developers targeting untapped markets (Latin America, South America, Africa) using skills (HTML5, JS, and CSS) that they already have is a large ocean of opportunity.

Firefox OS phones, a device our Telefonica Digital family have dedicated significant time to developing, may not compete on hardware specs with the iPhones we have in the US and Western Europe, but, as a platform for the masses, their reach could be much greater, open much larger markets, and open many more interesting opportunities in the years to come. Simultaneously, as mobile browsers continue to invest in speed and quality, a second wave of HTML5 apps wrapped in native application bindings could make a comeback.

The sum of these parts is that developers who have built HTML5-based apps will be well positioned to take advantage of the next generation of smartphone devices, but the panel agreed that today the burden lies on the developer to build multiple native applications.

2) Frameworks are only as powerful as the developer who wields them
Ted Drake spoke a lot about how accessibility has been pushed to new, and powerful, places with the emergence of smartphones and tablet devices. However, he decried frameworks as a programming paradigm where, “shit goes in, and shit comes out.”

This, I thought, was one of the better topics we debated.

It started with the thesis that frameworks are used by developers who don’t know what they’re doing as a shortcut or to quickly prototype their work. However, Jonathan Smiley spoke up for framework users by giving the example of his agency who had built a framework that they used to quickly, and robustly, build responsive applications for their clients. Zurb  knew what they were doing, and knew how to use the tool effectively.

Ultimately, I think the entire panel agreed that the issue with frameworks is speed more than complexity. If people are moving too fast, while not understanding the programming paradigm, and producing second-class applications as a result, then of course, “shit goes in, and shit comes out.” But a framework in the right hands used to solve the right problems can really help development, especially when it has to work across both mobile + web at launch.

I’m personally a huge fan of frameworks as they abstract the mess away from developers. Our OpenTok thesis is that we worry about the details so that our partners don’t have to. Instead they build really great applications that take advantage of a live-video platform without signing up for the hassle that usually comes with one.

3) Regardless of the target audience – mobile + web will both be part of the solution
One of the questions from the audience was about whether our discussions over the course of the hour applied equally to enterprise as well as consumer. At TokBox we’ve learned over the last two years of the OpenTok platform that solutions need to exist wherever the users are.

Doing a customer support app? People may want to call in from a website, a native app, or from the phone.

Heading to the doctor’s office remotely? Requiring patients to buy an iPad would make the solution impossible for most doctors, but having a solution that works as long as the patient has a web connection opens this door.

Mobile + Web is not a consumer only phenomenon, though the headlines are dominated by these types of products. The truth is that mobile + web is the new paradigm in which all products are being built.

I wanted to quickly thank Lisa Oshima who was the moderator of our panel, and just say again how much fun I had. I’d love to hear more about where you think mobile + web development is going so please comment or email me at melih@tokbox.com.