Archive for the ‘Development’ Category

I worked on Apple’s tablet

No, not that tablet.

The year was 1992. I was still at school doing some research on user interfaces based on speech and gestures. But I had this problem: I had all these cool ideas but no hardware to try them on.

I was young, naive, and an Apple fanboy before the term was even invented. In May I traveled to San Jose to attend Apple’s WWDC and at one of the sessions Ron Avitzur demoed a software that turned pen scribbles into neatly formatted equations. That was the coolest demo I saw at WWDC that year. I tracked down the Apple guy who was hosting the session, and asked if he was looking for a summer intern. There was only an extremely remote chance he would say yes, but fortune favors the bold. And that day fortune was very, very kind to me. I joined Apple’s Human Interface Team in August for an internship that was only supposed to last a couple of months. I would leave Apple 10 wonderful years later.

I soon found out that Apple was working on a project code-named PenLite. The hardware was based on the Macintosh Powerbook Duo. It shared the same motherboard and greyscale screen, and the same special dock connector for peripherals such as floppy disc drives and Ethernet (well… AppleTalk). It lacked a keyboard, but its display could detect the hovering and taps of a special pen.

Our job was to write PenMac, the system software extensions to Mac OS that would allow existing Mac applications to be used in this environment. It was a very interesting problem. For example, with the menu bar at the top of the screen your hand would hide the menu after you tapped on it. You also quickly realized how you take for granted the shift and commands keys.

I started working on the problem of gesture recognition, figuring that gestures would perhaps be a way of bringing back the convenience of keyboard shortcuts. I came up with a few interesting ways of doing this that ended up as patents for Apple (5,583,946, 5,590,219, 5,594,810 and 5,749,070).

As an aside, in 1993 we were paid a visit by a couple of guys from San Diego who were working on a drawing app they called SmartSketch. You could sketch four lines with a mouse and they would turn into a square. As much as it was fun to use with a mouse, it really shined when using a pen: it’s much easier to sketch with a pen than with a mouse and it made for a much more natural interaction. We gave them much encouragement to build pen-based software. They eventually founded a company and called it FutureWave. In 1995 they added support to create animations and called their new app FutureSplash Animator. Macromedia acquired FutureWave in 1996, and renamed FutureSplash Animator to Flash 1.0. Guess what I am working on these days… Small world.

Apple canceled the PenLite project in August 1993 as it decided to focus on Newton. We were all sad, but that experiment had also showed the limitations of repurposing a traditional desktop user interface. And yet, having a clipboard-sized computing device was deeply compelling. Despite the low resolution of the screen compared to what today’s technology can offer, the form factor of the device, the ability to cradle it in your arms, made for a fundamentally different, more intimate, experience than a laptop.

And yet, the Windows-based tablet computers of today suffer from the same problem: repurposing for a tablet a UI designed for a desktop is like eating pea soup with a fork.

Here’s hoping we will soon see clipboard-sized devices with a custom-designed user interface. I want to eat my ice cream with a spoon.

MAX 2009: Adobe AIR 2.0 and iPhone support

My team has been working on a couple of cool products that we’re unveiling at the MAX 2009 conference today. We have some related sessions I’m detailing below.

AIR 2.0

At MAX this week we’re showing a preview of AIR 2.0, codenamed Athena. We’ve added tons of cool new features, but first, we’ve worked on reducing the memory and CPU usage for a lot of AIR applications. We’ve also listened to what our developers were asking for and we’re adding some of the most popular features:

  • native code integration: you can invoke native code from your AIR application
  • native installers will give you the option to package and distribute your AIR app as a native installer
  • gesture recognition on Mac OS and Windows (multi-touch on Windows 7),
  • much improved printing APIs (you can skip the printing dialog, you can set tons of printing options programatically)
  • support to open documents with their native applications
  • detection of drive mounting and unmounting so you can tell when a USB mass storage device is connected for example
  • much improved networking APIs, including support for UDP, TLS (encrypted binary sockets), DNS lookups, IPv6 and more
  • raw microphone APIs so you can record from the microphone and process the data or store it locally without going through a server
  • improved accessibility support, including support for screen readers
  • finally, a small feature that will be popular with developers: we’re adding support for a global error handler so you can more easily catch and report problems in your application

You can find out more about AIR 2.0 at the following sessions:

Explore Deployment and Distribution Options for Adobe AIR Applications

by Oliver Goldman — Monday 10/5, 2pm, #adobemax51, Room 501A
Oliver will talk about how we continue to improve your options to deploy and distribute your AIR applications.

What’s Coming in Adobe AIR 2.0

by Christian Cantrell — Monday 10/5, 5pm, #adobemax335, Room 515A
Christian will give an overview of some of the new features in AIR 2.0, including some cool demos using some of the new APIs.

Meet the AIR and Flash Player Team

Monday 10/5, 8pm & 9pm, Room 512
Come to this session to ask your questions about AIR, Flash Player and the iPhone support.

What’s Coming in Adobe AIR 2

by Christian Cantrell — Tuesday 10/6, 4:30pm, #adobemax198, Room 411
Repeat of Christian’s session on Monday.

iPhone support

Opening the iPhone for Flash developers
For the past 6 months we’ve been quietly working on a surprise: Flash Pro CS5 can now build native application for the iPhone and iPod touch. This opens these devices to Flash developers and allows them to use Actionscript and the tools they are already familiar with to build native applications for the iPhone. The applications are compiled to native ARM machine code: there is no interpreter involved. You get access to many of the APIs available in Flash Player and AIR, plus a few extra ones. We have also added support in Actionscript for multi-touch, gestures, accelerometer, geolocation, etc…

We have been working with a few developers and they have built some really cool apps that are now available in the iTunes App Store for you to download and play with.

Apps in the iTunes App Store

Those apps are:

  • Chroma Circuit by Josh Tynjala of Bowler Hat Games
  • Trading Stuff by Ben Garney of PushButton Labs
  • Fickleblox by BlueskyNorth
  • That Roach by Break Design
  • Red Hood by Differences Games
  • Just Letters by muchosmedia
  • Word Zen by Greg Burch
  • Southpark Avatar Maker, by Southpark Studios

Deep thanks to all our pre-release developers for working with us and for their patience as we were working out the kinks.

To find out more: adobe.com/go/iphone or come to one of the sessions below, including my session Tuesday at 3pm.

Building Mobile Applications with Flash Pro

by Aditya Bansod — Monday 10/5, 2pm, #adobemax315, Room 402A
Aditya will present an overview of the tooling support and explain how you will be able to use Flash Pro CS5 to build applications for the iPhone. This will be a good introduction to how everything works.

Meet the AIR and Flash Player Team

Monday 10/5, 8pm & 9pm, Room 512,
Come to this session to ask your questions about AIR, Flash Player and the iPhone support.

Optimizing Flash Content for iPhone Applications

by Scott Petersen and Chris Brichford — Tuesday 10/6, 1:30pm, #adobemax402, Room TBD
Scott and Chris will describe in details how building applications for the iPhone using Flash Pro is different, and how you can optimize both your Actionscript to make sure the compiler produces the best possible result, as well as how to take advantage of hardware acceleration to make sure your app flies. This is an advanced session.

Designing Applications for Desktops and Mobile Devices with Adobe AIR

by Arno Gourdol — Tuesday 10/6, 3pm, #adobemax351, Room 515B
There’s a lot of things to consider when building apps for the iPhone. Applications on mobile devices are used differently than on a desktop. New modes of interaction and new features are available that open up new possibilities, such as geolocation or accelerometer. I’ll cover these points and give you tips on how you can start developing applications today on the desktop to make them available on the iPhone soon. I will focus mostly on the design aspects, with a few tips on how to optimize for performance.

Designing apps for the iPhone using Flash Pro

Building applications for the iPhone using Flash – Q&A

Wednesday 10/7, 9:30am, #adobemax159, Room 511C
Come join me and members of my team and ask us your questions about how we’re opening the iPhone to Flash developers

Building Mobile Applications with Flash Pro

by Aditya Bansod — Wednesday 10/7, 11am, #adobemax72, Room 501A
Repeat of Monday’s presentation

Optimizing Flash Content for iPhone Applications

by Scott Petersen and Chris Brichford — Wednesday 10/7, 11am, #adobemax402, Room TBD
Repeat of Tuesday’s presentation



We’re very excited to bring you these cool products and we’re looking forward to your feedback.

MAX 2009

I’ve made my way to Los Angeles to attend MAX 2009.

The Adobe logo decorates the entrance of the LA Convention Center, location of the MAX 2009 conference"

As can be expected with this kind of event, there’s still a lot of last minute work but everything is looking good now. The keynote tomorrow is looking out to be awesome. Lots of exciting announcements and great demos to show.

It’s the same kind of excitement I experienced just before the introduction of Mac OS X in 2000 at MacWorld San Francisco. I remember we were still fiddling with the demo machines late into the night before the keynote: we had one main machine and two backups tucked discreetly below the desk that Steve used. A KVM switch allowed him to switch to one of the backups if something bad happened. During the keynote, I was sweating bullets in the front row, hoping that everything would work fine. Everything ended up working splendidly, but I didn’t relax until the keynote was over and Steve high-fived me backstage. We were both relieved to have finally revealed publicly what we had been working on secretly for so many months.

I’m really looking forward to the keynote tomorrow. We have some nice surprises and I’ll be high-fiving everyone around me :-)

Thibault Imbert writes at ByteArray.org about Faster JPEG Encoding in Flash Player 10:

So what did I do ?

  • I used bitwise operators as much as possible.
  • I replaced all Arrays with Vectors.
  • I used pre-incrementation rather than post-incrementation (thanks Joa for this one ;-) ).
  • I casted to int all my Vector indices access.
  • Other minor stuff you always do to optimize your code :-)

The result: a 300% speedup. Not bad.

Writing well-behaved, efficient, AIR applications

The Adobe AIR platform makes it possible for many talented developers familiar with AJAX or Flash to build desktop applications. However, with great power comes great responsibility.

While an app running inside the browser can count on being short lived, it’s a different story with a desktop application. A desktop app may run for hours, if not days, at a time. Think of the many popular Twitter clients as an example.

As a desktop developer you have to give more thought to the resource usage of your application, such as the memory and the CPU, which is what I’m going to focus on here. The quantity of CPU cycles available on the machine is a resource shared by all the other apps (and system processes). Every CPU cycle your app use is a CPU cycle not available for something else. When the user is interacting directly with your application, using CPU cycles to present the most engaging experience possible is reasonable. However, when your application is in the background, your goal should be to reduce your CPU usage so that your app is as unobtrusive as possible and so that the application that the user is directly interacting with has more CPU cycles to use.

Keep in mind also that how much of the CPU is used affects the power consumption and therefore the battery life of laptops. Reduce the CPU usage of your app and save the planet.

What can you do? The first step is to measure. You can use some simple tools to get some rough data. For example, the Activity Monitor application on Mac OS, the top command line tool on Linux and the Task Manager on Windows.

Use these tools to watch out for the CPU usage of your application. Bring your application to the front, interact with it, and consider if the reported CPU usage is what you expect. While decoding high-def video you should expect that more of the CPU will be used than if your app is just doing some text editing.

Now, bring another app to the foreground. Watch out for the CPU usage of your app again. It should have dropped significantly. It’s OK for that number to jump back up from time to time (for example if you connect to a web service from time to time), but in general, lower is better.

As an aside, you can also use these tools to measure your app memory usage. Keep your app running for 24 hours, then measure the memory usage again. It should not have increased significantly, if at all.

How can you lower your CPU usage? Here are a few tips to consider.

1. Use the lowest framerate possible

The framerate of your application refers to how often the windows of your application are refreshed (the stage is redrawn on screen). With a framerate of 24, the content of your app is refreshed every 41ms (24 times per second). The framerate also defines how often the ENTER_FRAME event handler is invoked.

If you build a Flex-based application, the framerate by default will be 24. For many apps, that may be more than you really need. Try to lower the framerate as low as you can. Start with 7fps, then try to go lower. Once you’ve lowered the framerate see if there’s any interaction glitch while using your app. Watch out for animation and transitions in your app and make sure they’re still smooth.

To change the framerate of a Flex application, set the frameRate attribute in your mx:WindowedApplication tag.

For example:

<mx:WindowedApplication xmlns:mx=”http://www.adobe.com/2006/mxml”
   layout=”absolute”
   frameRate=”7″
>

If you are using Actionscript or Javascript, you can set the stage.frameRate property to the desired value.

In many cases a lower framerate has no discernible effect on the quality of animation or the interactivity of your application but will result in a decreased CPU usage. Notice also that the refresh of video is independent of the framerate of your application, so even if you’re playing video, try to lower your framerate.

2. Dynamically change the framerate to fit your application needs

In addition to lowering the framerate of your application overall, you should also consider changing the framerate depending on what your application needs at different point in time. For example, you may need a higher framerate when displaying an animation. However, this may happen infrequently and there’s no reason to set the framerate of your application to the maximum you will ever need. Instead, you can temporarily increase the framerate when you need it, then set it back to a lower value.

For example, you may need to increase the framerate (maybe to 12 or 24 frame per second) during a visual transition (a Flex state change), while responding to a dragging operation, while your window is being resized or the layout of your application changes, or while playing video.

On the other hand, consider lowering the framerate even further when your application goes in the background. If your app is in the background, that’s probably a good indication that the user is focussed on something else, and if it means pausing some animation, or having them occur with less fidelity, that may be a valuable tradeoff. You can set your framerate as low as 0.1 or even to 0. Even with a 0 framerate, your app is not completely paused. It will still be sent events to react to, such as activation, window move and resize, mouse and keyboard, etc…

While in the background, if you have an ENTER_FRAME handler or some timers, you might want to consider unregistering them to further reduce your CPU usage. Restore them when your application is brought back to the foreground.

For example:

// In your application initializer:
this.addEventListener(AIREvent.APPLICATION_DEACTIVATE,
applicationDeactivate);

// The application is being sent to the background
public function applicationDeactivate(event:Event):void {
    this.removeEventListener(Event.ENTER_FRAME, enterFrame);
    stage.frameRate = 0.1;
}

The above example uses Flex, but you can accomplish the same thing using the Event.DEACTIVATE event.

3. Only use Event.ENTER_FRAME handlers when necessary. Which is almost never.

You could think that Event.ENTER_FRAME handlers provide a convenient mean to perform some repetitive action. However, you must be mindful that they are not called just “once in a while”, but with every single frame that gets drawn on screen. That’s a lot. In general, you should only use an Event.ENTER_FRAME handler if you need to do some processing whenever the display is refreshed. That should be exceedingly rare. There’s usually a better (more efficient) way than using an Event.ENTER_FRAME handler. In fact, the only case I can think of where you could justify using Event.ENTER_FRAME is to calculate the effective framerate of your application.

For example, if you need to track the mouse, register for Event.MOUSE_MOVE events instead. Those events will only get dispatched when the mouse is actually moved. You should consider register for Event.MOUSE_ENTER and Event.MOUSE_LEAVE if that’s really the only thing you need to know.

If you need to do some processing not related to display, for example some sound processing, use a Timer instead of Event.ENTER_FRAME. You can choose how frequently to invoke the timer based on the amount of processing you need. You may be able to use a lower framerate for the stage, and a higher frequency for the timer. This will save considerably on the amount of CPU used.

Note that for animations, you are often better off using a Timer as well. The Event.ENTER_FRAME handler may not be called at exactly the framerate you expect anyway. As a result, if you base an animation on how frequently this handler is called, stutter may be visible. Instead, base your animation on absolute time, which will allow you to vary the framerate as needed without affecting the animation:

// Ease function to make the logo blink slowly
public function ease(t:Number): Number {
    return (Math.sin(Math.PI * (2 * t - 0.5)) + 1 ) / 2;
}

// This handler is called anytime a new frame is
// drawn. Use sparingly.
public function enterFrame(event:Event):void {
       logo.alpha = 0.2 + ease(new Date().time / 3000);
}

Whether you use a Timer or a Event.ENTER_FRAME handler make sure you stop and unregister them as soon as you can. Even a handler that does “nothing” will be consuming some significant amount of CPU cycles. That actually also applies to any kind of other event handlers, such as Event.MOUSE_MOVE. As soon as you’re done with them, make sure to unregister them.

The same advice that applied to your overall application framerate also applies to the frequency of timers: try to use the lowest frequency you can get away with. Experiment to find out what is acceptable. Also, vary the frequency of timers as needed, for example reducing them when your application is deactivated.

4. Have as few Event.ENTER_FRAME handlers and Timers as possible

You often need to do periodic operations for a variety of reason. It would be tempting to have multiple Timers or multiple Event.ENTER_FRAME handlers. However, there’s a non insignificant overhead with each Timer (or handler). Instead, try to group all the periodic operation that you need to perform in as few Timers as possible.

All the operations that need to happen at the same frequency should be handled in the same timer. Try to have at most a couple of Timers, maybe one for animation with a frequency of 40ms and another to do “background” operations every 2000ms or so. Turn off those timers whenever you’re done (for example, turn off the animation timer when the animation is done, and start it again if you need a new animation).

Note also that setInterval and clearInterval are implemented using Timers, so don’t think they are any lighter weight.

And to conclude…
AIR Idle Activity Monitor screenshot

Here’s a sample app that demonstrates some of those techniques. The result is an app that uses less than 1% of CPU on MacĀ OS when idle in the background. Your app can do that too!