I want handwriting recognition on the iPhone and iPad — even if it stinks

When I was a kid, I got my Dad's old Newton MessagePad 2100 — Apple's original personal digital assistant — as a "hand-me-down". I put that phrase in quotes because it really wasn't so much a hand-me-down as it was a "hand-it-over": After a year of intermittently playing with his 2100, I claimed it for my own.

While I may have... exaggerated some of the Newton's abilities to my awed classmates — I distinctly remember saying to a teacher "I'm going to scan the blackboard and have the Newton take a picture of your notes", ha ha — in 1999, there was nothing I loved more than demonstrating its handwriting recognition. It was almost guaranteed a "We're living in the future!", if not a "Oooo, let me try!"

Much as I hated longform writing and as terrible as my penmanship was, there was still something pretty magical and otherworldly about writing block or cursive letters and having them transform into digital text. By the year 2000, I had convinced most of my teachers that I could turn in typed work, but I probably saved future Serenity's penmanship from becoming truly atrocious because I had that Newton MessagePad. I was constantly writing and doodling with it.

Sixteen years later, the Newton lives again with my Dad (though probably in a storage box somewhere), and I've got both an iPhone and an iPad Pro to obsess over. But while there are many miraculous features I could demonstrate, handwriting recognition tricks have yet to make it to iOS.

Egg freckles and handwriting troubles

There are plenty of reasons why Apple hasn't yet implemented system-level handwriting recognition in iOS: For one, you need an exceptionally good pen input, and — until the introduction of the Pencil — no third-party stylus had the precision or the lag reduction to be a truly useful tool.

There's also both the processing power and stigma to consider. Like Siri, handwriting recognition works as a transcription service; in order for it to translate your physical marks into typed characters, however, it needs a substantial dictionary and the power to transcribe your words using your phone's processor or online servers. That means more cloud infrastructure or more powerful iOS devices — or both — as well as the engineers to craft a smart dictionary.

If you've ever dealt with Siri errors, imagine the problems that might result from poor handwriting, or trying to decipher the difference between lowercase, upper case, and cursive letters. And indeed, early Newton models got a lot of flack for mistranslating the written word, though later models — like my dear 2100 — saw significant improvement.

I also wonder, along those lines, if our collective penmanship hasn't become so disastrous that transcribing it into digital text would be an even bigger headache than previously imagined. Many primary schools no longer teach handwriting, and most of us write more words with a virtual keyboard than we do a pen and paper.

In the end, we're talking about a minor feature, hardly requested, that likely takes a backseat to all the more important projects Apple has in the pipeline — if the company's engineers are thinking about it at all.

Handwriting our way forward

But despite all the potential problems with handwriting recognition baked into iOS, there's a part of me that still wants to see the feature come to the iPad — even if it's only a beta baked into the Notes app.

We now have a top-tier stylus option for iPad in the form of the Pencil, and that option might be coming to smaller iPads soon. And there are system-level APIs in iOS that support great palm rejection, accuracy, and low-level latency.

Additionally, the iPad Pro's beefy A9X should be more than enough to process the transcription necessary for handwriting recognition, and there are there are two companies out there to prove it: Both indie developer Viet Tran and handwriting recognition leader MyScript have apps on the App Store that perform handwriting recognition.

The apps worked quickly in my tests, but unfortunately neither supports the Pencil's palm rejection, lag reduction, or pressure sensitivity, leading to a sub-par writing experience. But if those apps incorporate proper support for the Pencil, the experience would be downright pleasant. Both apps' underlying transcription engines are impressive, and if my handwriting weren't so sloppy on account of lag, they might be even better.

Maybe the solution is just that: Let those the third-party apps take control of the handwriting recognition market, and Apple can continue to work on other, more important aspects of iOS (like drag-and-drop between Split View panes). I hope I'm wrong, though. I don't want handwriting recognition limited to a single app, or as an option that requires a third-party keyboard. I want system-level support.

And for all the reasons reasons not to add this feature to iOS, there are some pretty compelling reasons to do it, too.

If nothing else, Apple has some pretty good handwriting recognition software lying around: The company's aging Inkwell) technology is based off the original Newton Rosetta framework, and lets people with a connected tablet transcribe letters and draw pictures. It hasn't been officially updated in an age, but it's still supported as of OS X El Capitan, and the recognition engine is halfway decent.

There's also Apple's tablet competitors: Windows 10 has handwriting recognition built in to the Surface line, while Google released a handwriting keyboard for Android in mid-2015.

Knowing all that, I could see an updated version of Inkwell making its way into the Notes app in iOS 10, even if it were only a feature for those with Apple Pencils and compatible iPads. Apple put a renewed focus on Notes with iOS 9, and even if that was partially just to demonstrate the Apple Pencil's capability, what better way to expound on that utility than handwriting recognition?

In Notes, adding a button next to the Sketch tool that lets you transcribe your written words would be useful for buisness professions that prefer handwriting to typing. Anyone walking around with an iPad in their arms may elect to write over trying to finger-type in their notes — doctors were my first thought, but architects and teachers could certainly benefit, too. And the more users playing with Apple's software, the faster and quicker it could improve.

To handwrite or not to handwrite?

What side do you come down on, iMore? Is it worth Apple devoting resources to handwriting recognition at a system level, now that we have the Pencil? Should we leave it up to the app developers? Let me know what side you come down on; meanwhile, I'll be over here in the corner doodling Egg Freckles.

Serenity Caldwell

Serenity was formerly the Managing Editor at iMore, and now works for Apple. She's been talking, writing about, and tinkering with Apple products since she was old enough to double-click. In her spare time, she sketches, sings, and in her secret superhero life, plays roller derby. Follow her on Twitter @settern.

  • Well, there IS already a third-party keyboard with handwriting recognition working in all apps: MyScript Stack. But still without official support for Apple Pencil (why?). I just don't see the need for an official Apple solution. That's why Apple introduced the system integration of third-party keyboards.
  • 1. Apple provides a very limited API for third party keyboards, it cannot consume more than a certain amount of memory and CPU, which means it can't use more resources that handwriting recognition will require. It also cannot do more than what Apple provides for its API. I suspect Apple does not allow Pencil in third party keyboards for various reasons.
    2. Third party keyboards has privacy issues and would not work in all fields. If such a feature requires offloading to the cloud, you are sharing the content of your writing to the developer and the rest of the world if they don't protect their stuff properly. I can trust Apple to do this but I do not trust anyone else.
    3. You need an official API support for Pencil integration via third party keyboards and if Apple is going to provide it, it might as well go all the way with the handwriting recognition anyway in areas throughout the OS like Notes.app. I would love Notes to transcribe my notes into OCR text.
    4. No one can provide a better implementation than Apple, everyone must use Apple's APIs. That means only Apple can provide super-fast responsive handwriting recognition that doesn't go through the higher levels on the iOS.
  • If that's an example of your handwriting, and you feel it causes recognition issues, I'd be in serious doo-doo! Lol!
    But if MS and Google have implemented the feature into (some) their devices, then it seems obvious there's a want for this type of service/feature. And that then means that, yes, Apple should bring this to iOS as a system wide feature for all to use if they so choose.
    But...doesn't this also mean that we'd then have to have Apple Pencil support on all new iPhone and iPad models?
  • Not sure about the usefulness of this. For long writing, keyboard is the best tool. For short note, I think even for people walking Dictation should work very well, and possibly much faster than hand-writing too.
    I'd say forget about hand writing recognition. How about leave hand writing be hand writing. Not JPG image of hand writing but rasterized hand writing data that will stay the same as original and not degraded whatever you'll do with it.
  • I'm using Myscript Stylus as the soft keyboard within iOS and using it with the pencil. It works across apps of course and I have found the recognition to be pretty frickin good ! https://itunes.apple.com/us/app/myscript-stylus-handwriting/id931394264?...
  • Check out GoodNotes! It uses the MyScript engine but embeds recognized text in the PDF when you export, making your exported PDFs searchable. It's great. https://itunes.apple.com/us/app/goodnotes-4-notes-pdf/id778658393?mt=8&u...
  • Are you doing this using the Myscript Stylus keyboard addition as I mentioned or is there a setting native in GoodNotes that I may be missing ? Thanks
  • If you go to "Search" when viewing a document you can select a language at the bottom. That's all I had to do to get it to start recognizing. Also you can use the lasso tool to select some text and tap "Convert" to get plain text that you can share!
  • I agree for long text the keyboard is best. However it's not an option for everyone. Keyboards have been banned from our meeting rooms. ( Not happy )They said they were a distraction. So now I have to handwrite notes then go back a type them. I would love to be able to use my iPad to transcribe my notes so I only have to do them once Sent from the iMore App
  • What about goodnotes? It'll do all this and I think it's based on MyScript
  • It would be nice to have a system wide implementation that worked like a pencil on paper no matter what app, no matter what "writing tool". Meaning, I'm browsing a floor plan, zooming & panning, think of a note, all I have to do is pick up the pencil and write on it- no tool selection- just default to "pencil". I could later adjust the color, width, etc. Or preemptively select the attributes if you'd like.
    I would prefer the handwriting be searchable, not converted to type.
    I realize this will never happen. Sent from the iMore App
  • I agree with this. Keep hand writing be hand writing but searchable.
  • Think Sticky Notes, for iOS.
    They could appear where they were created within an app. For instance, in Safari, you could markup a specific website, then when you go back to that site, your markups are still present.
    Perhaps also, in the Settings app, you could draw #1, #2, etc next to the 5 button presses to get to an often used setting.
    Or within the iMore app, draw a star next to an article to remind you to check for reply comments.
    Then the notes/markups are congregated in one app like Notes, where you can manage them, or click on one to take you to that note in that app. Sent from the iMore App
  • In which case Serenity, (love the name by the way), you are hugely at odds with the way Apple are purported to behave.
  • FWIW, I'm pretty sure that Apple could ship a world-class online(*) handwriting recognition engine for iPad(s) Pro that required absolutely no cloud backend support. They might be able to do some interesting things re: syncing a user's personal handwriting training data across their devices, but that's icing. If it really came to it, I could even see a Apple implementing a specialized coprocessor used for input processing: driving down touch and Pencil latencies, offloading stroke and gesture recognition work, etc. But that falls into Apple dialing in the engineering to a very high standard, not a functional barrier. Ultimately, it all boils down to the big question of whether Apple will bother. It'll be interesting what shows at WWDC, being the first one after the Pencil's announcement and release. (*) vs offline recognition: e.g. scanning a static document, such as USPS has to do with addresses. Online is a bit easier, since you have live stroke data to work with vs.
  • They should work on handwriting recognition. But not release it until it works pretty much all the time. We don't need a "**** you handwriting recognition!" website.
  • I have to laugh when you talk about good pen input and processing power. Can you be serious? That Newton messaged had an input recognition resolution far worse than today iOS devices and was doing what it was doing on an ARM chip a decade older than the original iPhone. It's dictionary (all 90,000 words) was contained in 4Mb of RAM. There is no infra structure to create, this would be such a joke for Apple to add that it is not for any reason except respect for Steve who hated Newton because it was the dream of the guy who kicked him out of his own company. I keep holding out hope that Rosetta will rise out the the ashes, but I don't expect anything.
  • G'day, nice article Serenity. Love the pencil and I can do marvelous penmanship too which no one can decipher. Hey what a great way to make your notes etc perform privately without the trouble of encryption. lol
    My understanding is that to get what you want, then the iPad would need such a large processor that the cost would be 2-3 times what we pay now!
    Not to mention the special screen.
    Better to download to a MAC and use that processing power to do the handwriting recognition.
    One point about palm rejection is easier to implement if you use a pencil or such consistently, to get a 'palm glove' so only the fingers or the pencil touch the screen.
    Another point I notice is that 'lefties' like yourself do a terrific job of handwriting coming from the top of the paper and kind of 'PUSHING' the pen across.
    I experiment using the right hand and lettering to the left or scribbling like we do with cursive. A subject too big for this article comments section.
    I have only an appreciation and understanding for the "lefties' who battle every day.
    BTW I have a need for a really useful OCR on the iPad to convert old type writer notes to useable text. LEADTOOLS OCR for iPad, does a terrific job on my iPad 2.
    See ya. From kangaroo land.
  • What I'm looking for is something that takes my handwriting and simply makes it look more presentable. Straighten it out, clean it up, etc. I like hand written notes. I just don't like MY hand written notes because my penmanship is simply atrocious.
  • Maybe rather than "patching" the problem you should just fix it. Practice writing your letters with guides online, it's much more beneficial to be able to handwrite neatly whether it be on paper or a device
  • Might be possible by choosing a handwriting font to convert to. It won't be your handwriting, but that's your point.. your handwriting isn't pretty. Don't know if Myscript Stylus allows you to do this yet, but it seems a reasonable feature upgrade to request.
  • The my script keyboard is a great way to do handwriting recognition in any app tag you can type in. It doesn't require extra rights to run and is pretty accurate. I have what some people call chicken scratch handwriting and it works well and is easy to correct the rare occasion of mistakes. Sent from the iMore App
  • The answer is simple. Apple buys Myscript and rolls it out system wide.
  • You should really encourage Apple to take a close look at recognition in OneNote on the Surface (or any pen-enabled Windows PC). Microsoft has been working on handwriting recognition for years, and it not only accurately converts handwriting to text, but you can search handwritten notes without conversion - both features you should encourage for the iPad. I've used OneNote since 2008, with hundreds if not thousands of pages of notes, and can search for almost any word in them, despite often messy fast written scrolls.
  • Handwriting recognition is welcome but there should be Graffiti handwriting recognition available. Its the only thing that really works. I used to write down lot of passwords and model numbers for various commercial AV equipment. Recently friend of mine picked up my old Message Pad 110 and asked if a dog got it, nope, it just got lot of use. The MP2100 looks lot better even it got lot of use too. That reminds me, I need to get a serial cable for it.
  • Personally I would rather not have handwriting recognition. The greatest thing about writing with the pencil is I know my own writing. I have a much faster processor than any iPad. I remember the terrible implementation of handwriting on my HP PDA. I had to type using their graffiti in order for the PDA to understand. I already know how to write why would I need to re-learn? Back then someone wanted to borrow your notes they had to decipher your writing. I would prefer not to bog down the hardware, OS or app with handwriting recognition. Sent from the iMore App
  • For one reason, Graffiti requires only one stroke per letter, it makes difference when you write lentokonesuihkuturbiinimoottoriapumekaanikkoaliupseerioppilas or other long words.
  • Has no one else tried the Mazec keyboard. Works on iPhone and iPad and recognises and converts to text in real time. Fab! Recognised my scrawl right out of the box! I wrote this with my finger but I often use a Jot Pro in meetings!!
  • It is also based on the MyScript engine, so just additional features, but the same technology as MyScript's own keyboard. And also no direct support for Apple Pencil and iPad Pro.
  • Thanks for letting me know about this. I learned also from 'gewappnet' that it's powered by MyScript Stylus which I do use now and is free. Question for me is does it provide a sufficiently superior user experience to pay for it. MyScript Stylus does not recognize when I enter numbers and it does not recognize the ampersand. I also miss real predictive text., all MyScript Stylus does is give me three options of what I just wrote might be; and that is only word by word. I was disappointed to see no reviews for the app on the app store. Looks like I may have to try it to find out.
  • I can see where this would be useful. However, I have gotten to the point that I never write anymore and the last thing I did actually started making my hand hurt! Strangely, I can draw for hours with no problems.
  • This is the only way I would buy an Apple Pencil. I love the technology and the use of it, but i can't draw and i don't think i would benefit from the Pencil. Since i love to write more than type, something like this would make me buy the new 9.7 inch iPad Pro with a Pencil. Until then, I will probably just buy it to say i have it.
  • I love the idea of writing cursively on my iPad which I'm doing now. For me it seems a natural fit with how I hold and use my iPad. The third-party app MyScript Stylus thankfully allows me to use mypencil in any app not just notes and while it's made the Pro and Pencil a great experience I'm convinced baking handwriting into ios itself would be an amazing user experience that would appeal to millions of older users who grew up with pen and paper and never got really fast with a keyboard. Can see lots of opportunities to sell all sorts of designer pens as well. Just think how much easier it would beto edit spreadsheets if you could navigate to a cell with your finger touch the cell and enter or edit info right in the cell itself with your Pencil. And that's just one advantage of baking handwriting right into ios.
  • I used to own a Nexus. And I absolutely loved the Google handwriting input. I have tried my stacks on iPhone. Google handwriting app blows it out of the water. Also where I find an application for handwriting input is when I input in non-English languages. Just write and it worked! Magic!! I hope Apple is listening or Google for that matter.