Let’s talk about text input
June 26, 2007
Hello everyone. Silence means busy – and busy is of course good, in many ways. We’re busy with a lot of new stuff. š
Personally I’ve also been busy with writing my master’s thesis document. And hooray: it’s finally done. Since the subject material is closely tied to the Maemo UI, I thought I’d post it about here. Basically the thesis deals with the design and principles behind the text input methods, more specifically the virtual keyboard, designed for the Nokia 770 and subsequent products. It discusses the topic of touch screen text input both from theoretical and practical sides.
It can be found from the Maemo site, here:
http://maemo.org/intro/presentations/text_input_on_mobile_devices-master-s_thesis_document.html
If you get any ideas, questions, rants/raves from reading this thesis, I would be happy to hear them and discuss. Additionally, if you have any thoughts you would like to share about the current touch screen text input methods found in the devices – irrespective of this thesis – I’d also like to discuss about those. This includes at least the virtual keyboard, handwriting recognition, full screen keyboard and the bluetooth keyboard.
Best regards,
Roope Rainisto
Maemo UI design 
Let’s not reinvent the wheel.
First of all, virtual keyboards are kludges and allways will be, until someone invents a way to give good tactile feedback from a virtual keyboard. But as far as those kludges go, Nokia’s fullscreen virtual thumbboard for the Internet Tablet ain’t half bad: I can actually manage to type recognizeable words with it.
Secondly, and however, Nokia’s “handwriting” “recognition” on the same Internet Tablet blows big chunks. It appears to have managed to incorporate all the bad points of Palm’s Graffiti and Jot (only cursive writing, very bad at recognizing shapes, extremely intolerant about the space between letters and dependant on a popup window on an already crowded screen) and obviously didn’t bother to get any of the (scarse) good points of those stroke recognizers.
It should be quite clear by now that the Internet Tablet is in dire need of a better HWR system, but rather than starting from scratch, Nokia should lobby with the people who know how to do it right. PhatWare Produces both Calligrapher and PenOffice, which are extremly capable products on top of ParaGraph, just about the best HWR engine in existence today (the fact that it is a direct descendant of the latter day Newton’s HWR engine is no coincidence).
Until now, PhatWare has only produced Windows products and AFAIK no user complaints have been able to persuade them to do otherwise (or maybe I’m the only one who bothers to send them angry emails. Who knows?), but, given that the original software came from a platform (Newton) that could not be more different from Windows, a port to Linux doesn’t seem to be unreasonable to assume.
I am pretty sure many people will be quite happy to fork over their shekels to PhatWare, if they’re given an opportunity to play with their HWR for a spell — I know I did — but it will take some major corporate convincing, kissing up and twisting arms to make them — well: make it.
How’s that for a rant then?
Correction:
That “cursive writing” above should obviously have been “printed writing”.
Danged Happy Hour…
Hi Karel, thanks for the comment. The handwriting is an interesting issue indeed. There is potential for much improvement, both from the engine perspective and also from the UI side.
Of the specific engine you mentioned, there currently isn’t a Linux implementation available. As an interaction designer I really cannot comment on how hard or easy it would be for them to provide this engine. Naturally we are eager to evaluate all available alternatives, but that particular one isn’t currently yet technically an alternative for us.
The text input method framework we are providing does also, with modest effort, allow 3rd party implementations of input methods. Companies could create their own input methods. Then again, I really can’t go into the business sides of these decisions. Pricing is an issue with all licensing. We as an open source platform have a strong desire to not rely on commercial proprietary elements, rather on open source components, whenever an open source alternative exists that would provide a good user experience. (Then again, for instance currently this is not the situation with the handwriting engine.)
Handwriting in general, considering the landscape of all touch screen mobile devices, hasn’t really made significant breakthroughs insofar that … At least I cannot really name any current popular major device that would provide a handwriting experience that the majority of users would actually want to use, would choose to use over the other methods that are available. (Even if considering some of the rather hyped up-and-coming competitor products – no handwriting at all… š
I personally think that such facts are a signal of _something_. For a feature that has existed for a long time, if it would be really the greatest thing since sliced bread, it would have already gained more popularity. Is is because the technology is not yet ready? It is too expensive? The majority of users are not ready to input text with their handwriting? There are faster input methods available? Companies just don’t “get it”? All of the above, or none of the above?
I agree that the state of the art is that real time “as naturally written” recognition is unreliable and too resource intensive. Two compromise techniques have gained enough adherents that they are widely known, practiced, and actually liked by some. These are “minikeyboards” and Graffiti.
“Virtual keyboards” are widely hated. Fitaly is indeed more efficient in time and in space, but it has not gathered enough adherents to be widely known.
Given that view, the course of action seems obvious.
(i) Implement the best compromise solutions, Graffiti (or another unistroke method which is consistent with low resource usage) and a minikeyboard, and use them while
(ii) Start a research project to make “as written” recognition work.
Luckily the n810 and n900 already have a minikeyboard.
my 2 cents
I have used 2 Newton Messagepads (model 2100) for five years as my only portable computers. I also bought a keyboard and combination carrying case for them, but I never used the keyboard for more than a handful of times.
Mind you, I was writing actual articles and stories on my Newtons at the time, often during long train commutes, and still the Newton HWR gave me 99+% of correct recognition (even in Dutch, with a much more restricted dictionary I came close to 99%); and that was exclusively writing in cursive.
I’ll readily admit that I can type much faster than I can write (I’m a touch typist, albeit a renegade one), but that’s not the point. The point is that I could write much faster with Newton’s HWR than I could input text with any of the “alternative” input methodes I’ve tried (and Bog knows I’ve tried a lot of them). The trick with good HWR is that you have to write as if for someone else to read it (which is essentially the case anyway). A minute effort to produce readable penmanship means a quantum leap in better recognition.
In my opinion, HWR is still suffering from the “doonesbury” effect (that guy should be drawn and quartered!) and is perceived as less than it actually is. Isn’t it telling that the donnesbury guy never made a cartoon about the HWR on NewtonOS 2.1? Since it just worked, it couldn’t possibly be funny, obviously.
I did a few mock-up tests on my N800, using a drawing program in portrait mode, and ergonomically at least, the N800’s screen is right in the money for cursive notetaking: easy to hold, just small enough to avoid resting one’s hand on the screen (and thus creating the infamous “jaggies”) and with a resolution that makes it very easy to produce well-formed input. Even the N800’s stylus is not too uncomfortable to hold for prolonged periods. All the tablet really needs, is the software.
As for the argument that none of the newer devices include HWR, just this one question: How many Internet Tablets were there before Nokia came out with the 770? You get my point?
I’ve written a small article with my thoughts on improved onscreen keyboarding, with reference to the iPod Touch keyboard (see link). My main thoughts were that keyboards should learn about their user and make allowances, along with using a vector-based word-recognition system which makes intelligent guesses at what you meant to write.
I find the n800 keyboard a bit tricky to use – not because the buttons are too small, but because I just miss the keys and it doesn’t predict the words I’m trying to type.
My biggest gripe is how many times you have to press the thing with your thumb flat on the screen to get the thing to pop up in the first place…
Phil
Hi Phil, thanks for your comments. I checked your link. The process you’re describing for prediction is quite valid (of looking at a radius around the actual keypress) – although it is already included in some of the commercially available engines. I think even the engine in the iPhone claims to that behaviour.
Also adapting the keyboard shapes visually on screen … I think I referenced that even somewhere in my master’s thesis. It is potentially interesting, then again the overall visuals very quickly get disjointed if you overdo it. And since errors cumulate, if the users tend to hit the left edge of a button, and the left edge keeps moving leftwards, the error can easily keep growing.
The larger argument is over prediction vs. non-prediction. Prediction as default isn’t always better. (There’s a reason for instance why PC keyboards have no prediction.) Naturally a small touch screen device doesn’t reach that threshold yet, but…
After a certain point, prediction doesn’t really make text input -faster-. It changes it substantially, by reducing a lot of errors, and creating some new errors, by the engine predicting something that the user didn’t really meant to input. Which then adds an extra verification-comprehension step for the user, of always checking whether the engine “didn’t do anything funny with my input”. This extra step then takes time and mental effort. Take any web2.0 service name, try entering their names with a predictive engine. š
Then again, this doesn’t mean that predictive engines wouldn’t be of any interest to us at all, but more that the overall set of gains vs. losses isn’t perhaps as clear as one might assume when first approaching this subject.
For the thumb detection, that is one issue we are working on to improve the touch screen response and sensitivity.
I came across to MaemoCJK project. Does Nokia have something “official” coming out regarding CJK or is that it? Maybe you can not comment though.
I have a Palm Tungsten T3, now four years old, that I have finally decided to replace with an N810. When I got the T3, which was my first PDA, I tried the virtual QWERTY keyboard. Seeing that this was a poor layout for tapping, I immediately set out to see if someone had created a tap-optimized layout. There was: it is called Fitaly (after the letters on one row). I installed it and practiced it a little bit, and fairly quickly got to the point where I could use it for note-taking in meetings — somewhere between 1/3 and 1/2 my touch-typing speed, I think, easily faster even than with pen on paper, and way faster than with Graffiti. (Admittedly, the error rate tended to be substantially higher than for touch-typing, but for note-taking this wasn’t a problem.)
Of course, the obvious downside of something like Fitaly, is that it’s unfamiliar, and it does take a bit of practice to feel facile with it. Still, the amount of improvement available compared to the amount of practice required seems to me a good trade. I think it greatly expanded what I could do with the T3, and without the hassle of an external keyboard.
I just got my N810 yesterday, so I’m still getting to know it, but I did try today typing some extended text with the slide-out keyboard. It works okay, certainly. I don’t think it’s any faster than I was with Fitaly on the Palm, though, and maybe a bit slower. I seem to mostly use my left thumb on the left three or four columns of keys, and my right forefinger for most everything else. I do notice that it feels rather busy, in that I have to move my hands around the keyboard a lot; it seems like it would be more distracting to others than just tapping on the screen with a stylus. Really, I’d just as soon have Fitaly back, or some similar layout (there’s another one called Opti II that may be slightly better).
But I also think that a different layout, one optimized for two-thumb typing with the frequently used letters on the sides, could work a lot better. Once again, there’s a lot of ultimate performance given up for the familiarity of QWERTY.
Anyway, the point of all this is, I wish manufacturers would realize that there are some of us out here, contrary to conventional wisdom, who are willing to invest a little time in learning a new text input skill for a handheld device if it’s something we anticipate using a lot and the new method is significantly easier and/or faster once one gets good at it. So for instance, if they offered something like the N810 with an alternative, thumb-optimized layout, there are some of us would buy it that way.
I’ve recently bought a N810 and although I generally love the device I’m very disappointed with all the text input methods. I find that the slide out keyboard is awkward to use (compare with a Psion Revo of around 10 years ago which was a pleasure).
My biggest gripe is the handwriting recognition, I have used several devices including a Palm Pilot, Compaq Ipaq with the MS recognition tool, a Sony Ericsson P910 and all were usuable. The P910 was the best and I could get up quite a speed even on the tiny screen. So 5 years on, Nokia have produced the worst system that I’ve ever used. It is accurate (generally) but painfully slow, sometimes it won’t even recognise that you’ve started to write causing you to repeat your stroke several times and other times it lets you get a couple of letters ahead only to forget the earlier ones and cause you to rub out and start again. It’s dreadful.
This machine has a 400MHz CPU, many times the power of those earlier machines and yet it can’t keep up whereas they could. Come on Nokia, commission someone to write a decent input program – it can be done !
All we have at the moment is the on screen keyboard which is reasonable but a slow way to input text.
Graffiti solves this problem very nicely.
It’s stable, low resource utilization, trivial to learn, trivial to come back to after you haven’t used it for a while. Why reinvent the flat tire with unreliable high resource utilization schemes, virtual keyboards?
I have a simple equation covering 4000 years in both (Gregory& Julian):
Year,Date / Month, Name of the day& the seven Music keys (Do Re Mi etc..)indicating the Days sequence + the billiard colours values indicating the dates value is there any intrest in developping such idea
My experience with Palm and PPC are that the “Graffiti” style block text entry wasn’t brilliant, but importantly, it was not frustrating. The N800 screen keyboard was slow and inaccurate, HWR was vile. The screen keyboard on the N900 is better than the fiddly little mechanical keyboard, although both are accurate enough for passwords and similar, but neither are fast. My eyesight is aging, and the screen keyboard is bigger and better illuminated for me. Graffiti is accurate, fairly fast and reliable, it’s a reasonable compromise, and it really isn’t hard to learn – coupla days. It would be a significant programming job for the N900, but no extra hardware would be needed! I can’t program, but I’m a hardware type. Colin