Portfolio

The E-Commerce Voice Assistant

{"name"=>"DIVA • Design & Management"} — Sat, 01 Jul 2017 00:00:00 +0000

Challenge:
A small remote team using the company's speech recognition software and NLU technology to build DIVA, a Siri-like e-commerce system, from scratch.

Approach:
Build, test, learn, build, freak out, learn, persevere, learn,…

Contribution:
Product Designer & Product Manager: Trying to keep a balance between my inner drive for perfection as a designer and my pragmatism as a product manager.

Year: 2016 - 2018

Situation

AppTek is a speech recognition company that licenses its speech recognition engine to its clients, tailored to their needs. AppTek is a strong believer that voice will be the primary interface for consumer computing within the next few years. So the company decided to offer a voice interface platform for e-commerce companies powered by their speech recognition and natural language understanding technologies. The outcome should be a platform that would make it possible for other e-commerce companies to offer voice-based services in their interfaces (web, mobile). To showcase the capabilities and to develop the platform, we built a shopping assistant called DIVA (Digital Intelligent Voice Assistant).

Challenge

The remote team in 3 different timezones

The input I received was, “Think of HAL 9000 or Jarvis and make that for e-commerce.” HAL 9000 and Jarvis are fictional intelligent assistants from science fiction movies. The theoretical possibilities were endless. The previous products that AppTek built were more B2B applications. The company’s own speech recognition was just recently enabled to stream the transcription back via the internet, which made the voice assistant possible in the first place. The team was scattered around the world (US East & West Coast & Germany). Most of the team members were only partially dedicated to the product. As the new product manager and designer, I only had the chance to meet part of the team in person, and I had no prior experience working in the field of AI or voice interfaces.

Our Team

The DIVA project team consisted of 7 members.

1 × iOS engineer
2 × backend developers
1 × speech recognition scientist
1 × NLU scientist
1 × QA manager
1 × me

Organizing the Team

To keep the team in sync, we met twice a week in a telephone conference and once per week in person with all the team members on the US West Coast. To avoid filling up our email inbox, we chose Slack. The team was already using JIRA. Since it integrated well with other Atlassian software, we chose Confluence for documentation.

Creating the Vision

The first thing was to figure out what we wanted to build. This was done in the first phase of the project and is described in the post: “Creating a Vision & Telling a Story”.

Keep Developers Closest

There is a saying made popular in the Godfather movies: “Keep your friends close, and your enemies closer!” and I would add: “and your developer closest.” Computer engineering is a profession that automates things and renders away redundancy. Designers generally can generate their designs quickly, and are therefore more open to changing things once they see they don’t work. Those changes for the developer often mean “redo the work you already did,” which can be annoying. Though it’s good to embrace a culture of change, that culture shouldn’t be stretched. To avoid that as much as possible, my goal is always to integrate the developer early in the process and to prepare the work toward him/her. I also like to have direct access to the code via a repository setup. This way I can do small incremental changes to text, animations or colors myself without bothering the developer. That also shows empathy from the designer toward the coder, and can generate a better bond between the two.

First Working Prototype

A simple prototype to tell the story

The first click prototype was designed for the user story. By building that interactive prototype, we had a couple of screens, colors and a design direction that we had checked out. The next step was to make the prototype real. As quickly as possible. To make a prototype app with a working speech recognition and language understanding, the team had to gather training data from the internet to create the needed models for interpreting the search queries and to improve the speech recognition quality. We decided that at the beginning we wouldn’t connect to an e-commerce API, but build our own product database connected to a Lucene search. That way we could act more quickly and were not dependent on other infrastructures. Meanwhile, the iOS engineer and I teamed up to bring the design prototype to life. After three weeks, we had the first working prototype in place that searched on our product database.

First functional prototype after 3 weeks

What you can see in this video is that the speech recognition isn’t very accurate. The reason for that is the improvements to the speech recognition weren’t in place yet.

Speed

When the tech was in place, we saw how slow our speech recognition was. There were several reasons for that. If you want to read how we tackled that problem, read: “Team, We Are Faster than Siri!”

Highlight what you Understand

For an AI based on voice recognition, two technologies are important:

If you say: I wanna buy … running shoes

An automatic speech recognition (ASR) service that transcribes what was just said.
- Text: I want to buy running shoes
A natural language understanding service (NLU) that extracts the correct intent and understands it.
- intent: buy
- object: running shoes

Some voice assistants don’t display the transcribed text at all, and just extract the relevant word.

Other assistants – like Hound, Siri, Google Assistant and Cortana – do transcribe the text.

We felt that showing the text is better than hiding it. Showing the text is a signal to the user, “I’m listening to what you are saying.” To emphasize that signal, we wanted to add another form of visualization. Therefore, we colored the intent and the object green, showing the user “Yes, we understood exactly what you want to do and what you are searching for.”

Highlighting the object in green

Later in the project, we also decided to push the color-coding paradigm further, and gave the refinement another color. That way the object and the refinement would be distinguished visually.

Highlighting the refinement (Feature is partially integrated in Downloadable App)

Improving the Speech Recognition

When we started with DIVA, the speech recognition models in use at the time lacked the optimization toward e-commerce. Speech recognition is a complex problem. Baidu, which updated its Chinese speech recognition in 2016, had 70 computer scientists working for two years on that particular problem. The project DIVA had one speech recognition scientist.

To improve the speech recognition, the scientist needs correctly transcribed audio data. We were recording audio data with each search query that was done with DIVA. To make the correction simple, I suggested using the iOS device directly. When something wasn’t transcribed correctly, the user could edit the sentence and send it back immediately.

Audio Correction Portal

This feature was the simplest way to generate corrected data immediately. That approach was okay for the start, but not a way to scale the process. Often people did not bother to correct what was said. The audio data needed to be transcribed. Therefore, we built an internal corrections system for the search queries. The priority for building the “Correction Portal” was a short time of development and a decent usability.

A simple portal to correct the audio utterances

Quality Assurance & Data Collection

To improve our recognition quality and our NLU, I worked with a remote quality assurance team on the US East Coast. We had several goals:

Use DIVA to search for products and refine the search, so we can draw conclusions about how people would use voice in e-commerce.
Check the quality of the speech recognition, and whether the intent was recognized.
Research e-commerce data (such as brands, colors, etc.) for the speech recognition vocabulary. Our QA team member would join one telephone conference call to tell the team how we were performing.

Working close together with QA team

Test in the Wild

I believe that a product benefits the most when all team members participate in usability tests and see what works and what doesn’t. Jared Spool, who has decades of experience in the field, states in his article Fast Path to a Great UX – Increased Exposure Hours

“Exposure hours. The number of hours each team member is exposed directly to real users interacting with the team’s designs or the team’s competitor’s designs. There is a direct correlation between this exposure and the improvements we see in the designs that team produces.”

With a remote team and limited resources, this was hard to implement. To give the team at least a bit of usability testing feedback, I tested DIVA in the wild. Whenever I went to a conference or meetup in the Silicon Valley, I demoed DIVA and asked people to try it out in front of me. Though most team members didn’t have any experience with usability testing, I asked them to do the same. This was not a structured usability test with researchers, but the feedback was very valuable. We learned a lot of things from these small exposure times.

People hold the mic button pressed when they speak. Siri does support that behavior. The Google Assistant doesn’t. We didn’t realize we should support that behavior also, until we figured out that we should.
People didn’t always recognize the green dot as a mic button. First, we added a little “Tap to Speak” tip that would be shown if no action was taken. Later, we added an icon to the button. It was especially important when we added image recognition capabilities to DIVA.

On the left is the old and on the right the improved start screen

Hands-Free Orders

One of our premises was to make DIVA a hands-free interface. You should be able to talk to it, “Okay, I want to buy it” or “Please go back” or “Show me the first one.” To implement that functionality, we programmed our NLU to interpret these orders and send them back to the client. The idea was that DIVA could be an interface in a TV, or the user could be in a situation where she only could speak and couldn’t touch the device.

Small numbers on the images let you choose the product via voice

Here you can see that we included numbers on the product list view. Now, would you want to have an ever-ongoing numbering for the products that you are showing? “Show me the 33rd one”? That’s not the best idea, and it would make the recognition more complicated. Together with the iOS engineer, we thought of just displaying the numbers when the user isn’t scrolling. And in this case, we could limit it to the number of products that are shown on the interface. The drawback of this approach was that the product list always needed to lock in, meaning if the user would display only half a product, the list would scroll up or down automatically. It worked. Later, we took that functionality out for the DIVA iOS client because people had DIVA in their hands, and the hands-free situation didn’t occur too often.

Thumb Orientation leads to Certain Design Decisions

The DIVA client is an app on a handheld device. If you are a designer, you might have seen this graphical representation from Luke W’s article “Designing for Large Screen Smartphones”, which shows that we interact mainly with our thumbs. The better an app is designed toward the thumbs, the better the usability is.

One of our features was the filter bar. The idea is that the attributes for a product come from the NLU knowledge base. In its functionality, it is similar to the filters that are shown in e-commerce apps. Filters are normally placed in the upper right corner.

Features like filters in apps are visually present

We thought that this interaction paradigm wasn’t the best solution for a thumb-oriented design. Therefore, we developed the filter bar. It was inspired by the pill-shaped buttons that were shown in other chat-based bot applications. We developed it a step further, making it possible to easily access more than just three or four options.

Screen recording showing the filter bar

Designing & Workflow

To organize my thoughts I use post it notes.

Post it’s are a designers best friend :)

To design the interfaces I use Sketch.

Sketch rocks!

To communicate the design specifications, I used a very simple annotation style for the developer.

Initial way to communicate measures with the developer

Later, we implemented the tool Zeplin in our workflow, which made the process better for everybody. You just upload specific designs directly from Sketch and the dimensions and images can be downloaded directly from Zeplin.

Later Zeplin made the workflow much smoother

Prototyping

There is no way of checking out if an interaction works well other than prototyping it. The quicker you can experience the interaction, the faster you can decide if it is a good one. For creating flows or testing out interactions I was using Flinto. I also used other prototyping software such as Marvel, Principle, or Framer but stuck with Flinto for most cases because it was able to generate the quickest output that I needed.

Overview of an interactive prototype built with Flinto

The Dirty Work

In every project there is the dirty work that needs to be done. Things that are tedious and take time. We had a couple of those tasks. Whenever it was possible, I coordinated with people who outsourced that work. But when it wasn’t possible, I did it myself.

Transcribing: Approving and correcting recorded audio utterances.
Correcting computer-generated phonemes: For example, if you say “USA,” you actually say “YOU ESS AY.” If you give the word to a computer to generate the phoneme automatically, it might generate it as a word (USAH), and not as single letters. Those computer-generated phonemes needed to be rechecked to improve the speech recognition.
Cleaning data: When data is scraped for training models it is dirty, especially when the data is user-generated. This means there is data you don’t want to have there. Between pictures of laptops, you may find boxes of laptops. The data needs to be cleaned up.

Speechtek 2016

In May 2016, we went to the Speech Technology Conference “Speechtek”. It was a good time to show what we were working on. Here are the slides that I presented.

Presentation of DIVA at Speechtek 2016

The Launch

Launch of DIVA after eight months

After eight months, we launched DIVA on the Apple AppStore in an Alpha stage. To give users a real benefit in trying out DIVA, we connected it with the eBay API and later with the Amazon API. Then users could search for a product, see it, and then open it in the web browser. The integration was a challenge in itself. Often we had to figure out how we had to structure the API calls to get the data we wanted. In some cases, we were not able to obtain the data in the way we needed it. Nevertheless, we wanted to get our product out in the wild and have people test it and play around with it. The initial idea was to make a purchase possible via voice. That wasn’t possible since the eBay API integration for that was too time-consuming.

Custom Apps

To demo the technology, we generated custom apps. DIVA’s design was simple and easy, skinnable. This enabled us to generate these apps with custom data in a very short amount of time.

Image Recognition

Simple user flow for image recognition

Having the application in the wild, we wanted to implement image recognition as another feature. The idea was that people could search for products by making a photo. While the team was gathering training data and building the models, I researched similar services and apps that offered a visual recognition service. I also checked the user flow of photo apps for taking a photo. With the best from both worlds, I designed a simple flow and added a camera button to the start page. When the prototype looked good and the image recognition technology was in place, we implemented the technology in the iOS client. Although we knew that the recognition technology needed to be improved, we wanted to learn how people would use such a feature.

Image recognition screen recording

Retrospect and Outlook

DIVA is a bold move for a small company like AppTek. Myself as an individual, we as a team, and the company as a whole, are learning a lot. With voice interaction becoming more and more important in the future, AppTek is showing that its technology is able to be part of a voice assistant. For a small company, it is an amazing start in that promising space.

Creating a Vision & Telling a Story

{"name"=>"DIVA • Storytelling"} — Mon, 01 May 2017 00:00:00 +0000

Challenge:
How to create a vision for a product and communicate it effectively.

Approach:
Tell them a story that allows them to visualize the process.

Contribution:
The storyteller and visualizer.

Year: 2016

Situation

We were at the very beginning of the DIVA our Digital Intelligent Voice Assistant. The task was to build a voice assistant for e-commerce. My entire career has taught me, again and again, “communication is key.” I knew I had to find a way to create a vision and communicate it to the team so everybody understood our goal. It needed to be simple, so that even non-technical people would grasp it.

Research

To understand the possibilities of intelligent assistants and voice recognition, I read many articles and downloaded every possible application to see how far things had been pushed. I completed several product purchases with these apps, and documented the process.

Documenting the buying process in a text based ecommerce assistant

The product arrived at home

Documenting the return process

With Alexa in my home and Siri on my phone, I tried as many interactions as possible. At the end of the research phase I had a good understanding of the landscape and the approaches that were used.

Experiencing voice interfaces with Alexa

Part of the research was to introduce my three-year-old daughter to voice assistants. Seeing how quickly she was able to search for specific videos without being able to write was amazing. This is when I truly grasped how important voice will be for the future of human-computer-interaction.

My three-year-old daughter interacting with Siri

Our Vision Statement

While doing the research, I already was working on the vision. Since the team was scattered around and focused on other tasks, it was difficult to integrate them into process. The general idea I have regarding interfaces is they are just a glass pane. We interact with computers and smartphones through different tools (mouse, keyboard and touch) just to have information displayed on the glass. The addition of voice recognition and language understanding would make the “glass” much faster. It all boils down to the following vision statement.

To communicate the vision, I came up with the idea of using an analogy of a salesman standing behind the glass, where the glass symbolizes the device’s screen. Here’s the email that I wrote to the team:

"There is a very good salesman. You know, those guys who can sell you exactly what you need. As if they could read your mind. We all have dealt with those guys. The problem with this salesman is he is standing behind a wall made out of very thick and durable frosted glass. He can’t see who is on the other side, but he can write on it and people can read his messages. So, he writes, "I sell things.” The only thing he is able to do is make a tiny slot in the glass. So, people write their orders, such as “nice jeans” or “hiking boots” on pieces of paper and slide them through the slot. The people can’t see or hear the salesman, though. They only can see the pictures that shows up on the glass wall, along with some descriptions he writes. They tap on the things they want. The salesman can see their taps. Then he holds a sign next to the picture: “I want to buy this” So, people tap on that then write down their address on a piece of paper and slide it through the slot. He takes it and let the postman deliver the order to the customer. It is a very complicated way of selling and buying stuff, but it works.

So then there comes a glass maker. He sees that and thinks: I can help the salesman by inventing better glass. So, he invents new glass and calls it DIVA.

DIVA's features are:
- The glass has a hole, so people can submit their orders via voice.
- The glass is semitransparent, so the salesman can see his customers.

Now the world of this salesman changes dramatically. He can see and hear his customers. People tell him, "I want boots for my daughter." So, he shows them the things they want. He also memorizes what people say when they buy their things, just like a good salesman.

This is what DIVA is. It is the new interface for the world of sales. People can talk with it, tap on the things they want, or just say them out loud. DIVA is so fast and accurate that people are amazed by it. They visit more often since buying things is so easy.

The feedback I received on the story was very good. “Wow! the best piece I read about a voice interface. Genius!!!!” The story was then also shared with partners to tell them about the product that we were working on.

The First Prototype

The vision statement was the North Star. The story was our binoculars to see the North Star more clearly. Now we needed something more tangible to know what we would try to build. The best way to do that was to make the product real. My starting point for designing interfaces is (if possible) to look at what is out there and learn from it. See the good things and the not so good things. You immediately can test interaction patterns without building them on your own. The next step was designing an interface. The goal wasn’t to have perfect usability, but an interface to tell a story. For a new project it is also a design exercise because the first version of the design language can be explored in a very simple way.

So, I started and designed the very first version of DIVA. I shared the first designs with the iOS engineer. It is always helpful to get feedback from peers, four eyes see more than two. For the first designs, I used Marvel to make them interactive and easily sharable.

Our User Story told with Personas

But the prototype was not enough, and didn’t tell the whole story. To show the benefits of such an interface, I used personas and embedded the interface in a real life situation. That way I also could show potential AI capabilities and discuss with the team what is possible for us. I extracted the features and highlighted what the AI would need to accomplish to deliver that experience.

My presentation communicating the vision with personas

Retrospect

The visualized story was used extensively, internally, and with external business partners. I later extended the story to other scenarios. It served us very well and gave us all a shared understanding of what we wanted to build.

Learning Bit by Bit

{"name"=>"DIVA • Interaction Design"} — Sat, 01 Apr 2017 00:00:00 +0000

Challenge:
Users don't know which kind of voice commands they can use.

Approach:
Design workshop with the engineer and implementation.

Contribution:
Researcher, Workshop Organizer, Product Designer

Year: 2017

The Problem

When we showed DIVA to others, we always needed to show the features that DIVA was capable of doing, such as “Search for price range” or ”Search for attributes”. The users said, “Ah, you can do that”.

No hints that teach the user what she can say

In visual interfaces, features that are important can be consistently shown. The user sees the feature. There is an external signal that reminds him of it. For example, if you search in a e-commerce website, you can see the filters, or when you type the keywords, suggestions will be shown, with autocomplete for each new letter that you type. Features of voice interfaces lack that ability. The question is how can the user be shown what capabilities the NLU has without making tours that everybody dismisses and nobody watches.

Workshop

DIVA has a display and is not a voice-only interface. There are quite a few similar voice assistants out there. All of them have the same problem and take slightly different approaches to solving it. Since this was a good problem that can be discussed in a workshop, the iOS engineer and I teamed up together.

First, I collected screenshots from all the other assistants and printed them out:

Cortana, Hound, Siri, Google, Ozlo, Alexa

We went through all the different solutions and discussed the pros and cons of each. We figured out the interaction patterns that were used in all solutions and decided together what could work for our case, or if we could come up with a better approach. We preferred when the information was minimal and not in a list form.

Cortana uses small hints here and there

Another thing we liked was when a skill was visualized as an icon.

Icons visualize skills better than just text

We felt that giving the user the information in small chunks of teaching would be the best approach. We had two opportunities to do that, the start screen and the product list view screen. We sketched on the printed-out screens how and where the visual-nuggets could be placed and what we would want to teach the user.

Screens where we could show the small nuggets

On our start screen we already had one example with a picture. We thought of having different visual examples that show different voice commands. And on our product list we could place small little examples on the bottom of the page.

Design & Prototype

I went straight to Sketch with the ideas and came up with different solutions. There was a back and forth between me and the engineer in Slack. The start screen looked like this:

V1: The high res pictures were too distracting

The start screen with the high-res images was too distracting and too concrete. It felt like advertising and the speak button wasn’t as prominent as before. So, I changed the images to very simple icons that I found on thenounproject.com. That way they looked more like examples of searches. To show the variety we used different examples with different text attributes, searching for color or price that changed randomly.

V2: Simple icons worked better as examples

The other opportunity to teach the user was the product list screen. To give the text-nuggets more space, I placed the microphone button on the right side. But looking at it together it didn’t feel right. So, I worked on making the text shorter and placing the microphone button in the middle.

The second version felt like the better approach. We also had ideas to make the suggestions relevant to the first question. Implementing this “intelligence” would be possible, but would cost us more time and resources from other engineers. Therefore, we decided to push that update through and implement the intelligent part later.

Retrospect

By doing a quick design workshop with the developer we were able to look at the problem as a team. Analyzing the problem together is not just more fun but can help to see the problem through each others eyes. Though we didn’t have the chance to test the results in user tests we felt that the solution was an improvement to what we had before.

“Team, We Are Faster than Siri!”

{"name"=>"DIVA • Design & Management"} — Sat, 01 Oct 2016 00:00:00 +0000

Challenge:
Our speech recognition takes up to 3 seconds or more until a word appears on the screen.

Approach:
Impress upon engineers why this is too slow and how we can make it faster together.

Contribution:
Product Designer and Product Manager

Year: 2016

Situation

When AppTek speech recognition technology debuted, it recorded the audio data, sent it to the servers, and then sent it back to the device as text. The ability to send text back while still recording / speaking was one of the relatively new features that made a voice assistant like DIVA possible in the first place. The crux was that it took 2.7 seconds or more until the uttered word was displayed on the device, which made the interface feel sluggish.

Early recording shows lag between utterance and display of the word

Especially when it was compared to other voice recognition systems. Back then the team was not aware that this might be too slow for the user. It “worked” and there were other problems the team needed to solve.

Approach

First, we had to understand as a team that this was a real issue. As Jakob Nielsen writes in his article Response Times about human-computer interaction, three time limits are crucial, and every interaction designer should know them:

0.1 second: Is about the limit for having the user feel that the system is reacting instantaneously.
1.0 second: Is about the limit for the user’s flow of thought to stay uninterrupted, even though the user will notice the delay.
10 seconds: Is about the limit for keeping the user’s attention focused on the dialogue.

The closer a response is to the 0.1 barrier, the more instantaneous it feels. The further away a response is from the 1.0 barrier, the more sluggish it will feel.

Everybody wants to be happy

After sharing this fact, I needed data points to show how slow the recognition was. I needed to come up with a simple and pragmatic way to do that without stressing out developers and binding their resources. So, I recorded the speech recognition with screen recording software, or I just took a video of two or three smartphones with DIVA and other voice recognition software next to each other.

Siri vs. Google vs. DIVA

Then I replayed the recording and counted the frames between the uttering of word and the word being displayed on the screen. Then I just needed to calculate the seconds based on the fps of the video file. I would do that with two different sentences every now and then. All competitive voice recognition software was faster than 1.0 second, with Google being the fastest. DIVA with up to 3 seconds was way behind. Fixing this was not an easy task. There are different systems that work together and different reasons why it was taking longer for DIVA.

Teamwork

Every time I did the test I shared the results in our weekly meetings and asked the team what we could do about it. The engineers came up with ideas. After those were deployed, I tested again and shared the results. It was important to keep working on the issue continuously. It is only when a problem haunts you that you think about solutions. To ensure this was the case, the team needed to be made aware of the issue. The process went on for nine months. Not every change we deployed made it better. But every change meant we were trying, and gave us a new way of looking at things.

DIVA shows outputs numbers for debugging the speech recognition

The engineers also would come up with ideas for simple visual indicators that the system was working or for debugging techniques so the user could see on the screen when something was stuck.

350% Faster

Nine months later we broke the 1.0 second barrier. It took DIVA only 0.6 seconds until an uttered word was displayed on the device. That was 350% faster than when we started (2.7 seconds). We were even faster than Apple Siri, and nearly as fast as Google speech recognition. This was a big achievement for the team.

Nine months later we were faster than Siri

Retrospect

Design is not how it looks, but how it works. And only close teamwork between engineering and design can make the dream work.

Flipkiss - A Game about Love

{"name"=>"Sideproject • Game Design"} — Sat, 01 Aug 2015 00:00:00 +0000

Challenge:
The idea was to create a small but different kind of matching game.

Approach:
Cuteness, a lovely story, an innovative matching game design and a social cause.

Contribution:
Executive Producer & Game Designer

Year: 2015 - Present

Summary

With a team of five I invented and launched a small but cute matching iOS game. Flipkiss was featured by Apple for “New Games We Love” in several countries like the USA, Germany, the United Kingdom and Spain. The game is connected to a social cause. With Flipkiss I wanted to raise awareness and funds for research for people affected by Lupus, an incurable autoimmune disease.

Flipkiss launched in October 2018

Team

Our team was composed of:

1 × 👨‍💻 iOS engineer (Germany 🇩🇪)
1 × 👩‍🎨 Illustrator (UK 🇬🇧)
1 × 👩‍🎤 Sound Designer (France 🇫🇷)
1 × 👨‍🍳 Music Composer (France 🇫🇷)
1 × 👨‍✈️ Producer & Designer (Germany 🇩🇪 & USA 🇺🇸)

Initial Idea & First Designs

The idea started in 2014 while I was having a shower and looking at the tiles in the bathroom.

“If I would turn one tile I could match two that have a similar mark and make them disappear… The match is a kiss… I flipped the tile… the game will be called Flipkiss!”

That’s pretty much what happened in my head. I immediately grabbed some Post-Its and Sharpies and created the first paper prototype.

Flipkiss Paper Prototype

It felt as if I was onto something. So I went straight to the computer and came up with a first design draft.

Early Designs for Flipkiss

The Journey

The idea and the designs were just the start for a journey that turned out to be longer than I thought it would be. Here are only some of the challenges that this project had to face:

the team had to be assembled
the story had to be created
the matching mechanism had to be refined
the game design needed a lot of work
the illustration style had to be figured out
the animations had to be created
the sounds and the music had to be composed
the way of publishing had to be figured out
the localization into eight languages

Here is a short video that shows the progress of the game:

From early sketches to game design to end result

Visual Impressions

Here are a couple of visual impressions of how the game looks like.

The initial idea was to raise awareness and funds for HIV. But after I met a Lupus warrior, I learned about Lupus. As Lupus has a smaller public appearance than HIV I chose to make Lupus our cause. I contacted the Lupus Research Alliance and they welcomed our approach warm-heartedly.

Raising awareness and funds for Lupus with Flipkiss

Launch & Success

Flipkiss was launched in October 2018. Flipkiss was featured by Apple in “New Games We Love” for several countries like the USA, Germany, the United Kingdom and Spain.

Launch Video

In the category “Puzzle” Flipkiss reached Top 10, Top 20 and Top 30 Ranks in the following countries:

Top 10: Italy, Saudi Arabia, Indonesia, Philippines, Malaysia, Egypt, Jordan, Tunisia, Portugal, Finland, Hong Kong.
Top 20: United Kingdom, Denmark, Belgium, Taiwan, Austria, Australia, Ghana, Ukraine.
Top 30: Germany, Spain, Canada (#31), USA (#33).

“New Games We Love” and “Rank 33” in the Apple US AppStore

Being a Midwife for an Interface Pattern Library

{"name"=>"nexum AG • UX Design"} — Wed, 01 Jul 2015 00:00:00 +0000

Challenge:
A visually outdated homegrown application world that was mainly built by developers, with very little user feedback, needed a unifying design.

Approach:
With a scrum team of five create the initial step for an interface pattern library, based on a specific software, with the user-centered design methodology.

Contribution:
UX Designer and Team Lead.

Year: 2015

Situation

One of Germany’s biggest retailers was working on updating its IT infrastructure. The frontend needed to be modernized, and the management wanted to introduce a more user-centered design approach to the company. A team of a frontend engineer, scrum master, visual designer, me as the senior UX designer and internal team lead (for the nexum team), and a product owner from the retail company were tasked to tackle that problem. Due to a shortage on internal personnel, the scrum master and the frontend developer were external.

Planning vs Reality

In this case study, I want to show some of the facts of real life. Projects can be planned beforehand, which is what you should do so you can allocate people and estimate costs. But reality very often challenges that plan.

Approach

The application world of the company was very big. One approach could have been to analyze the whole system and try unifying it with a pattern library. This approach is very time-consuming. Big projects not only take time, but the outcome is uncertain. Therefore, the choice was to focus on one singular software that was used in the retail stores and try generating a North Star. We would not only restructure and design the screens, but they would be presented as a working prototype with real data.

Agile & Peer Review

The project was set up as a scrum project. This was due to the way the HQ already was organizing their development process. Our plan was to be a satellite in the beginning, then later be integrated into the scrum processes in the HQ. One element of our agile process was the peer review. One consultant had a 30-minute meeting with the team at the end of the week, looking at the current results. It was a great help as it enabled the team to get an external opinion. Decisions were questioned and designs were critiqued in a constructive way. The team and the project benefitted a lot from that.

Designs and wireframes being peer reviewed

Who Are We

There is a great article by Oliver Reichenstein from Information Architects, “The interface of a cheeseburger”. It talks about how you can look at an interface. I think for a team working on digital products it is always important to know what they represent with their interface. So, this is why we asked ourselves this question. The interface of all these applications in the market is the digital interface that connects the store with the headquarters. So, the headquarters is sending a message with this channel that they are providing. The look and feel of an interface should reflect how the headquarters views its stores. So, we boiled down our vision statement to this sentence:

“We are the message that the headquarters sends to the stores.”

Our goal was that this message be a delightful message that respects the user as a human being by making the user interface not just look good, but also a pleasant experience.

Our war room with our vision statement

Sprint 0

Planned: 1 × 5 days
Reality: 1 × 3.5 days

The goal of sprint 0 was to get the team together and to figure out what we want to do and how we want to do it. We were briefed on how the software was used in the stores. Since this briefing was done by the developers, we wanted to get to know how store owners or the service people from HQ use the software. So, we started arranging appointments. Together with the product manager and input from the team, we thought of questions that we would like to ask our users. I started researching other interface pattern libraries and documentations. I also sketched the first screens. The frontend engineer started setting up his system. And we did set up our ticket system. And we came up with a draft concept for how code and design could be documented together. At the end of the very short sprint, we presented the outcome.

The first scribbles and wireframes after the research

Sprint 1

Planned: 2 × 5 days
Reality: 2 × 4 days

In sprint 1, we were able to reach out to two users. We interviewed one store owner in his retail store, which gave us some impression of how a store owner also was running his store. The other user we interviewed was from the Helpdesk in the HQ, who supported store owners with the software. This interview was conducted via telephone. Those two inputs weren’t enough to assume that we had gathered enough data, but they nevertheless gave us a good impression that there are very diverse needs to that interface and that those need to be taken into account.

As our “client” was the store owner, we chose to generate an empathy map so we could bring that persona more into the center of our project. With the feedback from the store owner, I worked on the wireframes for the next sprint.

Wireframe the Designs

As this project was very short, the time for design explorations was very tight. The visual designer, who also was working on another project, didn’t have the time to try out several versions. I offered to make the wireframes more like a design to explore another visual approach. The two design versions were very helpful in this starting phase, and were discussed constructively in the peer review.

Our first peer review

It was a very collaborative environment as we all were sharing one room and showed each other results or discussed solutions to problems that we saw.

Designs from 1st sprint

At the end of the sprint, we presented the results at the headquarters. We got valuable feedback that we took with us for the second sprint.

Sprint 2.1

Plan: 2 × 5 days
Reality: (1 × 4 days) + (1 × 4 days)

This second sprint was a challenge. In the first week, I got sick and missed three days, and in the second week the product owner had a planned holiday and was going to miss that part of the sprint. Therefore, we split the sprint into two parts, with two presentations at the end of each week. With the feedback from the presentations, the visual designer reworked the designs and made the appearance more professional and showed that the design concept was skin-able. Though I was not around, the wireframes I had generated in the first sprint were the basis for the work of the visual designer and the frontend developer in the first part of the second sprint. At the end of the week, we presented the results. The feedback was very positive. Our task for the next part of the sprint was to finalize what we had built and to document it.

Designs from 2nd sprint

Sprint 2.2

In the second part of the sprint, the visual designer and I built a system for documenting the pattern library. We separated the documentation into sections:

Colors and Color Usage
Grid
Screen Structure
Icon Concept
Typography
Skinning
Tables
Filters (First Pattern)

Overview of the documentation

The frontend developer worked on finalizing the working prototype. At the end of the week, we presented the final result in the HQ.

Retrospect

Putting the user in the center in B2B software environments can be a hard task. It’s not easy to get the feedback that is needed. And it’s not easy to restructure a developer-centric organization that had little to no direct connection with the user. But the client tried, and provided a good example of what can be achieved if the will is there. The client was very satisfied with the results and it was a major learning experience.

Usability Test it Baby

{"name"=>"nexum AG • UX Research"} — Sat, 01 Nov 2014 00:00:00 +0000

Challenge:
Our client, a German cinema operator, wanted to improve the current flow for booking cinema tickets on mobile and desktop and get insights from their users.

Approach:
Build a new flow with an interactive prototype and test that with an onsite usability testing.

Contribution:
UX Concept, Prototyper, UX researcher

Year: 2014

Customer Journey Mapping

nexum is a UX-centric agency. To determine the pain points of the cinema website, we organized a Customer Journey Mapping workshop. It helped us to understand the process better, and analyze what could be done to fix the problems.

Customer Journey Mapping Workshop result

Research

With the insights from the Customer Journey Mapping workshop, I started my research phase for the reorganization of the website and the reservation flow. How do other cinema websites or apps solve the reservation process? I also looked at other well-designed reservation flows, such as plane tickets or car reservations.

Researching other order flows

The reason is that those companies rely much more on well-designed flows since their revenue is derived, to a large extent, through online reservations. Therefore, they are much more willing to invest in the design and the ease of use than other companies. My research also showed that many of the cinema ticket booking flows that were available online were flawed, and that there was lot of room for improvement. In one of the calls with our client, I said that our goal should be to create the best booking flow in the world.

We have to design the best cinema booking flow in the world

It may sound pretentious, but if you don’t set high goals for yourself, you won’t create the best results.

Mobile First

The client told us that a big chunk of the reservations already were done on mobile (App and Website). As our agencies mobile expert, I proposed the Mobile First approach. The idea was to first fix the flow for mobile. Doing that, we would strip away unnecessary features and elements. Based on the mobile flow, we then would build a new reservation flow for the desktop website. As the time for creating the prototype was scarce, I teamed up with another UX designer. I started with the mobile flow. To create the interactive prototype, I used Axure RP.

Used Axure RP for wireframing and prototyping

Minitests while Prototyping

Everybody goes to the cinema, and many people do buy their tickets online. It was perfect, as I could ask everybody to test certain features or structures that I came up with. nexum has an open culture. So, every now and then, I asked a colleague: “Do you have five minutes? Could you please reserve a ticket for the film Planet of the Apes on Thursday evening?” This quick and dirty feedback helped the prototype to evolve very quickly, while being tested the whole time.

Quick 5 minutes usability testing

When the mobile prototype was at a certain stage, another UX designer hopped in to create the prototype for the desktop. The basis for her work was the mobile prototype. It was encouraged to break out from it and try out new things.

The desktop prototype

Test Guide

Our goal was twofold:

Check how people are using the current website.
See if and how the new flow improves the booking process. In cooperation with our client and my UX designer colleague, we prepared the test guide for our usability test.

Testing Marathon

35 usability tests in 10 hour session

For the testing, we went to one of the client’s movie theaters. Our client had contacted cinema-goers who had booked a ticket for that day with the app or from the website. They were asked if they would like to participate in a 30-minute usability test before or after their film screening and would receive two movie tickets and a big bucket of popcorn coupon. The incentive for that target group worked extraordinarily well. We had more than 35 sign-ups. We prepared two testing booths. Every booth had a laptop and a mobile device that had the prototypes saved locally. The testing for both flows were recorded with the screen recording software on the laptop and saved on external hard drives. One test took 20 - 25 minutes. So, every booth had to test for 10 hours. People who booked on mobile tested the mobile flow. Those who booked on the desktop tested the desktop.

The final booking flow on mobile

Result

Evaluation of the usability test results

The usability test did show us that some of the assumptions we had were true, and some were not. The new reservation flows were perceived as simple and fast. One of the big surprises was that more than 65% of the mobile users did make screenshots of their reservations. With the feedback from the usability test, we reorganized the prototypes, suggesting the improvements as a live document, instead of just summarizing them textually. For example, we organized all the information from the booking confirmation on one page so the user could save it easily with one screenshot.

Screenshot of final confirmation page

Retrospect

The approach of directly testing the prototype while creating it was a very effective way of creating a good prototype and getting rid of problems even before doing the big usability test. The big number of testers was good, but a smaller number would have led to the same results. After testing eight users, we were hearing mostly the same complaints and problems. The “Test at least 5 users” rules proved also to be true in this test scenario. The client was very pleased with the outcome. The results helped the client later improve the website booking process.

Portfolio

The E-Commerce Voice Assistant

Situation

Challenge

Our Team

Organizing the Team

Creating the Vision

Keep Developers Closest

First Working Prototype

Speed

Highlight what you Understand

Improving the Speech Recognition

Audio Correction Portal

Quality Assurance & Data Collection

Test in the Wild

Hands-Free Orders

Thumb Orientation leads to Certain Design Decisions

Designing & Workflow

Prototyping

The Dirty Work

Speechtek 2016

The Launch

Custom Apps

Image Recognition

Retrospect and Outlook

Creating a Vision & Telling a Story

Situation

Research

Our Vision Statement

The First Prototype

Our User Story told with Personas

Retrospect

Learning Bit by Bit

The Problem

Workshop

Design & Prototype

Retrospect

“Team, We Are Faster than Siri!”

Situation

Approach

Teamwork

350% Faster

Retrospect

Flipkiss - A Game about Love

Summary

Team

Initial Idea & First Designs

The Journey

Visual Impressions

Social Cause

Launch & Success

Being a Midwife for an Interface Pattern Library

Situation

Planning vs Reality

Approach

Agile & Peer Review

Who Are We

Sprint 0

Sprint 1

Wireframe the Designs

Sprint 2.1

Sprint 2.2

Retrospect

Usability Test it Baby

Customer Journey Mapping

Research

Mobile First

Minitests while Prototyping

Test Guide

Testing Marathon

Result

Retrospect