Chapter 4 The Technology Behind

Chapter 5 Validating The Prototype

For a couple of reasons, user interaction and feedback during development of the application was not very extensive. Naturally, there were time constrains, but the priority was also set on doing quick iterations to allow testing of new features. In addition, to be able to use the application a prior knowledge of the model is required. Therefore, for the first version only feedback from experts was possible. Even if the interactions are heavily inspired by the physical usage of sticky notes, there are new ways to interact with the application. Some of these have to be learned. Testing them too early when the whole system isn’t complete might make them get strongly rejected by users which are not in the habit of using them.

In the end, having made important design choices with limited user feedback, while creating the application, strengthens the need for testing usability as well as usefulness.

For the usability part, the questions are:

  • Will the user perceive the application as a web based page or interact with it like a desktop application?
  • Do the defined actions, menu labels, highlights have enough affordance to allow usage without having to read the complete manual?
  • Is the paradigm of the sticky note interactions still visible? And is it useful for people who did not use the paper based methodology?
  • Can the application be used on small screen as well as big screens?

Since we transformed and augmented a manual process into a digital one, the question about complexity and usefulness are also very prominent:

  • Is it as easy as with sticky notes?
  • Does the additional meta-information add sufficient value to be useful?
  • Can the tool support and help collaboration?
  • Can the tool stimulate the same thinking in a brainstorming session as does the paper based version?
  • How does the persistence and versioning affect new alternatives?

5.1 Testing Methodology

To allow for testing without needing the user to study the handbook, or become an expert of the model, testing was done using the walkthrough methodology.

A quick introduction to BMO was given as well as a fifteen minute demonstration of the tool. After which, the user was asked to perform different operations and his actions were carefully observed. When he was blocked or performed strange operations the tester asked for the reasoning and gave advice on how to continue. Participants were also encouraged to give feedback on ideas, expected interactions and other comments.

For all the following tests the premise is that the user knows at least the basics about BMO. The actions of creating an account or connecting to it are not tested.

5.2 Data Entry Test

5.2.1 Scenario 1

The business model is already defined. It only has to be input into the application to allow tracking of its evolution and allow annotation and detailed description of its elements. (SuperToast, Apple)

Tasks

  1. Create a new empty business model
  2. Add desired elements and layers
  3. Move elements, inside and between blocks
  4. Create links between strongly depended elements
  5. Display the links

5.2.2 Scenario 2

A business model is already created in the system. A new alternative or evolution has to be added. (Amazon: classic and S3, Apple: iPod and iTunes)

Tasks

  1. Open the business model that has to be edited.
  2. Add new layers and elements, and add existing elements to new layers as needed to represent the new aspect of the business model.

Alternative 1

  1. Create a new empty business model containing only the new parts.
  2. Merge the new business model with the initial model.
  3. Edit the duplicate elements.

Alternative 2

  1. Work with the snapshot manager to create alternatives.

Problem: the two alternatives are not visible at the same time.

5.3 Annotation Test

5.3.1 Scenario 3

Evaluate an existing business model by using annotations or answering to assessment questions.

Alternative (Direct)

  1. Add annotations directly to the model, the perspectives or the blocks
  2. Edit the annotations (change text, type)
  3. Filter annotations
  4. Delete annotations

Alternative (Wizard)

  1. Open the wizard
  2. Choose a question set either assessments or innovation.
  3. Answer some of the question, edit business model as needed.
  4. Add custom annotations.

5.4 Feedbacks

The three scenarios and other small interactions have been tested with multiple users on the final prototype during the month of September 2008. They all had knowledge of BMO ranging from novice to intermediate. Computer knowledge for the majority was intermediate, except for one who was an expert (see table 5.1 for a summary).

Table 5.1: User summary
User BMO knowledge Computer knowledge Tested Scenario
Assistant Intermediate Expert 1,2 and 3
Assistant Intermediate Intermediate 3
Assistant Intermediate Intermediate 1 and 2
Assistant Intermediate Intermediate 3
Consultant and a developer Intermediate Expert Defining a new Business Model and exploring future alternatives
Professor Expert Expert Miscellaneous Observations
Student group Novice Intermediate On TouchWall develop a new model
Consultant Expert Intermediate On TouchWall wizard, annotations and some editing

Globally feedback was positive. The testers managed to use the prototype without having too much trouble identifying the right action for their intensions. On the usefulness side of things, there were no real test, but the general opinion is that the reporting features as well as the versioning and layers features may certainly help in providing value over the paper based static solution.

Interestingly, users with greater knowledge of the model or who had more expertise in IT identified more defect in usability, and users with less experience were looking for features that were not available (usefulness). This combination of testers allowed identifying possible usability inconsistencies and also generating ideas for new features.

In the following section are analyzed mostly actions that were not providing the expected results, since covering the success of common actions like drag and drop or clicking a simple link seems to be pointless. Moreover, failures due to clearly identifiable bugs of coding and not choices of usability have not been taken in account and will simply be corrected in a new revision of the prototype.

5.4.1 Usability

An example of a bug that also influenced usability for a user was list highlighting which had not been turned off. Thereby, sometimes an element would get highlighted with a blue background without having any meaning or functionality. In this case, the user was informed not to pay attention to the artefact and testing was able to continue without interference of the bug.

Action

Clicking on an empty space inside the menu or perspective border toggles the appearance of the zone by resizing it. In collapsed form even the title of the perspective is hidden to maximize available space for the visible elements.

Observations

  • The toggle border for the menu is too hard to identify.
  • For novice user of the model hiding the perspective’s title is disturbing.
  • Wrongly collapsing a perspective can happen because target zone is every empty space.
  • A feedback indicating that the toggling will happen if clicked was expected.

Suggestions

Limit the toggle interactions to the title of the perspective. Display the title of the perspective also in collapsed mode. Generate a feedback on hover by changing the cursor or font of the title to highlight the possibility of an action.

5.4.3 Drag and Drop

Action

Drag and drop elements, from one place to the other or into the trash can.

Observations

There were no problem drag and dropping elements. Some concern was expressed concerning the ability to delete a whole layer with all its elements without warning.

Suggestions

Since the trash can permanently delete’s the object that is released onto it, it could be wise to ask for confirmation when the action has serious repercussions like when deleting a layer. Additionally, the dialog box could propose to take a snapshot before deletion.

It should also be taken into consideration to change the artifact’s image. A document shredder would perhaps be a better metaphor than the trash can based on the knowledge that ’Visual metaphors leverage real-world knowledge of objects, but metaphors that are very literal may introduce information that is irrelevant or misleading to a task’ [ Rosson and Carroll, 2002, Tradeoff 4.7 p. 129]

5.4.4 Click Interaction and Menu

Action

Clicking on an element displays a menu to view details, display links or change layers.

Double-clicking an element filters on the selected elements and displays its links.

Double-clicking an empty space in the building block (list of elements), creates a new element in this block or when displaying links switches back to normal view mode.

Observations

It is unclear if the double affectation of double-clicking a block is used. Quitting the link view was mostly done by clicking on the message bar, which also provides this functionality. There were also no attempts to create a new element in the link mode, although the creation of new elements by double-clicking in the normal mode was intuitive for everyone.

The biggest problem was the expectation from the action of double clicking an element. Every user novice or expert expected to display the detail window and not to display its links. Also since renaming an element can only be performed from the detail window, in contrast to prior prototypes where it could be done directly on an element generated some hindrance in the usage flow.

Advanced user had trouble with the fact that there are no right click possibilities, this indicates for one that even thought the application was running inside the browser it was perceived as a desktop application. Furthermore, these users were also more disturbed by the small delay there exist between clicking an element and the menu being displayed.

Some user did not know what action to expect behind the link menu, if it would create a link or display them.

Analysis

Some of the failure in usability observed in the mentioned click interactions can be traced back to the merging of the two modes create and link that were used in the first iteration of the second prototype. In hindsight, obviously the wrong interaction was favored for the double click action. This merger also removed the possibility to rename an element directly by simple click in favor of a menu. This menu although not optimal, is a necessity since we cannot use right click or any other interaction that would not be available across the multiple devices on which the application should be able to run. If there were to be special version for each input device the menu delay could be reduced and different actions could be chosen, the tradeoff being to lose consistency of use between inputs.

Suggestions

Change the action occurring when double-clicking an element to displaying the detail window. Try to reduce the delay of the menu appearing. Rename the label of the menu entry from link to something more explicit like view links. Try if adding a rename option in the menu with the possibility to directly rename the element without displaying the detail window can increase the flow in the model creation phase.

Action

Creating links by dragging one element over another in a different block if the model permits a relationship between these two blocks.

Deleting a link by drag and dropping a highlighted link onto the trash can symbol.

Observations

Link creation is well understood. After creation everyone does display the link he created.

One user thought that the highlighted link meant that a menu would appear if clicked, and not that it is ready to be dragged.

Suggestions

Automatically display the created link after creation, currently the model refreshes and does not filter on the newly created link.

Perhaps find an alternate way to delete links than to have to get them highlighted and then dragged onto the trash can.

In the current version there is no way to know if an element is linked except to display his links, perhaps some sort of highlight in the normal view mode could be of use.

5.4.6 Detail Dialog

Action

Edit attribute fields and close the dialog with the x in the upper right to save changes.

Observations

Most of the users were looking for a close/save button or typed enter to validate/confirm changes.

An advanced user wanted to interact with the listed links that are read-only.

Suggestions

Add a button to close the dialog. Perhaps to resolve problem with the link selection from previous case add possibility to edit links from this dialog as well as directly dragging the displayed links.

In the current version of the prototype the general purpose detail dialog does suffice and adds consistency, but in future iterations with more specialized need for each block, a different approach will have to be chosen to provide desired functionalities. [ Rosson and Carroll, 2002, Tradeoff 3.3 p. 84]

5.4.7 Layers

Action

A layer can be renamed by clicking its name.

Double clicking a border creates a new layer.

Visible layers define to which layers a newly created elements belongs.

Observations

Average users intuitively found the click to rename functionality, but had problems using the double click to create a new layer action.

Basic users didn’t really understand the layer selection and element creation dependence and the expert user was bother by it. He was expecting to have two types of selection one for dependence attribution and one for visibility like it is the case in design products from Adobe.

It is true that hiding layers for not getting their reference when creating new alternatives is disturbing because of the inability to see what has already been added to the model.

Suggestions

Add a button, menu or increase the zone that enables the creation of a new layer.

Make visibility, and layer dependence for new elements two separate options. Either by adding a second selection column or by selecting the layers in the new element dialog. Still it is a good idea to keep in mind that ’Displaying all active task entities enhances understanding and feelings of control, but each display elements adds to the complexity of the user design’ [ Rosson and Carroll, 2002, Tradeoff 4.1 p. 115]

5.4.8 Annotations

Action

Toggle new annotation pane by clicking on an empty space. If there are no annotations the pane is displayed, otherwise it is hidden by default.

Filter displayed annotations by type or level if they are to-dos. The range can be defined by moving both sliders.

Double click an annotation to edit it.

Observations

When looking at the annotations, many users did not find the answer annotations, because they are by default filtered. This can be related to ’Dynamic hiding of controls that are not in use conserves space and simplifies the display, but also conceals the controls’ affordances from the user’ [ Rosson and Carroll, 2002, Tradeoff 4.5 p. 125]

Most of the users expected to be able to click on an annotation to edit it, instead of having to double click it.

One tester was confused when creating a new annotation and mixed-up between the filter checkboxes and the drop down selection to specify the type of the new annotation.

When using the range filter for the first time it is unclear to most users that both sliders can be moved, but once the interaction is shown it is well understood.

One particular user imagined a correlation between visible layers and annotations displayed, which is not the case. This same user also wanted to be able to drag and drop annotations from one window to another.

Suggestions

It seems that hiding information is misleading and even if at start the screen is a bit cluttered, it would be better to display all annotations as well as the new annotation pane, and then let the user hide the elements he does not want to be displayed.

The range sliders is a new concept not seen in daily usage of computers, therefore showing a simple help information at first runtime could help to train the new users.

It would be interesting to study the usefulness of a possibility to drag and drop annotations, for now the same result can be achieved by copy pasting the old text into a new annotation.

5.4.9 Wizard

Action

Answer or comment a question to keep track of the thought process.

Observations

There was a mix-up between a question’s answer and creating new element as a response in the block displayed underneath.

Suggestions

Give the possibility to create a new element from a selected portion of the answer. Or perhaps change the placement of the block and the questions.

Although the wizard can guide novice user to identify missing elements and provide assessments question for more advanced users, a more open view should always be provided. This because ’Decomposing complex plans into chunks aids learning and application of action plans, but the sequence may create arbitrary or unnatural step boundaries.’ Rosson and Carroll, 2002, Tradeoff 5.5 p. 166]

5.5 Touch Interactions

To see if the prototype is useable and could be useful on a projected wall, this mainly in the intent to be as near as possible to the original sticky note experience, it was first tested on Tablet PC. Tablet PC gives the user the same display size as the normal screen experience, but instead of using a mouse the user can directly manipulate the objects on screen with a pen. This first experience revealed that most of the action are working as well as with the mouse, or even easier for some drag and drop operations. Naturally since the keyboard is still available it is the preferred input method for text, even if handwriting recognition is available. As explained in the technical chapter, early on such a test revealed that click interactions are somewhat imprecise and have to be taken into consideration for detecting the double click actions.

The next step was to test the prototype on a real big screen solution. It was choosen to augment a normal projector with the e-Beam interact solution to test the application. The setup can be seen in figure 5.1. This solution acts as a single touch cursor that can be calibrated over any flat surface onto which can be projected with a beamer. The idea was to test not only usability, but also to see how working with a projected solution would compare to the paper based system concerning group interactions, brainstorming possibilities and other design aspects. Unfortunately, during testing mostly usability problems were observed and there was not enough time to see the real impact on the usage side. Further testing with expert users and a greater time allocation is needed.

Prototype 2 projected onto a wall and eBeam interact as pointer

touchwall.jpg

As for the usability, it has become clear that touch solution are even more approximate for registering clicks and pointer location. For example, involuntary perspective collapse occurred frequently due to unintentional clicking on empty space of a perspective. Link selection was nearly impossible since the highlighting of a link is required and occurs only if the cursor is exactly over it.

First, text input was done through on screen handwriting recognition, which worked great, but was very slow. On screen keyboard is as well slow since keys have to be pushed one at a time. During testing these were abandoned in favor of entering the text directly on the laptop keyboard. This would suggest that for easiest usage a small wireless keyboard would be the optimal solution. Other possible solutions that could be tested would be mobile devices input via Bluetooth, or even voice recognition.

Bigger screen does not necessarily mean more space. In fact, screen resolution had to be turned down to eliminate some font size issues. During this testing it was the first time that users complained about the vertical text that is used for the wizard panes. This could be related to the font size problem.

In our temporary setup projection was done from the front which generated lots of shadows when people were trying to touch the wall. This had an impact on the number of persons that could be in front at the same time and limits testing of large group interactions. It would be ideal to find a better projection setup for further testing of interactions on the wall.

5.6 Usefulness

Usefulness was not tested with a statistical comparison between different methods and by defining meaningful metrics. The current prototype is more a proof of concept. Its usefulness was judged by the reactions and the interest of the participants and also by their eagerness of proposing new features.

One session which was very encouraging was done with an outside consultant comparing his model to ours. He was very interested by the possibility to generate a report from the acquired information, and would be delighted to see more detailed information in it. The parts that are missing from the BMO like environment and competitors can still be put onto our representation by using the annotation features. Also the layers proved to be very useful to input alternative evolution scenarios.

The introduction of annotations is certainly a useful feature judging by the interest it generated. One participant who has some experience with collaborative tools was especially interested to see the dates of annotation and changes since his last visit. This strengthens the fact that for a next iteration it would be interesting to explore more in deep the possibility of online collaborative features.

Even if mostly every element can be commented on, some tester wanted even more possibilities like annotating links, or to flag an element for later review directly on the element itself without creating an annotation on the block or adding a comment in the element’s description. In my opinion, adding too much visual flags could overload the model and take away its efficiency given by its simple visual layout.

The choice to let the layering system be as generic as possible has paid out. As predicted, people had different approaches to take advantage of the layering system. Some people prefer to use it to model evolution scenarios, while others identified different parts of their model with it. For example, in the case of a multisided business model, a user made the choice to separate upstream and downstream customers in different layers. The ability to use layers to model evolution does on short term testing limit the usage of the more advanced versioning possibilities, but it is clear that the possibility to freeze a state of a model in time and track the relations between states is also necessary as a long term alternative.

Merging has first been seen as a way to compare two BMO, but in the end is even more useful. It can be used to make a copy of a model by only merging one model. It can be used to merge more than two models. And finally it can also be used to merge two alternative states of the same model to more easily compare them.

In most tests the detailed attribute of the different elements were not used. I do not believe this is because there are not useful, but more due to the short testing run and the lack of knowledge about these attributes which were not used in the sticky note version. Another hint that confirms the usefulness of some of these attributes is the fact that one user suggested to display them directly on the creation dialog box so that he may be aware of them without having to go look for them afterwards in the detail window.

Some of the less knowledgeable user of BMO would have appreciated an overlay that explains the blocks their position and meaning of the links between them. This can be interpreted as interest for the model.

Lastly it is unclear how useful the wizard view mode actually is. It would be interesting to test its usage from beginning to end against a more sporadic usage. Naturally this can also be done on the paper based version.

Chapter 6 Further Research

tm/chapter5.txt · Dernière modification: d-m-Y H:i de boris
chimeric.de = chi`s home Creative Commons License Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0 Fristcher.ch