Task | Test User 1 - Observations | Test User 2 - Observations | Test User 3 - Observations | Tests 1/2/3 Findings and Take-Aways | Tests 1/2/3 Conclusions and changes | Action Items | |
Pre-Intro | Recording consent | Given, agreed | |||||
Intro | What is your name and age? | Björn, 32 | Pille, 40+ | Adel, 30 | |||
Could you please tell us about the tasks of your job and your job title? | Parlamentery advisor, works for regional parlament in Baden-Württemberg, consults in science, XXX and culture | eHealth since 2006, hospital, ehealth standardization management | GIZ Geo information manager | It was good, that they had same amount of information beforehand Unfortunately different backgrounds and roles/jobs made it hard for Oille and Adel to emphasize and understand general purpose of application | check participant profile and focus on politicians and their consultants if possible | ||
Did you participate in User Tests before? | Yes, due to his job | yes, no previous knowledge about govstack | no | ||||
Landing and Intro | Understanding GovStack and general intro feedback | not really sure what it is about, would click on upper links to learn more, though | no questions, "so far so good" | two out of three did not understand BB for the whole interview, introduction with Govstack general info did not help Adel thought he understoods the application and ist context (but didn't) and Pille communicated her insecurity (and didn't) |
| ||
Onboardingpage feedback | What’s your first impression and what can and would you do here now? | likes the amount of content, doesn't like too much text, therefore he likes it very much (first impression) "you have to have a certain pre-knowledge, right?" Artun gave context about pre-knowledge as Björn also gave hint that this was needed. Misses context about what GovStack is etc., spots more on it on bottom of the page what can you do? "get a glimpse of how it works", "very interesting!", "I can have both perspectives", "very necessary to have both perspectives" | thinks that first boxes are clickable would like to see something for the problem she is working on, i.e. health understands that USCT is the example use case "maybe too much information in one page" is wondering about skeleton in preview "clickable prototype would be easier to understand" | was trying to find main objective of interface, "mix of normal website and web based application" semms to have understanding about the potential motivation a targeted visitor of this page might have | Pille at least was hoping for a diversity of use cases to review it with use cases she can relate to | feedback this also to Nico/Farina |
|
Expectation | "Show in linear process different perspectives for one use case", "impression of layout etc." | "tool will be opened for me, not a video or sth. Like that" "page is overloaded with information", "not easy to catch", "dont understand some of the bars" | "experiment" "draw conclusions" "possibility to customize" "having phases, going one step to another" | Pilles and Adels expectations were off and they were not able to learn better later on we framed / nudged way too much on eGov simulation itself and the perspectives | Create better visuals for landing page, especially explaining BB background role in simulation and touch on frontend vs backend and role of architecture and BBs more explicitely nudge them towards GovStack principles, not the use case and not the eGov UI itself explicitly state that this is example use case for social welfare |
| |
Could you please start the simulation? | wouldn't like to read that much on the page, would like to go ahead | ||||||
OTHER | |||||||
Tutorial | How did you feel about the pacing and content and onboarding of the tutorial? | "Is this a planned tour?" thinks ist cool and helpful, pacing good and not too much, right amount of information | is wondering about consent and authentification details | (skipped tour directly) | Don't rely on users to READ the tutorial (and the onboarding page) | application itself has to be self-explanatory, guiding and exploratively accessible where needed and possible discuss if we want to change tutorial, i.e. to embedded tips or similar | |
How confident do you feel in your ability to use the simulation after completing the tutorial? | not really confident, feels like she missed something, "need for more clear information", step before (onboarding page) was not giving her enough useful information | ||||||
OTHER | |||||||
Simulation | Experience it and think out loud what you’re thinking. | ||||||
What is your first impression about this page? | "Am I logged in?" "That's a citizen perspective, am I right?" "I am unemployed" What was previous impression? "I am a citizen using services", "I am wondering what this icon means" Is insecure about perspective he had in the beginning | BBs "confusing me", doesn't understand why some of them are hidden thinks the information there is about "her" as a user sees no need for editing option next to email etc. Doesn't know what it is, lack of background information (concentrates a lot on simulation itself) | looks like a dashboard to him expected that he needed consent first for the login (Artun had to rephrase scenario) | Login as entry point confused two participants Consent BB was connected to actual UI login BBs were already misunderstood and questioned here BBs were misunderstood as tasks No participant got first perspective "right" (all thought they were a citizen or themselves) | redesign perspective status and strengthen connection to wireframes remove civil servant login from flow generally and start with Ongoing Case Management for the test, start with ongoing case management too if possible remove hovering state from insight bubbles redesign navigation, make it more simple, introduce arrows for back and forth navigation redesign DIAL button/starter |
| |
How would you describe the structure/layout of the simulation? (Let them explain the navigation & Ul elements, etc.) | "well structured first of all" likes amount of content "I do not know that Building Block here, never have seen something like this before" | "maybe there should be another layout" citizen vs. Servant, difference? Would like to see more accessible UI feeling of security | simplify eGov sim. Content, remove stuff that's not needed to understand BBs |
| |||
Please describe in your own words – what is a building block? | "Why is one highlighted, other is not?" What is BB? "BB is specifically designed for people who are interested in this product", "not clear if it's part of the program or not", "can imagine it's part of the program" "how would you describe BB role for page you are on?" - "it's not really clear", description in catalog is understood, but relation to page not clear to him, connection missing for him | "it's really hard to catch", not easily linkable doesn't understand why they are linked and about their roles | purpose of side panel? "is to structure major parts of app, in not confusing way, if I want to go to payment it gives me easier access to go to application parts quickly" "as I click it connects something in panel, guess it has something to do with main structure of interface What is BB? "...are major objectives of this prototype/app based on digital services" "have to do with major milestones to get where they want to be" | see above | DIAL link to BB explanation legend for BB activity, for example traffic light system connect active insights to BB overview think if we can show that they can be used for other use cases show more clearly that initial status of BBs shown relates to the whole page |
| |
How can you find out more about the information mediator and what is its role? | "no idea how I can learn more and how they relate to each other", confuses her at the moment | it's connected to frontend | intro can tell user with which perspective user is starting, too | introduce more properly to the first page | |||
How would you describe the role of the building blocks for this page? | not sure which side was shown, recommends to make it clearer! | "major things that are structuring the page" (after clarification of Artun) like foundations and design elements of this work | |||||
OTHER | additional question from jonas at end: got the connection after clicking and found it very helpful (seemed like the missing puzzle piece) BB term not telling, needs more explanation | Progress - don't think he recognized progress bar, was referring to content when asked for progress of phases | |||||
(Context change) | What changed on the screen? Could you explain it in your own words. | didn't realize perspective change | |||||
What does this change indicate for you? | "gives sense of security" (referring to login) | ||||||
Where would you expect to see the perspective changes on the screen? | Make perspective more clear, "not common that it (perspectives) looks the same" | ??? 47:30, didn't fully get it | |||||
End question: | How would you describe this product/simulation in your own words? | "think it's quite useful" "trapped me on the wrong foot" "if it's not 100% clear what you are looking at, confusion stays" prototypes he liked most in this context were those who had a straight aim, perspective, etc., simulation is nice, realistic but beginning, advertised something I couldn't find in simulation visualization how BB help me ... XX?! | "hard to understand how it works and how to use it as civil servant" doesn't know how it can help to build eServices if it would help me to find functionalities for my services it would be helpful "hard to understand in the beginning" | expectation was different, but thinks this can give perspective | even though it's not part of test, we should try to design proper offboarding | ||
If this was a real website, how would you rate the technical maturity of this product? Were there any indications of this? | 8 of 10 | 85%, because transitions are not perfect | make clear it's only for simulation purposes, make sure they understand there's no backend BUT BB activity is showing backend activity | ||||
@ | After experiencing the simulation - did you feel well introduced to it by the website and tutorial? | onboarding page was more comprehensive, logical connection was given, details missing on onboarding page visual vs. Text? -> "visual, good graphics with few words" | |||||
Offboarding: | Thank you very much for these great insights and of course for taking the time to help us! | ||||||
Do you have any questions or feedback for us? | |||||||
OTHER | "Would you share this?" - yes |
Summary
Difficulty understanding perspective changes in the simulation was a common issue among all participants. One participant relied on the header of Egov to identify the perspective but did not realize changes were being made.
The first participant, who had experience in digital government transformation and a political background, understood the aim of the project. This highlights the importance of recruiting testers with relevant expertise for the project.
The terminology used on the page, especially "Building Blocks," caused confusion for participants. Two out of the three participants did not understand the concept of building blocks and contextual panel functions.
Participants were mainly focused on the applicant/civil servant view for the onboarding page
One participant suggested having the option to select a use case they could relate to.
Users were overly nudged towards the eGov simulation and perspectives, instead of the Govstack approach and Building Blocks. In addition, the simulation did not align with their expectations created during the onboarding page.
The simulation was not self-explanatory enough for users, and one participant found the tutorial helpful while others quickly scanned or skipped it multiple times.
Egov wireframes caused confusion for users due to the heavy amount of information presented. Also the login page and updating personal information part caused confusion.
Despite the issues, all participants believed that the product was technically mature, scoring it 8 out of 10 in terms of technical readiness.
0 Comments