Task

Test User 1 - Observations

Test User 2 - Observations

Test User 3 - Observations

Tests 1/2/3 Findings and Take-Aways

Tests 1/2/3 Conclusions and changes

Action Items

Pre-Intro

Recording consent

Given, agreed

Intro

What is your name and age?

Björn, 32

Pille, 40+

Adel, 30

Could you please tell us about the tasks of your job and your job title?

Parlamentery advisor, works for regional parlament in Baden-Württemberg, consults in science, XXX and culture

eHealth since 2006, hospital, ehealth standardization management

GIZ Geo information manager

It was good, that they had same amount of information beforehand

Unfortunately different backgrounds and roles/jobs made it hard for Oille and Adel to emphasize and understand general purpose of application

check participant profile and focus on politicians and their consultants if possible

How they usually research new technology

Did you participate in User Tests before? 

Yes, due to his job

yes, no previous knowledge about govstack

no

Knowledge regarding Govstack

Landing and Intro

Understanding GovStack and general intro feedback

not really sure what it is about, would click on upper links to learn more, though

no questions, "so far so good"

two out of three did not understand BB for the whole interview, introduction with Govstack general info did not help

Adel thought he understoods the application and ist context (but didn't) and Pille communicated her insecurity (and didn't)

Onboardingpage feedback

What’s your first impression and what can and would you do here now?

likes the amount of content, doesn't like too much text, therefore he likes it very much (first impression)

"you have to have a certain pre-knowledge, right?"

Artun gave context about pre-knowledge as Björn also gave hint that this was needed.

Misses context about what GovStack is etc., spots more on it on bottom of the page

what can you do? "get a glimpse of how it works", "very interesting!", "I can have both perspectives", "very necessary to have both perspectives"

thinks that first boxes are clickable

would like to see something for the problem she is working on, i.e. health

understands that USCT is the example use case

"maybe too much information in one page"

is wondering about skeleton in preview

"clickable prototype would be easier to understand"

was trying to find main objective of interface,

"mix of normal website and web based application"

semms to have understanding about the potential motivation a targeted visitor of this page might have

Pille at least was hoping for a diversity of use cases to review it with use cases she can relate to

feedback this also to Nico/Farina

Expectation

"Show in linear process different perspectives for one use case", "impression of layout etc."

"tool will be opened for me, not a video or sth. Like that"

"page is overloaded with information", "not easy to catch", "dont understand some of the bars"

"experiment"

"draw conclusions"

"possibility to customize"

"having phases, going one step to another"

Pilles and Adels expectations were off  and they were not able to learn better later on

we framed / nudged way too much on eGov simulation itself and the perspectives

Create better visuals for landing page, especially explaining BB background role in simulation and touch on frontend vs backend and role of architecture and BBs more explicitely

nudge them towards GovStack principles, not the use case and not the eGov UI itself

explicitly state that this is example use case for social welfare

Could you please start the simulation?

wouldn't like to read that much on the page, would like to go ahead

OTHER

Tutorial 

How did you feel about the pacing and content and onboarding of the tutorial?

"Is this a planned tour?" thinks ist cool and helpful, pacing good and not too much, right amount of information

is wondering about consent and authentification details

(skipped tour directly)
always does this, is used to it

Don't rely on users to READ the tutorial (and the onboarding page)

application itself has to be self-explanatory, guiding and exploratively accessible where needed and possible

discuss if we want to change tutorial, i.e. to embedded tips or similar

How confident do you feel in your ability to use the simulation after completing the tutorial?

not really confident, feels like she missed something, "need for more clear information", step before (onboarding page) was not giving her enough useful information

OTHER

Simulation

Experience it and think out loud what you’re thinking.

What is your first impression about this page?

"Am I logged in?"

"That's a citizen perspective, am I right?"

"I am unemployed"

What was previous impression? "I am a citizen using services", "I am wondering what this icon means"

Is insecure about perspective he had in the beginning

BBs "confusing me", doesn't understand why some of them are hidden

thinks the information there is about "her" as a user

sees no need for editing option next to email etc.

Doesn't know what it is, lack of background information

(concentrates a lot on simulation itself)

looks like a dashboard to him

expected that he needed consent first for the login

(Artun had to rephrase scenario)

Login as entry point confused two participants

Consent BB was connected to actual UI login

BBs were already misunderstood and questioned here

BBs were misunderstood as tasks

No participant got first perspective "right" (all thought they were a citizen or themselves)

redesign perspective status and strengthen connection to wireframes

remove civil servant login from flow generally and start with Ongoing Case Management

for the test, start with ongoing case management too if possible

remove hovering state from insight bubbles

redesign navigation, make it more simple, introduce arrows for back and forth navigation

redesign DIAL button/starter

How would you describe the structure/layout of the simulation? (Let them explain the navigation & Ul elements, etc.)

"well structured first of all"

likes amount of content

"I do not know that Building Block here, never have seen something like this before"

"maybe there should be another layout" citizen vs. Servant, difference?

Would like to see more accessible UI

feeling of security

simplify eGov sim. Content, remove stuff that's not needed to understand BBs

Please describe in your own words – what is a building block?

"Why is one highlighted, other is not?"

What is BB? "BB is specifically designed for people who are interested in this product", "not clear if it's part of the program or not", "can imagine it's part of the program"

"how would you describe BB role for page you are on?" - "it's not really clear", description in catalog is understood, but relation to page not clear to him, connection missing for him 

"it's really hard to catch", not easily linkable

doesn't understand why they are linked and about their roles

purpose of side panel? "is to structure major parts of app, in not confusing way, if I want to go to payment it gives me easier access to go to application parts quickly"

"as I click it connects something in panel, guess it has something to do with main structure of interface

What is BB? "...are major objectives of this prototype/app based on digital services" "have to do with major milestones to get where they want to be"

see above

DIAL link to BB explanation

legend for BB activity, for example traffic light system

connect active insights to BB overview

think if we can show that they can be used for other use cases

show more clearly that initial status of BBs shown relates to the whole page

How can you find out more about the information mediator and what is its role?

"no idea how I can learn more and how they relate to each other", confuses her at the moment

it's connected to frontend

intro can tell user with which perspective user is starting, too

introduce more properly to the first page

How would you describe the role of the building blocks for this page?

not sure which side was shown, recommends to make it clearer!

"major things that are structuring the page" (after clarification of Artun)

like foundations and design elements of this work

OTHER

additional question from jonas at end: got the connection after clicking and found it very helpful (seemed like the missing puzzle piece)

BB term not telling, needs more explanation

Progress - don't think he recognized progress bar, was referring to content when asked for progress of phases

(Context change)

What changed on the screen? Could you explain it in your own words.

didn't realize perspective change

What does this change indicate for you?

"gives sense of security" (referring to login)

Where would you expect to see the perspective changes on the screen?

Make perspective more clear, "not common that it (perspectives) looks the same"

??? 47:30, didn't fully get it 

End question:

How would you describe this product/simulation in your own words?

"think it's quite useful" "trapped me on the wrong foot" "if it's not 100% clear what you are looking at, confusion stays"

prototypes he liked most in this context were those who had a straight aim, perspective, etc., simulation is nice, realistic

but beginning, advertised something I couldn't find in simulation

visualization how BB help me ... XX?!

"hard to understand how it works and how to use it as civil servant"

doesn't know how it can help to build eServices

if it would help me to find functionalities for my services it would be helpful

"hard to understand in the beginning"

expectation was different, but thinks this can give perspective

even though it's not part of test, we should try to design proper offboarding

If this was a real website, how would you rate the technical maturity of this product? Were there any indications of this?

8 of 10

85%, because transitions are not perfect

make clear it's only for simulation purposes, make sure they understand there's no backend BUT BB activity is showing backend activity

@

After experiencing the simulation - did you feel well introduced to it by the website and tutorial?

onboarding page was more comprehensive, logical connection was given,

details missing on onboarding page

visual vs. Text? -> "visual, good graphics with few words"

Offboarding:

Thank you very much for these great insights and of course for taking the time to help us!

Do you have any questions or feedback for us?

OTHER

"Would you share this?" - yes
they have principle of four eyes, so they like to share, and he would share it with colleagues