Add a "role" of Judge and/or Relevance Engineer to a user

[epugh]

Is your feature request related to a problem? Please describe.
Today the only role we have is "Administrator". We'd like to be able to invite users and potentially grant them a role of Judge or Relevance engineer. Not applying either means they are both....

Describe the solution you'd like
Only show case listings to Relevance Engineers.
Show books to both.
Change the homepage to be more helpful on the Judging call to arms.
Maybe when inviting someone to be a judge, the homepage shows their books to be judged? Add a table view, similar to the case view? Think about how to surface what's hidden in https://go.quepidapp.com/books.
Maybe the dropdown for books should have "Books Needing Judgements" with a sub menu?
Need to enhance the invite UI.
Only show Cases Dropdown, Scorers, Notebooks to Relevance Engineers.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[matt-bernhardt]

I'm hoping it is okay to chime in on this issue, as the underlying question is something I think is going to be pretty relevant for our organization. We are hoping to enlist subject matter experts (academic librarians) to judge query-doc pairs, but these experts are going to have no role in the configuration of cases or analysis of results.

We recently asked some colleagues (a UX professional and two software engineers) to try the judging workflow, and one of the big pieces of feedback we got was that there's a lot of information on the interfaces that aren't relevant from the perspective of someone only trying to judge query-doc pairs. I'm finding myself wondering whether having a user role of "rater" - who can only see the judging workflows - is somewhere on the roadmap for this application?

It feels like this issue is starting to poke at that potential, so I wanted to comment here in case I'm understanding where this issue is headed.

[epugh]

Thanks for chiming in @matt-bernhardt.. So, I am always careful to say, while we have a "wishlist", we don't have a strong "roadmap". Since Quepid is open source, features get added generally by either the community or we (OSC) are doing a project where we need something added to quepid to meet the goals of the project, and then it gets done.

So with that caveat out of the way, I'd LOVE to get a much simpler rater experience. Do you have some mockups or other ideas on what would make a much nicer judging workflow?

[matt-bernhardt]

I think the biggest suggestion I have at the moment for a simpler rater experience is to introduce a role that can only do rating. Based on some testing we've recently done with some colleagues here, I think there may be some additional tweaks that could be helpful. More details below:

Background:
Our organization (a university library system) is preparing to engage with subject liaisons to do the judging of query-document pairs for relevance. These colleagues are not part of any other part of the relevance workflow (which is happening largely in our technology department by software engineers) - but they are the people in our organization who are best positioned to look at a search query, a document that was returned by our application, and to judge the relevance based on their experience with information-seeking in academic contexts.

As we prepare to reach out to these colleagues, and get our Quepid instance set up for them to use, we asked a few members of our project team - a few software engineers and one user experience professional - to run through both the bulk and individual judging workflows, to see where they ran into issues. Our goal is to try and prepare sufficient documentation to help the liaisons get up to speed with minimal confusion.

Feedback by these initial testers:
The biggest feedback we heard was that there is a lot on the interface that our colleagues don't need to worry about, and which only opens the door to possible problems. These liaisons only need access to the Books area of the application (not Cases, Teams, Scorers, or Notebooks). Even within a Book, they only need access to the two judgement workflows - so basically everything on the book navigation menu (Query/Doc Pairs, Judgement Stats, Judgements, Import, Share book, Export, and Settings) is unhelpful. They also don't need to navigate to other teams, or the related cases that make up that book.

Preference was split between the testers about whether the bulk or individual workflows were better. I don't think this is a problem, and it makes me glad that both exist. Each workflow elicited some specific feedback - not all of which may be actionable, but I'll let you decide:

Individual workflow
For staff who preferred the bulk workflow, one stated reason was about the context switching when moving from one query-doc pair to another. It tripped users up to have to remember to look at the query each time - because they wanted to just look at the documents, keeping the same query in mind (this is probably why they resonated so strongly with the bulk workflow)

We also heard that the number of interactive elements on the screen was a bit overwhelming. There are buttons for relevance scoring at the bottom, two buttons on the right side (very far from the others), an expanding panel for the scoring guidelines, and also the explanation text area. All of this, combined with the need to parse both the query and the set of metadata records we're dealing with, was a lot of things to get used to in order to feel fluent with the tool. The users thought that they would likely get faster over time, but the hurdle to reach fluency felt pretty high.

One user was tripped up a bit by the fact that clicking on the rating buttons also submits the form - they wanted to click to register their rating (the primary focus) and then move on to secondary questions like answering "why did you choose this?" - but the way the form is built, they have to fill in the secondary information first, and then click the most important button last.

On the other hand, we had one user who loved the individual workflow, and the keyboard shortcuts specifically - they found it very easy to use this workflow, scan briefly, and vote quickly.

(My editorializing on this point... I'd be interested in trying to separate the query and document in different interface boxes, rather than grouped in the same box. I'd also be interested in placing the "judge later" and "can't tell" buttons lower down, below the document, so that the "things users can click on" are grouped separately from the "query" and "documents" area. I will try to mock up what I'm thinking about, but can't promise that it will be something I get to in the next week)

Bulk workflow
Users missed the checkbox to "show explanations", and thought that in this workflow they just couldn't answer "why did you vote this way?". That's not a problem for our organization, as we aren't sure we're able to really respond to this type of feedback on an ongoing basis, however.

We also heard that the feedback after voting was pretty easy to miss - the "saved" flash message disappears pretty quickly, and if you aren't warned about what's happening the user risks not realizing that their vote is recorded.

(My editorializing - I do see that there's a persistent notification that a judgement has been saved - although again this is pretty faint. I'm not sure how best to balance the need for persistent status updates and a clean UI that isn't unnecessarily complicated. I wonder about a UI affordance like a toggle that starts with "unjudged" and flips to "judgement recorded" after pressing that button - so there's a predicable UI element that we initially see in one state, that changes to the other, rather than something that starts invisible and then appears)

General feedback
I don't want this point to get lost - every user we tested with was glad to have this tool, and wanted to clarify that they would likely get used to using it. They realized that they were unfamiliar with tools like "judgement", "cases", "books", and the practice of relevance measurement in this way - so they weren't prevented from taking part by what they experienced.

Some of the difficulties which we heard may also be down to how I helped prepare them for the system (or failed to do so). We are attempting to figure out how best to use this capability in general (and this application specficially) - and we may be making some basic mistakes that are tripping ourselves up.

---

Thanks for considering our experiences. I'm going to try and mock up some thoughts about an interface, but can't promise anything on a given timeline.

[epugh]

@matt-bernhardt I think that these changes would be pretty straightforward to add.... And honestly, could be done with your AI coding buddy.

We already have a user.admin property that is a boolean. To simplify life, you could just add one more columns to the user table, user.judge. By default it would be null or zero. However, in the UI where you are adding people to your team, you can invite people. You could tweak that UI to be "Invite Bob to my team" or "Invite Bob as a Judge to my team". That would pick the setting.

Then, on the various UI's, we would conditionally show things if someone wasn't a judge, like the Cases drop down, and the homepage could be different.

I think some mockups would be very good to help articulate what you are looking for. Also, I wonder if there are some ideas on how to progressively show features in the judging UI? Or hide some things under a menu?