[zanata-users] Discussing RFEs [was: List-forking. A clarification about Zanata admin rights]

David Mason damason at redhat.com
Fri Sep 6 23:51:28 UTC 2013


----- Original Message -----
> From: "Sankarshan Mukhopadhyay" <sankarshan.mukhopadhyay at gmail.com>
> Cc: zanata-users at redhat.com
> Sent: Friday, 6 September, 2013 5:04:10 PM
> Subject: Re: [zanata-users] Discussing RFEs [was: List-forking. A clarification about Zanata admin rights]
> 
> I deeply appreciate that you took time to share a set of the RFEs. I
> went through them and, have a few comments, I'll in-line them against
> the respective RFEs.
> 
> On Fri, Sep 6, 2013 at 12:09 PM, David Mason <damason at redhat.com> wrote:
> > I have a few RFEs for the start of these changes (just a start of course):
> >
> >  - https://bugzilla.redhat.com/show_bug.cgi?id=919253
> 
> "User is able to enter not-accepted translations for any language,
> which would be queued for review by a user with higher permissions for
> that language."
> 
> This is an interesting piece. And, often during translation
> sprints/marathons this piece comes up. New users are unwilling to go
> through the process of signing up and, want to know why they cannot
> queue up translations for review and acceptance.
> 
> There is a particular reason why sign-up and acceptance into a team is
> often part of a language team's "new contributor orientation".
> Translations are expensive to maintain (in the same way, code is).
> And, incorrect or, inaccurate translations take a high toll on
> language communities who have to review (much in the same was as code
> review via systems like Gerrit often become a sore point due to the
> backlog it acquires; since reviewers are a scarce resource).
> 
> So, "seagull contributions" (ie. fly in, do something, fly away) are
> often more costly and looked at askance by language communities. The
> ability to sign-up and participate ensures that the coordinator can
> track existing contributors, they can assign tasks and, also do
> capacity planning. As awkward as it sounds, for language communities
> to be able to deliver a specific degree of coverage, language
> coordinators are effectively program managers who deal with a lot of
> moving pieces. Having unknown/untrained contributors put in small
> segments, even if they are intended to be qualifying tests, is an
> expensive proposition.

I agree that a review queue is probably not the best way to handle suggestions - these are evolving ideas, with plenty of room for improvement. The concept I have in mind at the moment is that suggested translations by someone with low reputation would be presented as an option for a more experienced translator to copy as a starting point or to use as-is, something like the way translation memory is presented at the moment. The intention is that the more experienced translator would be able to tell from looking at the suggestions for the first few strings in a document whether the suggestions from a particular user are of high enough quality for them to use, so that the overhead from lower quality suggestions is not high.

I think the concept is best communicated as a narrative - a bit long but I think it conveys the spirit of what I'm aiming for better than just listing implementation specifics:

Let us assume that 3 new users (let's call them A, B and C) have suggested translations for a language in for some or most of the strings in a document. An experienced translator (I'll call them T) then opens the document for translation. T focuses on the space for the first string and sees 3 suggestions next to the editor (and some TM matches, but I'll ignore those for brevity). The suggestion from B is just some rubbish, not even related to the translation, so T presses an [x] to hide that suggestion forever. The suggestion from A looks ok, but uses some unusual words - T just leaves it. The suggestion from C is quite good, so T clicks a button to copy it to the editor, changes a word, saves it as the translation and moves to the next string.
The next string has 2 suggestions. There's more rubbish from B, so T takes a moment to click B's name and and select the option to hide all of B's translations forever. The other suggestion is from A, which is again using some unusual word choices but is a decent grammatical structure, so T clicks to copy it to the editor, changes the words to be the more conventional words, saves it and moves to the next string.
The next string has a translation from C that looks exactly correct (and one from A, but T ignores that one since C seems to generally have higher quality), so T copies the string from C and saves it without any changes.
This continues through the document. T never sees any more rubbish from B. T uses a few translations from A (with modifications) but decides after a while that it feels slower than just writing them from scratch, so T uses an option to hide all translations from A temporarily (maybe for a few weeks or a month). For most of the strings, T uses translations from C with only small changes.

Now, there are some other sides to this story which is quite important.

First there is B's story: B is basically just a troll who is wasting people's time. B put rubbish on lots of documents in lots of projects, so a few other translators in addition to T chose to hide all of B's suggestions forever. This automatically reduces B's reputation, and it very quickly drops low enough for all of B's suggestions to be hidden from everyone (maybe B can still see them, but no one else's time is wasted).

Next there is C's story: C is an experienced translator who just hasn't used Zanata before so doesn't have any reputation in the system yet. Whenever T used one of C's translations without changing it, C's reputation automatically increased. When a suggestion was used but modified, C has a notification on their dashboard where they can see the difference between what they suggested and what was used (but does not get any reputation if the string was changed). Because T used enough strings without having to change them, C soon has enough reputation to have translator permissions in their language.

Finally there is A's story: A is a student translator who has studied for a semester or two and is trying to get some practice and learn. They carefully study all the notifications from when T used their suggestions with different word choices, and gets a better idea of which words are commonly used. A does not get enough reputation to have full translator permission yet, but the automatic feedback from T and other translators lets them improve. After about a month of making suggestions, finally one of A's suggestions is used without any changes, and A gets their first true reputation point towards getting translator permission. With this encouragement, A continues to make suggestions and study hard, and eventually has increasingly many suggestions used without modification, and gets full translation permission.

An important point I want to make is that a reputation system needs to be carefully tuned to make it work for the community and have a positive impact. The story above contains some important elements that I would like to be aiming for with a reputation system and other associated changes, and it is these and other important elements that I want to keep in mind when designing and adjusting such a system. I mention adjusting because this is the sort of thing that will be very difficult to get exactly right straight away, but can reach a good balance over time if we look carefully at how the community reacts to it.


I have responses to your other comments, but will leave those until Monday since I have a very full weekend.


Cheers,

David Mason
Software Engineer
L10n Engineering

Red Hat, Asia-Pacific Pty Ltd
Level 1, 193 North Quay
Brisbane 4000




More information about the zanata-users mailing list