CrowdMark--lost manual

Why CrowdMark

CM is a life-saver for large classes because it almost solves some problems:

  1. Accellates grading.
  2. Strudents cannot get their marks back since they either do not meet an instructor or TA or their work is misplaced after marked.
  3. Cheating when student adds something to his work and claims I did this!
  4. Cheating because students who sit nearby have close biooklets numbers and they are graded by default according to these numbers.
  5. Bloody mess at appeals: with CM all appeals go to Class Coordinator or any designated Test suprervisor and they trake much shorter time to process.


There are different errors:

  1. Page QR-code is not read (the possibility in the incorrect reading seems to be negligible. I never encountered it). Such errors are fixed manually really promptly.
  2. Server-side OCR cannot identify the student. Such errors are fixed manually really promptly. This usually happens when the email does not match the name 1;

  3. Server-side OCR misidentifies the student and misatributes the work. This is the worst possible error which happens rarely, and usually happens when the email does not match the name.

  4. Some pages are not scanned properly and should be rescanned. See below fail-safe.


  1. While CM server allows uploading pages in any order sometimes (very seldom) one needs to access the physical booklet. Therefore before cutting corners it is a good practice to sort booklets according to their numbers.
  2. During tests and exam while collecting signatures we also record booklet numbers. We also put booklet numbers for students who write with T&ES. Since the signature page from Dean's office does not contain such field, we make our own list. So, on Exam there are two lists–one for signatures from Dean's Office and one for number recording. (Dean's Office list is piece of junk: (a) They make it well in advance and it contains many names which are not in the class anymore. (b) Fonts are too small, vertical spaces insufficient, and they often make all bold. (c) The students are not numbered, each page contains unknown number of names. I create the list at the morning of exam or a night before, 14 points, sufficient spacing, numbers, 25 names per page (the lat pages of the groups are smaller if we divide students into groups according to locations)
  3. During Quizzes students are asked to record their booklet numbers.
  4. Using booklet numbers we can very easily locate booklets on CM and in the physical pile (since we sort). Some students fail to record.
  5. When marks are exported from CM to Quercus we at the same time send a Quercus announcement "Marks for Assesment X are posted. If you got email from CM and your mark is not on Quercus or there is a wrong please advise Class Coordinator and it will be done manually. If you wrote X and have not received email from CM, please advise Class Coordinator. In any case please indicate your booklet number.
  6. Actually on Tests and Exam I do an extra check: I export CM to csv file, it is ordered by booklet number; I reorder it by name and compare booklet numbers in this file with recorded numbers. For exam I consider it mandatory since CM marks are released when Dean's Office allows to do this and the misattribution error is not reported by students until then.

CM limitations and work-arounds

The major limitation of CM out-of-the-box is that each assessment has one variant. If I have several variants they recommend to make a separate CM assesment for each variant.

This hurts. Why? Assume that I have 9 variants for each Quiz (I have 16 tutorials offered during at 9 different time slots) this means 9 x 7 assessments for quizzes and

The work-around for the second problem is not to export marks to Quercus, but to download to csv files and then manually upload them to a Quercus the same mark column. I never tried this since I found much better solution. And it does not address the first problem anyway.

Mich better solution

How to create BIG pdf file to give to PrintShop? CM recommends:

make your pdf file , upload to server, say how many students, put full marks to problems. Then we clone this pdf file, put on each page QR-code and we give you this big file to download.

We do exactly this but the file we upload to the server has the same number of pages but is either empty or contains only common elements. Then we download this file and postprocess. What is it? My smart LaTeX script overprints each booklet by a corresponding problem set.

Easy case

I want to give 9 different versions of Quiz so each tutorial gets its own version. Then 1-100 get variant 1 for TUT0101, 101--200 Variant 1 for TUT0102 in the same time slot, 201--300 get variant 2 for TUT0201, ... . (I really want each tutorial section start from a good number). Finally I remove many unneeded booklets and go to Printshop.

Actually some booklets are not just removed but taken to a separate file which I emauil to T&ES if requested.

Ditto for Tests with Morning Sitting, Noon Sitting, may be Afternoon Sitting, Main (Night) Sitting and Deferred Sitting.

My script also prints Booklet Number (it knows it!) by Huge Digits in the top left corner of the front page, so students can record it easily.

Complicated case

Some (too many) students cheat on Quizzes. They cheat a lot. I doubt if this gives them a big unfair advantage but other students think so. And it is a huge demotivator. I worked so hard, and these cheaters...

So I give in each section several variants (it means A LOT work for me and a bit of extra work for TAs). And there are two seemingly conflicting requirements:

  1. I want in pdf and thus in the pile of printed Quizzes it was like this 1-st variant, 2-nd variant, ..., 5-th variant, 1-st variant, ..... (until booklet 100)
  2. TAs want booklets 1-20 be 1st variant, 21-40 2-nd variant ,... because the default grading is in the booklets order and they want to grade 1-st variant first, ... and they do not want to jump to the next booklet of the same variant.

My supersmart LaTeX script 2 overprints each booklet by a corresponding problem set, prints booklet numbers in huge digits and also shuffles the bookets. As a result in pdf file (and in the PrintShop delivery)the sequence is Booklet 1 with variant 1 Booklet 21 with variant 2 Booklet 41 with variant 3 Booklet 61 with variant 4 Booklet 81 with variant 5 Booklet 2 with variant 1 Booklet 22 with variant 2 .....

When I first gave (during the lectures) the multiversion quiz one student too smart for his own good approached me with the question if I know that different students got different problems 😂 I answered So what? And how do you know?!! Late some students expressed appreciation that cheaters were give a hard time. Does it eliminate cheating? Not completely but if you have two identiacal solutions of students sitting side-by-side I could not be sure if there was a cooperation or one of them simply snooped, now if I have two such solutions but there were say 2 students between them, I know that they cooperated and can give 0 to each without fear to punish an innocent.

Or, BTW, assume that I want to give them table of integrals. What is the best way to deliver? My LaTeX script also inserts these two pages in the end of each booklet. These pages do not have QR-codes and should not be uploaded to CM. Students are instructed to detach them from the test booklet (but some fail so one need to check the booklets before cutting corners).

People in PS (I have a good relations with them, we chat every time, were confused You say that the booklet has 16 pages but it says '1 out of 14'? 😕 I said look at the last 2 pages! They looked, saw no QR-code and said Oooh! You mathematicians are so smart!!! 😮.

Every man has a fool in his sleeve

Once I sent the same booklet into Printshop and to T&ES and both were used. What should I do?

You can guess which one I used.

  1. This should be forbidden, especially usage of non-utoronto mail servers as it violates privacy). 

  2. See sample