download inVideo quiz #683

coiby · 2018-07-05T14:14:15Z

Proposed changes

Download inVideo quiz and save it as a HTML file.

For example the video downloaded from https://www.coursera.org/learn/algorithms-part1/lecture/buZPh/course-introduction has the name 03_course-introduction.mp4. The name of HTML file will be 03_course-introduction_inVideoQuiz.html.

Implementing the feature of inVideo quiz is similar to implementing existing feature of quiz. So only some tweaks on existing code to deal with

a slightly different json structure
more question types

Types of changes

What types of changes does your code introduce?
Put an x in the boxes that apply

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating
the PR. If you're unsure about any of them, don't hesitate to ask. We're here
to help! This is simply a reminder of what we are going to look for before
merging your code.

I have read the CONTRIBUTING doc
I agree to contribute my changes under the project's LICENSE
I have checked that the unit tests pass locally with my changes
I have checked the style of the new code (lint/pep).
I have added tests that prove my fix is effective or that my feature works (but manually tested for the courses algorithms-part1 and learning-how-to-learn)
I have added necessary documentation (if appropriate)

Further comments

Reviewers

coveralls · 2018-07-05T14:16:12Z

Coverage increased (+0.9%) to 74.813% when pulling f37497d on coiby:master into a82edb9 on coursera-dl:master.

coveralls · 2018-07-05T14:16:12Z

Coverage decreased (-0.5%) to 73.483% when pulling 48f1c68 on coiby:master into a82edb9 on coursera-dl:master.

rbrito · 2018-07-06T20:57:04Z

I will take a look at your patch soon... If I don't manage to look at it until this next Wednesday, please, ping me.

Thanks,

Rogério.

balta2ar

Good job! Could you fix the things I mentioned and add tests?

balta2ar · 2018-07-06T21:26:46Z

coursera/api.py

+
+            # Question number and (if invideo) when quiz appear in the video
+
+            if invideo:


could you put this "if .. else" into a helper function? e.g.:

def _get_queue_point(question_json)

and then

cue_point = _get_queue_point(question_json) if invideo else ""

balta2ar · 2018-07-06T21:27:13Z

coursera/api.py

+               minutes = int(seconds / 60)
+               seconds = int(seconds % 60)
+               if minutes == 0:
+                   cuePoint = " at 0′{}″".format(seconds)


this below should be: cuePoint -> cue_point

balta2ar · 2018-07-06T21:28:03Z

coursera/api.py

-            return self._convert_quiz_json_to_links(quiz_json, 'quiz')
+            session_id = self._get_quiz_session_id(quiz_id, invideo)
+            quiz_json = self._get_quiz_json(quiz_id, session_id, invideo)
+            if invideo:


I'd prefer a one-liner, e.g.: tag = 'inVideoQuiz' if invideo else 'quiz'

1. one-liner style and use underscore naming 2. a separate helper method for getting&formatting cuePoint

coiby · 2018-07-08T15:44:00Z

@balta2ar @rbrito Thank you for reviewing the code. The suggested changes have been applied. As for the tests, I will add them in two days after getting more familiar with Python testing.

fix error UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 19: ordinal not in range(128)

fix error UnicodeDecodeError: 'ascii' codec can't decode byte

When testing implemented, invideo quiz feature for Python 2.7, I found the non-ascii characters in json file will lead to the error "UnicodeDecodeError: 'ascii' codec can't decode byte xxx in position 0: ordinal not in range(128)". Thus the tests will fail. To make the code compatible with Python 2.7, the solution is to treat every string as Unicode as suggested by https://www.azavea.com/blog/2014/03/24/solving-unicode-problems-in-python-2-7/. Before the fix, strings have mixed types of `str` and `Unicode`, for example, ``` (<type 'str'>, '<form>') (<type 'unicode'>, u'<label><input type="checkbox" name="0"><co-content>\n <span>\n Did you make a serious effort to understand the text?\n </span>\n</co-content><br></label>') ... (<type 'unicode'>, u'<label><input type="checkbox" name="0"><co-content>\n <span>\n Did you go over the study guide and problems with classmates and quiz one another?\n </span>\n</co-content><br></label>') (<type 'unicode'>, u'<label><input type="checkbox" name="0"><co-content>\n <span>\n If there was a review session, did you attend and ask questions about any concepts or ideas that you were unsure of?\n </span>\n</co-content><br></label>') (<type 'unicode'>, u'<label><input type="checkbox" name="0"><co-content>\n <span>\n Did you get a reasonable night\u2019s sleep before the test?\n </span>\n</co-content><br></label>') (<type 'str'>, '</form>') (<type 'str'>, '<hr>') ``` Now every string has the type `Unicode` in Python 2.7 which fixes UnicodeDecodeError.

… poll type of invideo quiz

coiby

Thank you! I've finished adding tests.

coiby · 2018-07-16T08:18:47Z

@rbrito @balta2ar More tests have been added to pass coverage tests. Btw, it seems there is something wrong with coveralls.io. For two almost identical commits, https://coveralls.io/builds/17935660 just adds more parameters to pytest.mark.parametrize than https://coveralls.io/builds/17934960. However, coveralls.io gives different coverage rates while the local coverage results shows the coverage rates are the same.

coiby · 2018-07-24T04:59:07Z

@balta2ar @rbrito Last week, I tried to improve this feature by also downloading feedbacks for the invideo quiz. But once the feedback for the correct answer is fetched, we have to refresh the web page to fetch remaining feedbacks, i.e. a new session has to be started. Otherwise coursera's API will forbid it by throwing a 409 error. I wonder how much overload will re-starting a new session for each question bring to the server? It is ethical?

balta2ar · 2018-07-24T20:24:06Z

I'm getting the following errors in your branch (learning-how-to-learn course):

coursera_dl version 0.11.4
Logged in on coursera.org.
Downloading class: learning-how-to-learn (1 / 1)
Parsing syllabus of on-demand course (id=GdeNrll1EeSROyIACtiVvg). This may take some time, please be patient ...
Processing module  what-is-learning
Processing section     introduction-to-focused-and-diffuse-modes
Processing lecture         1-1-introduction-to-the-focused-and-diffuse-modes (lecture)
Processing lecture         1-2-terrence-sejnowski-and-barbara-oakley-introduction-to-the-course-structure (lecture)
Error 404 Client Error: Not Found for url: https://api.coursera.org/api/opencourse.v1/user/4958/course/learning-how-to-learn/item/1bYD5/lecture/inVideoQuiz/session
 getting page https://api.coursera.org/api/opencourse.v1/user/4958/course/learning-how-to-learn/item/1bYD5/lecture/inVideoQuiz/session
The server replied: {"errorCode":null,"message":"Assessment id not available","details":null}
Could not download quiz 1bYD5: 404 Client Error: Not Found for url: https://api.coursera.org/api/opencourse.v1/user/4958/course/learning-how-to-learn/item/1bYD5/le
cture/inVideoQuiz/session
Processing lecture         1-3-using-the-focused-and-diffuse-modes-or-a-little-dali-will-do-you (lecture)
Processing lecture         1-4-what-is-learning (lecture)
Processing lecture         fun-introductory-quiz (quiz)
Processing section     procrastination-memory-and-sleep
Processing lecture         1-5-a-procrastination-preview (lecture)
Processing lecture         1-6-practice-makes-permanent (lecture)
Processing lecture         1-7-introduction-to-memory (lecture)
Processing lecture         1-8-the-importance-of-sleep-in-learning (lecture)
Processing lecture         1-9-interview-with-dr-terrence-sejnowski (lecture)
Error 404 Client Error: Not Found for url: https://api.coursera.org/api/opencourse.v1/user/4958/course/learning-how-to-learn/item/8IUbH/lecture/inVideoQuiz/session
 getting page https://api.coursera.org/api/opencourse.v1/user/4958/course/learning-how-to-learn/item/8IUbH/lecture/inVideoQuiz/session
The server replied: {"errorCode":null,"message":"Assessment id not available","details":null}
Could not download quiz 8IUbH: 404 Client Error: Not Found for url: https://api.coursera.org/api/opencourse.v1/user/4958/course/learning-how-to-learn/item/8IUbH/le
cture/inVideoQuiz/session

Does it work for you? Similar errors in the master branch:

Processing lecture         what-is-learning (exam)                                                                                                                 
Error 403 Client Error: Forbidden for url: https://api.coursera.org/api/onDemandExamSessions.v1 getting page https://api.coursera.org/api/onDemandExamSessions.v1  
The server replied: {"errorCode":"Not Authorized","message":null,"details":null}                                                                                   
Could not download exam CtdDH: 403 Client Error: Forbidden for url: https://api.coursera.org/api/onDemandExamSessions.v1

balta2ar · 2018-07-24T20:26:36Z

I've switched sessions but the errors remained... Has their API changed again?

coiby · 2018-07-26T10:09:07Z

The first error is expected for the lectures which don't have invideo quiz. But this message may confuse the user. I'll modify my code to better handle this exception.

Error 404 Client Error: Not Found for url: https://api.coursera.org/api/opencourse.v1/user/4958/course/learning-how-to-learn/item/8IUbH/lecture/inVideoQuiz/session
getting page https://api.coursera.org/api/opencourse.v1/user/4958/course/learning-how-to-learn/item/8IUbH/lecture/inVideoQuiz/session
The server replied: {"errorCode":null,"message":"Assessment id not available","details":null}
Could not download quiz 8IUbH: 404 Client Error: Not Found for url: https://api.coursera.org/api/opencourse.v1/user/4958/course/learning-how-to-learn/item/8IUbH/lecture/inVideoQuiz/session

As for the second error, I visited the assignment page and found such prompt,

The course has ended. Assignments may not be resubmitted.

This session has ended. To resubmit, enroll in the next session.

It seems once a course is finished, user can't view or re-submit the assignment.

…InVideoAssessment; sort invideo quizz questions by time

coiby · 2018-07-28T12:52:43Z

It turns out coursera's API has already provided the info on whether a lecture has invideo quiz.

{
        "contentSummary": {
          "typeName": "lecture",
          "definition": {
            "hasInVideoAssessment": true,
            "duration": 562882
          }
        },

My latest commit modifies the code to make use of this info. And it also sort the questions by the time they appear in the video.

download inVideo quiz

48f1c68

balta2ar requested changes Jul 6, 2018

View reviewed changes

Update api.py

f7ceb67

1. one-liner style and use underscore naming 2. a separate helper method for getting&formatting cuePoint

coiby and others added 8 commits July 10, 2018 23:39

Create question-type-checkboxpoll-input_invideo.json

0ab16ca

add invideo quiz json files

68f71bd

add tests for invideo quiz

283cac9

Update question-type-checkboxpoll-input_invideo.json

7fac588

Update question-type-checkboxpoll-input_invideo.json

cdf0f5e

fix error UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 19: ordinal not in range(128)

Update question-type-continue-input_invideo.json

b495c42

fix error UnicodeDecodeError: 'ascii' codec can't decode byte

fixing the error of uploading poll type josn as continue type and add…

2f8ee11

… poll type of invideo quiz

coiby commented Jul 11, 2018

View reviewed changes

coiby added 2 commits July 16, 2018 15:54

test _get_cue_point format; test lectures with no invideo quiz

462e7fd

fix error caused by Non-ASCII character for Python2.7

e53613d

determine whether to download invideo quizz together with the key has…

f37497d

…InVideoAssessment; sort invideo quizz questions by time

coiby mentioned this pull request Apr 8, 2019

Caching Proxy: A different approach to use coursera off-line #716

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

download inVideo quiz #683

download inVideo quiz #683

coiby commented Jul 5, 2018 •

edited

coveralls commented Jul 5, 2018 •

edited

coveralls commented Jul 5, 2018

rbrito commented Jul 6, 2018

balta2ar left a comment

balta2ar Jul 6, 2018

balta2ar Jul 6, 2018

balta2ar Jul 6, 2018

coiby commented Jul 8, 2018

coiby left a comment

coiby commented Jul 16, 2018

coiby commented Jul 24, 2018

balta2ar commented Jul 24, 2018

balta2ar commented Jul 24, 2018

coiby commented Jul 26, 2018 •

edited

coiby commented Jul 28, 2018


		# Question number and (if invideo) when quiz appear in the video

		if invideo:

download inVideo quiz #683

Are you sure you want to change the base?

download inVideo quiz #683

Conversation

coiby commented Jul 5, 2018 • edited

Proposed changes

Types of changes

Checklist

Further comments

Reviewers

coveralls commented Jul 5, 2018 • edited

coveralls commented Jul 5, 2018

rbrito commented Jul 6, 2018

balta2ar left a comment

Choose a reason for hiding this comment

balta2ar Jul 6, 2018

Choose a reason for hiding this comment

balta2ar Jul 6, 2018

Choose a reason for hiding this comment

balta2ar Jul 6, 2018

Choose a reason for hiding this comment

coiby commented Jul 8, 2018

coiby left a comment

Choose a reason for hiding this comment

coiby commented Jul 16, 2018

coiby commented Jul 24, 2018

balta2ar commented Jul 24, 2018

balta2ar commented Jul 24, 2018

coiby commented Jul 26, 2018 • edited

coiby commented Jul 28, 2018

coiby commented Jul 5, 2018 •

edited

coveralls commented Jul 5, 2018 •

edited

coiby commented Jul 26, 2018 •

edited