Alpha: Selenium v2

[cullzie]

Opening this discussion for anything that might arise during testing on the Alpha branch.
Please feel free to raise any issues found here. Thanks 🚀

[ninjatall12]

Still get error if i try from cmd i get error no module named selenium even thought i did pip install -r requirements.txt and saw it installed

[cullzie]

Hey @ninjatall12
Can you share the command you ran in cli and full output please? The more info the better

[ninjatall12]

managed to fix that error now i get this.

[nschepsen]

@ninjatall12 Do you pass --browser=chrome to the app?

[ninjatall12]

i tried it and got same thing

[lsgIT-HH]

Hi, I just tried with the Alpha branch and it seemed to work, although it did not find any unenrolled courses, what did not happen ever before...
But I saw starting Chrome and going through a lot of courses.

[cullzie]

Ye there are still some issues with the Alpha. Possibly some of the checks are no longer working. I pretty much took the old version of the UI enroller and re-jigged it to avoid the bot detection

What output are you seeing in the cli?
Are you receiving emails from Udemy to say you have subscribed?

[lsgIT-HH]

This is the whole output I get after a complete runthrough.
But I do not get any mails from Udemy as no new courses have been found. When the tool (latest release) was still working, it always found new courses...

Loading existing settings
[WDM] - 

[WDM] - ====== WebDriver manager ======
[WDM] - Current google-chrome version is 94.0.4606
[WDM] - Get LATEST driver version for 94.0.4606
[WDM] - Get LATEST driver version for 94.0.4606
[WDM] - Trying to download new driver from https://chromedriver.storage.googleapis.com/94.0.4606.61/chromedriver_mac64.zip
[WDM] - Driver has been saved in cache [/Users/lsgit/.wdm/drivers/chromedriver/mac64/94.0.4606.61]
freebiesglobal scraper is running
tutorialbar scraper is running
discudemy scraper is running
coursevania scraper is running
Page: 1 of 1078 scraped from tutorialbar.com
Error while running tutorialbar scraper: 'NoneType' object has no attribute 'find'
Page: 1 of 471 scraped from discudemy.com
Got 15 links from https://discudemy.com in 4.65 seconds
Error while running coursevania scraper: 'int' object has no attribute 'get'
Page: 1 of 1443 scraped from freebiesglobal.com
Got 29 links from https://freebiesglobal.com in 24.09 seconds
freebiesglobal scraper is running
tutorialbar scraper is running
discudemy scraper is running
coursevania scraper is running
Page: 2 of 1078 scraped from tutorialbar.com
Got 12 links from https://www.tutorialbar.com in 1.17 seconds
Page: 2 of 471 scraped from discudemy.com
Got 7 links from https://discudemy.com in 4.42 seconds
Error while running coursevania scraper: 'int' object has no attribute 'get'
Page: 2 of 1443 scraped from freebiesglobal.com
Got 30 links from https://freebiesglobal.com in 13.54 seconds
freebiesglobal scraper is running
tutorialbar scraper is running
discudemy scraper is running
coursevania scraper is running
Page: 3 of 1078 scraped from tutorialbar.com
Error while running coursevania scraper: 'int' object has no attribute 'get'
Got 12 links from https://www.tutorialbar.com in 1.26 seconds
Page: 3 of 471 scraped from discudemy.com
Got 0 links from https://discudemy.com in 3.99 seconds
Page: 3 of 1443 scraped from freebiesglobal.com
Got 30 links from https://freebiesglobal.com in 15.28 seconds
freebiesglobal scraper is running
tutorialbar scraper is running
discudemy scraper is running
coursevania scraper is running
Page: 4 of 1078 scraped from tutorialbar.com
Got 12 links from https://www.tutorialbar.com in 1.63 seconds
Page: 4 of 471 scraped from discudemy.com
Got 0 links from https://discudemy.com in 2.56 seconds
Error while running coursevania scraper: 'int' object has no attribute 'get'
Page: 4 of 1443 scraped from freebiesglobal.com
Got 30 links from https://freebiesglobal.com in 17.32 seconds
freebiesglobal scraper is running
tutorialbar scraper is running
discudemy scraper is running
coursevania scraper is running
Page: 5 of 1078 scraped from tutorialbar.com
Stopping loop. We have reached max number of pages to scrape: 5
tutorialbar scraper complete
Got 12 links from https://www.tutorialbar.com in 1.67 seconds
Error while running coursevania scraper: 'int' object has no attribute 'get'
Page: 5 of 471 scraped from discudemy.com
Stopping loop. We have reached max number of pages to scrape: 5
discudemy scraper complete
Got 0 links from https://discudemy.com in 4.30 seconds
Page: 5 of 1443 scraped from freebiesglobal.com
Stopping loop. We have reached max number of pages to scrape: 5
freebiesglobal scraper complete
Got 30 links from https://freebiesglobal.com in 20.10 seconds
coursevania scraper is running
Error while running coursevania scraper: 'int' object has no attribute 'get'
==================Run Statistics==================
Enrolled:                   0
Unwanted Category:          0
Unwanted Language:          219
Already Claimed:            0
Expired:                    0
Total Enrolments:           0
Savings:                    $0.00
==================Run Statistics==================
All scrapers complete
Closing browser

[cullzie]

Ye I didn't check validation. Probably is just some outdated xpaths.
Also the coursevania scraper is broken. That issue can wait though 😄
Thanks

[nschepsen]

The scraper gets local language titles instead of simplified english ones, e.g. Englisch for English

[infantannihilat0r]

works on manjaro + chromium. Thank you @cullzie

[cullzie]

Pushed some updates and fixes to alpha branch. Hopefully it should improve the experience for you all.

[Pos7alDude]

Here are my experiences with the new alpha (using chrome):

• udemy profile has to be set to English language before, otherwise it does not find the "enroll now" button
• the script works too fast! It skips many courses when the site has not already loaded completely and it already scans for the enroll-button. I see the button and code appearing for less than a second when the script already skips the course and loads the next one. I'm testing with 100M internet connection. Sometimes there are only 2 to 3 seconds until it skips.
• sometimes it calls the "purchase" button when a discount/code has expired. I have no credit card saved in my Udemy account so I cannot say if it would buy the course or stops at this point

It's very frustrating to see a lot of free courses being skipped so it would be great to let the script pause for some more seconds before checking if a course can be enrolled.

Just with the code pulled today.

[infantannihilat0r]

the script works too fast! [...] Sometimes there are only 2 to 3 seconds until it skips.

I can't confirm that. The WebDriver timeout is set to 10 seconds. I guess the origin of this problem is different. The switching to the english UI doesn't change the currency representation, or at least I am the only one who has this problem. So, the scraper tries to validate and then to format the course price and gets failed. He expects the raw price string e.g. $1,000.0 but gets 1.000,0 €. Thereupon he raises an exception and goes to the next course. The fix I'm using is formatted_price = float(raw_price[:-1].replace(',', '.'))

It does happen, I was watching it a couple of times and for some reason it does skip free courses :) Regardless, great work :)

[cullzie]

1. I'm not sure on the languages. It shouldn't be reliant on a specific xpath unless it differs by language but its something I'll keep an eye on
2. It shouldn't be loading too fast. In fact where it was doing that I added an explicit wait. There's still some tweaking to do though so I'm sure that this case is possible somewhere.
3. It calls the purchase since the price check is only done on the final screen at the moment. It shouldn't go through if the course is no longer free. I might be able to move it forward one screen but it takes time which I don't have much of at the moment 😞
4. Ye the price determination logic isn't great. I knew that localization's would be an issue. Its something I need to spend some more time on

[cullzie]

@Pos7alDude, @nschepsen and @infantannihilat0r Thanks for the feedback 👍

[cullzie]

@infantannihilat0r @nschepsen @Pos7alDude I've added a fix for the prices. That should fix the skipping free courses issue. Realized the 2 were linked just now 🤭

[cullzie]

New package was added https://pypi.org/project/price-parser/ so you will need to pip install the requirements or use poetry.

[ninjatall12]

I still get this i have deleted cookies and waited days im based in uk

[infantannihilat0r]

switch to alpha branch, update, then run with --bowser=chromium flag

[ninjatall12]

this worked

[cullzie]

@aapatre @fakeid30 Do you think we should cut a release and deprecate the REST based enrollment?
I don't see a simple way to get it working again.
I'll have a look at some of the issues raised here and see if I can get them sorted before a release

[fakeid30]

yes let's do it

[cullzie]

@fakeid30 Just trying to sort some of the bigger issues. Will try get them done as soon as I can

[cullzie]

Released now 🥳

[fakeid30]

Really good job bro. Your dedication to this project is commendable. We all owe you a beer. Or a chorus of "I'll tell me ma", the seminal Irish song, if we meet.

[cullzie]

Cheers 🍻 @fakeid30

[Pos7alDude]

It's still skipping free courses. Sometimes the script seems to run smoothly but then again it calls a few links in a short period of time (a bit more than one link per second). The site normally needs about two seconds to load completely so if it calls the links too fast I can see the "enroll" button for a few milliseconds while it's already loading the next course.
First I thought it may skip these because of unwanted category or language (these information are displayed faster than the rest of the site) and the script is even faster than I may check all Information with my eyes but if I run the script again immediately, it enrolls a few courses which it seems to have skipped before.

Also it still tries to enroll paid courses but that may be caused by my currency setting (EUR not USD). Otherwise I don't really care about that because it does not affect the functionality.

[cullzie]

It won't actually enrol in courses which cost money. It does go to the credit card screen because as in some regions it is a requirement that zipcode be entered.
The missed courses is possible if it is moving too quickly. Its not something I've observed recently though

[contextua1]

how do i install the alpha? im using python in anaconda

[cullzie]

You can get the latest version from pypi now which has these changes

[marain87]

I had made PR #346 to handle Free courses which no coupon code and no add to cart button on the course page.

Seem that the Free course page "Enroll now" and "Go to course" buttons with the exact same attributes, I made checking on span wording for English language only.

Hope someone could modify it for other language users.

[hoeni]

Alpha worked fine here (MacOS Monterey 12.0.1, Python 3.9.7/brew) and successfully enrolled me in a few courses 🎉 (after manually installing price-parser with pip).

[cullzie]

I've released a version to master and pypi now.
Can open bugs as normal from now on 😄

[Pos7alDude]

Cloned today from Master, the script waits several minutes after Got 12 links from https://discudemy.com in 5.14 seconds, I first thought it will not continue anymore. It seems like tutorialbar.com has some server issues, hopefully only today.
Error in get request: Cannot connect to host www.tutorialbar.com:443 ssl:default [Connect call failed ('192.241.140.217', 443)] Removing ad links from courses: {'https://bit.ly/3c8XlzC'} Got 10 links from https://www.tutorialbar.com in 130.58 seconds

Since the Selenium version keeps skipping many courses caused by "unwanted category" and "unwanted language" I'm wondering if maybe the config file has to be changed. I got the category names from the API documentation when using the old script version:
https://www.udemy.com/developers/affiliate/models/course-category/

my current config file:
udemy:
categories: [Development, Design, IT & Software, Software Engineering, Photography & Video, Business, Office Productivity, Personal Development, Teaching & Academics]
email: XXXXXXXXXXXXXX
languages: [English, German]
password: XXXXXXXX
zipcode: 'XXXXX'

Finally the logfile: app.log
Trying alpha again.