Iss1019feat #1030

yosoycentli · 2022-01-14T23:53:29Z

This Pull Request add the next feature related with the 1019 issue (#1019):

You can see data from 30 European countries related to the different COVID-19 variants, as you can see in the pictures.

Source of the data: https://www.ecdc.europa.eu/en/publications-data/download-todays-data-geographic-distribution-covid-19-cases-worldwide

There are two new endpoints:

/v3/covid-19/variants/countries/

/v3/covid-19/variants/countries/{country}

I'm still working on the test for this scraper, the swagger documentation and I need to remove the instance "Redis commander" that I install in order to see what I load to the Redis database http://joeferner.github.io/redis-commander/.

I'm sure that there will be comments and modifications requests from the maintainers but I think is a good start what I've done here.

Best Regards Centli Garcés

…e api_gov.spec.js and getGovData.js files

…etGovData.js since the variant scraper doesn't belong in there

…er with the endpoint v3/covid-19/variants/countries/{country}

ebwinters

The code looks pretty good, similar to the rest of the code base. Left a few comments, when you are ready to publish (and have docs+tests) I will review again. Thanks for the work here, I'm sure it will be picked up and used by people around the world!

ebwinters · 2022-01-15T14:30:32Z

routes/v3/covid-19/apiVariants.js

+
+router.get('/v3/covid-19/variants/countries/:country?', async (req, res) => {
+	const { allowNull } = req.query;
+	const { country: countryName, yearWeek, variant  } = req.params;


I don't see the yearWeek and variant parameters being used

I put them there since I wanted the API had to have these parameters, but I think it's better to remove them and later when they're created added again.

ebwinters · 2022-01-15T14:31:00Z

routes/v3/covid-19/apiVariants.js

+// *to here*
+
+// Take this code below as an example:
+// router.get('/v3/covid-19/countries/:query', async (req, res) => {
+// 	const { yesterday, twoDaysAgo, strict, allowNull } = req.query;
+// 	const { query } = req.params;
+// 	let countries = JSON.parse(await redis.get(wordToBoolean(yesterday) ? keys.yesterday_countries : wordToBoolean(twoDaysAgo) ? keys.twoDaysAgo_countries : keys.countries))
+// 		.filter(country => country.country.toLowerCase() !== 'world').map(fixApostrophe);
+// 	countries = splitQuery(query)
+// 		.map(country => nameUtils.getWorldometersData(countries, country, strict !== 'false'))
+// 		.filter(value => value).map(country => !wordToBoolean(allowNull) ? nameUtils.transformNull(country) : country);
+// 	if (countries.length > 0) res.send(countries.length === 1 ? countries[0] : countries);
+// 	else res.status(404).send({ message: 'Country not found or doesn\'t have any cases' });
+// });


remove once ready to publish draft

ebwinters · 2022-01-15T14:34:35Z

scrapers/covid-19/getVariants.js

+const csvUtils = require("../../utils/csvUtils");
+
+const PATH =
+    "https://opendata.ecdc.europa.eu/covid19/virusvariant/csv/data.csv";


Keep this in mind when writing the Swagger docs

ebwinters · 2022-01-15T14:36:07Z

scrapers/covid-19/getVariants.js

+                return obj;
+            }, {});
+
+        console.log(dataByCountry);


Remove when publishing draft, or alternatively put a log statement with the length of the result

You're right, I'll remove this "console.log()"

Thanks for checking this.

ebwinters · 2022-01-15T14:36:57Z

tests/v3/covid-19/api_gov.spec.js

@@ -71,6 +71,34 @@ describe.skip('TESTING /v3/covid-19/gov/canada', () => {
 	});
 });

+// here


Put into own testing file

You're right, this new variants endpoint needs its own test file, I'm working on that.

…n apiVariants.js file since they're never use

…og and commented code from the others files.

yosoycentli · 2022-01-18T00:28:46Z

Hi @ebwinters,
I think I cover all your comments about the code, I hope you can check the changes, especially the swagger part.

Best regards from Mexico City.

ebwinters · 2022-01-18T14:47:51Z

public/apidocs/swagger_v3.json

+      "variantsECDC": {
+        "type": "array",
+        "items": {
+          "type": "string"


This doesn't look right. See how other endpoints do this where they return an array of type x and then go on to declare x as the object structure of the element. Like

"updated": dateTime, "country": string, "yearWeek": string...

You're absolutely right, sorry about this.
I'm going to do fix it.

yosoycentli · 2022-01-18T18:12:02Z

@ebwinters
I think it's ok now :)

Thank you very much for your effort, time and comments, they're very helpful.

ebwinters · 2022-01-19T14:11:49Z

It's approved. Unit tests will fail for now until #1031 is merged. Once that is in, pull master, and the tests should all pass.

Also linting failed, check out why in the workflow run and fix that please

…feat

yosoycentli · 2022-01-19T18:01:37Z

Hi @ebwinters,
I fixed these eslint problems and I think it's finally everything ok.

Thank you so much for your patience with me :)

ebwinters · 2022-01-25T14:56:42Z

still seeing linting issues (unrelated to your PR but still fix please)

yosoycentli · 2022-01-25T16:05:15Z

All right then, I'll fix it right now :)

…ix the linting problem, as there are variables assigned that are never used.

yosoycentli · 2022-01-25T18:45:12Z

I commented the lines 2-5 to fix the linting problem. At least in my computer that is enough to pass the eslint check.
:)

ebwinters · 2022-01-26T01:11:08Z

now your unit tests are failing

yosoycentli · 2022-01-26T01:15:19Z

I'll check these, I truly don't know why this is happening.

Thanks for your patience.

ebwinters · 2022-02-03T15:36:17Z

@yosoycentli it's still failing, just checking back in

yosoycentli · 2022-02-04T13:27:01Z

I'm working on these tests, I still don't know why it's happening but I'll find out, I'm sure of that.
:)

…test' and the 'test-single' in the package.json file in order to get time enough to complete all tests

yosoycentli · 2022-02-14T21:22:28Z

Hi @ebwinters

I want to know if the limit of 200000 in the --timeout from the test and test-single scripts is something that I can change because I modified this number to 300000 and now all my tests are passing correctly.
:)

ebwinters · 2022-02-14T22:18:38Z

Hi @ebwinters

I want to know if the limit of 200000 in the --timeout from the test and test-single scripts is something that I can change because I modified this number to 300000 and now all my tests are passing correctly. :)

yea totally fine that's not a hard limitation

…erformance

…feat

yosoycentli · 2022-02-15T23:35:06Z

Hi, @ebwinters,

I finally understood why my tests don't pass, the reason is that the page where the scraper gets the information is usually under maintenance, so it's common that the tests don't pass.

I was wondering what is recommended to do in these cases, I noticed that there are tests that are pending, should I put my tests as pending?

Without anything else for the moment again I appreciate your patience with me.

Greetings from Mexico City

ebwinters · 2022-02-16T15:17:36Z

no it's fine if they are pending

yosoycentli · 2022-02-16T17:09:28Z

All right then @ebwinters ;).

yosoycentli · 2022-02-18T16:07:31Z

Hi @ebwinters ,
I realized that the tests finally passed so I would like to know if this is enough to merge the code or if it's better to put my tests as "pending".

yosoycentli added 8 commits January 10, 2022 19:25

[FEAT] create the scraper in the getEuropeVariants.js file, modify th…

648e3e4

…e api_gov.spec.js and getGovData.js files

[FEAT] create the scraper in the getEuropeVariants.js file, modify th…

69e948a

…e api_gov.spec.js and getGovData.js files

[MODIFY] Rename the getEuropeVariants to --> getVariants

51095ca

[MODIFY] Add the line 22 variants to the json

c88a70a

[MODIFY] change the values of keys.variants in the line 15 and 22

12611a6

[ADD/MODIFY] add the instance of the variant.js file and modify the g…

89d6cb2

…etGovData.js since the variant scraper doesn't belong in there

[MODIFY] change a several files in order to connect the variant scrap…

2dc3beb

…er with the endpoint v3/covid-19/variants/countries/{country}

[MODIFY] now the scraper works and return the data in the correct format

ac42848

yosoycentli mentioned this pull request Jan 15, 2022

[Feature Req] Variant tracking #1019

Open

ebwinters reviewed Jan 15, 2022

View reviewed changes

yosoycentli added 3 commits January 17, 2022 09:42

[MODIFY] remove the parameters yearWeek and variant from router.get i…

614077c

…n apiVariants.js file since they're never use

[MODIFY] add test for the getVariants.js scraper and remove console.l…

5d133ed

…og and commented code from the others files.

[MODIFY] add swagger documentation

2fb8e42

yosoycentli marked this pull request as ready for review January 18, 2022 00:22

ebwinters reviewed Jan 18, 2022

View reviewed changes

[MODIFY] fix swagger documentation

77816d7

Merge branch 'master' into iss1019feat

14ebd0e

ebwinters previously approved these changes Jan 19, 2022

View reviewed changes

yosoycentli added 2 commits January 19, 2022 11:40

[MODIFY] fix eslint style problems

c9c2f11

Merge branch 'iss1019feat' of github.com:yosoycentli/API into iss1019…

3478e5b

…feat

yosoycentli dismissed ebwinters’s stale review via 3478e5b January 19, 2022 17:43

Merge branch 'master' into iss1019feat

aecddba

[MODIFY] I commented the lines 2-5 in api_influenza.spec.js file to f…

46bfc6a

…ix the linting problem, as there are variables assigned that are never used.

[MODIFY] I change the --timeout value from 200000 to 300000 for the '…

e2a66aa

…test' and the 'test-single' in the package.json file in order to get time enough to complete all tests

ebwinters and others added 3 commits February 14, 2022 14:18

Merge branch 'master' into iss1019feat

67b85e2

[MODIFY] Put back the var in line 57 in order to get a right eslint p…

7442e60

…erformance

Merge branch 'iss1019feat' of github.com:yosoycentli/API into iss1019…

a4be9b4

…feat

ebwinters merged commit 09b5646 into disease-sh:master Feb 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Iss1019feat #1030

Iss1019feat #1030

yosoycentli commented Jan 14, 2022

ebwinters left a comment

ebwinters Jan 15, 2022

yosoycentli Jan 17, 2022

ebwinters Jan 15, 2022

ebwinters Jan 15, 2022

ebwinters Jan 15, 2022

yosoycentli Jan 17, 2022

ebwinters Jan 15, 2022

yosoycentli Jan 17, 2022

yosoycentli commented Jan 18, 2022

ebwinters Jan 18, 2022

yosoycentli Jan 18, 2022

yosoycentli commented Jan 18, 2022

ebwinters commented Jan 19, 2022

yosoycentli commented Jan 19, 2022

ebwinters commented Jan 25, 2022

yosoycentli commented Jan 25, 2022

yosoycentli commented Jan 25, 2022

ebwinters commented Jan 26, 2022

yosoycentli commented Jan 26, 2022

ebwinters commented Feb 3, 2022

yosoycentli commented Feb 4, 2022

yosoycentli commented Feb 14, 2022

ebwinters commented Feb 14, 2022

yosoycentli commented Feb 15, 2022

ebwinters commented Feb 16, 2022

yosoycentli commented Feb 16, 2022

yosoycentli commented Feb 18, 2022

@@ @@ -71,6 +71,34 @@ describe.skip('TESTING /v3/covid-19/gov/canada', () => { @@
               	});
               });
+              // here

Iss1019feat #1030

Iss1019feat #1030

Conversation

yosoycentli commented Jan 14, 2022

ebwinters left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yosoycentli commented Jan 18, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yosoycentli commented Jan 18, 2022

ebwinters commented Jan 19, 2022

yosoycentli commented Jan 19, 2022

ebwinters commented Jan 25, 2022

yosoycentli commented Jan 25, 2022

yosoycentli commented Jan 25, 2022

ebwinters commented Jan 26, 2022

yosoycentli commented Jan 26, 2022

ebwinters commented Feb 3, 2022

yosoycentli commented Feb 4, 2022

yosoycentli commented Feb 14, 2022

ebwinters commented Feb 14, 2022

yosoycentli commented Feb 15, 2022

ebwinters commented Feb 16, 2022

yosoycentli commented Feb 16, 2022

yosoycentli commented Feb 18, 2022