Sweep: refactoring index.js #31

Laeyoung · 2023-09-23T07:20:42Z

Checklist

api/jhu-edu.js ✅ Commit 1e9d63f

• Review the file and identify areas that can be improved or refactored. • Extract common logic into separate functions or modules. • Improve the structure and readability of the code.

Sandbox Execution Logs

trunk init 1/3 ✓

⡿ Downloading Trunk 1.16.0...
⡿ Downloading Trunk 1.16.0...
⢿ Downloading Trunk 1.16.0...
⣻ Downloading Trunk 1.16.0...
⣽ Downloading Trunk 1.16.0...
⣾ Downloading Trunk 1.16.0...
⣷ Downloading Trunk 1.16.0...
✔ Downloading Trunk 1.16.0... done
⡿ Verifying Trunk sha256...
✔ Verifying Trunk sha256... done
⡿ Unpacking Trunk...
✔ Unpacking Trunk... done
























✔ 10 linters were enabled (.trunk/trunk.yaml)
  checkov 2.4.9 (1 docker, 7 json, 2 yaml files)
  git-diff-check (16 files)
  hadolint 2.12.0 (1 docker file) (created .hadolint.yaml)
  markdownlint 0.37.0 (1 markdown file) (created .markdownlint.yaml)
  osv-scanner 1.4.0 (1 lockfile file)
  prettier 3.0.3 (3 javascript, 6 json, 1 markdown, 2 yaml files)
  terrascan 1.18.3 (1 docker file)
  trivy 0.45.1 (1 docker, 1 lockfile, 2 yaml files)
  trufflehog 3.56.1 (17 files)
  yamllint 1.32.0 (2 yaml files) (created .yamllint.yaml)
Next Steps
 1. Read documentation
    Our documentation can be found at https://docs.trunk.io
 2. Get help and give feedback
    Join the Trunk community at https://slack.trunk.io

trunk fmt api/jhu-edu.js 2/3 ✓

 ✔ Formatted api/jhu-edu.js
Re-checking autofixed files...





Checked 1 file
✔ No issues

trunk check --fix api/jhu-edu.js 3/3 ✓




Checked 1 file
✔ No issues

api/korea-kcdc.js ✅ Commit 3e9f448

• Review the file and identify areas that can be improved or refactored. • Extract common logic into separate functions or modules. • Improve the structure and readability of the code.

Sandbox Execution Logs

trunk init 1/3 ✓

⡿ Downloading Trunk 1.16.0...
⡿ Downloading Trunk 1.16.0...
⢿ Downloading Trunk 1.16.0...
⣻ Downloading Trunk 1.16.0...
⣽ Downloading Trunk 1.16.0...
⣾ Downloading Trunk 1.16.0...
⣷ Downloading Trunk 1.16.0...
✔ Downloading Trunk 1.16.0... done
⡿ Verifying Trunk sha256...
✔ Verifying Trunk sha256... done
⡿ Unpacking Trunk...
✔ Unpacking Trunk... done

























✔ 10 linters were enabled (.trunk/trunk.yaml)
  checkov 2.4.9 (1 docker, 7 json, 2 yaml files)
  git-diff-check (16 files)
  hadolint 2.12.0 (1 docker file) (created .hadolint.yaml)
  markdownlint 0.37.0 (1 markdown file) (created .markdownlint.yaml)
  osv-scanner 1.4.0 (1 lockfile file)
  prettier 3.0.3 (3 javascript, 6 json, 1 markdown, 2 yaml files)
  terrascan 1.18.3 (1 docker file)
  trivy 0.45.1 (1 docker, 1 lockfile, 2 yaml files)
  trufflehog 3.56.1 (17 files)
  yamllint 1.32.0 (2 yaml files) (created .yamllint.yaml)
Next Steps
 1. Read documentation
    Our documentation can be found at https://docs.trunk.io
 2. Get help and give feedback
    Join the Trunk community at https://slack.trunk.io

trunk fmt api/korea-kcdc.js 2/3 ✓

 ✔ Formatted api/korea-kcdc.js
Re-checking autofixed files...




Checked 1 file
✔ No issues

trunk check --fix api/korea-kcdc.js 3/3 ✓





Checked 1 file
✔ No issues

The text was updated successfully, but these errors were encountered:

sweep-ai · 2023-09-23T07:20:50Z

Here's the PR! #32.

⚡ Sweep Free Trial: I'm creating this ticket using GPT-4. You have 4 GPT-4 tickets left for the month and 2 for the day. For more GPT-4 tickets, visit [our payment portal.](https://buy.stripe.com/6oE5npbGVbhC97afZ4)

Actions (click)

↻ Restart Sweep

Step 1: 🔎 Searching

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I looked at (click to expand). If some file is missing from here, you can mention the path in the ticket description.

COVID-19-API/Dockerfile

Lines 1 to 7 in bea2941

    
           FROM node:12-alpine 
        
           WORKDIR /workspace 
        
           COPY package*.json ./ 
        
           RUN npm install 
        
           COPY . . 
        
           EXPOSE 80

COVID-19-API/api/jhu-edu.js

Lines 17 to 251 in bea2941

    
           const csv = require('csvtojson') 
        
           const csvPath = { 
        
             confirmed: 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv', 
        
             deaths: 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_global.csv', 
        
             recovered: 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_recovered_global.csv' 
        
           } 
        
           const fixedCountryCodes = require('../dataset/country-codes.json') 
        
           const iso2CountryLoc = require('../dataset/iso2-country-loc.json') 
        
           const schedule = require('node-schedule') 
        
           schedule.scheduleJob('42 * * * *', updateCSVDataSet) // Call every hour at 42 minutes 
        
           const responseSet = { 
        
             brief: '{}', 
        
             latest: '{}', 
        
             latestOnlyCountries: '{}', 
        
             timeseries: '{}', 
        
             timeseriesOnlyCountries: '{}' 
        
           } 
        
           let lastUpdate 
        
           router.get('/brief', function (req, res) { 
        
             res.status(200).json(responseSet.brief) 
        
           }) 
        
           router.get('/latest', function (req, res) { 
        
             const { iso2, iso3, onlyCountries } = req.query 
        
             let latest = onlyCountries ? responseSet.latestOnlyCountries : responseSet.latest 
        
             if (iso2) latest = filterIso2code(latest, iso2) 
        
             if (iso3) latest = filterIso3code(latest, iso3) 
        
             res.status(200).send(JSON.stringify(latest)) 
        
           }) 
        
           router.get('/timeseries', function (req, res) { 
        
             const { iso2, iso3, onlyCountries } = req.query 
        
             let timeseries = onlyCountries ? responseSet.timeseriesOnlyCountries : responseSet.timeseries 
        
             if (iso2) timeseries = filterIso2code(timeseries, iso2) 
        
             if (iso3) timeseries = filterIso3code(timeseries, iso3) 
        
             res.status(200).send(JSON.stringify(timeseries)) 
        
           }) 
        
           function updateCSVDataSet () { 
        
             console.log('Updated at ' + new Date().toISOString()) 
        
             const dataSource = { 
        
               confirmed: {}, 
        
               deaths: {}, 
        
               recovered: {} 
        
             } 
        
             lastUpdate = new Date().toISOString() 
        
             const queryPromise = [] 
        
             Object.entries(csvPath).forEach(([category, path]) => { 
        
               queryPromise.push(queryCsvAndSave(dataSource, path, category)) 
        
             }) 
        
             Promise.all(queryPromise) 
        
               .then((_values) => { 
        
                 const brief = { 
        
                   [column.CONFIRMED]: 0, 
        
                   [column.DEATHS]: 0, 
        
                   [column.RECOVERED]: 0 
        
                 } 
        
                 const latest = {} 
        
                 const timeseries = {} 
        
                 for (const [category, value] of Object.entries(dataSource)) { 
        
                   for (const [name, item] of Object.entries(value)) { 
        
                     const keys = Object.keys(item) 
        
                     const cell = item[keys[keys.length - 1]] 
        
                     const latestCount = cell ? Number(cell) : Number(item[keys[keys.length - 2]]) 
        
                     // For brief 
        
                     brief[category] += latestCount 
        
                     // For latest 
        
                     createPropertyIfNeed(latest, name, item) 
        
                     latest[name][category] = latestCount 
        
                     // For timeseries 
        
                     createPropertyIfNeed(timeseries, name, item) 
        
                     for (const date of keys.slice(4)) { 
        
                       // recovered csv file uses '3/24/2020' date format instead of '3/24/20' 
        
                       const fixedDatePropertyName = date.replace('/2020', '/20') 
        
                       nestedProperty.set(timeseries[name], `timeseries.${fixedDatePropertyName}.${category}`, Number(item[date])) 
        
                     } 
        
                   } 
        
                 } 
        
                 // Set 0 for unknown recovered value 
        
                 for (const item of Object.values(timeseries)) { 
        
                   for (const date of Object.values(item.timeseries)) { 
        
                     if (!date.recovered) date.recovered = 0 
        
                   } 
        
                 } 
        
                 responseSet.brief = brief 
        
                 responseSet.latest = Object.values(latest) 
        
                 responseSet.timeseries = Object.values(timeseries) 
        
                 // Uses deep copy 
        
                 // https://stackoverflow.com/a/122704/3614334 
        
                 const copyLatest = JSON.parse(JSON.stringify(latest)) 
        
                 responseSet.latestOnlyCountries = getMergedByCountry(Object.values(copyLatest)) 
        
                 const copyTimeseries = JSON.parse(JSON.stringify(timeseries)) 
        
                 responseSet.timeseriesOnlyCountries = getMergedByCountry(Object.values(copyTimeseries)) 
        
                 console.log(`Confirmed: ${brief.confirmed}, Deaths: ${brief.deaths}, Recovered: ${brief.recovered}`) 
        
               }) 
        
               .catch((error) => { 
        
                 console.log('Error on queryPromise: ' + error) 
        
               }) 
        
           } 
        
           function getMergedByCountry (list) { 
        
             const mergedList = { } 
        
             for (const item of list) { 
        
               const countryName = item.countryregion 
        
               if (mergedList[countryName]) { 
        
                 const country = mergedList[countryName] 
        
                 // for latest api 
        
                 mergeConfirmDeathRecover(country, item) 
        
                 // for timeseries api 
        
                 if (item.timeseries) { 
        
                   const timeseries = item.timeseries 
        
                   const mergedTimeseries = country.timeseries 
        
                   for (const key of Object.keys(timeseries)) { 
        
                     mergeConfirmDeathRecover(mergedTimeseries[key], timeseries[key]) 
        
                   } 
        
                 } 
        
                 // Overwrite location 
        
                 if (item.countrycode) { 
        
                   const iso2 = item.countrycode.iso2 
        
                   country.location = iso2CountryLoc[iso2] 
        
                 } 
        
               } else { 
        
                 delete item.provincestate 
        
                 mergedList[countryName] = item 
        
               } 
        
             } 
        
             return Object.values(mergedList) 
        
           } 
        
           function mergeConfirmDeathRecover (target, item) { 
        
             if (!(target && item)) return 
        
             if (item.confirmed) merge(target, item, 'confirmed') 
        
             if (item.deaths) merge(target, item, 'deaths') 
        
             if (item.recovered) merge(target, item, 'recovered') 
        
           } 
        
           function merge (target, item, key) { 
        
             const cur = target[key] ? target[key] : 0 
        
             const add = item[key] ? item[key] : 0 
        
             target[key] = cur + add 
        
           } 
        
           function filterIso2code (source, code) { 
        
             return source.filter(item => { 
        
               const countryCode = item.countrycode ? item.countrycode.iso2 : 'unknown' 
        
               return countryCode === code 
        
             }) 
        
           } 
        
           function filterIso3code (source, code) { 
        
             return source.filter(item => { 
        
               const countryCode = item.countrycode ? item.countrycode.iso3 : 'unknown' 
        
               return countryCode === code 
        
             }) 
        
           } 
        
           function queryCsvAndSave (dataSource, path, category) { 
        
             return csv() 
        
               .fromStream(request.get(path)) 
        
               .subscribe((json) => { 
        
                 if (json['Province/State']) { 
        
                   const provincestate = json['Province/State'] 
        
                   dataSource[category][provincestate] = json 
        
                 } else if (json['Country/Region']) { 
        
                   const countryregion = json['Country/Region'] 
        
                   dataSource[category][countryregion] = json 
        
                 } 
        
                 return new Promise((resolve, reject) => { 
        
                   resolve() 
        
                 }) 
        
               }) 
        
           } 
        
           function createPropertyIfNeed (target, name, item) { 
        
             if (!nestedProperty.has(target, name)) { 
        
               target[name] = { 
        
                 [column.PROVINCE_STATE]: item['Province/State'], 
        
                 [column.COUNTRY_REGION]: item['Country/Region'], 
        
                 [column.LAST_UPDATE]: lastUpdate, 
        
                 location: { 
        
                   lat: Number(item.Lat), 
        
                   lng: Number(item.Long) 
        
                 } 
        
               } 
        
               // Append country codes 
        
               const countryName = item['Country/Region'] 
        
               if (lookup.byCountry(countryName)) { 
        
                 appendCountryCode(lookup.byCountry(countryName)) 
        
               } else if (fixedCountryCodes[countryName]) { 
        
                 appendCountryCode(lookup.byCountry(fixedCountryCodes[countryName])) 
        
               } 
        
             } 
        
             function appendCountryCode (countryCode) { 
        
               target[name].countrycode = { 
        
                 iso2: countryCode.iso2, 
        
                 iso3: countryCode.iso3 
        
               } 
        
             } 
        
           } 
        
           updateCSVDataSet()

COVID-19-API/api/korea-kcdc.js

Lines 1 to 47 in bea2941

    
           const express = require('express') 
        
           const router = express.Router() 
        
           const fetch = require('node-fetch') 
        
           const fs = require('fs') 
        
           const url = 'https://raw.githubusercontent.com/LiveCoronaDetector/livecod/master/data/koreaRegionalData.js' 
        
           const tempFilePath = './dataset/temp-kcdc.js' 
        
           const tempModulePath = '../dataset/temp-kcdc.js' 
        
           const exportModule = 'module.exports = koreaRegionalData;' 
        
           const schedule = require('node-schedule') 
        
           schedule.scheduleJob('12 * * * *', updateDataSet) // Call every hour at 12 minutes 
        
           let koreaRegionalData = {} 
        
           router.get('/brief', function (req, res) { 
        
             res.status(200).json(koreaRegionalData) 
        
           }) 
        
           async function updateDataSet () { 
        
             console.log('Korea KCDC Updated at ' + new Date().toISOString()) 
        
             try { 
        
               const response = await fetch(url) 
        
               const body = await response.text() 
        
               fs.writeFile(tempFilePath, body + exportModule, function (err) { 
        
                 if (err) { 
        
                   console.error(err) 
        
                 } else { 
        
                   try { 
        
                     koreaRegionalData = require(tempModulePath) 
        
                     console.log('Korea KCDC data was saved!') 
        
                   } catch (e) { 
        
                     console.error(e) 
        
                     koreaRegionalData = {} 
        
                   } 
        
                 } 
        
               }) 
        
             } catch (e) { 
        
               console.error(e) 
        
               koreaRegionalData = {} 
        
             } 
        
           } 
        
           updateDataSet()

COVID-19-API/README.md

Lines 5 to 85 in bea2941

    
           ## Try it 
        
           [![Run on Ainize](https://ainize.ai/images/run_on_ainize_button.svg)](https://ainize.web.app/redirect?git_repo=https://github.com/laeyoung/covid-19-api) 
        
           ## API Document 
        
           >Recovered data from JHU CCSE will not be updated after March 23th. Check [this issue](https://github.com/Laeyoung/COVID-19-API/issues/16). 
        
           ### Endpoint 
        
           https://master-covid-19-api-laeyoung.endpoint.ainize.ai/ 
        
           ### Brief 
        
           **Request:** 
        
           ```json 
        
           GET /jhu-edu/brief 
        
           ``` 
        
           **Successful Response:** 
        
           ```json 
        
           HTTP/1.1 200 OK 
        
           Content-Type: application/json 
        
           { 
        
           	"confirmed": 7783, 
        
           	"deaths": 170, 
        
           	"recovered": 133 
        
           } 
        
           ``` 
        
           **Curl:** 
        
           ```sh 
        
           curl -X GET "https://master-covid-19-api-laeyoung.endpoint.ainize.ai/jhu-edu/brief" -H "accept: application/json" 
        
           ``` 
        
           **Browser:** 
        
           https://master-covid-19-api-laeyoung.endpoint.ainize.ai/jhu-edu/brief 
        
           ### Latest 
        
           **Request:** 
        
           ```json 
        
           GET /jhu-edu/latest 
        
           ``` 
        
           Options 
        
           - Filter by country's iso2 or iso3 code 
        
           ```json 
        
           GET /jhu-edu/latest?iso2=US 
        
           ``` 
        
           - Make the sum of all the cities or the states in those countries 
        
           ```json 
        
           GET /jhu-edu/latest?onlyCountries=true 
        
           ``` 
        
           You can uses both options together. 
        
           **Successful Response:** 
        
           ```json 
        
           HTTP/1.1 200 OK 
        
           Content-Type: application/json 
        
           [ 
        
             { 
        
               "provincestate": "Anhui", 
        
               "countryregion": "Mainland China", 
        
               "lastupdate": "2020-02-29T07:21:21.001Z", 
        
               "confirmed": 990, 
        
               "deaths": 6, 
        
               "recovered": 821, 
        
               "location": { 
        
                 "lat": 31.8257, 
        
                 "lng": 117.2264 
        
               }, 
        
               "countrycode": { 
        
                 "iso2": "CN", 
        
                 "iso3": "CHN" 
        
               } 
        
             }, { 
        
               "provincestate": "Beijing", 
        
               "countryregion": "Mainland China", 
        
               "lastupdate": "2020-02-29T07:21:21.001Z", 
        
               "confirmed": 410,

Step 2: ⌨️ Coding

api/jhu-edu.js ✅ Commit 1e9d63f

• Review the file and identify areas that can be improved or refactored. • Extract common logic into separate functions or modules. • Improve the structure and readability of the code.

Sandbox Execution Logs

trunk init 1/3 ✓

⡿ Downloading Trunk 1.16.0...
⡿ Downloading Trunk 1.16.0...
⢿ Downloading Trunk 1.16.0...
⣻ Downloading Trunk 1.16.0...
⣽ Downloading Trunk 1.16.0...
⣾ Downloading Trunk 1.16.0...
⣷ Downloading Trunk 1.16.0...
✔ Downloading Trunk 1.16.0... done
⡿ Verifying Trunk sha256...
✔ Verifying Trunk sha256... done
⡿ Unpacking Trunk...
✔ Unpacking Trunk... done
























✔ 10 linters were enabled (.trunk/trunk.yaml)
  checkov 2.4.9 (1 docker, 7 json, 2 yaml files)
  git-diff-check (16 files)
  hadolint 2.12.0 (1 docker file) (created .hadolint.yaml)
  markdownlint 0.37.0 (1 markdown file) (created .markdownlint.yaml)
  osv-scanner 1.4.0 (1 lockfile file)
  prettier 3.0.3 (3 javascript, 6 json, 1 markdown, 2 yaml files)
  terrascan 1.18.3 (1 docker file)
  trivy 0.45.1 (1 docker, 1 lockfile, 2 yaml files)
  trufflehog 3.56.1 (17 files)
  yamllint 1.32.0 (2 yaml files) (created .yamllint.yaml)
Next Steps
 1. Read documentation
    Our documentation can be found at https://docs.trunk.io
 2. Get help and give feedback
    Join the Trunk community at https://slack.trunk.io

trunk fmt api/jhu-edu.js 2/3 ✓

 ✔ Formatted api/jhu-edu.js
Re-checking autofixed files...





Checked 1 file
✔ No issues

trunk check --fix api/jhu-edu.js 3/3 ✓




Checked 1 file
✔ No issues

api/korea-kcdc.js ✅ Commit 3e9f448

• Review the file and identify areas that can be improved or refactored. • Extract common logic into separate functions or modules. • Improve the structure and readability of the code.

Sandbox Execution Logs

trunk init 1/3 ✓

⡿ Downloading Trunk 1.16.0...
⡿ Downloading Trunk 1.16.0...
⢿ Downloading Trunk 1.16.0...
⣻ Downloading Trunk 1.16.0...
⣽ Downloading Trunk 1.16.0...
⣾ Downloading Trunk 1.16.0...
⣷ Downloading Trunk 1.16.0...
✔ Downloading Trunk 1.16.0... done
⡿ Verifying Trunk sha256...
✔ Verifying Trunk sha256... done
⡿ Unpacking Trunk...
✔ Unpacking Trunk... done

























✔ 10 linters were enabled (.trunk/trunk.yaml)
  checkov 2.4.9 (1 docker, 7 json, 2 yaml files)
  git-diff-check (16 files)
  hadolint 2.12.0 (1 docker file) (created .hadolint.yaml)
  markdownlint 0.37.0 (1 markdown file) (created .markdownlint.yaml)
  osv-scanner 1.4.0 (1 lockfile file)
  prettier 3.0.3 (3 javascript, 6 json, 1 markdown, 2 yaml files)
  terrascan 1.18.3 (1 docker file)
  trivy 0.45.1 (1 docker, 1 lockfile, 2 yaml files)
  trufflehog 3.56.1 (17 files)
  yamllint 1.32.0 (2 yaml files) (created .yamllint.yaml)
Next Steps
 1. Read documentation
    Our documentation can be found at https://docs.trunk.io
 2. Get help and give feedback
    Join the Trunk community at https://slack.trunk.io

trunk fmt api/korea-kcdc.js 2/3 ✓

 ✔ Formatted api/korea-kcdc.js
Re-checking autofixed files...




Checked 1 file
✔ No issues

trunk check --fix api/korea-kcdc.js 3/3 ✓





Checked 1 file
✔ No issues

Step 3: 🔁 Code Review

I have finished reviewing the code for completeness. I did not find errors for sweep/refactor-api-files.

.

🎉 Latest improvements to Sweep:

Getting Sweep to run linters before committing! Check out Sweep Sandbox Configs to set it up.
Added support for self-hosting! Check out Self-hosting Sweep to get started.
[Self Hosting] Multiple options to compute vector embeddings, configure your .env file using VECTOR_EMBEDDING_SOURCE

💡 To recreate the pull request edit the issue title or description. To tweak the pull request, leave a comment on the pull request.
^{Join Our Discord}

sweep-ai bot added the sweep Sweep your software chores label Sep 23, 2023

sweep-ai bot linked a pull request Sep 23, 2023 that will close this issue

Refactor main API files #32

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sweep: refactoring index.js #31

Sweep: refactoring index.js #31

Laeyoung commented Sep 23, 2023 •

edited by sweep-ai bot

sweep-ai bot commented Sep 23, 2023 •

edited

Sweep: refactoring index.js #31

Sweep: refactoring index.js #31

Comments

Laeyoung commented Sep 23, 2023 • edited by sweep-ai bot

sweep-ai bot commented Sep 23, 2023 • edited

Here's the PR! #32.

Actions (click)

Step 1: 🔎 Searching

Step 2: ⌨️ Coding

Step 3: 🔁 Code Review

Laeyoung commented Sep 23, 2023 •

edited by sweep-ai bot

sweep-ai bot commented Sep 23, 2023 •

edited