Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new action to infer a tabular resource schema #82

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

aivuk
Copy link
Contributor

@aivuk aivuk commented Dec 6, 2022

Implement new action as described on #76.

  • New action logic
  • Add tests

@aivuk aivuk added the feature label Dec 6, 2022
@aivuk aivuk requested a review from amercader December 6, 2022 09:30
@aivuk aivuk self-assigned this Dec 6, 2022
@codecov
Copy link

codecov bot commented Dec 6, 2022

Codecov Report

Merging #82 (1c984dd) into master (1073c80) will decrease coverage by 0.72%.
The diff coverage is 13.63%.

❗ Current head 1c984dd differs from pull request most recent head 670d896. Consider uploading reports for the commit 670d896 to get more accurate results

@@            Coverage Diff             @@
##           master      #82      +/-   ##
==========================================
- Coverage   84.55%   83.82%   -0.73%     
==========================================
  Files          24       24              
  Lines        2117     2139      +22     
==========================================
+ Hits         1790     1793       +3     
- Misses        327      346      +19     
Impacted Files Coverage Δ
ckanext/validation/logic.py 64.30% <9.52%> (-3.97%) ⬇️
ckanext/validation/plugin/__init__.py 96.05% <100.00%> (+0.02%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Member

@amercader amercader left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good so far @aivuk, I added some comments

BTW this coverage messages are really annoying. I guess they'll go away when you add tests but I'd be happy to turn them off, if you find how to do it

ckanext/validation/logic.py Outdated Show resolved Hide resolved
@@ -6,6 +6,7 @@

from werkzeug.datastructures import FileStorage as FlaskFileStorage
import ckan.plugins as p
import ckan.lib.uploader as uploader
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not needed

resource = t.get_action('resource_show')(
{}, {u'id': data_dict['resource_id']})

source = None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the resource['format'] is not one of ckanext.validation.formats we should raise a ValidationError with : "Not a valid format to infer the resource schema"

# TODO: check for valid formats
fric_resource = Resource({'path': source, 'format': resource.get('format', 'csv').lower()})
fric_resource.infer()
resource['schema'] = fric_resource.schema.to_json()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get an error here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we can. an example, if you don't add the format, because the uploaded file don't have an extension, the frictionless throws a exception saying that it can't infer the schema because the file is not tabular.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok so let's catch these exceptions and wrap them in a raised ValidationError

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aivuk and others added 4 commits December 6, 2022 15:32
Co-authored-by: Adrià Mercader <amercadero@gmail.com>
…om:frictionlessdata/ckanext-validation into feature/76-add-resource-table-schema-infer
@aivuk
Copy link
Contributor Author

aivuk commented Dec 6, 2022

Looks good so far @aivuk, I added some comments

BTW this coverage messages are really annoying. I guess they'll go away when you add tests but I'd be happy to turn them off, if you find how to do it

removed it for now

@aivuk aivuk marked this pull request as ready for review December 16, 2022 12:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants