Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scispark-61 generating the openDap urls #66

Merged
merged 27 commits into from
Jul 4, 2016
Merged

Scispark-61 generating the openDap urls #66

merged 27 commits into from
Jul 4, 2016

Conversation

kwhitehall
Copy link
Member

No description provided.

@kwhitehall
Copy link
Member Author

Still working on this PR.

@chrismattmann
Copy link
Collaborator

LGTM as a start. @kwhitehall

kwhitehall and others added 24 commits June 28, 2016 11:54
@kwhitehall
allow for openDap subsetting.
7eda156
@kwhitehall
updated opendap version
18ca981
@kwhitehall
updated the urls paths and fixed the checkurls option
87a582c
@kwhitehall
update to write file to HDFS
91051c1
@kwhitehall
updated to check if file exists on hdfs.
0a75233
@kwhitehall
cleaned up some commit msgs, etc.
469b33e
@kwhitehall
updated SciSparkContext to read the OpenDapUrls
3b0a77f
@kwhitehall
removed log.info from map task
dcb097b
@rahulpalamuttam
update opendap dependency to 4.6.0. Note : 4.6.6 causes http dependen…
e28b6f6 
…cy conflicts
@kwhitehall
Merge pull request #70 from rahulpalamuttam/ssl_error_http_versioning
ae3146d 
[SCISPARK #69][ESIP workshop] update opendap to 4.6.0. Addresses issue #69
@kwhitehall
- update opendap url download link
cc7b514 
- allow user to provide start and end times
- allow for daily and 3hrly TRMM downloads
@kwhitehall
fixed time error related to 1st jan
0b4063e
@kwhitehall
updated filename in hrly url for change after 2011
810cb30
@kwhitehall
allow for openDap subsetting.
ed6fa7b
@kwhitehall
updated opendap version
6b754a4
@kwhitehall
updated the urls paths and fixed the checkurls option
7bc9ba3
@kwhitehall
update to write file to HDFS
b573ad4
@kwhitehall
updated to check if file exists on hdfs.
4f34c56
@kwhitehall
cleaned up some commit msgs, etc.
f431742
@kwhitehall
updated SciSparkContext to read the OpenDapUrls
e3967ee
@kwhitehall
removed log.info from map task
0fa01d6
@kwhitehall
reverted the cdm version that was updated in this PR
14acc3c
@kwhitehall
fixed rebasing conflicts
2d95aaa
@kwhitehall
allow user to pass hdfs url to test
6b074b1
@kwhitehall
Copy link
Member Author

kwhitehall commented Jul 2, 2016 •
edited

Ok @chrismattmann and @rahulpalamuttam
I think I'm finished with this PR.
To run the unit test, from command line the following is required
SBT_OPTS="-DhdfsURL=hdfs://url_host:port" sbt "test-only org.dia.urlgenerators.OpenDapTRMMURLGeneratorTest"

TODO: is to get the thread / waiting between calls to the openDap server in the SciSparkContext.scala line 106 going.

Finally, this PR addresses issue #61 and #67 (indirectly - the openDap and any url from file will now be read into a rdd of SciTensors)

@chrismattmann
Copy link
Collaborator

LGTM +1 to commit. @kwhitehall @rahulpalamuttam

@kwhitehall
Copy link
Member Author

Ack'ed @chrismattmann. Thanks!

@kwhitehall kwhitehall merged commit b416d53 into master Jul 4, 2016
@kwhitehall kwhitehall mentioned this pull request Jul 4, 2016
Closed
@kwhitehall kwhitehall deleted the scispark-61 branch July 4, 2016 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@kwhitehall @chrismattmann @rahulpalamuttam