registering service with HIS central #196
Comments
|
There was a bit of debugging of this try an issue filter with: |
|
hmm.. it seems like not working: I'm now trying https://github.com/ODM2/WOFpy/blob/master/docs/WOFpy_ODM2_SOAPandREST_Tests.ipynb referenced from #165 |
|
Below is my version of https://github.com/ODM2/WOFpy/blob/master/docs/WOFpy_ODM2_SOAPandREST_Tests.ipynb With lots of extra errors though! 😄 https://github.com/miguelcleon/ODM2API-ipython-notebooks/blob/master/wofpytest.ipynb |
This rings a bell, as something we ran into some time ago (a problem that happened in some circumstances only) and fixed. From the notes on the current release, it looks like those bug fixes are in the current release (v2.2.0). Based on the version number at the bottom of http://odm2admin.cuahsi.org/wofpy/odm2timeseries/, it looks like you are using the latest release. Thanks for testing with https://github.com/ODM2/WOFpy/blob/master/docs/WOFpy_ODM2_SOAPandREST_Tests.ipynb We'll look into it. |
|
BTW, before you register the service with WDC/HIS Central (once we get these errors figured out), I strongly advice you to pick a more meaningful network code. You're using the default, generic "odm2timeseries" network code, which is kind of useless in the context of the WDC catalog. Pick something like "lczots", "LCZO", "luquilloczo", etc. |
|
Ok yes, thank you for pointing that out |
|
@miguelcleon, you have an important error in your test notebook. In cell 15: # replace
networkcodes = ['wofpy/odm2timeseries']
# with
networkcodes = ['odm2timeseries']I'm not saying this will fix things. But at least the errors will be more helpful. Also, can we get access to your postgresql database, to examine values to help us debug? You can send us connection info offline. |
|
@emiliom actually when I use: http://odm2admin.cuahsi.org/odm2timeseries/soap/cuahsi_1_1/.wsdl but that gives a 404 error, but when I use Yes, I'll provide DB access. |
|
@emiliom @lsetiawan I sent you guys a link via google drive to download a backup of the database, does that work? |
|
Ok, here's what worked for the test notebook (I didn't actually test it, previously): server_base_url = 'http://odm2admin.cuahsi.org/wofpy/'
networkcodes = ['odm2timeseries']Note the slash at the end of server_base_url. The need for that has to do with the behavior of I ran this, and I got some successful requests and some errors (more successes than errors), matching what I found manually by clicking on the REST 1.1 examples on your end point. But at least the errors now are real errors ;) |
|
@miguelcleon Is the database backup that you sent us similar to |
|
@lsetiawan rest seems to work but I'm still having problems with soap |
|
@lsetiawan yes, it is the same database, just a more recent copy. |
|
I've updated the results of my test notebook |
If I work with this older database, will I see the same error do you think? |
|
latest stack trace from apache error log: |
|
I don't think the older database has sites associated with sampling features, which was a problem I fixed. |
|
Oh okay. I'll just spin up your backup then. Thanks! Stay tuned! Let's see if I can find and remove the 🐛 |
|
Any luck? I'm supposed to go see blade runner tonight and I need to leave soon. |
|
Just finished spinning up your db on the cloud, probably won't finish till tomorrow. Thanks. |
|
Ok, ttyl |
|
@lsetiawan I didn't end up going to the movie after all SOAP looks like it is working based on the results of the test notebook 😃 The hiscentral qa system is still generating errors though, I tested the same site that the notebook tests so that isn't it. I'll probably have to ask CUAHSI tomorrow. http://qa-hiscentral.cuahsi.org/testpage.aspx?n=5629 |
You don't have your priorities straight! BTW, I haven't seen it yet, but I intend to.
I wasn't aware of that test page until you mentioned it yesterday. Admittedly, the tests on our notebook are limited, but the fact that ulmo can consume the services at the WOFpy endpoint is a strong test in my book. Still, it'd be interesting to test one of our sample WOFpy endpoints with that CUAHSI tool. I'll try to do that tomorrow (Don doesn't work on my projects on Thursdays). But it'll be great to get more details from CUAHSI about the error that tool is running into. |
|
@miguelcleon, did you make any progress on this on Thursday, learning what exact errors the CUAHSI test tool is running into? I may try to register one of our test WOFpy endpoints tomorrow, just to see what happens. Of course, I won't make it live. |
|
@emiliom I emailed Martin at CUAHSI, yesterday morning but I haven't heard back from him. I tried changing the site in your test script to see if another one worked and it did. Yeah it would be good if you tried that. I just tried seeing if there is maybe some unicode in the sites or method records, that the CUAHSI test page might be choking on but I'm not seeing anything. Maybe it is expecting a method name returned from siteInfo, but I only see a method code in the response: |
In the example that is in the WaterML 1.1 Documentation Pg 68. Method Name is doesn't seem to be shown. So I think it should be fine if it's not returned. <method methodID="4">
<methodDescription>
Battery voltage measured by Campbell
Scientific CR206 datalogger.
</methodDescription>
</method> |
|
Yeah, I don't know we need to hear from them. |
Are you saying you tried a different site in the CUAHSI test system, and it worked? Or just in our test notebook? If I remember correctly the original (BTW, your endpoint http://odm2admin.cuahsi.org/wofpy/odm2timeseries/ seems to be down right now) |
|
Ok I got the below message from Martin, I changed the name of my site name and service name to odm2lczo and it worked! The new test service page is http://qa-hiscentral.cuahsi.org/testpage.aspx?n=5631
|
|
Woo hoo! That's awesome! I'm glad it was an easy fix. 😄 |
|
Awesome! I think I get what he's saying, but probably not 100%. I think it'd be helpful if we set up a conference call with you, Martin, Don and me, next week, if there's still some confusion. Also, I'm seeing many sites in your endpoint that have a site code that's the same or about the same as the site name. That's not a good practice. The site code should be a much more compact string, ideally (but not required) w/o blank spaces. FYI, I went to the CUAHSI test system for the link you sent us, and picked the Icacos site. That worked. Then I picked the water temperature variable, and that worked. Selected a small time range (1-2 months), and that worked too! |
|
To be clear: your new endpoint url is http://odm2admin.cuahsi.org/wofpy/odm2lczo/ |
|
@emiliom, yes, that is the new endpoint. Now I'm trying to see if they can populate their test hydroclient (http://qa-hiswebclient.azurewebsites.net/) with the data from the endpoint. It looks like they need to harvest the endpoint into hydroclient. We can setup a call. Hopefully you can test out your own endpoint to see what questions you may have for them. |
|
BTW, I noticed earlier this week that some of your sites don't have a valid lat-lon! At least one of them has (0.0, 0.0). You'll want to address that before going live as a public service 😉 Since things to be working on the CUAHSI end, maybe a call isn't needed. We can decide on Monday, after more tests, and specially if by then you've been able to test on the test hydroclient. |
|
Ok, I'll take a look at those lat-lons, thanks for that. |
|
@miguelcleon, we can close this issue, right? Or are we waiting to get confirmation that the service works with their test HydroClient? I know you're also going to address sites/sampling features with (0,0) lat-lons, but that's not a WOFpy issue 😉 Also, just to reiterate, before you go live with the service registration:
I would encourage you to use this opportunity to come up with short, computer-friendly site codes. But I realize you may have some dependencies on the current codes, that could cause some transition pain. If you DON'T have such dependencies, this really is a great time to create better site codes! You'll be grateful later. Still, that's not a WOFpy issue either. Just a good data management and "CUAHSI HIS/WOF/ODM1/ODM2" practice. |
|
@emiliom I'd like to keep this open until we can confirm that this is working on their test HydroClient. The new Luquillo CZO ODM2 service now shows up but I can't seem to make the filters work, so it seems like something went wrong. I emailed Martin about this. Yes, one problem I have with the lat-lons is I don't know some of them. They have related features where the related feature they are part of have lat-lons. Many of these are soil pits spread across multiple catenas where only 1 GPS measurement was recorded. I could make up some offsets but they wouldn't be correct. I thought about just having the single GPS measurement apply for all of them but that isn't really right either. Any thoughts? |
|
Sounds good, we'll keep it open.
Let us know via this issue what you learn from Martin.
Well, my first gut instinct is to filter out those complex sites so they're not served at all via WOFpy. They're an awkward fit for ODM1/CUAHSI HIS. We could filter them out now (via the DAO code, probably -- but we can discuss that), and then we'd have time to discuss options among ourselves, with the CUAHSI gang, and with Jeff. You can also ask Martin and Tony about this now, to see what they think. Personally I think it's a bad idea to include data subsets in a service if users will face a lot of ambiguity about how to interpret the data obtained via that service. |
|
@miguelcleon, any progress on HydroClient testing:
|
|
Martin got back to me this morning with:
I followed up asking if he could let me know which metadata fields are missing. It doesn't seem that 'unkown' is present in the WaterML so I asked if he meant they are blank. I know valuecounts aren't updated, I had some code to update them, but it's not working and it wasn't a priority. I don't see something called seriescount anywhere. Seems like I might have to fix my value counts but I'm not sure what else needs to be done at this point. |
|
Thanks for the update. Let's reconnect on Monday. But yes, it would be really helpful if we got from Martin et al a complete report of problems. I'll keep @lsetiawan out of this on Monday, though. He'll be offline the whole week except Monday, so he'll have other, more burning priorities on Monday 😸 |
|
@emiliom, @lsetiawan, and @miguelcleon - that I know of, there is no such thing as "seriescount" in WaterML. It's been a while for me and so I could just be spacing it. But, you should probably follow up and ask Martin for more information about what the harvester is expecting (which I'm sure you were planning on doing anyway). |
|
Thanks, @horsburgh. So, I think we've passed the first test phase, which is the CUAHSI service registration tests (that's what led to the last batch of fixes). Now Miguel is on the second (and last?) phase, confirming that everything works on HydroClient. Most things are working, some are not. We're waiting on feedback from Martin on specific errors. Actually, it didn't occur to me to ask Martin "what the harvester is expecting"; I guess I was being naive? Anyway, that's a good idea. But you've also inspired me to do something we hadn't done, which is to validate the WOFpy responses against the WaterML 1.1 XML schema (.xsd) file. I just tested the |
|
@miguelcleon, any progress in hearing back from CUAHSI about the errors? Also, while I examining your WaterML 1.1 responses (to test XML Schema validation on my end), I noticed that in your sample |
|
@miguelcleon, I assume there's no new info from CUAHSI about the problems you were running into with your service registration and HydroClient? FYI, we're wrapping up the last updates to WOFpy before issuing a new release (see #201); no more bug fixes or changes will be accepted. I plan to stamp a new release tomorrow morning, and a new conda package will be generated very soon after. |
|
@emiliom unfortunately no. I've been stuck in proposal and annual report writing mode for the past week. |
|
Oh well. Good luck! |
|
Based on conversation with Martin, The problem seems to lie on tags not being closed properly for empty attributes. For example |
|
Thanks @lsetiawan. I spoke with Martin a bit about this. What we're doing is not wrong (ie, |
I'm trying to register a WOFpy service with the QA version of HIS central and the ‘Get Site info’ seems to be erroring out:
http://qa-hiscentral.cuahsi.org/testpage.aspx?n=5629
The service should be viewable here (http://qa-hiswebclient.azurewebsites.net/ ) but it is not.
The text was updated successfully, but these errors were encountered: