RLS: 1.0.4 #33300

simonjayhawkins · 2020-04-05T10:44:42Z

we have several regressions reported since 1.0.3 (not due to 1.0.3, since 0.25.3) and a couple that had not been fixed.

https://github.com/pandas-dev/pandas/issues?page=1&q=is%3Aopen+is%3Aissue+label%3ARegression

Is there any that warrant a 1.0.4 release?

@pandas-dev/pandas-core

TomAugspurger · 2020-04-06T12:22:45Z

I probably won't be able to help with one this week, but the scripts at https://github.com/pandas-dev/pandas-release should be reasonably up to date.

We'll need to manually verify that all the necessary PRs have been backported (I don't think they have). It might be easier to do a 1.1 release soon.

simonjayhawkins · 2020-04-06T12:28:29Z

I probably won't be able to help with one this week

I don't think this is urgent. It's more a matter of whether we should be labelling PRs that fix regressions as 1.0.4 and backporting.

It might be easier to do a 1.1 release soon.

IMO we should be ensuring that the 1.0.x fixes regressions from 0.25.3. 1.1 could introduce new regressions. so this is about ensuring an upgrade path for affected users.

TomAugspurger · 2020-04-15T14:21:31Z

@simonjayhawkins I think in the call last week we decided to leave it up to you. If you have time to go through and backport the necessary PRs for a release we can go ahead and do it.

In addition to backporting, I think that

are blockers.

simonjayhawkins · 2020-04-17T08:28:01Z

potential backports (add to this list)

BUG: None converted to NaN after groupby first and last #33462 (regression in 1.0.2)

jreback · 2020-05-04T14:37:37Z

@simonjayhawkins still working on this?

simonjayhawkins · 2020-05-04T14:45:56Z

When this issue was opened there was no PRs merged to master that fixed regressions. There were also blockers to doing a release, #33300 (comment).

I've not been working on this, but now we are a few weeks on and the blockers are resolved, we now have a few PRs that could be backported.

I could start backporting these if we are likely to do a 1.0.4 release.

TomAugspurger · 2020-05-04T15:04:17Z

At this point, I think pushing for a 1.1 release in a week or two is probably easier.

jreback · 2020-05-04T15:49:08Z

I actually would be +1 on a 1.0.4; I also think this is orthogonal to doing 1.1, and good for others to try a release.

simonjayhawkins · 2020-05-04T17:37:24Z

In addition to backporting, I think that

MacPython/pandas-wheels#82

MacPython/pandas-wheels wheel builds failing #33210

are blockers.

#32737 wasn't backported, so #33210 shouldn't be an issue here?

simonjayhawkins · 2020-05-04T18:36:46Z

@jorisvandenbossche there is a to_feather test failing on 1.0.x for Linux py37_locale. pyarrow 0.17 is being installed. see https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=34788&view=logs&j=a3a13ea8-7cf0-5bdb-71bb-6ac8830ae35c

The logs for the previous builds on this branch where this test passed are unavailable to confirm the pyarrow version.

I can't find any recent PRs to fix ci for this. maybe #33422 fixes this.

on windows, conda is not installing pyarrow 0.17 so can't reproduce failure, tests pass with 0.16

$ conda install pyarrow==0.17.0 -c conda-forge
Collecting package metadata (repodata.json): done
Solving environment: \ 
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - conda-forge/win-64::fastparquet==0.3.3=py38hc8d92b1_0
  - conda-forge/win-64::pyarrow==0.16.0=py38h57df961_2
  - conda-forge/win-64::pyreadstat==0.3.3=py38h11373af_0
  - conda-forge/noarch::seaborn==0.10.1=py_0
  - conda-forge/win-64::statsmodels==0.11.1=py38h9de7a3e_1
  - conda-forge/noarch::xarray==0.15.1=py_0
done

## Package Plan ##

  environment location: C:\Users\simon\Anaconda3\envs\pandas-dev

  added / updated specs:
    - pyarrow==0.17.0


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    arrow-cpp-0.16.0           |py38h5a970ca_5_cpu        11.8 MB  conda-forge
    openssl-1.1.1g             |       he774522_0         5.7 MB  conda-forge
    ------------------------------------------------------------
                                           Total:        17.5 MB

The following NEW packages will be INSTALLED:

  pandas             conda-forge/win-64::pandas-1.0.3-py38he6e81aa_1

The following packages will be UPDATED:

  arrow-cpp                           0.16.0-py38h5a970ca_4 --> 0.16.0-py38h5a970ca_5_cpu
  ca-certificates     pkgs/main::ca-certificates-2020.1.1-0 --> conda-forge::ca-certificates-2020.4.5.1-hecc5488_0

The following packages will be SUPERSEDED by a higher-priority channel:

  certifi              pkgs/main::certifi-2020.4.5.1-py38_0 --> conda-forge::certifi-2020.4.5.1-py38h32f6830_0
  openssl                                         pkgs/main --> conda-forge


Proceed ([y]/n)? y


Downloading and Extracting Packages
openssl-1.1.1g       | 5.7 MB    | ################################################################################################################# | 100%  
arrow-cpp-0.16.0     | 11.8 MB   | ################################################################################################################# | 100%  
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(pandas-dev) 
$ conda list pyarrow
# packages in environment at C:\Users\simon\Anaconda3\envs\pandas-dev:
#
# Name                    Version                   Build  Channel
pyarrow                   0.16.0           py38h57df961_2    conda-forge
(pandas-dev)

jorisvandenbossche · 2020-05-04T20:33:48Z

If Simon is up to it, also +1 on doing a 1.0.4 release.

there is a to_feather test failing on 1.0.x for Linux py37_locale. pyarrow 0.17 is being installed

#33422 might indeed fix this, as I changed some tests there. I think that PR should be OK to backport (the actual code changes are limited, just about passing through keywords, or could also limit to only backport the test fixes)
I can take a look at this if you want.

simonjayhawkins · 2020-05-05T08:40:18Z

or could also limit to only backport the test fixes

will do that, just aiming to get ci passing on the 1.0.x branch for now. once the branch is ready to accept backports, we can discuss what can/should be backported. (for instance, #32611 fixed a regression (#33058), but may not be suitable for backporting.)

hroff-1902 · 2020-05-15T10:32:21Z

Guys, can you push the release of 1.0.4 (at least, please do not to postpone it to be after 1.1.0, as I saw at the last dev meeting minutes) or apply some more love to it?

The memory leak in rolling,min/max is serious, do you understand that thousands of apps using pandas crash worldwide in the meantime, till the fix is available for us, users of pandas?

jreback · 2020-05-15T10:40:47Z

Guys, can you push the release of 1.0.4 (at least, please do not to postpone it to be after 1.1.0, as I saw at the last dev meeting minutes) or apply some more love to it?

The memory leak in rolling,min/max is serious, do you understand that thousands of apps using pandas crash worldwide in the meantime, till the fix is available for us, users of pandas?

@hroff-1902

pandas is an all volunteer project; you are welcome to help out

comment like the above are simply not helpful

hroff-1902 · 2020-05-15T10:55:02Z

yes, a volunteer project, as many others, that's clear

the release procedure/regressions is not a part of the development process where a novice in a project could really help, I guess

sometimes (often) devs do not understand the severity of the issues so explicit expressions of concernments from the users of the project can help, at least sometimes.

please understand our concerns

TomAugspurger · 2020-05-15T11:10:34Z

@hroff-1902 have you been following the huge amount of work @simonjayhawkins has been doing to prep a 1.0.4 release?

Meanwhile, a release is blocked by our wheel building infrastructure at https://github.com/macpython/pandas-wheels. #34114 will need to be fixed before we can make a release. Can you look into it?

jreback · 2020-05-15T11:11:30Z

yes, a volunteer project, as many others, that's clear

the release procedure/regressions is not a part of the development process where a novice in a project could really help, I guess

sometimes (often) devs do not understand the severity of the issues so explicit expressions of concernments from the users of the project can help, at least sometimes.

please understand our concerns

there are many areas you could help, why are you not volunteering there?

I do understand severity of issues very well and am sympathetic but w/o any way to actually direct resources, your comments are not meaningful.

Its like, let’s not pay for anything, not contribute in-kind and yet have the audacity to demand something.

jreback · 2020-05-25T16:09:25Z

@simonjayhawkins I would be in favor of releasing this asap. no need to wait for any other patches. the timing is orthogonal to 1.1. If really really needed / wanted could always do a 1.0.5.

The only real requirements are passing CI & ability to build wheels (which i think are satisfied).

simonjayhawkins · 2020-05-25T17:10:55Z

I think both #32479 and #33644 are ready to be merged. These can then be backported. I think backporting #33644 is important as incorrect results are being produced.

jreback · 2020-05-25T18:41:55Z

we can finish the second (first is already merged) but generally tiny edge cases even if a regression are not worth holding things up

simonjayhawkins · 2020-05-28T18:55:23Z

maybe worked on re-run conda-forge/pandas-feedstock#81

TomAugspurger · 2020-05-28T19:41:15Z

Thanks. I think I've also had issues with git in the conda env.

I'll start on the doc build now.

simonjayhawkins · 2020-05-28T19:45:41Z

I'll start on the doc build now.

cool. just got rsync installed for git bash. would have needed ssh access to continue but calling it a day. will carry-on tomorrow with what's left to do.

TomAugspurger · 2020-05-29T11:55:50Z

Website and docs are uploaded. Let me know if you see any strange formatting issues. The layout was messed up the first time I visited a docs page, but seems to be OK now.

Wheels are on PyPI and conda-forge packages are building.

@simonjayhawkins do you want to send an email to the mailing lists with the announcement?

jorisvandenbossche · 2020-05-29T12:01:22Z

I still see the messed up layout for the symlinked ones, like https://pandas.pydata.org/docs/ (https://pandas.pydata.org/pandas-docs/version/1.0.4/ is fine).

TomAugspurger · 2020-05-29T12:05:21Z

Hmm, I have to run for a bit so I'll revert the symlink for now.

simonjayhawkins · 2020-05-29T12:11:24Z

@simonjayhawkins do you want to send an email to the mailing lists with the announcement?

happy to do this when we're happy all is OK

What's new in 1.0.4 on https://pandas.pydata.org/ points to https://pandas.pydata.org/docs/whatsnew/v1.0.0.html

Is this something that I need to change?

TomAugspurger · 2020-05-29T12:21:22Z

No, it’s been fixed on master so I’m not worried about that.

TomAugspurger · 2020-05-29T19:24:15Z

FYI, I'm debugging this at https://pandas.pydata.org/pandas-docs/debug/. It seems to only occur when the server has served a 1.0.3 request at that URL. So almost surely a CSS asset is being cached somewhere.

jorisvandenbossche · 2020-05-30T07:56:56Z

https://pandas.pydata.org/pandas-docs/debug/ now looks fine for me. Was that url supposed to be a "broken" one? (in which case it might be solved now)

WillAyd · 2020-06-01T02:20:00Z

Not sure if anyone has noticed but to @simonjayhawkins point clicking on the 1.0.4 whatsnew from the home page takes me to 1.0.0 with no option to see 1.0.4. It also mentions 1.0.1 in the top menu:

jorisvandenbossche · 2020-06-01T06:52:34Z

That's because the link seems to be hardcoded in the home page (while the text is automatically updated):

pandas/web/pandas/index.html

Lines 64 to 67 in 6eb34f1

    
           <h4>Latest version: {{ releases[0].name }}</h4> 
        
           <ul> 
        
               <li><a href="docs/whatsnew/v1.0.0.html">What's new in {{ releases[0].name }}</a></li> 
        
               <li>Release date:<br/>{{ releases[0].published.strftime("%b %d, %Y") }}</li>

So that needs a fix there.

(I also seem to remember some discussion about whether we found linking to 1.0.0 instead of 1.0.x more relevant, but in any case now the text and the link is inconsistent)

TomAugspurger · 2020-06-01T11:08:46Z

It's fixed on master: https://pandas.pydata.org/pandas-docs/dev/

TomAugspurger · 2020-06-01T11:10:05Z

I don't know what to do about the CSS issue. I set up https://pandas.pydata.org/pandas-docs/debug/ on Friday to replicate /docs and it did: I saw the messed up formatting for a while. Visiting now shows that it looks fine, so whatever was caching the CSS has since expired.

jorisvandenbossche · 2020-06-01T11:10:43Z

@TomAugspurger that's the docs (where we indeed removed the "Whatsnew" link in the top bar), but AFAIU it is about a wrong link on the home page (/web/)

TomAugspurger · 2020-06-01T11:32:35Z

Ah, thanks. Should that just link to the whatsnew index?

…

On Mon, Jun 1, 2020 at 6:10 AM Joris Van den Bossche < ***@***.***> wrote: @TomAugspurger <https://github.com/TomAugspurger> that's the docs (where we indeed removed the "Whatsnew" link in the top bar), but AFAIU it is about a wrong link on the home page (/web/) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#33300 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOITYNVIWYMGP5OXZTDTRUOEEDANCNFSM4MATQACQ> .

TomAugspurger · 2020-06-01T15:49:01Z

@aterrel I think this may be cached at the cloudflare CDN sitting between pandas' webserver and the internet.

$ http https://pandas.pydata.org/pandas-docs/debug/_static/css/index.css
HTTP/1.1 200 OK
Age: 123
CF-Cache-Status: HIT
CF-RAY: 59ca0e5778575c43-STL
Cache-Control: max-age=14400
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/css
Date: Mon, 01 Jun 2020 15:44:26 GMT
ETag: W/"5e724acc-156c"
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Last-Modified: Wed, 18 Mar 2020 16:22:36 GMT
Server: cloudflare
Set-Cookie: __cfduid=d1bde881be99bc9d3922b6701123103c61591026266; expires=Wed, 01-Jul-20 15:44:26 GMT; path=/; domain=.pydata.org; HttpOnly; SameSite=Lax
Transfer-Encoding: chunked
Vary: Accept-Encoding
cf-request-id: 0312274aa800005c43bab24200000001
...

Is giving me access to cloudflare so that I can purge some caches an option? https://support.cloudflare.com/hc/en-us/articles/200169246-Purging-cached-resources-from-Cloudflare.

TomAugspurger · 2020-06-01T15:54:38Z

Actually, hold on that. I might be able to instruct nginx to not use CF's cache. Won't have time to look into that until later though.

TomAugspurger · 2020-06-03T20:31:05Z

My idea was to add a cache-control to nginx to tell cloudflare to not cache anything, but I don't want to accidentally take down our webserver if the load is too high. I don't have a good idea.

@aterrel I think getting access to Cloudflare to manually purge the cache is best. Can you grant me access or can we sync up sometime for you to do it?

jorisvandenbossche · 2020-06-03T20:59:01Z

@TomAugspurger you didn't find out how to change the /doc symlink? (to link directly to the correct version, instead of through stable. Not sure if this would fix it of course)

aterrel · 2020-06-03T22:15:37Z

@TomAugspurger yes, create a cloudflare account and send me the email address you used.

aterrel · 2020-06-04T14:59:25Z

For those watching along, invite has been sent. Let me know if I can be of any other service.

TomAugspurger · 2020-06-04T16:07:21Z

Thanks Andy.

Possibly fixed with this purge list through the cloudflare console.

https://pandas.pydata.org/docs/_static/css/getting_started.css
https://pandas.pydata.org/docs/_static/css/index.css
https://pandas.pydata.org/docs/_static/css/pandas.css
https://pandas.pydata.org/pandas-docs/stable/_static/css/getting_started.css
https://pandas.pydata.org/pandas-docs/stable/_static/css/index.css
https://pandas.pydata.org/pandas-docs/stable/_static/css/pandas.css
https://pandas.pydata.org/docs/_static/js/index.js
https://pandas.pydata.org/pandas-docs/stable/_static/js/index.js

@jorisvandenbossche anything else you can think of that should be purged?

jorisvandenbossche · 2020-06-04T16:10:19Z

That should be all pandas doc specific js/css files, I think

TomAugspurger · 2020-06-04T20:55:46Z

Thanks.

And a huge thanks to @simonjayhawkins for managing the release!

simonjayhawkins added the Release label Apr 5, 2020

simonjayhawkins added this to the 1.0.4 milestone Apr 5, 2020

simonjayhawkins mentioned this issue Apr 5, 2020

REGR: Fix bug when replacing categorical value with self #33292

Merged

5 tasks

This was referenced Apr 17, 2020

Indexing broken inside groupby - apply #33058

Closed

REGR: DataFrame.mean(numeric_only=True) raises AttributeError on v1.0.3 #33256

Closed

simonjayhawkins mentioned this issue May 4, 2020

CI: Skip permissions test when running as sudo #33847

Merged

simonjayhawkins mentioned this issue May 5, 2020

CI: test_unsupported_other fails on pyarrow 0.17 #33990

Merged

This was referenced May 5, 2020

REGR: fix DataFrame reduction with EA columns and numeric_only=True #33761

Merged

BUG: Fix Categorical use_inf_as_na bug #33629

Merged

BUG: Groupby quantiles incorrect bins #33200 #33644

Merged

TomAugspurger mentioned this issue Jun 1, 2020

Release notes still missing for 1.0.4 #34513

Closed

jorisvandenbossche pinned this issue Jun 3, 2020

jorisvandenbossche mentioned this issue Jun 3, 2020

DOC: documentation link to whatsnew v1.0.4 not working #34541

Closed

TomAugspurger closed this as completed Jun 4, 2020

simonjayhawkins unpinned this issue Jun 5, 2020

RLS: 1.0.4 #33300

RLS: 1.0.4 #33300

Comments

simonjayhawkins commented Apr 5, 2020

TomAugspurger commented Apr 6, 2020

simonjayhawkins commented Apr 6, 2020

TomAugspurger commented Apr 15, 2020

simonjayhawkins commented Apr 17, 2020 • edited

jreback commented May 4, 2020

simonjayhawkins commented May 4, 2020

TomAugspurger commented May 4, 2020

jreback commented May 4, 2020

simonjayhawkins commented May 4, 2020

simonjayhawkins commented May 4, 2020

jorisvandenbossche commented May 4, 2020 • edited

simonjayhawkins commented May 5, 2020

hroff-1902 commented May 15, 2020

jreback commented May 15, 2020

hroff-1902 commented May 15, 2020

TomAugspurger commented May 15, 2020

jreback commented May 15, 2020

jreback commented May 25, 2020

simonjayhawkins commented May 25, 2020

jreback commented May 25, 2020

simonjayhawkins commented May 28, 2020

TomAugspurger commented May 28, 2020

simonjayhawkins commented May 28, 2020

TomAugspurger commented May 29, 2020

jorisvandenbossche commented May 29, 2020

TomAugspurger commented May 29, 2020

simonjayhawkins commented May 29, 2020

TomAugspurger commented May 29, 2020

TomAugspurger commented May 29, 2020

jorisvandenbossche commented May 30, 2020

WillAyd commented Jun 1, 2020

jorisvandenbossche commented Jun 1, 2020

TomAugspurger commented Jun 1, 2020

TomAugspurger commented Jun 1, 2020

jorisvandenbossche commented Jun 1, 2020 • edited

TomAugspurger commented Jun 1, 2020 via email

TomAugspurger commented Jun 1, 2020

TomAugspurger commented Jun 1, 2020

TomAugspurger commented Jun 3, 2020

jorisvandenbossche commented Jun 3, 2020

aterrel commented Jun 3, 2020

aterrel commented Jun 4, 2020

TomAugspurger commented Jun 4, 2020

jorisvandenbossche commented Jun 4, 2020

TomAugspurger commented Jun 4, 2020 • edited

simonjayhawkins commented Apr 17, 2020 •

edited

jorisvandenbossche commented May 4, 2020 •

edited

jorisvandenbossche commented Jun 1, 2020 •

edited

TomAugspurger commented Jun 4, 2020 •

edited