Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dash Fashion MNIST Explorer does not deploy #596

Open
xhluca opened this issue Mar 31, 2021 · 1 comment
Open

Dash Fashion MNIST Explorer does not deploy #596

xhluca opened this issue Mar 31, 2021 · 1 comment
Labels
bug Something isn't working.
Projects
Post-DE 4.1 release updates

Comments

@xhluca
Copy link
Collaborator

xhluca commented Mar 31, 2021

Seems like dash-fashion-mnist-explorer does not deploy on circleci. Will deployignore the app for now but we should investigate it later.

Show circleci failure logs 


====> Deploying dash-fashion-mnist-explorer
      dashr: false
      dash python version: dash==1.7.0
      exists: true
      remote-sha: 
====> Initializing repository
====> Ensuring code is up to date
      Copying updated app source
      Python app detected, injecting common python-specific files
====> Deploying
Counting objects: 34, done.
Delta compression using up to 36 threads.
Compressing objects: 100% (33/33), done.
Writing objects: 100% (34/34), 32.66 MiB | 2.94 MiB/s, done.
Total 34 (delta 0), reused 0 (delta 0)
remote: -----> DDS disk space information        
remote: Filesystem      Size  Used Avail Use% Mounted on        
remote: overlay         2.0T  1.5T  512G  75% /        
remote: /dev/sdc        206G  117G   79G  60% /data        
remote: -----> DASH_APP_NAME => dash-fashion-mnist-explorer        
remote: -----> DASH_DOMAIN_BASE => dash-gallery.plotly.host        
remote: -----> DASH_PATH_ROUTING => 1        
remote: -----> DASH_STREAMBED_DIRECT_IP => 172.17.0.1        
remote: -----> DASH_LOGOUT_URL => /Manager/api/logout        
remote: -----> SCRIPT_NAME => /dash-fashion-mnist-explorer        
remote: -----> Cleaning up...        
remote: -----> Building dash-fashion-mnist-explorer from herokuish...        
remote: rehash: warning: skipping ca-certificates.pem,it does not contain exactly one certificate or CRL        
remote: 1 added, 0 removed; done.        
remote: Running hooks in /etc/ca-certificates/update.d...        
remote:         
remote: updates of cacerts keystore disabled.        
remote: done.        
remote: -----> Adding BUILD_ENV to build environment...        
remote:        -----> Python app detected        
remote: -----> Installing pip        
remote: -----> Installing requirements with pip        
remote:        Collecting setuptools>=41.0.0 (from tensorboard<1.15.0,>=1.14.0->tensorflow->-r /tmp/build/requirements.txt (line 5))        
remote:        Downloading https://files.pythonhosted.org/packages/9e/d4/b99a960314121a003e9f39c61dfde01a1010bb47661e193a7722f7f32d52/setuptools-54.2.0-py3-none-any.whl (785kB)        
remote:        Installing collected packages: setuptools        
remote:        Found existing installation: setuptools 39.0.1        
remote:        Uninstalling setuptools-39.0.1:        
remote:        Successfully uninstalled setuptools-39.0.1        
remote:        Successfully installed setuptools-54.2.0        
remote:        -----> Discovering process types        
remote:        Procfile declares types -> web        
remote: -----> Releasing dash-fashion-mnist-explorer (dokku/dash-fashion-mnist-explorer:latest)...        
remote: -----> Deploying dash-fashion-mnist-explorer (dokku/dash-fashion-mnist-explorer:latest)...        
remote:  !     Predeploy command declared: 'python predeploy.py'        
remote: -----> App Procfile file found (/home/dokku/dash-fashion-mnist-explorer/DOKKU_PROCFILE)        
remote:        DOKKU_SCALE declares scale -> web=1        
remote: =====> Processing deployment checks        
remote:        No CHECKS file found. Simple container checks will be performed.        
remote:        For more efficient zero downtime deployments, create a CHECKS file. See http://dokku.viewdocs.io/dokku/deployment/zero-downtime-deploys/ for examples        
remote: -----> Attempting pre-flight checks (web.1)        
remote:        Waiting for 10 seconds ...        
remote: 3be80fce7b898273adbf023137d00167fcf63f2e82ba15a108a0f760089f2db6        
remote: App container failed to start!!        
remote: =====> dash-fashion-mnist-explorer web container output:        
remote:        [2021-03-31 21:32:51 +0000] [12] [INFO] Starting gunicorn 19.9.0        
remote:        [2021-03-31 21:32:51 +0000] [12] [INFO] Listening at: http://0.0.0.0:5000 (12)        
remote:        [2021-03-31 21:32:51 +0000] [12] [INFO] Using worker: sync        
remote:        [2021-03-31 21:32:51 +0000] [189] [INFO] Booting worker with pid: 189        
remote:        Using TensorFlow backend.        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_qint8 = np.dtype([("qint8", np.int8, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_quint8 = np.dtype([("quint8", np.uint8, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_qint16 = np.dtype([("qint16", np.int16, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_quint16 = np.dtype([("quint16", np.uint16, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_qint32 = np.dtype([("qint32", np.int32, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          np_resource = np.dtype([("resource", np.ubyte, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_qint8 = np.dtype([("qint8", np.int8, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_quint8 = np.dtype([("quint8", np.uint8, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_qint16 = np.dtype([("qint16", np.int16, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_quint16 = np.dtype([("quint16", np.uint16, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_qint32 = np.dtype([("qint32", np.int32, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          np_resource = np.dtype([("resource", np.ubyte, 1)])        
remote:        [2021-03-31 21:33:01 +0000] [189] [ERROR] Exception in worker process        
remote:        Traceback (most recent call last):        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker        
remote:            worker.init_process()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process        
remote:            self.load_wsgi()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi        
remote:            self.wsgi = self.app.wsgi()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi        
remote:            self.callable = self.load()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load        
remote:            return self.load_wsgiapp()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp        
remote:            return util.import_app(self.app_uri)        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app        
remote:            __import__(module)        
remote:          File "/app/app.py", line 30, in <module>        
remote:            model = load_model("trained_data/fashion_mnist_cnn.h5")        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/keras/engine/saving.py", line 492, in load_wrapper        
remote:            return load_function(*args, **kwargs)        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/keras/engine/saving.py", line 584, in load_model        
remote:            model = _deserialize_model(h5dict, custom_objects, compile)        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/keras/engine/saving.py", line 273, in _deserialize_model        
remote:            model_config = json.loads(model_config.decode('utf-8'))        
remote:        AttributeError: 'str' object has no attribute 'decode'        
remote:        [2021-03-31 21:33:01 +0000] [189] [INFO] Worker exiting (pid: 189)        
remote:        [2021-03-31 21:33:01 +0000] [12] [INFO] Shutting down: Master        
remote:        [2021-03-31 21:33:01 +0000] [12] [INFO] Reason: Worker failed to boot.        
remote:        [2021-03-31 21:33:04 +0000] [15] [INFO] Starting gunicorn 19.9.0        
remote:        [2021-03-31 21:33:04 +0000] [15] [INFO] Listening at: http://0.0.0.0:5000 (15)        
remote:        [2021-03-31 21:33:04 +0000] [15] [INFO] Using worker: sync        
remote:        [2021-03-31 21:33:04 +0000] [191] [INFO] Booting worker with pid: 191        
remote:        Using TensorFlow backend.        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_qint8 = np.dtype([("qint8", np.int8, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_quint8 = np.dtype([("quint8", np.uint8, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_qint16 = np.dtype([("qint16", np.int16, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_quint16 = np.dtype([("quint16", np.uint16, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_qint32 = np.dtype([("qint32", np.int32, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          np_resource = np.dtype([("resource", np.ubyte, 1)])        
remote:        [2021-03-31 21:33:05 +0000] [15] [INFO] Handling signal: term        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_qint8 = np.dtype([("qint8", np.int8, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_quint8 = np.dtype([("quint8", np.uint8, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_qint16 = np.dtype([("qint16", np.int16, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_quint16 = np.dtype([("quint16", np.uint16, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          _np_qint32 = np.dtype([("qint32", np.int32, 1)])        
remote:        /app/.heroku/python/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.        
remote:          np_resource = np.dtype([("resource", np.ubyte, 1)])        
remote:        [2021-03-31 21:33:09 +0000] [191] [ERROR] Exception in worker process        
remote:        Traceback (most recent call last):        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker        
remote:            worker.init_process()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process        
remote:            self.load_wsgi()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi        
remote:            self.wsgi = self.app.wsgi()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi        
remote:            self.callable = self.load()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load        
remote:            return self.load_wsgiapp()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp        
remote:            return util.import_app(self.app_uri)        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app        
remote:            __import__(module)        
remote:          File "/app/app.py", line 30, in <module>        
remote:            model = load_model("trained_data/fashion_mnist_cnn.h5")        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/keras/engine/saving.py", line 492, in load_wrapper        
remote:            return load_function(*args, **kwargs)        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/keras/engine/saving.py", line 584, in load_model        
remote:            model = _deserialize_model(h5dict, custom_objects, compile)        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/keras/engine/saving.py", line 273, in _deserialize_model        
remote:            model_config = json.loads(model_config.decode('utf-8'))        
remote:        AttributeError: 'str' object has no attribute 'decode'        
remote:        [2021-03-31 21:33:09 +0000] [191] [INFO] Worker exiting (pid: 191)        
remote:        Traceback (most recent call last):        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 225, in run        
remote:            handler()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 260, in handle_term        
remote:            raise StopIteration        
remote:        StopIteration        
remote:        During handling of the above exception, another exception occurred:        
remote:        Traceback (most recent call last):        
remote:          File "/app/.heroku/python/bin/gunicorn", line 11, in <module>        
remote:            sys.exit(run())        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 61, in run        
remote:            WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 223, in run        
remote:            super(Application, self).run()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 72, in run        
remote:            Arbiter(self).run()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 228, in run        
remote:            self.halt()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 345, in halt        
remote:            self.stop()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 393, in stop        
remote:            time.sleep(0.1)        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 245, in handle_chld        
remote:            self.reap_workers()        
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 525, in reap_workers        
remote:            raise HaltServer(reason, self.WORKER_BOOT_ERROR)        
remote:        gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>        
remote: =====> end dash-fashion-mnist-explorer web container output        
To ********************************/GIT/dash-fashion-mnist-explorer
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '********************************/GIT/dash-fashion-mnist-explorer'

Exited with code exit status 1
CircleCI received exit code 1

@xhluca xhluca added the bug Something isn't working. label Mar 31, 2021
@xhluca xhluca added this to TODO in Post-DE 4.1 release updates via automation Mar 31, 2021
@xhluca xhluca moved this from TODO to In progress in Post-DE 4.1 release updates May 27, 2021
@xhluca
Copy link
Collaborator Author

xhluca commented May 28, 2021

Seems like the issue still persists when run locally. The issue seems to stem from the version mismatch between tensorflow and the Keras-2.3.1. Unfortunately, I tried downgrading tensorflow to 1.x and change keras to tensorflow.keras, but the app still does not work correctly. I will keep it ignored until further investigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working.
Projects
Post-DE 4.1 release updates
In progress
Milestone
No milestone
Development

No branches or pull requests
1 participant
@xhluca