Skip to content
This repository has been archived by the owner on Oct 16, 2020. It is now read-only.

Error mounting NVME device on m5d instances #2613

Open
dhawal55 opened this issue Sep 12, 2019 · 1 comment
Open

Error mounting NVME device on m5d instances #2613

dhawal55 opened this issue Sep 12, 2019 · 1 comment

Comments

@dhawal55
Copy link

Issue Report

I'm trying to provision a m5d instance that comes with NVMe storage.
In my ignition config, I try to format the disk as follows:

filesystems:
  - name: etcddatafs
    mount:
      device: /dev/nvme1n1
      format: ext4
      wipe_filesystem: true

However, it fails with the following error:

[0;1;31m�[0mdisks: createFilesystems: op(f): op(11): op(13): [failed]   creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/nvme1n1": exit status 1: Cmd: "/usr/sbin/mkfs.ext4" "-F" "/run/ignition/dev_aliases/dev/nvme1n1" Stdout: "" Stderr: "mke2fs 1.44.5 (15-Dec-2018)
Found a gpt partition table in /run/ignition/dev_aliases/dev/nvme1n1
/run/ignition/dev_aliases/dev/nvme1n1 is apparently in use by the system; will not make a filesystem here!\n"

Bug

Container Linux Version

2191.5.0

Environment

AWS m5d.xlarge instance

Expected Behavior

/dev/nvme1n1 should be formatted and mounted

Actual Behavior

Format fails for nvme

Other Information

Full system log:

[   37.556774] audit: type=1131 audit(1568330461.401:21): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[�[0;32m  OK  �[0m] Started �[0;1;39mFile System Check on /dev/disk/by-label/ROOT�[0m.
[   37.276794] ignition[846]:     "disks": [
[   37.563522] audit: type=1130 audit(1568330461.410:22): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-fsck-root comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[�[0;32m  OK  �[0m] Started �[0;[   37.568427] audit: type=1130 audit(1568330461.417:23): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=verity-setup comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
1;39mVerity Setup for /dev/mapper/usr�[0m.
[   37.283428] ignition[846]:       {
[   37.289779] ignition[846]:         "device": "/dev/nvme1n1",
[   37.290935] ignition[846]:         "partitions": [
[   37.291902] ignition[846]:           {
[   37.292687] ignition[846]:             "label": "ETCDDATAFS",
[   37.293843] ignition[846]:             "size": 0,
[   37.294786] ignition[846]:             "start": 0
[   37.295742] ignition[846]:           }
[   37.296532] ignition[846]:         ],
[�[0;32m  OK  �[0m] Found device �[0;1;39m/dev/mapper/usr�[0m.
[   37.297328] ignition         [846]: Mounting �[0;1;39m/sysroot�[0m...        "wipeTable": true

         Starting �[0;1;39mEmergency Shell�[0m...
[   37.299878] ignition[846]:       },
[   37.300789] ignition[846]:       {
[   37.301461] ignition[846]:         "device": "/dev/xvdp",
[   37.302137] ignition[846]:         "partitions": [
[   37.302594] ignition[846]:           {
[   37.303048] ignition[846]:             "label": "IMAGEFS",
[   37.303508] ignition[846]:             "size": 0,
[   37.303959] ignition[846]:             "start": 0
[   37.304420] ignition[846]:           }
[   37.304872] ignition[846]:         ],
[   37.305114] ignition[846]:         "wipeTable": true
[   37.305567] ignition[846]:       },
[   37.305824] ignition[846]:       {
[   37.306268] ignition[846]:         "device": "/dev/xvdc",
[   37.306501] ignition[846]:         "partitions": [
[   37.306732] ignition[846]:           {
[   37.306963] ignition[846]:             "label": "NODEFS",
[   37.307407] ignition[846]:             "size": 0,
[   37.307644] ignition[846]:             "start": 0
[   37.307868] ignition[846]:           }
[   37.308096] ignition[846]:         ],
[   37.308541] ignition[846]:         "wipeTable": true
[   37.308775] ignition[846]:       }
[   37.309000] ignition[846]:     ],
[   37.309443] ignition[846]:     "files": [
[   37.309674] ignition[846]:       {
[   37.309899] ignition[846]:         "filesystem": "root",
[   37.310125] ignition[846]:         "path": "/etc/kubernetes/kubelet/kubeconfig",
[   37.319978] ignition[846]:         "contents": {
[   37.323192] ignition[846]:           "source": "data:,apiVersion%3A%20v1%0Akind%3A%20Config%0Aclusters%3A%0A-%20name%3A%20local%0A%20%20cluster%3A%0A%20%20%20%20server%3A%20https%3A%2F%2Faws-9405.platform-dev.r53.nordstrom.net%0A%20%20%20%20certificate-authority%3A%20%2Fetc%2Fkubernetes%2Fca.crt%0Ausers%3A%0A-%20name%3A%20kubelet%0A%20%20user%3A%0A%20%20%20%20client-certificate%3A%20%2Fetc%2Fkubernetes%2Ftls.crt%0A%20%20%20%20client-key%3A%20%2Fetc%2Fkubernetes%2Ftls.key%0Acontexts%3A%0A-%20context%3A%0A%20%20%20%20cluster%3A%20local%0A%20%20%20%20user%3A%20kubelet%0A",
[   37.323283] ignition[846]:           "verification": {}
[   37.323754] ignition[846]:         },
[   37.324221] ignition[846]:         "mode": 420
[   37.324676] ignition[846]:       },
[   37.325135] ignition[846]:       {
[   37.325600] igni[   37.618585] EXT4-fs (nvme0n1p9): mounted filesystem with ordered data mode. Opts: (null)
tion[846]:         "filesystem": "root",
[   37.325833] ignition[846]:         "path": "/etc/torcx/profiles/combined.json",
[   37.326224] ignition[846]:         "contents": {
[   37.326598] ignition[846]:           "source": "data:,%7B%22kind%22%3A%22profile-manifest-v0%22%2C%22value%22%3A%7B%22images%22%3A%5B%7B%22name%22%3A%22docker%22%2C%22reference%22%3A%2217.03%22%7D%2C%7B%22name%22%3A%22crio%22%2C%22reference%22%3A%22v1.14.6a%22%7D%5D%7D%7D%0A",
[   37.326973] ignition[846]:           "verification": {}
[   37.327374] ignition[846]:         },
[   37.328436] ignition[846]:         "mode": 420
[   37.328742] ignition[846]:       },
[   37.329120] ignition[846]:       {
[   37.329524] ignition[846]:         "filesystem": "root",
[   37.329894] ignition[846]:         "path": "/etc/torcx/next-profile",
[   37.330102] ignition[846]:         "contents": {
[   37.330515] ignition[846]:           "source": "data:,combined%0A",
[   37.330702] ignition[846]:           "verification": {}
[   37.331092] ignition[846]:         },
[   37.331469] ignition[846]:         "mode": 420
[   37.331880] ignition[846]:       },
[   37.332351] ignition[846]:       {
[   37.332737] ignition[846]:         "filesystem": "root",
[   37.333128] ignition[846]:         "path": "/var/lib/torcx/store/docker:17.03.torcx.tgz",
[   37.333536] ignition[846]:         "contents": {
[   37.336041] ignition[846]:           "source": "https://tectonic-torcx.release.core-os.net/pkgs/amd64-usr/docker/e3819eef5185486cd192ae03108003695e98e9cea4bc4b8082a0a79f212fc7eda263996c78fdac3690b0f2ebbfbdf5632f654483cc8bcd30bfb165ed9d322fbf/docker:17.03.torcx.tgz",[�[0;32m  OK  �[0m] 
Mounted �[0;1;39m/sysroot�[0m.
[   37.336524] ignition[846]:           "verification": {
[   37.336981] ignition[846]:             "hash": "sha512-e3819eef5185486cd192ae03108003695e98e9cea4bc4b8082a0a79f212fc7eda263996c78fdac3690b0f2ebbfbdf5632f654483cc8bcd30bfb165ed9d322fbf"
         Mounting �[0;1;39m/sysroot/usr�[0m...
[�[0;32m  OK  �[0m] Reached target �[0;1;39mInitrd Root File System�[0m.
[   37.337458] ignition[846]:           }
[   37.338390] ignition[846]:         },
[   37.338842] ignition[846]:         "mode": 420
[   37.339304] ignition[846]:       },
[   37.339[   37.649533] EXT4-fs (dm-0): mounted filesystem without journal. Opts: (null)
535] ignition[846]:       {
[   37.339765] ignition[846]:         "filesystem": "root",
[   37.339973] ignition[846]:         "path": "/etc/docker/daemon.json",
[   37.340373] ignition[846]:         "contents": {
[   37.340762] ignition[846]:           "source": "data:,%7B%0A%20%20%22metrics-addr%22%20%3A%20%220.0.0.0%3A9323%22%2C%0A%20%20%22experimental%22%20%3A%20true%0A%7D%0A",
[   37.341160] ignition[846]:           "verification": {}
[   37.341561] ignition[846]:         },
[   37.341941] ignition[846]:         "mode": 420
[   37.342332] ignition[846]:       },
[   37.342711] ignition[846]:       {
[   37.343101] ignition[846]:         "filesystem": "root",
[   37.343502] ignition[846]:         "path": "/var/lib/torcx/store/crio:v1.14.6a.torcx.tgz",
[   37.343868] ignition[846]:         "contents": {
[   37.344080] ignition[846]:           "source": "s3://platform-bootstrap-dev/binaries/crio:v1.14.6a.torcx.tgz",
[   37.344457] ignition[846]:           "verification": {}
[   37.344847] ignition[846]:         },
[   37.345240] ignition[846]:         "mode": 420
[   37.345614] ignition[846]:       },
[   37.345991] ignition[846]:       {
[   37.346388] ignition[846]:         "filesystem": "root",
[   37.346804] ignition[846]:         "path": "/etc/crio/crio.conf",
[   37.347197] ignition[846]:         "contents": {
[   37.347575] ignition[846]:           "source": "s3://platform-bootstrap-dev/binaries/etccrio/v1.14.6a/crio.conf",
[   37.347958] ignition[846]:           "verification": {}
[   37.348377] ignition[846]:         },
[   37.348768] ignition[846]:         "mode": 420
[   37.349170] ignition[846]:       },
[   37.349535] ignition[846]:       {
[   37.349743] ignition[846]:         "filesystem": "root",
[   37.350141] ignition[846]:         "path": "/etc/crio/seccomp.json",
[   37.350544] ignition[846]:         "contents": {
[   37.350923] ignition[846]:           "source": "s3://platform-bootstrap-dev/binaries/etccrio/v1.14.6a/seccomp.json",
[   37.351305] ignition[846]:           "verification": {}
[   37.351714] ignition[846]:         },
[   37.352083] ignition[846]:         "mode": 420
[   37.352481] ignition[846]:       },
[   37.352895] ignition[846]:       {
[   37.353110] ignition[846]:         "filesystem": "root",
[   37.353562] ignition[846]:         "path": "/opt/bin/generate_certs.sh",
[   37.354077] systemd[1]: ignition-disks.service: Main process exited, code=exited, status=1/FAILURE
[   37.354422] ignition[846]:         "contents": {
[   37.354696] ignition[846]:           "source": "s3://platform-bootstrap-dev-us-west-2/platform-instances/aws-9405/artifacts/generate_certs.sh",
[   37.354889] ignition[846]:           "verification": {}
[   37.355269] ignition[846]:         },
[   37.355656] ignition[846]:         "mode": 493
[   37.356027] ignition[846]:       },
[   37.356430] ignition[846]:       {
[   37.356621] ignition[846]:         "filesystem": "root",
[   37.356996] ignition[846]:         "path": "/root-ca/etcd-ca.crt",
[   37.357415] ignition[846]:         "contents": {
[   37.357654] ignition[846]:           "source": "s3://platform-bootstrap-dev-us-west-2/platform-instances/aws-9405/etcd-ca.crt",
[   37.358042] ignition[846]:           "verification": {}
[   37.358432] ignition[846]:         },
[   37.358790] ignition[846]:         "mode": 420
[   37.358983] ignition[846]:       },
[   37.359375] ignition[846]:       {
[   37.359755] ignition[846]:         "filesystem": "root",
[   37.360127] ignition[846]:         "path": "/root-ca/etcd-ca.key",
[   37.360536] ignition[846]:         "contents": {
[   37.360964] ignition[846]:           "source": "s3://platform-bootstrap-dev-us-west-2/platform-instances/aws-9405/etcd-ca.key",
[   37.361380] ignition[846]:           "verification": {}
[   37.361819] ignition[846]:         },
[   37.362285] ignition[846]:         "mode": 420
[   37.362745] ignition[846]:       },[
�[0;32m  OK  �[0m] Mounted �[0;1;39m/sysroot/usr�[0m.
[   37.411095] ignition[846]:       {
[�[0;32m  OK  �[0m] Reached target �[0;1;39mLocal File Systems�[0m.
[   37.411428] ignition[846]:         "filesystem": "root",
[   37.411815] ignition[846]:         "path": "/root-ca/k8s-ca.crt",
[   37.412203] ignition[846]:         "contents": {
[   37.412573] ignition[846]:           "source": "s3://platform-bootstrap-dev-us-west-2/platform-root-ca/k8s-ca.crt",
[   37.412962] ignition[846]:           "verification": {}
[   37.413348] ignition[846]:         },
[   37.413721] ignition[846]:         "mode": 420
[   37.414092] ignition[846]:       },
[   37.414477] ignition[846]:       {
[   37.414845] ignition[846]:         "filesystem": "root",
[   37.415426] ignition[846]:         "path": "/root-ca/k8s-ca.key",
[   37.415810] ignition[846]:         "contents": {
[   37.416190] ignition[846]:           "source": "s3://platform-bootstrap-dev-us-west-2/platform-root-ca/k8s-ca.key",
[   37.416556] ignition[846]:           "verification": {}
[   37.416930] ignition[846]:         },
[   37.417313] ignition[846]: [   37.714200] serial8250: too much work for irq4
        "mode": 420
[   37.417696] ignition[846]:       },
[   37.418082] ignition[846]:       {
[   37.418468] ignition[846]:         "filesystem": "root",
[   37.418854] ignition[846]:         "path": "/root/.docker/config.json",
[   37.419327] ignition[846]:         "contents": {
[   37.419813] ignition[846]:           "source": "data:,%7B%0A%09%22auths%22%3A%20%7B%0A%09%09%22gitlab-registry.nordstrom.com%22%3A%20%7B%0A%09%09%09%22auth%22%3A%20%22azhzLXJlYWQtb25seTotLXhrZVgySjZEeTRSOTZ3ZXhuVA%3D%3D%22%0A%09%09%7D%0A%09%7D%0A%7D%0A",
[   37.420286] ignition[846]:           "verification": {}
[   37.420746] ignition[846]:         },
         Starting �[0;1;39mRoot filesystem setup�[0m...
[   37.421920] ignition[846]:         "mode": 493
[   37.422385] ignition[846]:       },
[   37.422620] ignition[846]:       {
[   37.422849] ignition[846]:         "filesystem": "root",
[   37.423091] ignition[846]:         "path": "/etc/rkt/auth.d/gitlab-registry.json",
[   37.423554] ignition[846]:         "contents": {
[   37.423918] systemd[1]: ignition-disks.service: Failed with result 'exit-code'.
[   37.424334] ignition[846]:           "source": "data:,%7B%0A%09%22rktKind%22%3A%20%22dockerAuth%22%2C%0A%09%22rktVersion%22%3A%20%22v1%22%2C%0A%09%22registries%22%3A%20%5B%22gitlab-registry.nordstrom.com%22%5D%2C%0A%09%22credentials%22%3A%20%7B%0A%09%09%22user%22%3A%20%22k8s-read-only%22%2C%0A%09%09%22password%22%3A%20%22--xkeX2J6Dy4R96wexnT%22%0A%09%7D%0A%7D%0A",
[   37.424779] ignition[846]:           "verification": {}
[   37.425013] ignition[846]:         },
[   37.425490] ignition[846]:         "mode": 493
[   37.425891] ignition[846]:       },
[   37.426415] ignition[846]:       {
[   37.426643] ignition[846]:         "filesystem": "root",
[   37.427083] ignition[846]:         "path": "/etc/coreos/update.conf",
[   37.427477] ignition[846]:         "contents": {
[   37.427743] ignition[846]:           "source": "data:,%0AREBOOT_STRATEGY%3D%22off%22",
[   37.428121] ignition[846]:           "verification": {}
[   37.428546] ignition[846]:         },
[   37.428844] ignition[846]:         "mode": 420
[   37.429090] ignition[846]:       }
[   37.429498] ignition[846]:     ],
[   37.431038] ignition[846]:     "filesystems": [
[   37.431569] ignition[846]:       {
[   37.432558] ignition[846]:         "name": "root",
[   37.432996] ignition[846]:         "path": "/sysroot"
[   37.433426] ignition[846]:       },
[   37.433855] ignition[846]:       {
[   37.434285] ignition[846]:         "mount": {
[   37.434710] ignition[846]:           "device": "/dev/nvme1n1",
[   37.435159] ignition[846]:           "format": "ext4",
[   37.435582] ignition[846]:           "wipeFilesystem": true
[   37.436018] ignition[846]:         },
[   37.436446] ignition[846]:         "name": "etcddatafs"
[   37.436884] ignition[846]:       },
[   37.437113] ignition[846]:       {
[   37.437552] ignition[846]:         "mount": {
[   37.437969] ignition[846]:           "device": "/dev/xvdp",
[   37.438400] ignition[846]:           "format": "ext4",
[   37.438837] ignition[846]:           "wipeFilesystem": true
[   37.439276] ignition[846]:         },
[   37.439716] ignition[846]:         "name": "imagefs"
[   37.440138] ignition[846]:       },
[   37.440546] ignition[846]:       {
[   37.440976] ignition[846]:         "mount": {
[   37.441406] ignition[846]:           "device": "/dev/xvdc",
[   37.441848] ignition[846]:           "format": "ext4",
[   37.442302] ignition[846]:           "wipeFilesystem": true
[   37.442732] ignition[846]:         },
[   37.443177] ignition[846]:         "name": "nodefs"
[   37.443421] ignition[846]:       }
[   37.443839] ignition[846]:     ]
[   37.444074] ignition[846]:   },
[   37.444498] ignition[846]:   "systemd": {
[   37.444911] ignition[846]:     "units": [
[   37.445408] ignition[846]:       {
[   37.445863] ignition[846]:         "enabled": true,
[   37.446314] ignition[846]:         "name": "coreos-metadata-sshkeys@.service"
[   37.446757] ignition[846]:       },
[   37.447196] ignition[846]:       {
[   37.447467] ignition[846]:         "dropins": [
[   37.447933] ignition[846]:           {
[   37.448356] ignition[846]:             "contents": "[Service]\nEnvironment=\"DOCKER_OPTS=--log-opt max-size=20m --log-opt max-file=10\"",
[   37.448786] ignition[846]:             "name": "20-clct-docker.conf"
[   37.449230] ignition[846]:           }
[   37.449670] ignition[846]:         ],
[   37.450090] ignition[846]:         "enable": true,
[   37.450532] ignition[846]:         "name": "docker.service"
[   37.450962] ignition[846]:       },
[   37.451395] ignition[846]:       {
[   37.451884] ignition[846]:         "dropins": [
[   37.452132] ignition[846]:           {
[   37.452676] systemd[1]: Failed to start Ignition (disks).
[   37.453399] ignition[846]:             "contents": "[Unit]\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\n\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=\"ETCD_IMAGE_TAG=v3.3.15\"\nExecStart=\nExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS \\\n  --name=\"${COREOS_EC2_HOSTNAME}\" \\\n  --data-dir=\"/var/lib/etcd/data\" \\\n  --listen-peer-urls=\"https://${COREOS_EC2_IPV4_LOCAL}:2380\" \\\n  --listen-client-urls=\"https://0.0.0.0:2379\" \\\n  --initial-advertise-peer-urls=\"https://${COREOS_EC2_HOSTNAME}:2380\" \\\n  --advertise-client-urls=\"https://${COREOS_EC2_HOSTNAME}:2379\" \\\n  --cert-file=\"/etc/ssl/certs/tls.crt\" \\\n  --key-file=\"/etc/ssl/certs/tls.key\" \\\n  --client-cert-auth=true \\\n  --trusted-ca-file=\"/etc/ssl/certs/ca.crt\" \\\n  --peer-cert-file=\"/etc/ssl/certs/tls.crt\" \\\n  --peer-key-file=\"/etc/ssl/certs/tls.key\" \\\n  --peer-client-cert-auth=true \\\n  --peer-trusted-ca-file=\"/etc/ssl/certs/ca.crt\"",
[[   37.500954] �[0;32m  OK  �[0m] Started �[0;1;39mRoot filesystem setup�[0m.
ignition[846]:             "name": "20-clct-etcd-member.conf"
[   37.502631] ignition[846]:           }
[   37.503042] ignition[846]:         ],
         Starting �[0;1;39mReload Configuration from the Real Root�[0m...
[   37.503483] ignition[846]:         "enable": true,
[   37.503916] ignition[846]:         "name": "etcd-member.service"
[   37.504366] ignition[846]:       },
[   37.504807] ignition[846]:       {
[   37.505264] ignition[846]:         "mask": true,
[   37.505706] ignition[846]:         "name": "update-engine.service"
[   37.506162] ignition[846]:       },
[   37.506598] ignition[846]:       {
[   37.507033] ignition[846]:         "mask": true,
[   37.507482] ignition[846]:         "name": "locksmithd.service"
[   37.507923] ignition[846]:       },
[   37.508378] ignition[846]:       {
[   37.508821] ignition[846]:         "enable": true,
[   37.509267] ignition[846]:         "name": "fstrim.timer"
[   37.509694] ignition[846]:       },
[   37.510124] ignition[846]:       {
[   37.510568] ignition[846]:         "contents": "[Unit]\nDescription=Mount instance storage to /var/lib/etcd\n[Mount]\nWhat=/dev/nvme1n1\nWhere=/var/lib/etcd\nType=ext4\n",
[   37.511007] ignition[846]:         "name": "var-lib-etcd.mount"
[   37.511437] ignition[846]:       },
[   37.511864] ignition[846]:       {
[   37.512325] ignition[846]:         "contents": "[Unit]\nDescription=Mount ephemeral to /var/lib/docker\n[Mount]\nWhat=/dev/xvdp\nWhere=/var/lib/docker\nType=ext4\n",
[   37.512774] ignition[846]:         "name": "var-lib-docker.mount"
[   37.513235] ignition[846]:       },
[   37.513647] ignition[846]:       {
[   37.514072] ignition[846]:         "contents": "[Unit]\nDescription=Mount ephemeral to /var/lib/kubelet\n[Mount]\nWhat=/dev/xvdc\nWhere=/var/lib/kubelet\nType=ext4\n",
[   37.514492] ignition[846]:         "name": "var-lib-kubelet.mount"
[   37.514929] ignition[846]:       },
[   37.515369] ignition[846]:       {
[   37.515970] systemd[1]: Dependency failed for Initrd Default Target.
[   37.516590] ignition[846]:         "contents": "[Unit]\nDescription=Update kernel tuning params\n[Service]\nExecStartPre=/sbin/sysctl -w net.ipv4.neigh.default.gc_thresh3=15000 ;\\\n  /sbin/sys[   37.821764] serial8250: too much work for irq4
ctl -w net.ipv4.neigh.default.gc_thresh2=12500 ;\\\n  /sbin/sysctl -w net.ipv4.neigh.default.gc_thresh1=10000 ;\\\n  /sbin/sysctl -w fs.inotify.max_user_watches=131072\nExecStart=/bin/echo \"Set kernel tuning params\"\nRemainAfterExit=true\nRestart=on-failure\nRestartSec=10\n[Install]\nWantedBy=multi-user.target\n",
[   37.516929] ignition[846]:         "enable": true,
[   37.517593] ignition[846]:         "name": "params.service"
[   37.518023] ignition[846]:       },
[   37.518462] ignition[846]:       {
[   37.518899] ignition[846]:         "dropins": [
[   37.519338] ignition[846]:           {
[   37.519761] ignition[846]:             "contents": "[Unit]\nAfter=var-lib-docker.mount\nRequires=var-lib-docker.mount\n",
[   37.520193] ignition[846]:             "name": "20-await-var-lib-docker-mount.conf"
[   37.520622] ignition[846]:           }
[   37.521045] ignition[846]:         ],
[   37.521476] ignition[846]:         "name": "docker.service"
[   37.521906] ignition[846]:       },
[   37.522347] ignition[846]:       {
[   37.522813] systemd[1]: initrd.target: Job initrd.target/start failed with result 'dependency'.
[   37.523463] ignition[846]:         "contents": "[Unit]\nDescription=Generate etcd certs\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\nConditionFileNotEmpty=/opt/bin/generate_certs.sh\nConditionFileNotEmpty=!/etc/ssl/etcd/tls.crt\nConditionFileNotEmpty=!/etc/ssl/etcd/tls.key\nConditionFileNotEmpty=!/etc/ssl/etcd/ca.crt\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=ASSETS_DIR=/etc/ssl/etcd\nExecStartPre=/usr/bin/mkdir -p $ASSETS_DIR\nExecStartPre=/bin/sh -c \"\\\n  CERT_COMMON_NAME=etcd.aws-9405.platform-dev.r53.nordstrom.net \\\n  CERT_IP_SANS=${COREOS_EC2_IPV4_LOCAL},127.0.0.1 \\\n  CERT_ALT_NAMES=etcd.aws-9405.platform-dev.r53.nordstrom.net,${COREOS_EC2_HOSTNAME},localhost \\\n  ROOT_CA_KEY=\"/root-ca/etcd-ca.key\" \\\n  ROOT_CA_CERT=\"/root-ca/etcd-ca.crt\" \\\n  /opt/bin/generate_certs.sh\"\nExecStartPre=/bin/rm /root-ca/etcd-ca.key /root-ca/etcd-ca.crt\nExecStartPre=/usr/bin/curl -v https://discovery.etcd.io\nExecStartPre=/usr/bin/sh -c \[2019-09-12T23:21:02.140590]"echo | \\\n  openssl s_client -connect discovery.etcd.io:443 -prexit -showcerts 2\u003e /dev/null \\\n  | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' \\\n  \u003e /etc/ssl/etcd/ca-certificates.crt\"\nExecStart=/bin/echo \"Etcd certs generated\"\nRemainAfterExit=true\nRestart=on-failure\nRestartSec=10\n[Install]\nWantedBy=multi-user.target\n",
[   37.523892] ignition[846]:         "enable": true,
[   37.524314] ignition[846]:         "name": "generate-etcd-certs.service"
[   37.524742] ignition[846]:       },
[   37.525956] ignition[846]:       {
[   37.526530] ignition[846]:         "dropins": [
[   37.526966] ignition[846]:           {
[   37.527427] systemd[1]: initrd.target: Triggering OnFailure= dependencies.
[   37.528087] systemd-fsck[1074]: ROOT: clean, 565/553792 files, 37484/553472 blocks
[   37.528743] ignition[846]:             "contents": "[Unit]\nRequires=var-lib-etcd.mount\nAfter=var-lib-etcd.mount\nRequires=generate-etcd-certs.service\nAfter=generate-etcd-certs.service\n[Service]\nEnvironment=\"ETCD_IMAGE_URL=quay.io/coreos/etcd\"\nEnvironment=\"RKT_GLOBAL_ARGS=\"\nEnvironment=\"ETCD_STRICT_RECONFIG_CHECK=true\"\nEnvironment=\"ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:9379\"\nEnvironment=\"ETCD_DISCOVERY=https://discovery.etcd.io/9cc5ec5fe416532f25a4cde484d304f8\"\nEnvironment=\"ETCD_SSL_DIR=/etc/ssl/etcd\"\nOOMScoreAdjust=-999\nExecStartPre=/usr/bin/chown -R etcd:etcd /var/lib/etcd\nExecStartPre=/usr/bin/chown -R etcd:etcd /etc/ssl/etcd\n",
[   37.579942] ignition[846]:             "name": "01-conf.conf"
[   37.580596] ignition[846]:           }
[   37.581008] ignition[846]:         ],
[   37.581385] ignition[846]:         "enable": true,
[   37.581765] ignition[846]:         "name": "etcd-member.service"
[   37.582123] ignition[846]:       },
[   37.582495] ignition[846]:       {
[   37.582872] ignition[846]:         "contents": "[Unit]\nDescription=Generate kubelet certs\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\nConditionFileNotEmpty=/opt/bin/generate_certs.sh\nConditionFileNotEmpty=!/etc/kubernetes/tls.crt\nConditionFileNotEmpty=!/etc/kubernetes/tls.key\nConditionFileNotEmpty=!/etc/kubernetes/ca.crt\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=ASSETS_DIR=/etc/kubernetes\nExecStartPre=/usr/bin/mkdir -p $ASSETS_DIR\nExecStartPre=/bin/sh -c \"\\\n  CERT_COMMON_NAME=system:node:${COREOS_EC2_HOSTNAME} \\\n  CERT_ORGANIZATION=system:nodes \\\n  CERT_ALT_NAMES=${COREOS_EC2_HOSTNAME} \\\n  ROOT_CA_KEY=/root-ca/k8s-ca.key \\\n  ROOT_CA_CERT=/root-ca/k8s-ca.crt \\\n  /opt/bin/generate_certs.sh\"\nExecStartPre=/bin/rm /root-ca/k8s-ca.key /root-ca/k8s-ca.crt\nExecStart=/bin/echo \"Kubelet certs generated\"\nRemainAfterExit=true\nRestart=on-failure\nRestartSec=10\n[Install]\nWantedBy=multi-user.target\n",
[   37.583442] ignition[846]:         "enable": true,
[   37.583805] ignition[846]:         "name": "generate-kubelet-certs.service"
[   37.584363] ignition[846]:       },
[   37.584742] ignition[846]:       {
[   37.585246] systemd[1]: Dependency failed for Ignition (files).
[   37.585888] systemd[1]: Dependency failed for Populate torcx store to satisfy profile.
[   37.586745] ignition[846]:         "contents": "[Unit]\nDescription=Kubernetes Kubelet\nDocumentation=https://github.com/kubernetes/kubernetes\nRequires=generate-kubelet-certs.service\nAfter=generate-kubelet-certs.service\nRequires=var-lib-kubelet.mount\nAfter=var-lib-kubelet.mount\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=KUBELET_IMAGE=docker://gcr.io/google_containers/hyperkube:v1.15.3\n# additive to kubelet-wrapper\nEnvironment=\"RKT_GLOBAL_ARGS=--insecure-options=image\"\nEnvironment=\"RKT_RUN_ARGS= \\\n  --volume dns,kind=host,source=/etc/resolv.conf \\\n  --volume var-lib-cni,kind=host,source=/var/lib/cni \\\n  --volume var-log,kind=host,source=/var/log \\\n  --volume cni-bin,kind=host,source=/run/torcx/unpack/crio/usr/share/cni/bin/ \\\n  --mount volume=cni-bin,target=/opt/cni/bin \\\n  --mount volume=dns,target=/etc/resolv.conf \\\n  --mount volume=var-lib-cni,target=/var/lib/cni \\\n  --mount volume=var-log,target=/var/log\"\nExecStartPre=/bin/mkdir[2019-09-12T23:21:02.239909] -p /etc/kubernetes/manifests\nExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d\nExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets\nExecStartPre=/bin/mkdir -p /srv/kubernetes/manifests\nExecStartPre=/bin/mkdir -p /var/lib/cni\nExecStartPre=/usr/bin/mkdir -p /var/log/containers\nExecStartPre=/bin/cp /root/.docker/config.json /var/lib/kubelet/config.json\nExecStart=/usr/lib/coreos/kubelet-wrapper \\\n  --anonymous-auth=false \\\n  --read-only-port=0 \\\n  --cgroup-driver=cgroupfs \\\n  --cloud-provider=aws \\\n  --client-ca-file=/etc/kubernetes/ca.crt \\\n  --tls-cert-file=/etc/kubernetes/tls.crt \\\n  --tls-private-key-file=/etc/kubernetes/tls.key \\\n  --cluster_dns=${COREOS_EC2_IPV4_LOCAL},25.0.0.10 \\\n  --cluster_domain=cluster.local \\\n  --exit-on-lock-contention \\\n  --feature-gates=LocalStorageCapacityIsolation=true \\\n  --lock-file=/var/run/lock/kubelet.lock \\\n  --kubeconfig=/etc/kubernetes/kubelet/kubeconfig \\\n  --network-plugin=cni \\\n  --cni-conf-[2019-09-12T23:21:02.239929]dir=/etc/kubernetes/cni/net.d \\\n  --pod-manifest-path=/etc/kubernetes/manifests \\\n  --node-labels=master=true,\"beta.nordstrom.net/node-pool\"=\"etcd-pool\",\"node-role.kubernetes.io/master\"=true \\\n  --register-with-taints=dedicate=\"etcd:NoSchedule\" \\\n  --system-reserved=\"cpu=350m,memory=3Gi\" \\\n  --eviction-hard=\"memory.available\u003c10%,nodefs.available\u003c10%,nodefs.inodesFree\u003c10%,imagefs.available\u003c10%,imagefs.inodesFree\u003c10%\" \\\n  --eviction-soft=\"memory.available\u003c15%,nodefs.available\u003c20%,nodefs.inodesFree\u003c20%,imagefs.available\u003c20%,imagefs.inodesFree\u003c20%\" \\\n  --eviction-soft-grace-period=\"memory.available=2m,nodefs.available=30s,nodefs.inodesFree=30s,imagefs.available=30s,imagefs.inodesFree=30s\" \\\n  --eviction-minimum-reclaim=\"memory.available=0Mi,nodefs.available=5Gi,imagefs.available=5Gi\" \\\n  --e[   37.925411] serial8250: too much work for irq4
viction-max-pod-grace-period=\"300\" \\\n  --authentication-token-webhook \\\n  --cpu-manager-policy=static\nRestart=always\nRestartSec=5\n[Install]\nWantedBy=multi-user.target\n",
[   37.633021] ignition[846]:         "enable": true,
[   37.633418] ignition[846]:         "name": "kubelet.service"
[   37.633851] ignition[846]:       }
[   37.634505] ignition[846]:     ]
[   37.634941] ignition[846]:   }
[   37.635419] systemd[1]: torcx-profile-populate.service: Job torcx-profile-populate.service/start failed with result 'dependency'.
[   37.636127] ignition[846]: }
[   37.636557] systemd[1]: ignition-files.service: Job ignition-files.service/start failed with result 'dependency'.
[   37.637188] systemd[1]: Reached target Local File Systems (Pre).
[   37.637434] systemd[1]: Starting Verity Setup for /dev/mapper/usr...
[   37.637876] systemd[1]: Starting Setup Virtual Console...
[   37.638315] systemd[1]: Stopping Network Service...
[   37.638817] systemd[1]: dracut-pre-mount.service: Succeeded.
[   37.639213] systemd[1]: Stopped dracut pre-mount hook.
[   37.639635] systemd[1]: Starting File System Check on /dev/disk/by-label/ROOT...
[   37.639914] systemd[1]: dracut-initqueue.service: Succeeded.
[   37.640459] systemd[1]: Stopped dracut initqueue hook.
[   37.640756] systemd[1]: dracut-pre-trigger.service: Succeeded.
[   37.652578] systemd[1]: Stopped dracut pre-trigger hook.
[[   37.653284] systemd�[0;32m  OK  �[0m[1]: ] systemd-ask-password-console.path: Succeeded.Started �[0;1;39mReload Configuration from the Real Root�[0m.

[   37.653945] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch.
[�[0;32m  OK  �[0m] Reached target �[0;1;39mInitrd File Systems�[0m.
[   37.654252] systemd[1]: dracut-pre-udev.service: Succeeded.
[   37.654692] systemd[1]: Stopped dracut pre-udev hook.
[   37.654923] systemd[1]: dracut-cmdline.service: Succeeded.
         Starting �[0;1;39mCleaning Up and Shutting Down Daemons�[0m...
[   37.655390] systemd[1]: Stopped dracut cmdline hook.
[   37.655838] systemd[1]: systemd-vconsole-setup.service: Succeeded.
[   37.656164] systemd[1]: Started Setup Virtual Console.
[   37.656260] systemd[1]: Started File System Check on /dev/disk/by-label/ROOT.
[   37.656487] systemd[1]: Started Verity Setup for /dev/mapper/usr.
[   37.656722] systemd[1]: Found device /dev/mapper/usr.
[   37.656955] systemd[1]: Condition check resulted in File System Check on /dev/mapper/usr being skipped.
[   37.657413] systemd[1]: Mounting /sysroot...
[   37.657648] systemd[1]: Starting Emergency Shell...
[   37.657879] systemd[1]: Mounted /sysroot.
[   37.658114] systemd[1]: Mounting /sysroot/usr...
[   37.658571] systemd[1]: Reached target Initrd Root File System.
[   37.658809] systemd[1]: Mounted /sysroot/usr.
[   37.659043] systemd[1]: Reached target Local File Systems.
[   37.659499] systemd[1]: Starting Root filesystem setup...
[   37.659730] systemd[1]: initrd-setup-root.service: Succeeded.
[   37.659958] systemd[1]: Started Root filesystem setup.
[   37.660430] systemd[1]: Starting Reload Configuration from the Real Root...
[   37.660651] systemd[1]: Reloading.
[   37.660876] systemd[1]: initrd-parse-etc.service: Succeeded.
[   37.661105] systemd[1]: Started Reload Configuration from the Real Root.
[   37.661583] systemd[1]: Reached target Initrd File Systems.
[   37.662001] systemd[1]: Starting Cleaning Up and Shutting Down Daemons...
[   37.662756] systemctl[1234]: Failed to start initrd-switch-root.target: Transaction for initrd-switch-root.target/start is destructive (emergency.service has 'start' job queued, but 'stop' is included in transaction).
[   37.663021] systemctl[1234]: See system logs and 'systemctl status initrd-switch-root.target' for details.
[   37.663419] systemd[1]: Requested transaction contradicts existing jobs: Transaction for initrd-switch-root.target/start is destructive (emergency.service has 'start' job queued, but 'stop' is included in transaction).
[�[0;1;31mFAILED�[0m] Failed to start �[0;1;39mCleaning Up and Shutting Down Daemons�[0m.
See 'systemctl status initrd-cleanup.service' for details.
         Starting �[0;1;39mSetup Virtual Console�[0m...
[   37.666892] systemd[1]: initrd-cleanup.service: Main process exited, code=exited, status=4/NOPERMISSION
[   37.671334] systemd[1]: initrd-cleanup.service: Failed with result 'exit-code'.
[   37.823943] systemd[1]: Failed to start Cleaning Up and Shutting Down Daemons.
[�[0;32m  OK  �[0m] Started �[0;1;39mSetup Virtual Console�[0m.
[   37.827097] systemd[1]: initrd-cleanup.service: Triggering OnFailure= dependencies.
[   37.828783] systemd[1]: Starting Setup Virtual Console...
[   37.829608] systemd[1]: systemd-vconsole-setup.service: Succeeded.
[   37.830615] systemd[1]: Started Setup Virtual Console.
[   37.917350] systemd-networkd[266]: lo: Lost carrier
[   37.925219] systemd-networkd[266]: eth0: Lost carrier
[   37.925365] systemd-networkd[266]: eth0: DHCP lease lost
[   37.925889] systemd-networkd[266]: eth0: IPv6 successfully enabled
[   37.937444] systemd[1]: systemd-networkd.service: Succeeded.
[�[0;32m  OK  �[0m] Stopped �[0;1;39mNetwork Service�[0m.
[   37.937776] systemd[1]: Stopped Network Service.
cat: unrecognized option '-------------------------------------------------------------------------------'
Try 'cat --help' for more information.
�[0;1;31m�[0mIgnition v0.33.0
�[0;1;31m�[0mreading system config file "/usr/lib/ignition/base.ign"
�[0;1;31m�[0mparsing config with SHA512: 865c03baa79b8c74023d13a0b3666474fa06a165421a1e05731b76e0f557d42c5c89d4870a0b9c4182ad7d4d8209de20dca9c9da63d637e0410fbd60314cac6c
�[0;1;31m�[0mparsed url from cmdline: ""
�[0;1;31m�[0mno config URL provided
�[0;1;31m�[0mreading system config file "/usr/lib/ignition/user.ign"
�[0;1;31m�[0mno config at "/usr/lib/ignition/user.ign"
�[0;1;31m�[0mGET http://169.254.169.254/2009-04-04/user-data: attempt #1
�[0;1;31m�[0mGET result: OK
�[0;1;31m�[0mparsing config with SHA512: c192cb154c8d98f80afade2650befb222840b50419ba7876edfb4f2bcbb3a60f9f3d2430f868a163906d3d17da870dbde35e78aa5217a07454a05dad10914771
�[0;1;31m�[0mdisks: createPartitions: op(1): [started]  waiting for devices [/dev/nvme1n1 /dev/xvdp /dev/xvdc]
�[0;1;31m�[0mdisks: createPartitions: op(1): [finished] waiting for devices [/dev/nvme1n1 /dev/xvdp /dev/xvdc]
�[0;1;31m�[0mdisks: createPartitions: created device alias for "/dev/nvme1n1": "/run/ignition/dev_aliases/dev/nvme1n1" -> "/dev/nvme1n1"
�[0;1;31m�[0mdisks: createPartitions: created device alias for "/dev/xvdp": "/run/ignition/dev_aliases/dev/xvdp" -> "/dev/nvme2n1"
�[0;1;31m�[0mdisks: createPartitions: created device alias for "/dev/xvdc": "/run/ignition/dev_aliases/dev/xvdc" -> "/dev/nvme1n1"
�[0;1;31m�[0mdisks: createPartitions: op(2): [started]  partitioning "/run/ignition/dev_aliases/dev/nvme1n1"
�[0;1;31m�[0mdisks: createPartitions: op(2): wiping partition table requested on "/run/ignition/dev_aliases/dev/nvme1n1"
�[0;1;31m�[0mdisks: createPartitions: op(2): running sgdisk with options: [--zap-all /run/ignition/dev_aliases/dev/nvme1n1]
�[0;1;31m�[0mdisks: createPartitions: op(2): op(3): [started]  deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/nvme1n1"
�[0;1;31m�[0mdisks: createPartitions: op(2): op(3): executing: "/usr/sbin/sgdisk" "--zap-all" "/run/ignition/dev_aliases/dev/nvme1n1"
�[0;1;31m�[0mdisks: createPartitions: op(2): op(3): [finished] deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/nvme1n1"
�[0;1;31m�[0mdisks: createPartitions: op(2): op(4): [started]  reading partition table of "/run/ignition/dev_aliases/dev/nvme1n1"
�[0;1;31m�[0mdisks: createPartitions: op(2): op(4): [finished] reading partition table of "/run/ignition/dev_aliases/dev/nvme1n1"
�[0;1;31m�[0mdisks: createPartitions: op(2): running sgdisk with options: [--pretend --new=0:0:+0 /run/ignition/dev_aliases/dev/nvme1n1]
�[0;1;31m�[0mdisks: createPartitions: op(2): running sgdisk with options: [--new=0:0:+0 --change-name=0:ETCDDATAFS /run/ignition/dev_aliases/dev/nvme1n1]
�[0;1;31m�[0mdisks: createPartitions: op(2): op(5): [started]  deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/nvme1n1"
�[0;1;31m�[0mdisks: createPartitions: op(2): op(5): executing: "/usr/sbin/sgdisk" "--new=0:0:+0" "--change-name=0:ETCDDATAFS" "/run/ignition/dev_aliases/dev/nvme1n1"
�[0;1;31m�[0mdisks: createPartitions: op(2): op(5): [finished] deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/nvme1n1"
�[0;1;31m�[0mdisks: createPartitions: op(2): [finished] partitioning "/run/ignition/dev_aliases/dev/nvme1n1"
�[0;1;31m�[0mdisks: createPartitions: op(6): [started]  partitioning "/run/ignition/dev_aliases/dev/xvdp"
�[0;1;31m�[0mdisks: createPartitions: op(6): wiping partition table requested on "/run/ignition/dev_aliases/dev/xvdp"
�[0;1;31m�[0mdisks: createPartitions: op(6): running sgdisk with options: [--zap-all /run/ignition/dev_aliases/dev/xvdp]
�[0;1;31m�[0mdisks: createPartitions: op(6): op(7): [started]  deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/xvdp"
�[0;1;31m�[0mdisks: createPartitions: op(6): op(7): executing: "/usr/sbin/sgdisk" "--zap-all" "/run/ignition/dev_aliases/dev/xvdp"
�[0;1;31m�[0mdisks: createPartitions: op(6): op(7): [finished] deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/xvdp"
�[0;1;31m�[0mdisks: createPartitions: op(6): op(8): [started]  reading partition table of "/run/ignition/dev_aliases/dev/xvdp"
�[0;1;31m�[0mdisks: createPartitions: op(6): op(8): [finished] reading partition table of "/run/ignition/dev_aliases/dev/xvdp"
�[0;1;31m�[0mdisks: createPartitions: op(6): running sgdisk with options: [--pretend --new=0:0:+0 /run/ignition/dev_aliases/dev/xvdp]
�[0;1;31m�[0mdisks: createPartitions: op(6): running sgdisk with options: [--new=0:0:+0 --change-name=0:IMAGEFS /run/ignition/dev_aliases/dev/xvdp]
�[0;1;31m�[0mdisks: createPartitions: op(6): op(9): [started]  deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/xvdp"
�[0;1;31m�[0mdisks: createPartitions: op(6): op(9): executing: "/usr/sbin/sgdisk" "--new=0:0:+0" "--change-name=0:IMAGEFS" "/run/ignition/dev_aliases/dev/xvdp"
�[0;1;31m�[0mdisks: createPartitions: op(6): op(9): [finished] deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/xvdp"
�[0;1;31m�[0mdisks: createPartitions: op(6): [finished] partitioning "/run/ignition/dev_aliases/dev/xvdp"
�[0;1;31m�[0mdisks: createPartitions: op(a): [started]  partitioning "/run/ignition/dev_aliases/dev/xvdc"
�[0;1;31m�[0mdisks: createPartitions: op(a): wiping partition table requested on "/run/ignition/dev_aliases/dev/xvdc"
�[0;1;31m�[0mdisks: createPartitions: op(a): running sgdisk with options: [--zap-all /run/ignition/dev_aliases/dev/xvdc]
�[0;1;31m�[0mdisks: createPartitions: op(a): op(b): [started]  deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/xvdc"
�[0;1;31m�[0mdisks: createPartitions: op(a): op(b): executing: "/usr/sbin/sgdisk" "--zap-all" "/run/ignition/dev_aliases/dev/xvdc"
�[0;1;31m�[0mdisks: createPartitions: op(a): op(b): [finished] deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/xvdc"
�[0;1;31m�[0mdisks: createPartitions: op(a): op(c): [started]  reading partition table of "/run/ignition/dev_aliases/dev/xvdc"
�[0;1;31m�[0mdisks: createPartitions: op(a): op(c): [finished] reading partition table of "/run/ignition/dev_aliases/dev/xvdc"
�[0;1;31m�[0mdisks: createPartitions: op(a): running sgdisk with options: [--pretend --new=0:0:+0 /run/ignition/dev_aliases/dev/xvdc]
�[0;1;31m�[0mdisks: createPartitions: op(a): running sgdisk with options: [--new=0:0:+0 --change-name=0:NODEFS /run/ignition/dev_aliases/dev/xvdc]
�[0;1;31m�[0mdisks: createPartitions: op(a): op(d): [started]  deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/xvdc"
�[0;1;31m�[0mdisks: createPartitions: op(a): op(d): executing: "/usr/sbin/sgdisk" "--new=0:0:+0" "--change-name=0:NODEFS" "/run/ignition/dev_aliases/dev/xvdc"
�[0;1;31m�[0mdisks: createPartitions: op(a): op(d): [finished] deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/xvdc"
�[0;1;31m�[0mdisks: createPartitions: op(a): [finished] partitioning "/run/ignition/dev_aliases/dev/xvdc"
�[0;1;31m�[0mdisks: createFilesystems: op(e): [started]  waiting for devices [/dev/nvme1n1 /dev/xvdp /dev/xvdc]
�[0;1;31m�[0mdisks: createFilesystems: op(e): [finished] waiting for devices [/dev/nvme1n1 /dev/xvdp /dev/xvdc]
�[0;1;31m�[0mdisks: createFilesystems: created device alias for "/dev/nvme1n1": "/run/ignition/dev_aliases/dev/nvme1n1" -> "/dev/nvme1n1"
�[0;1;31m�[0mdisks: createFilesystems: created device alias for "/dev/xvdp": "/run/ignition/dev_aliases/dev/xvdp" -> "/dev/nvme2n1"
�[0;1;31m�[0mdisks: createFilesystems: created device alias for "/dev/xvdc": "/run/ignition/dev_aliases/dev/xvdc" -> "/dev/nvme1n1"
�[0;1;31m�[0mdisks: createFilesystems: op(f): [started]  determining filesystem type of "/dev/nvme1n1"
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(10): [started]  determining filesystem type of "/dev/xvdp"
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(10): op(11): [started]  determining filesystem type of "/dev/xvdc"
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(10): op(11): found  filesystem at "/dev/nvme1n1" with uuid "" and label ""
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(10): op(11): [finished] determining filesystem type of "/dev/nvme1n1"
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(10): op(12): [started]  creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/nvme1n1"
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(10): op(12): executing: "/usr/sbin/mkfs.ext4" "-F" "/run/ignition/dev_aliases/dev/nvme1n1"
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(10): op(12): found  filesystem at "/dev/xvdc" with uuid "" and label ""
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(10): op(12): [finished] determining filesystem type of "/dev/xvdc"
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(10): op(13): [started]  creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/xvdc"
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(10): op(13): executing: "/usr/sbin/mkfs.ext4" "-F" "/run/ignition/dev_aliases/dev/xvdc"
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(10): op(13): [failed]   creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/xvdc": exit status 1: Cmd: "/usr/sbin/mkfs.ext4" "-F" "/run/ignition/dev_aliases/dev/xvdc" Stdout: "" Stderr: "mke2fs 1.44.5 (15-Dec-2018)\nFound a gpt partition table in /run/ignition/dev_aliases/dev/xvdc\n/run/ignition/dev_aliases/dev/xvdc is apparently in use by the system; will not make a filesystem here!\n"
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(10): found  filesystem at "/dev/xvdp" with uuid "" and label ""
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(10): [finished] determining filesystem type of "/dev/xvdp"
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(14): [started]  creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/xvdp"
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(14): executing: "/usr/sbin/mkfs.ext4" "-F" "/run/ignition/dev_aliases/dev/xvdp"
�[0;1;31m�[0mdisks: createFilesystems: op(f): op(14): [finished] creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/xvdp"
�[0;1;31m�[0mdisks: createFilesystems: op(f): [finished] creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/nvme1n1"
�[0;1;31m�[0mIgnition failed: failed to create filesystems: mkfs failed: exit status 1: Cmd: "/usr/sbin/mkfs.ext4" "-F" "/run/ignition/dev_aliases/dev/xvdc" Stdout: "" Stderr: "mke2fs 1.44.5 (15-Dec-2018)\nFound a gpt partition table in /run/ignition/dev_aliases/dev/xvdc\n/run/ignition/dev_aliases/dev/xvdc is apparently in use by the system; will not make a filesystem here!\n"
�[0;1;31m�[0mdisks failedFull config:
�[0;1;31m�[0m{
�[0;1;31m�[0m  "ignition": {
�[0;1;31m�[0m    "config": {},
�[0;1;31m�[0m    "proxy": {},
�[0;1;31m�[0m    "security": {
�[0;1;31m�[0m      "tls": {}
�[0;1;31m�[0m    },
�[0;1;31m�[0m    "timeouts": {},
�[0;1;31m�[0m    "version": "2.4.0-experimental"
�[0;1;31m�[0m  },
�[0;1;31m�[0m  "networkd": {},
�[0;1;31m�[0m  "passwd": {},
�[0;1;31m�[0m  "storage": {
�[0;1;31m�[0m    "disks": [
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "device": "/dev/nvme1n1",
�[0;1;31m�[0m        "partitions": [
�[0;1;31m�[0m          {
�[0;1;31m�[0m            "label": "ETCDDATAFS",
�[0;1;31m�[0m            "size": 0,
�[0;1;31m�[0m            "start": 0
�[0;1;31m�[0m          }
�[0;1;31m�[0m        ],
�[0;1;31m�[0m        "wipeTable": true
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "device": "/dev/xvdp",
�[0;1;31m�[0m        "partitions": [
�[0;1;31m�[0m          {
�[0;1;31m�[0m            "label": "IMAGEFS",
�[0;1;31m�[0m            "size": 0,
�[0;1;31m�[0m            "start": 0
�[0;1;31m�[0m          }
�[0;1;31m�[0m        ],
�[0;1;31m�[0m        "wipeTable": true
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "device": "/dev/xvdc",
�[0;1;31m�[0m        "partitions": [
�[0;1;31m�[0m          {
�[0;1;31m�[0m            "label": "NODEFS",
�[0;1;31m�[0m            "size": 0,
�[0;1;31m�[0m            "start": 0
�[0;1;31m�[0m          }
�[0;1;31m�[0m        ],
�[0;1;31m�[0m        "wipeTable": true
�[0;1;31m�[0m      }
�[0;1;31m�[0m    ],
�[0;1;31m�[0m    "files": [
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/etc/kubernetes/kubelet/kubeconfig",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "data:,apiVersion%3A%20v1%0Akind%3A%20Config%0Aclusters%3A%0A-%20name%3A%20local%0A%20%20cluster%3A%0A%20%20%20%20server%3A%20https%3A%2F%2Faws-9405.platform-dev.r53.nordstrom.net%0A%20%20%20%20certificate-authority%3A%20%2Fetc%2Fkubernetes%2Fca.crt%0Ausers%3A%0A-%20name%3A%20kubelet%0A%20%20user%3A%0A%20%20%20%20client-certificate%3A%20%2Fetc%2Fkubernetes%2Ftls.crt%0A%20%20%20%20client-key%3A%20%2Fetc%2Fkubernetes%2Ftls.key%0Acontexts%3A%0A-%20context%3A%0A%20%20%20%20cluster%3A%20local%0A%20%20%20%20user%3A%20kubelet%0A",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 420
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/etc/torcx/profiles/combined.json",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "data:,%7B%22kind%22%3A%22profile-manifest-v0%22%2C%22value%22%3A%7B%22images%22%3A%5B%7B%22name%22%3A%22docker%22%2C%22reference%22%3A%2217.03%22%7D%2C%7B%22name%22%3A%22crio%22%2C%22reference%22%3A%22v1.14.6a%22%7D%5D%7D%7D%0A",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 420
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/etc/torcx/next-profile",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "data:,combined%0A",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 420
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/var/lib/torcx/store/docker:17.03.torcx.tgz",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "https://tectonic-torcx.release.core-os.net/pkgs/amd64-usr/docker/e3819eef5185486cd192ae03108003695e98e9cea4bc4b8082a0a79f212fc7eda263996c78fdac3690b0f2ebbfbdf5632f654483cc8bcd30bfb165ed9d322fbf/docker:17.03.torcx.tgz",
�[0;1;31m�[0m          "verification": {
�[0;1;31m�[0m            "hash": "sha512-e3819eef5185486cd192ae03108003695e98e9cea4bc4b8082a0a79f212fc7eda263996c78fdac3690b0f2ebbfbdf5632f654483cc8bcd30bfb165ed9d322fbf"
�[0;1;31m�[0m          }
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 420
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/etc/docker/daemon.json",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "data:,%7B%0A%20%20%22metrics-addr%22%20%3A%20%220.0.0.0%3A9323%22%2C%0A%20%20%22experimental%22%20%3A%20true%0A%7D%0A",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 420
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/var/lib/torcx/store/crio:v1.14.6a.torcx.tgz",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "s3://platform-bootstrap-dev/binaries/crio:v1.14.6a.torcx.tgz",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 420
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/etc/crio/crio.conf",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "s3://platform-bootstrap-dev/binaries/etccrio/v1.14.6a/crio.conf",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 420
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/etc/crio/seccomp.json",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "s3://platform-bootstrap-dev/binaries/etccrio/v1.14.6a/seccomp.json",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 420
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/opt/bin/generate_certs.sh",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "s3://platform-bootstrap-dev-us-west-2/platform-instances/aws-9405/artifacts/generate_certs.sh",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 493
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/root-ca/etcd-ca.crt",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "s3://platform-bootstrap-dev-us-west-2/platform-instances/aws-9405/etcd-ca.crt",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 420
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/root-ca/etcd-ca.key",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "s3://platform-bootstrap-dev-us-west-2/platform-instances/aws-9405/etcd-ca.key",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 420
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/root-ca/k8s-ca.crt",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "s3://platform-bootstrap-dev-us-west-2/platform-root-ca/k8s-ca.crt",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 420
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/root-ca/k8s-ca.key",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "s3://platform-bootstrap-dev-us-west-2/platform-root-ca/k8s-ca.key",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 420
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/root/.docker/config.json",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "data:,%7B%0A%09%22auths%22%3A%20%7B%0A%09%09%22gitlab-registry.nordstrom.com%22%3A%20%7B%0A%09%09%09%22auth%22%3A%20%22azhzLXJlYWQtb25seTotLXhrZVgySjZEeTRSOTZ3ZXhuVA%3D%3D%22%0A%09%09%7D%0A%09%7D%0A%7D%0A",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 493
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/etc/rkt/auth.d/gitlab-registry.json",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "data:,%7B%0A%09%22rktKind%22%3A%20%22dockerAuth%22%2C%0A%09%22rktVersion%22%3A%20%22v1%22%2C%0A%09%22registries%22%3A%20%5B%22gitlab-registry.nordstrom.com%22%5D%2C%0A%09%22credentials%22%3A%20%7B%0A%09%09%22user%22%3A%20%22k8s-read-only%22%2C%0A%09%09%22password%22%3A%20%22--xkeX2J6Dy4R96wexnT%22%0A%09%7D%0A%7D%0A",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 493
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "filesystem": "root",
�[0;1;31m�[0m        "path": "/etc/coreos/update.conf",
�[0;1;31m�[0m        "contents": {
�[0;1;31m�[0m          "source": "data:,%0AREBOOT_STRATEGY%3D%22off%22",
�[0;1;31m�[0m          "verification": {}
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "mode": 420
�[0;1;31m�[0m      }
�[0;1;31m�[0m    ],
�[0;1;31m�[0m    "filesystems": [
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "name": "root",
�[0;1;31m�[0m        "path": "/sysroot"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "mount": {
�[0;1;31m�[0m          "device": "/dev/nvme1n1",
�[0;1;31m�[0m          "format": "ext4",
�[0;1;31m�[0m          "wipeFilesystem": true
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "name": "etcddatafs"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "mount": {
�[0;1;31m�[0m          "device": "/dev/xvdp",
�[0;1;31m�[0m          "format": "ext4",
�[0;1;31m�[0m          "wipeFilesystem": true
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "name": "imagefs"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "mount": {
�[0;1;31m�[0m          "device": "/dev/xvdc",
�[0;1;31m�[0m          "format": "ext4",
�[0;1;31m�[0m          "wipeFilesystem": true
�[0;1;31m�[0m        },
�[0;1;31m�[0m        "name": "nodefs"
�[0;1;31m�[0m      }
�[0;1;31m�[0m    ]
�[0;1;31m�[0m  },
�[0;1;31m�[0m  "systemd": {
�[0;1;31m�[0m    "units": [
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "enabled": true,
�[0;1;31m�[0m        "name": "coreos-metadata-sshkeys@.service"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "dropins": [
�[0;1;31m�[0m          {
�[0;1;31m�[0m            "contents": "[Service]\nEnvironment=\"DOCKER_OPTS=--log-opt max-size=20m --log-opt max-file=10\"",
�[0;1;31m�[0m            "name": "20-clct-docker.conf"
�[0;1;31m�[0m          }
�[0;1;31m�[0m        ],
�[0;1;31m�[0m        "enable": true,
�[0;1;31m�[0m        "name": "docker.service"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "dropins": [
�[0;1;31m�[0m          {
�[0;1;31m�[0m            "contents": "[Unit]\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\n\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=\"ETCD_IMAGE_TAG=v3.3.15\"\nExecStart=\nExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS \\\n  --name=\"${COREOS_EC2_HOSTNAME}\" \\\n  --data-dir=\"/var/lib/etcd/data\" \\\n  --listen-peer-urls=\"https://${COREOS_EC2_IPV4_LOCAL}:2380\" \\\n  --listen-client-urls=\"https://0.0.0.0:2379\" \\\n  --initial-advertise-peer-urls=\"https://${COREOS_EC2_HOSTNAME}:2380\" \\\n  --advertise-client-urls=\"https://${COREOS_EC2_HOSTNAME}:2379\" \\\n  --cert-file=\"/etc/ssl/certs/tls.crt\" \\\n  --key-file=\"/etc/ssl/certs/tls.key\" \\\n  --client-cert-auth=true \\\n  --trusted-ca-file=\"/etc/ssl/certs/ca.crt\" \\\n  --peer-cert-file=\"/etc/ssl/certs/tls.crt\" \\\n  --peer-key-file=\"/etc/ssl/certs/tls.key\" \\\n  --peer-client-cert-auth=true \\\n  --peer-trusted-ca-file=\"/etc/ssl/certs/ca.crt\"",
�[0;1;31m�[0m            "name": "20-clct-etcd-member.conf"
�[0;1;31m�[0m          }
�[0;1;31m�[0m        ],
�[0;1;31m�[0m        "enable": true,
�[0;1;31m�[0m        "name": "etcd-member.service"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "mask": true,
�[0;1;31m�[0m        "name": "update-engine.service"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "mask": true,
�[0;1;31m�[0m        "name": "locksmithd.service"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "enable": true,
�[0;1;31m�[0m        "name": "fstrim.timer"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "contents": "[Unit]\nDescription=Mount instance storage to /var/lib/etcd\n[Mount]\nWhat=/dev/nvme1n1\nWhere=/var/lib/etcd\nType=ext4\n",
�[0;1;31m�[0m        "name": "var-lib-etcd.mount"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "contents": "[Unit]\nDescription=Mount ephemeral to /var/lib/docker\n[Mount]\nWhat=/dev/xvdp\nWhere=/var/lib/docker\nType=ext4\n",
�[0;1;31m�[0m        "name": "var-lib-docker.mount"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "contents": "[Unit]\nDescription=Mount ephemeral to /var/lib/kubelet\n[Mount]\nWhat=/dev/xvdc\nWhere=/var/lib/kubelet\nType=ext4\n",
�[0;1;31m�[0m        "name": "var-lib-kubelet.mount"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "contents": "[Unit]\nDescription=Update kernel tuning params\n[Service]\nExecStartPre=/sbin/sysctl -w net.ipv4.neigh.default.gc_thresh3=15000 ;\\\n  /sbin/sysctl -w net.ipv4.neigh.default.gc_thresh2=12500 ;\\\n  /sbin/sysctl -w net.ipv4.neigh.default.gc_thresh1=10000 ;\\\n  /sbin/sysctl -w fs.inotify.max_user_watches=131072\nExecStart=/bin/echo \"Set kernel tuning params\"\nRemainAfterExit=true\nRestart=on-failure\nRestartSec=10\n[Install]\nWantedBy=multi-user.target\n",
�[0;1;31m�[0m        "enable": true,
�[0;1;31m�[0m        "name": "params.service"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "dropins": [
�[0;1;31m�[0m          {
�[0;1;31m�[0m            "contents": "[Unit]\nAfter=var-lib-docker.mount\nRequires=var-lib-docker.mount\n",
�[0;1;31m�[0m            "name": "20-await-var-lib-docker-mount.conf"
�[0;1;31m�[0m          }
�[0;1;31m�[0m        ],
�[0;1;31m�[0m        "name": "docker.service"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "contents": "[Unit]\nDescription=Generate etcd certs\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\nConditionFileNotEmpty=/opt/bin/generate_certs.sh\nConditionFileNotEmpty=!/etc/ssl/etcd/tls.crt\nConditionFileNotEmpty=!/etc/ssl/etcd/tls.key\nConditionFileNotEmpty=!/etc/ssl/etcd/ca.crt\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=ASSETS_DIR=/etc/ssl/etcd\nExecStartPre=/usr/bin/mkdir -p $ASSETS_DIR\nExecStartPre=/bin/sh -c \"\\\n  CERT_COMMON_NAME=etcd.aws-9405.platform-dev.r53.nordstrom.net \\\n  CERT_IP_SANS=${COREOS_EC2_IPV4_LOCAL},127.0.0.1 \\\n  CERT_ALT_NAMES=etcd.aws-9405.platform-dev.r53.nordstrom.net,${COREOS_EC2_HOSTNAME},localhost \\\n  ROOT_CA_KEY=\"/root-ca/etcd-ca.key\" \\\n  ROOT_CA_CERT=\"/root-ca/etcd-ca.crt\" \\\n  /opt/bin/generate_certs.sh\"\nExecStartPre=/bin/rm /root-ca/etcd-ca.key /root-ca/etcd-ca.crt\nExecStartPre=/usr/bin/curl -v https://discovery.etcd.io\nExecStartPre=/usr/bin/sh -c \"echo | \\\n  ope[2019-09-12T23:21:07.839669]nssl s_client -connect discovery.etcd.io:443 -prexit -showcerts 2\u003e /dev/null \\\n  | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' \\\n  \u003e /etc/ssl/etcd/ca-certificates.crt\"\nExecStart=/bin/echo \"Etcd certs generated\"\nRemainAfterExit=true\nRestart=on-failure\nRestartSec=10\n[Install]\nWantedBy=multi-user.target\n",
�[0;1;31m�[0m        "enable": true,
�[0;1;31m�[0m        "name": "generate-etcd-certs.service"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "dropins": [
�[0;1;31m�[0m          {
�[0;1;31m�[0m            "contents": "[Unit]\nRequires=var-lib-etcd.mount\nAfter=var-lib-etcd.mount\nRequires=generate-etcd-certs.service\nAfter=generate-etcd-certs.service\n[Service]\nEnvironment=\"ETCD_IMAGE_URL=quay.io/coreos/etcd\"\nEnvironment=\"RKT_GLOBAL_ARGS=\"\nEnvironment=\"ETCD_STRICT_RECONFIG_CHECK=true\"\nEnvironment=\"ETCD_LISTEN_METRICS_URLS=http://0.0.0.0:9379\"\nEnvironment=\"ETCD_DISCOVERY=https://discovery.etcd.io/9cc5ec5fe416532f25a4cde484d304f8\"\nEnvironment=\"ETCD_SSL_DIR=/etc/ssl/etcd\"\nOOMScoreAdjust=-999\nExecStartPre=/usr/bin/chown -R etcd:etcd /var/lib/etcd\nExecStartPre=/usr/bin/chown -R etcd:etcd /etc/ssl/etcd\n",
�[0;1;31m�[0m            "name": "01-conf.conf"
�[0;1;31m�[0m          }
�[0;1;31m�[0m        ],
�[0;1;31m�[0m        "enable": true,
�[0;1;31m�[0m        "name": "etcd-member.service"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "contents": "[Unit]\nDescription=Generate kubelet certs\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\nConditionFileNotEmpty=/opt/bin/generate_certs.sh\nConditionFileNotEmpty=!/etc/kubernetes/tls.crt\nConditionFileNotEmpty=!/etc/kubernetes/tls.key\nConditionFileNotEmpty=!/etc/kubernetes/ca.crt\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=ASSETS_DIR=/etc/kubernetes\nExecStartPre=/usr/bin/mkdir -p $ASSETS_DIR\nExecStartPre=/bin/sh -c \"\\\n  CERT_COMMON_NAME=system:node:${COREOS_EC2_HOSTNAME} \\\n  CERT_ORGANIZATION=system:nodes \\\n  CERT_ALT_NAMES=${COREOS_EC2_HOSTNAME} \\\n  ROOT_CA_KEY=/root-ca/k8s-ca.key \\\n  ROOT_CA_CERT=/root-ca/k8s-ca.crt \\\n  /opt/bin/generate_certs.sh\"\nExecStartPre=/bin/rm /root-ca/k8s-ca.key /root-ca/k8s-ca.crt\nExecStart=/bin/echo \"Kubelet certs generated\"\nRemainAfterExit=true\nRestart=on-failure\nRestartSec=10\n[Install]\nWantedBy=multi-user.target\n",
�[0;1;31m�[0m        "enable": true,
�[0;1;31m�[0m        "name": "generate-kubelet-certs.service"
�[0;1;31m�[0m      },
�[0;1;31m�[0m      {
�[0;1;31m�[0m        "contents": "[Unit]\nDescription=Kubernetes Kubelet\nDocumentation=https://github.com/kubernetes/kubernetes\nRequires=generate-kubelet-certs.service\nAfter=generate-kubelet-certs.service\nRequires=var-lib-kubelet.mount\nAfter=var-lib-kubelet.mount\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=KUBELET_IMAGE=docker://gcr.io/google_containers/hyperkube:v1.15.3\n# additive to kubelet-wrapper\nEnvironment=\"RKT_GLOBAL_ARGS=--insecure-options=image\"\nEnvironment=\"RKT_RUN_ARGS= \\\n  --volume dns,kind=host,source=/etc/resolv.conf \\\n  --volume var-lib-cni,kind=host,source=/var/lib/cni \\\n  --volume var-log,kind=host,source=/var/log \\\n  --volume cni-bin,kind=host,source=/run/torcx/unpack/crio/usr/share/cni/bin/ \\\n  --mount volume=cni-bin,target=/opt/cni/bin \\\n  --mount volume=dns,target=/etc/resolv.conf \\\n  --mount volume=var-lib-cni,target=/var/lib/cni \\\n  --mount volume=var-log,target=/var/log\"\nExecStartPre=/bin/mkdir -p /etc/kubernet[2019-09-12T23:21:07.839995]es/manifests\nExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d\nExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets\nExecStartPre=/bin/mkdir -p /srv/kubernetes/manifests\nExecStartPre=/bin/mkdir -p /var/lib/cni\nExecStartPre=/usr/bin/mkdir -p /var/log/containers\nExecStartPre=/bin/cp /root/.docker/config.json /var/lib/kubelet/config.json\nExecStart=/usr/lib/coreos/kubelet-wrapper \\\n  --anonymous-auth=false \\\n  --read-only-port=0 \\\n  --cgroup-driver=cgroupfs \\\n  --cloud-provider=aws \\\n  --client-ca-file=/etc/kubernetes/ca.crt \\\n  --tls-cert-file=/etc/kubernetes/tls.crt \\\n  --tls-private-key-file=/etc/kubernetes/tls.key \\\n  --cluster_dns=${COREOS_EC2_IPV4_LOCAL},25.0.0.10 \\\n  --cluster_domain=cluster.local \\\n  --exit-on-lock-contention \\\n  --feature-gates=LocalStorageCapacityIsolation=true \\\n  --lock-file=/var/run/lock/kubelet.lock \\\n  --kubeconfig=/etc/kubernetes/kubelet/kubeconfig \\\n  --network-plugin=cni \\\n  --cni-conf-dir=/etc/kubernet[2019-09-12T23:21:07.840019]es/cni/net.d \\\n  --pod-manifest-path=/etc/kubernetes/manifests \\\n  --node-labels=master=true,\"beta.nordstrom.net/node-pool\"=\"etcd-pool\",\"node-role.kubernetes.io/master\"=true \\\n  --register-with-taints=dedicate=\"etcd:NoSchedule\" \\\n  --system-reserved=\"cpu=350m,memory=3Gi\" \\\n  --eviction-hard=\"memory.available\u003c10%,nodefs.available\u003c10%,nodefs.inodesFree\u003c10%,imagefs.available\u003c10%,imagefs.inodesFree\u003c10%\" \\\n  --eviction-soft=\"memory.available\u003c15%,nodefs.available\u003c20%,nodefs.inodesFree\u003c20%,imagefs.available\u003c20%,imagefs.inodesFree\u003c20%\" \\\n  --eviction-soft-grace-period=\"memory.available=2m,nodefs.available=30s,nodefs.inodesFree=30s,imagefs.available=30s,imagefs.inodesFree=30s\" \\\n  --eviction-minimum-reclaim=\"memory.available=0Mi,nodefs.available=5Gi,imagefs.available=5Gi\" \\\n  --eviction-max-pod-grace-period=\"300\" \\\n  --authentication-token-webhook \\\n  --cpu-manager-policy=static\nRestart=always\nRes[2019-09-12T23:21:07.840040]tartSec=5\n[Install]\nWantedBy=multi-user.target\n",
�[0;1;31m�[0m        "enable": true,
�[0;1;31m�[0m        "name": "kubelet.service"
�[0;1;31m�[0m      }
�[0;1;31m�[0m    ]
�[0;1;31m�[0m  }
�[0;1;31m�[0m```
@dharmab
Copy link

dharmab commented Oct 30, 2019

Try reparitioning the device in Ignition in addition to formatting the filesystem:

  disks:
    # Repartition NVME Instance Storage
    - device: /dev/nvme1n1
      wipe_table: True
      partitions:
        - number: 0
          size: 0
          start: 0
  filesystems:
    # Format NVME Instance Storage
    - mount:  # Not actually mounting, just formatting
        device: /dev/nvme1n1p1
        format: ext4

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dhawal55 @dharmab