Secure Content from server pod of nvflare==2.1.2

[Writam18]

I am trying to stimulate an experiment with the NVIDIA FLARE framework on Kubernetes(K8s). I am trying this on a private cluster. I have installed nvflare==2.1.2 on the container used by K8s to create pods.
After creating the overseer, server, and client using provision API the job submission is failing. And I am getting the following error from the server pod.
The following files are not secure content
fed_server.json

This is the following part is added as spec in the server pod.

spec:
containers:

• args:
  • -u
  • • m
  • nvflare.private.fed.app.server.server_train
  • -m
  • {project_path}/prod_00/server
  • -s
  • fed_server.json
  • ---set
  • secure_train=True
  • config_folder=config
    command:
  • /usr/local/bin/python3

[YuanTingHsieh]

@Writam18 that warning show up when users modify the content of fed_server.json AFTER the provision.

It is meant to protect users from bad people modifying the content, kind of like a checksum stuff.

Right now users need to follow the following steps for a version upgrade:

1. stop your current system with old versions
2. remove/move your old provision startup folders into other places
3. re-run the provision command with the nvflare new version
4. starts the overseer, server, and client

We will enhance it to be backward compatible in the future releases.

[Writam18]

@YuanTingHsieh Is there a way to resolve this issue without upgrading the nvflare version? This nvflare==2.1.2 was running fine for me for over a year and suddenly this warning started coming up. And upgrading it to nvflare==2.4 will force me to change my scripts everywhere and that will be an overhead. So, is there a way to work with nvflare==2.1.2 and resolve this issue?

[YuanTingHsieh]

@Writam18 thanks for raising the issue.

@IsaacYangSLA @yhwen Do you know is there any way the user can resolve this issue when upgrading from 2.1.2 to 2.4?

[YuanTingHsieh]

@Writam18 ok, so you want to keep using 2.1.2 for all your servers and clients, I don't see any problems with that.

That warning show up when users modify the content of fed_server.json AFTER the provision.
It is meant to protect users from bad people modifying the content, kind of like a checksum stuff.

You just need to make sure that, your server, clients, overseer and admin client that you are using they are generated using the same version of NVFlare and same project.yml

[IsaacYangSLA]

The certificates generated by provisioning tools expire after 360 days. @Writam18 , I guess the root CA certificate expired and when server loaded the secure contents and built the certificate chains, the entire chain of trust could not be established. One way to resolve this issue is to re-generate the startup kits of all participants (server, overseer, client and admin).