Need help: 2 Short questions

[daviddelriod]

Hi everybody,

I am currently developing and FL experiment trying to execute a binnary classification problem by detecting Pneumonia cases using the well-known Chest x-ray dataset. I have already created the FederatedDataset and FederatedNet objects but now I find myself with 2 questions as I do not understand 100% the official documentation. Taking into account that I want to run the experiment in a single notebook and i want to make a director-based workflow:

1. Which are the main settings that I would have to define in the fl workspace (I mean, a director, an envoy, a plan.yaml...) and how should I do it
2. Can I track the model's performance by any MLOps framework?

Thanks in advance to everybody!

[psfoley]

Thanks for the great questions! The FederatedModel and FederatedDataset were originally developed for the native Python API built on top of the Aggregator-based Workflow. The starting point for experiments that make use of the Director-based workflow is in the interactive API
tutorials folder. Given that you want to run everything in a single notebook, I would recommend to start by looking at the Linear Regression with Numpy and OpenFL Example, and launching it in Google colab. This notebook will actually launch the director and envoys through the notebook instead of the command line, and after the Federated experiment has been launched, you can see model accuracy interactively through an embedded Tensorboard window.

[psfoley]

Regarding your questions:

Which are the main settings that I would have to define in the fl workspace (I mean, a director, an envoy, a plan.yaml...) and how should I do it

With the interactive API (the API tied to the Director-based workflow), modifying the plan is strictly optional, and only necessary if you want to make changes to compression or another low level detail. The director has a config file, each envoy has their own config file, and then the person defining the experiment in a jupyter notebook just needs to connect to the director, define their model / how the dataset will be preprocessed, then submit the experiment to the director.

Can I track the model's performance by any MLOps framework?

The model's performance can be tracked through Tensorboard today (see the previous numpy example I posted). We don't have support for other MLOps experiment tracking platforms like wandb or Neptune.AI yet. These are things we've talked about adding support for, but haven't been able to get around to. If you're looking for a particular MLOps tool integration, please submit a feature request to OpenFL and our development team will look into it! 👍

[daviddelriod]

Thanks @psfoley I will try to implement my solution with the interactive API. Appreciate the help!!!

[CasellaJr]

Look at my repos, there are short READMEs that can explain you how to run experiments:

• https://github.com/CasellaJr/Multi-Input-Neural-Networks-in-Federated-Learning
• https://github.com/CasellaJr/Benchmarking-FedAvg-and-FedCurv-for-Image-Classification-Tasks
• https://github.com/CasellaJr/streamflow-fl

Moreover, I have submitted an article to IEEE Transactions on Medical Imaging where I do the same task: binary classification on pneumonia datasets: https://arxiv.org/pdf/2206.10048.pdf

[daviddelriod]

Thank you so much @CasellaJr, appreciate!!