flip.nvflare.controllers.fed_evaluation
=======================================

.. py:module:: flip.nvflare.controllers.fed_evaluation


Classes
-------

.. autoapisummary::

   flip.nvflare.controllers.fed_evaluation.ModelEval


Module Contents
---------------

.. py:class:: ModelEval(task_check_period=0.5, submit_model_timeout=600, validation_timeout: int = 6000, model_locator_id='', formatter_id='', submit_model_task_name=AppConstants.TASK_SUBMIT_MODEL, evaluation_task_name=PTConstants.EvalTaskName, cleanup_models=False, participating_clients=None, wait_for_clients_timeout=300)

   Bases: :py:obj:`nvflare.app_common.workflows.cross_site_model_eval.CrossSiteModelEval`


   FLIP model-collection evaluation: evaluate a set of server-provided models on every client.

   Subclasses NVFLARE's :class:`CrossSiteModelEval` to inherit its constructor validation,
   unknown-task routing, and stop logic. Unlike cross-site validation, where each client's
   submitted model is validated by every
   other client, evaluation loads a *collection* of models server-side and bundles them into a
   single evaluation task broadcast to all clients.

   Sharing the base also defines the ``_validation_task_name`` / ``_cross_val_dir`` attributes the
   previous standalone fork referenced but never assigned.


   .. py:attribute:: _evaluation_task_name
      :value: 'evaluation'



   .. py:attribute:: _eval_results_dir
      :value: 'evaluation_results'



   .. py:attribute:: _eval_results
      :type:  dict


   .. py:attribute:: _all_models_dxo
      :value: None



   .. py:method:: start_controller(fl_ctx: nvflare.apis.fl_context.FLContext)


   .. py:method:: control_flow(abort_signal: nvflare.apis.signal.Signal, fl_ctx: nvflare.apis.fl_context.FLContext)


   .. py:method:: _locate_server_models(fl_ctx: nvflare.apis.fl_context.FLContext) -> bool

      Load the whole collection of models to evaluate from the model locator.

      Unlike stock (one DXO located per model name), evaluation loads every model in a single
      ``locate_model`` call and keeps the collection DXO to broadcast to clients.



   .. py:method:: _before_send_validate_task_cb(client_task: nvflare.apis.controller_spec.ClientTask, fl_ctx: nvflare.apis.fl_context.FLContext)

      Bundle the whole model collection into the task data sent to each client.



   .. py:method:: _accept_val_result(client_name: str, result: nvflare.apis.shareable.Shareable, fl_ctx: nvflare.apis.fl_context.FLContext)

      Record each client's evaluation result, keyed flatly by client.



   .. py:method:: handle_event(event_type: str, fl_ctx: nvflare.apis.fl_context.FLContext)