Developer Guide: How To Add An Algorithm Or Method

This tutorial describes how a developer may add a new preference learning algorithm, feature selection method, or evaluation method to the Preference Learning Toolbox API and/or GUI. The tutorial is divided into 4 steps:

  1. Create the class
  2. Update the corresponding enum class
  3. Create a corresponding GUI menu (OPTIONAL)
  4. Use the new algorithm/method

For more detailed information on any of the methods and parameters mentioned throughout the tutorial, one may check out the corresponding section of the API Reference.

Step 1: Create the class

In order to implement a new algorithm or method in PLT, we must declare it by extending the appropriate base class. Depending on what type of method one intends to add, the following base classes are available:

  • pyplt.plalgorithms.base.PLAlgorithm -- for preference learning algorithms.
  • pyplt.fsmethods.base.FeatureSelectionMethod -- for feature selection methods. If the feature selection method is a wrapper method, one can instead extend the class pyplt.fsmethods.wrappers.WrapperFSMethod.
  • pyplt.evaluation.base.Evaluator -- for evaluation methods.

These base classes (PLAlgorithm, FeatureSelectionMethod and Evaluator) are structured very similarly. They each have a constructor through which the name and description of the algorithm/method are specified/passed together with the values of any additional parameters. The classes each contain a number of basic getter functions for these attributes. They also contain a varying number of abstract methods (functions) that all subclasses must implement. Additionally, there are a number of base methods (functions) which implement default behaviours which may be overwritten by subclasses (e.g., for faster computation). Below is a breakdown of the most important of these methods (functions) for each class.

Extending PLAlgorithm

Abstract methods which must be implemented by all subclasses:

  • init_train() -- implement any initializations prior to the training stage (e.g., weight initialization in Backpropagation). N.B. This method is called automatically by the run() method of the Experiment class prior to each call to train().
  • train() -- implement the training stage.
  • predict() -- implement how the inferred model is used to predict the output for an unseen data instance.
  • save_model() -- implement how the inferred model may be stored or saved to file.
  • transform_data() -- implement how a given sample from the dataset (i.e., a row from a pandas.DataFrame) is to be transformed for use by this method.

Important base methods which may optionally be overwritten by subclasses:

  • calc_train_accuracy() -- calculates the training accuracy of the inferred model.
  • test() -- calculates the test accuracy of the inferred model on a given test set (unseen data instances).
  • clean_up() -- overwrite this method to implement any final commands required by the algorithm implementation (e.g., closing the tensorflow session in Backpropagation). The base method simply returns None. This method is called automatically by the run() method of the Experiment class after each call to train() (or test(), if applicable).

Extending FeatureSelectionMethod

Abstract methods which must be implemented by all subclasses:

  • select() -- implement the feature selection procedure.
  • _evaluate() -- implement how the effectiveness of a given feature subset is measuered. In the WrapperFSMethod class, this method is implemented to suit wrapper-type feature selection methods.

Extending Evaluator

There are no abstract methods for this class. However, if the evaluation method is intended to split the dataset into subsets or folds, then a split() method similar to those in pyplt.evaluation.holdout.HoldOut and pyplt.evaluation.cross_validation.KFoldCrossValidation should be implemented. As with Holdout and K-Fold Cross Validation, the way this split() method (function) is used on the dataset should then be specified in the run() method of the Experiment class.

Step 2: Update the corresponding enum class

Once the new algorithm/method has been implemented, the corresponding enum class must be updated. That is:

  • FSMethod -- for feature selection methods.
  • PLAlgo -- for preference learning algorithms.
  • EvaluatorType -- for evaluation methods.

These enum classes are all located in the pyplt.util.enums module. In order to update the corresponding enum class with the new algorithm/method, a new enumerated constant must be created for it by assigning an unused integer to a variable named after the algorithm/method.

For example, say we have implemented the Bootstrap validation method. Updating the EvaluatorType class would be updated to look similarly to the following:

In [ ]:
class EvaluatorType(Enum):
    """Class specifying enumerated constants for evaluators.

    Extends `enum.Enum`.
    NONE = 0
    HOLDOUT = 1
    KFCV = 2

Step 3: Create a corresponding GUI menu (OPTIONAL)

N.B. This step requires some knowledge of the tkinter Python package and its tkinter.ttk module.

One may wish to create a GUI menu corresponding to the new algorithm/method where users may specify parameters to the algorithm/method. In order to create such a menu, a tkinter widget (class) extending the tkinter.Frame class should be created.

The widget/class may be structured as desired however it must implement a get_params() method which returns a dict specifying the values of the parameters chosen through the menu.

The classes declaring the GUI menus for the current in-built algorithms/methods are located in the pyplt.gui.experiment.preflearning subpackage and may be referred to for guidance.

These widgets are used by the pyplt.gui.experiment.featureselection.featselectiontab.FeatureSelectionTab class and the pyplt.gui.experiment.preflearning.pltab.PLFrame class. However these classes need not be modified in order for the new GUI menu to be added to the Feature Selection and Preference Learning tabs of the PLT GUI (see Step 4 below).

Step 4: Use the new algorithm/method

The new algorithm or method may now be used via the API in the same way that the in-built algorithms/methods are used via the API (as explained in Steps 4-6 of the How To Set Up And Run An Experiment Developer Tutorial).

However, an additional step is required for the new algorithm/method, together with its corresponding GUI menu (if applicable), to be added to the Preference Learning Toolbox GUI.

Update supported algorithms/methods dictionaries (for GUI)

That is, it must be specified as a supported algorithm/method by appropriately modifying the dictionaries declared in the pyplt.gui.util.supported_methods module:

  • supported_fs_methods -- for feature selection methods in Advanced Mode.
  • supported_algorithms -- for preference learning algorithms in Advanced Mode (available both in the Feature Selection tab and the Preference Learning tab of the GUI).
  • supported_algorithms_beginner -- for preference learning algorithms in Beginner Mode.
  • supported_evaluation_methods -- for evaluation methods in Advanced Mode (available both in the Feature Selection tab and the Preference Learning tab of the GUI)

The dictionaries used for the Advanced Mode contain the algorithm/method enum types as keys and tuples of size 2 as values. Each tuple contains a reference to the given algorithm/method class, followed by a reference to the corresponding GUI menu (if applicable). For example, say we have implemented a neuroevolutionary algorithm defined in a class called 'NeuroEvo', together with a corresponding GUI menu in a class called 'NeuroevoGUIMenu'. The supported_algorithms dict should be updated to look similarly to the following:

In [ ]:
supported_algorithms = {
    PLAlgo.RANKSVM: (ranksvm.RankSVM, ranksvm_menu.RankSVMMenu),  # RankSVM
    PLAlgo.BACKPROPAGATION: (backprop_tf.BackpropagationTF, backprop_menu.BackpropMenu),  # Backpropagation (tensorflow)
    PLAlgo.NEUROEVOLUTION: (Neuroevo, NeuroevoGUIMenu)

On the other hand, the dictionaries used for the Beginner Mode contain the algorithm/method enum types as keys and references to the corresponding algorithm/method classes as values. GUI menus are ignored since parameter value setup is excluded from the design of the Beginner Mode. For example, to add the 'NeuroEvo' algorithm to the Beginner Mode, the supported_algorithms_beginner dict should be updated to look similarly to the following:

In [ ]:
supported_algorithms_beginner = {
    PLAlgo.RANKSVM: ranksvm.RankSVM,  # RankSVM
    PLAlgo.BACKPROPAGATION: backprop_tf.BackpropagationTF,  # Backpropagation (tensorflow)

Additional notes (for GUI)

If the GUI menu of the new algorithm/method requires additional parameters (as with the Backpropagation Menu), make sure to also update accordingly the _update_algo_menu() methods (functions) (in the case of an algorithm) or the _update_eval_menu() methods (functions) (in the case of an evaluation method) in the classes pyplt.gui.experiment.featureselection.featselectiontab.FeatureSelectionFrame and pyplt.gui.experiment.preflearning.pltab.PLFrame.

Furthermore, to specify values other than the class defaults for the parameters of the new algorithm/method to be used in the Beginner Mode, make sure to update the preference learning section of the _run_exp() method in the pyplt.gui.beginnermenu.BeginnerMenu class accordingly.