# RPC PS Benchmark ## How to add your experiment 1. Data - Create a data class and add it to the data directory - Update benchmark_class_helper.py to include your data class in the data_map - Add configurations to data_configurations.json in the configurations directory 2. Model - Create a model class and add it to the model directory - Update benchmark_class_helper.py to include your model class in the model_map - Add configurations to model_configurations.json in the configurations directory 3. Trainer - Create a trainer class and add it to the trainer directory - Update benchmark_class_helper.py to include your trainer class in the trainer_map - Add configurations to trainer_configurations.json in the configurations directory 4. Parameter Server - Create a parameter server class and add it to the parameter_servers directory - Update benchmark_class_helper.py to include your parameter_server class in the ps_map - Add configurations to parameter_server_configurations.json in the configurations directory 5. Script - Create a bash script for your experiment and add it to the experiment_scripts directory 6. Testing - Add a test method for your script to test_scripts.py ## Trainer class The trainer directory contains base classes to provide a starting point for implementing a trainer. Inherit from a base class and implement your trainer. The benchmark has two requirements for trainers. 1. It must implement a __init__ method that takes rank, trainer_count, and ps_rref as arguments ```python def __init__(self, rank, trainer_count, ps_rref, backend, use_cuda_rpc): ``` 2. It must implement a train method that takes model and data as arguments. ```python def train(self, model, data): ``` ## Parameter Server class The parameter_server directory contains base classes to provide a starting point for implementing a parameter server. Inherit from a base class and implement your parameter server. The benchmark has two requirements for parameter servers. 1. It must implement a __init__ method that takes rank and ps_trainer_count as arguments ```python def __init__(self, rank, ps_trainer_count, backend, use_cuda_rpc): ``` 2. It must implement a reset_state method ```python def reset_state(ps_rref): ``` ## Testing Use `pytest` to run the test methods added to test_scripts.py. To test all the scripts added use `pytest test_scripts.py`.