lagom.experiment¶
Config¶
-
class
lagom.experiment.
Config
(items, num_sample=1, keep_dict_order=False)[source]¶ Defines a set of configurations for the experiment.
The configuration includes the following possible items:
- Hyperparameters: learning rate, batch size etc.
- Experiment settings: training iterations, logging directory, environment name etc.
All items are stored in a dictionary. It is a good practice to semantically name each item e.g. network.lr indicates the learning rate of the neural network.
For hyperparameter search, we support both grid search (
Grid
) and random search (Sample
).Call
make_configs()
to generate a list of all configurations, each is assigned with a unique ID.note:
For random search over small positive float e.g. learning rate, it is recommended to use log-uniform distribution, i.e. .. math:: \text{logU}(a, b) \sim \exp(U(\log(a), \log(b))) An example: `np.exp(np.random.uniform(low=np.log(low), high=np.log(high)))` Because direct uniform sampling is very `numerically unstable`_.
Warning
The random seeds should not be set here. Instead, it should be handled by
BaseExperimentMaster
andBaseExperimentWorker
.Example:
>>> config = Config({'log.dir': 'some path', 'network.lr': Grid([1e-3, 5e-3]), 'env.id': Grid(['CartPole-v1', 'Ant-v2'])}, num_sample=1, keep_dict_order=False) >>> import pandas as pd >>> print(pd.DataFrame(config.make_configs())) ID env.id log.dir network.lr 0 0 CartPole-v1 some path 0.001 1 1 Ant-v2 some path 0.001 2 2 CartPole-v1 some path 0.005 3 3 Ant-v2 some path 0.005
Parameters: - items (dict) – a dictionary of all configuration items.
- num_sample (int) – number of samples for random configuration items.
If grid search is also provided, then the grid will be repeated
num_sample
of times. - keep_dict_order (bool) – if
True
, then each generated configuration has the same key ordering withitems
.
Run experiment¶
-
lagom.experiment.
run_experiment
(run, config, seeds, log_dir, max_workers, chunksize=1, use_gpu=False, gpu_ids=None)[source]¶ A convenient function to parallelize the experiment (master-worker pipeline).
It is implemented by using concurrent.futures.ProcessPoolExecutor
It automatically creates all subfolders for each pair of configuration and random seed to store the loggings of the experiment. The root folder is given by the user. Then all subfolders for each configuration are created with the name of their job IDs. Under each configuration subfolder, a set subfolders are created for each random seed (the random seed as folder name). Intuitively, an experiment could have following directory structure:
- logs - 0 # ID number - 123 # random seed - 345 - 567 - 1 - 123 - 345 - 567 - 2 - 123 - 345 - 567 - 3 - 123 - 345 - 567 - 4 - 123 - 345 - 567
Parameters: - run (function) – a function that defines an algorithm, it must take the arguments (config, seed, device, logdir)
- config (Config) – a
Config
object defining all configuration settings - seeds (list) – a list of random seeds
- log_dir (str) – a string to indicate the path to store loggings.
- max_workers (int) – argument for ProcessPoolExecutor. if None, then all experiments run serially.
- chunksize (int) – argument for Executor.map()
- use_gpu (bool) – if True, then use CUDA. Otherwise, use CPU.
- gpu_ids (list) – if None, then use all available GPUs. Otherwise, only use the GPU device defined in the list.