hpc05 package

class hpc05.Client(profile='pbs', hostname='hpc05', username=None, password=None, culler=True, culler_args=None, env_path=None, local=False, *args, **kwargs)[source]

Bases: ipyparallel.client.client.Client

Return an ipyparallel.Client and connect to a remote ipcluster over ssh if local=False and start the engine culler.

Parameters
  • profile (str) – profile name, default is ‘pbs’ which results in the folder ipython/profile_pbs.

  • hostname (str) – Hostname of machine where the ipcluster runs. If connecting via the headnode use: socket.gethostname() or set local=True.

  • username (str) – Username to log into hostname. If not provided, it tries to look it up in your ssh/config.

  • password (str) – password for ssh username@hostname.

  • culler (bool) – Controls starting of the culler. Default: True.

  • culler_args (str) – Add arguments to the culler. e.g. ‘–timeout=200’

  • env_path (str, default: None) – Path of the Python environment, ‘/path/to/ENV/’ if Python is in /path/to/ENV/bin/python. Examples ‘~/miniconda3/envs/dev/’, ‘miniconda3/envs/dev’, ‘~/miniconda3’. Defaults to the environment that is sourced in bashrc or bash_profile.

  • local (bool, default: False) – Connect to the client locally or over ssh. Set it False if a connection over ssh is needed.

json_filename

file name of tmp local json file with connection details.

Type

str

tunnel

ssh tunnel for making connection to the hpc05.

Type

pexpect.spawn object

Notes

You need a profile with PBS (or SLURM) settings in your ipython folder on the cluster. You can generate this by running:

hpc05.create_remote_pbs_profile(username, hostname)

Then setup a ipcluster on the hpc05 by starting a screen and running

ipcluster start –n=10 –profile=pbs.

hpc05.create_local_pbs_profile(profile='pbs', local_controller=False, custom_template=None)[source]

Creata a PBS profile for ipyparallel.

Parameters
  • profile (str) – Profile name.

  • local_controller (bool) – Create a ipcontroller on a seperate node if True and locally if False.

  • custom_template (str) – A custom job script template, see the example below.

Examples

By default no memory is specified, using the following custom_template allows you to request a certain amount of memory.

import hpc05
import sys
custom_template = f'''\
    #!/bin/sh
    #PBS -t 1-{{n}}
    #PBS -V
    #PBS -N ipengine
    #PBS -l mem=15GB
    {sys.executable} -m ipyparallel.engine --profile-dir="{{profile_dir}}" --cluster-id=""
'''
hpc05.create_local_pbs_profile('pbs_15GB',
                               local_controller=False,
                               custom_template=custom_template)
hpc05.create_remote_pbs_profile(hostname='hpc05', username=None, password=None, profile='pbs', local_controller=False, custom_template=None)[source]
hpc05.create_local_slurm_profile(profile='slurm', local_controller=False, custom_template=None)[source]

Creata a SLURM profile for ipyparallel.

Parameters
  • profile (str) – Profile name.

  • local_controller (bool) – Create a ipcontroller on a seperate node if True and locally if False.

  • custom_template (str) – A custom job script template, see the example below.

Examples

By default no memory is specified, using the following custom_template allows you to request a certain amount of memory.

import hpc05
custom_template = '''\
    #!/bin/sh
    #SBATCH --ntasks={n}
    #SBATCH --mem-per-cpu=4G
    #SBATCH --job-name=ipy-engine-
    srun ipengine --profile-dir='{profile_dir}' --cluster-id=''
'''
hpc05.create_local_pbs_profile('pbs', False, custom_template)
hpc05.create_remote_slurm_profile(hostname='hpc05', username=None, password=None, profile='slurm', local_controller=False, custom_template=None)[source]
hpc05.check_difference_in_envs(local_env_name=None, remote_env_name=None)[source]

Only works when setting the Python env in your ~/.bash_rc on the remote machine.

hpc05.start_ipcluster(n, profile, env_path=None, timeout=300)[source]

Start an ipcluster locally.

Parameters
  • n (int) – Number of engines to be started.

  • profile (str, default 'pbs') – Profile name of IPython profile.

  • env_path (str, default=None) – Path of the Python environment, ‘/path/to/ENV/’ if Python is in /path/to/ENV/bin/python. Examples ‘~/miniconda3/envs/dev/’, ‘miniconda3/envs/dev’, ‘~/miniconda3’. Defaults to the environment that is sourced in bashrc or bash_profile.

  • timeout (int) – Time limit after which the connection attempt is cancelled.

Returns

Return type

None

hpc05.start_remote_ipcluster(n, profile='pbs', hostname='hpc05', username=None, password=None, env_path=None, timeout=300)[source]

Starts an ipcluster over ssh on hostname and wait untill it’s successfully started.

Parameters
  • n (int) – Number of engines to be started.

  • profile (str, default 'pbs') – Profile name of IPython profile.

  • hostname (str) – Hostname of machine where the ipcluster runs.

  • username (str) – Username to log into hostname. If not provided, it tries to look it up in your ssh/config.

  • password (str) – Password for ssh username@hostname.

  • env_path (str, default=None) – Path of the Python environment, ‘/path/to/ENV/’ if Python is in /path/to/ENV/bin/python. Examples ‘~/miniconda3/envs/dev/’, ‘miniconda3/envs/dev’, ‘~/miniconda3’. Defaults to the environment that is sourced in bashrc or bash_profile.

  • timeout (int) – Time for which we try to connect to get all the engines.

Returns

Return type

None

hpc05.connect_ipcluster(n, profile='pbs', hostname='hpc05', username=None, password=None, culler=True, culler_args=None, env_path=None, local=True, timeout=300, folder=None, client_kwargs=None)[source]

Connect to an ipcluster on the cluster headnode.

Parameters
  • n (int) – Number of engines to be started.

  • profile (str, default 'pbs') – Profile name of IPython profile.

  • hostname (str) – Hostname of machine where the ipcluster runs. If connecting via the headnode use: socket.gethostname() or set local=True.

  • username (str) – Username to log into hostname. If not provided, it tries to look it up in your ssh/config.

  • password (str) – Password for ssh username@hostname.

  • culler (bool) – Controls starting of the culler. Default: True.

  • culler_args (str) – Add arguments to the culler. e.g. ‘–timeout=200’

  • env_path (str, default: None) – Path of the Python environment, ‘/path/to/ENV/’ if Python is in /path/to/ENV/bin/python. Examples ‘~/miniconda3/envs/dev/’, ‘miniconda3/envs/dev’, ‘~/miniconda3’. Defaults to the environment that is sourced in bashrc or bash_profile.

  • local (bool, default: True) – Connect to the client locally or over ssh. Set it False if a connection over ssh is needed.

  • timeout (int) – Time for which we try to connect to get all the engines.

  • folder (str, optional) – Folder that is added to the path of the engines, e.g. “~/Work/my_current_project”.

  • client_kwargs (dict) – Keyword arguments that are passed to hpc05.Client().

Returns

  • client (ipython.Client object) – An IPyparallel client.

  • dview (ipyparallel.client.view.DirectView object) – Direct view, equivalent to client[:].

  • lview (ipyparallel.client.view.LoadBalancedView) – LoadedBalancedView, equivalent to client.load_balanced_view().

hpc05.start_and_connect(n, profile='pbs', hostname='hpc05', culler=True, culler_args=None, env_path=None, local=True, timeout=300, folder=None, client_kwargs=None, kill_old_ipcluster=True)[source]

Start an ipcluster locally and connect to it.

Parameters
  • n (int) – Number of engines to be started.

  • profile (str, default 'pbs') – Profile name of IPython profile.

  • hostname (str) – Hostname of machine where the ipcluster runs. If connecting via the headnode use: socket.gethostname() or set local=True.

  • culler (bool) – Controls starting of the culler. Default: True.

  • culler_args (str) – Add arguments to the culler. e.g. ‘–timeout=200’

  • env_path (str, default: None) – Path of the Python environment, ‘/path/to/ENV/’ if Python is in /path/to/ENV/bin/python. Examples ‘~/miniconda3/envs/dev/’, ‘miniconda3/envs/dev’, ‘~/miniconda3’. Defaults to the environment that is sourced in bashrc or bash_profile.

  • local (bool, default: True) – Connect to the client locally or over ssh. Set it False if a connection over ssh is needed.

  • timeout (int) – Time for which we try to connect to get all the engines.

  • folder (str, optional) – Folder that is added to the path of the engines, e.g. “~/Work/my_current_project”.

  • client_kwargs (dict) – Keyword arguments that are passed to hpc05.Client().

  • kill_old_ipcluster (bool) – If True, it cleansup any old instances of ipcluster and kills your jobs in qstat or squeue.

Returns

  • client (ipython.Client object) – An IPyparallel client.

  • dview (ipyparallel.client.view.DirectView object) – Direct view, equivalent to client[:].

  • lview (ipyparallel.client.view.LoadBalancedView) – LoadedBalancedView, equivalent to client.load_balanced_view().

hpc05.start_remote_and_connect(n, profile='pbs', hostname='hpc05', username=None, password=None, culler=True, culler_args=None, env_path=None, timeout=300, folder=None, client_kwargs=None, kill_old_ipcluster=True)[source]

Start a remote ipcluster on hostname and connect to it.

Parameters
  • n (int) – Number of engines to be started.

  • profile (str, default 'pbs') – Profile name of IPython profile.

  • hostname (str) – Hostname of machine where the ipcluster runs. If connecting via the headnode use: socket.gethostname() or set local=True.

  • username (str) – Username to log into hostname. If not provided, it tries to look it up in your ssh/config.

  • password (str) – Password for ssh username@hostname.

  • culler (bool) – Controls starting of the culler. Default: True.

  • culler_args (str) – Add arguments to the culler. e.g. ‘–timeout=200’

  • env_path (str, default: None) – Path of the Python environment, ‘/path/to/ENV/’ if Python is in /path/to/ENV/bin/python. Examples ‘~/miniconda3/envs/dev/’, ‘miniconda3/envs/dev’, ‘~/miniconda3’. Defaults to the environment that is sourced in bashrc or bash_profile.

  • timeout (int) – Time for which we try to connect to get all the engines.

  • folder (str, optional) – Folder that is added to the path of the engines, e.g. “~/Work/my_current_project”.

  • client_kwargs (dict) – Keyword arguments that are passed to hpc05.Client().

  • kill_old_ipcluster (bool) – If True, it cleansup any old instances of ipcluster and kills your jobs in qstat or squeue.

Returns

  • client (ipython.Client object) – An IPyparallel client.

  • dview (ipyparallel.client.view.DirectView object) – Direct view, equivalent to client[:].

  • lview (ipyparallel.client.view.LoadBalancedView) – LoadedBalancedView, equivalent to client.load_balanced_view().

hpc05.kill_remote_ipcluster(hostname='hpc05', username=None, password=None, env_path=None)[source]

Kill your remote ipcluster and cleanup the files.

This should do the same as the following bash function (recommended: add this in your bash_profile / bashrc): ```bash del() {

qselect -u $USER | xargs qdel rm -f .hpc05.hpc ipengine* ipcontroller* pbs_* pkill -f hpc05_culler 2> /dev/null pkill -f ipcluster 2> /dev/null pkill -f ipengine 2> /dev/null pkill -f ipyparallel.controller 2> /dev/null pkill -f ipyparallel.engines 2> /dev/null