aiida.transports
#
Module for classes and utilities to define transports to other machines.
Package Contents#
Classes#
Support connection, command execution and data transfer to remote computers via SSH+SFTP. |
|
Abstract class for a generic transport (ssh, local, β¦) contains the set of minimal methods. |
Functions#
Convert a string passed in the CLI to a valid bool. |
|
Return the ssh configuration for a given computer name. |
API#
- class aiida.transports.SshTransport(*args, **kwargs)#
Bases:
aiida.transports.transport.Transport
Support connection, command execution and data transfer to remote computers via SSH+SFTP.
Initialization
Initialize the SshTransport class.
- Parameters:
machine β the machine to connect to
load_system_host_keys β (optional, default False) if False, do not load the system host keys
key_policy β (optional, default = paramiko.RejectPolicy()) the policy to use for unknown keys
Other parameters valid for the ssh connect function (see the self._valid_connect_params list) are passed to the connect function (as port, username, password, β¦); taken from the accepted paramiko.SSHClient.connect() params.
- _valid_connect_options = [('username',), ('port',), ('look_for_keys',), ('key_filename',), ('timeout',), ('allow_agent',), ('...#
- _valid_connect_params = None#
- _valid_auth_options = None#
- _MAX_EXEC_COMMAND_LOG_SIZE = None#
- classmethod _get_username_suggestion_string(computer)#
Return a suggestion for the specific field.
- classmethod _get_port_suggestion_string(computer)#
Return a suggestion for the specific field.
- classmethod _get_key_filename_suggestion_string(computer)#
Return a suggestion for the specific field.
- classmethod _get_timeout_suggestion_string(computer)#
Return a suggestion for the specific field.
Provide 60s as a default timeout for connections.
- classmethod _get_allow_agent_suggestion_string(computer)#
Return a suggestion for the specific field.
- classmethod _get_look_for_keys_suggestion_string(computer)#
Return a suggestion for the specific field.
- classmethod _get_proxy_command_suggestion_string(computer)#
Return a suggestion for the specific field.
- classmethod _get_proxy_jump_suggestion_string(_)#
Return an empty suggestion since Paramiko does not parse ProxyJump from the SSH config.
- classmethod _get_compress_suggestion_string(computer)#
Return a suggestion for the specific field.
- classmethod _get_load_system_host_keys_suggestion_string(computer)#
Return a suggestion for the specific field.
- classmethod _get_key_policy_suggestion_string(computer)#
Return a suggestion for the specific field.
- classmethod _get_gss_auth_suggestion_string(computer)#
Return a suggestion for the specific field.
- classmethod _get_gss_kex_suggestion_string(computer)#
Return a suggestion for the specific field.
- classmethod _get_gss_deleg_creds_suggestion_string(computer)#
Return a suggestion for the specific field.
- classmethod _get_gss_host_suggestion_string(computer)#
Return a suggestion for the specific field.
- open()#
Open a SSHClient to the machine possibly using the parameters given in the __init__.
Also opens a sftp channel, ready to be used. The current working directory is set explicitly, so it is not None.
- Raises:
aiida.common.InvalidOperation β if the channel is already open
- _close_proxies()#
Close all proxy connections (proxy_jump and proxy_command)
- close()#
Close the SFTP channel, and the SSHClient.
- Todo:
correctly manage exceptions
- Raises:
aiida.common.InvalidOperation β if the channel is already open
- property sshclient#
- property sftp#
- __str__()#
Return a useful string.
- chdir(path)#
Change directory of the SFTP session. Emulated internally by paramiko.
Differently from paramiko, if you pass None to chdir, nothing happens and the cwd is unchanged.
- normalize(path='.')#
Returns the normalized path (removing double slashes, etcβ¦)
- stat(path)#
Retrieve information about a file on the remote system. The return value is an object whose attributes correspond to the attributes of Pythonβs
stat
structure as returned byos.stat
, except that it contains fewer fields. The fields supported are:st_mode
,st_size
,st_uid
,st_gid
,st_atime
, andst_mtime
.- Parameters:
path (str) β the filename to stat
- Returns:
a paramiko.sftp_attr.SFTPAttributes object containing attributes about the given file.
- lstat(path)#
Retrieve information about a file on the remote system, without following symbolic links (shortcuts). This otherwise behaves exactly the same as stat.
- Parameters:
path (str) β the filename to stat
- Returns:
a paramiko.sftp_attr.SFTPAttributes object containing attributes about the given file.
- getcwd()#
Return the current working directory for this SFTP session, as emulated by paramiko. If no directory has been set with chdir, this method will return None. But in __enter__ this is set explicitly, so this should never happen within this class.
- makedirs(path, ignore_existing=False)#
Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not just the rightmost) will be created if it does not exist.
NOTE: since os.path.split uses the separators as the host system (that could be windows), I assume the remote computer is Linux-based and use β/β as separators!
- Parameters:
path β directory to create (string)
ignore_existing β if set to true, it doesnβt give any error if the leaf directory does already exist (bool)
- Raises:
OSError β If the directory already exists.
- mkdir(path, ignore_existing=False)#
Create a folder (directory) named path.
- Parameters:
path β name of the folder to create
ignore_existing β if True, does not give any error if the directory already exists
- Raises:
OSError β If the directory already exists.
- rmtree(path)#
Remove a file or a directory at path, recursively Flags used: -r: recursive copy; -f: force, makes the command non interactive;
- Parameters:
path β remote path to delete
- Raises:
IOError β if the rm execution failed.
- rmdir(path)#
Remove the folder named βpathβ if empty.
- abstract chown(path, uid, gid)#
Change owner permissions of a file.
For now, this is not implemented for the SSH transport.
- isdir(path)#
Return True if the given path is a directory, False otherwise. Return False also if the path does not exist.
- chmod(path, mode)#
Change permissions to path
- Parameters:
path β path to file
mode β new permission bits (integer)
- static _os_path_split_asunder(path)#
Used by makedirs. Takes path (a str) and returns a list deconcatenating the path
- put(localpath, remotepath, callback=None, dereference=True, overwrite=True, ignore_nonexisting=False)#
Put a file or a folder from local to remote. Redirects to putfile or puttree.
- Parameters:
localpath β an (absolute) local path
remotepath β a remote path
dereference β follow symbolic links (boolean). Default = True (default behaviour in paramiko). False is not implemented.
overwrite β if True overwrites files and folders (boolean). Default = False.
- Raises:
ValueError β if local path is invalid
OSError β if the localpath does not exist
- putfile(localpath, remotepath, callback=None, dereference=True, overwrite=True)#
Put a file from local to remote.
- Parameters:
localpath β an (absolute) local path
remotepath β a remote path
overwrite β if True overwrites files and folders (boolean). Default = True.
- Raises:
ValueError β if local path is invalid
OSError β if the localpath does not exist, or unintentionally overwriting
- puttree(localpath, remotepath, callback=None, dereference=True, overwrite=True)#
Put a folder recursively from local to remote.
By default, overwrite.
- Parameters:
localpath β an (absolute) local path
remotepath β a remote path
dereference β follow symbolic links (boolean) Default = True (default behaviour in paramiko). False is not implemented.
overwrite β if True overwrites files and folders (boolean). Default = True
- Raises:
ValueError β if local path is invalid
OSError β if the localpath does not exist, or trying to overwrite
IOError β if remotepath is invalid
Note
setting dereference equal to True could cause infinite loops. see os.walk() documentation
- get(remotepath, localpath, callback=None, dereference=True, overwrite=True, ignore_nonexisting=False)#
Get a file or folder from remote to local. Redirects to getfile or gettree.
- Parameters:
remotepath β a remote path
localpath β an (absolute) local path
dereference β follow symbolic links. Default = True (default behaviour in paramiko). False is not implemented.
overwrite β if True overwrites files and folders. Default = False
- Raises:
ValueError β if local path is invalid
IOError β if the remotepath is not found
- getfile(remotepath, localpath, callback=None, dereference=True, overwrite=True)#
Get a file from remote to local.
- Parameters:
remotepath β a remote path
localpath β an (absolute) local path
overwrite β if True overwrites files and folders. Default = False
- Raises:
ValueError β if local path is invalid
OSError β if unintentionally overwriting
- gettree(remotepath, localpath, callback=None, dereference=True, overwrite=True)#
Get a folder recursively from remote to local.
- Parameters:
remotepath β a remote path
localpath β an (absolute) local path
dereference β follow symbolic links. Default = True (default behaviour in paramiko). False is not implemented.
overwrite β if True overwrites files and folders. Default = False
- Raises:
ValueError β if local path is invalid
IOError β if the remotepath is not found
OSError β if unintentionally overwriting
- get_attribute(path)#
Returns the object Fileattribute, specified in aiida.transports Receives in input the path of a given file.
- copyfile(remotesource, remotedestination, dereference=False)#
- copytree(remotesource, remotedestination, dereference=False)#
- copy(remotesource, remotedestination, dereference=False, recursive=True)#
Copy a file or a directory from remote source to remote destination. Flags used:
-r
: recursive copy;-f
: force, makes the command non interactive;-L
follows symbolic links- Parameters:
remotesource β file to copy from
remotedestination β file to copy to
dereference β if True, copy content instead of copying the symlinks only Default = False.
recursive (bool) β if True copy directories recursively, otherwise only copy the specified file(s)
- Raises:
IOError β if the cp execution failed.
Note
setting dereference equal to True could cause infinite loops.
- _exec_cp(cp_exe, cp_flags, src, dst)#
Execute the
cp
command on the remote machine.
- static _local_listdir(path, pattern=None)#
Acts on the local folder, for the rest, same as listdir
- listdir(path='.', pattern=None)#
Get the list of files at path.
- Parameters:
path β default = β.β
pattern β returns the list of files matching pattern. Unix only. (Use to emulate
ls *
for example)
- remove(path)#
Remove a single file at βpathβ
- rename(oldpath, newpath)#
Rename a file or folder from oldpath to newpath.
- Parameters:
- Raises:
IOError β if oldpath/newpath is not found
ValueError β if sroldpathc/newpath is not a valid string
- isfile(path)#
Return True if the given path is a file, False otherwise. Return False also if the path does not exist.
- _exec_command_internal(command, combine_stderr=False, bufsize=-1)#
Executes the specified command in bash login shell.
Before the command is executed, changes directory to the current working directory as returned by self.getcwd().
For executing commands and waiting for them to finish, use exec_command_wait.
- Parameters:
command β the command to execute. The command is assumed to be already escaped using
aiida.common.escaping.escape_for_bash()
.combine_stderr β (default False) if True, combine stdout and stderr on the same buffer (i.e., stdout). Note: If combine_stderr is True, stderr will always be empty.
bufsize β same meaning of the one used by paramiko.
- Returns:
a tuple with (stdin, stdout, stderr, channel), where stdin, stdout and stderr behave as file-like objects, plus the methods provided by paramiko, and channel is a paramiko.Channel object.
- exec_command_wait_bytes(command, stdin=None, combine_stderr=False, bufsize=-1)#
Executes the specified command and waits for it to finish.
- Parameters:
command β the command to execute
stdin β (optional,default=None) can be a string or a file-like object.
combine_stderr β (optional, default=False) see docstring of self._exec_command_internal()
bufsize β same meaning of paramiko.
- Returns:
a tuple with (return_value, stdout, stderr) where stdout and stderr are both bytes and the return_value is an int.
- gotocomputer_command(remotedir)#
Specific gotocomputer string to connect to a given remote computer via ssh and directly go to the calculation folder.
- _symlink(source, dest)#
Wrap SFTP symlink call without breaking API
- Parameters:
source β source of link
dest β link to create
- symlink(remotesource, remotedestination)#
Create a symbolic link between the remote source and the remote destination.
- Parameters:
remotesource β remote source. Can contain a pattern.
remotedestination β remote destination
- path_exists(path)#
Check if path exists
- class aiida.transports.Transport(*args, **kwargs)#
Bases:
abc.ABC
Abstract class for a generic transport (ssh, local, β¦) contains the set of minimal methods.
Initialization
__init__ method of the Transport base class.
- Parameters:
safe_interval β (optional, default self._DEFAULT_SAFE_OPEN_INTERVAL) Minimum time interval in seconds between opening new connections.
use_login_shell β (optional, default True) if False, do not use a login shell when executing command
- DEFAULT_MINIMUM_JOB_POLL_INTERVAL = 10#
- _DEFAULT_SAFE_OPEN_INTERVAL = 30.0#
- _valid_auth_params = None#
- _MAGIC_CHECK = 'compile(...)'#
- _common_auth_options = [('use_login_shell',), ('safe_interval',)]#
- __enter__()#
For transports that require opening a connection, opens all required channels (used in βwithβ statements).
This object can be used in nested with statements and the connection will only be opened once and closed when the final with scope finishes e.g.:
t = Transport() with t: # Connection is now open.. with t: # ..still open.. pass # ..still open.. # ...closed
- __exit__(type_, value, traceback)#
Closes connections, if needed (used in βwithβ statements).
- property is_open#
- abstract open()#
Opens a local transport channel
- abstract close()#
Closes the local transport channel
- __repr__()#
- __str__()#
- set_logger_extra(logger_extra)#
Pass the data that should be passed automatically to self.logger as βextraβ keyword. This is typically useful if you pass data obtained using get_dblogger_extra in aiida.orm.utils.log, to automatically log also to the DbLog table.
- Parameters:
logger_extra β data that you want to pass as extra to the self.logger. To write to DbLog, it should be created by the aiida.orm.utils.log.get_dblogger_extra function. Pass None if you do not want to have extras passed.
- classmethod get_short_doc()#
Return the first non-empty line of the class docstring, if available
- classmethod get_valid_auth_params()#
Return the internal list of valid auth_params
- auth_options() collections.OrderedDict #
Return the authentication options to be used for building the CLI.
- Returns:
OrderedDict of tuples, with first element option name and second dictionary of kwargs
- classmethod _get_safe_interval_suggestion_string(computer)#
Return as a suggestion the default safe interval of this Transport class.
This is used to provide a default in
verdi computer configure
.
- classmethod _get_use_login_shell_suggestion_string(computer)#
Return a suggestion for the specific field.
- property logger#
Return the internal logger. If you have set extra parameters using set_logger_extra(), a suitable LoggerAdapter instance is created, bringing with itself also the extras.
- get_safe_open_interval()#
Get an interval (in seconds) that suggests how long the user should wait between consecutive calls to open the transport. This can be used as a way to get the user to not swamp a limited number of connections, etc. However it is just advisory.
If returns 0, it is taken that there are no reasons to limit the frequency of open calls.
In the main class, it returns a default value (>0 for safety), set in the _DEFAULT_SAFE_OPEN_INTERVAL attribute of the class. Plugins should override it.
- Returns:
The safe interval between calling open, in seconds
- Return type:
- abstract chdir(path)#
Change directory to βpathβ
- abstract chmod(path, mode)#
Change permissions of a path.
- abstract chown(path, uid, gid)#
Change the owner (uid) and group (gid) of a file. As with pythonβs os.chown function, you must pass both arguments, so if you only want to change one, use stat first to retrieve the current owner and group.
- abstract copy(remotesource, remotedestination, dereference=False, recursive=True)#
Copy a file or a directory from remote source to remote destination (On the same remote machine)
- Parameters:
remotesource (str) β path of the remote source directory / file
remotedestination (str) β path of the remote destination directory / file
dereference (bool) β if True copy the contents of any symlinks found, otherwise copy the symlinks themselves
recursive (bool) β if True copy directories recursively, otherwise only copy the specified file(s)
- Raises:
IOError, if one of src or dst does not exist
- abstract copyfile(remotesource, remotedestination, dereference=False)#
Copy a file from remote source to remote destination (On the same remote machine)
- Parameters:
- Raises:
IOError β if one of src or dst does not exist
- abstract copytree(remotesource, remotedestination, dereference=False)#
Copy a folder from remote source to remote destination (On the same remote machine)
- Parameters:
- Raises:
IOError β if one of src or dst does not exist
- copy_from_remote_to_remote(transportdestination, remotesource, remotedestination, **kwargs)#
Copy files or folders from a remote computer to another remote computer.
- Parameters:
transportdestination β transport to be used for the destination computer
remotesource (str) β path to the remote source directory / file
remotedestination (str) β path to the remote destination directory / file
kwargs β keyword parameters passed to the call to transportdestination.put, except for βdereferenceβ that is passed to self.get
Note
the keyword βdereferenceβ SHOULD be set to False for the final put (onto the destination), while it can be set to the value given in kwargs for the get from the source. In that way, a symbolic link would never be followed in the final copy to the remote destination. That way we could avoid getting unknown (potentially malicious) files into the destination computer. HOWEVER, since dereference=False is currently NOT supported by all plugins, we still force it to True for the final put.
Note
the supported keys in kwargs are callback, dereference, overwrite and ignore_nonexisting.
- abstract _exec_command_internal(command, **kwargs)#
Execute the command on the shell, similarly to os.system.
Enforce the execution to be run from the cwd (as given by self.getcwd), if this is not None.
If possible, use the higher-level exec_command_wait function.
- Parameters:
command (str) β execute the command given as a string
- Returns:
stdin, stdout, stderr and the session, when this exists (can be None).
- abstract exec_command_wait_bytes(command, stdin=None, **kwargs)#
Execute the command on the shell, waits for it to finish, and return the retcode, the stdout and the stderr as bytes.
Enforce the execution to be run from the pwd (as given by self.getcwd), if this is not None.
The command implementation can have some additional plugin-specific kwargs.
- Parameters:
command (str) β execute the command given as a string
stdin β (optional,default=None) can be a string or a file-like object.
- Returns:
a tuple: the retcode (int), stdout (bytes) and stderr (bytes).
- exec_command_wait(command, stdin=None, encoding='utf-8', **kwargs)#
Executes the specified command and waits for it to finish.
- Note:
this function also decodes the bytes received into a string with the specified encoding, which is set to be
utf-8
by default (for backward-compatibility with earlier versions) of AiiDA. Use this method only if you are sure that you are getting a properly encoded string; otherwise, use theexec_command_wait_bytes
method that returns the undecoded byte stream.- Note:
additional kwargs are passed to the
exec_command_wait_bytes
function, that might use them depending on the plugin.- Parameters:
command β the command to execute
stdin β (optional,default=None) can be a string or a file-like object.
encoding β the encoding to use to decode the byte stream received from the remote command execution.
- Returns:
a tuple with (return_value, stdout, stderr) where stdout and stderr are both strings, decoded with the specified encoding.
- abstract get(remotepath, localpath, *args, **kwargs)#
Retrieve a file or folder from remote source to local destination dst must be an absolute path (src not necessarily)
- Parameters:
remotepath β (str) remote_folder_path
localpath β (str) local_folder_path
- abstract getfile(remotepath, localpath, *args, **kwargs)#
Retrieve a file from remote source to local destination dst must be an absolute path (src not necessarily)
- abstract gettree(remotepath, localpath, *args, **kwargs)#
Retrieve a folder recursively from remote source to local destination dst must be an absolute path (src not necessarily)
- abstract getcwd()#
Get working directory
- Returns:
a string identifying the current working directory
- abstract get_attribute(path)#
Return an object FixedFieldsAttributeDict for file in a given path, as defined in aiida.common.extendeddicts Each attribute object consists in a dictionary with the following keys:
st_size: size of files, in bytes
st_uid: user id of owner
st_gid: group id of owner
st_mode: protection bits
st_atime: time of most recent access
st_mtime: time of most recent modification
- Parameters:
path (str) β path to file
- Returns:
object FixedFieldsAttributeDict
- get_mode(path)#
Return the portion of the fileβs mode that can be set by chmod().
- Parameters:
path (str) β path to file
- Returns:
the portion of the fileβs mode that can be set by chmod()
- abstract isdir(path)#
True if path is an existing directory.
- Parameters:
path (str) β path to directory
- Returns:
boolean
- abstract isfile(path)#
Return True if path is an existing file.
- Parameters:
path (str) β path to file
- Returns:
boolean
- abstract listdir(path='.', pattern=None)#
Return a list of the names of the entries in the given path. The list is in arbitrary order. It does not include the special entries β.β and β..β even if they are present in the directory.
- listdir_withattributes(path='.', pattern=None)#
Return a list of the names of the entries in the given path. The list is in arbitrary order. It does not include the special entries β.β and β..β even if they are present in the directory.
- Parameters:
- Returns:
a list of dictionaries, one per entry. The schema of the dictionary is the following:
{ 'name': String, 'attributes': FileAttributeObject, 'isdir': Bool }
where βnameβ is the file or folder directory, and any other information is metadata (if the file is a folder, a directory, β¦). βattributesβ behaves as the output of transport.get_attribute(); isdir is a boolean indicating if the object is a directory or not.
- abstract makedirs(path, ignore_existing=False)#
Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not just the rightmost) will be created if it does not exist.
- abstract mkdir(path, ignore_existing=False)#
Create a folder (directory) named path.
- abstract normalize(path='.')#
Return the normalized path (on the server) of a given path. This can be used to quickly resolve symbolic links or determine what the server is considering to be the βcurrent folderβ.
- abstract put(localpath, remotepath, *args, **kwargs)#
Put a file or a directory from local src to remote dst. src must be an absolute path (dst not necessarily)) Redirects to putfile and puttree.
- abstract putfile(localpath, remotepath, *args, **kwargs)#
Put a file from local src to remote dst. src must be an absolute path (dst not necessarily))
- abstract puttree(localpath, remotepath, *args, **kwargs)#
Put a folder recursively from local src to remote dst. src must be an absolute path (dst not necessarily))
- abstract remove(path)#
Remove the file at the given path. This only works on files; for removing folders (directories), use rmdir.
- abstract rename(oldpath, newpath)#
Rename a file or folder from oldpath to newpath.
- Parameters:
- Raises:
IOError β if oldpath/newpath is not found
ValueError β if oldpath/newpath is not a valid string
- abstract rmdir(path)#
Remove the folder named path. This works only for empty folders. For recursive remove, use rmtree.
- Parameters:
path (str) β absolute path to the folder to remove
- abstract rmtree(path)#
Remove recursively the content at path
- Parameters:
path (str) β absolute path to remove
- abstract gotocomputer_command(remotedir)#
Return a string to be run using os.system in order to connect via the transport to the remote directory.
Expected behaviors:
A new bash session is opened
A reasonable error message is produced if the folder does not exist
- Parameters:
remotedir (str) β the full path of the remote directory
- abstract symlink(remotesource, remotedestination)#
Create a symbolic link between the remote source and the remote destination.
- Parameters:
remotesource β remote source
remotedestination β remote destination
- whoami()#
Get the remote username
- Returns:
list of username (str), retval (int), stderr (str)
- abstract path_exists(path)#
Returns True if path exists, False otherwise.
- glob(pathname)#
Return a list of paths matching a pathname pattern.
The pattern may contain simple shell-style wildcards a la fnmatch.
- iglob(pathname)#
Return an iterator which yields the paths matching a pathname pattern.
The pattern may contain simple shell-style wildcards a la fnmatch.
- glob1(dirname, pattern)#
Match subpaths of dirname against pattern.
- glob0(dirname, basename)#
Wrap basename i a list if it is empty or if dirname/basename is an existing path, else return empty list.
- has_magic(string)#
- _gotocomputer_string(remotedir)#
command executed when goto computer.
- aiida.transports.convert_to_bool(string)#
Convert a string passed in the CLI to a valid bool.
- Returns:
the parsed bool value.
- Raises:
ValueError β If the value is not parsable as a bool
- aiida.transports.parse_sshconfig(computername)#
Return the ssh configuration for a given computer name.
This parses the
.ssh/config
file in the home directory and returns the part of configuration of the given computer name.- Parameters:
computername β the computer name for which we want the configuration.