materials_commons.cli.tree_functions module

materials_commons.cli.tree_functions.clipaths_to_local_abspaths(proj_local_path, clipaths, working_dir)[source]

Convert CLI paths input to local absolute paths

Parameters

  • proj_local_path (str) – Path to Materials Commons project

  • clipaths (List of str) – Indicates files and directories, either absolute paths or relative to current working directory

  • working_dir (str) – Directory cli_paths are relative to.

Returns

List local absolute paths to upload, excluding the .mc directory.

Raises

  • MCCLIException, if any path in clipaths is not within the local project –

  • directory. –

materials_commons.cli.tree_functions.clipaths_to_mcpaths(proj_local_path, clipaths, working_dir)[source]

Convert CLI paths input to Materials Commons standardized paths

Parameters

  • proj_local_path (str) – Path to Materials Commons project

  • clipaths (List of str) – Indicates files and directories, either absolute paths or relative to current working directory

  • working_dir (str) – Directory cli_paths are relative to.

Returns

List Materials Commons paths (does not include project top directory, starts with “/”) to upload, excluding the .mc directory.

Raises

  • MCCLIException, if any path in clipaths is not within the local project –

  • directory. –

materials_commons.cli.tree_functions.get_types(path, files_data, dirs_data)[source]

Use treecompare output to get local and remote types

Parameters

  • path (str) – Path to check for type

  • files_data – The “files_data” output from treecompare()

  • dirs_data – The “dirs_data” output from treecompare()

Returns

Tuple with (local_type, remote_type) of path.

materials_commons.cli.tree_functions.is_child_data_mismatch(child_data)[source]

Check treecompare child_data file comparison for type mismatch

materials_commons.cli.tree_functions.is_type_mismatch(path, files_data, dirs_data)[source]

Check treecompare filds_data and dirs_data output to check for type mismatch

materials_commons.cli.tree_functions.make_local_abspaths_for_upload(proj_local_path, paths)[source]

Clean paths for uploads

This is written for identifying uploads. If the top directory is included it replaces it with all children except .mc.

Parameters

  • proj_local_path (str) – Path to project

  • paths (iterable of str) – Local absolute paths to filter

Returns

List of str – Materials Commons paths, filtered as described above.

materials_commons.cli.tree_functions.make_mcpaths_for_upload(proj_local_path, paths)[source]

Clean paths for uploads

This is written for identifying uploads. If the top directory is included it replaces it with all children except .mc.

Parameters

  • proj_local_path (str) – Path to project

  • paths (iterable of str) – Paths to filter and convert to absolute paths

Returns

List of str – Materials Commons paths, filtered as described above.

materials_commons.cli.tree_functions.mkdir(proj, path, remote_only=False, create_intermediates=False, remotetree=None)[source]

Make directories

proj: mcapi.Project

Project instance with proj.local_path indicating local project location

path: str

Materials Commons style path (absolute path, not including project name directory) of directory to make.

create_intermediates: bool (optional, default=False)

If True, make intermediate directories as necessary when they do not exist.

remote_only: bool (optional, default=False)

If True, only make directories on remote. If False, make on local and remote.

remotetree: RemoteTree object (optional, default=None)

A RemoteTree object stores remote file and directory information to minimize API calls and data transfer. Will be used and updated if provided.

result: mcapi.File or None

mcapi.File object representing the created directory, if successful.

Raises MCCLIException if unsuccessful with one of following messages:

  • path + “: is a local file”:

    If attempting to create “/A/B/C” and any of “/A”, “/A/B”, or “/A/B/C” is an existing file locally and remote_only==False.

  • path + “: is a remote file”:

    If attempting to create “/A/B/C” and any of “/A”, “/A/B”, or “/A/B/C” is an existing file on Materials Commons.

  • parent_path + “: parent directory does not exist”:

    If attempting to create “/A/B/C” and the parent directory, “/A/B” does not exist on Materials Commons and create_intermediates==False.

materials_commons.cli.tree_functions.move(proj, paths, remote_only=False, localtree=None, remotetree=None)[source]

Move files and directories

proj: mcapi.Project

Project instance with proj.local_path indicating local project location

paths: List of str

List of Materials Commons style paths (absolute path, not including project name directory) to move.

remote_only: bool (optional, default=False)

If True, only move files and directories on remote. If False, move on local and remote.

localtree: LocalTree object (optional, default=None)

A LocalTree object stores local file checksums to avoid unnecessary hashing. Will be used and updated if provided and checksum == True.

remotetree: RemoteTree object (optional, default=None)

A RemoteTree object stores remote file and directory information to minimize API calls and data transfer. Will be used and updated if provided.

materials_commons.cli.tree_functions.remove(proj, paths, recursive=False, no_compare=False, remote_only=False, localtree=None, remotetree=None)[source]

Remove files and directories

proj: mcapi.Project

Project instance with proj.local_path indicating local project location

paths: List of str

List of Materials Commons style paths (absolute path, not including project name directory) to remove.

recursive: bool (optional, default=False)

If True, remove directories recursively. Otherwise, will not remove directories.

no_compare: bool (optional, default=False)

If True, remove files and directories without checking for equality between local and remote.

remote_only: bool (optional, default=False)

If True, only remove files and directories on remote. If False, remove on local and remote.

localtree: LocalTree object (optional, default=None)

A LocalTree object stores local file checksums to avoid unnecessary hashing. Will be used and updated if provided and checksum == True.

remotetree: RemoteTree object (optional, default=None)

A RemoteTree object stores remote file and directory information to minimize API calls and data transfer. Will be used and updated if provided.

materials_commons.cli.tree_functions.standard_upload(proj, paths, working_dir, recursive=False, limit=50, no_compare=False, upload_as=None, localtree=None, remotetree=None)[source]

Upload files to Materials Commons

Parameters

  • proj (materials_commons.api.Project) – Project instance with proj.local_path indicating local project location

  • paths (List of str) – List of paths to upload. Expects local absolute paths, or paths relative to working_dir.

  • working_dir (str) – Current working directory, used for finding relative paths and printing messages.

  • recursive (bool) – If True, remove directories recursively. Otherwise, will not remove directories.

  • limit (int) – The limit in MB on the size of the file allowed to be uploaded.

  • no_compare (bool) – By default, this function checks local and remote file checksum to avoid downloading files that already exist. If no_compare is True, this check is skipped and all specified files are downloaded, even if an equivalent file already exists locally.

  • upload_as (str) – Materials Commons style path specifying where to upload. Requires len(paths) == 1.

  • localtree (LocalTree) – A LocalTree object stores local file checksums to avoid unnecessary hashing. Optional, will be used and updated if provided and checksum == True.

  • remotetree (RemoteTree) – A RemoteTree object stores remote file and directory information to minimize API calls and data transfer. Optional, will be used and updated if provided.

Returns

(file_results, error_results) –

file_results: dict of path: file

Successfully uploaded files

error_results: dict of path: str

Error messages for unsuccessful file uploads

materials_commons.cli.tree_functions.treecompare(proj, paths, checksum=False, localtree=None, remotetree=None)[source]

Compare files and directories on the local and remote trees.

proj: mcapi.Project

Project instance with proj.local_path indicating local project location

paths: List of str

List of Materials Commons style paths (absolute path, not including project name directory) to query.

checksum: bool (optional, default=False)

If True, calculate MD5 checksum of local files and compared to remote. If False, ‘eq’ will not be included in the output data.

localtree: LocalTree object (optional, default=None)

A LocalTree object stores local file checksums to avoid unnecessary hashing. Will be used and updated if provided and checksum == True.

remotetree: RemoteTree object (optional, default=None)

A RemoteTree object stores remote file and directory information to minimize API calls and data transfer. Will be used and updated if provided.

(files_data, dirs_data, child_data, not_existing):

files_data: dict of filepath: file or directory comparison

Contains file comparisons

dirs_data: dict of dirpath: file or directory comparison

Contains directory comparisons

child_data: dict of dirpath: childpath: file or directory comparison

Contains directory children comparisons

not_existing: list of str

Paths that do not exist locally or remotely

For each file or directory the comparison data is:

‘l_mtime’: float, local file modify time (seconds since epoch) ‘l_size’: int, local file size in bytes ‘l_type’: str, local file type (‘file’ or ‘directory’) ‘l_checksum’: str, local file md5 hash ‘r_mtime’: float, remote file modify time (seconds since epoch) ‘r_size’: int, remote file size in bytes ‘r_type’: remote file type (‘file’ or ‘directory’) ‘r_checksum’: str, remote file md5 hash ‘r_obj’: File or Directory, remote object ‘eq’: bool, whether the local and remote files are equivalent ‘path’: str, path to file or directory (including the project top) ‘id’: str, Materials Commons ID, if exists ‘parent_id’: str, Materials Commons ID, if exists

Values are None if the file does not exist in the relevant tree.

The equivalence check (‘eq’ in data) is only done for files. For directories, it is always None.

When directories are updated in localtree and remotetree their children are also updated (not recursively).

Remote objects, ‘r_obj’, are only returned if remotetree is None.