materials_commons.cli.tree_functions module

materials_commons.cli.tree_functions.check_and_upload_directory(proj, local_abspath, working_dir, limit=50, no_compare=False, upload_as=None, localtree=None, remotetree=None, parent_id=None)[source]

Checks validity and uploads a directory and contents recursively

Notes

Checks that target is not a file, with message:
remote is file (skipping)
Options allow providing the remote parent directory ID to reduce the number of API calls

Parameters:

proj (materials_commons.api.Project) – Project instance with proj.local_path indicating local project location
local_abspath (str) – Local absolute path to directory to be uploaded recursively
working_dir (str) – Current working directory, used for making relative paths and printing messages.
limit (int) – The limit in MB on the size of the files allowed to be uploaded.
no_compare (bool) – By default, this function checks local and remote file checksum to avoid downloading files that already exist. If no_compare is True, this check is skipped and all specified files are downloaded, even if an equivalent file already exists locally.
upload_as (str or None) – Materials Commons style path specifying where to upload. If None, use the equivalent location within the project.
localtree (LocalTree) – A LocalTree object stores local file checksums to avoid unnecessary hashing. Optional, will be used and updated if provided and checksum == True.
remotetree (RemoteTree) – A RemoteTree object stores remote file and directory information to minimize API calls and data transfer. Optional, will be used and updated if provided.
parent_id (str) – ID of parent directory where the file will be uploaded. May be None, in which case the directory will be created if necessary.

Returns:

(file_results, error_results) –

file_results: dict of path: file: Successfully uploaded files
error_results: dict of path: str: Error messages for unsuccessful or skipped uploads

materials_commons.cli.tree_functions.check_and_upload_file(proj, local_abspath, working_dir, limit=50, no_compare=False, upload_as=None, localtree=None, remotetree=None, parent_id=None, child_data=None, update_remotetree=True)[source]

Checks validity and upload one file

Notes

Checks that target is not a directory, with message:
remote is directory (skipping)
Depending on options given, checks if existing remote file is equivalent, with message:
`local is equivalent to remote (skipping)”
Options allow providing the remote parent directory ID, or parent directory child_data treecompare output to reduce the number of API calls

Parameters:

proj (materials_commons.api.Project) – Project instance with proj.local_path indicating local project location
local_abspath (str) – Local absolute path to file to be uploaded
working_dir (str) – Current working directory, used for making relative paths and printing messages.
limit (int) – The limit in MB on the size of the file allowed to be uploaded.
no_compare (bool) – By default, this function checks local and remote file checksum to avoid downloading files that already exist. If no_compare is True, this check is skipped and all specified files are downloaded, even if an equivalent file already exists locally.
upload_as (str) – Materials Commons style path specifying where to upload. Requires len(paths) == 1.
localtree (LocalTree) – A LocalTree object stores local file checksums to avoid unnecessary hashing. Optional, will be used and updated if provided and checksum == True.
remotetree (RemoteTree) – A RemoteTree object stores remote file and directory information to minimize API calls and data transfer. Optional, will be used and updated if provided.
parent_id (str) – ID of parent directory where the file will be uploaded. May be None, in which case the directory will be created if necessary.
child_data (dict or None) – If available, the child_data output from treecompare. If this file is being uploaded as part of a directory upload, the child_data comparing the local and remote files might already be available.
update_remotetree (bool) – Set to False to skip updating remotetree for the uploaded file. Used when updating via parent directory is preferrable.

Returns:

(file_result, error_result) –

file_result: file: Successfully uploaded files
error_results: str: Error messages for unsuccessful file uploads

materials_commons.cli.tree_functions.clipaths_to_local_abspaths(proj_local_path, clipaths, working_dir)[source]

Convert CLI paths input to local absolute paths

Parameters:

proj_local_path (str) – Path to Materials Commons project
clipaths (List of str) – Indicates files and directories, either absolute paths or relative to current working directory
working_dir (str) – Directory cli_paths are relative to.

Returns:

List local absolute paths to upload, excluding the .mc directory.

Raises:

MCCLIException, if any path in clipaths is not within the local project –
directory. –

materials_commons.cli.tree_functions.clipaths_to_mcpaths(proj_local_path, clipaths, working_dir)[source]

Convert CLI paths input to Materials Commons standardized paths

Parameters:

proj_local_path (str) – Path to Materials Commons project
clipaths (List of str) – Indicates files and directories, either absolute paths or relative to current working directory
working_dir (str) – Directory cli_paths are relative to.

Returns:

List Materials Commons paths (does not include project top directory, starts with “/”) to upload, excluding the .mc directory.

Raises:

MCCLIException, if any path in clipaths is not within the local project –
directory. –

materials_commons.cli.tree_functions.filter_local_abspaths(proj_local_path, local_abspaths, working_dir)[source]

Filter local_abspaths, skipping .mc and those specified by .mcignore

Parameters:

proj_local_path (str) – Path to project
local_abspaths (List of str) – Local absolute paths to file or directories to be uploaded
working_dir (str) – Current working directory, used for making relative paths and printing messages.

Returns:

_local_abspaths –

List of str: Filtered local absolute paths

materials_commons.cli.tree_functions.get_types(path, files_data, dirs_data)[source]

Use treecompare output to get local and remote types

Parameters:

path (str) – Path to check for type
files_data – The “files_data” output from treecompare()
dirs_data – The “dirs_data” output from treecompare()

Returns:

Tuple with (local_type, remote_type) of path.

materials_commons.cli.tree_functions.is_child_data_mismatch(child_data)[source]

Check treecompare child_data file comparison for type mismatch

Notes

Not existing is not a type mismatch

Returns:: _is_type_mismatch (bool) – This is True if l_type and r_type are different and not None, otherwise it is False.

materials_commons.cli.tree_functions.is_type_mismatch(path, files_data, dirs_data)[source]

Check treecompare filds_data and dirs_data output to check for type mismatch

Notes

Not existing is not a type mismatch

Returns:: _is_type_mismatch (bool) – This is True if l_type and r_type are different and not None, otherwise it is False.

materials_commons.cli.tree_functions.make_local_abspaths_for_upload(proj_local_path, paths)[source]

Clean paths for uploads

This is written for identifying uploads. If the top directory is included it replaces it with all children except .mc.

Parameters:

proj_local_path (str) – Path to project
paths (iterable of str) – Local absolute paths to filter

Returns:

List of str – Materials Commons paths, filtered as described above.

materials_commons.cli.tree_functions.make_mcpaths_for_upload(proj_local_path, paths)[source]

Clean paths for uploads

This is written for identifying uploads. If the top directory is included it replaces it with all children except .mc.

Parameters:

proj_local_path (str) – Path to project
paths (iterable of str) – Paths to filter and convert to absolute paths

Returns:

List of str – Materials Commons paths, filtered as described above.

materials_commons.cli.tree_functions.mkdir(proj, path, remote_only=False, create_intermediates=False, remotetree=None, parent_id=None)[source]

Make directories

Arguments

proj: mcapi.Project: Project instance with proj.local_path indicating local project location
path: str: Materials Commons style path (absolute path, not including project name directory) of directory to make.
create_intermediates: bool (optional, default=False): If True, make intermediate directories as necessary when they do not exist.
remote_only: bool (optional, default=False): If True, only make directories on remote. If False, make on local and remote.
remotetree: RemoteTree object (optional, default=None): A RemoteTree object stores remote file and directory information to minimize API calls and data transfer. Will be used and updated if provided.
parent_id (str): ID of parent directory where the directory should be created, if already: known. May be None, in which case the parent directory will be found using path.

Returns

result: mcapi.File or None: mcapi.File object representing the created directory, if successful.

Raises

Raises MCCLIException if unsuccessful with one of following messages:

path + “: is a local file”:
If attempting to create “/A/B/C” and any of “/A”, “/A/B”, or “/A/B/C” is an existing file locally and remote_only==False.

path + “: is a remote file”:
If attempting to create “/A/B/C” and any of “/A”, “/A/B”, or “/A/B/C” is an existing file on Materials Commons.

parent_path + “: parent directory does not exist”:
If attempting to create “/A/B/C” and the parent directory, “/A/B” does not exist on Materials Commons and create_intermediates==False.

materials_commons.cli.tree_functions.move(proj, paths, remote_only=False, localtree=None, remotetree=None)[source]

Move files and directories

Arguments

proj: mcapi.Project: Project instance with proj.local_path indicating local project location
paths: List of str: List of Materials Commons style paths (absolute path, not including project name directory) to move.
remote_only: bool (optional, default=False): If True, only move files and directories on remote. If False, move on local and remote.
localtree: LocalTree object (optional, default=None): A LocalTree object stores local file checksums to avoid unnecessary hashing. Will be used and updated if provided and checksum == True.
remotetree: RemoteTree object (optional, default=None): A RemoteTree object stores remote file and directory information to minimize API calls and data transfer. Will be used and updated if provided.

materials_commons.cli.tree_functions.remove(proj, paths, recursive=False, no_compare=False, remote_only=False, localtree=None, remotetree=None)[source]

Remove files and directories

Arguments

proj: mcapi.Project: Project instance with proj.local_path indicating local project location
paths: List of str: List of Materials Commons style paths (absolute path, not including project name directory) to remove.
recursive: bool (optional, default=False): If True, remove directories recursively. Otherwise, will not remove directories.
no_compare: bool (optional, default=False): If True, remove files and directories without checking for equality between local and remote.
remote_only: bool (optional, default=False): If True, only remove files and directories on remote. If False, remove on local and remote.
localtree: LocalTree object (optional, default=None): A LocalTree object stores local file checksums to avoid unnecessary hashing. Will be used and updated if provided and checksum == True.
remotetree: RemoteTree object (optional, default=None): A RemoteTree object stores remote file and directory information to minimize API calls and data transfer. Will be used and updated if provided.

materials_commons.cli.tree_functions.standard_upload(proj, paths, working_dir, recursive=False, limit=50, no_compare=False, upload_as=None, localtree=None, remotetree=None)[source]

Upload files to Materials Commons

Parameters:

proj (materials_commons.api.Project) – Project instance with proj.local_path indicating local project location
paths (List of str) – List of paths to upload. Expects local absolute paths, or paths relative to working_dir.
working_dir (str) – Current working directory, used for finding relative paths and printing messages.
recursive (bool) – If True, remove directories recursively. Otherwise, will not remove directories.
limit (int) – The limit in MB on the size of the file allowed to be uploaded.
no_compare (bool) – By default, this function checks local and remote file checksum to avoid downloading files that already exist. If no_compare is True, this check is skipped and all specified files are downloaded, even if an equivalent file already exists locally.
upload_as (str) – Materials Commons style path specifying where to upload. Requires len(paths) == 1.
localtree (LocalTree) – A LocalTree object stores local file checksums to avoid unnecessary hashing. Optional, will be used and updated if provided and checksum == True.
remotetree (RemoteTree) – A RemoteTree object stores remote file and directory information to minimize API calls and data transfer. Optional, will be used and updated if provided.

Returns:

(file_results, error_results) –

file_results: dict of path: file: Successfully uploaded files
error_results: dict of path: str: Error messages for unsuccessful file uploads

materials_commons.cli.tree_functions.standard_upload_v2(proj, paths, working_dir, recursive=False, limit=50, no_compare=False, upload_as=None, localtree=None, remotetree=None)[source]

Upload files and directories to Materials Commons

Parameters:

proj (materials_commons.api.Project) – Project instance with proj.local_path indicating local project location
paths (List of str) – List of paths to upload. Expects local absolute paths, or paths relative to working_dir.
working_dir (str) – Current working directory, used for finding relative paths and printing messages.
recursive (bool) – If True, remove directories recursively. Otherwise, will not remove directories.
limit (int) – The limit in MB on the size of the file allowed to be uploaded.
no_compare (bool) – By default, this function checks local and remote file checksum to avoid downloading files that already exist. If no_compare is True, this check is skipped and all specified files are downloaded, even if an equivalent file already exists locally.
upload_as (str) – Materials Commons style path specifying where to upload. Requires len(paths) == 1.
localtree (LocalTree) – A LocalTree object stores local file checksums to avoid unnecessary hashing. Optional, will be used and updated if provided and checksum == True.
remotetree (RemoteTree) – A RemoteTree object stores remote file and directory information to minimize API calls and data transfer. Optional, will be used and updated if provided.

Returns:

(file_results, error_results) –

file_results: dict of path: file: Successfully uploaded files
error_results: dict of path: str: Error messages for unsuccessful file uploads

materials_commons.cli.tree_functions.treecompare(proj, paths, checksum=False, localtree=None, remotetree=None, get_children=True)[source]

Compare files and directories on the local and remote trees.

Arguments

proj: mcapi.Project: Project instance with proj.local_path indicating local project location
paths: List of str: List of Materials Commons style paths (absolute path, not including project name directory) to query.
checksum: bool (optional, default=False): If True, calculate MD5 checksum of local files and compared to remote. If False, ‘eq’ will not be included in the output data.
localtree: LocalTree object (optional, default=None): A LocalTree object stores local file checksums to avoid unnecessary hashing. Will be used and updated if provided and checksum == True.
remotetree: RemoteTree object (optional, default=None): A RemoteTree object stores remote file and directory information to minimize API calls and data transfer. Will be used and updated if provided.
get_children: bool (optional, default=True): If True, compare children of directories.

Returns

(files_data, dirs_data, child_data, not_existing):

files_data: dict of filepath: file or directory comparison
Contains file comparisons

dirs_data: dict of dirpath: file or directory comparison
Contains directory comparisons

child_data: dict of dirpath: childpath: file or directory comparison
Contains directory children comparisons, if get_children==True, else empty dict.

not_existing: list of str
Paths that do not exist locally or remotely

For each file or directory the comparison data is:
‘l_mtime’: float, local file modify time (seconds since epoch) ‘l_size’: int, local file size in bytes ‘l_type’: str, local file type (‘file’ or ‘directory’) ‘l_checksum’: str, local file md5 hash ‘r_mtime’: float, remote file modify time (seconds since epoch) ‘r_size’: int, remote file size in bytes ‘r_type’: remote file type (‘file’ or ‘directory’) ‘r_checksum’: str, remote file md5 hash ‘r_obj’: File or Directory, remote object ‘eq’: bool, whether the local and remote files are equivalent ‘path’: str, path to file or directory (including the project top) ‘id’: str, Materials Commons ID, if exists ‘parent_id’: str, Materials Commons ID, if exists

Values are None if the file does not exist in the relevant tree.

Notes

The equivalence check (‘eq’ in data) is only done for files. For directories, it is always None.

When directories are updated in localtree and remotetree their children are also updated (not recursively).

Remote objects, ‘r_obj’, are only returned if remotetree is None.

materials_commons.cli.tree_functions.upload_file(proj, local_abspath, mcpath, working_dir, parent_id=None, limit=50, remotetree=None, update_remotetree=True)[source]

Upload one file

Notes

Creates parent and intermediate directories as necessary
Does not allow filename change, with message:
–upload-as file name changed (skipping)
Will not upload files over the size limit, with message:
file too large (size={1}MB, limit={0}MB) (not uploaded)

Parameters:

proj (materials_commons.api.Project) – Project instance with proj.local_path indicating local project location
local_abspath (str) – Local absolute path to file to be uploaded
mcpath (dict) – Path where the file will be uploaded. Currently, the basename must be the same as local_abspath.
parent_id (str) – ID of parent directory where the file will be uploaded. May be None, in which case the directory os.path.dirname(mcpath) will be created.
working_dir (str) – Current working directory, used for making relative paths and printing messages.
limit (int) – The limit in MB on the size of the file allowed to be uploaded.
remotetree (RemoteTree) – A RemoteTree object stores remote file and directory information to minimize API calls and data transfer. Optional, will be used and updated if provided.
update_remotetree (bool) – Set to False to skip updating remotetree for the uploaded file. Used when updating via parent directory is preferrable.

Returns:

(file_result, error_result) –

file_result: file: Successfully uploaded files
error_results: str: Error messages for unsuccessful file uploads