Create a new Materials Commons project

This example demonstrates creating a new Materials Commons project from a Jupyter notebook. To try running it locally, download the notebook from here.

To install the necessary dependencies (requires Python 3):

pip install materials-commons-cli

Notes:

• If you do not yet have a Materials Commons account, it can be created here.
• If you have not yet configured the Materials Commons client to access your account, you will be prompted to enter a password.
• Only one project per owner may have the same name. If the same name as an existing project is given, then the existing project is returned instead of creating a new project.

# Login information (Edit here or be prompted by the next cell)

email = None
mcurl = "https://materialscommons.org/api"

# Construct a Materials Commons client

from materials_commons.cli.user_config import make_client_and_login_if_necessary

if email is None:
    print("Account (email):")
    email = input()

client = make_client_and_login_if_necessary(email=email, mcurl=mcurl)

Account (email):
user@domain.edu
Login to: user@domain.edu https://materialscommons.org/api
password: ········

Added APIKey for user@domain.edu at https://materialscommons.org/api to /Users/username/.materialscommons/config.json

import materials_commons.api as mcapi

# Project name
name = "ExampleProjectFromJupyter"

# Projct summary - short description to show in tables
summary = "Example project created via Jupyter notebook"

# Project description - describes the project, may be more detailed
description = "This project was created as an example of how to create "\
"and use Materials Commons projects from within a Jupyter notebook"

# Create a new project (or return existing one with same name)
request = mcapi.CreateProjectRequest(description=description, summary=summary)
remote_mc_proj = client.create_project(name, request)

print(str(remote_mc_proj))
print("URL:", client.base_url)
print("Project ID:", remote_mc_proj.id)
print("Project name:", remote_mc_proj.name)

<materials_commons.api.models.Project object at 0x7fe991784940>
URL: https://materialscommons.org/api
Project ID: 657
Project name: ExampleProjectFromJupyter

Cloning a project

"Cloning" a project creates a local directory which is used as a place to upload and download project files. There are three construction options:

1. Clone the project in a temporary directory (default)
2. Clone the project in a particular location or open the project if it already exists. This option makes use of the "parent_path" and "name" constructor arguments to specify where the local project directory will be constructed if it doesn't already exist.
3. Open an existing local cloned project. This option uses the "path" constructor argument to It can be at a particular location that a user chooses to reuse (by constructing with the

import os
import pathlib
import shutil
from materials_commons.cli.cloned_project import ClonedProject

Example 1: Clone the project - using a temporary directory

This example clones the project using a temporary directory. Downloaded files will be eventually be cleaned up by the system when the ClonedObject instance is no longer in use.

cloned_mc_proj = ClonedProject(email=email, mcurl=mcurl, proj_id=remote_mc_proj.id)

print(str(cloned_mc_proj))
print("Cloned project local path:", cloned_mc_proj.local_path)

<materials_commons.cli.cloned_project.ClonedProject object at 0x7fe991a91940>
Cloned project local path: /var/folders/v7/lq3_7hgj7n95d1xrct505ryh0000gp/T/tmpbq88nimt/ExampleProjectFromJupyter

Example 2: Clone the project - specifying the location

This example clones the project to ~/mc_projects/ExampleProjectFromJupyter.

parent_path = pathlib.Path.home() / "mc_projects"
os.makedirs(parent_path, exist_ok=True)

cloned_mc_proj = ClonedProject(email=email,
                               mcurl=mcurl,
                               proj_id=remote_mc_proj.id,
                               parent_path=parent_path, # must exist
                               name=None) # default uses project name

print(str(cloned_mc_proj))
print("Cloned project local path:", cloned_mc_proj.local_path)

<materials_commons.cli.cloned_project.ClonedProject object at 0x7fe991aa0ca0>
Cloned project local path: /Users/username/mc_projects/ExampleProjectFromJupyter

Example 3: Open an existing cloned project

This example opens a local project that has already been cloned.

cloned_mc_proj = ClonedProject(email=email,
                               mcurl=mcurl,
                               proj_id=remote_mc_proj.id,
                               path=pathlib.Path.home() / "mc_projects" / "ExampleProjectFromJupyter")

print(str(cloned_mc_proj))
print("Cloned project local path:", cloned_mc_proj.local_path)

<materials_commons.cli.cloned_project.ClonedProject object at 0x7fe991aa0310>
Cloned project local path: /Users/username/mc_projects/ExampleProjectFromJupyter

Using the ClonedProject

The ClonedProject instance provides access to Client and Project objects from the Materials Commons API (materials_commons.api) along with the location of the cloned local project directory local_path.

print(str(cloned_mc_proj.proj))
print(str(cloned_mc_proj.proj.remote))
print(type(cloned_mc_proj.local_path), str(cloned_mc_proj.local_path))

<materials_commons.api.models.Project object at 0x7fe991aa0d60>
<materials_commons.api.client.Client object at 0x7fe991aa0bb0>
<class 'pathlib.PosixPath'> /Users/username/mc_projects/ExampleProjectFromJupyter

File transfer

The ClonedProject instance from the CLI also provides methods for uploading and downloading files using features beyond those included in materials_commons.api.

For example, transfers can include checks to skip transferring files that are equivalent, support recursive upload and download, globus transfer, etc.

Other methods implemented by the CLI will be added to ClonedProject in the future.

Setup for upload examples:

This creates a directory and writes some files used in the upload examples.

example_file1 = cloned_mc_proj.local_path / "example_file1.txt"
with open(example_file1, 'w') as f:
    f.write("Hello World!\n")

example_file2 = cloned_mc_proj.local_path / "example_file2.txt"
with open(example_file2, 'w') as f:
    f.write("Hello World, again!\n")

example_dir = cloned_mc_proj.local_path / "dir"
os.makedirs(example_dir, exist_ok=True)

example_file3 = example_dir / "example_file3.txt"
with open(example_file3, 'w') as f:
    f.write("Got some data here!\n")

example_file4 = example_dir / "example_file4.txt"
with open(example_file4, 'w') as f:
    f.write("So much data!\n")

Upload one file

cloned_mc_proj.upload(example_file1)

uploaded: example_file1.txt

By default, files that already exist will be skipped

• Use no_compare=True to transfer without comparing checksums. Materials Commons will still check for file equivalence and only create a new file version if the file is different.

cloned_mc_proj.upload(example_file1)
cloned_mc_proj.upload(example_file1, no_compare=True)

example_file1.txt: local is equivalent to remote (skipping)
uploaded: example_file1.txt

Upload multiple files

cloned_mc_proj.upload(example_file1, example_file2)

example_file1.txt: local is equivalent to remote (skipping)
uploaded: example_file2.txt

Upload files and directories, recursively

• Use the recursive=True argument to transfer files and directories recursively

cloned_mc_proj.upload(example_dir, recursive=True)

uploaded: dir/example_file3.txt
uploaded: dir/example_file4.txt

Uploading the notebook itself / use of "upload_as"

It is possible to upload a notebook, from within the notebook itself. To do so, we can use the "upload_as" option which allows uploading files that do not exist in the local cloned project directory. The following cells demonstrate getting the notebook's name, nb_name, and then uploading the notebook itself to the Materials Commons project. It is placed in a "notebooks" directory. Note that it uploads the last saved version of the notebook, not the current state.

Note: Getting the notebook file path from `os.path.join(os.getcwd(), nb_name)` may not work in all cases

nb_name = "MaterialsCommons-Project-Example.ipynb"
notebook_local_abspath = os.path.join(os.getcwd(), nb_name)
notebook_upload_as = cloned_mc_proj.local_path / "notebooks" / nb_name

cloned_mc_proj.upload(notebook_local_abspath, upload_as=notebook_upload_as)

uploaded: ../../Work/codes/mccli/doc/source/assets/examples/MaterialsCommons-Project-Example.ipynb as notebooks/MaterialsCommons-Project-Example.ipynb

Setup for download examples:

This removes the existing local files and directories to demonstrate downloading from Materials Commons.

for file in [example_file1, example_file2]:
    if os.path.exists(file):
        os.remove(file)

if os.path.exists(example_dir):
    shutil.rmtree(example_dir)

print("Local project directory contents:", os.listdir(cloned_mc_proj.local_path))

Local project directory contents: ['.mc']

Download one file

cloned_mc_proj.download(example_file1)

downloaded: example_file1.txt

By default, files that already exist will be skipped

• Use no_compare=True to transfer without comparing checksums.
• Use force=True to force overwriting existing files without prompting.

cloned_mc_proj.download(example_file1, no_compare=True)

cloned_mc_proj.download(example_file2)
shutil.copyfile(example_file2, example_file1)
cloned_mc_proj.download(example_file1, force=True)

Overwrite 'example_file1.txt'?
y/n: y
downloaded: example_file1.txt
downloaded: example_file2.txt
downloaded: example_file1.txt

Download multiple files

cloned_mc_proj.download(example_file1, example_file2, force=True)

example_file1.txt: local is equivalent to remote (skipping)
example_file2.txt: local is equivalent to remote (skipping)

Download files and directories, recursively

• Use the recursive=True argument to transfer files and directories recursively

cloned_mc_proj.download(example_dir, recursive=True)

downloaded: dir/example_file3.txt
downloaded: dir/example_file4.txt

Download with different name

• Use the output argument to output one file or directory to a different location

cloned_mc_proj.download(example_file1, output=cloned_mc_proj.local_path / "example_file3.txt")

downloaded: example_file1.txt as example_file3.txt

Using Globus file transfer

• Use the globus=True argument to peform the file transfer using the current globus upload or download directory
• Use the label argument to give the transfer a label for easier tracking
• Globus configuration and transfer management is not currently supported via ClonedProject but it can be done using shell commands or the Materials Commons website
• See the online documentation for more information on Globus file transfers.

cloned_mc_proj.upload(example_file1, globus=True)

Created new globus upload (name=cavern-jaob-tryst, id=385).
Globus task_id: 70e9548c-b298-11eb-a161-5fad80e6400b
Globus transfer task initiated.
Use `globus task list` to monitor task status.
Use `mc globus upload` to manage Globus uploads.
Multiple transfer tasks may be initiated.
When all tasks finish uploading, use `mc globus upload --id 385 --finish` to import all uploaded files into the Materials Commons project.

cloned_mc_proj.download(example_file2, globus=True, force=True)

Created new globus download (name=oxbane-ureido-rifter, id=386).
Checking if download is ready..
Download is ready.
task_id = 7597269e-b298-11eb-a161-5fad80e6400b
Globus transfer task initiated.
Use `globus task list` to monitor task status.
Use `mc globus download` to manage Globus downloads.
Multiple transfer tasks may be initiated.
When all tasks finish downloading, use `mc globus download --id 386 --delete` to close the download.

Monitor transfer status

! cd {cloned_mc_proj.local_path} && mc globus upload && mc globus download
! globus task list

    project_name                 project_id  type    name                 id  created               status
--  -------------------------  ------------  ------  -----------------  ----  --------------------  --------
*   ExampleProjectFromJupyter           657  upload  cavern-jaob-tryst   385  2021 May 11 20:35:33  Ready

    project_name                 project_id  type      name                    id  created               status
--  -------------------------  ------------  --------  --------------------  ----  --------------------  --------
*   ExampleProjectFromJupyter           657  download  oxbane-ureido-rifter   386  2021 May 11 20:35:40  Ready

Task ID                              | Status    | Type     | Source Display Name        | Dest Display Name          | Label
------------------------------------ | --------- | -------- | -------------------------- | -------------------------- | ----------------------------------------------
7597269e-b298-11eb-a161-5fad80e6400b | SUCCEEDED | TRANSFER | materials-commons-2-upload | user_computer        | ExampleProjectFromJupyter-oxbane-ureido-rifter
70e9548c-b298-11eb-a161-5fad80e6400b | SUCCEEDED | TRANSFER | user_computer        | materials-commons-2-upload | ExampleProjectFromJupyter-cavern-jaob-tryst

Finish the transfer

• Uploads must be "finished" to transfer files into the project. Once processing is finished, the upload directory will no longer appear in mc globus upload results, and all files should appear in the project directory. The processing time required before files appear in your project will depend on the size of the transfer.
• Globus download directories should be "deleted" when download tasks are finished. The download directory may be left as long as desired, but it will not reflect any file or directory changes to the project since the time the download directory was created.

from materials_commons.cli.functions import read_project_config

project_config = read_project_config(cloned_mc_proj.local_path)

! cd {cloned_mc_proj.local_path} && mc globus upload --id {project_config.globus_upload_id} --finish --force

Finishing current Globus upload

! cd {cloned_mc_proj.local_path} && mc globus download --id {project_config.globus_download_id} --delete --force

    project_name                 project_id  type      name                    id  created               status
--  -------------------------  ------------  --------  --------------------  ----  --------------------  --------
*   ExampleProjectFromJupyter           657  download  oxbane-ureido-rifter   386  2021 May 11 20:35:40  Ready

Permanently deleting with --force...
Deleted current Globus download

Example cleanup

The delete_project call will delete a project on Materials Commons.

Notes:

• Only the project owner can delete a project
• A project that has published datasets may not be deleted
• Be careful, there is no undo! Deleting a project deletes all project files and data.
• Deleting the remote project does not delete the local project files.
• Deleting the local project files does not delete the remote project

# Delete the remote project
projs = client.get_all_projects()
for proj in projs:
    if proj.name == "ExampleProjectFromJupyter":
        client.delete_project(proj.id)

# Delete the local project
local_project_path = pathlib.Path.home() / "mc_projects" / "ExampleProjectFromJupyter"
if os.path.exists(local_project_path):
    shutil.rmtree(local_project_path)