Create a new Materials Commons project

This example demonstrates creating a new Materials Commons project from a Jupyter notebook. To try running it locally, download the notebook from here.

To install the necessary dependencies (requires Python 3):

pip install materials-commons-cli

For this notebook, you must configure mc beforehand:

 mc remote --add user@domain.edu https://materialscommons.org/api
 mc remote --set-default user@domain.edu https://materialscommons.org/api

Notes:

• If you do not yet have a Materials Commons account, it can be created here.
• Only one project per owner may have the same name. If the same name as an existing project is given, then the existing project is returned instead of creating a new project.

import os
import pathlib
import shutil

parent_path = pathlib.Path.home() / "mc_projects"
os.makedirs(parent_path, exist_ok=True)

# Project name
name = "ExampleProjectFromJupyter"
project_path = parent_path / name

# Projct summary - short description to show in tables
summary = "Example project created via Jupyter notebook"

# Project description - describes the project, may be more detailed
description = "This project was created as an example of how to create "\
"and use Materials Commons projects from within a Jupyter notebook"

! cd {parent_path} && mkdir {name}
! cd {project_path} && mc init --desc "{description}"

Created new project on: https://materialscommons.org/api
    name                         owner    id  modified_at
--  -------------------------  -------  ----  --------------------
*   ExampleProjectFromJupyter       65   660  2021 May 12 03:50:31

File Transfer

These examples demonstrate the mixed use of Python with shell commands (! ...) to make use of the Materials Commons CLI in a Jupyter notebook.

Setup for upload examples:

This creates a directory and writes some files used in the upload examples.

example_file1 = project_path / "example_file1.txt"
with open(example_file1, 'w') as f:
    f.write("Hello World!\n")

example_file2 = project_path / "example_file2.txt"
with open(example_file2, 'w') as f:
    f.write("Hello World, again!\n")

example_dir = project_path / "dir"
os.makedirs(example_dir, exist_ok=True)

example_file3 = example_dir / "example_file3.txt"
with open(example_file3, 'w') as f:
    f.write("Got some data here!\n")

example_file4 = example_dir / "example_file4.txt"
with open(example_file4, 'w') as f:
    f.write("So much data!\n")

Upload one file

! cd {project_path} && mc up example_file1.txt

uploaded: example_file1.txt

By default, files that already exist will be skipped

• Use --no-compare to transfer without checking. Materials Commons will still check for file equivalence and only create a new file version if the file is different.

! cd {project_path} && mc up example_file1.txt
! cd {project_path} && mc up --no-compare example_file1.txt

example_file1.txt: local is equivalent to remote (skipping)
uploaded: example_file1.txt

Upload multiple files

! cd {project_path} && mc up example_file1.txt example_file2.txt

example_file1.txt: local is equivalent to remote (skipping)
uploaded: example_file2.txt

Upload files and directories, recursively

• Use the -r,--recursive argument to transfer files and directories recursively

! cd {project_path} && mc up -r dir

uploaded: dir/example_file3.txt
uploaded: dir/example_file4.txt

Uploading the notebook itself / use of "upload_as"

It is possible to upload a notebook, from within the notebook itself. To do so, we can use the --upload-as option which allows uploading files that do not exist in the local cloned project directory. The following cells demonstrate getting the notebook's name, nb_name, and then uploading the notebook itself to the Materials Commons project. It is placed in a "notebooks" directory. Note that it uploads the last saved version of the notebook, not the current state.

_Note: Getting the notebook file path from os.path.join(os.getcwd(), nb_name) may not work in all cases_

nb_name = "MaterialsCommons-Project-Shell-Example.ipynb"
notebook_local_abspath = os.path.join(os.getcwd(), nb_name)
notebook_upload_as = project_path / "notebooks" / nb_name

! cd {project_path} && mc up {notebook_local_abspath} --upload-as {notebook_upload_as}

uploaded: ../../Work/codes/mccli/doc/source/assets/examples/MaterialsCommons-Project-Shell-Example.ipynb as notebooks/MaterialsCommons-Project-Shell-Example.ipynb

Setup for download examples

This removes the existing local files and directories to demonstrate downloading from Materials Commons.

for file in [example_file1, example_file2]:
    if os.path.exists(file):
        os.remove(file)

if os.path.exists(example_dir):
    shutil.rmtree(example_dir)

print("Local project directory contents:", os.listdir(project_path))

Local project directory contents: ['.mc']

Download one file

! cd {project_path} && mc down example_file1.txt

downloaded: example_file1.txt

By default, files that already exist will be skipped

• Use --no-compare to transfer without comparing checksums.
• Use -f,--force to force overwriting existing files without prompting.

! cd {project_path} && mc down --no-compare --force example_file1.txt
! cd {project_path} && mc down --no-compare --force example_file2.txt

! cd {project_path} && cp example_file2.txt example_file1.txt
! cd {project_path} && mc down --force example_file1.txt

downloaded: example_file1.txt
downloaded: example_file2.txt
downloaded: example_file1.txt

Download multiple files

! cd {project_path} && mc down example_file1.txt example_file2.txt

example_file1.txt: local is equivalent to remote (skipping)
example_file2.txt: local is equivalent to remote (skipping)

Download files and directories, recursively

• Use the -r,--recursive argument to transfer files and directories recursively

! cd {project_path} && mc down -r dir

downloaded: dir/example_file3.txt
downloaded: dir/example_file4.txt

Download with different name

• Use -o,--output to output one file or directory to a different location

! cd {project_path} && mc down example_file1.txt -o example_file3.txt

downloaded: example_file1.txt as example_file3.txt

Using Globus file transfer

• Use the --globus argument to peform the file transfer using the current globus upload or download directory
• Use the --label argument to give the transfer a label for easier tracking
• Globus configuration and transfer management can be done using shell commands or the Materials Commons website
• See the online documentation for more information on Globus file transfers.

! cd {project_path} && mc up --globus example_file1.txt

Created new globus upload (name=knoppy-ref-tympan, id=391).
Globus task_id: 6f4b61b4-b2d5-11eb-afd1-e1e7a67e00c1
Globus transfer task initiated.
Use `globus task list` to monitor task status.
Use `mc globus upload` to manage Globus uploads.
Multiple transfer tasks may be initiated.
When all tasks finish uploading, use `mc globus upload --id 391 --finish` to import all uploaded files into the Materials Commons project.

! cd {project_path} && mc down --globus --force example_file2.txt

Created new globus download (name=menial-tahil-per, id=392).
Checking if download is ready..
Download is ready.
task_id = 72b2ab5a-b2d5-11eb-a161-5fad80e6400b
Globus transfer task initiated.
Use `globus task list` to monitor task status.
Use `mc globus download` to manage Globus downloads.
Multiple transfer tasks may be initiated.
When all tasks finish downloading, use `mc globus download --id 392 --delete` to close the download.

Monitor transfer status

! cd {project_path} && mc globus upload && mc globus download
! globus task list

    project_name                 project_id  type    name                 id  created               status
--  -------------------------  ------------  ------  -----------------  ----  --------------------  --------
*   ExampleProjectFromJupyter           660  upload  knoppy-ref-tympan   391  2021 May 12 03:52:10  Ready

    project_name                 project_id  type      name                id  created               status
--  -------------------------  ------------  --------  ----------------  ----  --------------------  --------
*   ExampleProjectFromJupyter           660  download  menial-tahil-per   392  2021 May 12 03:52:14  Ready

Task ID                              | Status    | Type     | Source Display Name        | Dest Display Name          | Label
------------------------------------ | --------- | -------- | -------------------------- | -------------------------- | ----------------------------------------------
72b2ab5a-b2d5-11eb-a161-5fad80e6400b | SUCCEEDED | TRANSFER | materials-commons-2-upload | user_computer        | ExampleProjectFromJupyter-menial-tahil-per
6f4b61b4-b2d5-11eb-afd1-e1e7a67e00c1 | SUCCEEDED | TRANSFER | user_computer        | materials-commons-2-upload | ExampleProjectFromJupyter-knoppy-ref-tympan

Finish the transfer

• Uploads must be "finished" to transfer files into the project. Once processing is finished, the upload directory will no longer appear in mc globus upload results, and all files should appear in the project directory. The processing time required before files appear in your project will depend on the size of the transfer.
• Globus download directories should be "deleted" when download tasks are finished. The download directory may be left as long as desired, but it will not reflect any file or directory changes to the project since the time the download directory was created.

from materials_commons.cli.functions import read_project_config

project_config = read_project_config(project_path)

! cd {project_path} && mc globus upload --id {project_config.globus_upload_id} --finish --force

Finishing current Globus upload

! cd {project_path} && mc globus download --id {project_config.globus_download_id} --delete --force

    project_name                 project_id  type      name                id  created               status
--  -------------------------  ------------  --------  ----------------  ----  --------------------  --------
*   ExampleProjectFromJupyter           660  download  menial-tahil-per   392  2021 May 12 03:52:14  Ready

Permanently deleting with --force...
Deleted current Globus download

Example cleanup

The delete_project call will delete a project on Materials Commons.

Notes:

• Only the project owner can delete a project
• A project that has published datasets may not be deleted
• Be careful, there is no undo! Deleting a project deletes all project files and data.
• Deleting the remote project does not delete the local project files.
• Deleting the local project files does not delete the remote project

# Delete the remote project (-f, force delete is required to avoid prompt)
print("Deleting remote project...")
! mc proj --delete -f --id {project_config.project_id}

# Delete the local project
print("Deleting local project...")
if os.path.exists(project_path):
    shutil.rmtree(project_path)

Deleting remote project...
    name                       owner                 id  updated_at
--  -------------------------  ------------------  ----  --------------------
    ExampleProjectFromJupyter  user@domain.edu      660  2021 May 12 03:50:59

Permanently deleting with --force...
Deleted project: ExampleProjectFromJupyter 660
Note that this only deletes the project remotely and does not delete any local files.
Deleting local project...