Upload My Analysis to Flywheel

Date: 26-Sept-2022

Description:

This notebook provides a walk through to upload analyses generated on a local filesystem to Flywheel. Some neuroimaging analyses and workflows are not yet supported as Flywheel gears. For these workflows, the current workaround is to download your Flywheel analyses (or workflow inputs) run the workflow on your local machine, then upload the restuls (workflow outputs) back to flywheel as a new analysis.
It should be possible to run this notebook in any jupyter-compatible thrid party-platforms such as google collab or mybinder.org.

Requirements

University of Colorado at Boulder Research Computing (CURC) account
Access to University of Colorado Flywheel Instance

The following workbook should be run on CURC Blanca Compute. If you are unsure you have the correct permission or access to these resources please contact INC Data and Analysis team: Amy Hegarty [amy.hegarty@colorado.edu] or Lena Sherbakov [lena.sherbakov@colorado.edu].

CURC Jupyterhub

Before launching this jupyter notebook, users should launch this session using Open OnDemand.

TIP: Follow the instructions on INC Documentation to get started with Jupyter Notebooks.

We will be working on a large scratch system mounted only on Blanca compute nodes in this tutorial. If you do not have access to this filesystem you should select a similar large capacity scratch enviornment for analysis.

Setup

TIP: Please use the “flywheel” kernel for this tutorial. If you do not see a “flywheel” kernel, contact INC Data and Analysis team to install this environment.

[1]:

print("Welcome to Intermountain Neuroimaging Consortium!")

Welcome to Intermountain Neuroimaging Consortium!

[ ]:

# Python standard package come first
import logging
import os, platform, sys
from zipfile import ZipFile
from datetime import datetime

# Third party packages come second
import flywheel

# add software paths
sys.path.append('/projects/ics/software/flywheel-python/bids-client/')
sys.path.append('/projects/ics/software/flywheel-python/')

Lets intialize a logger to keep track of the progress of our job (e.g. useful to keep track of runtime).

[ ]:

# Instantiate a logger
logging.basicConfig(level=logging.INFO)
log = logging.getLogger('root')

Lets check we are on the correct computing system.

[ ]:

host = os.getenv('HOSTNAME', os.getenv('COMPUTERNAME', platform.node())).split('.')[0]

if "bnode" not in host:
    log.error("Tutorial should be run on CURC high performance compute nodes: blanca")

Flywheel API Key and Client

You can get you API_KEY by following the steps described in the Flywheel SDK doc here.

DANGER: Do NOT share your API key with anyone for any reason - it is the same as sharing your password and may break human subject participant confidentiality. ALWAYS obscure credentials from your code, especially when sharing with others/commiting to a shared repository.

[ ]:

API_KEY = getpass('Enter API_KEY here: ')

Instantiate the Flywheel API client either using the API_KEY provided by the user input above or by reading it from the environment variable FW_KEY.

[ ]:

fw = flywheel.Client(API_KEY if 'API_KEY' in locals() else os.environ.get('FW_KEY'))

You can check which Flywheel instance you have been authenticated against with the following:

[ ]:

log.info('You are now logged in as %s to %s', fw.get_current_user()['email'], fw.get_config()['site']['api_url'])

Constants

Often you will have to define a few constants in your notebook which serve as the inputs. Such constant for instance is the API_KEY that was used to instantiate the Flywheel client. Other examples could be a PROJECT_ID or PROJECT_LABEL that will be used to identify a specific project.

[ ]:

PROJECT_LABEL = 'MyProject'

Helper functions

Here are all the custom helper functions we have developed for use in this example.

[ ]:

def get_project_id(fw, project_label):
    """Return the first project ID matching project_label

    Args:
       fw (flywheel.Client): A flywheel client
       project_label (str):  A Project label

    Returns:
       (str): Project ID or None if no project found
    """
    project = fw.projects.find_first(f'label={project_label}')
    if project:
        return project.id
    else:
        return None

Upload My Analysis

First, lets point to a project in Flywheel.

[ ]:

project_id = get_project_id(fw, PROJECT_LABEL)
if project_id:
    print(f'Project ID is: {project_id}.')
else:
    print(f'No Project with label {PROJECT_LABEL} found.')

Next we need to zip the contents of our analysis then upload those zipped directories to a flywheel analysis.

[ ]:

project = fw.get_project(project_id)

# create new project level analysis
analysis = project.add_analysis(label='CONN Analysis: ' + datetime.now(" %x %X"))

# zip outputs...
os.system('zip -R conn_analysis conn_analysis/ conn_analysis.mat')

# upload output zipped directories
 analysis.upload_output('conn_analysis.zip')

if os.path.exists('conn_inputs.zip'):
    os.system('zip -R conn_inputs conn_inputs/ ')
    analysis.upload_output('conn_inputs.zip')

Finally, you can open Flywheel GUI to check your analysis was sucessfully uploaded.