Flyhweel Software Developer Kit (SDK) Basics and Helpful Hints
Date modified: 28-July-2022 Authors: Amy Hegarty, Lena Sherbakov, Intermountain Neuroimaging Consortium
Description: The following includes a set of basic SDK commands for Flywheel. Examples included below:
Basic Flywheel Heirarchy (finding a session)
Pulling Data To Scatch workspace
Uploading Analyses to Flywheel
Links
[3]:
print("Welcome to Intermountain Neuroimaging Consortium!")
Welcome to Intermountain Neuroimaging Consortium!
Import Python Dependencies
[ ]:
from pathlib import Path
import sys, subprocess, os, nipype, datetime, logging
sys.path.append('/projects/ics/software/flywheel-python/bids-client/')
sys.path.append('/projects/ics/software/flywheel-python/')
from getpass import getpass
import flywheel, flywheel_gear_toolkit
from flywheel_gear_toolkit.interfaces.command_line import (
build_command_list,
exec_command,
)
from flywheel_gear_toolkit.utils.zip_tools import unzip_archive, zip_output
from utils.zip_htmls import zip_htmls
from flywheel_bids.export_bids import export_bids
from flywheel_bids.export_bids import download_bids_dir
import subprocess
from datetime import datetime
from zipfile import ZipFile
Define Supporting Functions
[ ]:
# supporting functions
def analysis_exists(session, analysis_name, exclude_list=[]):
# Returns True if analysis already exists with a running or complete status, else false
# make sure to pass full session object (use fw.get_session(session.id))
#
#Get all analyses for the session
flag=False
for analysis in session.analyses:
#only print ones that match the analysis label
if analysis_name in analysis.label:
#filter for only successful job
analysis_job=analysis.job
if any(analysis_job.state in string for string in ["complete","running","pending"]):
if analysis_job.failure_reason is None:
flag=True
#check if session is in exclude list
if any(session.id in string for string in exclude_list):
flag=True
return flag
Flywheel API Key and Client
An API Key is required to interact with the datasets on flywheel db. More on this in the Flywheel SDK doc here. Here we pull the API key directly from the user’s os enviornment, set within the flywheel command line interface.
[ ]:
# Instantiate a logger
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s')
log = logging.getLogger('root')
[ ]:
# Create client, using CLI credentials
fw = flywheel.Client()
# who am I logged in as?
log.info('You are now logged in as %s to %s', fw.get_current_user()['email'], fw.get_config()['site']['api_url'])
Viewing a Flywheel Project
The fw.lookup function can be used to find any container in flywheel by path. This is an easy way to find projects, subjects, or sessions. Lets view a Flywheel project and list all sessions and their scan date.
[ ]:
project = fw.lookup('<group>/<project>')
project.reload()
for session in project.sessions.find():
# Loop through all sessions in the subject container.
dt = session.timestamp
if not dt:
dt = "NAN"
else:
dt = dt.strftime(" %x %X")
print('%s: Subject: %s Session: %s\tScanning Date: %s' % (session.id, session.subject.label, session.label, dt))
Viewing a Flywheel Analysis
Similarly, we can view analyses that have been either uploaded to Flywheel or generated by running flywheel gears. Here, lets look for all analyses attached to a single session.
[ ]:
# start by finding a session of interest
session = fw.lookup('<group>/<project>/<subject>/<session>')
# we need to take an extra step to get the full session object, not just basic info
session_object = fw.get_session(session.id)
# print all the analyses for this session
for analysis in session_object.analyses:
if hasattr(analysis.job,'state'):
print('%s: %s %s' % (analysis.id, analysis.label, analysis.job.state))
else:
print('%s: %s' % (analysis.id, analysis.label))
Download Data From Flywheel to Local Scratch (Part 1, Analyses)
Often times you need to use data in flywheel as inputs for group level analyses. At present this is still most easily accomplished by exporting the data from file to a scatch filesystem.
[ ]:
# path to scratch directory
username=os.getenv('USER')
scratch='/scratch/summit/'+username'/''
os.chdir(scratch)
# find the analysis you wish to download
analysis_id='629f8f5cbffdbee5eb7dcb37'
analysis=fw.get(id=analysis_id)
# Download the data to scratch
for fl in analysis.files:
fl.download(scratch+fl['name'])
# unzip files
if '.zip' in fl['name']:
zipfile = ZipFile(scratch+fl['name'], "r")
zipfile.extractall(scratch)
# Run your computations! Here lets just list the contents of the directory and store that in a text file
os.system('mkdir -p output ; ls -l '+scratch+' > output/out.log')
Upload Results to Flywheel Analysis (Part 1, Session Analyses)
[ ]:
# Once you have finished your analysis, we need to upload your files and store some critical metadata in flywheel
# session add analysis, zip outputs, upload
# ...#zip output except for scratch
log.info('Zipping contents of directory %s', scratch+'output')
zip_output(scratch,'output', "output.zip", exclude_files=['scratch'])
#create an analysis for that session
timestamp=os.path.getmtime(scratch+'output')
dtobject = datetime.fromtimestamp(timestamp)
analysis = session_object.add_analysis(label='Sample Upload '+dtobject.strftime(" %x %X"))
#upload analysis files
analysis.upload_output(scratch+'output.zip')
Upload Results to Flywheel Analysis (Part 2, Project Analyses)
[ ]:
# same thing for project
#create an analysis for that project
timestamp=os.path.getmtime(scratch+'output')
dtobject = datetime.fromtimestamp(timestamp)
analysis = project.add_analysis(label='Sample Upload '+dtobject.strftime(" %x %X"))
#upload analysis files
analysis.upload_output(scratch+'output.zip')
Some Examples Looping Through All Sessions
Here is some example code where we loop through all sessions to check if an analysis exists. We will use a function we defined at the top of the notebook.
[ ]:
for session in project.sessions.find():
full_session = fw.get_session(session.id)
flag = analysis_exists(full_session, "hcp")
print('%s: %s %s' % (session.subject.label, session.label, flag))