{ "cells": [ { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "# Download and Run Analysis Locally\n", "**Date**: 26-Sept-2022 \n", "**Description**: \n", "\n", " - This notebook provides a walk through to download analyses stored in Flywheel to a local file system. Some neuroimaging analyses and workflows are not yet supported as Flywheel gears. For these workflows, the current workaround is to download your Flywheel analyses (or workflow inputs) run the workflow on your local machine, then upload the restuls (workflow outputs) back to flywheel as a new analysis.\n", " - In this example, we will be downloading data into a CURC scratch filesystem and run a group CONN analysis on our high performance compute.\n", " - It should be possible to run this notebook in any jupyter-compatible thrid party-platforms such as [google collab](https://colab.research.google.com/) or [mybinder.org](https://mybinder.org/)." ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## Requirements\n", " - University of Colorado at Boulder Research Computing (CURC) account\n", " - Access to University of Colorado Flywheel Instance\n", "\n", "The following workbook should be run on CURC Blanca Compute. If you are unsure you have the correct permission or access to these resources please contact INC Data and Analysis team: Amy Hegarty [amy.hegarty@colorado.edu] or Lena Sherbakov [lena.sherbakov@colorado.edu]." ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "### CURC Jupyterhub \n", "Before launching this jupyter notebook, users should launch this session using Open OnDemand.\n", "\n", "
\n", "TIP: Follow the instructions on INC Documentation to get started with Jupyter Notebooks.\n", "
\n", "\n", "We will be working on a large scratch system mounted only on Blanca compute nodes in this tutorial. If you do not have access to this filesystem you should select a similar large capacity scratch enviornment for analysis.\n" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## Setup\n", "\n", "
\n", "TIP: Please use the \"flywheel\" kernel for this tutorial. If you do not see a \"flywheel\" kernel, contact INC Data and Analysis team to install this environment.
" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Welcome to Intermountain Neuroimaging Consortium!\n" ] } ], "source": [ "print(\"Welcome to Intermountain Neuroimaging Consortium!\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# Python standard package come first\n", "import logging\n", "import os, platform, sys\n", "from zipfile import ZipFile\n", "\n", "# Third party packages come second\n", "import flywheel\n" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Lets initialize a logger to keep track of the progress of our job (e.g. useful to keep track of runtime). " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# Instantiate a logger\n", "logging.basicConfig(level=logging.INFO)\n", "log = logging.getLogger('root')" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Lets check we are on the correct computing system. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "host = os.getenv('HOSTNAME', os.getenv('COMPUTERNAME', platform.node())).split('.')[0]\n", "\n", "if \"bnode\" not in host:\n", " log.error(\"Tutorial should be run on CURC high performance compute nodes: blanca\")\n", " \n" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## Flywheel API Key and Client" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "You can get you API_KEY by following the steps described in the Flywheel SDK doc [here](https://flywheel-io.gitlab.io/product/backend/sdk/branches/master/python/getting_started.html#api-key)." ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "
\n", "DANGER: \n", " Do NOT share your API key with anyone for any reason - it is the same as sharing your password and may break human subject participant confidentiality. ALWAYS obscure credentials from your code, especially when sharing with others/commiting to a shared repository.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "API_KEY = getpass('Enter API_KEY here: ')" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Instantiate the Flywheel API client either using the API_KEY provided by the user input above or by reading it from the environment variable `FW_KEY`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "fw = flywheel.Client(API_KEY if 'API_KEY' in locals() else os.environ.get('FW_KEY'))" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "You can check which Flywheel instance you have been authenticated against with the following:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "log.info('You are now logged in as %s to %s', fw.get_current_user()['email'], fw.get_config()['site']['api_url'])" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## Constants" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Often you will have to define a few constants in your notebook which serve as the inputs. Such constant for instance is the API_KEY that was used to instantiate the Flywheel client. Other examples could be a PROJECT_ID or PROJECT_LABEL that will be used to identify a specific project." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "PROJECT_LABEL = 'MyProject'" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## Helper functions" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Here are all the custom helper functions we have developed for use in this example." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "def get_project_id(fw, project_label):\n", " \"\"\"Return the first project ID matching project_label\n", " \n", " Args:\n", " fw (flywheel.Client): A flywheel client\n", " project_label (str): A Project label\n", " \n", " Returns:\n", " (str): Project ID or None if no project found\n", " \"\"\"\n", " project = fw.projects.find_first(f'label={project_label}')\n", " if project:\n", " return project.id\n", " else:\n", " return None" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## Main script" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "We will be using the Flywheel SDK to identify and reterive specific analysis files stored in Flywheel for download. Importantly, since the original analysis files are still retained in Flywheel we can use our local copy of the data as a temporary or scratch workspace and remove all files at the end of this workflow." ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "First, lets point to a project in Flywheel." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "project_id = get_project_id(fw, PROJECT_LABEL)\n", "if project_id:\n", " print(f'Project ID is: {project_id}.')\n", "else:\n", " print(f'No Project with label {PROJECT_LABEL} found.')" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Lets start by getting some information about the analyses in our flywheel project. We will loop through all the sessions in our desired project, and log the number of complete, failed, and cancelled jobs. This structure will be the same that we will use when downloading the list of analyses next." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# get project object\n", "project = fw.get_project(project_id)\n", "\n", "gear_name = 'bids-fmriprep'\n", "\n", "icomplete = 0\n", "ifailed = 0\n", "icancelled = 0\n", "isessions = 0\n", "\n", "# loop through all sessions in the project. More detailed filters could be \n", "# used to specify a subset of sessions\n", "for session in project.sessions.find():\n", "\n", " full_session = fw.get_session(session.id)\n", " isessions += 1\n", " for analysis in full_session.analyses:\n", " \n", " analysis_job=analysis.job\n", " \n", " #only print ones that match the analysis label\n", " if gear_name in analysis.label:\n", " if any(analysis_job.state in string for string in [\"complete\"]):\n", " icomplete += 1\n", "\n", " elif any(analysis_job.state in string for string in [\"failed\"]):\n", " ifailed += 1\n", " log.info(\"subject: %s session: %s %s job: %s %s\", session.subject.label, session.label, session.id, analysis.id, analysis_job.state)\n", "\n", " elif any(analysis_job.state in string for string in [\"cancelled\"]):\n", " icancelled += 1\n", " log.info(\"subject: %s session: %s %s job: %s %s\", session.subject.label, session.label, session.id, analysis.id, analysis_job.state)\n", "\n", "log.info('%s Sessions, gear %s: %s complete, %s failed, %s cancelled', str(isessions),gear_name,str(icomplete), str(ifailed), str(icancelled)) \n" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Lets point to a file location on our local machine (in this case Blanca Compute) to store the analyses locally.\n", "\n", "
\n", "TIP: Point to a large scratch filesystem for fast read and write operations.
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# path to scratch directory\n", "username=os.getenv('USER')\n", "scratch='/scratch/alpine/'+username+'/'\n", "os.chdir(scratch)" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## Download Analyses to Local Filesystem\n", "Next we are going download analyses of interest. In this example, we will download all fmriprep output directories. There is plenty of customization you can use here to be sure you are downloading only the sessions and analyses of interest. Check out some filtering examples in our tutorial here .\n", "\n", "
\n", "WARNING: This will take some time!\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# loop through all sessions in the project. More detailed filters could be \n", "# used to specify a subset of sessions\n", "for session in project.sessions.find():\n", "\n", " full_session = fw.get_session(session.id)\n", " isessions += 1\n", " for analysis in full_session.analyses:\n", " \n", " analysis_job=analysis.job\n", " \n", " #only download ones that match the analysis label\n", " if gear_name in analysis.label:\n", " if any(analysis_job.state in string for string in [\"complete\"]):\n", " \n", " # Download the data to scratch\n", " for fl in analysis.files:\n", " fl.download(scratch+fl['name'])\n", "\n", " # unzip files\n", " if '.zip' in fl['name']:\n", " os.system('unzip -d '+scratch+\" \"+scratch+fl['name'])\n", " \n", " log.info('Downloaded analysis: %s for Subject: %s Session: %s', analysis.label,session.subject.label, session.label) \n", "\n" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Now that your data is stored in local scratch, its time to run your analysis as you always would, for example... CONN analysis. To get started with your CONN analysis, you will need to return to Open OnDemand and start a new core desktop session. For more information on how to do this, please visit our [documentation](https://inc-documentation.readthedocs.io/en/dev/pl_and_blanca_basics.html#high-performance-compute-portal)\n" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "After you are happy with your analysis don't forget to upload your analysis results to flywheel. Follow the instructions in this tutorial to get started: [upload-my-analysis.ipynb](upload-my-analysis.ipynb)" ] } ], "metadata": { "kernelspec": { "display_name": "Python (myenv)", "language": "python", "name": "myenv" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.12" } }, "nbformat": 4, "nbformat_minor": 4 }