tutorials/General_move_data_to_from_Object_Storage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# How to move data to/from Object Storage.\n",
    "\n",
    "This tutorial shows you how to use the `python-swiftclient` to move data to/from your Object Storge account. \n",
    "\n",
    "This will be useful in a variety of ways. \n",
    "\n",
    "If you're at the hackathon and running on a PowerAI system, using the `swift` client will be the best way to move data from your Nimbix cloud machine to an IBM Object Storage account.  \n",
    "\n",
    "**Important for hackathon participants using the PowerAI systems. When those machines are shut down, all data in your local user space will be lost. So, be sure to save your work somewhere!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#!pip install --user --upgrade python-keystoneclient\n",
    "#!pip install --user --upgrade python-swiftclient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Find Your Object Storage Credentials\n",
    "\n",
    "If you have an IBM Object Storage account, then you probably have either signed up with IBM Bluemix or IBM Data Science Experience (DSX). If you signed up with DSX, then a Bluemix account was created for you automatically. \n",
    "\n",
    "Instructions for finding your Object Storage credentials [are found here](https://github.com/ibm-cds-labs/seti_at_ibm#object-storage-credentials) and also on the DSX Docs page.\n",
    "\n",
    "**NB: If you follow the instructions that take you through your IBM Bluemix account, when you land on the Object Storage dashboard, you'll be able to create new containers from that interface. You might want to create different containers to hold different kinds of data.** IBM Object Storage comes with 5 GB of free storage. \n",
    "\n",
    "**NB 2: From the Object Storage Dashboard in both DSX and Bluemix, you can download any file there through your web browser. Thus, if you move a .csv file to your Object Storage from Spark, you can download that .csv file to your local machine through the dashboard.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "credentials = {\n",
    "  'auth_uri':'',\n",
    "  'global_account_auth_uri':'',\n",
    "  'username':'xx',\n",
    "  'password':\"xx\",\n",
    "  'auth_url':'https://identity.open.softlayer.com',\n",
    "  'project':'xx',\n",
    "  'project_id':'xx',\n",
    "  'region':'dallas',\n",
    "  'user_id':'xx',\n",
    "  'domain_id':'xx',\n",
    "  'domain_name':'xx',\n",
    "  'tenantId':'xx'\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import swiftclient.client as swiftclient\n",
    "\n",
    "conn = swiftclient.Connection(\n",
    "    key=credentials['password'],\n",
    "    authurl=credentials['auth_url']+\"/v3\",\n",
    "    auth_version='3',\n",
    "    os_options={\n",
    "        \"project_id\": credentials['project_id'],\n",
    "        \"user_id\": credentials['user_id'],\n",
    "        \"region_name\": credentials['region']})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Now use the SwiftClient connection to programmatically\n",
    "\n",
    "* put_object(container, objectname, data)\n",
    "* get_object(container, objectname)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "examplefile = 'my_team_name_data_folder/zipfiles/classification_1_narrowband.zip'\n",
    "etag = conn.put_object('some_container', 'classification_1_narrowband.zip', open(examplefile).read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "classification_results_file = 'my_team_name_data_folder/results/my_final_testset_classes.csv'\n",
    "etag = conn.put_object('some_container', 'my_final_testset_classes.csv', open(examplefile).read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Using SwiftClient from the Command Line.\n",
    "\n",
    "When you install `python-swiftclient` on your local machine, this also installs the CLI tool, `swift`. \n",
    "\n",
    "You can use this to create new containers on your Object Storage account, upload and download data files.\n",
    "\n",
    "The easiest way to use the `swift` CLI tool is to set the following environment variabls in your `bash` shell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "```\n",
    "> pip install python-keystoneclient\n",
    "> pip install python-swiftclient\n",
    "```\n",
    "\n",
    "Copy and paste this into your `bash` shell. Get your credentials from above.\n",
    "\n",
    "```\n",
    "export OS_PROJECT_ID=xx\n",
    "export OS_PASSWORD=xx\n",
    "export OS_USER_ID=xx\n",
    "export OS_AUTH_URL=https://identity.open.softlayer.com/v3\n",
    "export OS_REGION_NAME=dallas\n",
    "export OS_IDENTITY_API_VERSION=3\n",
    "export OS_AUTH_VERSION=3\n",
    "```\n",
    "\n",
    "Then from the command line, you can\n",
    "\n",
    "```\n",
    "> swift list\n",
    "```\n",
    "\n",
    "Create a new container\n",
    "\n",
    "```\n",
    "> swift post new_container_name\n",
    "```\n",
    "\n",
    "Upload a file \n",
    "\n",
    "```\n",
    "> swift upload some_container_name some_local_file\n",
    "```\n",
    "\n",
    "Download a file\n",
    "\n",
    "```\n",
    "> swift download some_container_name some_file_in_container\n",
    "```\n",
    "\n",
    "This tool can be used from the shell prompt of the PowerAI system as well to move data from those instances to and from your Object Storage account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}