{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "slide_helper": "subslide_end",
     "slide_type": "subslide"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Reading netCDF data\n",
    "- requires [numpy](http://numpy.scipy.org) and netCDF/HDF5 C libraries.\n",
    "- Github site: https://github.com/Unidata/netcdf4-python\n",
    "- Online docs: http://unidata.github.io/netcdf4-python/\n",
    "- Based on Konrad Hinsen's old [Scientific.IO.NetCDF](http://dirac.cnrs-orleans.fr/plone/software/scientificpython/) API, with lots of added netcdf version 4 features.\n",
    "- Developed by Jeff Whitaker at NOAA, with many contributions from users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Interactively exploring a netCDF File\n",
    "\n",
    "Let's explore a netCDF file from the *Atlantic Real-Time Ocean Forecast System*\n",
    "\n",
    "first, import netcdf4-python and numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "internals": {
     "frag_number": 2,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import netCDF4\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 2,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Create a netCDF4.Dataset object\n",
    "- **`f`** is a `Dataset` object, representing an open netCDF file.\n",
    "- printing the object gives you summary information, similar to *`ncdump -h`*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 4,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'netCDF4._netCDF4.Dataset'>\n",
      "root group (NETCDF4_CLASSIC data model, file format HDF5):\n",
      "    Conventions: CF-1.0\n",
      "    title: HYCOM ATLb2.00\n",
      "    institution: National Centers for Environmental Prediction\n",
      "    source: HYCOM archive file\n",
      "    experiment: 90.9\n",
      "    history: archv2ncdf3z\n",
      "    dimensions(sizes): MT(1), Y(850), X(712), Depth(10)\n",
      "    variables(dimensions): float64 MT(MT), float64 Date(MT), float32 Depth(Depth), int32 Y(Y), int32 X(X), float32 Latitude(Y, X), float32 Longitude(Y, X), float32 u(MT, Depth, Y, X), float32 v(MT, Depth, Y, X), float32 temperature(MT, Depth, Y, X), float32 salinity(MT, Depth, Y, X)\n",
      "    groups: \n"
     ]
    }
   ],
   "source": [
    "f = netCDF4.Dataset('data/rtofs_glo_3dz_f006_6hrly_reg3.nc')\n",
    "print(f) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 4,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Access a netCDF variable\n",
    "- variable objects stored by name in **`variables`** dict.\n",
    "- print the variable yields summary info (including all the attributes).\n",
    "- no actual data read yet (just have a reference to the variable object with metadata)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 6,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['MT', 'Date', 'Depth', 'Y', 'X', 'Latitude', 'Longitude', 'u', 'v', 'temperature', 'salinity'])\n",
      "<class 'netCDF4._netCDF4.Variable'>\n",
      "float32 temperature(MT, Depth, Y, X)\n",
      "    coordinates: Longitude Latitude Date\n",
      "    standard_name: sea_water_potential_temperature\n",
      "    units: degC\n",
      "    _FillValue: 1.2676506e+30\n",
      "    valid_range: [-5.078603  11.1498995]\n",
      "    long_name:   temp [90.9H]\n",
      "unlimited dimensions: MT\n",
      "current shape = (1, 10, 850, 712)\n",
      "filling on\n"
     ]
    }
   ],
   "source": [
    "print(f.variables.keys()) # get all variable names\n",
    "temp = f.variables['temperature']  # temperature variable\n",
    "print(temp) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 6,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## List the Dimensions\n",
    "\n",
    "- All variables in a netCDF file have an associated shape, specified by a list of dimensions.\n",
    "- Let's list all the dimensions in this netCDF file.\n",
    "- Note that the **`MT`** dimension is special (*`unlimited`*), which means it can be appended to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 8
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('MT', <class 'netCDF4._netCDF4.Dimension'> (unlimited): name = 'MT', size = 1)\n",
      "('Y', <class 'netCDF4._netCDF4.Dimension'>: name = 'Y', size = 850)\n",
      "('X', <class 'netCDF4._netCDF4.Dimension'>: name = 'X', size = 712)\n",
      "('Depth', <class 'netCDF4._netCDF4.Dimension'>: name = 'Depth', size = 10)\n"
     ]
    }
   ],
   "source": [
    "for d in f.dimensions.items():\n",
    "    print(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 9
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Each variable has a **`dimensions`** and a **`shape`** attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 10
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('MT', 'Depth', 'Y', 'X')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "temp.dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 11,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 10, 850, 712)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "temp.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 11,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Each dimension typically has a variable associated with it (called a *coordinate* variable).\n",
    "- *Coordinate variables* are 1D variables that have the same name as dimensions.\n",
    "- Coordinate variables and *auxiliary coordinate variables* (named by the *coordinates* attribute) locate values in time and space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 13,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'netCDF4._netCDF4.Variable'>\n",
      "float64 MT(MT)\n",
      "    long_name: time\n",
      "    units: days since 1900-12-31 00:00:00\n",
      "    calendar: standard\n",
      "    axis: T\n",
      "unlimited dimensions: MT\n",
      "current shape = (1,)\n",
      "filling on, default _FillValue of 9.969209968386869e+36 used\n",
      "<class 'netCDF4._netCDF4.Variable'>\n",
      "int32 X(X)\n",
      "    point_spacing: even\n",
      "    axis: X\n",
      "unlimited dimensions: \n",
      "current shape = (712,)\n",
      "filling on, default _FillValue of -2147483647 used\n"
     ]
    }
   ],
   "source": [
    "mt = f.variables['MT']\n",
    "depth = f.variables['Depth']\n",
    "x,y = f.variables['X'], f.variables['Y']\n",
    "print(mt)\n",
    "print(x)                 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 13,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Accessing data from a netCDF variable object\n",
    "\n",
    "- netCDF variables objects behave much like numpy arrays.\n",
    "- slicing a netCDF variable object returns a numpy array with the data.\n",
    "- Boolean array and integer sequence indexing behaves differently for netCDF variables than for numpy arrays. Only 1-d boolean arrays and integer sequences are allowed, and these indices work independently along each dimension (similar to the way vector subscripts work in fortran)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 15
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[41023.25]\n"
     ]
    }
   ],
   "source": [
    "time = mt[:]  # Reads the netCDF variable MT, array of one element\n",
    "print(time) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 16
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   0.  100.  200.  400.  700. 1000. 2000. 3000. 4000. 5000.]\n"
     ]
    }
   ],
   "source": [
    "dpth = depth[:] # examine depth array\n",
    "print(dpth) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 17,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape of temp variable: (1, 10, 850, 712)\n",
      "shape of temp slice: (6, 425, 356)\n"
     ]
    }
   ],
   "source": [
    "xx,yy = x[:],y[:]\n",
    "print('shape of temp variable: %s' % repr(temp.shape))\n",
    "tempslice = temp[0, dpth > 400, yy > yy.max()/2, xx > xx.max()/2]\n",
    "print('shape of temp slice: %s' % repr(tempslice.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 17,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What is the sea surface temperature and salinity at 50N, 140W?\n",
    "### Finding the latitude and longitude indices of 50N, 140W\n",
    "\n",
    "- The `X` and `Y` dimensions don't look like longitudes and latitudes\n",
    "- Use the auxiliary coordinate variables named in the `coordinates` variable attribute, `Latitude` and `Longitude`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 19
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'netCDF4._netCDF4.Variable'>\n",
      "float32 Latitude(Y, X)\n",
      "    standard_name: latitude\n",
      "    units: degrees_north\n",
      "unlimited dimensions: \n",
      "current shape = (850, 712)\n",
      "filling on, default _FillValue of 9.969209968386869e+36 used\n"
     ]
    }
   ],
   "source": [
    "lat, lon = f.variables['Latitude'], f.variables['Longitude']\n",
    "print(lat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 20,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Aha!  So we need to find array indices `iy` and `ix` such that `Latitude[iy, ix]` is close to 50.0 and `Longitude[iy, ix]` is close to -140.0 ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 20,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# extract lat/lon values (in degrees) to numpy arrays\n",
    "latvals = lat[:]; lonvals = lon[:] \n",
    "# a function to find the index of the point closest pt\n",
    "# (in squared distance) to give lat/lon value.\n",
    "def getclosest_ij(lats,lons,latpt,lonpt):\n",
    "    # find squared distance of every point on grid\n",
    "    dist_sq = (lats-latpt)**2 + (lons-lonpt)**2  \n",
    "    # 1D index of minimum dist_sq element\n",
    "    minindex_flattened = dist_sq.argmin()    \n",
    "    # Get 2D index for latvals and lonvals arrays from 1D index\n",
    "    return np.unravel_index(minindex_flattened, lats.shape)\n",
    "iy_min, ix_min = getclosest_ij(latvals, lonvals, 50., -140)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 22
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Now we have all the information we need to find our answer.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 23
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "```\n",
    "|----------+--------|\n",
    "| Variable |  Index |\n",
    "|----------+--------|\n",
    "| MT       |      0 |\n",
    "| Depth    |      0 |\n",
    "| Y        | iy_min |\n",
    "| X        | ix_min |\n",
    "|----------+--------|\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 24
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### What is the sea surface temperature and salinity at the specified point?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 25,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 6.4631 degC\n",
      "32.6572 psu\n"
     ]
    }
   ],
   "source": [
    "sal = f.variables['salinity']\n",
    "# Read values out of the netCDF file for temperature and salinity\n",
    "print('%7.4f %s' % (temp[0,0,iy_min,ix_min], temp.units))\n",
    "print('%7.4f %s' % (sal[0,0,iy_min,ix_min], sal.units))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 25,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Remote data access via openDAP\n",
    "\n",
    "- Remote data can be accessed seamlessly with the netcdf4-python API\n",
    "- Access happens via the DAP protocol and DAP servers, such as TDS.\n",
    "- many formats supported, like GRIB, are supported \"under the hood\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 27
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The following example showcases some nice netCDF features:\n",
    "\n",
    "1. We are seamlessly accessing **remote** data, from a TDS server.\n",
    "2. We are seamlessly accessing **GRIB2** data, as if it were netCDF data.\n",
    "3. We are generating **metadata** on-the-fly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 28,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p5deg/GFS_Global_0p5deg_20230525_1200.grib2/GC\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "date = datetime.datetime.now()\n",
    "# build URL for latest synoptic analysis time\n",
    "URL = 'https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p5deg/GFS_Global_0p5deg_%04i%02i%02i_%02i%02i.grib2/GC' %\\\n",
    "(date.year,date.month,date.day,6*(date.hour//6),0)\n",
    "# keep moving back 6 hours until a valid URL found\n",
    "validURL = False; ncount = 0\n",
    "while (not validURL and ncount < 10):\n",
    "    print(URL)\n",
    "    try:\n",
    "        gfs = netCDF4.Dataset(URL)\n",
    "        validURL = True\n",
    "    except RuntimeError:\n",
    "        date -= datetime.timedelta(hours=6)\n",
    "        ncount += 1       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 28,
     "slide_helper": "subslide_end",
     "slide_type": "subslide"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'netCDF4._netCDF4.Variable'>\n",
      "float32 Temperature_surface(time1, lat, lon)\n",
      "    long_name: Temperature @ Ground or water surface\n",
      "    units: K\n",
      "    abbreviation: TMP\n",
      "    missing_value: nan\n",
      "    grid_mapping: LatLon_Projection\n",
      "    coordinates: reftime time1 lat lon \n",
      "    Grib_Variable_Id: VAR_0-0-0_L1\n",
      "    Grib2_Parameter: [0 0 0]\n",
      "    Grib2_Parameter_Discipline: Meteorological products\n",
      "    Grib2_Parameter_Category: Temperature\n",
      "    Grib2_Parameter_Name: Temperature\n",
      "    Grib2_Level_Type: 1\n",
      "    Grib2_Level_Desc: Ground or water surface\n",
      "    Grib2_Generating_Process_Type: Forecast\n",
      "    Grib2_Statistical_Process_Type: UnknownStatType--1\n",
      "unlimited dimensions: \n",
      "current shape = (129, 361, 720)\n",
      "filling off\n",
      "<class 'netCDF4._netCDF4.Variable'>\n",
      "float64 time1(time1)\n",
      "    units: Hour since 2023-05-25T12:00:00Z\n",
      "    standard_name: time\n",
      "    long_name: GRIB forecast or observation time\n",
      "    calendar: proleptic_gregorian\n",
      "    _CoordinateAxisType: Time\n",
      "unlimited dimensions: \n",
      "current shape = (129,)\n",
      "filling off\n",
      "<class 'netCDF4._netCDF4.Variable'>\n",
      "float32 lat(lat)\n",
      "    units: degrees_north\n",
      "    _CoordinateAxisType: Lat\n",
      "unlimited dimensions: \n",
      "current shape = (361,)\n",
      "filling off\n",
      "<class 'netCDF4._netCDF4.Variable'>\n",
      "float32 lon(lon)\n",
      "    units: degrees_east\n",
      "    _CoordinateAxisType: Lon\n",
      "unlimited dimensions: \n",
      "current shape = (720,)\n",
      "filling off\n"
     ]
    }
   ],
   "source": [
    "# Look at metadata for a specific variable\n",
    "# gfs.variables.keys() will show all available variables.\n",
    "sfctmp = gfs.variables['Temperature_surface']\n",
    "# get info about sfctmp\n",
    "print(sfctmp)\n",
    "# print coord vars associated with this variable\n",
    "for dname in sfctmp.dimensions:   \n",
    "    print(gfs.variables[dname])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 28,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##Missing values\n",
    "- when `data == var.missing_value` somewhere, a masked array is returned.\n",
    "- illustrate with soil moisture data (only defined over land)\n",
    "- white areas on plot are masked values over water."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 31
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape=(361, 720), type=<class 'numpy.ma.core.MaskedArray'>, missing_value=nan\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 35268 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "soilmvar = gfs.variables['Volumetric_Soil_Moisture_Content_depth_below_surface_layer']\n",
    "# flip the data in latitude so North Hemisphere is up on the plot\n",
    "soilm = soilmvar[0,0,::-1,:] \n",
    "print('shape=%s, type=%s, missing_value=%s' % \\\n",
    "      (soilm.shape, type(soilm), soilmvar.missing_value))\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "cs = plt.contourf(soilm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 32,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "##Packed integer data\n",
    "There is a similar feature for variables with `scale_factor` and `add_offset` attributes.\n",
    "\n",
    "- short integer data will automatically be returned as float data, with the scale and offset applied.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 32,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Dealing with dates and times\n",
    "- time variables usually measure relative to a fixed date using a certain calendar, with units specified like ***`hours since YY:MM:DD hh-mm-ss`***.\n",
    "- **`num2date`** and **`date2num`** convenience functions provided to convert between these numeric time coordinates and handy python datetime instances.  \n",
    "- **`date2index`** finds the time index corresponding to a datetime instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 34
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name of time dimension = time1\n",
      "units = Hour since 2023-05-25T12:00:00Z, values = [  0.   3.   6.   9.  12.  15.  18.  21.  24.  27.  30.  33.  36.  39.\n",
      "  42.  45.  48.  51.  54.  57.  60.  63.  66.  69.  72.  75.  78.  81.\n",
      "  84.  87.  90.  93.  96.  99. 102. 105. 108. 111. 114. 117. 120. 123.\n",
      " 126. 129. 132. 135. 138. 141. 144. 147. 150. 153. 156. 159. 162. 165.\n",
      " 168. 171. 174. 177. 180. 183. 186. 189. 192. 195. 198. 201. 204. 207.\n",
      " 210. 213. 216. 219. 222. 225. 228. 231. 234. 237. 240. 243. 246. 249.\n",
      " 252. 255. 258. 261. 264. 267. 270. 273. 276. 279. 282. 285. 288. 291.\n",
      " 294. 297. 300. 303. 306. 309. 312. 315. 318. 321. 324. 327. 330. 333.\n",
      " 336. 339. 342. 345. 348. 351. 354. 357. 360. 363. 366. 369. 372. 375.\n",
      " 378. 381. 384.]\n"
     ]
    }
   ],
   "source": [
    "from netCDF4 import num2date, date2num, date2index\n",
    "timedim = sfctmp.dimensions[0] # time dim name\n",
    "print('name of time dimension = %s' % timedim)\n",
    "times = gfs.variables[timedim] # time coord var\n",
    "print('units = %s, values = %s' % (times.units, times[:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 35,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['2023-05-25 12:00:00', '2023-05-25 15:00:00', '2023-05-25 18:00:00', '2023-05-25 21:00:00', '2023-05-26 00:00:00', '2023-05-26 03:00:00', '2023-05-26 06:00:00', '2023-05-26 09:00:00', '2023-05-26 12:00:00', '2023-05-26 15:00:00']\n"
     ]
    }
   ],
   "source": [
    "dates = num2date(times[:], times.units)\n",
    "print([date.strftime('%Y-%m-%d %H:%M:%S') for date in dates[:10]]) # print only first ten..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 35,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "###Get index associated with a specified date, extract forecast data for that date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 37
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-05-28 15:57:27.760935\n",
      "index = 25, date = 2023-05-28 15:00:00\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime, timedelta\n",
    "date = datetime.now() + timedelta(days=3)\n",
    "print(date)\n",
    "ntime = date2index(date,times,select='nearest')\n",
    "print('index = %s, date = %s' % (ntime, dates[ntime]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 38
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "###Get temp forecast for Boulder (near 40N, -105W)\n",
    "- use function **`getcloses_ij`** we created before..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 39,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Boulder forecast valid at 2023-05-28 15:00:00 UTC = 297.6 K\n"
     ]
    }
   ],
   "source": [
    "lats, lons = gfs.variables['lat'][:], gfs.variables['lon'][:]\n",
    "# lats, lons are 1-d. Make them 2-d using numpy.meshgrid.\n",
    "lons, lats = np.meshgrid(lons,lats)\n",
    "j, i = getclosest_ij(lats,lons,40,-105)\n",
    "fcst_temp = sfctmp[ntime,j,i]\n",
    "print('Boulder forecast valid at %s UTC = %5.1f %s' % \\\n",
    "      (dates[ntime],fcst_temp,sfctmp.units))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 39,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##Simple multi-file aggregation\n",
    "\n",
    "What if you have a bunch of netcdf files, each with data for a different year, and you want to access all the data as if it were in one file?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 41
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-r-- 1 8985332 May 17 15:27 data/prmsl.2000.nc\r\n",
      "-rw-rw-r-- 1 8968789 May 17 15:27 data/prmsl.2001.nc\r\n",
      "-rw-rw-r-- 1 8972796 May 17 15:27 data/prmsl.2002.nc\r\n",
      "-rw-rw-r-- 1 8974435 May 17 15:27 data/prmsl.2003.nc\r\n",
      "-rw-rw-r-- 1 8997438 May 17 15:27 data/prmsl.2004.nc\r\n",
      "-rw-rw-r-- 1 8976678 May 17 15:27 data/prmsl.2005.nc\r\n",
      "-rw-rw-r-- 1 8969714 May 17 15:27 data/prmsl.2006.nc\r\n",
      "-rw-rw-r-- 1 8974360 May 17 15:27 data/prmsl.2007.nc\r\n",
      "-rw-rw-r-- 1 8994260 May 17 15:27 data/prmsl.2008.nc\r\n",
      "-rw-rw-r-- 1 8974678 May 17 15:27 data/prmsl.2009.nc\r\n",
      "-rw-rw-r-- 1 8970732 May 17 15:27 data/prmsl.2010.nc\r\n",
      "-rw-rw-r-- 1 8976285 May 17 15:27 data/prmsl.2011.nc\r\n"
     ]
    }
   ],
   "source": [
    "!ls -ldgG data/prmsl*nc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 42
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**`MFDataset`** uses file globbing to patch together all the files into one big Dataset.\n",
    "You can also pass it a list of specific files.\n",
    "\n",
    "Limitations:\n",
    "\n",
    "- It can only  aggregate the data along the leftmost dimension of each variable.\n",
    "- only works with `NETCDF3`, or `NETCDF4_CLASSIC` formatted files.\n",
    "- kind of slow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 43,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "starting date = 2000-01-01 00:00:00\n",
      "ending date = 2011-12-31 00:00:00\n",
      "times shape = 4383\n",
      "prmsl dimensions = ('time', 'lat', 'lon'), prmsl shape = (4383, 91, 180)\n"
     ]
    }
   ],
   "source": [
    "mf = netCDF4.MFDataset('data/prmsl*nc')\n",
    "times = mf.variables['time']\n",
    "dates = num2date(times[:],times.units)\n",
    "print('starting date = %s' % dates[0])\n",
    "print('ending date = %s'% dates[-1])\n",
    "prmsl = mf.variables['prmsl']\n",
    "print('times shape = %s' % times.shape)\n",
    "print('prmsl dimensions = %s, prmsl shape = %s' %\\\n",
    "     (prmsl.dimensions, prmsl.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 43,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Closing your netCDF file\n",
    "\n",
    "It's good to close netCDF files, but not actually necessary when Dataset is open for read access only.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 45
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f.close()\n",
    "gfs.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 45,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "##That's it!\n",
    "\n",
    "Now you're ready to start exploring your data interactively.\n",
    "\n",
    "To be continued with **Writing netCDF data** ...."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
