"
],
"text/plain": [
":Scatter [TIME] (DEPTH,TEMP)"
]
},
"execution_count": 6,
"metadata": {
"application/vnd.holoviews_exec.v0+json": {}
},
"output_type": "execute_result"
}
],
"source": [
"# Hourly averages x 12 depths x 13+ yr = over a million points to plot!\n",
"# Let's just look at a year's worth to speed things up...\n",
"ds_mooring.sel(TIME=\"2022\").hvplot.scatter(x=\"TIME\", y=\"DEPTH\", c=\"TEMP\",\n",
" cmap=\"coolwarm\", alpha=0.2,\n",
" flip_yaxis=True, hover=False)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "80eb6d7d-3110-459c-822f-1d613f883128",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
":Curve [TIME] (TEMP)"
]
},
"execution_count": 7,
"metadata": {
"application/vnd.holoviews_exec.v0+json": {}
},
"output_type": "execute_result"
}
],
"source": [
"# ... or we can look at the full timeseries of temperature at a single depth\n",
"ds_mooring.TEMP.sel(DEPTH=30).hvplot()"
]
},
{
"cell_type": "markdown",
"id": "bc26abf8-1638-4036-98a7-79e86bcfcab2",
"metadata": {
"slideshow": {
"slide_type": "skip"
},
"tags": []
},
"source": [
"### Another example\n",
"See also the OPeNDAP example in [last year's data access tutorial](https://github.com/oceanhackweek/ohw-tutorials/blob/OHW22/01-Tue/00-data-access-in-python/00-data-access.ipynb), or explore that [dataset](http://tds.marine.rutgers.edu/thredds/dodsC/roms/doppio/2017_da/avg/Averages_Best.html) in the OPeNDAP Access Form."
]
},
{
"cell_type": "markdown",
"id": "571909e3-71b4-427d-aefb-3d64dbd6158c",
"metadata": {
"slideshow": {
"slide_type": "slide"
},
"tags": []
},
"source": [
"# Web Feature Service (WFS)\n",
"\n",
"* A [standard](http://www.opengeospatial.org/standards/wfs) of the [Open Geospatial Consortium](http://www.opengeospatial.org/) (OGC)\n",
"* Allows tabular geospatial data to be accessed via the Web.\n",
"* A _feature_ has a _geometry_ (e.g. a point/line/polygon) indicating a geographic location, and a set of properties (e.g. temperature) \n",
"* WFS allows filtering based on geometry or properties.\n",
"* In Python WFS and other OGC Web Services (OWS) can be accessed using the [`owslib`](https://pypi.org/project/OWSLib/) library"
]
},
{
"cell_type": "markdown",
"id": "1b1d1270-159f-46da-b6b8-b2162b0401d6",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"For example, most of the tabular hosted by the AODN is available via WFS."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "6250dfb4-fc51-4db7-9088-17bd2bd9e8c7",
"metadata": {
"slideshow": {
"slide_type": "fragment"
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"'AODN Web Feature Service (WFS)'"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"wfs = WebFeatureService(url=\"https://geoserver-123.aodn.org.au/geoserver/wfs\",\n",
" version=\"1.1.0\")\n",
"wfs.identification.title"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "af41cd03-3895-490a-8621-95a569a53a35",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"There are 397 fature types, e.g.\n"
]
},
{
"data": {
"text/plain": [
"['imos:anmn_ctd_profiles_data',\n",
" 'imos:anmn_ctd_profiles_map',\n",
" 'imos:anmn_velocity_timeseries_map',\n",
" 'imos:anmn_nrs_rt_meteo_timeseries_data',\n",
" 'imos:anmn_nrs_rt_meteo_timeseries_map',\n",
" 'imos:anmn_nrs_rt_bio_timeseries_data',\n",
" 'imos:anmn_nrs_rt_bio_timeseries_map',\n",
" 'imos:anmn_nrs_rt_wave_timeseries_data',\n",
" 'imos:anmn_nrs_rt_wave_timeseries_map',\n",
" 'imos:anmn_acoustics_map']"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Each dataset is served as a separate \"feature type\":\n",
"print(f\"There are {len(wfs.contents)} fature types, e.g.\")\n",
"list(wfs.contents)[:10]"
]
},
{
"cell_type": "markdown",
"id": "08076cde-3bec-4199-932a-61ee43e8df64",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"For now we'll assume we already know which featuretype we want. In this example we'll look at a dataset containing condicutivity-temperature-depth (CTD) profiles obtained at the National Reference Stations around Australia ([here](https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/7b901002-b1dc-46c3-89f2-b4951cedca48)'s a detailed metadata record)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "87f370e7-a79c-431e-9059-fee45837b0ef",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"{'file_id': 'int',\n",
" 'site_code': 'string',\n",
" 'cruise_id': 'string',\n",
" 'time_coverage_start': 'dateTime',\n",
" 'time_coverage_end': 'dateTime',\n",
" 'TIME': 'dateTime',\n",
" 'INSTANCE': 'int',\n",
" 'DIRECTION': 'string',\n",
" 'TIME_quality_control': 'string',\n",
" 'LATITUDE': 'double',\n",
" 'LATITUDE_quality_control': 'string',\n",
" 'LONGITUDE': 'double',\n",
" 'LONGITUDE_quality_control': 'string',\n",
" 'DEPTH': 'float',\n",
" 'DEPTH_quality_control': 'string',\n",
" 'BOT_DEPTH': 'float',\n",
" 'BOT_DEPTH_quality_control': 'string',\n",
" 'PRES_REL': 'float',\n",
" 'PRES_REL_quality_control': 'string',\n",
" 'TEMP': 'float',\n",
" 'TEMP_quality_control': 'string',\n",
" 'PSAL': 'float',\n",
" 'PSAL_quality_control': 'string',\n",
" 'DOX2': 'float',\n",
" 'DOX2_quality_control': 'string',\n",
" 'TURB': 'float',\n",
" 'TURB_quality_control': 'string',\n",
" 'CHLF': 'float',\n",
" 'CHLF_quality_control': 'string',\n",
" 'CHLU': 'float',\n",
" 'CHLU_quality_control': 'string',\n",
" 'CPHL': 'float',\n",
" 'CPHL_quality_control': 'string',\n",
" 'CNDC': 'float',\n",
" 'CNDC_quality_control': 'string',\n",
" 'DESC': 'float',\n",
" 'DESC_quality_control': 'string',\n",
" 'DENS': 'float',\n",
" 'DENS_quality_control': 'string'}"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"typename = 'imos:anmn_ctd_profiles_data'\n",
"wfs.get_schema(typename)['properties']"
]
},
{
"cell_type": "markdown",
"id": "0c02cee3-558e-4515-aaae-c5353f4c3f29",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"We can read in a subset of the data by specifying a bounding box (in this case near Sydney, Australia).\n",
"We'll get the result in CSV format so it's easy to read into a Pandas DataFrame.\n",
"\n",
"First we'll ask for just 10 features, for a quick look at the data."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "dee02762-2045-415f-b10c-c572b9e11899",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" FID file_id site_code \\\n",
"0 anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... 754 PH100 \n",
"1 anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... 754 PH100 \n",
"2 anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... 754 PH100 \n",
"3 anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... 754 PH100 \n",
"4 anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... 754 PH100 \n",
"5 anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... 754 PH100 \n",
"6 anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... 754 PH100 \n",
"7 anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... 754 PH100 \n",
"8 anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... 754 PH100 \n",
"9 anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... 754 PH100 \n",
"\n",
" cruise_id time_coverage_start time_coverage_end TIME \\\n",
"0 PHNRS_1108 2011-08-29T00:03:40 2011-08-29T00:03:40 2011-08-29T00:03:40 \n",
"1 PHNRS_1108 2011-08-29T00:03:40 2011-08-29T00:03:40 2011-08-29T00:03:40 \n",
"2 PHNRS_1108 2011-08-29T00:03:40 2011-08-29T00:03:40 2011-08-29T00:03:40 \n",
"3 PHNRS_1108 2011-08-29T00:03:40 2011-08-29T00:03:40 2011-08-29T00:03:40 \n",
"4 PHNRS_1108 2011-08-29T00:03:40 2011-08-29T00:03:40 2011-08-29T00:03:40 \n",
"5 PHNRS_1108 2011-08-29T00:03:40 2011-08-29T00:03:40 2011-08-29T00:03:40 \n",
"6 PHNRS_1108 2011-08-29T00:03:40 2011-08-29T00:03:40 2011-08-29T00:03:40 \n",
"7 PHNRS_1108 2011-08-29T00:03:40 2011-08-29T00:03:40 2011-08-29T00:03:40 \n",
"8 PHNRS_1108 2011-08-29T00:03:40 2011-08-29T00:03:40 2011-08-29T00:03:40 \n",
"9 PHNRS_1108 2011-08-29T00:03:40 2011-08-29T00:03:40 2011-08-29T00:03:40 \n",
"\n",
" INSTANCE DIRECTION TIME_quality_control ... CHLU_quality_control CPHL \\\n",
"0 NaN D NaN ... NaN NaN \n",
"1 NaN D NaN ... NaN NaN \n",
"2 NaN D NaN ... NaN NaN \n",
"3 NaN D NaN ... NaN NaN \n",
"4 NaN D NaN ... NaN NaN \n",
"5 NaN D NaN ... NaN NaN \n",
"6 NaN D NaN ... NaN NaN \n",
"7 NaN D NaN ... NaN NaN \n",
"8 NaN D NaN ... NaN NaN \n",
"9 NaN D NaN ... NaN NaN \n",
"\n",
" CPHL_quality_control CNDC CNDC_quality_control DESC \\\n",
"0 NaN 4.6266 0 0.228 \n",
"1 NaN 4.6246 0 0.574 \n",
"2 NaN 4.6224 0 0.741 \n",
"3 NaN 4.6190 0 0.803 \n",
"4 NaN 4.6138 0 0.749 \n",
"5 NaN 4.6089 0 0.687 \n",
"6 NaN 4.6067 0 0.722 \n",
"7 NaN 4.6048 0 0.773 \n",
"8 NaN 4.6023 0 0.788 \n",
"9 NaN 4.5982 0 0.846 \n",
"\n",
" DESC_quality_control DENS DENS_quality_control \\\n",
"0 0 1025.8478 0 \n",
"1 0 1025.8652 0 \n",
"2 0 1025.8737 0 \n",
"3 0 1025.8790 0 \n",
"4 0 1025.8892 0 \n",
"5 0 1025.9072 0 \n",
"6 0 1025.9241 0 \n",
"7 0 1025.9321 0 \n",
"8 0 1025.9385 0 \n",
"9 0 1025.9432 0 \n",
"\n",
" geom \n",
"0 POINT (-34.1161666667 151.218) \n",
"1 POINT (-34.1161666667 151.218) \n",
"2 POINT (-34.1161666667 151.218) \n",
"3 POINT (-34.1161666667 151.218) \n",
"4 POINT (-34.1161666667 151.218) \n",
"5 POINT (-34.1161666667 151.218) \n",
"6 POINT (-34.1161666667 151.218) \n",
"7 POINT (-34.1161666667 151.218) \n",
"8 POINT (-34.1161666667 151.218) \n",
"9 POINT (-34.1161666667 151.218) \n",
"\n",
"[10 rows x 41 columns]"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"xmin, xmax = 151.2, 151.25 # Port Hacking, near Sydney, NSW\n",
"ymin, ymax = -34.2, -34.1\n",
"\n",
"response = wfs.getfeature(typename=typename,\n",
" bbox=(xmin, ymin, xmax, ymax),\n",
" maxfeatures=10,\n",
" outputFormat='csv')\n",
"df = pd.read_csv(response)\n",
"response.close()\n",
"\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "309726d2-a129-4fa2-a4ff-b450a0356384",
"metadata": {
"slideshow": {
"slide_type": "skip"
},
"tags": []
},
"outputs": [],
"source": [
"## Load local copy of CSV file returned...\n",
"\n",
"# local_csv = os.path.join(DATA_BASEPATH, 'wfs_response1.csv')\n",
"# df = pd.read_csv(local_csv)\n",
"# df"
]
},
{
"cell_type": "markdown",
"id": "af9bcbb6-605c-4aa9-8be1-8370a8d4b44e",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"We can also filter the data based on the values in specified columns (properties) and ask for only a subset of the columns to be returned. The filters need to be provided in XML format, but the `owslib` library allows us to construct them in a more Pythonic way.\n",
"\n",
"Here we select only the profiles associated with the Port Hacking 100m mooring site, and only the data points flagged as \"good data\" by automated quality-control procedures."
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "1953a518-7a7f-4706-aedf-a1bc399fb4a0",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"outputs": [],
"source": [
"from owslib.etree import etree\n",
"from owslib.fes import PropertyIsEqualTo, And\n",
"\n",
"filter = And([PropertyIsEqualTo(propertyname=\"site_code\", literal=\"PH100\"),\n",
" PropertyIsEqualTo(propertyname=\"PRES_REL_quality_control\", literal=\"1\"),\n",
" PropertyIsEqualTo(propertyname=\"TEMP_quality_control\", literal=\"1\"),\n",
" PropertyIsEqualTo(propertyname=\"PSAL_quality_control\", literal=\"1\"),\n",
" PropertyIsEqualTo(propertyname=\"CPHL_quality_control\", literal=\"1\")\n",
" ])\n",
"filterxml = etree.tostring(filter.toXML(), encoding=\"unicode\")\n",
"\n",
"response = wfs.getfeature(typename=typename, filter=filterxml, outputFormat=\"csv\",\n",
" propertyname=[\"TIME\", \"DEPTH\", \"TEMP\", \"PSAL\", \"CPHL\"]\n",
" )\n",
"df = pd.read_csv(response, parse_dates=[\"TIME\"])\n",
"response.close()\n",
"\n",
"# the server adds a feature ID column we don't really need\n",
"df.drop(columns='FID', inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fbcf91fe-1111-4313-9fef-b7650e351caf",
"metadata": {
"slideshow": {
"slide_type": "skip"
},
"tags": []
},
"outputs": [],
"source": [
"## Load local copy of CSV file returned...\n",
"\n",
"# local_csv = os.path.join(DATA_BASEPATH, 'wfs_response2.csv')\n",
"# df = pd.read_csv(local_csv, parse_dates=[\"TIME\"]).drop(columns='FID')"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "c9b1860c-b573-4410-a45a-216abf9439e3",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | FID | \n", "file_id | \n", "site_code | \n", "cruise_id | \n", "time_coverage_start | \n", "time_coverage_end | \n", "TIME | \n", "INSTANCE | \n", "DIRECTION | \n", "TIME_quality_control | \n", "... | \n", "CHLU_quality_control | \n", "CPHL | \n", "CPHL_quality_control | \n", "CNDC | \n", "CNDC_quality_control | \n", "DESC | \n", "DESC_quality_control | \n", "DENS | \n", "DENS_quality_control | \n", "geom | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... | \n", "754 | \n", "PH100 | \n", "PHNRS_1108 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "NaN | \n", "D | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "4.6266 | \n", "0 | \n", "0.228 | \n", "0 | \n", "1025.8478 | \n", "0 | \n", "POINT (-34.1161666667 151.218) | \n", "
| 1 | \n", "anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... | \n", "754 | \n", "PH100 | \n", "PHNRS_1108 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "NaN | \n", "D | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "4.6246 | \n", "0 | \n", "0.574 | \n", "0 | \n", "1025.8652 | \n", "0 | \n", "POINT (-34.1161666667 151.218) | \n", "
| 2 | \n", "anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... | \n", "754 | \n", "PH100 | \n", "PHNRS_1108 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "NaN | \n", "D | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "4.6224 | \n", "0 | \n", "0.741 | \n", "0 | \n", "1025.8737 | \n", "0 | \n", "POINT (-34.1161666667 151.218) | \n", "
| 3 | \n", "anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... | \n", "754 | \n", "PH100 | \n", "PHNRS_1108 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "NaN | \n", "D | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "4.6190 | \n", "0 | \n", "0.803 | \n", "0 | \n", "1025.8790 | \n", "0 | \n", "POINT (-34.1161666667 151.218) | \n", "
| 4 | \n", "anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... | \n", "754 | \n", "PH100 | \n", "PHNRS_1108 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "NaN | \n", "D | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "4.6138 | \n", "0 | \n", "0.749 | \n", "0 | \n", "1025.8892 | \n", "0 | \n", "POINT (-34.1161666667 151.218) | \n", "
| 5 | \n", "anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... | \n", "754 | \n", "PH100 | \n", "PHNRS_1108 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "NaN | \n", "D | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "4.6089 | \n", "0 | \n", "0.687 | \n", "0 | \n", "1025.9072 | \n", "0 | \n", "POINT (-34.1161666667 151.218) | \n", "
| 6 | \n", "anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... | \n", "754 | \n", "PH100 | \n", "PHNRS_1108 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "NaN | \n", "D | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "4.6067 | \n", "0 | \n", "0.722 | \n", "0 | \n", "1025.9241 | \n", "0 | \n", "POINT (-34.1161666667 151.218) | \n", "
| 7 | \n", "anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... | \n", "754 | \n", "PH100 | \n", "PHNRS_1108 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "NaN | \n", "D | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "4.6048 | \n", "0 | \n", "0.773 | \n", "0 | \n", "1025.9321 | \n", "0 | \n", "POINT (-34.1161666667 151.218) | \n", "
| 8 | \n", "anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... | \n", "754 | \n", "PH100 | \n", "PHNRS_1108 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "NaN | \n", "D | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "4.6023 | \n", "0 | \n", "0.788 | \n", "0 | \n", "1025.9385 | \n", "0 | \n", "POINT (-34.1161666667 151.218) | \n", "
| 9 | \n", "anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b... | \n", "754 | \n", "PH100 | \n", "PHNRS_1108 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "2011-08-29T00:03:40 | \n", "NaN | \n", "D | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "4.5982 | \n", "0 | \n", "0.846 | \n", "0 | \n", "1025.9432 | \n", "0 | \n", "POINT (-34.1161666667 151.218) | \n", "
10 rows × 41 columns
\n", "\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" TIME DEPTH TEMP PSAL CPHL\n",
"0 2014-12-08 22:28:54 1.986 21.6432 35.5080 0.9365\n",
"1 2014-12-08 22:28:54 2.979 21.6441 35.5085 0.9560\n",
"2 2014-12-08 22:28:54 3.971 21.6417 35.5085 0.9644\n",
"3 2014-12-08 22:28:54 4.964 21.6314 35.5089 0.9963\n",
"4 2014-12-08 22:28:54 5.957 21.6077 35.5102 0.9844\n",
"... ... ... ... ... ...\n",
"11377 2023-05-15 22:08:05 82.398 18.0130 35.5832 0.1554\n",
"11378 2023-05-15 22:08:05 83.391 18.0008 35.5841 0.1417\n",
"11379 2023-05-15 22:08:05 84.384 17.9824 35.5843 0.1345\n",
"11380 2023-05-15 22:08:05 85.376 17.9343 35.5821 0.0937\n",
"11381 2023-05-15 22:08:05 86.368 17.8612 35.5799 0.0300\n",
"\n",
"[11382 rows x 5 columns]"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "f0a36ec4-b928-4000-b80c-a6b6738e7deb",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | TIME | \n", "DEPTH | \n", "TEMP | \n", "PSAL | \n", "CPHL | \n", "
|---|---|---|---|---|---|
| 0 | \n", "2014-12-08 22:28:54 | \n", "1.986 | \n", "21.6432 | \n", "35.5080 | \n", "0.9365 | \n", "
| 1 | \n", "2014-12-08 22:28:54 | \n", "2.979 | \n", "21.6441 | \n", "35.5085 | \n", "0.9560 | \n", "
| 2 | \n", "2014-12-08 22:28:54 | \n", "3.971 | \n", "21.6417 | \n", "35.5085 | \n", "0.9644 | \n", "
| 3 | \n", "2014-12-08 22:28:54 | \n", "4.964 | \n", "21.6314 | \n", "35.5089 | \n", "0.9963 | \n", "
| 4 | \n", "2014-12-08 22:28:54 | \n", "5.957 | \n", "21.6077 | \n", "35.5102 | \n", "0.9844 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 11377 | \n", "2023-05-15 22:08:05 | \n", "82.398 | \n", "18.0130 | \n", "35.5832 | \n", "0.1554 | \n", "
| 11378 | \n", "2023-05-15 22:08:05 | \n", "83.391 | \n", "18.0008 | \n", "35.5841 | \n", "0.1417 | \n", "
| 11379 | \n", "2023-05-15 22:08:05 | \n", "84.384 | \n", "17.9824 | \n", "35.5843 | \n", "0.1345 | \n", "
| 11380 | \n", "2023-05-15 22:08:05 | \n", "85.376 | \n", "17.9343 | \n", "35.5821 | \n", "0.0937 | \n", "
| 11381 | \n", "2023-05-15 22:08:05 | \n", "86.368 | \n", "17.8612 | \n", "35.5799 | \n", "0.0300 | \n", "
11382 rows × 5 columns
\n", "
"
],
"text/plain": [
":Layout\n",
" .NdOverlay.I :NdOverlay [TIME]\n",
" :Curve [TEMP] (DEPTH)\n",
" .NdOverlay.II :NdOverlay [TIME]\n",
" :Curve [PSAL] (DEPTH)\n",
" .NdOverlay.III :NdOverlay [TIME]\n",
" :Curve [CPHL] (DEPTH)"
]
},
"execution_count": 27,
"metadata": {
"application/vnd.holoviews_exec.v0+json": {}
},
"output_type": "execute_result"
}
],
"source": [
"# We can explore the temperature, salinity and chlorophyll profiles\n",
"# Change \"by\" to \"groupby\" to view one profile at a time, with time selected interactively\n",
"temp_plot = df.hvplot(x=\"TEMP\", y=\"DEPTH\", by=\"TIME\", flip_yaxis=True, legend=False, width=200)\n",
"psal_plot = df.hvplot(x=\"PSAL\", y=\"DEPTH\", by=\"TIME\", flip_yaxis=True, legend=False, width=200)\n",
"cphl_plot = df.hvplot(x=\"CPHL\", y=\"DEPTH\", by=\"TIME\", flip_yaxis=True, legend=False, width=200)\n",
"\n",
"(temp_plot + psal_plot + cphl_plot).opts(tight=True)"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "6d8ab58b-a465-4800-8172-c7b2cb7e4411",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
":Overlay\n",
" .Curve.I :Curve [TIME] (TEMP)\n",
" .Scatter.I :Scatter [TIME] (TEMP)"
]
},
"execution_count": 29,
"metadata": {
"application/vnd.holoviews_exec.v0+json": {}
},
"output_type": "execute_result"
}
],
"source": [
"# We can also extract the temperature measurements at a fixed depth\n",
"# and compare to the timeseries from the mooring \n",
"comp_depth = 20 # metres\n",
"\n",
"df_sub = df[df.DEPTH.round() == comp_depth]\n",
"ctd_plot = df_sub.hvplot.scatter(x=\"TIME\", y=\"TEMP\", c=\"red\")\n",
"\n",
"mooring_plot = ds_mooring.TEMP.sel(DEPTH=comp_depth).hvplot()\n",
"\n",
"mooring_plot * ctd_plot"
]
},
{
"cell_type": "markdown",
"id": "8bae0efa-8466-421d-aace-39f5259cd695",
"metadata": {
"slideshow": {
"slide_type": "slide"
},
"tags": []
},
"source": [
"# Direct access to files on cloud storage\n",
"\n",
"Data files made available to the public on cloud storage such as [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) (Simple Storage Service) can be accessed over the web as if they were stored locally. You just need to find the exact URL for each file.\n",
"\n",
"In Python, we can access S3 storage in a very similar way to a local filesystem using the `s3fs` library."
]
},
{
"cell_type": "markdown",
"id": "78fdff28-e958-42a8-8a07-43ebd5583020",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"For example, all the public data files hosted by the Australian Ocean Data Network are stored in an [S3 bucket](https://www.techtarget.com/searchaws/definition/AWS-bucket) called `imos-data`. You can browse the contents of the bucket and download individual files [here](https://imos-data.aodn.org.au). \n",
"\n",
"Below we'll look at a [high-resolution regional SST product](https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/a4170ca8-0942-4d13-bdb8-ad4718ce14bb) from IMOS (based on satellite and in-situ observations). This product is a collection of daily gridded NetCDF files covering the Australian region."
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "00b9e6e6-c345-46bf-a325-1f9e98059303",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"['imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230713120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230714120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230715120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230716120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230717120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230718120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230719120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230720120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230721120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230722120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230723120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230725120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230726120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230727120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230728120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230729120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230730120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230731120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230801120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',\n",
" 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230802120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc']"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"s3 = s3fs.S3FileSystem(anon=True)\n",
"\n",
"# List the most recent files available\n",
"sst_files = s3.ls(\"imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023\")\n",
"sst_files[-20:]"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "285652d9-de6c-48a8-917a-a7447a7c95b3",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": [
"DRAFT"
]
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
"
],
"text/plain": [
"<xarray.Dataset>\n",
"Dimensions: (time: 1, lat: 1081, lon: 1561)\n",
"Coordinates:\n",
" * time (time) datetime64[ns] 2023-08-02T12:00:00\n",
" * lat (lat) float32 -70.0 -69.92 -69.83 ... 19.83 19.92 20.0\n",
" * lon (lon) float32 60.0 60.08 60.17 60.25 ... 189.8 189.9 190.0\n",
"Data variables:\n",
" sea_ice_fraction (time, lat, lon) float32 ...\n",
" analysed_sst (time, lat, lon) float32 ...\n",
" analysis_error (time, lat, lon) float32 ...\n",
" mask (time, lat, lon) float32 ...\n",
" crs int32 -2147483647\n",
"Attributes: (12/65)\n",
" id: RAMSSA_09km-ABOM-L4-AUS-v01\n",
" Conventions: CF-1.6, ACDD-1.3, ISO 8601\n",
" title: RAMSSA v1.1 Analysed high resolution foundati...\n",
" summary: AMSR2-JAXA nobs=946601 obsesd: avg=0.693 min=...\n",
" source: AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VII...\n",
" references: Beggs H., A. Zhong, G. Warren, O. Alves, G. B...\n",
" ... ...\n",
" geospatial_lat_max: 20.0\n",
" geospatial_lat_min: -70.0\n",
" geospatial_lon_max: 190.0\n",
" geospatial_lon_min: 60.0\n",
" geospatial_bounds: POLYGON((-70 60, 20 60, 20 190, -70 190, -70 ...\n",
" geospatial_bounds_crs: EPSG:4326
"
],
"text/plain": [
":Overlay\n",
" .Image.I :Image [lon,lat] (analysed_sst)\n",
" .Land.I :Feature [Longitude,Latitude]"
]
},
"execution_count": 32,
"metadata": {
"application/vnd.holoviews_exec.v0+json": {}
},
"output_type": "execute_result"
}
],
"source": [
"# Plot a subset of the dataset around Australia\n",
"sst_var = 'analysed_sst'\n",
"gds = gv.Dataset(ds.sel(lat=slice(-50, 0), lon=slice(105, 175)),\n",
" kdims=['lon', 'lat'],\n",
" vdims=[sst_var]\n",
" )\n",
"sst_plot = (gds.to(gv.Image)\n",
" .opts(cmap='coolwarm', colorbar=True, aspect=1.4, title=ds.title))\n",
"sst_plot * gf.land"
]
},
{
"cell_type": "markdown",
"id": "69f9e6cf-4e07-42e2-ad13-fbb282f568d6",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"It's worth understanding a little about how this works. \n",
"\n",
"The above example only makes use of the metadata from the file, one of the 4 data variables, and the `lon` and `lat` coordinates. On a local filesystem, it would be easy to read only these specific parts of the file from disk. "
]
},
{
"cell_type": "markdown",
"id": "1de52f75-90ca-49be-a644-5deb139077bc",
"metadata": {
"slideshow": {
"slide_type": "fragment"
},
"tags": []
},
"source": [
"However, on cloud storage services like S3 (also called \"object storage\") the basic read/write functions operate on the entire file (object), so at least in the backend, the entire file is read**. If you only need a small subset of a large file, this can be a very inefficient way to get it.\n",
"\n",
"** _Note: it is possible to request only a subset of an S3 object to be read, but this is more advanced usage than what we're doing here._"
]
},
{
"cell_type": "markdown",
"id": "73e731bc-985f-4d29-8647-8f0e2c476fe7",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": [
"DRAFT"
]
},
"source": [
"For example, if we wanted to plot a timeseries of the above satellite SST product at a given point, we would only need a single value out of each file (corresponding to one point in the timeseries), but the entire file would need to be read each time.\n",
"\n",
"For a quick demo we'll try this with last month's files. `xarray` has a handy `open_mfdataset` function that can create a single `Dataset` object out of a series of files (with similar structure)."
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "ac703357-680d-4ada-b536-93f81a3b3daf",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": [
"DRAFT"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 4.78 s, sys: 119 ms, total: 4.9 s\n",
"Wall time: 15.4 s\n"
]
},
{
"data": {
"text/html": [
"\n",
"
"
],
"text/plain": [
"<xarray.Dataset>\n",
"Dimensions: (time: 30, lat: 1081, lon: 1561)\n",
"Coordinates:\n",
" * time (time) datetime64[ns] 2023-07-01T12:00:00 ... 2023-07-3...\n",
" * lat (lat) float32 -70.0 -69.92 -69.83 ... 19.83 19.92 20.0\n",
" * lon (lon) float32 60.0 60.08 60.17 60.25 ... 189.8 189.9 190.0\n",
"Data variables:\n",
" sea_ice_fraction (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>\n",
" analysed_sst (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>\n",
" analysis_error (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>\n",
" mask (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>\n",
" crs (time) int32 -2147483647 -2147483647 ... -2147483647\n",
"Attributes: (12/65)\n",
" id: RAMSSA_09km-ABOM-L4-AUS-v01\n",
" Conventions: CF-1.6, ACDD-1.3, ISO 8601\n",
" title: RAMSSA v1.1 Analysed high resolution foundati...\n",
" summary: AMSR2-JAXA nobs=****** obsesd: avg=0.693 min=...\n",
" source: AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VII...\n",
" references: Beggs H., A. Zhong, G. Warren, O. Alves, G. B...\n",
" ... ...\n",
" geospatial_lat_max: 20.0\n",
" geospatial_lat_min: -70.0\n",
" geospatial_lon_max: 190.0\n",
" geospatial_lon_min: 60.0\n",
" geospatial_bounds: POLYGON((-70 60, 20 60, 20 190, -70 190, -70 ...\n",
" geospatial_bounds_crs: EPSG:4326\n",
"
"
],
"text/plain": [
"<xarray.DataArray 'analysed_sst' (time: 30, lat: 1081, lon: 1561)>\n",
"dask.array<concatenate, shape=(30, 1081, 1561), dtype=float32, chunksize=(1, 1081, 1561), chunktype=numpy.ndarray>\n",
"Coordinates:\n",
" * time (time) datetime64[ns] 2023-07-01T12:00:00 ... 2023-07-31T12:00:00\n",
" * lat (lat) float32 -70.0 -69.92 -69.83 -69.75 ... 19.75 19.83 19.92 20.0\n",
" * lon (lon) float32 60.0 60.08 60.17 60.25 ... 189.8 189.8 189.9 190.0\n",
"Attributes:\n",
" valid_min: -300\n",
" valid_max: 4500\n",
" clip_min: 269.30999398231506\n",
" clip_max: 304.8399931881577\n",
" units: kelvin\n",
" long_name: analysed sea surface temperature\n",
" standard_name: sea_surface_foundation_temperature\n",
" comment: Optimally interpolated analysis of SST observations.\n",
" source: AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VIIRS_N...\n",
" coverage_content_type: physicalMeasurement\n",
" grid_mapping: crs\n",
"
"
],
"text/plain": [
"<xarray.Dataset>\n",
"Dimensions: (time: 30, lat: 1081, lon: 1561)\n",
"Coordinates:\n",
" * time (time) datetime64[ns] 2023-07-01T12:00:00 ... 2023-07-3...\n",
" * lat (lat) float32 -70.0 -69.92 -69.83 ... 19.83 19.92 20.0\n",
" * lon (lon) float32 60.0 60.08 60.17 60.25 ... 189.8 189.9 190.0\n",
"Data variables:\n",
" sea_ice_fraction (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>\n",
" analysed_sst (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>\n",
" analysis_error (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>\n",
" mask (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>\n",
" crs (time) int32 -2147483647 -2147483647 ... -2147483647\n",
"Attributes: (12/65)\n",
" id: RAMSSA_09km-ABOM-L4-AUS-v01\n",
" Conventions: CF-1.6, ACDD-1.3, ISO 8601\n",
" title: RAMSSA v1.1 Analysed high resolution foundati...\n",
" summary: AMSR2-JAXA nobs=****** obsesd: avg=0.693 min=...\n",
" source: AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VII...\n",
" references: Beggs H., A. Zhong, G. Warren, O. Alves, G. B...\n",
" ... ...\n",
" geospatial_lat_max: 20.0\n",
" geospatial_lat_min: -70.0\n",
" geospatial_lon_max: 190.0\n",
" geospatial_lon_min: 60.0\n",
" geospatial_bounds: POLYGON((-70 60, 20 60, 20 190, -70 190, -70 ...\n",
" geospatial_bounds_crs: EPSG:4326
"
],
"text/plain": [
":Curve [time] (analysed_sst)"
]
},
"execution_count": 36,
"metadata": {
"application/vnd.holoviews_exec.v0+json": {}
},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"mds[sst_var].sel(lat=-42, lon=150, method=\"nearest\").hvplot()"
]
},
{
"cell_type": "markdown",
"id": "1c89076b-480b-4a04-b7ca-c582786879fb",
"metadata": {
"slideshow": {
"slide_type": "slide"
},
"tags": []
},
"source": [
"## Zarr - a cloud-optimised data format\n",
"\n",
"Zarr is a relatively new data format specifically developed for efficient access to multi-dimensional data in the cloud. Each dataset is broken up into many smaller files containing \"chunks\" of the data, organised in a standard hierarchy. The metadata are stored in separate files. When reading such a dataset, only the required information is read for each operation."
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "3b0aabac-3b27-4a93-8c68-8d8004c719e2",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
"
],
"text/plain": [
"<xarray.Dataset>\n",
"Dimensions: (time: 178, lat: 4500, lon: 6000)\n",
"Coordinates:\n",
" * lat (lat) float32 19.99 19.97 19.95 ... -69.97 -69.99\n",
" * lon (lon) float32 70.01 70.03 70.05 ... 190.0 190.0\n",
" * time (time) datetime64[ns] 2021-01-01T15:20:00 ... 20...\n",
"Data variables:\n",
" dt_analysis (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>\n",
" l2p_flags (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>\n",
" quality_level (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>\n",
" satellite_zenith_angle (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>\n",
" sea_surface_temperature (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>\n",
" sses_bias (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>\n",
" sses_count (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>\n",
" sses_standard_deviation (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>\n",
"Attributes: (12/47)\n",
" Conventions: CF-1.6\n",
" Metadata_Conventions: Unidata Dataset Discovery v1.0\n",
" Metadata_Link: TBA\n",
" acknowledgment: Any use of these data requires the following ...\n",
" cdm_data_type: grid\n",
" comment: HRPT AVHRR experimental L3 retrieval produced...\n",
" ... ...\n",
" summary: Skin SST retrievals produced from stitching t...\n",
" time_coverage_end: 20210101T151752Z\n",
" time_coverage_start: 20210101T095824Z\n",
" title: IMOS L3S Nighttime gridded multiple-sensor mu...\n",
" uuid: 4d02ee75-876d-4ff0-8956-ab68917c9001\n",
" westernmost_longitude: 70.01000213623047\n",
"
"
],
"text/plain": [
"<xarray.DataArray 'sea_surface_temperature' (time: 178, lat: 4500, lon: 6000)>\n",
"dask.array<open_dataset-5ed7711d911bd370bffa8b7f9a31ccf1sea_surface_temperature, shape=(178, 4500, 6000), dtype=float32, chunksize=(64, 64, 64), chunktype=numpy.ndarray>\n",
"Coordinates:\n",
" * lat (lat) float32 19.99 19.97 19.95 19.93 ... -69.95 -69.97 -69.99\n",
" * lon (lon) float32 70.01 70.03 70.05 70.07 ... 189.9 189.9 190.0 190.0\n",
" * time (time) datetime64[ns] 2021-01-01T15:20:00 ... 2021-07-25T15:20:00\n",
"Attributes:\n",
" _Netcdf4Dimid: 2\n",
" comment: The skin temperature of the ocean at a depth of approxima...\n",
" long_name: sea surface skin temperature\n",
" standard_name: sea_surface_skin_temperature\n",
" units: kelvin\n",
" valid_max: 32767\n",
" valid_min: -32767\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" site qc_val\n",
"time \n",
"2015-01-02 07:30:00+00:00 Geographe Bay 24.3728\n",
"2015-01-02 07:00:00+00:00 Geographe Bay 24.3728\n",
"2015-01-02 06:30:00+00:00 Geographe Bay 24.3238\n",
"2015-01-02 06:00:00+00:00 Geographe Bay 24.2518\n",
"2015-01-02 05:30:00+00:00 Geographe Bay 24.1798"
]
},
"execution_count": 49,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Here's the path to the dataset on AWS S3\n",
"parquet_path = \"s3://rimrep-data-public/091-aims-sst/test-50-64-spatialpart/\"\n",
"\n",
"# Let's see if there are any temperature loggers near us (in Dunsborough, Western Australia)\n",
"filters = [('lon', '>', 114.5),\n",
" ('lon', '<', 115.5),\n",
" ('lat', '>', -34.),\n",
" ('lat', '<', -33.)]\n",
"\n",
"df = dd.read_parquet(parquet_path,\n",
" filters=filters,\n",
" # only read the site names and QC'd temperature values\n",
" columns = ['site', 'qc_val'],\n",
" index='time',\n",
" engine='pyarrow',\n",
" storage_options = {\"anon\": True}\n",
" )\n",
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": 50,
"id": "74a58bb0-6d9d-4b84-9ec2-677792a2a25f",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"94896"
]
},
"execution_count": 50,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(df)"
]
},
{
"cell_type": "code",
"execution_count": 51,
"id": "39a28be2-8a69-4b9d-b394-2f8feb432293",
"metadata": {
"slideshow": {
"slide_type": "fragment"
},
"tags": []
},
"outputs": [],
"source": [
"# This subset should fit into memory, so let's turn it into a regular DataFrame\n",
"df = df.compute()"
]
},
{
"cell_type": "code",
"execution_count": 52,
"id": "6a2f66ff-fe9b-4218-b7bc-da31614478eb",
"metadata": {
"slideshow": {
"slide_type": "fragment"
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"array(['Geographe Bay', 'Cowaramup Bay', 'Canal Rocks'], dtype=object)"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Let's see how many sites we have...\n",
"df.site.unique()"
]
},
{
"cell_type": "code",
"execution_count": 54,
"id": "0e7bea2d-e1e8-4a2b-8cbd-3dba7f5521a0",
"metadata": {
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | site | \n", "qc_val | \n", "
|---|---|---|
| time | \n", "\n", " | \n", " |
| 2015-01-02 07:30:00+00:00 | \n", "Geographe Bay | \n", "24.3728 | \n", "
| 2015-01-02 07:00:00+00:00 | \n", "Geographe Bay | \n", "24.3728 | \n", "
| 2015-01-02 06:30:00+00:00 | \n", "Geographe Bay | \n", "24.3238 | \n", "
| 2015-01-02 06:00:00+00:00 | \n", "Geographe Bay | \n", "24.2518 | \n", "
| 2015-01-02 05:30:00+00:00 | \n", "Geographe Bay | \n", "24.1798 | \n", "
"
],
"text/plain": [
":NdOverlay [site]\n",
" :Curve [time] (qc_val)"
]
},
"execution_count": 54,
"metadata": {
"application/vnd.holoviews_exec.v0+json": {}
},
"output_type": "execute_result"
}
],
"source": [
"# Now we can plot the temperature timeseries for all these sites\n",
"df.hvplot(by='site')"
]
},
{
"cell_type": "markdown",
"id": "8ec4f51b-2293-4d91-a3fc-48d7fcdbee7e",
"metadata": {
"slideshow": {
"slide_type": "skip"
},
"tags": []
},
"source": [
"### Alternative dataset\n",
"\n",
"Another Parquet example using data from the Ocean Biodiversity Information System (OBIS) is shown in [this notebook](https://github.com/MathewBiddle/globe/blob/main/organismQuantity_check.ipynb)"
]
},
{
"cell_type": "markdown",
"id": "98aa13e5-5f12-4895-936a-3dff24cf4819",
"metadata": {
"slideshow": {
"slide_type": "slide"
},
"tags": [
"TODO",
"DRAFT"
]
},
"source": [
"# Other methods\n",
"\n",
"## ERDDAP\n",
"* Supports searching, subsetting, and downloads in a wide range of formats\n",
"* [Example](https://ioos.github.io/ioos_code_lab/content/code_gallery/data_access_notebooks/2016-10-12-fetching_data.html?highlight=erddap)\n",
"* Also covered in [this OHW22 tutorial](https://github.com/oceanhackweek/ohw-tutorials/blob/OHW22/01-Tue/00-data-access-in-python/00-data-access.ipynb)\n",
"\n",
"## New OGC APIs\n",
"* [New standards](https://ogcapi.ogc.org/#standards) from the Open Geospatial Consortium\n",
"* OGC Features (replacement for WFS) - [example](https://github.com/aodn/rimrep-examples/blob/main/Python_based_scripts/ogcfeatures.ipynb)\n",
"* OGC Coverages [example](https://github.com/aodn/rimrep-examples/blob/main/Python_based_scripts/ogccoverages.ipynb)\n",
"\n",
"## OGC Web Map Service (WMS)\n",
"* Also covered in [this OHW22 tutorial](https://github.com/oceanhackweek/ohw-tutorials/blob/OHW22/01-Tue/00-data-access-in-python/00-data-access.ipynb)\n",
"\n",
"## Further resources\n",
"* Lots of data access examples at [IOOS CodeLab](https://ioos.github.io/ioos_code_lab/content/code_gallery/data_access.html)\n",
"* Examples in both Python and R from [Reef 2050 Integrated Monitoring and Reporting Program Data Management System (RIMReP DMS)](https://github.com/aodn/rimrep-examples)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.16"
},
"toc-autonumbering": false,
"toc-showcode": false,
"toc-showmarkdowntxt": false,
"toc-showtags": true
},
"nbformat": 4,
"nbformat_minor": 5
}