Technical Details As Acknowledgements

The gauge on my patio has collected over twelve inches of rain since the last week of June. So this week's extended break from the monsoon was welcome, and it provided an opportunity to document some topics neglected in the previous two posts. Also, suggestions and lessons learned might be helpful for anyone wanting to create their own custom weather maps. The first lesson learned is that the link I provided in the previous post quickly turned bad. Here are replacement links to brief summary descriptions about the GFS, which includes the GDAS. Those summaries are on the NOMADS page, which provides public access to gridded data from a variety of National Weather Service models, as well as links to data from other modeling centers. Separately the National Weather Service provides a Model Analysis and Guidance page, where the gridded data have already been processed into standard weather map images of many varieties. If those standard maps are adequate, it might not be necessary for a user to deal with the gridded data.

My custom journeys through the model data are spread out across the screen of my MacBook in the form of maps generated through the python packages matplotlib and Cartopy. Other python packages are involved, which I'll discuss later. I like to keep track of what I'm installing and why. So my starting point is the Homebrew package manager. The starting point for Homebrew is its installation instruction web page. After Homebrew is installed, installation of a particular piece of software under Homebrew is handled by a Formula. Among the Homebrew formulae I've installed and kept updated for a long time are python itself (the formula includes installation of pip) and ipython. Some python packages require access to a library, which can be installed with Homebrew. A tricky detail is that Cartopy requires an older version of the proj library, which happily Homebrew provides as formula proj@7. A recent discovery for me is the formula eccodes, which installs the ECMWF package for decoding GRIB files. This library is needed for a python package to be discussed later. Independent of its companion python package, eccodes installs a set of command line tools. One of these tools is grib_ls, which comes in handy for figuring out what is actually in the file that was downloaded from the NOMADS site.

I keep the standard python packages (cartopy, pandas, numpy, matplotlib, scipy, etc.) updated manually with pip. There is one more tricky thing with Cartopy. A required package, Shapely, needs to be installed using the pip option --no-binary. For easy reading of GRIB files, the python packages cfgrib (interfaces with the eccodes library) and xarray are needed. I came to cfgrib through xarray, and to xarray through the python package MetPy. While intending to eventually use all of MetPy's computational features, for now I'm especially taking advantage of its addition to the map features available in Cartopy (i.e., from metpy.plots import USCOUNTIES).

Here's the easy reading part, illustrated by a line from my python script:
ds = xr.open_dataset(myFilename, engine="cfgrib", filter_by_keys={'typeOfLevel': levelType[levelTypeIndex], 'level':level})
where xarray was imported as xr, and myFilename was set to one of the files I downloaded from NOMADS. The optional argument filter_by_keys gets passed to cfgrib. This filtering is generally necessary, and will be discusssed later.

Jumping to the end of the process for now, assume that a downloaded GRIB file, or at least a subset of it, has been successfully ingested by xarray. Then it's only a matter of adapting from the many MetPy examples available. My personal preference for just viewing the fields is to rely on two matplotlib functions: pcolormesh for the scalars and streamplot for the wind components. For some reason pcolormesh understands the lon/lat/variable inputs when they are supplied as xarray DataArray structures, but streamplot requires extracting the .values property in order to supply ordinary numpy arrays.

So let's back up in the process to discuss downloading. As mentioned near the beginning of this post, the NOMADS page provides access to a wide variety of model gridded fields. I've been routinely looking at the GDAS analysis and the GFS forecast fields on their 0.25 degree grid. Those are just 2 of the nearly 100 datasets listed on the NOMADS page. Most of the datasets are equipped with the grib filter option. Clicking on the link in the previous sentence takes you to the NOMADS page of general instructions. As the instruction page explains, grib filter is best used for creating regional subsets. Following the instructions and finally reaching the "Extract" page, I initially skip past the Extract Levels and Variables section, and so that I don't forget about it scroll down to the Extract Subregion section. It's not enough to just fill in the top/bottom/left/right lat/lon. Also be sure to check the box after the words "make subregion." Returning to the select variables section, if I want just the variables corresponding to standard radiosonde observations I'll select the abbreviated names HGT, TMP, RH, UGRD and VGRD. Depending on the dataset you chose, there might be many more variables. You may need to view Parameter Table Version 2 to figure out the abbreviations. For the levels desired it might be safest to just select "all". Instead I select each level that I want. Some of the level options are actually levelTypes, as can be verified later by running grib_ls on the downloaded file. The variable PWAT needs the level (levelType) "entire atmosphere (considered as a single layer)," while REFC needs "entire atmosphere."

Here is the promised later discussion about the filtering that is generally necessary in connection with xarray's opening of a grib file to create a dataset. For an explanation of why filtering is needed, see in this early documentation for CFGRIB the section Devil In The Details. Basically each message (i.e., each 2_D grid) in a file downloaded from one of the NOMADS "Extract" pages will be a unique combination of the three keys typeOfLevel/level/parameter. But xarray tries to merge all the levels and all the parameters in the file into a level/parameter array. In order to keep xarray happy, it's often enough to restrict xarray's attention to one typeOfLevel. But it may be necessary to also restrict attention to one level, as in my line of code above. An example is when TMP and RH are at level 2 m above ground but UGRD and VGRD are at level 10 m. Xarray with cfgrib's help tries to create a table with 2 levels and 4 parameters, but is disappointed to find that half of the table's entries would be empty. Working out what is in the GRIB file and thus what filtering is needed is where the command line tool grib_ls comes in handy.

Once into a routine of what I want to download, I follow the suggestion at the bottom of the grib filter instruction page, the section "Scripting file retrievals." However instead of a script I just enter the commands interactively in the terminal window. At first I was always copying from my text editor (BBEdit) and pasting to the terminal window, with the first paste being the mutiple lines where I change to my local directory and set the date and time parameters, and the second paste being the curl line. But then I learned to just use the terminal window: ctrl-r to search the history file for text in the last use of the multi-line command, move with arrow keys and make minor edits, ctrl-o to execute that multi-line and bring up the next history line, which is the curl line ready to substitute the new ${hr} ${fhr} and ${date}. Hit return. It takes a little over 5 seconds to do the typing, and another 5 seconds for my 12 by 12 degree lat/lon regional subset 630 Kbyte file of 182 grib messages to download.