{"id":754,"date":"2011-02-18T19:42:58","date_gmt":"2011-02-19T00:42:58","guid":{"rendered":"http:\/\/lukemiller.org\/?p=754"},"modified":"2013-05-19T15:34:10","modified_gmt":"2013-05-19T19:34:10","slug":"accessing-noaa-tide-data-with-r","status":"publish","type":"post","link":"https:\/\/lukemiller.org\/index.php\/2011\/02\/accessing-noaa-tide-data-with-r\/","title":{"rendered":"Accessing NOAA tide data with R"},"content":{"rendered":"
\"\"<\/a>
Boston Harbor tides for December 2010, retrieved from NOAA CO-OPS server.<\/figcaption><\/figure>\n

The National Oceanic and Atmospheric Administration makes the data from their many tide monitoring stations around the continent available for download. One way to access these data is through NOAA’s Center for Operational Oceanographic Products and Services (CO-OPS)<\/a> website, which provides several access methods including an OPeNDAP server. The OPeNDAP server allows you to construct a fairly simple URL query to submit in a web browser, and it will return the requested data as ascii text. The basics of making an OPeNDAP query are explained here<\/a>, but you can probably just figure out how it works from the code below and by visiting the CO-OPS OPeNDAP server linked above. Some NOAA tide stations also record water temperature, air temperature, wind speed, wind direction, and conductivity, all of which could be accessed through the OPeNDAP server.<\/p>\n

I wrote the following R script to build up an OPeNDAP query, submit it to the CO-OPS server using the RCurl <\/a>package, and then parse the resulting text into a data frame that can be saved to disk. The R script is available in my GitHub repository<\/a>. Either copy that script to your clipboard (there’s a button to do so) and save it on your computer, or navigate to the main repository page (Misc_R_scripts<\/a>) and download everything as a zip file. When you’re downloading 6-minute verified tide height data, the server will only let you retrieve 1 month’s worth of data at a time. You can only access 6-minute verified data that is older than 2 months.<\/p>\n

To find your local tide station, try the http:\/\/opendap.co-ops.nos.noaa.gov\/stations\/index.jsp<\/a> site or the main NOAA Tides and Currents webpage<\/a>.<\/p>\n

Edit July 2011: I have also put together a stripped-down version of the above script that will automatically step through a set of specified years and download the tide data for each month into separate files. That script is available on GitHub as well<\/a>. If you want to get several years’ worth of data all at once for a single site, use that script.<\/p>\n

The script:<\/p>\n

\n
\n
# coops_tide_ht_retrieval.R<\/span>\r\n# Author: Luke Miller Feb 2011<\/span>\r\n################################################################################<\/span>\r\n# Run this script as-is. It will prompt you to enter the appropriate<\/span>\r\n# station and date values when needed. You can only request less than 1 month of <\/span>\r\n# data at a time, so if you get an error returned, try asking for slightly less<\/span>\r\n# data.<\/span>\r\n\r\n# This script will download a set of verified tide height data from a NOAA <\/span>\r\n# CO-OPS DODS\/OPeNDAP server and parse it into a data frame to be saved to disk <\/span>\r\n#<\/span>\r\n# Station list: http:\/\/opendap.co-ops.nos.noaa.gov\/stations\/index.jsp<\/span>\r\n# Scroll through that list to find your station ID. <\/span>\r\n\r\n# OPeNDAP server gateway: http:\/\/opendap.co-ops.nos.noaa.gov\/dods\/<\/span>\r\n# The gateway has links to several other data types available including <\/span>\r\n# water temperature, air temperature, wind etc. <\/span>\r\n# See http:\/\/docs.opendap.org\/index.php\/UserGuideOPeNDAPMessages for info on<\/span>\r\n# structuring OPeNDAP queries.<\/span>\r\n#<\/span>\r\n# Six-minute water level data can only be retrieved 1 month at a time. Other<\/span>\r\n# fields such as water temperature could return up to 1 year at a time. This <\/span>\r\n# script only deals with 6-minute tide height data, but could serve as a guide <\/span>\r\n# for accessing other data types from NOAA CO-OPS. <\/span>\r\n# The tide height is reported in meters and time in UTC (Greenwich).<\/span>\r\n\r\n###############################################################################<\/span>\r\nrequire<\/span><\/a>(<\/span>RCurl<\/span><\/a>)<\/span>\r\n\r\nnoquote<\/span><\/a>(<\/span>print<\/span><\/a>(<\/span>\"Enter NOAA tide station number (i.e. Monterey = 9413450): \"<\/span>)<\/span>)<\/span>\r\nstation = scan<\/span><\/a>(<\/span>\"\"<\/span>,<\/span>what = character<\/span><\/a>(<\/span>)<\/span>,<\/span>nlines = 1<\/span>)<\/span>\r\nnoquote<\/span><\/a>(<\/span>print<\/span><\/a>(<\/span>\"Enter start date (format: 20080123 = Jan 23, 2008): \"<\/span>)<\/span>)<\/span>\r\nstartdate = scan<\/span><\/a>(<\/span>\"\"<\/span>,<\/span>what = character<\/span><\/a>(<\/span>)<\/span>,<\/span>nlines = 1<\/span>)<\/span> #get one line of values<\/span>\r\nnoquote<\/span><\/a>(<\/span>print<\/span><\/a>(<\/span>\"Enter end date (format: 20081231 = Dec 31, 2008): \"<\/span>)<\/span>)<\/span>\r\nenddate = scan<\/span><\/a>(<\/span>\"\"<\/span>,<\/span>what = character<\/span><\/a>(<\/span>)<\/span>,<\/span>nlines = 1<\/span>)<\/span>\r\n\r\n#OPeNDAP query for 6-minute verified water level looks like this (on 1 line):<\/span>\r\n#http:\/\/opendap.co-ops.nos.noaa.gov\/dods\/IOOS\/<\/span>\r\n#SixMin_Verified_Water_Level.ascii?<\/span>\r\n#WATERLEVEL_6MIN_VFD_PX._STATION_ID,<\/span>\r\n#WATERLEVEL_6MIN_VFD_PX._DATUM,<\/span>\r\n#WATERLEVEL_6MIN_VFD_PX.DATE_TIME,<\/span>\r\n#WATERLEVEL_6MIN_VFD_PX.WL_VALUE,<\/span>\r\n#WATERLEVEL_6MIN_VFD_PX.I,<\/span>\r\n#WATERLEVEL_6MIN_VFD_PX.F,<\/span>\r\n#WATERLEVEL_6MIN_VFD_PX.R,<\/span>\r\n#WATERLEVEL_6MIN_VFD_PX.T<\/span>\r\n#&WATERLEVEL_6MIN_VFD_PX._STATION_ID=%229449880%22<\/span>\r\n#&WATERLEVEL_6MIN_VFD_PX._DATUM=%22MLLW%22<\/span>\r\n#&WATERLEVEL_6MIN_VFD_PX._BEGIN_DATE=%2220080801%22<\/span>\r\n#&WATERLEVEL_6MIN_VFD_PX._END_DATE=%2220080808%22<\/span>\r\n\r\n#####################################################<\/span>\r\n######## DON'T CHANGE ANYTHING BELOW THIS LINE ####################<\/span>\r\n#The parts of the url to be assembled:<\/span>\r\nurl1 = \"http:\/\/opendap.co-ops.nos.noaa.gov\/dods\/IOOS\/\"<\/span>\r\nurl2 = \"SixMin_Verified_Water_Level.ascii?\"<\/span>\r\nurl3 = \"WATERLEVEL_6MIN_VFD_PX._STATION_ID,\"<\/span> #return stationId<\/span>\r\nurl4 = \"WATERLEVEL_6MIN_VFD_PX._DATUM,\"<\/span> #return datum<\/span>\r\nurl5 = \"WATERLEVEL_6MIN_VFD_PX.DATE_TIME,\"<\/span> #return record date-time<\/span>\r\nurl6 = \"WATERLEVEL_6MIN_VFD_PX.WL_VALUE,\"<\/span> #return water level value<\/span>\r\nurl7 = \"WATERLEVEL_6MIN_VFD_PX.I,\"<\/span> #return quality flag<\/span>\r\nurl8 = \"WATERLEVEL_6MIN_VFD_PX.F,\"<\/span> #return quality flag<\/span>\r\nurl9 = \"WATERLEVEL_6MIN_VFD_PX.R,\"<\/span> #return quality flag<\/span>\r\nurl10 = \"WATERLEVEL_6MIN_VFD_PX.T\"<\/span> #return quality flag<\/span>\r\n#The remaining parts of the url specify how to filter the data on the server <\/span>\r\n#to only retrieve the desired station and date range. Values must be enclosed<\/span>\r\n#in ascii double-quotes, which are represented by the code %22<\/span>\r\n# Do not change any values here, do not insert your own values here.<\/span>\r\nurl11 = \"&WATERLEVEL_6MIN_VFD_PX._STATION_ID=%22\"<\/span> # station ID goes here<\/span>\r\nurl12 = \"%22\"<\/span>\r\nurl13 = \"&WATERLEVEL_6MIN_VFD_PX._DATUM=%22MLLW%22\"<\/span>#we want MLLW as the datum<\/span>\r\nurl14 = \"&WATERLEVEL_6MIN_VFD_PX._BEGIN_DATE=%22\"<\/span> # start date gets put in here<\/span>\r\nurl15 = \"%22\"<\/span>\r\nurl16 = \"&WATERLEVEL_6MIN_VFD_PX._END_DATE=%22\"<\/span> # end date gets put in here<\/span>\r\nurl17 = \"%22\"<\/span>\r\n####### DON'T CHANGE ANYTHING ABOVE THIS LINE ############<\/span>\r\n##############################################<\/span>\r\n\r\n#Assemble the URL<\/span>\r\nurltotal = paste<\/span><\/a>(<\/span>url1,<\/span>url2,<\/span>url3,<\/span>url4,<\/span>url5,<\/span>url6,<\/span>url7,<\/span>url8,<\/span>url9,<\/span>url10,<\/span>url11,<\/span>\r\n\t\tstation,<\/span>url12,<\/span>url13,<\/span>url14,<\/span>startdate,<\/span>url15,<\/span>url16,<\/span>enddate,<\/span>url17,<\/span>sep =\"\"<\/span>)<\/span>\r\n\r\n#Download the data<\/span>\r\ncat<\/span><\/a>(<\/span>\"Contacting server...\\n<\/span>\"<\/span>)<\/span>;<\/span> flush.console<\/span><\/a>(<\/span>)<\/span>\r\ndat = getURL(<\/span>urltotal)<\/span> #use RCurl to retrieve text into a vector 'dat'<\/span>\r\ncat<\/span><\/a>(<\/span>\"Data returned...\\n<\/span>\"<\/span>)<\/span>;<\/span> flush.console<\/span><\/a>(<\/span>)<\/span>\r\nSys.sleep<\/span><\/a>(<\/span>2<\/span>)<\/span> #If you access data in a loop, be courteous and give the server<\/span>\r\n#a short break between requests<\/span>\r\n#cleanup<\/span>\r\nrm<\/span><\/a>(<\/span>url1,<\/span>url2,<\/span>url3,<\/span>url4,<\/span>url5,<\/span>url6,<\/span>url7,<\/span>url8,<\/span>url9,<\/span>url10,<\/span>url11,<\/span>url12,<\/span>url13,<\/span>url14)<\/span>\r\nrm<\/span><\/a>(<\/span>url15,<\/span>url16,<\/span>url17)<\/span>\r\n\r\ncon = textConnection<\/span><\/a>(<\/span>dat)<\/span> #create text Connection to dat vector<\/span>\r\nall.lines = readLines<\/span><\/a>(<\/span>con)<\/span> #read lines of text into separate slots in a vector<\/span>\r\nclose<\/span><\/a>(<\/span>con)<\/span> #close connection to dat vector<\/span>\r\n\r\nif<\/span> (<\/span>length<\/span><\/a>(<\/span>grep<\/span><\/a>(<\/span>'^Error'<\/span>,<\/span>all.lines)<\/span>)<\/span>><\/span>0<\/span>)<\/span> {<\/span> #check for error in retrieval<\/span>\r\n\tcat<\/span><\/a>(<\/span>\"There was an error...\\n<\/span>\"<\/span>)<\/span>\r\n\tcat<\/span><\/a>(<\/span>dat,<\/span>\"\\n<\/span>\"<\/span>)<\/span> #print contents of dat to show error<\/span>\r\n\tflush.console<\/span><\/a>(<\/span>)<\/span>\r\n}<\/span> else<\/span> {<\/span> #retrieval was successful, parse the text<\/span>\r\n\r\n\t#The column headers are typically preceded by a line of dashes<\/span>\r\n\theaderlines = grep<\/span><\/a>(<\/span>\"^--------\"<\/span>,<\/span>all.lines)<\/span> #find index of headers (-1)<\/span>\r\n\r\n\t#read column header names into a vector<\/span>\r\n\tcon = textConnection<\/span><\/a>(<\/span>dat)<\/span>\r\n\theaders = scan<\/span><\/a>(<\/span>con,<\/span> skip = headerlines,<\/span> nlines = 1<\/span>,<\/span> sep = \",\"<\/span>,<\/span>\r\n\t\t\twhat = \"character\"<\/span>,<\/span> strip.white = TRUE<\/span>)<\/span>\r\n\tclose<\/span><\/a>(<\/span>con)<\/span>\r\n\r\n\t#read rest of the data into a data frame 'df'<\/span>\r\n\tcon = textConnection<\/span><\/a>(<\/span>dat)<\/span>\r\n\tdf<\/span><\/a> = read.table<\/span><\/a>(<\/span>con,<\/span> skip = headerlines+<\/span>1<\/span>,<\/span> sep = \",\"<\/span>,<\/span> header = FALSE<\/span>,<\/span>\r\n\t\t\tquote<\/span><\/a> = \"\\\"<\/span>\"<\/span>,<\/span> col.names = headers,<\/span> strip.white = TRUE<\/span>,<\/span>\r\n\t\t\tstringsAsFactors = FALSE<\/span>)<\/span>\r\n\tclose<\/span><\/a>(<\/span>con)<\/span>\r\n\r\n\t###########################################################################<\/span>\r\n\t#The following operations will need to be altered if you change the <\/span>\r\n\t#fields or data type being returned by the OPeNDAP server<\/span>\r\n\r\n\t#Convert the time column to POSIX time (seconds since 1970-01-01 00:00:00)<\/span>\r\n\tdf<\/span><\/a>[<\/span>,<\/span>3<\/span>]<\/span> = as.POSIXct<\/span><\/a>(<\/span>strptime(<\/span>df<\/span><\/a>[<\/span>,<\/span>3<\/span>]<\/span>,<\/span>format<\/span><\/a> = \"%b %d %Y %I:%M%p\"<\/span>,<\/span>\r\n\t\t\t\t\ttz = \"GMT\"<\/span>)<\/span>)<\/span>\r\n\r\n\t#Give the columns shorter names<\/span>\r\n\tnames<\/span><\/a>(<\/span>df<\/span><\/a>)<\/span> = c<\/span><\/a>(<\/span>\"stationId\"<\/span>,<\/span>\"datum\"<\/span>,<\/span>\"TimeUTC\"<\/span>,<\/span>\"TideHT\"<\/span>,<\/span>\"Flag.Inferred\"<\/span>,<\/span>\r\n\t\t\t\"Flag.Flat.Tol\"<\/span>,<\/span>\"Flag.Rate.Tol\"<\/span>,<\/span>\"Flag.Temp.Tol\"<\/span>)<\/span>\r\n\r\n\t#Uncomment this if you want to plot the data<\/span>\r\n#\tplot(df$TimeUTC, df$TideHT, type = \"l\",<\/span>\r\n#\t\t\txlab = \"Date\",ylab = \"Tide Height, meters\")<\/span>\r\n\r\n\t#Save data automatically to a .csv file. <\/span>\r\n\tfilename = paste<\/span><\/a>(<\/span>\"Station_\"<\/span>,<\/span>station,<\/span>\"_\"<\/span>,<\/span>startdate,<\/span>\"-\"<\/span>,<\/span>enddate,<\/span>\r\n\t\t\t\".csv\"<\/span>,<\/span>sep = \"\"<\/span>)<\/span> #make file name<\/span>\r\n\twrite.csv<\/span><\/a>(<\/span>df<\/span><\/a>,<\/span>filename,<\/span>row.names<\/span><\/a> = FALSE<\/span>,<\/span> quote<\/span><\/a> = FALSE<\/span>)<\/span> #write file to disk<\/span>\r\n\tcat<\/span><\/a>(<\/span>\"Saved to \"<\/span>,<\/span>getwd<\/span><\/a>(<\/span>)<\/span>,<\/span>\"\/\"<\/span>,<\/span>filename,<\/span>\"\\n<\/span>\"<\/span>,<\/span>sep = \"\"<\/span>)<\/span>\r\n\tflush.console<\/span><\/a>(<\/span>)<\/span>\r\n\r\n\t#Alternate file save method lets user specify file name at run time<\/span>\r\n\t#Uncomment this if you wish to use it instead of the automated file <\/span>\r\n\t#output above<\/span>\r\n#\twrite.csv(df,file.choose(),row.names = FALSE, quote = FALSE)<\/span>\r\n\r\n\t#cleanup<\/span>\r\n\trm<\/span><\/a>(<\/span>dat,<\/span>con,<\/span>all.lines,<\/span>startdate,<\/span>enddate,<\/span>filename,<\/span>headerlines,<\/span> headers)<\/span>\r\n\r\n}<\/span> #end of if-else statement<\/span><\/pre>\n<\/div>\n<\/div>\n
Created by Pretty R at inside-R.org<\/a><\/p>\n
I stumbled across the Pretty R site at some point on http:\/\/www.r-bloggers.com<\/a>.<\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
Here’s some R code that can be used to download archived tide height data from NOAA’s CO-OPS OPeNDAP server. The code makes use of RCurl to send a URL query to the server, and then splits apart the resulting data into a data frame. <\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[218],"tags":[65,64,63,7,58,62,34],"class_list":["post-754","post","type-post","status-publish","format-standard","hentry","category-r-project","tag-historical-data","tag-noaa","tag-opendap","tag-r","tag-r-project","tag-tide-height","tag-water-temperature"],"_links":{"self":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts\/754","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/comments?post=754"}],"version-history":[{"count":28,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts\/754\/revisions"}],"predecessor-version":[{"id":2606,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts\/754\/revisions\/2606"}],"wp:attachment":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/media?parent=754"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/categories?post=754"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/tags?post=754"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}