{"id":2607,"date":"2021-11-10T11:27:42","date_gmt":"2021-11-10T19:27:42","guid":{"rendered":"https:\/\/lukemiller.org\/?p=2607"},"modified":"2021-11-10T11:27:42","modified_gmt":"2021-11-10T19:27:42","slug":"accessing-noaa-tide-data-with-r-updated-2021","status":"publish","type":"post","link":"https:\/\/lukemiller.org\/index.php\/2021\/11\/accessing-noaa-tide-data-with-r-updated-2021\/","title":{"rendered":"Accessing NOAA tide data with R (updated 2021)"},"content":{"rendered":"\n

It’s been a decade since I wrote a script<\/a> to retrieve observed tide height data from NOAA’s CO-OPS server, which lets you grab up to a month of tide data for a given tide station at a time. In that time, they’ve migrated over to a new server and slightly different API, so it’s probably time to update the script. <\/p>\n\n\n\n

The updated version of this script<\/a> is available in my miscellaneous public R Github as “coops_tide_ht_retrieval_monthly_v3.R”. The current version is copied below. To use it, you’ll need to install the ‘curl’ package in R, and you’ll need to figure out the station number of the tide monitoring station. You can find that at http:\/\/opendap.co-ops.nos.noaa.gov\/stations\/index.jsp<\/a> . Once you have your station number and the year(s) that you want to retrieve, enter them in the lines below where it says ENTER YOUR STATION ID AND YEARS HERE (around lines 25-28). Then highlight the whole script and run it. The result will be a series of monthly csv files in your current working directory, with 6 minute water level observations (height in meters), with time stamps in UTC (Greenwich Mean Time), and a series of 4 data quality flags for each observation that should hopefully all be ‘0’ if the data quality is sufficient. <\/p>\n\n\n\n

# coops_tide_ht_retrieval_monthly_v3.R\n# Author: Luke Miller July 2011 - updated 2021\n################################################################################\n# This script will download a set of verified tide height data from a NOAA \n# CO-OPS ERDDAP server and parse it into a data frame to be saved to disk \n#\n# Station list: http:\/\/opendap.co-ops.nos.noaa.gov\/stations\/index.jsp\n# Current server link\n# https:\/\/opendap.co-ops.nos.noaa.gov\/erddap\/tabledap\/IOOS_SixMin_Verified_Water_Level.html\n# Old OPeNDAP server gateway: http:\/\/opendap.co-ops.nos.noaa.gov\/dods\/\n# The gateway has links to several other data types available including \n# water temperature, air temperature, wind etc. \n#\n# Six-minute water level data can only be retrieved 1 month at a time. Other\n# fields such as water temperature can return up to 1 year at a time.\n# The tide height is reported in meters.\n\n###############################################################################\nrequire(curl) # If you don't have this, type install.packages(\"curl\") at the \n\t\t\t   # R command line before running this script.\nrequire(chron) # for the leap.year function\n\n################################################################################\n## ENTER YOUR STATION ID AND YEARS HERE ##\nstation = 9451600 #Enter your desired station ID here\nyear = 2018  #Enter the first year of data to get here\nyear2 = 2018 #Enter the last year of data to get here (can be same as 1st year)\n################################################################################\n\n# Example station ID's. \n# See http:\/\/opendap.co-ops.nos.noaa.gov\/stations\/index.jsp for a list of all\n# stations\n#Monterey, CA \t\t\t\t9413450\n#La Jolla, CA\t\t\t\t9410230\n#Los Angeles\t\t\t\t9410660 (water) 9410647(air\/wind)\n#San Francisco\t\t\t\t9414290\n#Point Arena, CA\t\t\t9416841\n#Crescent City, CA\t\t\t9419750\n#Port Orford, OR\t\t\t9431647\n#Charleston, OR\t\t\t\t9432780\n#Yaquina River, Newport OR\t9435380\n#Toke Point, WA\t\t\t\t9440910\n#Neah Bay, WA\t\t\t\t9443090\n#Sitka, AK\t\t\t\t\t9451600\n\n\nfor (yr in year:year2) {\n\tleap = leap.year(yr) #test if desired year is a leap year\n\t\n\tfor (mo in 1:12) { #start of mo for-loop\n\t\t\n\t\t#create text string for month value\n\t\tif (mo < 10) {month = paste(\"0\",as.character(mo),sep=\"\")} else {\n\t\t\tmonth = as.character(mo)\n\t\t}\n\t\t#figure out number of days in month\n\t\tif ((mo == 4) | (mo == 6) | (mo == 9) | (mo == 11)) {nday = 30} else {\n\t\t\tif (mo == 2 & leap == TRUE) {nday = 29} else {\n\t\t\t\tif (mo == 2 & leap == FALSE) {nday = 28} else nday = 31 }\n\t\t} \n\t\t\n\t\tstartdate = paste(yr,month,\"01\",sep = \"\")\n\t\tenddate = paste(yr,month,nday,sep = \"\")\n\t\t\n\t\t\n#ERDDAP query for 6-minute verified water level looks like this (on 1 line):\n#https:\/\/opendap.co-ops.nos.noaa.gov\/erddap\/tabledap\/\n#IOOS_SixMin_Verified_Water_Level.asc?\n#STATION_ID%2C\n#DATUM%2C\n#time%2C\n#WL_VALUE%2C\n#I%2CF%2CR%2CT&\n#STATION_ID=%229451600%22\n#&DATUM%3E=%22MLLW%22\n#&BEGIN_DATE%3E=%2220190101%22\n#&END_DATE%3E=%2220190131%22\n\n\n########################################################\n###### DON'T CHANGE ANY OF THE CODE BELOW THIS LINE ####\n#The parts of the url\n\t\turl1 = \"https:\/\/opendap.co-ops.nos.noaa.gov\/erddap\/tabledap\/\"\n\t\turl2 = \"IOOS_SixMin_Verified_Water_Level.asc?\"\n\t\turl3 = \"STATION_ID%2C\" #return stationId\n\t\turl4 = \"DATUM%2C\" #return datum\n\t\turl5 = \"time%2C\" #return record date-time\n\t\turl6 = \"WL_VALUE\" #return water level value\n\t\turl7 = \"%2CI\" #return quality flag\n\t\turl8 = \"%2CF\" #return quality flag\n\t\turl9 = \"%2CR\" #return quality flag\n\t\turl10 = \"%2CT\" #return quality flag\n#The remaining parts of the url specify how to filter the data on the server \n#to only retrieve the desired station and date range. Values must be enclosed\n#in ascii double-quotes, which are represented by the code %22\n\t\turl11 = \"&STATION_ID=%22\" #station gets added after this\n\t\turl12 = \"%22\"  # closing quote for station ID\n\t\turl13 = \"&DATUM=%22MLLW%22\"#we want MLLW as the datum\n\t\turl14 = \"&BEGIN_DATE%3E=%22\" #start date gets added here\n\t\turl15 = \"%22\"\n\t\turl16 = \"&END_DATE%3E=%22\" #end date gets added here\n\t\turl17 = \"%22\"\n##### DON'T CHANGE ANY CODE ABOVE THIS LINE ###########\n########################################################################\n\t\turltotal = paste(url1,url2,url3,url4,url5,url6,url7,url8,url9,url10,url11,\n\t\t\t\tstation,url12,url13,url14,startdate,url15,url16,enddate,url17,sep =\"\")\n\t\tcat(\"Contacting server...\\n\"); flush.console()\n\t\tcon = curl(urltotal) # Open a connection\n\t\tdat = readLines(con) # Read the returned data\n\t\tclose(con)\n\t\tcat(\"Data returned...\\n\"); flush.console()\n\t\tSys.sleep(2) #pause for a few seconds to avoid overloading server\n\t\t\n\t\t#cleanup\n\t\trm(url1,url2,url3,url4,url5,url6,url7,url8,url9,url10,url11,url12,url13,url14)\n\t\trm(url15,url16,url17)\n\t\t\n\t\tcon = textConnection(dat) #create text Connection to dat vector\n\t\tall.lines = readLines(con) #read lines of text into separate slots in a vector\n\t\tclose(con) #close connection to dat vector\n\t\t\n\t\tif (length(grep('^Error',all.lines))>0) { #check for error in retrieval\n\t\t\tcat(\"There was an error...\\n\")\n\t\t\tcat(dat,\"\\n\") #print contents of dat to show error\n\t\t\tflush.console()\n\t\t} else {\n\t\t\t#The column headers are typically preceded by a line of dashes\n\t\t\theaderlines = grep(\"^--------\",all.lines) #find index of headers (-1)\n\t\t\t\n\t\t\t#read column header names into a vector\n\t\t\tcon = textConnection(dat)\n\t\t\theaders = scan(con, skip = headerlines, nlines = 1, sep = \",\",\n\t\t\t\t\twhat = \"character\", strip.white = TRUE)\n\t\t\tclose(con)\n\t\t\t\n\t\t\t#read rest of the data into a data frame 'df'\n\t\t\tcon = textConnection(dat)\n\t\t\tdf = read.table(con, skip = headerlines+1, sep = \",\", header = FALSE,\n\t\t\t\t\tquote = \"\\\"\", col.names = headers, strip.white = TRUE,\n\t\t\t\t\tstringsAsFactors = FALSE)\n\t\t\tclose(con)\n\t\t\t\n\t\t\t###########################################################################\n\t\t\t#The following operations will need to be altered if you change the \n\t\t\t#fields or data type being returned by the OPeNDAP server\n\t\t\t\n\t\t\t#Convert the time column to POSIX time (seconds since 1970-01-01 00:00:00)\n\t\t\tdf[,3] = as.POSIXct(df[,3], tz = 'UTC', origin = '1970-1-1')\n\n\t\t\t\n\t\t\t#Give the columns shorter names\n\t\t\tnames(df) = c(\"stationId\",\"datum\",\"TimeUTC\",\"TideHT.m\",\"Flag.Inferred\",\n\t\t\t\t\t\"Flag.Flat.Tol\",\"Flag.Rate.Tol\",\"Flag.Temp.Tol\")\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t#Uncomment this if you want to plot the data\n\t\t\t#plot(df$TimeUTC, df$TideHT, type = \"l\",\n\t\t\t#\t\txlab = \"Date\",ylab = \"Tide Height, meters\")\n\t\t\t\n\t\t\t#Save data automatically to a .csv file. \n\t\t\tfilename = paste(\"Station_\",station,\"_tide_ht_\",startdate,\"-\",enddate,\n\t\t\t\t\t\".csv\",sep = \"\")\n\t\t\twrite.csv(df,filename,row.names = FALSE, quote = FALSE)\n\t\t\tcat(\"Saved to \",filename,\"\\n\")\n\t\t\tflush.console()\n\t\t\t\n\t\t\t#Alternate file save method lets user specify file name at run time\n\t\t\t#write.csv(df,file.choose(),row.names = FALSE, quote = FALSE)\n\t\t\t\n\t\t\t#cleanup\n\t\t\trm(dat,con,all.lines,startdate,enddate,filename,headerlines, headers,df,\n\t\t\t\t\turltotal)\n\t\t\t\n\t\t} #end of if-else statement\n\t\t\n\t} #end of mo for-loop\n\n} #end of yr for-loop\n\n###############################################################################\n###############################################################################\n\ncat(\"Finished\\n\\a\")\n\n\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
It’s been a decade since I wrote a script to retrieve observed tide height data from NOAA’s CO-OPS server, which lets you grab up to a month of tide data for a given tide station at a time. In that time, they’ve migrated over to a new server and slightly different API, so it’s probably […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[271,270,64,58,62],"class_list":["post-2607","post","type-post","status-publish","format-standard","hentry","category-journal","tag-co-ops","tag-erddap","tag-noaa","tag-r-project","tag-tide-height"],"_links":{"self":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts\/2607","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/comments?post=2607"}],"version-history":[{"count":2,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts\/2607\/revisions"}],"predecessor-version":[{"id":2609,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts\/2607\/revisions\/2609"}],"wp:attachment":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/media?parent=2607"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/categories?post=2607"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/tags?post=2607"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}