{"id":1536,"date":"2013-05-17T23:02:18","date_gmt":"2013-05-18T03:02:18","guid":{"rendered":"http:\/\/lukemiller.org\/?p=1536"},"modified":"2016-09-06T17:32:12","modified_gmt":"2016-09-07T00:32:12","slug":"interfacing-xtide-and-r","status":"publish","type":"post","link":"https:\/\/lukemiller.org\/index.php\/2013\/05\/interfacing-xtide-and-r\/","title":{"rendered":"Interfacing XTide and R"},"content":{"rendered":"

Edit: <\/strong>There is now a full-fledged R package, rtide<\/a>, to accomplish the same basic task of generating time series of tide predictions that is outlined here. See this more recent post<\/a> for information.<\/p>\n

XTide<\/a> is an open-source program that predicts tide heights and current speeds for hundreds of tide and current stations around the United States. It can be used to produce tide predictions in the past and future for a site at your chosen interval (down to the minute), as well as producing sunrise and sunset times, moon phases, and times when the tide level passes a chosen level mark. It has a user-friendly GUI version (xtide.exe), but also comes in a command line version (tide.exe) that can be interfaced with other programs like R.<\/p>\n

Getting R and XTide to talk to each other is a matter of producing a command line query to pass from R to XTide, and then using R to parse the results into a handy format for further manipulation. I have put together an example script that walks through the process, and shows how to parse the results into a data frame with time stamps. The script, xtide_interface.R<\/a>, is available in this GitHub repository<\/a>.<\/p>\n

\n
\n
# xtide_interface.R<\/span>\r\n# An example R script to show how to build XTide queries, submit them to the <\/span>\r\n# system command line, and then parse the results into a data frame. There are<\/span>\r\n# several different examples outlined below. <\/span>\r\n# Author: Luke Miller May 17, 2013<\/span>\r\n###############################################################################<\/span>\r\n# Site names: http:\/\/www.flaterco.com\/xtide\/locations.html <\/span>\r\n# List of command line flags: http:\/\/www.flaterco.com\/xtide\/settings.html<\/span>\r\n\r\n# Obviously XTide, or a port like tide.exe (for Windows) must be downloaded from <\/span>\r\n# the XTide site and installed. It is a standalone program, separate from R.<\/span>\r\n\r\n# In Windows, the XTide directory that contains the tide.exe program needs to be<\/span>\r\n# added to the system search PATH. It's easiest if you also put the <\/span>\r\n# harmonics.tcd file in the same directory as tide.exe.<\/span>\r\n###############################################################################<\/span>\r\n\r\n# Sys.setenv(TZ=\"UTC\") \t# Useful if you're getting data for sites in multiple <\/span>\r\n\t\t\t\t\t\t# time zones and need to standardize on one time zone.<\/span>\r\n\r\n# Start and end times, supplied as character strings<\/span>\r\nstartchar = '2013-05-16 16:00'<\/span>\r\nendchar = '2013-05-20 16:00'<\/span>\r\n# Site name, taken from http:\/\/www.flaterco.com\/xtide\/locations.html<\/span>\r\nsitename = 'Bodega Harbor entrance, California'<\/span>\r\n\r\n# Example switches for Xtide command line program<\/span>\r\n# -l\tGive the site name after this switch<\/span>\r\n# -b\tBeginning time, in the format YYYY-MM-DD HH:MM<\/span>\r\n# -e \tEnding time, in the format YYYY-MM-DD HH:MM<\/span>\r\n# -f \toutput format, c = csv<\/span>\r\n# -u \tunits, i.e. m or ft<\/span>\r\n# -em\tevent mask, suppress output of sunrise, sunset, moon phase etc<\/span>\r\n# -z \tUTC (zulu) time zone<\/span>\r\n# -ml\tmark level, only returns info in plain mode when tide crosses this level<\/span>\r\n# -m\toutput information: r = raw, m = medium rare, p = plain<\/span>\r\n#\t\traw mode returns the site, unix time code, and height<\/span>\r\n#\t\tmedium rare mode returns site, date, time, and height<\/span>\r\n#\t\tplain mode returns site, date, time, height, and event<\/span>\r\n# \t\t\ti.e. High Tide, Low Tide, mark level crossing, sunrise, sunset etc<\/span>\r\n\r\n################################################################################<\/span>\r\n# Example 1<\/span>\r\n# Create query to return high and low tide times only, with units in feet and<\/span>\r\n# using the current system time zone. <\/span>\r\ntidecommand = paste<\/span><\/a>(<\/span>'tide -l \"'<\/span>,<\/span>sitename,<\/span>'\" -b \"'<\/span>,<\/span>\r\n\t\tstartchar,<\/span> '\" -e \"'<\/span>,<\/span> endchar,<\/span>\r\n\t\t'\" -f c -em pSsMm -m p -u ft '<\/span>,<\/span> sep = ''<\/span>)<\/span>\r\n\r\nss = system<\/span><\/a>(<\/span>tidecommand,<\/span> intern = TRUE<\/span>)<\/span> #invoke tide.exe and return results<\/span>\r\n\r\n# Convert the character strings in 'ss' into a data frame<\/span>\r\nhilow = read.table<\/span><\/a>(<\/span>textConnection<\/span><\/a>(<\/span>ss)<\/span>,<\/span> sep = ','<\/span>,<\/span> colClasses = 'character'<\/span>)<\/span>\r\n# Add column names to the data frame<\/span>\r\nnames<\/span><\/a>(<\/span>hilow)<\/span> = c<\/span><\/a>(<\/span>'Site'<\/span>,<\/span>'Date'<\/span>,<\/span>'Hour'<\/span>,<\/span>'TideHt'<\/span>,<\/span>'Event'<\/span>)<\/span>\r\n# Combine the Date & Hour columns into a POSIX time stamp<\/span>\r\nhilow$Time = as.POSIXlt<\/span><\/a>(<\/span>paste<\/span><\/a>(<\/span>hilow$Date,<\/span>hilow$Hour)<\/span>,<\/span> \r\n\t\tformat<\/span><\/a> = \"%Y-%m-%d %I:%M %p\"<\/span>)<\/span>\r\n# Strip off the height units and convert tide height to numeric values<\/span>\r\nhilow$TideHt = as.numeric<\/span><\/a>(<\/span>gsub<\/span><\/a>(<\/span>'[ [:alpha:]]'<\/span>,<\/span>' '<\/span>,<\/span>hilow$TideHt)<\/span>)<\/span>\r\n\r\nhead<\/span><\/a>(<\/span>hilow)<\/span>\r\n\r\n################################################################################<\/span>\r\n# Example 2<\/span>\r\n# Create query to return high and low tide times, along with any time when <\/span>\r\n# the tide crosses the +2.0ft level. (High\/low tides are always included).<\/span>\r\n# Time values will be in the system time zone. Be careful of daylight savings<\/span>\r\n# transitions.<\/span>\r\ntidecommand = paste<\/span><\/a>(<\/span>'tide -l \"'<\/span>,<\/span>sitename,<\/span>'\" -b \"'<\/span>,<\/span>\r\n\t\tstartchar,<\/span> '\" -e \"'<\/span>,<\/span> endchar,<\/span>\r\n\t\t'\" -f c -em pSsMm -m p -ml 2.0ft -u ft '<\/span>,<\/span> sep = ''<\/span>)<\/span>\r\n\r\nss = system<\/span><\/a>(<\/span>tidecommand,<\/span> intern = TRUE<\/span>)<\/span> #invoke tide.exe and return results<\/span>\r\nss2 = grep<\/span><\/a>(<\/span>'Mark Falling'<\/span>,<\/span> ss)<\/span> #find lines with 'Mark Falling' <\/span>\r\n# Convert the character strings in 'ss' into a data frame<\/span>\r\nevents = read.table<\/span><\/a>(<\/span>textConnection<\/span><\/a>(<\/span>ss[<\/span>ss2]<\/span>)<\/span>,<\/span> sep = ','<\/span>,<\/span> \r\n\t\tcolClasses = 'character'<\/span>)<\/span>\r\n# Add column names to the data frame<\/span>\r\nnames<\/span><\/a>(<\/span>events)<\/span> = c<\/span><\/a>(<\/span>'Site'<\/span>,<\/span>'Date'<\/span>,<\/span>'Hour'<\/span>,<\/span>'TideHt'<\/span>,<\/span>'Event'<\/span>)<\/span>\r\n# Combine the Date & Hour columns into a POSIX time stamp<\/span>\r\nevents$Time = as.POSIXlt<\/span><\/a>(<\/span>paste<\/span><\/a>(<\/span>events$Date,<\/span>events$Hour)<\/span>,<\/span> \r\n\t\tformat<\/span><\/a> = \"%Y-%m-%d %I:%M %p\"<\/span>)<\/span>\r\n# Strip off the height units and convert tide height to numeric values<\/span>\r\nevents$TideHt = as.numeric<\/span><\/a>(<\/span>gsub<\/span><\/a>(<\/span>'[ [:alpha:]]'<\/span>,<\/span>' '<\/span>,<\/span>events$TideHt)<\/span>)<\/span>\r\n\r\nhead<\/span><\/a>(<\/span>events)<\/span>\r\n# Calculate time between each event.<\/span>\r\ndiff<\/span><\/a>(<\/span>events$Time)<\/span>\r\n\r\n################################################################################<\/span>\r\n# Example 3<\/span>\r\n# Create query for returning times of high, low tide, plus Sunrise and Sunset,<\/span>\r\n# all in UTC timezone, for your site. Height units are meters.<\/span>\r\ntidecommand = paste<\/span><\/a>(<\/span>'tide -l \"'<\/span>,<\/span>sitename,<\/span>'\" -b \"'<\/span>,<\/span>\r\n\t\tstartchar,<\/span> '\" -e \"'<\/span>,<\/span> endchar,<\/span> \r\n\t\t'\" -f c -em pMm -m p -u m -z'<\/span>,<\/span> sep = ''<\/span>)<\/span>\r\n\r\nss = system<\/span><\/a>(<\/span>tidecommand,<\/span> intern = TRUE<\/span>)<\/span> #invoke Xtide and return results<\/span>\r\n\r\n# Convert the character strings in 'ss' into a data frame<\/span>\r\nhilowSun = read.table<\/span><\/a>(<\/span>textConnection<\/span><\/a>(<\/span>ss)<\/span>,<\/span> sep = ','<\/span>,<\/span> colClasses = 'character'<\/span>)<\/span>\r\n# Add column names to the data frame<\/span>\r\nnames<\/span><\/a>(<\/span>hilowSun)<\/span> = c<\/span><\/a>(<\/span>'Site'<\/span>,<\/span>'Date'<\/span>,<\/span>'Hour'<\/span>,<\/span>'TideHt'<\/span>,<\/span>'Event'<\/span>)<\/span>\r\n# Combine the Date & Hour columns into a POSIX time stamp<\/span>\r\nhilowSun$Time = as.POSIXlt<\/span><\/a>(<\/span>paste<\/span><\/a>(<\/span>hilowSun$Date,<\/span>hilowSun$Hour)<\/span>,<\/span> \r\n\t\tformat<\/span><\/a> = \"%Y-%m-%d %I:%M %p\"<\/span>,<\/span> tz = \"UTC\"<\/span>)<\/span>\r\n# Strip off the height units and convert tide height to numeric values<\/span>\r\nhilowSun$TideHt = as.numeric<\/span><\/a>(<\/span>gsub<\/span><\/a>(<\/span>'[ [:alpha:]]'<\/span>,<\/span>' '<\/span>,<\/span>hilowSun$TideHt)<\/span>)<\/span>\r\n\r\n# Convert Time values to current R session time zone, overwriting old timestamps<\/span>\r\nhilowSun$LocalTime = c<\/span><\/a>(<\/span>hilowSun$Time)<\/span>\r\nhead<\/span><\/a>(<\/span>hilowSun)<\/span>\r\n\r\n################################################################################<\/span>\r\n# Example 4<\/span>\r\n# Create query to return tide height at fixed time intervals (10 minutes here),<\/span>\r\n# in UTC time zone, units of meters.<\/span>\r\ntidecommand = paste<\/span><\/a>(<\/span>'tide -l \"'<\/span>,<\/span>sitename,<\/span>'\" -b \"'<\/span>,<\/span>\r\n\t\tstartchar,<\/span> '\" -e \"'<\/span>,<\/span> endchar,<\/span>\r\n\t\t'\" -f c -m m -s 00:10 -u m -z'<\/span>,<\/span> sep = ''<\/span>)<\/span>\r\n\r\nss = system<\/span><\/a>(<\/span>tidecommand,<\/span> intern = TRUE<\/span>)<\/span> #invoke tide.exe and return results<\/span>\r\n# Convert the character strings in 'ss' into a data frame<\/span>\r\ntides = read.table<\/span><\/a>(<\/span>textConnection<\/span><\/a>(<\/span>ss)<\/span>,<\/span> sep = ','<\/span>,<\/span> colClasses = 'character'<\/span>)<\/span>\r\n# Add column names to the data frame<\/span>\r\nnames<\/span><\/a>(<\/span>tides)<\/span> = c<\/span><\/a>(<\/span>'Site'<\/span>,<\/span>'Date'<\/span>,<\/span>'Hour'<\/span>,<\/span>'TideHt'<\/span>)<\/span>\r\n# Combine the Date & Hour columns into a POSIX time stamp<\/span>\r\ntides$Time = as.POSIXlt<\/span><\/a>(<\/span>paste<\/span><\/a>(<\/span>tides$Date,<\/span>tides$Hour)<\/span>,<\/span> \r\n\t\tformat<\/span><\/a> = \"%Y-%m-%d %I:%M %p\"<\/span>,<\/span> tz = \"UTC\"<\/span>)<\/span>\r\n# Strip off the height units and convert tide height to numeric values<\/span>\r\ntides$TideHt = as.numeric<\/span><\/a>(<\/span>gsub<\/span><\/a>(<\/span>'[ [:alpha:]]'<\/span>,<\/span>' '<\/span>,<\/span>tides$TideHt)<\/span>)<\/span>\r\n# Create a column of time stamps in the current R session time zone<\/span>\r\ntides$LocalTime = c<\/span><\/a>(<\/span>tides$Time)<\/span>\r\nhead<\/span><\/a>(<\/span>tides)<\/span>\r\n\r\n################################################################################<\/span>\r\n# Example 5<\/span>\r\n# Create query to return times of high\/low tide, sunrise and sunset, in the <\/span>\r\n# local time zone. Extract the time for the Sunsets and place in a data frame<\/span>\r\ntidecommand = paste<\/span><\/a>(<\/span>'tide -l \"'<\/span>,<\/span>sitename,<\/span>'\" -b \"'<\/span>,<\/span>\r\n\t\tstartchar,<\/span> '\" -e \"'<\/span>,<\/span> endchar,<\/span>\r\n\t\t'\" -f c -em pMm -m p -u ft '<\/span>,<\/span> sep = ''<\/span>)<\/span>\r\n\r\nss = system<\/span><\/a>(<\/span>tidecommand,<\/span> intern = TRUE<\/span>)<\/span> #invoke tide.exe and return results<\/span>\r\n\r\nss2 = grep<\/span><\/a>(<\/span>'Sunset'<\/span>,<\/span>ss)<\/span> #find lines with Sunset<\/span>\r\n# Convert the character strings in 'ss' into a data frame<\/span>\r\nsunsets = read.table<\/span><\/a>(<\/span>textConnection<\/span><\/a>(<\/span>ss[<\/span>ss2]<\/span>)<\/span>,<\/span> sep = ','<\/span>,<\/span> \r\n\t\tcolClasses = 'character'<\/span>)<\/span>\r\n# Add column names to the data frame<\/span>\r\nnames<\/span><\/a>(<\/span>sunsets)<\/span> = c<\/span><\/a>(<\/span>'Site'<\/span>,<\/span>'Date'<\/span>,<\/span>'Hour'<\/span>,<\/span>'TideHt'<\/span>,<\/span>'Event'<\/span>)<\/span>\r\n# Combine the Date & Hour columns into a POSIX time stamp<\/span>\r\nsunsets$Time = as.POSIXlt<\/span><\/a>(<\/span>paste<\/span><\/a>(<\/span>sunsets$Date,<\/span>sunsets$Hour)<\/span>,<\/span> \r\n\t\tformat<\/span><\/a> = \"%Y-%m-%d %I:%M %p\"<\/span>)<\/span>\r\nhead<\/span><\/a>(<\/span>sunsets)<\/span><\/pre>\n<\/div>\n<\/div>\n
The output from the first example would look something like this (I chopped off the last column so it would fit here):<\/p>\n
                                Site       Date         Hour TideHt     Event\r\n1 Bodega Harbor entrance| California 2013-05-16  5:30 PM PDT   4.23 High Tide\r\n2 Bodega Harbor entrance| California 2013-05-16 11:16 PM PDT   2.82  Low Tide\r\n3 Bodega Harbor entrance| California 2013-05-17  4:05 AM PDT   4.05 High Tide\r\n4 Bodega Harbor entrance| California 2013-05-17 11:06 AM PDT   0.56  Low Tide\r\n5 Bodega Harbor entrance| California 2013-05-17  6:14 PM PDT   4.41 High Tide\r\n6 Bodega Harbor entrance| California 2013-05-18 12:23 AM PDT   2.47  Low Tide<\/pre>\n
The output from Example 4 above resembles the following (I’ve cut out some of the columns again to save space):<\/p>\n
        Date        Hour   TideHt           LocalTime\r\n1 2013-05-16 4:00 PM UTC 0.180493 2013-05-16 09:00:00\r\n2 2013-05-16 4:10 PM UTC 0.158821 2013-05-16 09:10:00\r\n3 2013-05-16 4:20 PM UTC 0.140198 2013-05-16 09:20:00\r\n4 2013-05-16 4:30 PM UTC 0.124632 2013-05-16 09:30:00\r\n5 2013-05-16 4:40 PM UTC 0.112148 2013-05-16 09:40:00\r\n6 2013-05-16 4:50 PM UTC 0.102741 2013-05-16 09:50:00<\/pre>\n
 <\/p>\n
The interface between R and XTide is the easy part. The hard part is getting XTide properly installed on your computer.<\/p>\n
Oddly enough, this is one of those cases where the Windows version (download link) \u00a0<\/a>might actually be easier than the Mac install, since the Mac version only seems to exist as source code that must be built via the command line <\/a>(found on this page<\/a>, someone point me to a pre-rolled Mac binary please). To build from source you need both the xtide source and libtcd source. In either case, on Windows, Mac, or Linux, you also need to download a harmonics file (download link) .<\/a><\/p>\n
On Windows,\u00a0 after installing the binary XTide file, it’s probably easiest to just put the unzipped harmonics .tcd file in the same directory where you installed XTide (where xtide.exe and tide.exe are found). You also need to edit the Windows environmental variable PATH to include the path to the folder where xtide.exe and tide.exe are installed (see here<\/a> for an example dealing with Matlab).<\/p>\n
On Mac\/Linux, it’s easiest to create a xtide.conf file in the \/etc directory found inside the system root directory. The screenshot below shows the process of getting to the \/etc directory and starting the nano editor to create the xtide.conf file.<\/p>\n
\"Screen<\/a>
Navigating to the \/etc folder on the Mac.<\/figcaption><\/figure>\n
In nano, on the very first line, enter the path to your harmonics .tcd file. Then hit Control+O, Control+X to save and exit nano.<\/p>\n
\"Screen<\/a>
An example of the path to the harmonics .tcd file that goes on the first line of the xtide.conf file. Change your path to suit wherever your harmonics file ended up.<\/figcaption><\/figure>\n
 <\/p>\n
If you got Xtide properly built from source, the xtide.conf file should get it working by helping XTide find the harmonics file. The installation pages for XTide reference the export HFILE_PAGE=…. prior to building the source files, but I didn’t have any success with that method. The xtide.conf method finally got things working for me on the Mac.<\/p>\n
When XTide is installed and running, you should be able to open up a system terminal and do something simple like<\/p>\n
tide -l \"Bodega Harbor entrance, California\"<\/pre>\n
and get a string of tide predictions out. If that works in your system terminal, the R script above should work as well, since R is simply calling the tide program via the system shell.<\/p>\n
Finally, if you’re not interested in predictions for Bodega Harbor and instead want data for your local site, you need to find the XTide name for your site here: http:\/\/www.flaterco.com\/xtide\/locations.html. <\/a>There are tons of sites there, so use Ctrl+F or Command+F to use your browser’s find function to search for the name of your local tide station. If your station has some crazy long name on that page, you’ll need to use that same crazy long name (For example, “Dumbarton Highway Bridge, 0.28 nmi. SE of, San Francisco Bay, California Current (25d)”) for the sitename<\/code> variable.<\/p>\n
 <\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
Edit: There is now a full-fledged R package, rtide, to accomplish the same basic task of generating time series of tide predictions that is outlined here. See this more recent post for information. XTide is an open-source program that predicts tide heights and current speeds for hundreds of tide and current stations around the United […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[218,240],"tags":[154,58,155,156,62,153],"class_list":["post-1536","post","type-post","status-publish","format-standard","hentry","category-r-project","category-tide-prediction","tag-current-speed","tag-r-project","tag-sunrise","tag-sunset","tag-tide-height","tag-xtide"],"_links":{"self":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts\/1536","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/comments?post=1536"}],"version-history":[{"count":14,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts\/1536\/revisions"}],"predecessor-version":[{"id":2318,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts\/1536\/revisions\/2318"}],"wp:attachment":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/media?parent=1536"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/categories?post=1536"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/tags?post=1536"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}