{"id":1190,"date":"2012-07-11T20:42:49","date_gmt":"2012-07-12T00:42:49","guid":{"rendered":"http:\/\/lukemiller.org\/?p=1190"},"modified":"2012-07-11T20:42:49","modified_gmt":"2012-07-12T00:42:49","slug":"r-scripts-for-downloading-ibutton-thermochron-dataloggers","status":"publish","type":"post","link":"https:\/\/lukemiller.org\/index.php\/2012\/07\/r-scripts-for-downloading-ibutton-thermochron-dataloggers\/","title":{"rendered":"R scripts for downloading iButton Thermochron dataloggers"},"content":{"rendered":"

Last time,<\/a> I posted some R code to help quickly launch many iButton Thermochron temperature dataloggers with the same mission parameters. The R code makes use of a publicly-available command line utility released by the iButton’s manufacturer, Maxim.\u00a0 Of course, Maxim also has a command line utility for downloading the data from those iButtons that you launched already. The code below will make use of that program to download an iButton, give the file a unique name of your choosing, parse the data out into a simple comma-separated-value file (for easy opening in R or Excel), and then immediately be ready to download another iButton (repeat ad infinitum).<\/p>\n

A word of warning, I’ve only tested this on Windows 7, but Maxim does make command line utilities for Linux\/OSX that could probably be adapted into my R script. I downloaded the 1-Wire Public Domain software development kit, in this case the pre-compiled binary version for Win64 USB Preliminary Version 3.11 Beta 2 that came as a zip file called\u00a0winusb64vc311Beta2_r2.zip (downloaded 2012-03-15) from this site:<\/p>\n

http:\/\/www.maxim-ic.com\/products\/ibutton\/software\/1wire\/wirekit.cfm<\/a><\/p>\n

Inside the \\builds\\winusb64vc\\release folder you’ll find thermoms.exe and thermodl.exe, which are the command line utilities required for launching and downloading iButtons in Windows. They should be copied into the R working directory where these R scripts will be run.<\/p>\n

My iButton scripts are available in a Github.com repository:<\/p>\n

http:\/\/github.com\/millerlp\/ibuttons<\/a><\/p>\n

The R script for launching iButtons (mission_launch_multi.R) covered last time is available there, as are two scripts for downloading iButtons (ibutton_download_custom_names.R and ibutton_download_increment_names.R). If you have any helpful changes to these scripts, feel free to send a pull request via Github. <\/p>\n

The concept here is fairly straightforward. I’m using R (because it’s what I know), to ask the user to provide a name for their downloaded data file, then calling the thermodl.exe utility to download an iButton that is attached to a DS9490R USB adapter, and parsing the raw data file produced by the utility to extract the useful time\/date and temperature data. The script also includes the iButton’s serial number (in case you use that) and your unique ID for the filename in the comma-separated-value file that is output at the end.<\/p>\n

The script creates two directories, one to hold the raw downloaded files from the iButton created by the thermodl.exe (“_raw_downloads”) in case you want to hold on to those. The second directory (“_iButton_data”), holds csv files that can be easily opened in Excel. <\/p>\n

\"\"<\/a>
Two directories will be created in your R working directory to hold the output files.<\/figcaption><\/figure>
\n<\/br>
\nThe “raw” data files have lots of information about the mission parameters and include the temperature histogram along with the time-stamped temperature data. These files can’t be easily imported into a spreadsheet or data frame. <\/p>\n

\"\"<\/a>
The “raw” datafiles downloaded from the iButton. An example of some of the information stored in the raw data file is shown.<\/figcaption><\/figure>
\n<\/br>
\nThe parsed csv files throw away most of that extra information and just retain the time stamps and temperature in Celsius in the first two columns. The iButton’s serial number is included in the 3rd column, as is the name I gave to this file when the script downloaded the iButton. Either of those two columns might be useful for sorting data when you start collating data from multiple iButtons in the same file.
\n<\/br>
\n
\"\"<\/a>
The csv files created in the 2nd directory are formatted for easy access to the important time and temperature information.<\/figcaption><\/figure>
\n<\/br>
\nYou’ll notice the weird jumbles of numbers peppered throughout the filenames and directory names. Hopefully you’ve been able to discern that these are simply the date (20120711) and time (1705) expressed in a compact format. The aim here is to help organize files if you’re accumulating data over many months, and will hopefully prevent you from overwriting previous data from one site with newly downloaded data from that same site. Each time you run the script, it will update with the current date and time and insert those into your newly created filenames. <\/p>\n

When you run the script, it will initially ask you for an ID to insert into the downloaded file name. This could be anything, such as “PointSur_High_Zone_A”, but avoid blank spaces in the name. Once you enter this ID, the script will immediately look for the iButton attached to your USB adapter, and download its data. After it’s finished, the temperature data will be flashed on the R console, and the script will sit and wait for you to enter the ID for the next iButton you want to download (go ahead and swap in the new iButton before you enter the next ID). Repeat this as much as necessary, and when you’re ready to quit, just type “q” instead of an ID. Hopefully you don’t have a site simply named “q”, because that will make your life difficult. <\/p>\n

\n
\n
# Filename: ibutton_download_custom_names.R<\/span>\r\n#<\/span>\r\n# Author: Luke Miller Jul 11, 2012<\/span>\r\n###############################################################################<\/span>\r\n# This script will allow the user to download iButtons, giving a unique name<\/span>\r\n# to each file. It will parse the raw data file and extract the useful bits<\/span>\r\n# (to me at least): date\/time, temperature (Celsius), user's ID for this<\/span>\r\n# iButton, and the serial number for this iButton, and place all of these in a<\/span>\r\n# comma-separated-value file that can be easily opened in Excel or R. Each<\/span>\r\n# downloaded iButton data file has the current date and time included in the<\/span>\r\n# filename to avoid overwriting older data files. Raw data files are stored in a<\/span>\r\n# directory with today's date on it, while the parsed csv files go in a 2nd<\/span>\r\n# directory.<\/span>\r\n \r\n# The thermodl.exe file was originally downloaded as part of the Maxim iButton<\/span>\r\n# 1-Wire Public Domain Kit.<\/span>\r\n# There are several versions of the Kit available, including<\/span>\r\n# versions with pre-compiled binaries (executables) for Windows\/Linux\/OSX.<\/span>\r\n# http:\/\/www.maxim-ic.com\/products\/ibutton\/software\/1wire\/wirekit.cfm<\/span>\r\n# On my Windows 7 x64 computer using the DS9490B USB ibutton adapter, I used the<\/span>\r\n# precompiled binary build for Win64 USB (DS9490 + WinUSB) Preliminary Version<\/span>\r\n# 3.11 Beta 2,<\/span>\r\n# filename: winusb64vc311Beta2_r2.zip, downloaded 2012-03-15<\/span>\r\n# Unzip this file and find the .exe files thermoms.exe and thermodl.exe in the<\/span>\r\n# builds\\winusb64vc\\release folder. Copy these to your R working directory.<\/span>\r\n# The drivers for the DS9490 USB iButton adapter must also be downloaded and<\/span>\r\n# installed:<\/span>\r\n# http:\/\/www.maxim-ic.com\/products\/ibutton\/software\/tmex\/<\/span>\r\n# I downloaded and installed the file "install_1_wire_drivers_x64_v403.msi"<\/span>\r\n# The Java OneWireViewer app can also be downloaded and used to verify that your<\/span>\r\n# drivers work and that you can communicate with iButtons successfully through<\/span>\r\n# the USB adapter. You can download this app here:<\/span>\r\n# http:\/\/www.maxim-ic.com\/products\/ibutton\/software\/1wire\/OneWireViewer.cfm<\/span>\r\n \r\n \r\n# NOTE: the Maxim program "thermodl.exe" must be present in the current R<\/span>\r\n# working directory. Uncomment the setwd() line below to change the R working<\/span>\r\n# directory. Enter your working directory location inside the quotes.<\/span>\r\n# setwd('D:\/R\/ibuttons')<\/span>\r\n \r\ncur.date = Sys.Date<\/span><\/a>(<\/span>)<\/span> # Get current date<\/span>\r\n# Assemble a directory name to store raw downloaded data into<\/span>\r\ndir.name = paste<\/span><\/a>(<\/span>'.\\\\<\/span>'<\/span>,<\/span>as.character<\/span><\/a>(<\/span>cur.date)<\/span>,<\/span> '_raw_downloads'<\/span>,<\/span>sep = ''<\/span>)<\/span>\r\n# Assemble a directory name to store the parsed csv output files in<\/span>\r\ndir.name2 = paste<\/span><\/a>(<\/span>'.\\\\<\/span>'<\/span>,<\/span>as.character<\/span><\/a>(<\/span>cur.date)<\/span>,<\/span>'_iButton_data'<\/span>,<\/span> sep = ''<\/span>)<\/span>\r\n# Check to see if that folder already exists, if not then create it.<\/span>\r\nif<\/span> (<\/span>is.na<\/span><\/a>(<\/span>file.info<\/span><\/a>(<\/span>dir.name)<\/span>$<\/span>isdir)<\/span>)<\/span> {<\/span>\r\ndir.create<\/span><\/a>(<\/span>dir.name)<\/span>\t\r\n}<\/span>\r\n# Check to see if the other folder already exists, if not then create it.<\/span>\r\nif<\/span> (<\/span>is.na<\/span><\/a>(<\/span>file.info<\/span><\/a>(<\/span>dir.name2)<\/span>$<\/span>isdir)<\/span>)<\/span> {<\/span>\r\ndir.create<\/span><\/a>(<\/span>dir.name2)<\/span>\t\r\n}<\/span>\r\n#######################################<\/span>\r\n# Enter main while loop to repeatedly download buttons until user decides to<\/span>\r\n# quit.<\/span>\r\n# Have user enter their own ibutton ID to be included in the filename<\/span>\r\ncat<\/span><\/a>(<\/span>'Enter starting number for file name: \\n<\/span>'<\/span>)<\/span>\r\nfnameID = scan<\/span><\/a>(<\/span>file<\/span><\/a> = ''<\/span>,<\/span> what = character<\/span><\/a>(<\/span>)<\/span>,<\/span> n = 1<\/span>)<\/span>\r\nloop = TRUE<\/span>\r\nwhile<\/span>(<\/span>loop)<\/span> {<\/span>\r\n# Get current time to insert in filename so we don't overwrite old data<\/span>\r\ncurrTime = strftime<\/span><\/a>(<\/span>Sys.time<\/span><\/a>(<\/span>)<\/span>,<\/span> format<\/span><\/a> = "%Y%m%d_%H%M"<\/span>)<\/span>\r\n# Assemble filename<\/span>\r\nfname = paste<\/span><\/a>(<\/span>dir.name,<\/span>'\\\\<\/span>'<\/span>,<\/span>fnameID,<\/span>'_'<\/span>,<\/span>currTime,<\/span>'.dat'<\/span>,<\/span> sep = ''<\/span>)<\/span>\r\n# Call the thermodl.exe program provided by Maxim. This must be in the<\/span>\r\n# current R working directory. Start by assembling a command to be issued<\/span>\r\nsys.com = paste<\/span><\/a>(<\/span>'thermodl.exe ds2490-0 '<\/span>,<\/span> fname,<\/span> sep = ''<\/span>)<\/span>\r\n# Now issue the command to the system using the system() function<\/span>\r\ntemp = system<\/span><\/a>(<\/span>sys.com,<\/span> wait = TRUE<\/span>,<\/span> intern = TRUE<\/span>)<\/span>\r\n# The raw downloaded data should now be in the "raw_downloads" directory<\/span>\r\n# created at the start of the script.<\/span>\r\n# Output the status updates from thermodl.exe<\/span>\r\ncat<\/span><\/a>(<\/span>temp[<\/span>7<\/span>]<\/span>,<\/span> '\\n<\/span>'<\/span>)<\/span>\r\ncat<\/span><\/a>(<\/span>temp[<\/span>9<\/span>]<\/span>,<\/span>'\\n<\/span>'<\/span>)<\/span>\r\ncat<\/span><\/a>(<\/span>temp[<\/span>18<\/span>]<\/span>,<\/span> '\\n<\/span>'<\/span>)<\/span>\r\ncat<\/span><\/a>(<\/span>temp[<\/span>20<\/span>]<\/span>,<\/span> '\\n<\/span>'<\/span>)<\/span>\r\n# Open the data file created by thermodl.exe<\/span>\r\nx = readLines<\/span><\/a>(<\/span>fname)<\/span> # read data into a character vector<\/span>\r\n# Extract serial number from line 4 of file<\/span>\r\nserialnum = substr<\/span><\/a>(<\/span>x[<\/span>4<\/span>]<\/span>,<\/span> 26<\/span>,<\/span> nchar<\/span><\/a>(<\/span>x[<\/span>4<\/span>]<\/span>)<\/span>)<\/span>\r\n \r\n# Parse the iButton data file to extract the relevant temp\/time data<\/span>\r\nlog.start = grep<\/span><\/a>(<\/span>'Log Data'<\/span>,<\/span> x)<\/span> # Find what line the log data starts below<\/span>\r\nlog.start = log.start +<\/span> 3<\/span> # change value to first line of log data<\/span>\r\nlog.end = grep<\/span><\/a>(<\/span>'Debug Dump'<\/span>,<\/span> x)<\/span> # Find where the log data ends (roughly)<\/span>\r\nlog.end = log.end -<\/span> 2<\/span> # change value to last line of log data<\/span>\r\n# Check if there were any logged temperature data<\/span>\r\nif<\/span> (<\/span>!<\/span>(<\/span>log.end -<\/span> log.start <<\/span> 0<\/span>)<\/span>)<\/span> {<\/span>\r\ntemps = x[<\/span>log.start:<\/span>log.end]<\/span> # extract log data, still as characters<\/span>\r\n \r\n# Convert time stamps to POSIX object<\/span>\r\ntimes = as.POSIXct<\/span><\/a>(<\/span>strptime(<\/span>substr<\/span><\/a>(<\/span>temps,<\/span>1<\/span>,<\/span>17<\/span>)<\/span>,<\/span>\r\nformat<\/span><\/a> = '%m\/%d\/%Y %H:%M'<\/span>)<\/span>)<\/span>\r\ntemp.data = data.frame<\/span><\/a>(<\/span>Time<\/span><\/a> = times)<\/span> # stick into a new data frame<\/span>\r\n# convert temps to numeric and insert into temp.data<\/span>\r\ntemp.data$<\/span>TempC = as.numeric<\/span><\/a>(<\/span>substr<\/span><\/a>(<\/span>temps,<\/span>20<\/span>,<\/span>26<\/span>)<\/span>)<\/span>\r\n# Insert column with iButton's unique serial number<\/span>\r\ntemp.data$<\/span>Serial.number = serialnum\r\n# Insert new column with ibutton ID from file name<\/span>\r\ntemp.data$<\/span>ID = fnameID\r\n# Output temperature data to console<\/span>\r\ncat<\/span><\/a>(<\/span>'Temperature summary data: \\n<\/span>'<\/span>)<\/span>\r\nsprintf<\/span><\/a>(<\/span>'%s'<\/span>,<\/span> temp.data)<\/span>\r\nprint<\/span><\/a>(<\/span>temp.data)<\/span>\r\nflush.console<\/span><\/a>(<\/span>)<\/span>\r\n# Output temperature data to a comma-separated-value file for easy<\/span>\r\n# reading in Excel or R.<\/span>\r\n# Start by assembling new filename, sticking output file in the 2nd<\/span>\r\n# directory created at the start of the script.<\/span>\r\noutputfile = paste<\/span><\/a>(<\/span>dir.name2,<\/span>'\\\\<\/span>'<\/span>,<\/span>fnameID,<\/span>'_'<\/span>,<\/span>currTime,<\/span>'.csv'<\/span>,<\/span> sep = ''<\/span>)<\/span>\r\n# Write temp.data to a comma-separated-value file<\/span>\r\nwrite.csv<\/span><\/a>(<\/span>temp.data,<\/span> file<\/span><\/a> = outputfile,<\/span> quote<\/span><\/a> = FALSE<\/span>,<\/span>\r\nrow.names<\/span><\/a> = FALSE<\/span>)<\/span>\r\n}<\/span> else<\/span> {<\/span>\r\ncat<\/span><\/a>(<\/span>'\\n<\/span>\\n<\/span>*****No temperature data*****\\a<\/span>\\n<\/span>\\n<\/span>'<\/span>)<\/span>\r\n}<\/span>\r\n \r\ncat<\/span><\/a>(<\/span>temp[<\/span>18<\/span>]<\/span>,<\/span> '\\n<\/span>'<\/span>)<\/span>\r\ncat<\/span><\/a>(<\/span>temp[<\/span>20<\/span>]<\/span>,<\/span> '\\n<\/span>'<\/span>)<\/span>\r\ncat<\/span><\/a>(<\/span>'\\a<\/span>\\n<\/span>---------------------\\n<\/span>'<\/span>)<\/span>\r\ncat<\/span><\/a>(<\/span>'Swap in next iButton and hit enter. Enter q to quit.\\n<\/span>'<\/span>)<\/span>\r\n# Read next iButton ID from user's input<\/span>\r\nfnameID = scan<\/span><\/a>(<\/span>file<\/span><\/a> = ''<\/span>,<\/span> what =character<\/span><\/a>(<\/span>)<\/span>,<\/span> n = 1<\/span>)<\/span>\r\nif<\/span> (<\/span>length<\/span><\/a>(<\/span>fnameID)<\/span> ><\/span> 0<\/span>)<\/span> {<\/span>\r\nif<\/span> (<\/span>fnameID ==<\/span> 'q'<\/span>)<\/span> loop = FALSE<\/span> # quit out of while loop<\/span>\r\n}<\/span> else<\/span> loop = TRUE<\/span> # return to start of while loop to download next iButton<\/span>\r\n \r\n}<\/span>\r\n \r\ncat<\/span><\/a>(<\/span>'Finished\\n<\/span>'<\/span>)<\/span><\/pre>\n<\/div>\n<\/div>\n
Created by Pretty R at inside-R.org<\/a><\/p>\n
The other version of this script, called ibutton_download_increment_names.R, also available in the github repository (http:\/\/github.com\/millerlp\/ibuttons<\/a>) is for people that have assigned simple sequential numeric values to their iButtons. Maybe you have sites or iButtons that you simply refer to as 1,2,3,4,5 etc. You can use this alternate script to enter the number for the 1st ibutton (23 for example), and it will automatically increment the filename for each subsequent iButton (24, then 25, then 26 and so on) that you download. <\/p>\n
By the way, iButtons still aren’t waterproof, so you’d better do your own waterproofing if you’re going to be putting them anywhere with lots of moisture (in the ocean, buried in soil, in an animal, etc.). <\/p>\n","protected":false},"excerpt":{"rendered":"
Last time, I posted some R code to help quickly launch many iButton Thermochron temperature dataloggers with the same mission parameters. The R code makes use of a publicly-available command line utility released by the iButton’s manufacturer, Maxim.\u00a0 Of course, Maxim also has a command line utility for downloading the data from those iButtons that […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[218],"tags":[125,131,7,45,130],"class_list":["post-1190","post","type-post","status-publish","format-standard","hentry","category-r-project","tag-ibutton","tag-maxim","tag-r","tag-script","tag-thermochron"],"_links":{"self":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts\/1190","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/comments?post=1190"}],"version-history":[{"count":12,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts\/1190\/revisions"}],"predecessor-version":[{"id":1206,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/posts\/1190\/revisions\/1206"}],"wp:attachment":[{"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/media?parent=1190"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/categories?post=1190"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lukemiller.org\/index.php\/wp-json\/wp\/v2\/tags?post=1190"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}