AmeriFlux - Uploading High Frequency Data

Steps to Upload Data

Choose an Operating System

Windows
MacOS / Linux

NOTE: if you have multiple sites, please use one SSH key per site.

Windows

Generate SSH key
On Windows, we suggest using PuTTY tools for secure file transfer. You’ll need only the puttygen and pscp tools. You can get them here:
http://the.earth.li/~sgtatham/putty/latest/x86/puttygen.exe
http://the.earth.li/~sgtatham/putty/latest/x86/pscp.exe
But if you ever downloaded the full PuTTY package before, you might already have these two tools.To generate the keys, run puttygen.exe (e.g., double click the file). It will show you the PuTTY Key Generator window. Select “SSH-2 RSA” for type of key — sometimes this shows as just “RSA”; don’t use “SSH-1 RSA” — and “4096” for number of bits in a key (these are probably the defaults). Click generate key, and move the mouse around while it processes the key (this is to generate a key that has better randomization); this should take 20-30 seconds.Then click on “Save public key” and save the file in a known folder (the file name can be anything, but we usually use .pub as the extension). Then click on “Save private key”, it will ask you if you want to save it without a passphrase. We usually say yes here and save this file in a safe folder in a single computer. If you prefer to use a passphrase to protect your private key, you’ll have to type the passphrase every time you send us data. This file name can also be anything, but usually the extension .ppk is used.

Send public key
Send us the the public key only (usually has a .pub file extension), so we can register it in our system. The private key should be kept in a safe place, only accessible to those doing the transfers. You email us your key here: ameriflux-support@lbl.gov

Upload data
After we confirm your key is setup in our system, you can use pscp.exe to send us data. Open a Command Prompt (Start->Programs->Accessories->Command Prompt, or Start->Run, then type cmd and click OK). The command to send the data will be something like this:C:\> c:\PATH\TO\PSCP\pscp.exe -i c:\PATH\TO\KEYS\KEY.ppk -scp c:\PATH\TO\DATA\DATAFILE.CSV fluxnet@dtn01.nersc.gov:where:
- C:\>
  is the prompt (shown by windows, can also be showing a folder, no need to type it)
- c:\PATH\TO\PSCP\pscp.exe
  is the full path to the pscp executable
- -i c:\PATH\TO\KEYS\KEY.ppk
  the flag to use and full path to the private key you generated
- -scp
  is a command flag telling pscp to use the scp protocol
- c:\PATH\TO\DATA\DATAFILE.CSV
  is the full path to the data file you want to transfer. If you want to send all files within a folder use * instead of DATAFILE.CSV. You can also transfer all the contents of a folder (including sub-folders); in this case use only c:\PATH\TO\DATA\ here, and add the -r flag right after the -scp flag (see example below)
- fluxnet@dtn01.nersc.gov:
  is the destination of the data transfer. Note: The colon “:” at the end is necessary.
Other command examples
- Transfer all files in a folder:
  C:\> c:\PATH\TO\PSCP\pscp.exe -i c:\PATH\TO\KEYS\KEY.ppk -scp c:\PATH\TO\DATA\* fluxnet@dtn01.nersc.gov:
- Transfer all .csv files in a folder:
  C:\> c:\PATH\TO\PSCP\pscp.exe -i c:\PATH\TO\KEYS\KEY.ppk -scp c:\PATH\TO\DATA\*.csv fluxnet@dtn01.nersc.gov:
- Transfer an entire folder and subfolders:
  C:\> c:\PATH\TO\PSCP\pscp.exe -i c:\PATH\TO\KEYS\KEY.ppk -scp -r c:\PATH\TO\DATA fluxnet@dtn01.nersc.gov:(Note the lack of a backslash at the end of the path; if a backslash is present, the contents of the directory will be copied, but not the directory itself)

Mac OS / Linux

Generate SSH key
Open a Terminal and execute the command:
$ ssh-keygen -t rsa -b 4096 -f /PATH/TO/KEYS/KEYwhere:
- $
  is the prompt (shown by the Terminal, can have different formats with other characters and information on current host or working directory, no need to type it)
- ssh-keygen
  is the command to generate an SSH key on Unix systems (MacOS and Linux)
- -t rsa
  is a flag selecting the RSA-2 as key type
- -b 4096
  is a flag selecting 4096 bits as the key length
- -f /PATH/TO/KEY/FILENAME
  is a flag pointing to the target filename where the both the private and public parts of the key will be stored. Note: if you don’t specify this option, the default location (~/.ssh/id_rsa) will be used, which could overwrite your existing key, if you already have one.
The program will ask you for a passphrase — something like “Enter passphrase (empty for no passphrase):”. The suggestion is use a empty passphrase (just hit enter) and save this file in a safe folder in a single computer. If you prefer to use a passphrase to protect your private key, you’ll have to type the passphrase every time you send us data. The file name for the key can also be anything, but we suggest using the site ID (e.g., CC-SSS).

Send public key
Send us the the public key only (usually has a .pub file extension), so we can register it in our system. The private key should be kept in a safe place, only accessible to those doing the transfers. You email us your key here: ameriflux-support@lbl.gov

Upload data
After we confirm your key is setup in our system, you can use the scp command to send us data. The command to send the data will be something like this:$ scp -i /PATH/TO/KEYS/KEY /PATH/TO/DATA/DATAFILE.CSV fluxnet@dtn01.nersc.gov:where:
- $
  is the prompt (shown by the Terminal, can have different formats with other characters and information on current host or working directory, no need to type it)
- -i /PATH/TO/KEYS/KEY
  is the path to the file holding your private key. This should be the same as the path you specified to create the keys. Just note if the path was absolute (starting with a forward slash “/”) or relative (starting with other naming characters) and specify the path accordingly.
- /PATH/TO/DATA/DATAFILE.CSV
  is the full path to the data file you want to transfer. If you want to send all files within a folder use * instead of DATAFILE.CSV. You can also transfer all the contents of a folder (including sub-folders); in this case use only /PATH/TO/DATA/ here, and add the -r flag right before the -i flag (see example below)
- fluxnet@dtn01.nersc.gov:
  is the destination of the data transfer. Note: The colon “:” at the end is necessary.
Other command examples
- Transfer all files in a folder:
  $ scp -i /PATH/TO/KEYS/KEY /PATH/TO/DATA/* fluxnet@dtn01.nersc.gov:
- Transfer all .csv files in a folder:
  $ scp -i /PATH/TO/KEYS/KEY /PATH/TO/DATA/*.csv fluxnet@dtn01.nersc.gov:
- Transfer an entire folder and subfolders:
  $ scp -i /PATH/TO/KEYS/KEY -r /PATH/TO/DATA fluxnet@dtn01.nersc.gov:

Notes on the data servers

There are four servers setup to receive AmeriFlux data: dtn01.nersc.gov, dtn02.nersc.gov, dtn03.nersc.gov, and dtn04.nersc.gov. This is to allow some redundancy and avoid a single server being overloaded. You can use any of these servers, and even setup your transfers to try different servers if one of them is not responding or is slow. However, using the first server (as in the examples) should work correctly most of the time.

If you have any problems or questions, please contact us: ameriflux-support@lbl.gov