1. Flux-met data processing
2. More to come!
1. Flux-met data processing
- 1.1 General
- 1.2 Format QA/QC
- 1.1.1 Why can some variable names with “_PI” qualifier be autocorrected and others can’t?
- 1.1.2 Why are there variable names with “d1”, “d2”, “d3”, etc?
- 1.1.3 Are some variables required?
- 1.1.4 Can I submit gap-filled variables?
- 1.3 Data QA/QC
1.1 General
1.1.1 Are some variables required?
A carbon flux variable (e.g., FC or FCH4) is required. We encourage submission of water and heat fluxes as well (LE and H). These flux variables must be submitted without gap-filling. Gap-filled flux variables may be submitted in addition to the non-filled variables.
1.1.2 Can I submit gap-filled variables?
Yes! We encourage submission of both non-filled and gap-filled variables. Please use the appropriate qualifiers described in the FP-In Instructions (a.k.a., Half-Hourly / Hourly Data Upload Format).
1.2 Format QA/QC
1.2.1 Why can some variable names with “_PI” qualifier be autocorrected and others can’t?
The AmeriFlux data team adds the _PI qualifier to some calculated variables as well as gap-filled variables submitted by the site. We automatically remove the PI qualifier from variable names to which we add it during the publishing process. We will not remove the PI qualifier from other variables.
1.2.2 Why are there variable names with “d1”, “d2”, “d3”, etc?
If duplicate variable names are submitted within a file, we add a _d# suffix to the variable name so that we can continue assessing the file format. Empty columns or extra rows at the beginning of the file are often interpreted as empty variable names that are temporarily renamed _d1, _d2, etc to continue Format QA/QC processing.
1.2.3 What can cause the test result is “Error reading data from file.”?
Missing variable names and rows with different column numbers are common causes of this error.
1.2.4 Excel is saving my timestamps in scientific notation. How can I fix this?
Often when converting Excel files to .csv files, formatting can be an issue. The Ameriflux FP-In formatting standards for timestamps (YYYYMMDDHHMM) is commonly reformatted within Excel to scientific notation.
For example, inputting “201704031430” will be converted to “2.01704E+11”.
When reading this file in Excel, the data is often still there, but when the .csv is opened in a text editor, such as Notepad or TextEdit, the data is saved in scientific notation. Since the data pipeline reads the files similar to a text editor, timestamp data saved in scientific notation will fail the format check.
To save Timestamp data properly and prevent reformatting:
- Select all data in Timestamp data columns
- Under “Number Formatting”, select “Number” from the drop down menu and decrease decimal to create a whole number
- Save file as a .csv file
- Open the .csv file in a text editor to verify the changes have been made and saved
Please create duplicates of your data in case of issues while reformatting. Saving data in scientific notation as a .csv can accidentally result in total loss of timestamp data. By following these instructions and verifying that the .csv is properly formatted, your data can be read by the AmeriFlux QA/QC processing code.
1.3 Data QA/QC
more coming soon!
2. More to come!
Contact ameriflux-support@lbl.gov if you didn’t find an answer to your question.