SOLARNET Metadata Recommendations for Solar Observations

Stein Vidar Hagfors Haugan, Terje Fredvik

This document has received funding from the European Union’s Horizon 2020 and FP7 programmes under grant agreements No 824135 and 31295. Version 2.0 of this document was the final version produced under these grants, but it keeps evolving as further needs and issues arise.

This specification is licensed under the Creative Commons Attribution 4.0 International Public License. For more information see the LICENSE.txt in our GitHub Repository or the Creative Commons website.

If you want to find the latest released or past version, see the releases. To provide comments or suggestions, please create an issue or create a pull request.

New changes since the most recent release are highlighted in yellow.

Abstract

Until the advent of the SOLARNET recommendations, metadata descriptions of Solar observations have been standardized for space-based observations, but the standards have been mostly within a single space mission at a time, at times with significant differences between different mission standards. In the context of ground-based Solar observations, data has typically not been made freely available to the general research community, resulting in an even greater lack of standards for metadata descriptions. This situation makes it difficult to construct multi-instrument archives/virtual observatories with anything more than the most basic metadata available for searching, as well as making it difficult to write generic software for instrument-agnostic data analysis. This document describes the metadata recommendations developed under the SOLARNET EU project, which aims foster more collaboration and data sharing between both ground-based and space-based Solar observatories. The recommendations will be followed by data pipelines developed under the SOLARNET project and others see footnote for a full list [1] . These recommendations are meant to function as a common reference to which even existing diverse data sets may be related, for ingestion into solar virtual observatories and for analysis by generic software.

[1]

Full list per July 2021: Solar Orbiter SPICE, SST CHROMIS/CRISP (SSTRED), AISAS/Lomnicky Stit COMP-S/SCD, SAMNET, Gregor HiFI/GFPI, ROB USET, Alma pipeline for Solar data (SOAP), INAF IBIS-A (IBIS data Archive), PADRE/MeDDEA

Part A. Description of FITS Keywords

• 1. About file formats
• 2. Header and Data Units (HDUs) in FITS files
  • 2.1 Naming of HDUs in SOLARNET FITS files
  • 2.2 Fully and partially SOLARNET-compliant Obs-HDUs
  • 2.3 Other HDUs
  • 2.4 Comments
• 3 The World Coordinate System (WCS) and related keywords
  • 3.1 Fundamental WCS coordinate specification
  • 3.2 WCS positional keywords and relative radial velocity
• 4 Time-related WCS keywords
  • 4.1 Specifying WCS time coordinates
• 5 Description of data contents
  • 5.1 Data type/units (BTYPE/BNAME/BUNIT)
  • 5.2 Exposure time, binning factors
  • 5.3 Cadence
  • 5.4 Instrument/data characteristics etc
  • 5.4.1 Polarimetric data reference system
  • 5.5 Quality aspects
  • 5.6 Data statistics
  • 5.6.1 Missing and saturated pixels, spikes/cosmic rays, padding, etc
  • 5.6.2 Explicit listing of missing, saturated, spike/cosmic ray pixels etc
• 6 Metadata about affiliation, origin, acquisition, etc
• 7 Grouping
  • 7.1 Pointing ID separates observations into groups
  • 7.2 Further separating observations with identical pointing ID
  • 7.3 Group ID ties together logical groups of observations
  • 7.4 Mosaics
• 8 Pipeline processing applied to the data
  • 8.1 Basic description of processing software
  • 8.2 Detailed description of all processing steps
• 9 Integrity and administrative information
• 10 Reporting of events detected by the pipeline/spacecraft
• Appendix I. Variable-keyword mechanism
  • Appendix I-a. Variable keywords using coordinate association
  • Appendix I-b. Variable keywords using pixel-to-pixel association
  • Appendix I-c. Array-valued keywords (no association)
  • Appendix I-d. Using image extensions instead of binary tables
• Appendix II. Pixel list mechanism for flagging pixels
• Appendix III. Meta-HDU mechanism
  • Appendix III-a. Extension to multiple split dimensions
• Appendix IV. Adaptation to binary table extensions
• Appendix V. Other recommendations or suggestions
  • Appendix V-a. File naming suggestions
  • Appendix V-b. Storing data in a single file or in separate files
  • Appendix V-c. Obs-HDU content guidelines
• Appendix VI. Extended mechanism for distortion corrections
  • Appendix VI-a. Multiple distortion corrections to the same coordinate
• Appendix VII. External Extension References
  • Placeholder extensions
  • Virtual external extensions
• Appendix VIII. File list wildcard patterns and sorting
• Appendix IX. Higher-level data: parameterized components

Part B. Lists of Mandatory and Optional FITS Keywords with Example Values

• 12 Mandatory keyword for all HDUs (Section 2.1)
• 13 Mandatory keywords for all Obs-HDUs (Section 2.2)
• 14 Mandatory WCS keyword for all HDUs with a UTC (time) coordinate (Section 4.1)
• 15 Mandatory keywords for fully SOLARNET-compliant Obs-HDUs
  • 15.1 Mandatory general keywords (Sections 8, 9, and Appendix V-a)
  • 15.2 Fundamental WCS coordinate keywords (Section 3.1)
  • 15.3 Mandatory WCS positional keywords (Section 3.2)
  • 15.3.1 Mandatory for ground-based observatories (Section 3.2)
  • 15.3.2 Mandatory for Earth orbiting satellites (Section 3.2)
  • 15.3.3 Mandatory for deep space missions (not Earth orbiting satellites) (Section 3.2)
  • 15.4 Mandatory data description keywords (Sections 5.1, 5.2 and 5.6.2)
  • 15.5 Mandatory keywords identifying the origin of the observations (Section 6)
  • 15.6 Mandatory keywords for spectrographs and filter instruments (Sections 3.2 and 5.4)
  • 15.7 Mandatory keyword for spectrographs (Section 5.4)
  • 15.8 Mandatory keywords for polarimetric data (Section 5.4.1)
  • 15.9 Mandatory keyword for grouping (Sections 7 and Appendix V-b )
• 16 Mandatory keyword for SOLARNET HDUs that contain keywords with a definition in conflict with the specifications in this document (Section 2.2)
• 17 Mandatory keywords for all HDUs that uses any of the variable-keyword, pixel list or meta-observation mechanism (Sections 2.1, 2.2, 2.3, Appendix I, Appendix II, and Appendix III)
  • 17.1 Mandatory keyword for binary table extension value columns that use pixel-to-pixel association (Appendix I-b)
  • 17.2 Mandatory keywords for binary table extensions that use the SOLARNET pixel lists mechanism for flagging pixels (Appendix I-d)
• 18 Optional keywords for all Obs-HDUs
  • 18.1 Optional keyword for deep space missions (not Earth orbiting satellites) (Section 3.2)
  • 18.2 Optional date and time keywords (Section 4)
  • 18.3 Optional keywords describing cadence (Section 5.3)
  • 18.4 Optional keyword for spectrographs (Section 5.4)
  • 18.5 Optional data description keywords (Section 5.1)
  • 18.6 Optional keywords characterizing the instrument/data (Sections, 5.4 and 5.5)
  • 18.7 Optional quality aspects keywords (Sections 3.1 and 5.5)
  • 18.8 Optional data statistics keywords (Section 5.6)
  • 18.9 Optional keywords for missing and saturated pixels (Section 5.6.1)
  • 18.10 Optional pipeline processing keywords (Sections 8, 8.1 and 8.2)
  • 18.11 Optional keyword for administrative information (Section 9)
  • 18.12 Optional keywords for grouping (Section 7)
  • 18.13 Optional keyword for binary table extensions using the variable-keyword, pixel list or meta-observation mechanism (Appendix I, Appendix I-d and Appendix III)

Part C. Part C. Alphabetical listings of FITS keywords with section references

• 19 Alphabetical listing of all keywords with section references

Version Guidelines

• Version Guidelines for SOLARNET Metadata Recommendations
  • Purpose
  • Versioning Approach
    • Semantic Versioning
    • Synchronized Updates
    • Version Increment Rules
  • Implementation Notes
  • References

References

• Reference List

• Index

• Search Page