Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
filereference:output:start [2019/05/31 13:12] – [Level 2: Classification by filename] enviadminfilereference:output:start [2023/02/26 18:43] (current) – [Level 1: Storage by folder name] enviadmin
Line 4: Line 4:
 Each ENVI-met simulation creates a huge amount of data which is organized in different files and folders.  Each ENVI-met simulation creates a huge amount of data which is organized in different files and folders. 
 In general, there are two different types of output files regardless their content: In general, there are two different types of output files regardless their content:
-  * **Simple Text Files**: Files which contain data in pure ASCII text format. Usually, the content of the file are described in the first row of the file. These file can be read and visiualized with any software that can read text files (which should be about every piece of software that can visiualize data)+  * **Simple Text Files**: Files which contain data in pure ASCII text format seperated with "comma" aka CSV DataThe content of the file (variable names) are described in the first row of the file. These file can be read and visiualized with any software that can read text files.  From Version 5 on, ENVI-met provides different options to analyse data using Python e.g. the DataStudio inside LEONARDO. With respect to Python, the format of the text output files have been changed from V5 on to allow a seamless integration with Python and Pandas. 
-  * **Binary Files (EDX/EDT)**: The main ENVI-met output files are stored in a binary format and require special software to be read. The [[leonardo:start|LEONARDO]] software included in the ENVI-met system offers a comfortable and visual access to all data and give various options for 2D and 3D visualisation. In addition, with the help of the reference for the [[filereference:edx_edi|EDX/EDT file format]], own reader routines can easily be developed.+ 
 + 
 +  * **Binary Files (EDX/EDT)**: The main ENVI-met output files are stored in a binary format and require special software to be read. The [[leonardo:start|LEONARDO]] software included in the ENVI-met system offers a comfortable and visual access to all data and give various options for 2D and 3D visualisation. In addition, with the help of the reference for the [[filereference:edx_edi|EDX/EDT file format]], own reader routines can easily be developed.  Moreover, ENVI-met provides the option to write simulation data in addition using the [[wp>NetCDF|NetCDF format]].  
 + 
 +**In order to use advanced analysis functions like Time Series in LEONARDO, it is strongly recommended to store only one simulation run in the same folder and not to mix different model runs. While the files remain usable itself, the logic of the different applications might not be able to sort out the structure of your simulation data.**
  
 ===== Name and storage logic ===== ===== Name and storage logic =====
Line 11: Line 15:
  
   * Level 1: Storage by folder name   * Level 1: Storage by folder name
-  * Level 2: Classification by filename+  * Level 2: Classification by filename or file extension
   * Level 3: Metadata information in the [[filereference:edx_edi|. EDX File]]   * Level 3: Metadata information in the [[filereference:edx_edi|. EDX File]]
 +
 +For simple Text files, Level 3 data are not available, so that the filename and filename extension provide information about the content of the files.
  
 ==== Level 1: Storage by folder name ==== ==== Level 1: Storage by folder name ====
  
-{{ :filereference:output:outputfolders.png ?nolink 600 |Root output folders of ENVI-met 4}} +The uppermost level of file organisation starts by sorting the output files into subfolders of the selected output folders. Some folders are only existing in the full version of ENVI-met, others may be added in the course of time. All files in one folder are of the same structure and contain the same set of information, some of the output folders are further organized in sub folders. The following list summarizes the basic content of the folders. For more information on the file types, see sections below.
- +
-The uppermost level of file organisation starts by sorting the output files into subfolders of the selected output folders. The figure above shows the folder structure of a typical ENVI-met output folder (some folders are only existing in the Advanced version of ENVI-met). All files in one folder are of the same structure and contain the same set of information, some of the output folders are further organized in sub folders. The following list summarizes the basic content of the folders. For more information, click on the "→Read more..." below+
- +
-  -  **[[filereference:output:atmosphere | → Atmosphere:]]**  All information about the state of the atmosphere at different times of model simulation. Definitely the largest set of data and mostly considered as "the" model result (which is not true as the other data are very important, too).  +
-  - **Buildings**   Data along the canvas (facades and roofs) of the buildings. Includes static data such as material, albedo (folder ''STATIC'') as well as dynamic data of meteorology along the building and building physics data (folder ''DYNAMIC'', not in BASIC). →Read more... +
-  - **Inflow**  Data about the inflow model of ENVI-met used to represent the surroundings of the model domain that cannot be modelled in the given scale. →Read more...  +
-  - **Log**  A copy of the ENVI-met outputs generated during the model run. →Read more... +
-  - **Pollutants**  Data about pollutant concentration in the model atmosphere (only present if pollutants are used in the model). →Read more... +
-  - **Radiation**  Detailed data sets about the different radiation fluxes in the model. →Read more... +
-  - **Receptors**  Time series and profile data at defined receptor points in the model (if defined)   →Read more... +
-  - **Soils**   State of the ENVI-met soil model  →Read more... +
-  - **SolarAccess**   Detailed analysis on solar access data such as solargrams, sun hours, etc (ENVI-met PRO/SCIENCE only) →Read more... +
-  - **Surface**   State of the soil surface as an interface between the atmosphere and the soil model  →Read more... +
-  - **Vegetation**   State of the vegetation including detailed log of observed 3D plants →Read more...+
  
 +  - **InputData:**   //This folder contains a copy of all relevant simulation files (.INX, .SIMX,...),  database files and forcing data used in the simulation. Using this folder, you can reconstruct your model settings as they have been at the time of simulation even if they have been changed afterwards. This folder is required by several tools such as BIO-met. (Added with V5.0.3)//
 +  -  **Atmosphere:**  All information about the state of the atmosphere at different times of model simulation. Definitely the largest set of data and mostly considered as "the" model result (which is not true as the other data are very important, too). 
 +  - **Buildings:**   Detailed data for the facades and roofs of the buildings. Folder includes static data such as material, albedo (folder ''STATIC'') as well as dynamic data of meteorology along the building and building physics data like temperatures and fluxes (folder ''DYNAMIC'', not in ENVI-met LITE). In addition, summary files about the different buildings are included.
 +  - **[[filereference:output:inflow | → Inflow:]]**  Data about the inflow model of ENVI-met used to represent the surroundings of the model domain that cannot be modelled in the given scale. 
 +  - **Log:**  A copy of the ENVI-met outputs generated during the model run. 
 +  - **Pollutants:**  Data about pollutant concentration in the model atmosphere (only present if pollutants are used in the model). 
 +  - **Radiation:**  Detailed data sets about the different radiation fluxes in the model.
 +  - **[[filereference:output:receptors | → Receptors:]]**  Time series and profile data at defined receptor points in the model (if defined)  
 +  - **Soil:**   State of the ENVI-met soil model 
 +  - **Solar Access:**   Detailed analysis on solar access data such as solargrams, sun hours, etc (Full versions of ENVI-met  only)
 +  - **Surface:**   State of the soil surface as an interface between the atmosphere and the soil model 
 +  - **Vegetation:**   State of the vegetation including detailed log of observed 3D plants. Contains 3D data and text data for observed plants which are organized in a ''STATIC'' folder for constant properties and a ''DYNAMIC'' folder with time-dependent properties of the plants.
 + 
 In addition, the BIOMet tool will create further folders, by default labled "**Biomet**", but with a free choice of names. In addition, the BIOMet tool will create further folders, by default labled "**Biomet**", but with a free choice of names.
-Future versions or additional modules of ENVI-met will create further folders and file, but the general concept of storge will be the same. +Future versions or additional modules of ENVI-met will create further folders and file, but the general concept of storage will be the same. 
  
-==== Level 2: Classification by filename ====+==== Level 2: Classification by file name or extension ====
 Sorting the output files into folders provides a first system of order in the output data. However, once a file has been moved out of its folder or if several simulations come together, this system is not unique and not persistent. As a solution, the ENVI-met file name generation scheme allows a direct identification of the simulation files and their content. Sorting the output files into folders provides a first system of order in the output data. However, once a file has been moved out of its folder or if several simulations come together, this system is not unique and not persistent. As a solution, the ENVI-met file name generation scheme allows a direct identification of the simulation files and their content.
  
-{{ :filereference:output:filenamegeneration.png ?nolink 300 |Logic of ENVI-met output files file name generation}}+{{ :filereference:output:filescheme_new.png ?nolink 300 |Logic of ENVI-met output files file name generation}} 
  
 Each ENVI-met output filename consists of 3 parts:  Each ENVI-met output filename consists of 3 parts: 
   - **Simulation Base Name**: This is the given working name for the simulation. You define this name in the .SIMX simulation configuration file. It should be not too log (as more data are added) and unique. In the example above,  ''urbanLayout'' has been selected  as basename.   - **Simulation Base Name**: This is the given working name for the simulation. You define this name in the .SIMX simulation configuration file. It should be not too log (as more data are added) and unique. In the example above,  ''urbanLayout'' has been selected  as basename.
   - **Type identifier**:  The type identifier describes the content of the file following the categories listed above. For example ''_AT_'' defines that this file holds atmospheric data. The different content ids are listed in the description of the output files categories and at the end of this page.    - **Type identifier**:  The type identifier describes the content of the file following the categories listed above. For example ''_AT_'' defines that this file holds atmospheric data. The different content ids are listed in the description of the output files categories and at the end of this page. 
-  - **Time of data**: The last part of the filename represents the model time the data were taken. In the example above, the file represents the state of the atmosphere (''_AT_'') at 13:00 model time. If the data in the file are not time sensitive, this part of the filename is empty. From Version 4.4.3 (Summer 19) onwards, the order of the time step has been changed to Year-Month-Date (YYYY-MM-DD + Time)+  - **Time of data**: The last part of the filename represents the model time the data were taken. This part of the filename is generated base on the "YYYY-MM-DD+'_'+hh.mm.ss" scheme. In the example shown above, the file represents the state of the atmosphere (''_AT_'') at 08:00:01 model time on the 15th July (07) 2018. If the data in the file are not time sensitive, this part of the filename is empty.  
 + 
 +===Classification by file extension === 
 +For pure text fileswhich might be processed in other programs or using the Python-based DataStudio, it is not possible to include metadata or other addtional data in the files except a header line defining the variables stored in the different colums of the file. 
 + 
 +From ENVI-met Version 5 on, all text files are comma-seperated (CSVfiles that can be processed directly in Python/Pandas or other software. In order to identify the contents of the file and suggest the correct scripts in DataStudio, the file extension is requried for these file types.  
  
  
Line 51: Line 63:
  
  
-==== Sumary on File Identifiers ====+===== Binary Files: Overview and description =====
  
-<WRAP box left 80%> +The list below lists and links to the basic main files in binary format generated by the ENVI-met simulations. 
-^ ID ^ Content ^ Remark ^ + 
-_AT_ | Atmospheric data  |  +<WRAP box left 99%> 
-_BLDG_ Building data  | Both STATIC and DYNAMIC   | +^ID  ^Content  ^Remark  ^Details 
-_POLU_ Pollutants  |  +AT | Atmospheric data  | 3D Data Main state of the atmosphere with most of the variables in it |   [[filereference:output:atmosphere|{{:filereference:output:about_24.png?nolink |}}]] 
-_SO_ | Soil data  |  | +FX Surface data and surface fluxes  2D Data describing state and fluxes at the ground surface    [[filereference:output:surface|{{:filereference:output:about_24.png?nolink |}}]] 
-_SA_ Solar Access  Main 3D Model | +SO | Soil data  | 3D Data  about the state of the soil model, mainly temperature and soil water |    [[filereference:output:soil|{{:filereference:output:about_24.png?nolink |}}]] 
-_SAFAC_ Solar Accerss   Facade and Single Wall data  | +VEG Vegetation data  Plant data like Leaf Temperature in the context of the 3D Model, Detailed data for observed plants are additional text files of ''.VEG_status'' type (see below)|    [[filereference:output:vegetation|{{:filereference:output:about_24.png?nolink |}}]] 
-_FX_ Surface data and surface fluxes  +BLDG_static Building data static  Static building data like material properties (Subfolder ''STATIC'' | [[filereference:output:building_static|{{:filereference:output:about_24.png?nolink |}}]] 
-_VEG_ Vegetation data  Main 3D Model, Observed plants are text files |+BLDG Building data dynamic  Dynamic building data like wall temperatures or state of facade greeings (Subfolder ''DYNAMIC'') | [[filereference:output:building_dynamic|{{:filereference:output:about_24.png?nolink |}}]] | 
 +| POLU | Pollutants  | 3D Data containing air pollutants including chemical conversion rates if used  |   [[filereference:output:pollutants|{{:filereference:output:about_24.png?nolink |}}]] 
 +RD Radiation Data  | 3D Data with detailed information about shortwave and longwave radiation fluxes|    [[filereference:output:radiation|{{:filereference:output:about_24.png?nolink |}}]] | 
 +| BIO | BIO-met Data  | Data generated by BIO-met | 
 +| SA | Solar Access   | Solar Access Data for the ground and Biomet level |   [[filereference:output:solar|{{:filereference:output:about_24.png?nolink |}}]] | 
 +| SAFAC | Solar Access on Facades and Roofs  | Solar Access for Facade/ Roofs and Single Walls  |   [[filereference:output:solarfacade|{{:filereference:output:about_24.png?nolink |}}]] |
 </WRAP> </WRAP>
  
 <WRAP clear></WRAP> <WRAP clear></WRAP>
  
-Other files are ASCII files. 
  
  
 +===== Text Files: Overview and description =====
 +
 +This list gives an overview over the different text output files generated by ENVI-met. All files are CSV text files that can be loaded directly in to Python (Pandas). Not all files may be generated in each simulation. This list refers to ENVI-met V5 and newer.
 +
 +<WRAP box left 99%>
 +
 +^ID   ^Content  ^Remark  ^
 +| .AT_1DR | Receptor Atmospheric data  (Profile) | Vertical profile atmosphere at receptor at a given model time |
 +| .AT_1DT | Receptor Atmospheric data  (Time Series)  | Vertical profile atmosphere at receptor for all model times of simulation in one file |
 +| .FX_1DT | Receptor Surface data (Time Series)  | Time series of ground surface state at receptor for all model times |
 +| .SO_1DR | Receptor Soil data (Profile)  | Vertical profile soil at receptor at a given model time  |
 +| .SO_1DT | Receptor Soil data (Time Series) | Vertical profile soil at receptor for all model times of simulation in one file  |
 +| .BLDG_status | Single Building Status (Time Series) | Actual status of individual buildings for all model times of simulation in one file (one for each building) |
 +| .BLDG_statistics | Building Statistics | Summary of static parameters for the buildings (one for each building) |
 +| .BLDG_list | Buildings Inventory | List of all buildings in model (all buildings in one file) |
 +| .VEG_status | Vegetation Status (Time Series)  | Actual status of 3D vegetation for all model times of simulation (only for observed plants) |
 +| .VEG_list | Vegetation Inventory  | List of all 3D plants and their key properties in the model |
 +| .Inflow1D | Profile Atmosphere Model at inflow boundary  | State of the 1D inflow boundary layer model at a given time up to 2500 m |
 +</WRAP>
 +<WRAP clear></WRAP>
 +
 +Please note, for all receptor files, you can produce output files with a higher resolution than the main output file intervall. All other files will be generated when the main model files are saved.
 +
 +=== Import text file into Python Pandas ===
 +
 +To import ENVI-met output text files with time series into Python Pandas, use:
 +
 +<WRAP box left 99%>             
 +<code Python>
 +import pandas as pd
 +
 +# this imports the (ENVI-met) text file "mysourcefile" into a Panda dataframe object.
 +# Comma "," is selected as seperator sign, 
 +# the "DateTime" column is flaged to be handeld as date information 
 +# and also is set as the default index column for data processing
 +                  
 +df = pd.read_csv("mysourcefile", sep=",", parse_dates=['DateTime'], index_col=['DateTime'])     
 +
 +</code>
 +</WRAP>
 +<WRAP clear></WRAP>
 +
 +Each Time Series files now contains a Python-compatible //DataTime// column to allow a direct use of the different Date and Time functions supplied by Python.
 +
 +
 +The LEONARDO DataStudio automatically recognizes the content of the ENVI-met output text files and offers specific scripts matching the file types.