File types: Difference between revisions
Jump to navigation
Jump to search
| Line 265: | Line 265: | ||
'''Some example (PBF with BINARY and SPARSEXY – test_49_ToA.pbf.dsc):''' | '''Some example (PBF with BINARY and SPARSEXY – test_49_ToA.pbf.dsc):''' | ||
<syntaxhighlight> | <syntaxhighlight> | ||
B000000001 B=binary / A=ASCII and file | B000000001 B=binary / A=ASCII and number = count of frames in multiframe file | ||
[F0] Index of frame in the file = 0 | [F0] Index of frame in the file = 0 | ||
Type=double [X,Y,C] width=256 height=256 | Type=double [X,Y,C] width=256 height=256 Data with type double, format X,Y,C = only hit pixels saved and has XY pos. | ||
"Acq Serie Index" ("Acquisition serie index"): Some metadata item name and (description) | "Acq Serie Index" ("Acquisition serie index"): Some metadata item name and (description) | ||
u32[1] Type of the item data [number of values] | u32[1] Type of the item data [number of values] | ||
| Line 300: | Line 300: | ||
[F1] Start of the second subframe | [F1] Start of the second subframe | ||
Type=i16 [X,C] width=256 height=256 | Type=i16 [X,C] width=256 height=256 Data with type i16, format X,C = only hit pixels saved and has matrix index. | ||
"Acq Serie Index" ("Acquisition serie index"): | "Acq Serie Index" ("Acquisition serie index"): | ||
u32[1] | u32[1] | ||
Revision as of 10:28, 9 January 2024
File type and extensions constants
These constants are file types and extensions. It can be used for filenames testing or with acquisition functions.
Python example:
# measure and save one 0.25 second frame to png file named "testFile.png"
dev.doSimpleAcquisition(1, 0.25, pixet.PX_FTYPE_PNG, "testFile")
dev.doSimpleAcquisition(1, 0.25, pixet.PX_FTYPE_AUTODETECT, "testFile.png")
| File type constants | File extensions constants | Exts |
|---|---|---|
| PX_FTYPE_NONE | (No direct file saving – data stored only in memory) | |
| PX_FTYPE_AUTODETECT | (FTYPE detected by extension in a filename) | |
| PX_FTYPE_ASCII_FRAME | PX_EXT_ASCII_FRAME | "txt" |
| PX_FTYPE_BINARY_FRAME | PX_EXT_BINARY_FRAME | "pbf" |
| PX_FTYPE_MULTI_FRAME | PX_EXT_MULTI_FRAME | "pmf" |
| PX_FTYPE_BINARY_MULTIFRAME | PX_EXT_BINARY_MULTI_FRAME | "bmf" |
| PX_FTYPE_TPX3_PIXELS | PX_EXT_TPX3_PIXELS | "t3p" |
| PX_FTYPE_TPX3_PIXELS_ASCII | PX_EXT_TPX3_PIXELS_ASCII | "t3pa" |
| PX_FTYPE_CLUSTER_LOG | PX_EXT_CLUSTER_LOG | "clog" |
| PX_FTYPE_PIXEL_LOG | PX_EXT_PIXEL_LOG | "plog" |
| PX_FTYPE_PNG | PX_EXT_PNG | "png" |
| PX_FTYPE_TPX3_RAW_DATA | PX_EXT_TPX3_RAW_DATA | "t3r" |
| PX_FTYPE_PIXET_RAW_DATA | PX_EXT_PIXET_RAW_DATA | "prd" |
| PX_FTYPE_EXTERNAL | (reserved) | |
| (description file saved automatically with pmf/txt) | PX_EXT_FRAME_DESC | "dsc" |
| (index file saved automatically with pmf/txt) | PX_EXT_INDEX | "idx" |
| txt | ASCII matrix: Text files with img lines converted to text lines with numbers separated by spaces. |
| pbf | Simple binary files, numbers only. |
| pmf | Multiple frames. Default is same as the txt, but multiple frames on top of each other. Can use BINARY flag. |
| t3pa | Tpx3 pixels ASCII. Text format, tab-separated columns with the header in the first row. Biggest to saving. |
| t3p | Tpx3 pixels. Binary format. Lower saved size, more complex to understand. |
| t3r | Tpx3 raw data. Fastest to saving, difficult to understand, slow to processing and can cause processing errors. |
| bstg | Binary settings file: Measured data with all configuration. See Binary Spectral Imaging API: BSTG files |
| clog, plog | Clusters/pixels logs. Text files contains clusters separated to frames with pixels lists. Historic formats for saving a data with few hited pixels in a frames. |
| h5 | HDF5, hierarchical data format 5. Used as one of multi-frame formats. |
| info | Text file with "[FileInfo]" head and all metadata list. |
| dsc | Text file with frame index first and all metadata list. |
| idx | Binary index for multi-frame files. Usesfull for fast access to n-th frame. |
File saving flags summary
File saving flags can do additional settings for file(s) saving.
- Can be used in saving files or in doAdvancedAcquisition python methods, for example.
- Flags can be combined.
- Default frame file settings is set of separate subframes text files, with all pixels include zeros, each subframe with idx+dsc files:
- file_ToT.pmf, file_ToT.pmf.dsc, file_ToT.pmf.idx, file_ToA.pmf, file_ToA.pmf.dsc, file_ToA.pmf.idx
| Flag constant base name | Description |
|---|---|
| PX_FRAMESAVE_BINARY | Use binary format in pmf. |
| PX_FRAMESAVE_SPARSEX | Index + non-zero pixels in file. # separates (sub)frs. |
| PX_FRAMESAVE_SPARSEXY | X, Y + non-zero pixel in file. # separates (sub)frames. |
| PX_FRAMESAVE_NODSC | Do not add dsc file. |
| PX_FRAMESAVE_NOSUBFRAMES | Do not use subframes, save main frame only. |
| PX_FRAMESAVE_SUBFRAMES_ONEFILE | Save all subframes to a single file. |
| PX_FRAMESAVE_SUBFRAMES_SAVEMAINFRAME | Save separate all subframes and main frame extra. |
- The file saving flags can be used in
- Python API: Use pixet.PX_FRAMESAVE_... constants in flags parameter of some measuring/saving methods.
- Binary API: Use PX_FRAMESAVE_... constants in flags parameter of some measuring functions.
- The Pixet program. Available in the More measurement settings, after filename was selected
 File saving flags in More measurement settings in Pixet |
File extensions and flags: TXT/PBF/PMF/BMF details
The files formats
| txt | Single frame in the text file. |
| pbf | Pixet Binary Frame - Single frame in the binary file. |
| pmf | Pixet Multi Frame - Multiframe file with text or binary format, depends on flags used with saving. |
| bmf | Binary Multi Frame - Multiframe file with binary format, depends on flags used with saving. |
| idx | Index for seeking - Binary array of 64b pointers to start of frames and subframes. |
| dsc | Description. List of all metadata for each frame and subframe. Actual device and acquisition parameters, data types, etc. The "Frame name" item can be helpful to orientation in pmf structure if the ONEFILE flag used. The Type=item is helpful to understanding the structure of data if the BINARY flag used. |
Multi-files names generation
- Note
- All the next examples are for Timepix3, single chip, opm = TPX3_OPM_TOATOT
- flags 0 (default), input filename = "name", acqCount = 1
- name_ToA.txt, name_ToA.txt.dsc, name_ToT.txt, name_ToT.txt.dsc
- acqCount = 6
- name_0_ToA.txt, name_0_ToA.txt.dsc, name_0_ToT.txt, ...
- ...
- name_5_ToA.txt, name_5_ToA.txt.dsc, ...
- PMF note
- With each pmf generating .pmf.idx binary file, other is same as TXT with acqCount = 1.
Files with flags=0
- Note
- All the next examples are for Timepix3, single chip, opm = TPX3_OPM_TOATOT
- TXT file data, default
- 0 0 0 5 0 0 0 … 256 numbers (int for non-calibrated values or float if the calibration used) and enter
- 0 872 0 0 0 … 256 numbers (int for non-calibrated values or float if the calibration used) and enter
- (256 lines)
- PMF file data, default
- 0.00000 78.65742 0.00000 … 256 numbers (int for non-calibrated values or float if the calibration used) and enter
- 0.00000 0.00000 999785.5 … 256 numbers (int for non-calibrated values or float if the calibration used) and enter
- (256 lines * acqCount)
| The PMF.IDX files generated beside the PMFs. Contains the simple binary array of little-endian qword addresses of the each frame start.
|
 .pmf.idx file example |
Flags influence to files
| _ToA.txt file | _ToT.txt file | ||
|---|---|---|---|
| px index | ToA | px index | ToT |
| 0 | 227212.500000 | 0 | 20 |
| 17 | 310685.937500 | 17 | 13 |
| 255 | 265487.500000 | 255 | 11 |
| 1274 | 105728.125000 | 1274 | 9 |
| - Lists of all hited pixels - ToT: int for non-calibrated data or float if the calibration used | |||
| _ToA.txt file | _ToT.txt file | ||||||
|---|---|---|---|---|---|---|---|
| X | Y | ToA | X | Y | ToT | ||
| 247 | 3 | 189851.562500 | 247 | 3 | 16 | ||
| 250 | 4 | 140042.187500 | 250 | 4 | 12 | ||
| 5 | 9 | 317195.312500 | 5 | 9 | 5 | ||
| - Lists of all hited pixels - ToT: int for non-calibrated data or float if the calibration used | |||||||
PMF file data, pixet.PX_FRAMESAVE_SPARSEX(Y) flag
Same as TXT, but containing single lines with only # to separate frames
| X | Y | ToA | Line description |
|---|---|---|---|
| 232 | 139 | 321620.312500 | frame 1, px 1 |
| 4 | 252 | 340231.250000 | frame 1, px 2 |
| # | frames separator | ||
| 39 | 0 | 258270.312500 | frame 2, px 1 |
| 201 | 0 | 76593.750000 | frame 2, px 2 |
| 92 | 1 | 268642.187500 | frame 2, px 3 |
- PX_FRAMESAVE_SUBFRAMES_ONEFILE
- All the data is in one file, subframes are placed one behind the other. If the measurement result has 10 frames with 2 subframes A/B, each _n TXT file contains 2 subrfames and the PMF contains 20 frames in order:
- sfr0A, sfr0B, sfr1A, sfr1B, ...
- The exact order and names of type of (sub)frames is listed in the DSC file. The DSC have separate records [Fn] for all the items.
- PX_FRAMESAVE_SUBFRAMES_SAVEMAINFRAME
- The group of the saved files contains the main frame and all subframes. Subframe files end in _sfrName, the main frame does not. In DSC file accompanying the TXT with main frame is not the "Frame name" item.
- Not applicable if combined with the ONEFILE flag.
- PX_FRAMESAVE_BINARY
- If the file type supports text and binary format, ex. PMF, save the binary.
- Not applicable to TXT, must use PBF instead.
- Data in the file are the simple array of non-calibrated 16 or 32b integers or calibrated doubles. See the DSC file for used data type.
| BINARY + SPARSEXY examples: |
 Example of data saved if flags BINARY+SPARSEXY used |
DSC files details
The first 3 lines have special functions:
- File type selection
- Index of the frame in this file
- Arrangement and type of the data
Other lines are a list of metadata items separated by blank lines.
In txt.dsc and pbf.dsc, end of the frame is end of the file.
In the pmf.dsc, next item 2 and 3 and frame or subframe data follows.
Some example (PBF with BINARY and SPARSEXY – test_49_ToA.pbf.dsc):
B000000001 B=binary / A=ASCII and number = count of frames in multiframe file
[F0] Index of frame in the file = 0
Type=double [X,Y,C] width=256 height=256 Data with type double, format X,Y,C = only hit pixels saved and has XY pos.
"Acq Serie Index" ("Acquisition serie index"): Some metadata item name and (description)
u32[1] Type of the item data [number of values]
49 The value
(more metadata items separated by blank lines …)
"Frame name" ("Frame name"):
char[3]
ToA This is the ToA frame
(more metadata items separated by blank lines …)
(end of the file)
Other example (PMF with BINARY+SPARSEX+ONEFILE – test.pmf.dsc):
B000000010
[F0] Start of the first subframe
Type=double [X,C] width=256 height=256 Pixel index and double type pixel data (ToA in ns)
"Acq Serie Index" ("Acquisition serie index"):
u32[1]
0
(more metadata items separated by blank lines …)
"Frame name" ("Frame name"):
char[3]
ToA
(more metadata items separated by blank lines …)
[F1] Start of the second subframe
Type=i16 [X,C] width=256 height=256 Data with type i16, format X,C = only hit pixels saved and has matrix index.
"Acq Serie Index" ("Acquisition serie index"):
u32[1]
0
(and the ToT frame metadata, [F2] and ToA subframe, [F3] and ToT sfr, … [Fn] and ToT sfr of (n/2)th frame)
Complete one frame DSC example (BINARY+SPARSEX – test_15_ToA.pbf.dsc):
B000000001
[F0]
Type=double [X,C] width=256 height=256
"Acq Serie Index" ("Acquisition serie index"):
u32[1]
15
"Acq Serie Start time" ("Acquisition serie start time"):
double[1]
1639059034.903085
"Acq time" ("Acquisition time [s]"):
double[1]
0.500000
"ChipboardID" ("Chipboard ID"):
char[9]
I08-W0060
"DACs" ("DACs"):
u16[19]
16 8 128 10 120 1301 501 5 16 8 16 8 40 128 128 128 256 128 128
"Frame name" ("Frame name"):
char[3]
ToA
"HV" ("High voltage [V]"):
double[1]
-500
"Interface" ("Readout interface"):
char[7]
MiniPIX
"Mpx type" ("Medipix type (1-MXR, 2-TPX, 3-MPX3, 4-TPX3, 5-TPX2)"):
i32[1]
4
"Pixet version" ("Pixet version"):
char[5]
1.7.8
"Start time" ("Acquisition start time"):
double[1]
1639059042.934810
"Start time (string)" ("Acquisition start time (string)"):
char[64]
Thu Dec 9 15:10:42.934809 2021
"Threshold" ("Threshold [keV]"):
double[1]
5.026744CLOG and CLOG.IDX files details
The CLOG format was developed to facilitate further processing of cluster data by the user programs. This is a text file divided to the frame records and the records can contain a clusters. Frames and clusters are separated by the line breaks. Frames can be separated by whole free line.
- The record format
- Frame FN (frameStart, frameAcqTime s)
- [x, y, energy, ToA] [x, y, energy, ToA] [x, y, energy, ToA] …
| FN | Frame index number. First 0 or 1. |
| frameStart | Start time of the frame. There are variants:
1. If it from measuring or from replay frame-based data with metadata available:
2. If it from pixel-based data with metadata available (file.t3pa + file.t3pa.info):
3. If it from replay data and metadata not available:
|
| frameAcqTime | Duration of the frame, float in seconds. Always 0.000000 in data from data-driven sources. |
| x, y | Position of the pixel. |
| energy* | Energy deposited in the pixel. Integer ToT counter value if not calibrated, float in keV if calibrated. |
| ToA* | Time of arrival, relative to frameStart. Integer in CLK ticks if ToA conversion is disabled, float in ns if ToA conversion is enabled. |
*ToA+energy records can be created from source that supports combined ToA+ToT modes, like as OPM_TOATOT on the Timepix3. If the data source supports only single modes, only one value is in this position.
Clog from data-driven source not contains free frames.
Clog from frame-based source can contains free frames.
Example records (Timepix3, Frame2 with two clusters by 2 and 4 pixels, Frame3 with single 2-pixel cluster)
Frame 2 (273697060.937500, 0.000000 s)
[214, 195, 43.1598, 0] [220, 191, 20.6515, 7.8125]
[224, 182, 21.8018, 31.25] [223, 186, 4.58576, 31.25] [222, 183, 38.2381, 31.25] [226, 185, 14.7623, 34.375]
Frame 3 (371034565.625000, 0.000000 s)
[151, 33, 32.5745, 0] [151, 34, 13.8135, 17.1875]
Example records (Timepix)
Frame 6 (1639143482.765164, 0.200000 s)
[87, 134, 5.75352] [217, 58, 14.8396]
Frame 7 (1639143483.019154, 0.200000 s)
Frame 8 (1639143483.261158, 0.200000 s)
Frame 9 (1639143483.513150, 0.200000 s)
The CLOG.IDX files generated beside the CLOGs. Contains the simple binary array of little-endian qword addresses of the "F" at each record start.
|
 Example .clog.idx file |
T3PA files details
The T3PA is timepix3 data file in text format with lines and tabs. Contains the information line and data lines with record index, pixel index, Time of arrival, Time over threshold, Fine ToA and Overflow.
Example:
Index Matrix Index ToA ToT FToA Overflow
0 1028 1918 14 22 0
1 1028 3126 8 28 0
2 1028 3778 5 23 0
...
156003 39793 98473646054 38 9 0
156004 190 98492090610 19 3 0- The Index is simple index of measurement line. This growing while measurement is running. If you append new measurement to existing file, new index is 0 again and again growing while new measurement is running.
Physical position of the x=0, y=0 pixel on the Minipix (1,1 in Pixet view)
- The Matrix Index is index of the pixel. On the Minipix Tpx3 is 0 at the left-down (see image)
- The ToA is time of arrival in units 25 ns, mod by limit, for example Minipix 264 (14600y), Advapix-single 230 (26s), Advapix-Quad 228 (6.5s).
- Note: The ToA on-chip implementation in the pixels is limited to 14 bits (409.6 µs).
- The ToA in T3PA is extended by device. But there is inherent uncertainty around the borders. These values may be incorrectly assigned. Users not comfortable with our extension can apply AND with (uint64)16383 to extended ToA to get original ToA from the chip.
- The ToT is time over threshold in units 25 ns.
- The FToA stands for "fine ToA" and it is the finest step of the ToA measurement. To properly account for this step in the conversion of ToA to time, it is necessary to subtract the amount of counts of fToA in the following manner:
- Time [ns] = 25*ToA - (25/16)*fToA
- The original range of this fToA value in the chip is 4 bits, or 16 values. This is extended in the post-processing of the data into 5 bits, or 32 values to include a correction for the delay of the clock propagation in the chip. The final value exported into t3pa files has a range of 5 bits, or 32 values, but the previous equation still stands.
- The Overflow is sign of data transfer overflow. If the line has this 1:
- index = 0x74: start of lost data
- index = 0x75: end of lost data, toa is length of the missing time
- (this can occurs with rates over megahits per seconds for Minipix)
- Note: In data from multichip devices, there is not Overflow, replaced by Chip index (But column name is still Overflow).
Pixel matrix configuration files
| bpc | Binary Pixel Configuration | All PM config in one file, meaning of the bits depends on the chip. |
| txt | Ascii Mask Matrix | Text file with pixel mask |
| txt | Ascii Test Bit Matrix | Text file with test bits |
| txt | Ascii THL adj. bits Matrix | Text file with threshold values adjustment |
Other files
- H5: The HDF5 files. Binary containers for structured data. If used to save, contains both measured data and metadata. To access theese files, use third party tools and libraries like as h5py for Python or HDF5 C++ API form HDF Group.
- See: Files and directories of the Pixet and SDK: pixet.ini
- See: Files and directories of the Pixet and SDK: Configuration XML files
- See: Files and directories of the Pixet and SDK: Device configuration ini files
- See: Files and directories of the Pixet and SDK: Device firmware files
- See: Binary Spectral Imaging API: BSTG files
- User XML settings: See: The ISetting object chapter in the Python API manual