Publications of the computing group for ultrafast imaging
Stevanovic U., Caselle M., Cecilia A., Chilingaryan S., Farago T., Gasilov S., Herth A., Kopmann A., Vogelgesang M., Balzer M., Baumbach T., Weber M.
in IEEE Transactions on Nuclear Science (2015). DOI:10.1109/TNS.2015.2425911
High-speed X-ray imaging applications play a crucial role for non-destructive investigations of the dynamics in material science and biology. On-line data analysis is necessary for quality assurance and data-driven feedback, leading to a more efficent use of a beam time and increased data quality. In this article we present a smart camera platform with embedded Field Programmable Gate Array (FPGA) processing that is able to stream and process data continuously in real-time. The setup consists of a Complementary Metal-Oxide-Semiconductor (CMOS) sensor, an FPGA readout card, and a readout computer. It is seamlessly integrated in a new custom experiment control system called Concert that provides a more efficient way of operating a beamline by integrating device control, experiment process control, and data analysis. The potential of the embedded processing is demonstrated by implementing an image-based trigger. It records the temporal evolution of physical events with increased speed while maintaining the full field of view. The complete data acquisition system, with Concert and the smart camera platform was successfully integrated and used for fast X-ray imaging experiments at KIT’s synchrotron radiation facility ANKA.
Caselle M., Brosi M., Chilingaryan S., Dritschler T., Judin V., Kopmann A., Mueller A.-S., Raasch J., Smale N.J., Steinmann J., Vogelgesang M., Wuensch S., Siegel M., Weber M.
in 2014 19th IEEE-NPSS Real Time Conference, RT 2014 – Conference Records (2015), 7097535. DOI:10.1109/RTC.2014.7097535
© 2014 IEEE. Since a few years Coherent Synchrotron Radiation (CSR) generated by short electron bunches is provided at the ANKA synchrotron light source. To study the THz emission characteristics over multiple revolutions superconducting YBa2Cu3O7-δ (YBCO) thin-film detectors can be used. The intrinsic response time of YBCO thin films is in the order of a few picoseconds only. For fast and continuous sampling of this individual ultra-short terahertz pulses a novel digitizer system has been developed with programmable sampling times in the range of 3 to 100 ps. The Real-time system is based on a heterogeneous FPGA and GPU architecture for on-line pulse reconstruction and evaluations of the peak amplitudes and the time between consecutive bunches. The data is transmitted to a GPU computing node by a fast data transfer link based on a bus master DMA engine connected to PCI express endpoint logic. This new DMA architecture ensures a continuous high data throughput of up to 4 GByte/s. The presented DAQ system is able to resolve the bursting behavior of single bunches even in a multi-bunch environment and to study the bunch-bunch-interactions.
Rota L., Caselle M., Chilingaryan S., Kopmann A., Weber M.
in 2014 19th IEEE-NPSS Real Time Conference, RT 2014 – Conference Records (2015), 7097561. DOI:10.1109/RTC.2014.7097561
© 2014 IEEE. PCI Express (PCIe) is a high-speed serial point-to-point interconnect that delivers high-performance data throughput. KIT has developed a Direct Memory Access (DMA) engine compatible with the Xilinx PCIe core to provide a smart and low-occupancy alternative logic to expensive commercial solutions. In order to maximize the PCIe throughput the DMA engine adopts a new strategy, where the DMA descriptor list is stored inside the FPGA and not in the central memory system. The FPGA design package is complemented with a simple register access to control the DMA engine by a Linux driver. A handshaking sequence between the DMA engine and the Linux driver ensures that no errors occure, even in data transfers of several hundreds of Gigabytes. The design has been tested with Xilinx FPGA Families 6 and 7, and operates with the Xilinx PCIe endpoint generation 1 and 2 with all lane configurations (x1, x2, x4, x8, x16). Data throughput of more than 3.4 GB/s has been achieved with a PCIe Gen 2 ×8 lanes endpoint. The proposed DMA is currently used in several experiments at the ANKA synchrotron light source.
Amsbaugh, J.F. et al.
in Nuclear Instruments and Methods in Physics Research, Section A: Accelerators, Spectrometers, Detectors and Associated Equipment Volume 778, 1 April 2015, Pages 40-60
The focal-plane detector system for the KArlsruhe TRItium Neutrino (KATRIN) experiment consists of a multi-pixel silicon p-i-n-diode array, custom readout electronics, two superconducting solenoid magnets, an ultra high-vacuum system, a high-vacuum system, calibration and monitoring devices, a scintillating veto, and a custom data-acquisition system. It is designed to detect the low-energy electrons selected by the KATRIN main spectrometer. We describe the system and summarize its performance after its final installation. © 2015 Elsevier B.V. All rights reserved.
Shkarin A., Ametova E., Chilingaryan S., Dritschler T., Kopmann A., Vogelgesang M., Shkarin R., Tsapko S.
in Fundamenta Informaticae, 141 (2015) 259-274. DOI:10.3233/FI-2015-1275
© 2015 Fundamenta Informaticae 141. The recent developments in detector technology made possible 4D (3D + time) X-ray microtomographywith high spatial and time resolutions. The resolution and duration of such experiments is currently limited by destructive X-ray radiation. Algebraic reconstruction technique (ART) can incorporate a priori knowledge into a reconstruction model that will allow us to apply some approaches to reduce an imaging dose and keep a good enough reconstruction quality. However, these techniques are very computationally demanding. In this paper we present a framework for ART reconstruction based on OpenCL technology. Our approach treats an algebraic method as a composition of interacting blocks which performdifferent tasks, such as projection selection, minimization, projecting and regularization. These tasks are realised using multiple algorithms differing in performance, the quality of reconstruction, and the area of applicability. Our framework allows to freely combine algorithms to build the reconstruction chain. All algorithms are implemented with OpenCL and are able to run on a wide range of parallel hardware. As well the framework is easily scalable to clustered environment with MPI. We will describe the architecture of ART framework and evaluate the quality and performance on latest generation of GPU hardware from NVIDIA and AMD.
Shkarin R., Ametova E., Chilingaryan S., Dritschler T., Kopmann A., Mirone A., Shkarin A., Vogelgesang M., Tsapko S.
in Fundamenta Informaticae, 141 (2015) 245-258. DOI:10.3233/FI-2015-1274
© 2015 Fundamenta Informaticae 141.On-line monitoring of synchrotron 3D-imaging experiments requires very fast tomographic reconstruction. Direct Fourier methods (DFM) have the potential to be faster than standard Filtered Backprojection. We have evaluated multiple DFMs using various interpolation techniques. We compared reconstruction quality and studied the parallelization potential. A method using Direct Fourier Inversion (DFI) and a sinc-based interpolation was selected and parallelized for execution on GPUs. Several optimization steps were considered to boost the performance. Finally we evaluated the achieved performance for the latest generation of GPUs from NVIDIA and AMD. The results show that tomographic reconstruction with a throughput of more than 1.5 GB/sec on a single GPU is possible.
Gehrke R., Kopmann A., Wintersberger E., Beckmann F.
in Synchrotron Radiation News, 28 (2015) 36-42. DOI:10.1080/08940886.2015.1013420
© Taylor & Francis. The Helmholtz Association is the largest scientific organization in Germany. It operates all major German research infrastructures involved in research with photons, neutrons, and ions. These are DESY in Hamburg; the Karlsruhe Institute of Technology (KIT); the Research Centre Jülich (FZJ); the Helmholtz Centres in Geesthacht (HZG), Berlin (HZB), and Dresden-Rossendorf (HZDR); and the GSI Centre for research with heavy ions in Darmstadt. In common, all these centers are facing similar challenges related to dramatically increasing data rates and volumes generated with more and more powerful radiation sources together with larger and faster detectors. On the other hand, each center has its own specific portfolio of long-lasting technical expertise in areas like data analysis, information technology, or hardware development. Therefore, it was obvious to address the challenges by acting in concert. This was the main motivation in 2010 for the launch of a joint project among the partners called the “High Data Rate Processing and Analysis Initiative (HDRI).” The initiative is organized into three basic work packages: “Data Management,” “Real-time Data Processing,” and “Data Analysis, Modelling, and Simulation.” The aim is to carry out the development of methods, hardware components, and software for data acquisition, real-time and offline analysis, documentation and archiving, and for remote access to data. The solutions are finally meant to be integrated at the various experimental stations and thus have to be versatile and flexible to cope with the heterogeneous requirements of the different experiments. The claim to create standard solutions makes it mandatory to closely collaborate with large international activities in the field of data handling, like the European PaNdata project (see article in this issue), but also with vendors of detectors, data evaluation software, etc., as well as with corresponding standardization bodies.
Brosi M., Caselle M., Hertle E., Hiller N., Kopmann A., Muller A.-S., Schonfeldt P., Schwarz M., Steinmann J.L., Weber M.
in 6th International Particle Accelerator Conference, IPAC 2015 (2015) 882-884.
Copyright © 2015 CC-BY-3.0 and by the respective authors. The ANKA storage ring of the Karlsruhe Institute of Technology (KIT) operates in the energy range from 0.5 to 2.5 GeV and generates brilliant coherent synchrotron radiation in the THz range with a dedicated bunch length reducing optic. The producing of radiation in the so-called THz-gap is challenging, but this intense THz radiation is very attractive for certain user experiments. The high degree of compression in this so-called low-alpha optics leads to a complex longitudinal dynamics of the electron bunches. The resulting micro-bunching instability leads to time dependent fluctuations and strong bursts in the radiated THz power. The study of these fluctuations in the emitted THz radiation provides insight into the longitudinal beam dynamics. Fast THz detectors combined with KAPTURE, the dedicated KArlsruhe Pulse Taking and Ultrafast Readout Electronics system developed at KIT, allow the simultaneous measurement of the radiated THz intensity for each bunch individually in a multibunch environment. This contribution gives an overview of the first experience gained using this setup as an online diagnostics tool.
Caselle M., Brosi M., Chilingaryan S., Dritschler T., Hiller N., Judin V., Kopmann A., Muller A.-S., Raasch J., Rota L., Petzold L., Smale N.J., Steinmann J.L., Vogelgesang M., Wuensch S., Siegel M., Weber M.
in International Beam Instrumentation Conference, IBIC 2014 (2014).
The ANKA storage ring generates brilliant coherent synchrotron radiation (CSR) in the THz range due to a dedicated low-ac-optics with reduced bunch length. At higher electron currents the radiation is not stable but is emitted in powerful bursts caused by micro-bunching instabilities. This intense THz radiation is very attractive for users. However, the experimental conditions cannot be easily reproduced due to those power fluctuations. To study the bursting CSR in multi-bunch operation an ultra- fast and high-accuracy data acquisition system for recording of individual ultra-short coherent pulses has been developed. The Karlsruhe Pulse Taking Ultra-fast Readout Electronics (KAPTURE) is able to monitor all buckets turn-by-turn in streaming mode. KAPTURE provides real-time sampling of the pulse with a minimum sampling time of 3 ps and a total time jitter of less than 1.7 ps. The KAPTURE system, the synchrotron operation modes and beam test results are presented in this paper.
Stevanovic U., Caselle M., Balzer M., Cecilia A., Chilingaryan S., Farago T., Gasilov S., Herth A., Kopmann A., Vogelgesang M., Weber M.
in 2014 19th IEEE-NPSS Real Time Conference, RT 2014 – Conference Records (2014), 7097495. DOI:10.1109/RTC.2014.7097495
© 2014 IEEE. High-speed X-ray imaging applications such as radiography and tomography play a crucial role for non-destructive investigations in material and biology sciences. For data-intensive applications, on-line analysis of the data is necessary for initial quality assurance and data-driven feedback. In this article we will present a new smart camera platform, with embedded FPGA processing that is able to stream and process data continuously in real-time. It is used in the new imaging beamline IMAGE, in ANKA. The new smart camera platform consists of a CMOS sensor, an FPGA readout card connected with a high speed PCIe interface to the GPU-based readout computer. It is tightly coupled to a newly implemented control system, called Concert. Concert enables efficient operation of the beamline by integrating devices and experiment process control, as well as data analysis. A key feature of smart cameras is embedded image processing. In this article we will demonstrate the potential of this approach with the implementation of an image-based self-event trigger. The algorithm automatically restricts the readout to selected regions with changed content. Application dependent trigger parameters are hidden by our control system which sets them automatically according to experiment requirements and conditions.