The Marenostrum Numerical The Marenostrum Numerical
Cosmology Project Cosmology Project
-MNCP- -MNCP-
Gustavo Yepes Gustavo Yepes
Universidad Autónoma de Madrid Universidad Autónoma de Madrid
http://astro.ft.uam.es/marenostrum http://astro.ft.uam.es/marenostrum
The MareNostrum The MareNostrum
Numerical Cosmology Project Numerical Cosmology Project
The MNCP is an international collaboration of different institutions The MNCP is an international collaboration of different institutions
spread across the world with the aim of producing state-of-the-art spread across the world with the aim of producing state-of-the-art
Grand Challenge Cosmological Simulations Grand Challenge Cosmological Simulations of structure formation in the Universe.
of structure formation in the Universe.
This collaboration was established after the installation of the
Marenostrum supercomputer in
Barcelona in 2004 in which most of
the simulations have been done
People behind MNCP
(P.I.)Stefan Gottlöber (AIP)
(P.I. )Gustavo Yepes
(UAM)
MNCP
• In this collaboration, we use state-of-the art parallel MPI numerical codes to treat simultaneously:
• Gravity using N-body solvers
• Gas dynamics: SPH and AMR
• Cooling and heating from atomic processes
• Star formation and stellar feedbacks
MNCP
•
All codes are fully MPI parallel using different domain decomposition techinques:•
Fully Lagrangian codes:Fully Lagrangian codes:•
GADGET (TREEPM + SPH)•
GASOLINE ( TREECODE + SPH)•
GRID BASED CODE with Adaptive Mesh refinement GRID BASED CODE with Adaptive Mesh refinement techniques:techniques:
•
Adaptive Refinement Tree (ART)MNCP FIRST MNCP SIMULATION WAS DONE DURING THE TESTING PERIOD OF MARENOSTRUM I (2004-2005)
THE MARENOSTRUM UNIVERSE
Still the world’s largest gasdynamical SPH simulation of
large scale structure formation in the Universe
The The MareNostrum Universe MareNostrum Universe TREEPM+SPH simulation TREEPM+SPH simulation
Concordance cosmological Model
•
1,500 Million light-years computational box•
GADGET 2 code (Springel 2005)•
Adiabatic SPH+TREEPM Nbody•
10243 FFT for the PM force.•
15 kpc force resolution.•
2x10243 dark and sph particles•
109.33 partículas•
8x109 M dark matter•
109 M for gas particles•
1 million dark halos bigger thana typical galaxy (1012 Mo)
•
Simulation done at MareNostrum 1 (2005)•
512 processors (1/20th total power)•
1Tbyte RAM•
500 wallclock hrs (29 cpu years)•
Output: 8600 Gbytes of data.The The MareNostrum Universe MareNostrum Universe TREEPM+SPH simulation TREEPM+SPH simulation
Movie:
Movie:
•
The time evolution of the gas component at the position of the largestgalaxy cluster formed in the simulation.
Colors
correspond to logarithm of the gas density
The The MareNostrum Universe MareNostrum Universe TREEPM+SPH simulation TREEPM+SPH simulation
ovieovie: A fly-by : A fly-by through the through the
simulation box.
simulation box.
The movie The movie shows the shows the
density of gas density of gas
color coded color coded according to according to
logarithm of the logarithm of the
density density..
MNCP
Most computationaly expensive simulation done up to now by MNCP:
THE MARENOSTRUM HIGH-Z GALAXY FORMATION SIMULATION
After more than 2 centuries of CPU time the simulation has just evolved a patch of universe from an age of few hundred
million years to few billion years, still too young compared with its present age.
The MareNostrum High-z The MareNostrum High-z
GALAXY FORMATION SIMULATION GALAXY FORMATION SIMULATION
Gasdynamics and N-body with 2 billion particles
+
Detailed modelling of baryonic physics.
To study in detail the galaxy formation process we need to account at least for To study in detail the galaxy formation process we need to account at least for
• Radiative and Compton coolingRadiative and Compton cooling
• UV-photoionizationUV-photoionization
• Multiphase ISM.Multiphase ISM.
• Star Formation.Star Formation.
• Star-Gas backreactionsStar-Gas backreactions..
Use Springel-Hernquist (2003) implementation of multiphase SPH modeling Use Springel-Hernquist (2003) implementation of multiphase SPH modeling
in GADGET-2.
in GADGET-2.
z = 5.6 z = 5.6
MareNostrum galaxy formation simulation MareNostrum galaxy formation simulation
Box 1.5x106 light-years 2x 109 particles gas+dark ΛCDM model
Mgas=1.4 x106 Msun Mdark=107 Msun.
Mhalos > 109 Msun Picture:
A slice of Gas density
z = 5.6 z = 5.6
MareNostrum galaxy formation simulation MareNostrum galaxy formation simulation
Box 50/h Mpc 2x 10
9
gas+dark
LCDM model Mgas=1.4 x10
6
Msun
Mdark=10
7
Msun.
Mhalos > 10
9
Msun
Mass Function:
Abundance of galaxies when the Universe was 1/3 of its
present age
z = 11.2
z = 11.2 z = 7.3z = 7.3
z = 5.6 z = 5.6
MareNostrum galaxy formation simulation MareNostrum galaxy formation simulation
Picture of the Simulation taken from the light coming from stars formed in the simulated volume. Note the enormous amount of tiny objects much smaller than present day galaxies. These are the protogalaxies that will eventually form the present day galaxies by mergers.
his is a mock CCD image a big-enough telescope could take from the real universe proviided it could point to the same position in the sky for more than 10
6
seconds
CODE COMPARISON
MNCP GADGET (SPH) MNCP GADGET (SPH)
800 processors of MN Resolution: 500 pc.
400 YEARS of CPU
http://astro.ft.uam.es/marenostrum
HORIZON -RAMSES (AMR) HORIZON -RAMSES (AMR)
More than 2000 processors Resolution: 2 kpc
150 YEARS CPU
http://www.projet-horizon.fr
MNCP
Projects under development:
High resolution simulations of the formation and High resolution simulations of the formation and
evolution of evolution of
Clusters and Groups of Galaxies Clusters and Groups of Galaxies
A different approach:
Select a representative sample of galaxy groups and clusters from a large box and resimulate them with very accurate
physics and resolution
Adaptive multi-mass to achieve high resolution:
Re-Simulated areas from large
computational boxes by resampling
particles of increasing mass away from the refined region:
Original initial conditions up to 40963 particles in a big box.
Trace back particles of selected objects to identify
region to be resimulated with very high resolution
Very easy way of parallelization.
GALAXY CLUSTERS
ART eulerian Nbody+
Hydro code :
A Kravtsov & D. Rudd
ART Cluster Sample
2 4 0 M p c/ h
Low Resolution simulation produced sample of ~250
groups and clusters
Run using 128 Marenostrum processors using a total of
30k CPU hours
5123 particles, over 2 billion
AMR grid cells
High-Resolution Re-simulations
3.6 Mpc/ h
13 most massive
objects
re-simulated at high mass & spatial
resolution (~5-7 kpc, 7.5x108 M☉)
1-2 million particles,
~40 million grid cells per cluster
Run using 384 Marenostrum
processors using 340k CPU hours in
the past year
High-Resolution Re-simulations
Dark Matter
Gas
Stars
High-Resolution Re-simulations
Dark Matter
Gas
Stars
z = 1 z = 0.5 z = 0
SIMULATING GROUPS OF GALAXIES :
1 CODES used:
2 GASOLINE and GADGET 3 In collab. with R. Feldmann
and L. Mayer (Zurich)
•
Galaxy Groups
•
Bound structures with masses 10
12
-10
14
M
⊙
, typically up to ~50 observable galaxies, sizes ~ 1 Mpc; likely dominant place of Galaxy evolution
Set-up of the simulations
• TreeSPH code Gasoline (Wadsley et al. 2004)
• Collisionless DM particles, Eq. of Motion solved efficiently with a Tree-Code
• Gas component followed with Smooth Particle Hydrodynamics (SPH), a particle-based Lagrangian formulation of the Euler-equations
• Star formation, SN feedback, UV-background, metal enrichment
Initial conditions • Several layers of different resolution
• SPH particles only in the highest-res region
x1 x8 x64
grafic-2
Simulation Runs at the MN
Sim Remark CPUh Proc
G2-VHR-DM Group G2; DM-only; very high resolution 60’000 128 G2-hE-hF SPH; 2x standard softening; high feedback 30’000 64
G2-hE-lF SPH; 2x standard softening; low feedback 30’000 64 G2-hT SPH; high initial gas temperature, high time
resolution
60’000 64
G1 Group G1 (isolated group) 64’000 128
G3 Group G3 (near cluster) 250’000 128
Sum ~500’000
memory allocation of all runs except G2-VHR-DM: ~16 GByte memory allocation of G2-VHR-DM (estimated): ~100 GByte
in total 3 Groups: G1-G3
Central galaxies
High-z galaxies:
Observations indicate evolved stable disks at z~2-3, with high SFR (Foerster-Schreiber et al. 2006, Genzel et al. 2006, Stark et al. 2008
)
What do self-consistent Hydrodynamical simulations predict?
We are currently studying our high-z central
galaxies:
high SFR ~ 30 Msol/yr at z~2.5(but less than the Genzel et al. 2006 object ~140 Msol/yr, but the uncertainty on the latter is huge +->80 Msol/yr)
Central object has a cold-gas disk of
~3kpc radius. M~5x10^9 Msol at z=2.5.
(left) Face-on, (right) Edge-on
z=2.5
Feldmann et al. in prep
Satellite Galaxies
We see indeed the action of the group environment!
Merging
Tidal tails Ram-pressure
500 kpc (com)
gas star
z=2.5
Edge-on view on disk. DM (blue), stellar (red) and cold gas (green).
The stripping of the gas disk is clearly visible.
The group at z~2. Tidal streams from mergers between satellites and the central object are visible.
A high-redshift merger between gaseous disks
Galaxy Group Formation
Movie: This movie shows the time evolution of one of our
galaxy group objects
simulated with GASOLINE at high resolution. We display the sum of dark matter (blue),
stars (red) and gas(green) is displayed
SIMULATING OUR OWN LOCAL GROUP
WE LIVE HERE
Constrained Simulations of the Constrained Simulations of the
Local Universe Local Universe
The perfect tool to study the formation of individual objects, that look like those close to us, starting from cosmological initial conditions and in a realistic environment.
•
Eg. Virgo, Coma, the Local filament .. or the Local Group.• An excellent laboratory to investigate how dark matter
is distributed and structured in a similar environment
than our own galaxy .
Observational Constrains Observational Constrains
Mass and velocity constrains Masses of nearby X-ray clusters
Reiprich & Bohringer 2002 Peculiar Velocities taken from
MARK3, SBF (large scale)
(YH, Klypin,Gottlober,Kravtsov ,2002)
+
Karantchenstev et al. ( LG) Cosmological Model:
WMAP3 parameters
Simulated Volumes Simulated Volumes
Box 160/h Mpc:
CS: 2563 density field
COMA, LSC, PP, GA, Virgo
Resimulated box with much higher
resolution:
Make random realization of LCDM P(K) in a 40963 mesh.
Substitute fourier modes corresponding to those from the 2563 CR.
Apply Zeldovich approx to find displacement fields
Fill box with arbitrary number of particles up to the 40963 maximum.
COMA
G.A. P-P
LSC
Simulated Volumes Simulated Volumes
Box 160/h Mpc:
CS: 2563 density field
COMA, LSC, PP, GA, Virgo
Our biggest runs:
Our biggest runs:
10243 particles filling the box 1.2 kpc, 2.5x108 Msun Resimulated area around LSC
40963 particles (4x106 Mʘ), 300 pc resolution.
300 million particles total.
ART N-body code.
COMA
G.A. P-P
LSC
Simulating the Local Universe
Simulations including observational constrains in the initial conditions from the distribution of galaxies in our neighbourhood (R <100Mpc) can reproduce the mass structures observed around us
300 kpc
250K Msun:
40963 effective particles ~ 7x1010 in
whole box.
Only 60 million with zoomed
simu.
Galaxy formation in the LG
• Overall, the Local Group object found in the constrained simulation looks quite realistic
•
Environment, internal dynamics, halo structures•
It can be used as a cosmological lab for galaxy formation to It can be used as a cosmological lab for galaxy formation to test different modeling of the various baryonic processes:test different modeling of the various baryonic processes:
(“gastrophysics”) and compare results with observations:
(“gastrophysics”) and compare results with observations:
Disk structure, Star formation history, HI and metal distributions, Local UV sources, surviving satellites…
Disk galaxies in DM halos
Effects of baryons on dark halos
• M31
• DM only
• 40963 res
• DARK MATTER DENSITY
Effects of baryons on dark halos
M31
GAS+DM+STARS
20483 res
Adiabatic Compression More compact dark mater halos Spherical objects
Indirect Dark Matter Detection
Make simulated maps of γ -ray detection by FERMI (former GLAST)
Study effects of substructure
Signal from nearby galaxies in the LG.
Better than Via Lactea simulation because we include baryons
.
LG 20483 Dark Matter Only
LG 20483 dark + baryonsNATURE OF THE DARK MATTER NATURE OF THE DARK MATTER
cold or warm ? cold or warm ?
M
wdm
=3 keV M
wdm
=2 keV
M
wdm
=1 keV
RESIMULATING THE Local Group in a different
cosmological setting:
WDM particles:
Mwdm = 3keV – 1 keV Comparison with ΛCDM:
•density profiles
•substructure mass functions
MILKY WAY IN CDM vs WDM
Counting the number of
substructures
surviving within the halos and
comparing with those found in the Milky Way, one can put limits to the mass of the particles
composing the dark matter:
THE MISSING SATELLITE THE MISSING SATELLITE
PROBLEM PROBLEM
CDM vs WDM CDM vs WDM
Λ WDM 3keV Λ WDM 1keV Λ CDM
Constraints on the Mass of the dark matter
particle from simulations: M > 3kev
SUMARIO SUMARIO
•
La Creación de Universos Virtuales realistas en una de las herramientas La Creación de Universos Virtuales realistas en una de las herramientas fundamentales para considerar a la Astrofísica y la Cosmología como fundamentales para considerar a la Astrofísica y la Cosmología comoverdaderas ciencias experimentales:
verdaderas ciencias experimentales:
•
Es el laboratorio natural donde hacer experimentos con las componentes del universo y sus interacciones físicas.• Nos permite adentrarnos en épocas todavía no accesibles a la observación y predecir que podemos esperar ver.
•
Las simulaciones cosmológicas son uno de los desafíos computacionales más Las simulaciones cosmológicas son uno de los desafíos computacionales másimportantes. Debido a que la gravedad es una fuerza no saturante, es muy difícil derivar importantes. Debido a que la gravedad es una fuerza no saturante, es muy difícil derivar
algoritmos capaces de distribuir los elementos computacionales que describen los algoritmos capaces de distribuir los elementos computacionales que describen los
distintos fluidos de forma eficiente entre miles o decenas de miles de procesadores distintos fluidos de forma eficiente entre miles o decenas de miles de procesadores
•
Es necesario realizar un trabajo de desarrollo de códigos paralelos considerable. Es necesario realizar un trabajo de desarrollo de códigos paralelos considerable.•
El paralelismo en este campo está en sus comienzos, menos de 10 años de vida…El paralelismo en este campo está en sus comienzos, menos de 10 años de vida…•
Grid super-computing puede ayudar a resolver el problema..Grid super-computing puede ayudar a resolver el problema..•
The The•
MareNostrum Numerical Cosmology ProjectMareNostrum Numerical Cosmology Project, pretende unir esfuerzos a nivel , pretende unir esfuerzos a nivel internacional para abordar problemas deinternacional para abordar problemas de grand challenge grand challenge que necesitan de capacidades que necesitan de capacidades computacionales extremas y de recursos humanos suficientes para el análisis de los
computacionales extremas y de recursos humanos suficientes para el análisis de los datos numérico
datos numéricos.s.