GridFactory

Taking citizen cyberscience to the next level

admin — Fri, 19 Aug 2011 11:32:37 +0000

Recently I’ve stumbled upon the terms citizen science and citizen cyberscience. The last term was apparently invented by Shuttleworth Fellow Francois Grey as a label for BOINC-based distributed computing projects like seti@home, folding@home and his own lhc@home. Grey is also behind the Citizen Cyberscience Centre in Geneva - on the web pages of which there’s more information to be found about his motivation and ideas.

All fascinating stuff IMO. Not so much because of the direct scientific impact these distributed computing projects may or may not have, but because of the potential of getting more people interested in science and “academic” knowledge. Like the Khan Academy, an example of how the Internet really is improving the world.

Now, what does this have to do with GridFactory?

A lot actually! I’ll argue that GridFactory is conceptually a logical evolution of the ideas behind distributed computing software like BOINC (there is also the older distributed.net). If you visit the BOINC web site or the Wikipedia list of distributed computing projects, you’ll notice that despite what Grey and others, say, distributed computing is not really enabling “ordinary” citizens to do science… yet.

With BOINC-based systems citizens are passively witnessing their computer crunching away on some professional scientist’s problem. Yes, that might stimulate interest in the scientific problem at hand and science in general and that seems to be the hope and ambition of the enthusiasts behind these projects. A very commendable ambition IMO.

But wouldn’t it be nice if the citizen could actually take a look at the script/code he’s executing, try to understand it, change some parameters, run the modified code himself on his own PC… heck, create his own computing grid and run his modified code on all participating computers?

With GridFactory, all this and more is possible: collaborations are formed and destroyed on the fly; initiators of a collaboration can set up a software catalog (probably starting with a copy of someone else’s), a shared storage area and a compute cloud using their own, someone else’s, or even a central common server.

Of course, if GridFactory were to be used for large-scale, voluntary, distributed computing, like BOINC - with legions of workers carrying out the computations of a few, i.e. used to create a large grid of untrusted workers, it probably would not be a good idea to make it too easy for the legionnaires to mess with the code. In such cases, jobs can simply consist in booting up a locked-down virtual machine from a trusted software catalog.

That said, one well-known problem of some BOINC projects is precisely the scale and the fact that contributors are so eager to contribute more than their peers, that some cheat and produce fake results. Therefore, smaller collaborations are not necessarily bad. The world may need global collaborations to solve global problems, but some level of fragmentation or compartmentalization may have its merits too. In smaller groups, people know each other and there’s less incentive to fake results and more opportunity for real collaboration and involvement. Sure, you’ll not be running on 500′000 CPUs like seti@home, but there are many “smaller” problems out there that deserve attention and that may not need millions of CPU-hours to yield useful results.

The GridFactory equivalent of the BOINC client is the GridWorker. Like BOINC, GridFactory also uses Apache and MySQL on the server side, but where BOINC implements the actual web service as CGI scripts, GridFactory’s web services are implemented as Apache modules and designed to “talk” to each other, i.e. pull jobs from each other, and allow horizontal scale-out. But what really sets GridFactory aside, as compared to BOINC, is the integrated software catalog, the group functionality and the GridPilot GUI for creating and managing compute jobs.

Finally, I’ll add that I’m not saying GridFactory could or should replace BOINC in large-scale voluntary computing projects - it is a far less mature software product; but I am hoping that the ideas and concepts behind GridFactory may serve as a source of inspiration for future developments of citizen cyberscience and help the overall democratization of the enterprise of science.

The vision of GridFactory encompasses precisely this: democratization of citizen science - allowing citizens to not only passively contribute to science, but to engage and actually do science.

CERN School of Computing 2011 was… awesome

admin — Fri, 19 Aug 2011 08:24:25 +0000

As exemplified on the pictures above (from the photo gallery of one of the students), CERN School of Computing includes a good deal of extracurricular activities. Which probably goes a long way in explaining the good and humorous atmosphere in the classroom.

Having spent a lot of energy preparing the exercise with the great CSC team, it really was great fun to see my software in the hands of 60 energetic students from all over the world. Here you have the exercise as presented by CSC:

http://cernvm.cern.ch/portal/csc#ex2

In just a few minutes, these guys (mostly non-physicists) carried out the vision of GridFactory: created a classroom cloud and started simulating and anlyzing 500′000 collision events in the Large Hadron Collider at CERN.

The actual student exercise was formulated as a competition (see http://cernvm.cern.ch/portal/csc#ex2) - the winner being the first to create a histogram showing the result of an analysis of 500′000 simulated TTBar events in the Large Hadron Collider.

Worker nodes were fired up and jobs submitted. Obviously, if all had been equally fast in doing this and no collaboration was taking place, each might as well have run the exercise on her own PC. Equally clear, if there had been an overall aim of just generating one histogram with 500′000 events as fast as possible, all should have fired up worker nodes and only one should’ve submitted jobs. In the end what happened was that a few were very fast in submitting jobs and quickly 25 different jobs were queued and running, whereas the jobs of the rest were submitted later and ended up sitting in the queue. Interestingly, the ones who won were apparently not among the fast submitters. Instead they realized that all were running with the same credentials on the server and that therefore they could simply download the output files of the first 25 different jobs to finish and make a histogram. Congrats to the winners

CERN School of Computing 2011 - Exercise 2

admin — Thu, 07 Jul 2011 12:53:23 +0000

In this exercise we’ll solve the same assignment as in exercise 1, but using a prepackaged GridPilot application.

Download and install GridPilot
Start GridPilot and answer the initial questions - enable only the computing system “GridFactory” and set the submission host to gridfactory.nbi.dk
Import the app “ttbar_exercise”
Select the application/dataset “ttbar_exercise-100k-events” and click “Run”
Select the application/dataset “ttbar_exercise-merge” and click “Run”
Open the merged output file with Root: root ttbar-analysis.root - inside Root, type TBrowser b;

CERN School of Computing 2011 - Exercise 1

admin — Thu, 07 Jul 2011 12:53:17 +0000

Exercise 1: LHC Monte Carlo event generation and analysis

One of the final plots.

This exercise was created for the 2011 CERN CERN School of Computing, hosted by the Niels Bohr Institute, University of Copenhagen.

Credits: physics content of this exercise by Jørgen Beck Hansen from the Niels Bohr Institute.

Assignment: Generate 500′000 LHC t-tbar events with PYTHIA and analyze the data: find bosons and top hadrons, run jet-‐algorithm: cone R<0.7

Prerequisites

Download and install VirtualBox.
Download and fire up a CernVM. It’s probably easiest to use the installer.
Configure your CernVM to use the repositories hepsoft, grid and sft and setup a user account.
Login on your new CernVM.

Setup GridFactory:

export PATH=$PATH:\
/cvmfs/sft.cern.ch/lcg/external/experimental/gridfactory.org/gridfactory_ui:\
/cvmfs/sft.cern.ch/lcg/external/experimental/gridfactory.org/gridworker:\
/cvmfs/sft.cern.ch/lcg/external/Java/JDK/1.6.0/ia32/bin

Start a gridworker:
in a separate shell:
```
gridworker.sh [-n] https://gridfactory.nbi.dk/db/
```
Hint: If you have X enabled, you can run in graphical mode by leaving out the -n

Setup Root:
In a 32-bit VM:

source /cvmfs/sft.cern.ch/lcg/external/gcc/4.3.2/i686-slc5-gcc43-opt/setup.sh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:\
/cvmfs/sft.cern.ch/lcg/external/ROOT/5.28.00/slc4_ia32_gcc34/root/lib
export PATH=$PATH:\
/cvmfs/sft.cern.ch/lcg/external/ROOT/5.28.00/slc4_ia32_gcc34/root/bin

In a 64-bit VM:

source /cvmfs/sft.cern.ch/lcg/external/gcc/4.3.2/x86_64-slc5-gcc43-opt/setup.sh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:\
/cvmfs/sft.cern.ch/lcg/external/ROOT/5.28.00/x86_64-slc5-gcc43-opt/root/lib
export PATH=$PATH:\
/cvmfs/sft.cern.ch/lcg/external/ROOT/5.28.00/x86_64-slc5-gcc43-opt/root/bin

Get and unpack the exercise tarball TTBarExercise.tar.gz.

Help

Compile PYTHIA + analysis code

cd TTBarExercise/siscone-2.0.0 
      
./configure --prefix=$PWD/../siscone
make clean     
make
make install       
cd ..

rm -f *.o
make

g77 -o pythia pythia_exercise.f pythia-6.4.25.f

Tests

Run PYTHIA: ./pythia
Analyze PYTHIA output and save result to root histogram: ./AsciiReader Eventsgen.ascii

Create batch jobs

In a 32-bit VM:

mkdir jobs
for n in {1..25}; do
cat > jobs/job$n.sh <

In a 64-bit VM:

mkdir jobs
for n in {1..25}; do
cat > jobs/job$n.sh <



Submit batch jobs

First check that GridFactory is setup correctly:

psub -h


If all is well, continue with submitting all the jobs:

rm jobs.txt
for n in {1..25}; do
 psub -b gridfactory.nbi.dk jobs/job$n.sh -i pythia -i AsciiReader -e pythia -e AsciiReader -o ttbar_analysis_$n.root | grep -v submitted >> jobs.txt
 echo "submitted job $n"
done


Hint: if you have problems with the above, try running a single job:
psub -b gridfactory.nbi.dk jobs/job1.sh -i pythia -i AsciiReader -e pythia -e AsciiReader -o ttbar_analysis_1.root




Monitor jobs
pstat `cat jobs.txt`





Get and merge output histograms
mkdir results
for n in `cat jobs.txt`; do
 dir=`echo $n | awk -F / '{print $NF}'`
 mkdir results/$dir
 pget -o results/$dir $n
done

hadd ttbar_analysis.root results/*/*.root




Open the final histogram with Root
root ttbar-analysis.root
.
.
.
root [1] TBrowser b;




Clean up
pclean `cat jobs.txt`
rm -rf jobs results

SSL/certificates ok again.

admin — Mon, 16 May 2011 11:05:52 +0000

I’ve updated the CA certificates on the public GridFactory servers and their attached worker nodes and made new versions of the software with new CA certificates available for download.

SSL trouble - NorduGrid CA certificate expired 12/5, 12:00

admin — Sun, 15 May 2011 12:59:20 +0000

As the title says, there’s trouble. Since I’ve not updated neither the servers nor the software, the GridFactory servers will not trust any NorduGrid client or certificates including their own - so they are effectively down - and also the software is useless until I get things upgraded. Sorry for any incovenience.

Public beta!

admin — Mon, 09 May 2011 08:28:39 +0000

Dear grid warriors: more tools are now available to assist you in your battles. The GridFactory software including GridPilot is now publicly available for download. Please take it for a spin and report back - either by commenting on the pages of this site, or by email to gridfactory@gridfactory.org. Good luck!

CERN/ATLAS data processing on grids and GridFactory

admin — Sat, 07 May 2011 16:48:59 +0000

In this post I’ll report on running the application “mc09_7TeV.107691….” from the GridPilot app store. In the case of NorduGrid and WLCG, the ATLAS software is preinstalled on the resources. In the case of GridFactory, the jobs run inside a CernVM appliance with ATLAS software loaded through the AFS network file system. The input dataset consisted of 26 files totaling 36 GB, i.e. each input file was rather large. The files were physically located at atlassrm-fzk.gridka.de.

Timing results are summarized below. There’s no timing for general NorduGrid because I did not manage run on other clusters than our own tier-3.

Summary of runs

	NorduGrid tier-3 cluster (A)	WLCG (B)	GridFactory / virt. - 4 nodes - run 1 (C)	GridFactory / virt. - 4 nodes - run 2 (D)	GridFactory - 4 nodes - run 1 (E)	GridFactory - 4 nodes - run 2 (F)	GridFactory - 1 node (G)
Average submission time per job (s)	1.34	3.69	0.538	0.731	0.731	0.731	0.692
Summed CPU time (s)	3259	7691	4445	4390	2244	1926	1887
Summed download time (s)	-	72041	8755	11478	8998	7170	6295
User real waiting time (submission, processing and data transfer time) (s)	6356	37528	1595	1702	1354	962	2515
Number of available cores	~100	-	16	16	16	16	4

Contrary to the simulation runs and more in line with the boildown runs, this time our tier-3 cluster performed substantially better than the average WLCG resource, but still substantially worse than a new desktop PC. In fact, a user is better off running 26 such jobs on such a desktop PC (with 4 Intel i7 cores) - they finish 2.5 times faster than on the tier-3 cluster with 160 cores (of which a few were busy with other jobs). This is partly because each job runs 1.7 times faster on the desktop PC. The remaining gap must be due to the desktop PC having a faster internet connection and/or overhead incurred by the grid system.

On WLCG, 16 out of 26 jobs failed and had to be resubmitted - with various reasons reported: “user timeout” (.es, .it, .tw), “server responded with and error - transfer aborted” (.tw), atlas misconfiguration (.ru, .za).

On the GridFactory cluster, I ran the same production twice with VirtualBox virtualization and twice without. In both cases, the ATLAS software was read from AFS. In the latter case, a noticeable speedup was observed in the second run - this is presumably due to AFS having cached the ATLAS software. In the former case, the speedup was much smaller - probably the cache in the virtual machine is too small to make a difference. Download times varied quite a lot - presumably for reasons outside of our control (the file server, network). Interestingly, the performance penalty incurred by virtualization appears to be almost a factor 2 - much higher than e.g. in the case of the simulation runs. We ascribe this to the I/O penalty from running in a VirtualBox shared folder on a rather large input file.

CERN/ATLAS Monte Carlo simulation on grids and clouds

admin — Tue, 03 May 2011 21:53:17 +0000

In previous posts we saw that I/O bound jobs ran ~3 faster on standard SATA disks than on network file systems and block devices (GPFS, NFS, EBS). This post reports on CPU bound jobs. I ran standard ATLAS Monte Carlo simulation on both grid and cloud resources: imported the ATLAS simulation app and ran the default 100 small jobs, each generating 100 events and writing a file of about 0.8 MB in size. On the two clouds (C and D - in so far a virtualized batch system qualify as a cloud), the jobs ran inside a CernVM appliance with ATLAS software loaded through the CVMFS network file system. The timing results are summarized in the table and chart below.

Summary of runs

	NorduGrid tier-3 cluster (A)	WLCG (B)	GridFactory / Irigo (C)	GridFactory / virt. (D)	GridFactory (E)
Average submission time per job (s)	0.59	6.43	0.59	0.56	0.59
Average running time (s)	393	130	83.9	85.6	75.9
User real waiting time (submission, processing and data transfer time) (s)	2689	18000	1773	2730	1700
Number of available cores	~100	-	8	8	8

Notes

On the GridFactory cluster, switching on virtualizatilon incurred an 11.3% performance penalty and a penalty of 37.7% in “User real waiting time”. This rather substantial last penalty is primarily due to the short running time of each job, i.e. to running jobs via SSH and staging files in and out of virtual machines.
On WLCG (gLite) the “User real waiting time” was exceedingly long because of the “tail” problem mentioned in previous posts: some jobs took a very long time to start.
On WLCG (gLite) 28 out of 100 jobs failed for a variety of reasons - most prominently because the ATLAS setup script was not found in the standard location “$VO_ATLAS_SW_DIR/software/[release]/setup.sh”. I don’t know if I should look for the setup script somewhere else - if someone out there does, please post a comment or drop me a line.
Despite the many cores, the tier-3 cluster did not do too well. Presumably one reason for this is simply that its processors are rather old Xeons, at least compared to the Core i7’s of the Irigo and GridFactory clusters. Another reason could be that other jobs were running on the cluster - using other ATLAS software releases. If some jobs running different ATLAS releases happened to run on the same (8-core) node, GPFS may have had trouble serving the software. This last hypothesis is supported by the large spread in the CPU times of the jobs: 59 s - 774 s, compared to e.g. 42 s - 132 s on the GridFactory cluster (without virtualization).

CERN/ATLAS boildown on clouds

admin — Fri, 15 Apr 2011 08:32:45 +0000

In this post, I’ll take a look at some more runs of the “atlas_d3pd_boildown” application available in the GridPilot app store. The difference w.r.t. the runs described in a previous post is that this time I ran on cloud as opposed to grid resources. On dedicated hardware and on two public clouds, Amazon’s EC2 and Cabo’s Irigo cloud, I fired up a GridFactory cluster, and changed my preferences to use each one in turn. On the dedicated hardware I made sure the jobs would run in virtual machines (VirtualBox with shared folder) - 2 on each of the 4 worker nodes, each running one job, and on both clouds I ran on 8 worker nodes, each running one job. Notice that in all cases, the jobs run inside a CernVM appliance, but whereas on the two clouds, ATLAS software is accessed through the CVMFS network file system, on GridFactory, the software is accessed over the AFS network file system. Notice also that to avoid the notorious “tail” problem (see previous posts, e.g. this one), this time I ran a “private” cluster on EC2, allowing only my own GridWorkers. On EC2 I chose instances of type “small” (1.7 GB of RAM, I/O performance moderate, 1 virtual core). On Irigo and the dedicated hardware with virtualization I chose a matching setup with instances with the same amount of RAM and 1 virtual core. On the dedicated hardware without virtualization I ran 2 jobs at a time on each 4-core physical machine. The results are summarized below.

Summary of runs

	EC2	Irigo	Dedicated hardware with virtualization	Dedicated hardware without virtualization
Average submission time per job (s)	1.24	1.09	0.531	0.541
Summed running time (s)	55788	13715	9166	7755
Summed CPU time (s)	9501	8644	3220	2517
User real waiting time (submission, processing and data transfer time) (s)	9940	6926	4980	4853

Notes:

This was primarily an I/O exercise: with I/O referring to both network (download of input file) and disk (reading the file) I/O.
Irigo and EC2 have comparable disk I/O and CPU performance with Irigo having a ~10% edge.
The virtual machines on both Irigo and EC2 are booted from a shared file system: Irigo from its image store and EC2 from Amazon’s EBS. Apparently the technology underlying EBS is very similar in performance to that that used by Irigo for its image store - NFS (on top of ZFS).
The “moderate” network I/O of the “small” EC2 instances chosen, apparently is really moderate. Download of input files takes ~9 times longer than on Irigo and ~8 times longer than on the dedicated hardware (at CERN). These fast downloads are likely bounded mostly by the performance of the file server (in Germany) where the input files reside.
The disk images of the virtual machines on both EC2 and Irigo are hosted on a shared file system. While this was expected to limit performance, it is still a bit surprising that the raw disks of the dedicated hardware with (VirtualBox) virtualization give an almost 3 times better performance.
From the two last columns it is seen that the virtualization layer itself apparently incurs ~27% overhead. This is presumably mainly due to the fact that the input data file is read from a VirtualBox shared folder.