Difference between revisions of "MPI4YOU"

From Radio Amteur Station OH5BZR - WIKI
Jump to: navigation, search
(→‎OLDER)
(→‎http://www.msg.ameslab.gov/graphics/other.html)
 
(9 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
=For Ubuntu 14.04=
 +
 +
==Computtaional Chemistry paclages==
 +
http://www1.bbiq.jp/zzzfelis/Facio.html<br>
 +
http://classic.chem.msu.su/gran/gamess<br>
 +
===Databses===
 +
https://www.cfa.harvard.edu/hitran/<br>
 +
http://www.nist.gov/srd/<br>
 +
====http://www.msg.ameslab.gov/graphics/other.html====
 +
gabedit<br>
 +
https://sites.google.com/site/allouchear/Home/gabedit/gabedittutorialfiles/start<br>
 +
https://sites.google.com/site/allouchear/Home/gabedit/installation-of-computational-chemistry-programs<br>
 +
 +
molpro<br>
 +
facio<br>
 +
gamess<br>
 +
firfly<br>
 +
 +
==Elmerfem==
 +
https://github.com/elmercsc/elmerfem <br>
 +
 +
===External Solvers & BASIC COMPONENTS===
 +
 +
LAPACK, BLAS from AMD
 +
 +
http://graal.ens-lyon.fr/MUMPS/index.php MUMPS<br>
 +
 +
http://acts.nersc.gov/hypre/ HYPRE <br>
 +
 +
http://www.pardiso-project.org/ PARDISO <br>
 +
 +
http://trilinos.org/ TRILINOS <br>
 +
 +
http  SuperLU <br>
 +
 +
==basic software needed==
 +
===blcr===
 +
 +
===mpich3===
 +
===PBS - Torque===
 +
http://en.wikipedia.org/wiki/Portable_Batch_System <br>
 +
http://docs.adaptivecomputing.com/suite/8-0/basic/help.htm#topics/torque/1-installConfig/initializeConfigOnServer.htm <br>
 +
 +
==Netgen==
 +
 +
At 14.04 again must compile by own:
 +
Togl1.7 ( no support 2.o at netgen )
 +
Tcl / Tk 8.6
 +
Tix
 +
 +
New occ dose not define PI so add:
 +
 +
Partition_Loop.cxx
 +
#ifndef PI
 +
# include <math.h>
 +
# define PI M_PI
 +
#endif
 +
 +
=OLD=
 +
 
=== Running Installation ===
 
=== Running Installation ===
  
Line 48: Line 108:
 
-DHAVE_HYPRE <br>
 
-DHAVE_HYPRE <br>
 
-DHAVE_PARDISO <br>
 
-DHAVE_PARDISO <br>
 +
 +
==== ALL SPARSE SOLVERS ====
 +
 +
http://www.cise.ufl.edu/research/sparse/codes/<br>
  
 
=== Installation ===
 
=== Installation ===

Latest revision as of 10:42, 15 April 2015

For Ubuntu 14.04

Computtaional Chemistry paclages

http://www1.bbiq.jp/zzzfelis/Facio.html
http://classic.chem.msu.su/gran/gamess

Databses

https://www.cfa.harvard.edu/hitran/
http://www.nist.gov/srd/

http://www.msg.ameslab.gov/graphics/other.html

gabedit
https://sites.google.com/site/allouchear/Home/gabedit/gabedittutorialfiles/start
https://sites.google.com/site/allouchear/Home/gabedit/installation-of-computational-chemistry-programs

molpro
facio
gamess
firfly

Elmerfem

https://github.com/elmercsc/elmerfem

External Solvers & BASIC COMPONENTS

LAPACK, BLAS from AMD 

http://graal.ens-lyon.fr/MUMPS/index.php MUMPS


http://acts.nersc.gov/hypre/ HYPRE 


http://www.pardiso-project.org/ PARDISO 


http://trilinos.org/ TRILINOS 


http  SuperLU

basic software needed

blcr

mpich3

PBS - Torque

http://en.wikipedia.org/wiki/Portable_Batch_System
http://docs.adaptivecomputing.com/suite/8-0/basic/help.htm#topics/torque/1-installConfig/initializeConfigOnServer.htm

Netgen

At 14.04 again must compile by own: Togl1.7 ( no support 2.o at netgen ) Tcl / Tk 8.6 Tix

New occ dose not define PI so add:

Partition_Loop.cxx 

#ifndef PI
# include <math.h>
# define PI M_PI
#endif

OLD

Running Installation

Parraler elmerfem

mpiexec -v -np <how many core> -host <hosts> -ckpointlib blcr -ckpoint-prefix <chekpoint files storage> -ckpoint-interval <chekponting interval seconds> /mpi<mpi4you version>/C/elmer_<elmer revision>/bin/ElmerSolver_mpi  
mpiexec -v -np 3 -host ugh -ckpointlib blcr -ckpoint-prefix /mpi3/chekpoint -ckpoint-interval 10 /mpi3/C2/elmer_6003/bin/ElmerSolver_mpi

Netgen

/mpi3/C2/netgen_623/bin/netgen -geofile=Palkki_tila.brep -meshfile=Mesh_4 -meshsizefile=meshaize4.msz -meshfiletype="Elmer Format" -batchmode
/mpi3/C2/netgen_623/bin/netgen -geofile=<geometry> -meshfile=<result mesh file> -meshsizefile=<size parameters> -meshfiletype="Elmer Format" -batchmode
  • .msz:

<how many points> <X> <Y> <Z> <Mesh Size> <how many lines> <X1> <Y1> <Z1> <X2> <Y2> <Z2> <Mesh Size>

example mesh file: 

1
0 0 0 0.001
7
0 0 0 0 0 0.208 0.001
0 0 0.208 0 0.100 0.208 0.001
0 0.100 0.208 0 0.100 0.200 0.001 
0 0.100 0.200 0 0.008 0.200 0.001
0 0.008 0.200 0 0.008 0.008 0.001
0 0.008 0.008 0 0.100 0.008 0.001
0 0.100 0.008 0 0.100 0 0.001
0 0.100 0 0 0 0 0.001

New librrayes for elmerfem

-DHAVE_CHOLMOD http://faculty.cse.tamu.edu/davis/suitesparse.html
-Dhave_TRILINOS http://trilinos.org/oldsite/TrilinosBuildQuickRef.html
-DHAVE_SUPERLU http://crd-legacy.lbl.gov/~xiaoye/SuperLU/#superlu_dist

OLDER

-DHAVE_MUMPS
-DHAVE_HYPRE
-DHAVE_PARDISO

ALL SPARSE SOLVERS

http://www.cise.ufl.edu/research/sparse/codes/

Installation

THIS IS COPY, NOT DONE YET,... FROM https://sites.google.com/site/jpsdatareviewstheboy007/ubuntu-lts-12-4-companion-whit-ltsp-mpich2-elmer-openfoam


hi,

This page is under construction it has 'recepy' used to create easily manageable environment to run parallel solvers ( netgen,OpenFOAM and Elmer,... )
whit AMD Buldozer hardware at Ubuntu 12.4 LTS

This system is not fully tested, it's a like installation note's storage - it can help you.
I have done lot's experiments and to get all work together is not easy task, one software work's whit one and not all sofware's, compatibelity is real problem.

It can be Ubuntu is not good floor to build scientific calculations, based past experience but that's  what I'm doing now.
It's worth to start from <a href="https://www.scientificlinux.org/" target="_blank">Scientific Linux Distribution</a> - but I can not recommend personally due no experience from that anyway it looks better!!

Files and future development of scripts, makefiles and changes can be done at <a href="https://launchpad.net/mpi4you">https://launchpad.net/mpi4you</a> . You are welcome !

Idea's:
-one installation, easy and fast to distribute to cluster/workstations ( that's <a href="http://www.ltsp.org/" target="_blank">ltsp</a> )
-parallel (mpich2), threads, OpenMP
-multi purpose use, all node's as well server can work as workstation or black box ( not recommended practice by Ubuntu )
-energy, resource efficiency, ecological solutions (hardware configuration, sleeping,..)

Tested:

-one installation and shareing it via netboot, it works whit ltsp but at moment ltsp closes out energy management options

-separately installed, more management work but energy saving options are working and that's money !


Generalideas, guide lines

Hardware in use:

old server:
- CPU AMD x 4
-ram 8G
-ASROCK
-0.5T raid 1
-nfs4 file server RAID1, syslog-nq, bacula
-bought from <a href="http://www.smarit.fi">http://www.smartit.fi</a>

new server:
-<a href="http://www.amd.com/us/products/desktop/processors/amdfx/Pages/amdfx.aspx" rel="nofollow" target="_blank">CPU AMD FX(tm)-8150 Eight-Core Processor × 8</a> it's <a href="http://en.wikipedia.org/wiki/Bulldozer_%28processor%29" target="_blank">Bulldozer CPU</a>
-ram 16G
-main board <a href="http://www.asrock.com/mb/AMD/990FX%20Extreme3/?cat=Specifications" target="_blank">AsROCK990FXextreme3</a>
-1T raid 1
-120G raid 0 , two 2x 300-500kt/s = 600-1000kt/s SATA3 Kingston 60G SSD's
-ltsp-server standalone,nfs4, file server RAID1, ssd RAID0
-bought from <a href="http://www.hardwareversand.de">http://www.hardwareversand.de</a>

Work Stations:

-<a href="http://www.amd.com/us/products/desktop/processors/amdfx/Pages/amdfx.aspx" target="_blank">CPU AMD FX(tm)-8150 Eight-Core Processor × 8</a> it's <a href="http://en.wikipedia.org/wiki/Bulldozer_%28processor%29" target="_blank">Bulldozer CPU</a>
-ram 16G
-main board AsROCK990FXextreme3
-bought from <a href="http://www.hardwareversand.de">http://www.hardwareversand.de</a> , yes price is ok, they deliver quite promptly whitin
     2-4week's from order but if you have problem's whit product's it takes time - avoid them if you can!

System specifications:

Two identical servers, if another out of use other will take responsibility to run services.
- Ubuntu 12.4 LTS, kernel 3.2.0-36-generic
- DNS, bind9
- DHCP, ISC DHCP, <a href="http://consultancy.edvoncken.net/index.php/HOWTO_Configure_DHCP_failover" target="_blank">Fail-over primary and secondary servers </a>
- <a href="https://help.ubuntu.com/community/SettingUpNFSHowTo" target="_blank">NFS4</a>
- <a href="http://www.mcs.anl.gov/research/projects/mpich2" target="_blank">mpich2</a> for parraler calculation's
- <a href="http://www.ltsp.org/" target="_blank">ltsp</a> , ltsp-server-standalone
- syslog-ng, <a href="http://www.quietearth.us/articles/2006/09/27/Logging-remote-host-to-specific-logfile-with-syslogng" target="_blank">install ubuntu</a>
- bacula
- <a href="https://help.ubuntu.com/10.04/serverguide/openssh-server.html" target="_blank">openssh</a>
- apt-casher-ng ( you need to log )
- quata ( to avoid system crash due no room at hard disk )
- <a href="http://www.openfoam.com/" target="_blank">OpenFOAM</a>
- Elmer ( rev. 6033 ) whit <a href="http://www.pardiso-project.org" target="_blank">pardiso</a>
- <a href="http://www.code-aster.org/" target="_blank">Salomè meca</a> ( 6.5 )
- <a href="http://rene.bz/graphing-your-servers-motherboard-temperature-and-fan-rpm-speeds/" target="_blank">cacti</a> - to manage hardware, <a href="http://www.pitt-pladdy.com/blog/_20091031-160232_0000_LM_Sensors_stats_on_Cacti_via_SNMP_/" target="_blank">scripts </a>needed temp/fan/voltage
- netgen ( rev. 623 )

pre-build binaryes:
-rsh-redone

sources & library installation:

mpich2
Togl
Tcl
TK
parmetis
acml (blas,lapack),
MUMPS
PARDISO
Scalpack ( BLACS )
elmerfem ( rev 5912 )
netgen ( rev 596 )

Hierarchy:

<tbody> </tbody>
 Ubuntu   DHCP. DNS, bind9
<tbody> </tbody>
  


<tbody> </tbody>
<tbody> </tbody>
bash
RSH-redone / SSH
 mipch2


<a href="https://help.ubuntu.com/community/SettingUpNFSHowTo" target="_blank">NFS4</a>



elmer
<a href="http://www.openfoam.com/" target="_blank">OpenFOAM</a>
Salomé
Netgen
  parraler solvers		 <a href="http://en.wikipedia.org/wiki/Network_block_device" target="_blank">nbd</a>


libraryes

  <a href="http://en.wikipedia.org/wiki/Preboot_Execution_Environment" target="_blank">PXE</a>


core operateing system boot

 
   <a href="http://www.ltsp.org/" target="_blank"> ltsp integration</a>
   



<a href="http://en.wikipedia.org/wiki/Simple_Network_Management_Protocol" target="_blank">SNMP</a> to see what happens at hardware



<a href="http://www.ltsp.org/" target="_blank">Ltsp</a> is set of script's and package to manage and create
configuration files, operating system distribution
over network (<a href="http://en.wikipedia.org/wiki/Preboot_Execution_Environment" target="_blank">netboot</a>). Orginaly it was used to supply software
to terminals (light computers) but why not use it to create cluster!!!
My Idea is to make one installation to sever and share it
to workstation's and nodes instead of having separate computers
installed separately. This cut downs installation and management duties
remarkably. Compared MAAS, EC2 & JUJU could cause unwanted overhead at parallel
computing at local network as well more management tasks.

Unsolved problem's whit installation ( tested kernel 3.2.0-35-generic ) :

1.a  blcr supports kernel's older that 2.6.38, NO SUPPORT TO KERNEL - NO CHECKPOINTS <a href="https://code.launchpad.net/~ubuntu-branches/ubuntu/precise/blcr/precise-updates">https://code.launchpad.net/~ubuntu-branches/ubuntu/precise/blcr/precise-updates</a> , solution under work <a href="http://packages.debian.org/experimental/blcr-dkms">http://packages.debian.org/experimental/blcr-dkms currently it dose not work!!!</a>
1.b DMTCP dose not work eighter, it dose memory violation. 
Change: blcr 0.5.0 compiles well whit 3.2.0-49-generic but not fully tested yet !

2. wakeonlan dose not work whit LTPS,  <a href="https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1051325" target="_blank"> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1051325</a>

Fixe's coming soon to precise: <a href="http://crd.lbl.gov/groups-depts/ftg/projects/current-projects/BLCR/berkeley-lab-checkpoint-restart-for-linux-blcr-downloads/" target="_blank">new blcr version</a>, <a href="http://wiki.xbmc.org/index.php?title=How-to:Enable_Wake-On-Device_for_Ubuntu" target="_blank">ACPI corrections to kernel</a>.

Installationnotes



Server's have IP 192.168.0.40 and 192.168.0.41.
ltsp client's are useing IP's starting 192.168.0.42

Basic Server Installation tricks:



Compatible Usage Rights


At moment best solution have been:
nsh drives:  group = nogroup
user must belong nogroup
/etc/passwd must be same ltsp and Ubuntu !!!
Both server and ltps nodes have same /home/user/.ssh/ssh-keys


To Install


-install basic configuration from CD-ROM only. This way you avoid re-installation due broken packages that can be tricky to find
-install server version so you can configure RAID disks, ext,... then

apt-get install Ubuntu-Desktop

-last step is graphics card drivers.If you have nvidia based graphics card first:

apt-get kernel-headers kernel-sources
and after that

apt-get nvidia-current nvidia-common
 

Install system related applications:

apt-get install apt-cacher-ng

apt-get install nfs-common nfs-kernel





apt-get install quota

apt-get install ltsp-server-standalone

apt-get install syslog-ng apt-cacher-ng bacula ntp


apt-get install ubuntu-desktop

apt-get install linux-headers kernel-source

apt-get install nvidia-current nvidia-common

nvidia-xconfigure


  1. ADD/USE FIREWALL /NAT TO ROUTER THE DISABLE ALL LINUX FIREWALL'S

apt-get remove ufw
apt-install firestarter


  1. for fun

apt-get install samba
apt-get install wine jstest-gtk wine playonlinux ubuntu-restricted-extras mythplugins



apt-get install openjdk-7-jre icedtea-7-plugin



Network

/etc/network/interfaces
======================================================

  1. This file describes the network interfaces available on your system
  2. and how to activate them. For more information, see interfaces(5).


  1. The loopback network interface

auto lo
iface lo inet loopback

  1. The primary network interface

auto eth0
iface eth0 inet static
    address 192.168.0.41
    netmask 255.255.255.0
    gateway 192.168.0.254
=======================================================

ipv6:
<a href="http://www.shorewall.net/IPv6Support.html#idp67096" target="_blank">IPV6, Shorewall, Webmin</a>

<a href="http://codeghar.wordpress.com/2012/08/03/ipv6-in-ubuntu-12-04/" target="_blank">Ubuntu IPv6 </a>- by HS

<a href="https://wiki.ubuntu.com/IPv6#Configure_your_Ubuntu_box_as_a_IPv6_router" target="_blank">Ubuntu IPV6</a> - Ubuntu Wiki
<a href="http://www.sixxs.net" target="_blank">Sixx - tunnel from IPV4</a>

<a href="http://www.csc.fi/funet/status/tools/funet_ipv6_status" target="_blank">Real Use IPV6</a>

ssh

Links:
<a href="http://www.open-mpi.org/faq/?category=rsh" target="_blank">Ssh OpenMPI</a>
<a href="https://help.ubuntu.com/10.04/serverguide/openssh-server.html" target="_blank">openssh</a>

  1. install

apt-get install openssh-client openssh-server

gedit /etc/ssh/sshd_config
======================================================================

  1. Package generated configuration file
  2. See the sshd_config(5) manpage for details


  1. What ports, IPs and protocols we listen for

Port 22

  1. Use these options to restrict which interfaces/protocols sshd will bind to
  2. ListenAddress ::
  3. ListenAddress 0.0.0.0

Protocol 2

  1. HostKeys for protocol version 2

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key

  1. Privilege Separation is turned on for security

UsePrivilegeSeparation yes

  1. Lifetime and size of ephemeral version 1 server key

KeyRegenerationInterval 3600
ServerKeyBits 768

  1. Logging

SyslogFacility AUTH
LogLevel INFO

  1. Authentication:

LoginGraceTime 120
PermitRootLogin yes
StrictModes yes

RSAAuthentication yes
PubkeyAuthentication yes

  1. AuthorizedKeysFile    %h/.ssh/authorized_keys


  1. Don't read the user's ~/.rhosts and ~/.shosts files

IgnoreRhosts yes

  1. For this to work you will also need host keys in /etc/ssh_known_hosts

RhostsRSAAuthentication no

  1. similar for protocol version 2

HostbasedAuthentication no

  1. Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
  2. IgnoreUserKnownHosts yes


  1. To enable empty passwords, change to yes (NOT RECOMMENDED)

PermitEmptyPasswords no

  1. Change to yes to enable challenge-response passwords (beware issues with
  2. some PAM modules and threads)

ChallengeResponseAuthentication no

  1. Change to no to disable tunnelled clear text passwords
  2. PasswordAuthentication yes


  1. Kerberos options
  2. KerberosAuthentication no
  3. KerberosGetAFSToken no
  4. KerberosOrLocalPasswd yes
  5. KerberosTicketCleanup yes


  1. GSSAPI options
  2. GSSAPIAuthentication no
  3. GSSAPICleanupCredentials yes


X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes

  1. UseLogin no


  1. MaxStartups 10:30:60
  2. Banner /etc/issue.net


  1. Allow client to pass locale environment variables

AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

  1. Set this to 'yes' to enable PAM authentication, account processing,
  2. and session processing. If this is enabled, PAM authentication will
  3. be allowed through the ChallengeResponseAuthentication and
  4. PasswordAuthentication.  Depending on your PAM configuration,
  5. PAM authentication via ChallengeResponseAuthentication may bypass
  6. the setting of "PermitRootLogin without-password".
  7. If you just want the PAM account and session checks to run without
  8. PAM authentication, then enable this but set PasswordAuthentication
  9. and ChallengeResponseAuthentication to 'no'.

UsePAM yes
======================================================================

gedit /etc/ssh/ssh_config
======================================================================

  1. This is the ssh client system-wide configuration file.  See
  2. ssh_config(5) for more information.  This file provides defaults for
  3. users, and the values can be changed in per-user configuration files
  4. or on the command line.


  1. Configuration data is parsed as follows:
  2.   1. command line options
  3.   2. user-specific file
  4.   3. system-wide file
  5. Any configuration value is only changed the first time it is set.
  6. Thus, host-specific definitions should be at the beginning of the
  7. configuration file, and defaults at the end.


  1. Site-wide defaults for some commonly used options.  For a comprehensive
  2. list of available options, their meanings and defaults, please see the
  3. ssh_config(5) man page.


Host *

  1.    ForwardAgent no
  2.    ForwardX11 no
  3.    ForwardX11Trusted yes
  4.    RhostsRSAAuthentication no
  5.    RSAAuthentication yes
  6.    PasswordAuthentication yes
  7.    HostbasedAuthentication no
  8.    GSSAPIAuthentication no
  9.    GSSAPIDelegateCredentials no
  10.    GSSAPIKeyExchange no
  11.    GSSAPITrustDNS no
  12.    BatchMode no
  13.    CheckHostIP yes
  14.    AddressFamily any
  15.    ConnectTimeout 0
  16.    StrictHostKeyChecking ask
  17.    IdentityFile ~/.ssh/identity
  18.    IdentityFile ~/.ssh/id_rsa
  19.    IdentityFile ~/.ssh/id_dsa
  20.    Port 22
  21.    Protocol 2,1
  22.    Cipher 3des
  23.    Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
  24.    MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
  25.    EscapeChar ~
  26.    Tunnel no
  27.    TunnelDevice any:any
  28.    PermitLocalCommand no
  29.    VisualHostKey no
  30.    ProxyCommand ssh -q -W %h:%p gateway.example.com

    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials no

===========================================================================

Then generate public/private rsa for user mpi and copy public one to authoritzed one's.

sudo adduser mpi
sudo login mpi

<span><b>cd .ssh<br>ssh-keygen -t rsa</b></span>

cat id_rsa.pub >> authorized_keys

Later at LTSP install mpi user's ~/.ssh directory will copyed to ltsp-client chroot /home/mpi/.ssh
That's due when client is has finished network boot it has image based chroot and after user at
client login's ltsp-client will link user's home directory. While useing mpi we are takeing connection
remotely trough ssh and thus sshserver at client side needs ssh sertificate's and those should be
in ltsp chroot image!!!

To get rid off keypasphere  enter

ssh-add $HOME/.ssh/id_rsa and give passphere once
before launching applications.

rsh-redone

Ubuntu default rsh is fake rsh, just link's to ssh package.
sudo apt-get remove rsh
sudo apt-get install rsh-redone-server rsh-redone-client

apt-cacher-ng

sudo apt-get install apt-cacher-ng

- fill cache from CD-ROM
- After install http://localhost:3142 you find instructions from there
- create and add following file to server /etc/apt/apt.conf.d/ as well to ltsp image root /opt/ltsp/amd64/etc/apt

sudo gedit /etc/apt/apt.conf.d/02proxy
=================
Acquire::http { Proxy "http://192.168.0.40:3142"; };
=================

BIND

<a href="https://help.ubuntu.com/community/BIND9ServerHowto" target="_blank">Bind9 Installation</a>
<a href="http://www.sixxs.net/faq/dns/?faq=reverse" target="_blank">IPv6 Sixx'x Reverse Domain</a>


Source Installation's

<a href="https://launchpad.net/mpi4you" rel="nofollow">https://launchpad.net/mpi4you</a> is under construction and you find there:

-build cripts, dir /BX

-general flag's, dir /Bx/flag

-download code script, dir /Bx/download

-changed source file files's, dir /Sx


Basic "compile from sources" compatibility solutions


To get sources to work each other we must standardize communication between libraries and I have done it:
<a href="http://www.unix.org/version2/whatsnew/lp64_wp.html" target="_blank">Memory model LP64</a>  int 32bit real 64bit pointers/float 64bit and to get it work we use -m64 whit fortran and metis -DIDXSIZE32, fortran -fdefault-real-8 -fdefault-double-8 dose the work.

Calling convetion, actual name of function at Linker Level should be same and flag -DAdd_  adds one underscore to fort name's and dose the job.

Unfortunately the way gcc treats linking has changed, to get it remember all libraries to bare end of linking sometime's you need to add -Wl,--no-as-needed
as well  <a href="http://gcc.gnu.org/onlinedocs/gfortran/Code-Gen-Options.html" target="_blank">-fno-align-commons</a> gives some help.

Ubuntu 12.4 come whit GCC 4.6.3 and GCC 4.6 dose not support Buldozer architecture optimisation it's possible whit GCC 4.7.

mpic2

Got elmer working first time but only whit mpich2
compiled from <a href="http://www.mcs.anl.gov/research/projects/mpich2" target="_blank">sources</a> and ALL mpi realted distribution file's removed!

apt-get install mpich2 mpich2-doc libmpich2-dev libmpich2-3


CACTI

To find botlenecks you need some statistics from system. CACTI can collect needed information.

sudo apt-get snmpd

sudo apt-get install snmp-mibs-downloader

sudo apt-get cacti

<a href="http://rene.bz/graphing-your-servers-motherboard-temperature-and-fan-rpm-speeds/" rel="nofollow" target="_blank">cacti</a> - to manage hardware, <a href="http://www.pitt-pladdy.com/blog/_20091031-160232_0000_LM_Sensors_stats_on_Cacti_via_SNMP_/" rel="nofollow" target="_blank">scripts </a>needed temp/fan/voltage


LTSP install Links:

<a href="https://help.ubuntu.com/community/UbuntuLTSP/FatClients" target="_blank">ltsp-fat client</a>

sudo gedit /etc/ltsp/ltsp-build-client.conf

==================================================

# The chroot architecture.

ARCH=amd64

LOCALE="fi_FI.UTF-8"

# ubuntu-desktop and edubuntu-desktop are tested.

# If you test with [k|x]ubuntu-desktop, edit this page and mention if it worked OK.

# kubuntu lucid (10.10) working okay.

FAT_CLIENT_DESKTOPS="ubuntu-desktop"


# Space separated list of programs to install.

# The java plugin installation contained in ubuntu-restricted-extras

# needs some special care, so let's use it as an example.

LATE_PACKAGES="

kde-l10n-fi


language-selector-gnome

language-selector-common


language-pack-gnome-fi-base

language-pack-gnome-en-base

language-pack-gnome-de-base

language-pack-gnome-fr-base


language-pack-fi-base

language-pack-de-base

language-pack-fr-base

language-pack-fen-base


language-pack-fi

language-pack-fr

language-pack-de

language-pack-en


language-pack-gnome-de

language-pack-gnome-fr

language-pack-gnome-en

language-pack-gnome-fi


language-pack-en-base

language-pack-fi-base

language-pack-de-base

language-pack-fr-base



firefox-locale-fi

thunderbird-locale-fi


voikko-fi


libreoffice-l10n-fr

libreoffice-l10n-de

libreoffice-l10n-sv

libreoffice-l10n-fi

libreoffice-help-fi

hyphen-de

hyphen-fr

mythes-de

mythes-de-ch

mythes-fr

mythes-en-us

hyphen-en-us



"


# This is needed to answer "yes" to the Java EULA.

# We'll create that file in the next step.

DEBCONF_SEEDS="/etc/ltsp/debconf.seeds"


# This uses the server apt cache to speed up downloading.

# This locks the servers dpkg, so you can't use apt on

# the server while building the chroot.

MOUNT_PACKAGE_DIR="/var/cache/apt/archives/"

===============================================================



# created to /opt/ltsp

sudo ltsp-build-client

sudo ltsp-chroot -c -p


#timezone

dpkg-reconfigure tzdata

#local languages

dpkg-reconfigure localeconf

  1. system softwareapt-get

#apt-get install openmpi-common openmpi-bin

#apt-get install libscotch-dev libopenmpi-dev

apt-get install nfs-client nfs-common samba cifs-utils

apt-get install playonlinux

#powerwake powernap

apt-get install mythtv-frontend

apt-get install gimp darktable

apt-get install mytharchive mythgallery mythgame mythmusic mythnews mythweather


#IMAGE UPDATE
sudo ltsp-chroot -c -p
adduser root
adduser mpi

passwd root

passwd mpi


apt-get update
exit

sudo mkdir /opt/ltsp/amd64/mpi

sudo mkdir /opt/ltsp/amd64/home/mpi

sudo cp -R /home/mpi/.ssh /opt/ltsp/amd64/home/mpi #enviroment

sudo cp /home/mpi/.bashrc /opt/ltsp/amd64/home/mpi
sudo cp /home/mpi/.bash_login /opt/ltsp/amd64/home/mpi

#sudo cp -R /root/.ssh /opt/ltsp/amd64/root

sudo ltsp-chroot -c -p


cd /home

chown -R mpi:mpi mpi

cd /home/mpi

chmod 700 .ssh

chmod 600 .ssh/id*

chmod 644 .ssh/known_hosts

chmod 644 .ssh/authorized_keys


cd /root

chmod 700 .ssh

chmod 600 .ssh/id*

chmod 644 .ssh/known_hosts

chmod 644 .ssh/authorized_keys


mkdir /mpi

mkdir /mpi2

mkdir /mp3


apt-get install quota

apt-get install wine jstest-gtk wine playonlinux ubuntu-restricted-extras mythplugins

apt-get install ltsp-server-standalone

apt-get syslog-ng bacula-fd ntp


# COMMAND'S TO UPDATE IMAGE

sudo ltsp-update-sshkeys

sudo ltsp-update-image -n

sudo ltsp-update-kernels

Tip's:

-diagnostics, apt-get wireshark


# NFS filesystem

#SERVER

/etc/fstab :

/mpi2    /export/mpi2   none    bind  0  0
/mpi3    /export/mpi3   none    bind  0  0

/etc/exports :

/export/mpi2 192.168.0.0/24(rw,nohide,insecure,no_subtree_check,sync)
/export/mpi3 192.168.0.0/24(rw,nohide,insecure,no_subtree_check,sync)

service idmapd restart

#CLIENT info at server

/var/lib/tftpbootp/ltsp/amd64/lts.conf :

 FSTAB_1="192.168.0.41:/mpi3 /mpi3 nfs4 _netdev,auto 0 0"
        FSTAB_2="192.168.0.40:/ /mpi nfs4 _netdev,auto 0 0"
        FSTAB_3="//192.168.0.51/OpenShare /media/ST1 cifs guest,rw,iocharset=utf8,file_mode=0777,dir_mode=0777 0 0"
        FSTAB_4="192.168.0.41:/mpi2 /mpi2 nfs4 _netdev,auto 0 0"


=========================================================================================





# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/md0 during installation
UUID=24d52ebb-9a49-4949-831a-0702ca032406 /               ext4    errors=remount-ro 0       1
# /boot/efi was on /dev/sda1 during installation
UUID=9AD9-4C15  /boot/efi       vfat    defaults        0       1
# /home was on /dev/md1 during installation
UUID=a9091c8f-31e2-43af-8167-88d8191a4f2c /home           ext4    usrquota,grpquota 0       2
# /var was on /dev/md2 during installation
UUID=3dae13d4-6310-417a-bc67-e987638dbe54 /var            ext4    defaults        0       2
# swap was on /dev/sda5 during installation
UUID=c5de965b-1adc-40e4-ab8b-cddb8bd153c6 none            swap    sw              0       0
# swap was on /dev/sdb5 during installation
UUID=bc109a85-d598-41e4-8805-a8bce1c68aa2 none            swap    sw              0       0
/dev/md127 /mpi3 ext4 rw,nosuid,nodev,uhelper=udisks 0 0
192.168.0.40:/ /mpi nfs4 _netdev,auto 0 0
//192.168.0.51/OpenShare /media/ST1 cifs guest,rw,iocharset=utf8,file_mode=0777,dir_mode=0777 0 0
#/mpi2    /export/mpi2   none    bind  0  0
/dev/sr0 /media/cdrom iso9660,udf user,noauto,exec,utf8,uid=1000,mode=0777 0 0
#//path to your windows share/    /your_mount_path cifs username=your_username,password=your_password,workgroup=your_workgroup,users,auto,user_xattr 0 0

<a>file:///etc/fstab</a>

===================================================================================================

DHCP, ISC DHCP

Links: <a href="http://consultancy.edvoncken.net/index.php/HOWTO_Configure_DHCP_failover" target="_blank">Failover primary and secondary servers </a>

<a href="http://mirrors.bieringer.de/Linux+IPv6-HOWTO/hints-daemons-isc-dhcp.html" target="_blank">FOR   IPv6</a> you need to run second ISC DHCP deamon whit own configuration for IPv6 <a href="http://www.buntschu.ch/blog/?p=344" target="_blank">more</a>.

Other solution is radvd deamon but I did not get it work whit ISC DHCP serving IPv4 clients.


# check out that isc-dhcp is only dhcp server installed at server


gedit /etc/ltsp/dhcpd.conf

=======================================

#dhcp


  2. Default LTSP dhcpd.conf config file.

INTERFACES="eth0";

authoritative;

failover peer "dhcp-failover" {
  primary; # declare this to be the primary server
  address 192.168.0.41;
  port 647;
  peer address 192.168.0.40;
  peer port 647;
  max-response-delay 30;
  max-unacked-updates 10;
  split 128;
  mclt 3600;
  load balance max seconds 3;
}

subnet 192.168.0.0 netmask 255.255.255.0 {

pool {
    failover peer "dhcp-failover";
    deny dynamic bootp clients;
    max-lease-time 1800; # 30 minutes
    range 192.168.0.100 192.168.0.200;
   }

pool {
      max-lease-time 1800; # 30 minutes
    range 192.168.0.201 192.168.0.210;
   }
    option domain-name "kurrola.dy.fi";
  1.     option domain-name-servers 193.229.0.40, 193.229.0.42;
  option domain-name-servers 192.168.0.40,192.168.0.41;
    option broadcast-address 192.168.0.255;
    option routers 192.168.0.254;

  1.     get-lease-hostnames true;
    option subnet-mask 255.255.255.0;
    option root-path "/opt/ltsp/amd64";

    if substring( option vendor-class-identifier, 0, 9 ) = "PXEClient" {
        filename "/ltsp/amd64/pxelinux.0";
    } else {
        filename "/ltsp/amd64/nbi.img";
    }

     next-server 192.168.0.41;
   allow booting;
   allow bootp;
}
  1. subnet ends

  1. staic lease

  host ugh {
                hardware ethernet  xxxxxxxxxxxxxxxx;
                fixed-address 192.168.0.42;
                }
  host kaak {
                hardware ethernet  xxxxxxxxxxxxxxxx;
                fixed-address 192.168.0.43;
                }

  host     kurrola.dy.fi {
                hardware ethernet xxxxxxxxxxxxxxxx;
                fixed-address 192.168.0.40;
                }
  host JuuliaHP {
                hardware ethernet  xxxxxxxxxxxxxxxx;
                fixed-address 192.168.0.43;
                }
  host     JoniXperiaActiv {
                hardware ethernet  xxxxxxxxxxxxxxxx;
                fixed-address 192.168.0.90;
                }
  host     ipadHanna {
                hardware ethernet xxxxxxxxxxxxxxxx;
                fixed-address 192.168.0.91;
                }
  host     KINDLEJoni {
                hardware ethernet xxxxxxxxxxxxxxxx;
                fixed-address 192.168.0.92;
                }
  host KurrolaStorage1 {
                hardware ethernet  xxxxxxxxxxxxxxxx;
                fixed-address 192.168.0.51;
                }
  host paula {
                hardware ethernet  xxxxxxxxxxxxxxxx;
                fixed-address 192.168.0.81;
                }
  host EPSONSX420W {
                hardware ethernet  xxxxxxxxxxxxxxxx;
                fixed-address 192.168.0.50;
                }
  host mpi1 {
                hardware ethernet  xxxxxxxxxxxxxxxx;
                fixed-address 192.168.0.41;
                }


=======================================

HPC

More About <a href="http://en.wikipedia.org/wiki/Supercomputer" target="_blank">HPC</a>. Most work done to compiled at make script distribution <a href="https://launchpad.net/mpi4you">https://launchpad.net/mpi4you</a> .

Needed Calculation libraries shortlist:-

<a href="http://www.cmake.org/cmake/help/install.html" target="_blank">CMAKE</a>
<a href="http://www.vtk.org/Wiki/VTK/Building/Linux" target="_blank">VTK</a>
<a href="http://developer.amd.com/membership/licenseagreement.aspx?id=4c8a0be8-f366-418e-b8b6-85ffdebca990&ReferUrl=" target="_blank">AMCL AMD ( BLAS, LAPACK )</a>

<a href="http://graal.ens-lyon.fr/MUMPS/index.php?page=dwnld" target="_blank">MUMPS</a>

<a href="http://acts.nersc.gov/hypre/" target="_blank">HYPRE</a>

<a href="http://www.pardiso-project.org/branch" target="_blank">PARDISO</a>


Before production use, one way to to check basic installation is to take part <a href="http://icl.cs.utk.edu/hpcc/" target="_blank">HPCC Speed Chalange</a>

and benchmark your system against others, here is my very first trial out at  <a href="http://icl.cs.utk.edu/hpcc/hpcc_results.cgi?vid=051M03bTda&ea=the@boy007.dy.fi" target="_blank">cluster.kurroila.dy.fi - mpi4you</a> !

That way you recognize bottle neck's in your system and have realistic understanding what it can do for you.


Compile methods:

a) CMAKE is used whit VTK, HDF5.  , paraview, parmetis, metis

b) make  / Makefile.inc ARPACK, MUMPS 4.1, scalapck ( scalapac_installer ), scotch_esmumps

c) ./configure  hypre, mpich, netgen, elmerfem


Paraview

Links: <a href="http://www.paraview.org/paraview/resources/software.php" target="_blank">download and compile,</a> <a href="http://www.vtk.org/Wiki/ParaView:Build_And_Install" target="_blank">Instructions VTK-Wiki</a>

<a href="http://www.vtk.org/Wiki/ParaView:Build_And_Install" target="_blank"></a>

To get space ball working install vrpn config it for spaceball (uncomment vrpn_3DConnexion_Navigator device0 and comment activated).

#hidapi
sudo apt-get install libudev-dev libusb-dev libfox-dev
sudo apt-get install autotools-dev

git clone git://github.com/signal11/hidapi.git

cd /mpi3/S2

./bootstrap
./configure --prefix=/mpi3/C2/hidapi
make
sudo make install

cd ..

  1. download vrpn
  2. mkdir vrpn-BUILD
  3. cmake -i ../vrpn
cd vrpn-BUILD
make
sudo make install

Download Paraview

mkdir /mpi3/S2/ParaView6
cd /mpi3/S2/ParaView6
git clone git://paraview.org/ParaView.git /mpi3/S2/ParaView6
git checkout master
git pull origin master
git submodule update --init

Run SpaceBall <a href="http://review.source.kitware.com/#/c/10740/" target="_blank">patch</a>  ( Thanks for Cory from this patch ) before compilationt of paraview

git fetch http://review.source.kitware.com/p/ParaView refs/changes/40/10740/3 && git checkout FETCH_HEAD -b add_space_navigator_grab_world_style

cd /mpi3/S2
mkdir ParaView6-BUILD
cd ParaView6-BUILD
cmake -i ../ParaView6

  1. gedit cmake

make
make install




Salomè

<a href="http://www.code-aster.org/V2/spip.php?article303" target="_blank">Links: Salomé Meca code Aster </a>work's but crashes time to time and
GUI dose not work  correctly.

<a href="http://www.salome-platform.org/" target="_blank">Salomé Source: Debian 6.0 Squeeze 64bit</a> can be compiled as
binary. Can not build from SOURCES at moment.
Just download InstallWizard, extract it and run install.

Read: VTK & SpaceBall


download <a href="http://www.salome-platform.org/" target="_blank">Salomé Source: Debian 6.0 Squeeze 64bit</a>
cd /mpi3/S2/InstallWizard_6.6.0_Debian_6.0_64bit
./runInstall

<a href="http://www.salome-platform.org/" target="_blank">Salomé Source: Debian 6.5 Squeeze 64bit</a> can be compiled as
binary. Can not build from SOURCES at moment.

Prerequisites for binary build:
sudo apt-get install freeglut3
sudo ln -s /usr/lib/libicuuc.so.48 /usr/lib/libicuuc.so.44
sudo ln -s /usr/lib/libicui18n.so.48 /usr/lib/icui18n.so.44
sudo ln -s /usr/lib/libicudata.so.48 /usr/lib/libicudata.so.44
....


Elmer parraler for Ubuntu 12.4

Links:
<a href="http://www.elmerfem.org/elmerwiki/index.php/Compiling_Elmer_parallel_version_%28Linux%29">Compling Elmer parallel</a>
<a href="https://launchpad.net/%7Etormodvolden/+archive/ppa" target="_blank">Update ELMER binary for ubuntu 12.4 ppa by Tormod Volden</a>

At system  Elmer is installed RAID0  SDD disk,
elmer will be under /mpi3 directory.

You should install elmer as user:mpi to get it fully
work at cluster.

Elmer binaryes will after installation at /mpi3/elmer/bin and source /mpi3/elmerfem.

Trunk source will be used at directory /mpi3/elmerfem/trunk

To build binaryes:

a) first download all code, libraryes commented out #sudo #svn
at first section of buildelmer.sh once.

b) do changes to ElmerGUI.pri

c) After that run ./buildelmer.sh.  I use  buildelmer.sh
as storage of infromation to download and all libraryes,...

To start gui use elmerG.sh

If you need to compile again do ./clean.sh first.


  1. Load Elmer sources:

mkdir /mpi3
cd /mpi3
mkdir elmer
svn co https://elmerfem.svn.sourceforge.net/svnroot/elmerfem elmerfem

  1. pardiso library and licence file to directory elmer/lib
  2. pardiso available by separate licence from   <a href="http://www.pardiso-project.org/download/bb6ex6hze7cvjhuz15" target="_blank">http://www.pardiso-project.org</a>


  1. ==============================
  2. Do changes /mpi3/elmerfem/trunk/ElmerGUI/ElmerGUI.pri
  3. cd /usr/include
  4. sudo ln -s vtk-5.8/* ./*


  1. =DO CHANGES BEFORE COMPILE =======

gedit /mpi3/elmerfem/trunk/ElmerGUI/ElmerGUI.pri

  1. ===================================
  2. ==============================================================================

  4.                        ElmerGUI: configuration file

  6. ==============================================================================


  1. ------------------------------------------------------------------------------
  2. Optional components (undefine or comment out to exclude from compilation):
  3. ------------------------------------------------------------------------------

DEFINES += EG_QWT      # Use QWT for convergence monitor?
DEFINES += EG_VTK      # Use VTK for postprocessing?
DEFINES += EG_MATC     # Use MATC for internal operations in postprocessing?
DEFINES += EG_OCC      # Use OpenCASCADE 6.3 for importing CAD files? Needs VTK.
DEFINES -= EG_PYTHONQT # Use PythonQt for scripting in post processor?

  1. ------------------------------------------------------------------------------
  2. 64 bit system?
  3. ------------------------------------------------------------------------------

BITS = 64

  1. ------------------------------------------------------------------------------
  2. Python library:
  3. ------------------------------------------------------------------------------

unix {
   PY_INCLUDEPATH = /usr/include/python3.2
   PY_LIBPATH = /usr/lib
   PY_LIBS = -lpython3.2
}

  1. ------------------------------------------------------------------------------
  2. VTK library:libQVTK.so.5.8.0
  3. ------------------------------------------------------------------------------

unix {
   VTK_INCLUDEPATH = /usr/include/vtk-5.8
   VTK_LIBPATH = /usr/lib
   VTK_LIBS = -lQVTK \
              -lvtkCommon \
              -lvtkDICOMParser \
              -lvtkFiltering \
              -lvtkGenericFiltering \
              -lvtkGraphics \
              -lvtkHybrid \
              -lvtkIO \
              -lvtkImaging \
              -lvtkInfovis \
              -lnetcdf \
              -lvtkRendering \
              -lvtkViews \
              -lvtkVolumeRendering \
              -lvtkWidgets
}


=== TO WIPE OUT COMPILE FILE'S ======

run at install directory

cd /mpi3

===================================

===================================

build script - at first time do uncommented

parts by copy paste at terminal


===================================
/mpi3/buildelmer.sh
==============================

  1. !/bin/sh -f

export ElmerMPI_DIR="$( cd "$( dirname "$0" )" && pwd )"

  1. svn co https://elmerfem.svn.sourceforge.net/svnroot/elmerfem elmerfem
  2. sudo apt-get install libvtk5-qt4-dev

cd elmerfem/trunk

  1. Choose the installation directory and set up the environment:

export ELMER_HOME=$ElmerMPI_DIR/elmer
export ELMER_POST_HOME=$ELMER_HOME/share/elmerpost
export ELMER_FRONT_HOME=$ELMER_HOME/share/elmerfront
export ELMERGUI_HOME=$ELMER_HOME/bin
export PATH=$PATH:$ELMER_HOME/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ELMER_HOME/lib

  1. the compilers

export CC=mpicc
export CXX=mpic++
export FC=mpif90
export F77=mpif90

  1. the compiler flags

export CFLAGS="-I/usr/include/tcl8.5"
export CXXFLAGS="-I/usr/include/tcl8.5"
export FCFLAGS=""
export F77FLAGS=""
export FFLAGS=""

  1. linking
  2. export LDFLAGS=""


  1. if PARDISO available
  2. export FCPPFLAGS="-DHAVE_PARDISO"
  3. export LDFLAGS="-L/mpi4/elmer/lib/ -lpardiso412-GNU443-MPI-X86-64 -lgfortran -fopenmp -lpthread -lm"


  1. -Ofast -ipa -lacml -lfortran -lffio -lacml mv -lmv

export OMP_NUM_THREADS=3
export PARDISO_LIC_PATH=$ELMER_HOME/lib
export MKL_SERIAL=YES

  1. paths


  1. Modules definition and configure options setting:

modules="matc umfpack mathlibs elmergrid meshgen2d eio hutiter fem"
for m in $modules; do
  cd $m
  ./configure --prefix=$ELMER_HOME --with-64bits=yes --with-mpi=yes
  make
  make install
  cd ..
done

cd post
export CXXFLAGS="-I/usr/include/FTGL -I/usr/include/tcl8.5"
./configure --prefix=$ELMER_HOME
make
make install
cd ..

cd front
./configure --prefix=$ELMER_HOME
make
make install
cd ..

cd ElmerGUI
echo "now in ElmerGUI"
qmake ElmerGUI.pro
make
cp ./Application/ElmerGUI $ELMERGUI_HOME/ElmerGUI
cp -r ./Application/edf $ELMERGUI_HOME

cd ..




======== START UP ElmerGUI  ==========
/mpi3/elmerG.sh
====================================

  1. !/bin/sh -f

export ElmerMPI_DIR="$( cd "$( dirname "$0" )" && pwd )"

export ELMER_HOME=$ElmerMPI_DIR/elmer
export PATH=$PATH:$ELMER_HOME/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ELMER_HOME/lib
export ELMERGUI_HOME=$ELMER_HOME/bin
export ELMER_POST_HOME=$ELMER_HOME/bin
export OMP_NUM_THREADS=3
export PARDISO_LIC_PATH=$ELMER_HOME/lib
export MKL_SERIAL=YES

$ELMERGUI_HOME/ElmerGUI



====================================

===================================
/mpi3/clean.sh
===================================

  1. !/bin/sh -f


cd elmerfem/trunk

  1. Modules definition and configure options setting:

modules="matc umfpack mathlibs elmergrid meshgen2d eio hutiter fem"
for m in $modules; do
  cd $m
    make clean
  cd ..
done

cd front
make clean
cd ..

cd post
make clean
cd ..

cd ElmerGUI
make clean

cd Application
make clean
cd..
cd ..


==============

  1. All ok?
  2. go:


cd elmer/bin
ElmerSolver

MAIN: =============================================================

MAIN: ElmerSolver finite element software, Welcome!

MAIN: This program is free software licensed under (L)GPL

MAIN: Copyright 1st April 1995 - , CSC - IT Center for Science Ltd.

MAIN: Webpage http://www.csc.fi/elmer, Email elmeradm@csc.fi

MAIN: Library version: 7.0 (Rev: 5825)

MAIN: Running in parallel using 18 tasks.

MAIN: HYPRE library linked in.

MAIN: MUMPS library linked in.

MAIN: PARDISO library linked in.

MAIN: =============================================================


==============

  1. TEST

cd /mpi3/elmerfem/trunk/fem/tests
.runtest

joni@mpi1:/mpi3/elmerfem/trunk/fem/tests$ ./runtests
$ELMER_HOME undefined, setting it to ../src
test 1 :                   1dtests         [PASSED], CPU time=0.06
test 2 :                   1sttime         [PASSED], CPU time=0.43
test 3 :                   2ndtime         [PASSED], CPU time=0.74
test 4 :               adaptivity1         [PASSED], CPU time=0.98
test 5 :               adaptivity2         [PASSED], CPU time=1.19
test 6 :               adaptivity3         [PASSED], CPU time=1.74
test 7 :               adaptivity4         [PASSED], CPU time=2
test 8 :               adaptivity5         [PASSED], CPU time=2.22
test 9 :                 adv_diff1         [PASSED], CPU time=2.82
test 10 :                 adv_diff2         [PASSED], CPU time=3.46
test 11 :                 adv_diff3         [PASSED], CPU time=5.68
test 12 :                 adv_diff4         [PASSED], CPU time=9.11
test 13 :            AdvDiffHandles         [PASSED], CPU time=9.33
test 14 :           AdvDiffHandles2         [PASSED], CPU time=10.25
test 15 :           AdvDiffHandles3         [PASSED], CPU time=10.46
test 16 :                AdvReactDG         [PASSED], CPU time=10.81
test 17 :              AdvReactDG_P         [PASSED], CPU time=11.24
test 18 :                    amultg         [PASSED], CPU time=12.36
test 19 :                   amultg2         [PASSED], CPU time=12.79
test 20 :              beam-springs         [PASSED], CPU time=13.64
test 21 :                 bentonite         [PASSED], CPU time=13.72
test 22 :            BlockLinElast1         [PASSED], CPU time=14.04
test 23 :            BlockLinElast2         [PASSED], CPU time=14.18
test 24 :           BlockLinElast2b         [PASSED], CPU time=14.31
test 25 :            BlockLinElast3         [PASSED], CPU time=15.79
test 26 :             BlockPoisson1         [PASSED], CPU time=15.85
test 27 :             BlockPoisson2         [PASSED], CPU time=15.99
test 28 :             BlockPoisson3         [PASSED], CPU time=16.28
test 29 :                   bodydir         [PASSED], CPU time=16.37
test 30 :                  bodydir2         [PASSED], CPU time=20.36
test 31 :                  bodyload         [PASSED], CPU time=20.44
test 32 :                  buckling         [PASSED], CPU time=24.36
test 33 :         CapacitanceMatrix         [PASSED], CPU time=24.54
test 34 :                 CavityLid         [PASSED], CPU time=25.04
test 35 :                CavityLid2         [PASSED], CPU time=28.68
test 36 :               channel_v2f         [PASSED], CPU time=34.9
test 37 :                    cmultg         [PASSED], CPU time=38.89
test 38 :                   coating         [PASSED], CPU time=42.06
test 39 :         CoordinateScaling         [PASSED], CPU time=42.14
test 40 :           CoupledPoisson1         [PASSED], CPU time=42.21
test 41 :          CoupledPoisson1b         [PASSED], CPU time=42.29
test 42 :           CoupledPoisson2         [PASSED], CPU time=43.58
test 43 :          CoupledPoisson2b         [PASSED], CPU time=45
test 44 :           CoupledPoisson3         [PASSED], CPU time=45.08
test 45 :           CoupledPoisson4         [PASSED], CPU time=45.15
test 46 :           CoupledPoisson5         [PASSED], CPU time=45.25
test 47 :           CoupledPoisson6         [PASSED], CPU time=45.32
test 48 :           CoupledPoisson7         [PASSED], CPU time=45.4
test 49 :           CoupledPoisson8         [PASSED], CPU time=45.48
test 50 :           CoupledPoisson9         [PASSED], CPU time=45.56
test 51 :                   current         [PASSED], CPU time=45.76
test 52 :      current_heat_control         [PASSED], CPU time=46.2
test 53 :           CurvedBndryPFEM         [PASSED], CPU time=46.26
test 54 :                 dft-water         [PASSED], CPU time=46.26
test 55 :               diffuser_sa         [PASSED], CPU time=54.64
test 56 :              diffuser_sst         [PASSED], CPU time=60.6
test 57 :              diffuser_v2f         [PASSED], CPU time=75.99
test 58 :      DivergenceAnalytic2D         [PASSED], CPU time=76.87
test 59 :             el_adaptivity         [PASSED], CPU time=77.01
test 60 :         ElastElstat1DBeam         [PASSED], CPU time=77.16
test 61 :           ElastElstatBeam         [PASSED], CPU time=77.46
test 62 :                elasticity         [PASSED], CPU time=77.99
test 63 :        ElasticLubrication         [PASSED], CPU time=80.55
test 64 :                ElastPelem         [PASSED], CPU time=85.2
test 65 :           Electrokinetics         [PASSED], CPU time=97.87
test 66 :                    elstat         [PASSED], CPU time=102.1
test 67 :              elstat_infty         [PASSED], CPU time=105.89
test 68 :             elstat_source         [PASSED], CPU time=106.03
test 69 :     ExtrusionStructured2D         [PASSED], CPU time=108.68
test 70 :             FlowResNoslip         [PASSED], CPU time=117.48
test 71 :               FlowResSlip         [PASSED], CPU time=126.22
test 72 :                fluxsolver         [PASSED], CPU time=126.31
test 73 :               fluxsolver2         [PASSED], CPU time=126.39
test 74 :               fluxsolver3         [PASSED], CPU time=126.69
test 75 :                  freesurf         [PASSED], CPU time=127.4
test 76 :              freesurf_axi         [PASSED], CPU time=128.71
test 77 :              freesurf_int         [PASSED], CPU time=129.73
test 78 :              freesurf_ltd         [PASSED], CPU time=138.1
test 79 :                  fsi_beam         [PASSED], CPU time=139.49
test 80 :         fsi_beam_optimize         [PASSED], CPU time=146.08
test 81 :                   fsi_box         [PASSED], CPU time=148.21
test 82 :                  fsi_box2         [PASSED], CPU time=149.63
test 83 :                 geomstiff         [PASSED], CPU time=149.87
test 84 :                    gmultg         [PASSED], CPU time=150.16
test 85 :               HeatControl         [PASSED], CPU time=150.23
test 86 :              HeatControl2         [PASSED], CPU time=150.37
test 87 :                    heateq         [PASSED], CPU time=150.74
test 88 :               heateq_bdf2         [PASSED], CPU time=150.97
test 89 :               heateq_bdf3         [PASSED], CPU time=151.19
test 90 :            heateq_newmark         [PASSED], CPU time=151.35
test 91 :     heateq_newmark_global         [PASSED], CPU time=151.5
test 92 :                heateq-par         [PASSED], CPU time=151.59
test 93 :                   HeatGap         [PASSED], CPU time=151.69
test 94 :              HelmholtzBEM         [PASSED], CPU time=152.35
test 95 :             HelmholtzEdge         [PASSED], CPU time=152.64
test 96 :             HelmholtzFace         [PASSED], CPU time=152.94
test 97 :              HelmholtzFEM         [PASSED], CPU time=153.15
test 98 :       HelmholtzPlaneWaves         [PASSED], CPU time=153.66
test 99 :   HelmholtzPlaneWavesAxis         [PASSED], CPU time=154.2
test 100 :        HelmholtzStructure         [PASSED], CPU time=154.85
test 101 :       HelmholtzStructure2         [PASSED], CPU time=155.33
test 102 :       HelmholtzStructure3         [PASSED], CPU time=156.88
test 103 :          InductionHeating         [PASSED], CPU time=157.36
test 104 :         InductionHeating2         [PASSED], CPU time=158.15
test 105 :                    L2norm         [PASSED], CPU time=160.29
test 106 :                 levelset1         [PASSED], CPU time=165.11
test 107 :                 levelset2         [PASSED], CPU time=167.31
test 108 :                 levelset3         [PASSED], CPU time=174.16
test 109 :                levelset3b         [PASSED], CPU time=180.73
test 110 :         LimitDisplacement         [PASSED], CPU time=184.43
test 111 :          LimitTemperature         [PASSED], CPU time=189.43
test 112 :         LimitTemperature2         [PASSED], CPU time=192.22
test 113 :             linearsolvers         [PASSED], CPU time=192.45
test 114 :       linearsolvers_cmplx         [PASSED], CPU time=193.49
test 115 :     LubricationTunedForce         [PASSED], CPU time=194.21
test 116 :                 marangoni         [PASSED], CPU time=194.34
test 117 :         MeshRefineGrading         [PASSED], CPU time=197.73
test 118 :              mgdyn_3phase         [PASSED], CPU time=201.81
test 119 :                  mgdyn_bh         [PASSED], CPU time=232.21
test 120 :            mgdyn_harmonic         [PASSED], CPU time=246.3
test 121 :              mgdyn_steady         [PASSED], CPU time=256.66
test 122 :               mgdyn_torus         [PASSED], CPU time=262.11
test 123 :      mgdyn_torus_harmonic         [PASSED], CPU time=269.4
test 124 :           mgdyn_transient         [PASSED], CPU time=291.99
test 125 :                       mhd         [PASSED], CPU time=294.85
test 126 :                      mhd2         [PASSED], CPU time=298.2
test 127 :                 multimesh         [PASSED], CPU time=298.52
test 128 :         NaturalConvection         [PASSED], CPU time=304.29
test 129 :   NonnewtonianChannelFlow         [PASSED], CPU time=305
test 130 :                   normals         [PASSED], CPU time=305.07
test 131 :        NormalTangentialBC         [PASSED], CPU time=307.2
test 132 : OptimizeSimplexFourHeaters         [PASSED], CPU time=311.06
test 133 :          ParticleAdvector         [PASSED], CPU time=312.96
test 134 :         ParticleAdvector2         [PASSED], CPU time=316.34
test 135 :         ParticleAdvector3         [PASSED], CPU time=323.75
test 136 :   ParticleAdvectorZalesak         [PASSED], CPU time=328.92
test 137 :           ParticleFalling         [PASSED], CPU time=329.07
test 138 :           ParticleHeating         [PASSED], CPU time=339.2
test 139 :                   passive         [PASSED], CPU time=339.92
test 140 :                 periodic1         [PASSED], CPU time=340.04
test 141 :                 periodic2         [PASSED], CPU time=340.17
test 142 :                 periodic3         [PASSED], CPU time=340.25
test 143 :         periodic_explicit         [PASSED], CPU time=341.49
test 144 :    periodic_nonconforming         [PASSED], CPU time=341.59
test 145 :           periodic_offset         [PASSED], CPU time=341.7
test 146 :          periodic_offset2         [PASSED], CPU time=342.42
test 147 :              periodic_rot         [PASSED], CPU time=342.53
test 148 :               PhaseChange         [PASSED], CPU time=346.09
test 149 :              PhaseChange2         [PASSED], CPU time=346.94
test 150 :              PhaseChange3         [PASSED], CPU time=347.83
test 151 :                     piezo         [PASSED], CPU time=347.97
test 152 :                    plates         [PASSED], CPU time=348.07
test 153 :                    pmultg         [PASSED], CPU time=349.36
test 154 :                  pointdir         [PASSED], CPU time=349.43
test 155 :                 pointload         [PASSED], CPU time=349.51
test 156 :                pointload2         [PASSED], CPU time=351.69
test 157 :                PoissonBEM         [PASSED], CPU time=351.79
test 158 :          PoissonBoltzmann         [PASSED], CPU time=356.47
test 159 :                 PoissonDG         [PASSED], CPU time=356.66
test 160 :               PoissonPFEM         [PASSED], CPU time=356.74
test 161 :                PorousPipe         [PASSED], CPU time=357.54
test 162 :                      Q1Q0         [PASSED], CPU time=357.7
test 163 :                 radiation         [PASSED], CPU time=357.9
test 164 :                radiation2         [PASSED], CPU time=358.09
test 165 :               radiation2d         [PASSED], CPU time=360.01
test 166 :               radiation3d         [PASSED], CPU time=364.17
test 167 :                    reload         [PASSED], CPU time=364.97
test 168 :                 reynolds1         [PASSED], CPU time=367.86
test 169 :                 reynolds2         [PASSED], CPU time=368.55
test 170 :                 reynolds3         [PASSED], CPU time=373.73
test 171 :                reynolds3b         [PASSED], CPU time=378.24
test 172 :                  rgdblock         [PASSED], CPU time=380.04
test 173 :              RichardsDyke         [PASSED], CPU time=384.67
test 174 :             RichardsDyke2         [PASSED], CPU time=394.69
test 175 :          RigidMeshMapper1         [PASSED], CPU time=395.36
test 176 :          RigidMeshMapper2         [PASSED], CPU time=395.73
test 177 :                 rot_aniso         [PASSED], CPU time=399.02
test 178 :              RotatingFlow         [PASSED], CPU time=399.8
test 179 :                   rotflow         [PASSED], CPU time=400.55
test 180 :               savescalars         [PASSED], CPU time=400.84
test 181 :      savescalars_boundary         [PASSED], CPU time=400.96
test 182 :          savescalars_flux         [PASSED], CPU time=401.05
test 183 :            ShallowWaterNS         [PASSED], CPU time=403.49
test 184 :                     shell         [PASSED], CPU time=403.71
test 185 :                    shell2         [PASSED], CPU time=404.31
test 186 :                staged_sim         [PASSED], CPU time=405.64
test 187 :                   Step_ke         [PASSED], CPU time=407.57
test 188 :                   Step_ns         [PASSED], CPU time=408
test 189 :                   Step_sa         [PASSED], CPU time=416.5
test 190 :            Step_sst-kw-wf         [PASSED], CPU time=434.12
test 191 :               Step_stokes         [PASSED], CPU time=434.24
test 192 :         Step_stokes_block         [PASSED], CPU time=434.45
test 193 :                  Step_v2f         [PASSED], CPU time=458.62
test 194 :                StokesPFEM         [PASSED], CPU time=458.75
test 195 :                StokesProj         [PASSED], CPU time=459.18
test 196 :       StrainCalculation01         [PASSED], CPU time=459.46
test 197 :       StrainCalculation02         [PASSED], CPU time=465.41
test 198 :       StrainCalculation03         [PASSED], CPU time=472.1
test 199 :               streamlines         [PASSED], CPU time=472.44
test 200 :                    stress         [PASSED], CPU time=472.74
test 201 :                 structmap         [PASSED], CPU time=472.85
test 202 :                structmap2         [PASSED], CPU time=474.1
test 203 :                structmap3         [PASSED], CPU time=486.62
test 204 :           ThermalActuator         [PASSED], CPU time=551.68
test 205 :            ThermalBiMetal         [PASSED], CPU time=551.84
test 206 :           ThermalBiMetal2         [PASSED], CPU time=552.04
test 207 :           ThermalCompress         [PASSED], CPU time=552.89
test 208 :            ThermoElectric         [PASSED], CPU time=553.02
test 209 :                 TimeAdapt         [PASSED], CPU time=555.13
test 210 :                  TimeFunc         [PASSED], CPU time=557.59
test 211 :                    tresca         [PASSED], CPU time=557.76
test 212 :                  vortex2d         [PASSED], CPU time=558.9
test 213 :                  vortex3d         [PASSED], CPU time=560.19
test 214 :                   WaveEqu         [PASSED], CPU time=560.33
test 215 :         WeightComputation         [PASSED], CPU time=560.41
Tests completed, passed: 215 out of total 215 tests
Cumulative CPU time used in test: 560.41 s
joni@mpi1:/mpi3/elmerfem/trunk/fem/tests$


Netgen parraler

NOT TESTED
Source:<a href="http://code.google.com/p/parallel-netgen/" target="_blank"> netgen-paraler</a>

UNDER TEST:
apt-get libtogl1 libtogl-dev "tcl" "tix" "occ" "jpeglib" "ffmpeglib"

http://sourceforge.net/projects/netgen-mesher/files/latest/download?source=files
./configure --enable-parallel  --with-tcl=/usr/lib/tcl8.5 --with-tk=/usr/lib/tk8.5 --with-occ=/usr/lib/ --with-togl=/usr/lib --enable-jpeglib --enable-ffmpeg

build instruction:
http://sourceforge.net/apps/mediawiki/netgen-mesher/index.php?title=Installing_on_Ubuntu
http://glaros.dtc.umn.edu/gkhome/metis/parmetis/download

<img src="https://www.google.com/chart?chc=sites&cht=d&chdp=sites&chl=%5B%5BGoogle+Gadget%27%3D20%27f%5Cv%27a%5C%3D0%2710%27%3D499%270%27dim%27%5Cbox1%27b%5CF6F6F6%27fC%5CF6F6F6%27eC%5C0%27sk%27%5C%5B%22Google+Group%22%27%5D%27a%5CV%5C%3D12%27f%5C%5DV%5Cta%5C%3D10%27%3D0%27%3D500%27%3D597%27dim%27%5C%3D10%27%3D10%27%3D500%27%3D597%27vdim%27%5Cbox1%27b%5Cva%5CF6F6F6%27fC%5CC8C8C8%27eC%5C%27a%5C%5Do%5CLauto%27f%5C&sig=DUR0sa0cgFwYKZ2Gld2wIgZ7yrE" data-igsrc="http://153.gmodules.com/ig/ifr?mid=153&synd=trogedit&url=http%3A%2F%2Fwww.gstatic.com%2Fsites-gadgets%2Fforum%2Fsingle_group.xml&up_GroupUrl=https%3A%2F%2Fgroups.google.com%2Fforum%2F%23!forum%2Fubuntuparraler&up_ShowSearch=true&up_ShowPopout=true&up_Transparency=false&h=600&w=100%25" data-type="ggs-gadget" data-props="align:left;borderTitle:Google Group Ubuntu Parraler;height:600;igsrc:http#58//153.gmodules.com/ig/ifr?mid=153&synd=trogedit&url=http%3A%2F%2Fwww.gstatic.com%2Fsites-gadgets%2Fforum%2Fsingle_group.xml&up_GroupUrl=https%3A%2F%2Fgroups.google.com%2Fforum%2F%23!forum%2Fubuntuparraler&up_ShowSearch=true&up_ShowPopout=true&up_Transparency=false&h=600&w=100%25;mid:153;scrolling:no;showBorder:true;showBorderTitle:true;spec:http#58//www.gstatic.com/sites-gadgets/forum/single_group.xml;up_GroupUrl:https#58//groups.google.com/forum/#35!forum/ubuntuparraler;up_ShowPopout:true;up_ShowSearch:true;up_Transparency:false;width:100%;" style="display:block;text-align:left;margin-right:auto;" class="igm" height="600" width="500">

<img src="https://www.google.com/chart?chc=sites&cht=d&chdp=sites&chl=%5B%5BGoogle+Gadget%27%3D20%27f%5Cv%27a%5C%3D0%2710%27%3D119%270%27dim%27%5Cbox1%27b%5CF6F6F6%27fC%5CF6F6F6%27eC%5C0%27sk%27%5C%5B%22Include+gadget+(iframe)%22%27%5D%27a%5CV%5C%3D12%27f%5C%5DV%5Cta%5C%3D10%27%3D0%27%3D120%27%3D237%27dim%27%5C%3D10%27%3D10%27%3D120%27%3D237%27vdim%27%5Cbox1%27b%5Cva%5CF6F6F6%27fC%5CC8C8C8%27eC%5C%27a%5C%5Do%5CLauto%27f%5C&sig=gUycQn5kgb7bG9PkOC0UpjtwoTI" data-igsrc="http://0.gmodules.com/ig/ifr?mid=0&synd=trogedit&url=http%3A%2F%2Fwww.gstatic.com%2Fsites-gadgets%2Fiframe%2Fiframe.xml&up_iframeURL=http%3A%2F%2Frcm.amazon.com%2Fe%2Fcm%3Ft%3Dprodurevie049-20%26o%3D1%26p%3D8%26l%3Das1%26asins%3DB000LB7G00%26ref%3Dqf_sp_asin_til%26fc1%3D000000%26IS2%3D1%26lt1%3D_blank%26m%3Damazon%26lc1%3D0000FF%26bc1%3D000000%26bg1%3DFFFFFF%26f%3Difr&up_scroll=no&w=120px&h=240px" data-type="ggs-gadget" data-props="align:left;height:240px;igsrc:http#58//0.gmodules.com/ig/ifr?mid=0&synd=trogedit&url=http%3A%2F%2Fwww.gstatic.com%2Fsites-gadgets%2Fiframe%2Fiframe.xml&up_iframeURL=http%3A%2F%2Frcm.amazon.com%2Fe%2Fcm%3Ft%3Dprodurevie049-20%26o%3D1%26p%3D8%26l%3Das1%26asins%3DB000LB7G00%26ref%3Dqf_sp_asin_til%26fc1%3D000000%26IS2%3D1%26lt1%3D_blank%26m%3Damazon%26lc1%3D0000FF%26bc1%3D000000%26bg1%3DFFFFFF%26f%3Difr&up_scroll=no&w=120px&h=240px;mid:0;spec:http#58//www.gstatic.com/sites-gadgets/iframe/iframe.xml;up_iframeURL:http#58//rcm.amazon.com/e/cm?t=produrevie049-20&o=1&p=8&l=as1&asins=B000LB7G00&ref=qf_sp_asin_til&fc1=000000&IS2=1&lt1=_blank&m=amazon&lc1=0000FF&bc1=000000&bg1=FFFFFF&f=ifr;up_scroll:no;width:120px;wrap:true;" style="display:inline;float:left;margin:5px 20px 5px 0;" class="igm" height="240" width="120">

<img src="https://www.google.com/chart?chc=sites&cht=d&chdp=sites&chl=%5B%5BGoogle+Gadget%27%3D20%27f%5Cv%27a%5C%3D0%2710%27%3D119%270%27dim%27%5Cbox1%27b%5CF6F6F6%27fC%5CF6F6F6%27eC%5C0%27sk%27%5C%5B%22Include+gadget+(iframe)%22%27%5D%27a%5CV%5C%3D12%27f%5C%5DV%5Cta%5C%3D10%27%3D0%27%3D120%27%3D237%27dim%27%5C%3D10%27%3D10%27%3D120%27%3D237%27vdim%27%5Cbox1%27b%5Cva%5CF6F6F6%27fC%5CC8C8C8%27eC%5C%27a%5C%5Do%5CLauto%27f%5C&sig=gUycQn5kgb7bG9PkOC0UpjtwoTI" data-igsrc="http://0.gmodules.com/ig/ifr?mid=0&synd=trogedit&url=http%3A%2F%2Fwww.gstatic.com%2Fsites-gadgets%2Fiframe%2Fiframe.xml&up_iframeURL=http%3A%2F%2Frcm.amazon.com%2Fe%2Fcm%3Ft%3Dprodurevie049-20%26o%3D1%26p%3D8%26l%3Das1%26asins%3DB006H2RP12%26ref%3Dqf_sp_asin_til%26fc1%3D000000%26IS2%3D1%26lt1%3D_blank%26m%3Damazon%26lc1%3D0000FF%26bc1%3D000000%26bg1%3DFFFFFF%26f%3Difr&up_scroll=no&w=120px&h=240px" data-type="ggs-gadget" data-props="align:left;height:240px;igsrc:http#58//0.gmodules.com/ig/ifr?mid=0&synd=trogedit&url=http%3A%2F%2Fwww.gstatic.com%2Fsites-gadgets%2Fiframe%2Fiframe.xml&up_iframeURL=http%3A%2F%2Frcm.amazon.com%2Fe%2Fcm%3Ft%3Dprodurevie049-20%26o%3D1%26p%3D8%26l%3Das1%26asins%3DB006H2RP12%26ref%3Dqf_sp_asin_til%26fc1%3D000000%26IS2%3D1%26lt1%3D_blank%26m%3Damazon%26lc1%3D0000FF%26bc1%3D000000%26bg1%3DFFFFFF%26f%3Difr&up_scroll=no&w=120px&h=240px;mid:0;spec:http#58//www.gstatic.com/sites-gadgets/iframe/iframe.xml;up_iframeURL:http#58//rcm.amazon.com/e/cm?t=produrevie049-20&o=1&p=8&l=as1&asins=B006H2RP12&ref=qf_sp_asin_til&fc1=000000&IS2=1&lt1=_blank&m=amazon&lc1=0000FF&bc1=000000&bg1=FFFFFF&f=ifr;up_scroll:no;width:120px;wrap:true;" style="display:inline;float:left;margin:5px 20px 5px 0;" class="igm" height="240" width="120"><img src="https://www.google.com/chart?chc=sites&cht=d&chdp=sites&chl=%5B%5BGoogle+Gadget%27%3D20%27f%5Cv%27a%5C%3D0%2710%27%3D119%270%27dim%27%5Cbox1%27b%5CF6F6F6%27fC%5CF6F6F6%27eC%5C0%27sk%27%5C%5B%22Include+gadget+(iframe)%22%27%5D%27a%5CV%5C%3D12%27f%5C%5DV%5Cta%5C%3D10%27%3D0%27%3D120%27%3D237%27dim%27%5C%3D10%27%3D10%27%3D120%27%3D237%27vdim%27%5Cbox1%27b%5Cva%5CF6F6F6%27fC%5CC8C8C8%27eC%5C%27a%5C%5Do%5CLauto%27f%5C&sig=gUycQn5kgb7bG9PkOC0UpjtwoTI" data-igsrc="http://0.gmodules.com/ig/ifr?mid=0&synd=trogedit&url=http%3A%2F%2Fwww.gstatic.com%2Fsites-gadgets%2Fiframe%2Fiframe.xml&up_iframeURL=http%3A%2F%2Frcm.amazon.com%2Fe%2Fcm%3Ft%3Dprodurevie049-20%26o%3D1%26p%3D8%26l%3Das1%26asins%3DB0016OOS16%26ref%3Dqf_sp_asin_til%26fc1%3D000000%26IS2%3D1%26lt1%3D_blank%26m%3Damazon%26lc1%3D0000FF%26bc1%3D000000%26bg1%3DFFFFFF%26f%3Difr&up_scroll=no&w=120px&h=240px" data-type="ggs-gadget" data-props="align:left;height:240px;igsrc:http#58//0.gmodules.com/ig/ifr?mid=0&synd=trogedit&url=http%3A%2F%2Fwww.gstatic.com%2Fsites-gadgets%2Fiframe%2Fiframe.xml&up_iframeURL=http%3A%2F%2Frcm.amazon.com%2Fe%2Fcm%3Ft%3Dprodurevie049-20%26o%3D1%26p%3D8%26l%3Das1%26asins%3DB0016OOS16%26ref%3Dqf_sp_asin_til%26fc1%3D000000%26IS2%3D1%26lt1%3D_blank%26m%3Damazon%26lc1%3D0000FF%26bc1%3D000000%26bg1%3DFFFFFF%26f%3Difr&up_scroll=no&w=120px&h=240px;mid:0;spec:http#58//www.gstatic.com/sites-gadgets/iframe/iframe.xml;up_iframeURL:http#58//rcm.amazon.com/e/cm?t=produrevie049-20&o=1&p=8&l=as1&asins=B0016OOS16&ref=qf_sp_asin_til&fc1=000000&IS2=1&lt1=_blank&m=amazon&lc1=0000FF&bc1=000000&bg1=FFFFFF&f=ifr;up_scroll:no;width:120px;wrap:true;" style="display:inline;float:left;margin:5px 20px 5px 0;" class="igm" height="240" width="120">




Retrieved from "http://oh5bzr.hamshack.info/wiki/index.php?title=MPI4YOU&oldid=654"