Cluster Kurrola

From Radio Amteur Station OH5BZR - WIKI
Revision as of 20:24, 29 November 2015 by 192.168.0.41 (talk) (→‎RESTORE whitout DATABASE)
Jump to: navigation, search

Precise mpi4you

hi,

Upgrade under work, many things changes when moveing from precise to trusty. Same whit notes, those are mixed now. Below precise notes and trusty note's after that. Some trusty update mixed at precise to make everything easyer?

Installation

Cluster Kurrola desing targets: easy to manage ( webmin and LTSP are tested and compared ) , get parallel solvers work ( bind9,dhcp,nfs4 ) giganet whit IP4, stabilize system ( Twin Machine , "4G Cantenna mobile internet Antenna,.." server to work outside world at IPV6 and inside IPV4 / IPV6 (wwdial,hostapd,radvd,shoreline,shoreline6) basic servers ( mysql, postresql, apache2, mediawiki, mythweb, calibre ) and wlan services to workstations

To get parallel software like elmerfem and openfoam to work correctly you need full stack ( name services DNS and routing ) of network service and as learnt at practice at twin machine configuration. Twin machine configuration for DHCP ( both ipv4 and ipv6 ), BIND9 ( master and slave ). BIND9 namaserice is needed to have stable routing even connection to global internet breaks. DHCP is used to networkboot and network address managment. To get IPV6 and IPV4 filtered and routed correctly Shoreline and Shoreline6 are simple tool to configure firewall to Linux kernel. You need device level softaware like wvdial to get 4G modem/router to talk whit server routing and brdegeing software as well hostapd to open wlan services. To manage configuration of servers as well workstations webmin is good tool. To get network working well whit software you need common usernamre as well password base to get communication work between machines and services like NFS4 and rsh-redone.

For post prosessing you need Paraview as well support for 3d mouse. That's possible by building Praview 4.0 from sources whit vrpn and hidapi.

Cluster Kurrola ( MPI4YOU ) base parameters: 

/mpi3 is directory shared via NFS4 to all nodes
/mpi3/B3 Build Script's for version 3
/mpi3/S3 Source Code version 3
/mpi3/C3 Code Verion 3

Each node has Ubuntu 12.04 LTS ( Long Term Suport version, no need to change OS for 7 years )

Connections

NetworkConfiguration.jpg

Installing NetworkManager is simple way to get all right for single machine network. I got big supprise when I understood how complex it come to setup STABILE beawulf parallel network to home, you need to setup all network components.


Ubuntu 12.04 have much automation at protocol stack, it finds devices when connected ([udev]) and most cases setups hardware drivers and creates device like eth and ttyUSB. At Cluster Kurrola Installation NetworkManager is totaly removed due instability and some server software incompatibility issues (calibre,mpich3,rsh-redone). What have to do is to glue kernel IPV4 and IPV6 part's together whit devices and give correct parameter's to devices for operation.

Physical: 4G mobile network Saunanlahti & ELISA, HSPA+DC

Modem  : Huawei 398

Protocol stack:

DC-HSPA ppp 

   ipv4: rounting, bridge  Ubuntu kernel
         -shorewall, vwdial, isc-dhcp., bind9

   ipv6: System Six IPV6 tunnel over IPV4 network
         -shorewall6, isc-dhcp, bind9,aiccu


Server mpi2:


ppp0: wvdialD, wvdial, aiccu br0: eth0 shoreline shoreline6 aiccu raddvd, wlan0 (hostapd)

80 / http, drupal 443 / https, drupal 81 / nominatim-osm-MTK MAPs 8082 / owncloud 

/ bind9
/ dhcp
/ dhcp ipv6

3142 / apt-cacher-ng 8080 / calibre 8081 / internal drupal site 7000-7200 / mpi4you mpich3


apt-get

due self compiled packages HOLD LINUX KERNEL : 

sudo apt-mark hold linux-generic linux-headers-generic linux-image-generic

to unhold: 

sudo apt-mark unhold linux-generic linux-headers-generic linux-image-generic

hostapd

/etc/hostapd/hostapd.conf : 

interface=wlan0
bridge=br0
driver=nl80211
macaddr_acl=1
accept_mac_file=/etc/hostapd.accept
ssid=KURROAIR
hw_mode=g
channel=3 
wpa=2
wpa_passphrase=MyEncryptionKey
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP CCMP
wpa_ptk_rekey=600

For next step

https://github.com/Janhouse/4g-connect-linux/blob/master/huawei-4g-linux.pl
https://answers.launchpad.net/ubuntu/+source/gnome-nettool/+question/211095
waiting 12.04 LTS -> 14.04 LTS upgrade


http://dd-wrt.ca/phpBB2/viewtopic.php?t=139831&postdays=0&postorder=asc&start=15&sid=6f91400a4c33da5e04c1206c11d847da

echo "12d1 1506" > /sys/bus/usb-serial/drivers/option1/new_id
sleep 20
usb_modeswitch -v 12d1 -p 1505 -V 12d1 -P 1506 -M "55534243123456780000000000000011062000000100000000000000000000" -n 1 -W
sleep 30
nvram set 3gdata=/dev/usb/tts/0
Then it should get online when you restart the router given that the apn are set correctly.
3. cron (if you get disconnected)
* * * * * [ "`/usr/sbin/nvram get 3gdata`" == "/dev/usb/tts/2" ] && /usr/sbin/nvram set 3gdata=/dev/usb/tts/0


Basic setup:

sudo apt-get install libqmi-utils /udev/rules.d/*huawei*

wvdial

/etc/wvdial.conf : 

[Dialer connect]
Modem Type = Analog Modem
Modem = /dev/ttyUSB0
Phone = *99#
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
Init3 = AT^SYSCFGEX="0302",3fffffff,1,4,7fffffffffffffff,,
Init4 = AT+CGDCONT=1,"ip","internet",,0,0
ISDN = off
Ask Username = off
Username = no
Ask Password = off
Password = no
Baud = 9600
Stupid mode = on
Auto DNS = on
[Dialer signal]
Modem = /dev/ttyUSB1
Init1 = AT+CSQ
Init2 = AT^SYSINFO
Init3 = AT+CIND?

/etc/init.d/wvdialD : 

#!/bin/sh
NAME="wvdialD"
DESC="wvdial keep alive and startup"
test -x $DAEMON || exit 0
case "$1" in
start)
date >> /var/log/syslog
echo "[wvdialD] *** Starting Mobile Broadband Connection. wvdial up" >> /var/log/syslog
date >> /var/log/3g4gsession 
while true; do
/home/joni/webon >> /var/log/3g4gsession &
wvdial connect >> /var/log/3g4gsession
echo "RESTART" >> /var/log/3g4gsession 
date >> /var/log/3g4gsession
sleep 5 
done
date >> /var/log/syslog
/home/joni/weboff
echo "[wvdialD] Mobile Broadband Connection closed *******." >> /var/log/syslog
echo "STOP-DROP" > /var/log/3g4gsession
date >> /var/log/3g4gsession
;;
stop)
echo "[wvdialD] Stopping wvdial." >> /var/log/syslog  
killall wvdial
echo "[wvdialD] wvdial down." >> /var/log/syslog
echo "STOPED BY COMMAND" > /var/log/3g4gsession
date >> /var/log/3g4gsession
killall wvdialD
;;
status)
# Check network status with nmcli 
cat /var/log/3g4gsession
;;
*)
echo "[wvdialD] Keeps wvdial up"
echo $"Usage: $0 {start|stop|status}"
exit 1
esac
exit 0

IPV4,IPV6

/etc/network/interface : 

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
#Primary internet connection
auto ppp0
iface ppp0 inet wvdial
# The primary network interface
auto eth0
iface eth0 inet manual
manual wlan0
#address 192.168.0.40
iface wlan0 inet manual
auto br0
iface br0 inet static
network 192.168.0.0
address 192.168.0.XX
broadcast 192.168.0.255
netmask 255.255.255.0
bridge_ports eth0 wlan0
dns-nameservers 192.168.0.40 192.168.0.XX 
dns-search X.kurrola.dy.fi X.kurrola.dy.fi
dns-domain X.kurrola.dy.fi X.kurrola.dy.fi
gateway 192.168.0.254
#ipv6
iface eth0 inet6 static
address 2001:XXXX:100:363::40
netmask 64

ipv6 routing local wlan node: 

sysctl -w net.ipv6.conf.br0.disable_ipv6=0 ( echo1 > /proc/sys/net/ipv6/conf/br0/disable_ipv6 )
sudo ip -6 addr add 2001:14b8:100:363::41/64 dev br0

ipv6 routing at router whit aiccu and firewall: 

sysctl -w net.ipv6.conf.eth0.disable_ipv6=0
sudo ip -6 addr add 2001:14b8:100:363::40/64 dev eth0
sudo ip -6 route add 2001:14b8:100:363::40/64 dev eth0


[postfix] https://help.ubuntu.com/community/PostfixAmavisNew
https://help.ubuntu.com/community/Postfix

Orginal: http://christian.skala.me/blog/gmail-why-are-you-doing-this-to-me/#.Unt2oEOJRwY

Forcing IPv4 when sending to Gmail

If we want to force postfix to use IPv4 instead IPv6 when sending to Gmail, we need to add a line in the main configuration file of postfix like so 1 2

$ sudo nano /etc/postfix/main.cf transport_maps = hash:/etc/postfix/transport

Let’s create this new file and add a new entry in the transport table. Basically, we fore mail for gmail.com to use smtp-ipv4 (which we have to define later, see below) 1 2

$ sudo nano /etc/postfix/transport gmail.com smtp-ipv4:

Here comes the key part. We create a new rule for our new entry and define to use IPv4 protocol explicitly when this rule applies: 1 2 3

$ sudo nano /etc/postfix/master.cf smtp-ipv4 unix .. .. .. .. smtp 

-o inet_protocols=ipv4

We need to run the postmap command after the change and reload postfix 1 2

$ sudo postmap /etc/postfix/transport $ sudo postfix reload


Ltps

ACPI - SUSPEND - HIBERNATE START

/etc/udev/rules.d/

Install: 

apt-get install usb-modeswitch

http://www.draisberghof.de/usb_modeswitch/#install http://www.siptune.net/tiki-index.php?page=udev+rules+mokkuloille http://sysadminnotebook.blogspot.fi/2012/03/vodafone-k5005-huawei-e389-4g-modem-on.html

/lib/udev/rules.d/40-usb_modeswitch.rules : 

# Huawei 398 oma
ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="1506", RUN+="usb_modeswitch '%b/%k'"

/etc/usb_modeswitch.d : 

# Huawei E398
TargetVendor= 0x12d1
TargetProduct= 0x1506
MessageContent="55534243123456780000000000000011062000000100000000000000000000"

usb_modeswitch -R -v 12d1 -p 1505 

#!/bin/bash
rmmod option
modprobe option
echo "12d1 1506" > /sys/bus/usb-serial/drivers/option1/new_id
usb_modeswitch -v 12d1 -p 1505 -V 12d1 -P 1506 -M "55534243123456780000000000000011062000000100000000000000000000" -n 1

New testing:

echo "12d1 1506" > /sys/bus/usb-serial/drivers/option1/new_id echo -e "AT^NDISDUP=1,1,\"online.telia.se\"\r" > /dev/ttyUSB0 dhclient wwan0

rsh-redone

apt-get install rsh-redone-client rsh-redone-server

/etc/hosts.equiv

  1. command access to your system .

+ + ~


/etc/inetd.conf

  1. BSD: Shell, login, exec and talk are BSD protocols.

shell stream tcp nowait root /usr/sbin/tcpd /usr/sbin/in.rshd login stream tcp nowait root /usr/sbin/tcpd /usr/sbin/in.rlogind


/home/joni/.rhosts 

kalle
mika
hei
paakone
192.168.0.77
192.168.0.78
192.168.0.132
192.168.0.233

/etc/pam.d/rsh 

#
# The PAM configuration file for the rsh (Remote Shell) service
#
# Due to limitations in the rsh protocol, modules depending on the conversation
# function to work cannot be used.  This includes authentication modules such
# as pam_unix.so.
#auth   required        pam_rhosts.so
auth    sufficient        pam_rhosts.so
#auth   required        pam_nologin.so
#auth   required        pam_env.so
#account        required        pam_unix.so
#session        required        pam_unix.so
auth    sufficient        pam_rhosts.so

Local Time keyboard

https://help.ubuntu.com/community/Howto%3A%20Custom%20keyboard%20layout%20definitions 

sudo apt-get install loacles kde-l10n-fi language-selector-gnome language-selector-common language-pack-gnome-fi-base language-pack-gnome-en-base language-pack-gnome-de-base language-pack-gnome-fr-base language-pack-fi-base language-pack-de-base language-pack-fr-base language-pack-en-base language-pack-fi language-pack-fr language-pack-de language-pack-en language-pack-gnome-de language-pack-gnome-fr language-pack-gnome-en language-pack-gnome-fi language-pack-en-base language-pack-fi-base language-pack-de-base language-pack-fr-base firefox-locale-fi thunderbird-locale-fi voikko-fi 
#timezone
dpkg-reconfigure tzdata
#local languages


Files:

/etc/default/locale: 

LANG=fi_FI.UTF-8

/etc/default/keyboard 
/etc/X11/xorg.conf
udevadm trigger --subsystem-match=input --action=change

THEN: 

 locale-gen purge

Debian not unbuntu: 


dpkg-reconfigure localeconf
sudo dpkg-reconfigure console-data
sudo dpkg-reconfigure console-setup



sudo dpkg-reconfigure keyboard-configuration - work's after reboot
sudo dpkg-reconfigure kdb

setxkbmap fi - change keyboard command line

Monitoring

sudo apt-get install lm-sensors gkrellmd
sudo sensors dedect
sudo service module-init-tools start
sudo nano /etc/gkrell*.conf
sudo /etc/init.d/gkrellmd start

Nvidia most recent

http://www.nvidia.com/download/driverResults.aspx/69378/en-us 

wget *

update-grub /dev/sda
update-grub /dev/sdb

MediaWiki

Running MediaWiki on Ubuntu [1]
PDF support [2]
Latex support sudo apt-get install ocml texlive and changes [3]

Drupal

Mysql error message

http://drupal.stackexchange.com/questions/34915/alternate-page-for-mysql-connection-error 

$mysqli = new mysqli($databases['default']['default']['host'],$databases['default']['default']['username'],$databases['default']['default']['password'],$databases['default']['default']['database']);
   if (mysqli_connect_error()) {
    echo "Your HTML here";
    exit();
   }

g++-4.7

sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install g++-4.7

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-4.6
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.7 40 --slave /usr/bin/g++ g++ /usr/bin/g++-4.7
sudo update-alternatives --config gcc


IPV6

https://www.sixxs.net/wiki/Installing_a_Subnet


Sixxs How To[4]

Ipv6 How to[5]

http://www.shorewall.net/IPv6Support.html

http://tldp.org/HOWTO/Linux+IPv6-HOWTO/x1083.html

http://www.shorewall.net/6to4.htm

Space Navigator

See vrpn how to get it work whit Paraview.

http://docs.salome-platform.org/salome_4_1_5/visu/user/navigation_in_gauss_viewer_page.html http://code.google.com/p/liquid-galaxy/wiki/LinuxSpaceNavigator

ODPDS

SOURCES:
ODPDS wiki
[Orginal J-P's wiki https://sites.google.com/site/jpsdatareviewstheboy007/kindle-new-kindle-3/jetbook-whit-calibre-and-calibre2odpds]

Source Under construction: 

JetBook whit Calibre and calibre2odpds Installations note's

Download & install: Ubuntu 12.4 LTS Calibre calibre2odpds - manual version apache server - run: sudo apt-get install apache2 JAVA - run: sudo apt-get install openjdk-7-jre icedtea-7-plugin

Extra: Ectaco instructions to create by hand library

Directory structure of example: /home/calibre/CalibreBooks - calibre databse directory /var/www/odpds - link to /home/calibre/CalibreBooks, run: sudo ln -s /home/calibre/CalibreBooks /var/www/odpdss


Calibre2odpds configuration file is located /home/.calibre2opds/default.profile.xml just manually edit it:

gedit /home/joni/.calibre2opds/default.profile.xml

LINE: <entry key="UrlBase">http://yourip/</entry>


run: ./rungui.sh

Hard Disk

USB 3.0 - Sandberg Docking Station 705 Mb/s 

WD20EARX 52-122 Mb/s average 89 Mb/s
ST3 WD20EZRX 74-140 Mb/s average 120 Mb/s


SATA 2 

 sdb WD10EFRX 68-152 Mb/s average 59 Mb/s
myjohn2 ST 3500AS 42-91 Mb/s average 66 Mb/s
myjohn3 WD6402A 52-165 Mb/s 92 Mb/s
var/lib/mysql 268-278 Mb/s 273 Mb/s

Evolution & SMART CARDS

a) install OpenSC & related 

apt-get install opensc

Add security device: 

nano .pki/nssdb/pkcs11.txt 

library=/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
name=OpenSC


b) at Firefox add security device, 

/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so

c) copy firefox installation to evolution: 

cd ~/.mozilla/firefox/*.default  # warining, there can be multiple configurations
cp cert8.db key3.db secmod.db /home/joni/.local/share/evolution/


add ldap's

Julha address: ldapp://ldap.julha.fi port: 389 dmd=julkishallinto, c=fi

Finid address: ldapp://ldap.fineid.fi port: 389 c=fi

tai

dmdname=fineid,c=fi subtree

MythTV

http://wiki.team-mediaportal.com/1_MEDIAPORTAL_1/11_Preparing_Your_System/01_Supported_TV_Cards http://parker1.co.uk/mythtv_freesat.php http://www.lyngsat.com/Astra-1KR-1L-1M-2C.html http://www.tbsdtv.com/products/tbs6928-dvb-s2-tv-tuner-ci-pcie-card.html

!!!No Trans bug solution https://github.com/MythTV/mythweb/commit/6e251346675db00a889fc922ab7f270c1d9b66c9

digikam

http://linuxg.net/how-to-install-digikam-3-3-0-on-ubuntu-linux-mint-debian-and-derivates/

precise dose not have:

OpenCV enough fresh
http://docs.opencv.org/doc/tutorials/introduction/linux_install/linux_install.html#linux-installation

CAN NOT WORK 12.4 due depencyes,...easyly!

sageserver

http://wiki.sagemath.org/SageServer
http://wiki.sagemath.org/SageServer
http://www.sagemath.org/pdf/en/installation/installation.pdf

above should work out whit below changes

You need extra user's sageserver, sage0 .. sage9 You need extra groups sageserver, sageuser Chesk that user sageserver and sage0..sage9 can access /mpi3/S3/sage-6.1 (sage source directory) 

sudo apt-get install gnutsl
sudo apt-get install graphicsmagick-imagemagick-compat 
sudo apt-get install libpango1.0-dev libcairo-dev 
sudo apt-get install texlive-full 
sudo apt-get install libreadline-dev


bash script for userspace at server: 

#!/bin/bash
sudo -u sageserver@192.168.0.41 -i ssh-keygen -t dsa
for i in {0..9}
do
mkdir /home/sage$i/.ssh
cat /home/sageserver/.ssh/id_dsa.pub > /home/sage$i/.ssh/authorized_keys
chown -R sage$i:sageserver /home/sage$i/.ssh
chmod 700 -R /home/sage$i/.ssh
chmod 755 -R /home/sage$i/.ssh/authorized_keys
done
chmod 755 -R /home/sageserver/.ssh/authorized_keys
chmod 700 -R /home/sageserver/.ssh

notebook startup script script /home/sageserver/startnotebook: 

#!/bin/sh
~/sage/sage -c "notebook(interface='localhost', 
directory='./sage_notebook.sagenb',
automatic_login=false,
secure=false,
port=9999, 
accounts=false, 
timeout=3600, 
server_pool=['sage%d@localhost'%i for i in range(10)], 
ulimit='-u 100 -t 36000 -v 500000')"


upstart for server: 

#!/bin/sh
NAME="sageserverD"
DESC="sageserver keep alive and startup"
test -x $DAEMON || exit 0
case "$1" in

start) 

          date >> /var/log/syslog

echo "[sageserverD] *** Starting SageServer" >> /var/log/syslog 

           exec su -l sageserver /home/sageserver/startnotebook > /dev/null
    	;;

stop) 

      echo "[sageserverD] Stopping sageserverD." >> /var/log/syslog  
       killall sage

;; status) 

     	;;

*) echo "SageServer" 

      echo $"Usage: $0 {start|stop|status}"
      exit 1
esac
exit 0

Running Server

MPI4YOU - elmer, netgen and openfoam whit mpich3
HYDRA
GIS - Installation openstreetmap, gdal, ogr2ogr, psql2osm
Library - Installation calibre, mendelay, recol
GPG - Finish

Server start & stop

/etc/init.d/networking restart
/etc/init.d/snmpd restart
service bind9 restart
sudo service isc-dhcp-server start
/etc/init.d/apache2 restart


SSL Update & create =

SERVER CERTIFICATE UPDATE at SSL DIRECTORY 

sudo openssl genrsa -out privkey.pem 4096
sudo openssl req -new -key privkey.pem -out cert.csr
  GO CACERT
sudo nano kurrola.dy.fi.crt

sudo cp privkey.pem kurrola.dy.fi.insecure
wget http://www.cacert.org/certs/root.crt

CLIENT SERTIFICATE GENERATION

BACULA

http://wiki.bacula.org/doku.php?id=faq 

What to backup:
/etc/bacula - you need configuration
/var/lib/bacula/bacula.sql - you need backup from database

Crete index at MySql database: 

   File.PathId
   File.FilenameId
   Job.FileSetId
   Job.ClientId

apt-cacher-ng

Import New Upgrade from CD-ROM

mount -t smbfs //192.168.0.41/cdrom /var/cache/apt-cacher-ng/_import/ http://mpi2:3142/acng-report.html Import

MariaDB - under test

Due pronlem's whit mysql started testong MariaDB

https://downloads.mariadb.org/mariadb/repositories/#mirror=netinch&distro=Ubuntu&distro_release=precise&version=10.0

<a href="http://mariadb.org"> 

 <img src="http://badges.mariadb.org/mariadb-badge-125x50.png"
   width="125" height="50" border="0" alt="Powered by MariaDB"
   title="Powered by MariaDB" />

</a> 

SELECT concat('ALTER TABLE `',TABLE_NAME,'` ENGINE=tokudb;')
FROM Information_schema.TABLES 
WHERE ENGINE != 'tokudb' AND TABLE_TYPE='BASE TABLE' 
AND TABLE_SCHEMA='bacula'

ALTER TABLE `BaseFiles` ENGINE=tokudb; 

ALTER TABLE `Client` ENGINE=tokudb;                  
ALTER TABLE `Counters` ENGINE=tokudb;                 
ALTER TABLE `Device` ENGINE=tokudb;                   
ALTER TABLE `File` ENGINE=tokudb;                     
ALTER TABLE `FileSet` ENGINE=tokudb;                  
ALTER TABLE `Filename` ENGINE=tokudb;                 
ALTER TABLE `Job` ENGINE=tokudb;                      
ALTER TABLE `JobHisto` ENGINE=tokudb;                
ALTER TABLE `JobMedia` ENGINE=tokudb;                 
ALTER TABLE `Location` ENGINE=tokudb;                 
ALTER TABLE `LocationLog` ENGINE=tokudb;              
ALTER TABLE `Log` ENGINE=tokudb;                      
ALTER TABLE `Media` ENGINE=tokudb;                    
ALTER TABLE `MediaType` ENGINE=tokudb;                
ALTER TABLE `Path` ENGINE=tokudb;                     
ALTER TABLE `PathHierarchy` ENGINE=tokudb;            
ALTER TABLE `PathVisibility` ENGINE=tokudb;           
ALTER TABLE `Pool` ENGINE=tokudb;                     
ALTER TABLE `RestoreObject` ENGINE=tokudb;            
ALTER TABLE `Status` ENGINE=tokudb; 
ALTER TABLE `Storage` ENGINE=tokudb;     
ALTER TABLE `UnsavedFiles` ENGINE=tokudb;
ALTER TABLE `Version` ENGINE=tokudb;

or 

SET @DATABASE_NAME = 'name_of_your_db';

SELECT  CONCAT('ALTER TABLE ', table_name, ' ENGINE=tokudb;') AS sql_statements
FROM    information_schema.tables AS tb
WHERE   table_schema = @DATABASE_NAME
AND     `ENGINE` = 'MyISAM'
AND     `TABLE_TYPE` = 'BASE TABLE'
ORDER BY table_name DESC;

MYSQL to SSD

Whit TokuDB prevent sw&hw buffering: 

hdparam -W0 /dev/SDD

Mysql run on two SSD at server mpi1 and mpi2, asyncronouse mode mpi2 as master.

Innodb's can not be copyed palce to another, directory name must be same if so you can copy from system to other. So do mysqldump, setup new storage system and re-install databases.


Setup:

/etc/mysql/my.cnf datadir=<sdd dir> 

datadir         = /media/dbs/mysql

/etc/apparmor.d/abstractions/mysql 

  /var/lib/mysql/mysql.sock rw,
  /media/myjohn2/mysql/mysql.sock rw,

/etc/apparmor.d/usr.sbin.mysqld 

 /media/dbs/mysql/ r,
 /media/dbs/mysql/** rwk,

/etc/passwd 

mysql:x:129:139:MySQL Server,,,:/media/dbs/mysql:/bin/false

cd /media/dbs/mysql/
cp -Rp /var/lib/mysql ./
chown -R mysql:mysql /media/dbs/mysql/
/etc/init.d/apparmor restart
stop mysql
start mysql

Mysql Replication

http://dev.mysql.com/doc/refman/5.6/en/replication-howto.html http://www.howtoforge.com/how-to-set-up-database-replication-in-mysql-on-ubuntu-9.10-p2 http://dev.mysql.com/doc/refman/5.0/en/replication-howto-masterbaseconfig.html http://www.howtoforge.com/mysql_database_replication

master my.cnf 

server-id               = 1
log_bin                 = /var/log/mysql/mysql-bin.log
expire_logs_days        = 10
max_binlog_size         = 100M
binlog_do_db            = bacula, mediawiki, kurrola

rights for slave to replicate: 

GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'%' IDENTIFIED BY 'slave_password';
FLUSH PRIVILEGES;
quit;

LOCK & UNLOCK DATABASE FOR COPYING: 

USE wanted; ### IF ONLY ONE
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;

....COPY DATABASE 

UNLOCK TABLES;
quit;

slave my.cnf 

server-id               =2
master-connect-retry    =60
#replicate-do-db         =bacula, mediawiki, kurrola

where to replicate and used user , password: 

STOP SLAVE;
RESET SLAVE;
UNLOCK TABLES;
CHANGE MASTER TO MASTER_HOST='192.168.0.100', MASTER_USER='slave_user', MASTER_PASSWORD='slave_password', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=19467;

I have also found that I needed to UNLOCK TABLES on the slave before starting the slave

In trouble do not forget: 

 mysql_install_db  -  can fix missing files sometimes

 nano /etc/apparmor.d/local/usr.sbin.mysqld  -  add rights to access new locations for mysql
 service apparmor reload

TUNE MYSQL

./tuning-primer.sh ./mysqltuner.pl

Emergency Repair

GPG

gedit ~/.gnupg/gpg.conf 

keyserver http://keyserver.ubuntu.com

run: 

apt-get update 2> /tmp/keymissing; for key in $(grep "NO_PUBKEY" /tmp/keymissing |sed "s/.*NO_PUBKEY //"); do echo -e "\nProcessing key: $key"; sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys $key ; done

HARD DISK'S whit bad blocks

repair bad blokcs: 

sudo badblocks /dev/sdc > /tmp/iambad
sudo fsck -l /tmp/iambad /dev/sdc

DPKG , packages

http://www.thepowerbase.com/2012/04/how-to-fix-broken-packages-in-ubuntu-or-debian/

Copy Installation: 

dpkg --get-selections > mypackages.txt
sudo dpkg --set-selections < mypackages.txt
sudo apt-get dselect-upgrade

or

Repair broken: 

debsums -s -a 2> /tmp/broken.log
sed -n 's/^.*\(checksum mismatch\|changed file\) \([^ ]*\) file.*$/\1/p;s/^.*t open \([^ ]*\) file.*$/\1/p' < /tmp/broken.log |  sort -u > /tmp/broken.pkgs
cd /var/cache/apt/archives
rm  `cat /tmp/broken.pkgs`
apt-get --reinstall install `cat /tmp/broken.pkgs`

or

Try organize dpkg: https://help.ubuntu.com/community/PackageManagerTroubleshootingProcedure 

ubuntu-support-status
sudo grep -R proxy /etc/apt/*
grep proxy  /etc/environment
echo $http_proxy
echo $ftp_proxy
grep proxy /etc/bash.bashrc
grep proxy ~/.bashrc
cat /etc/apt/apt.conf
sudo fuser -vvv /var/lib/dpkg/lock
cat /etc/lsb-release
uname -a
sudo rm /var/lib/apt/lists/lock 
sudo rm /var/lib/dpkg/lock
sudo cp -arf /var/lib/dpkg /var/lib/dpkg.backup
sudo mv /var/lib/dpkg/status /var/lib/dpkg/status-bad
sudo cp /var/lib/dpkg/status-old /var/lib/dpkg/status  ||  sudo cp /var/backups/apt.extended_states.0 /var/lib/dpkg/status
sudo mv /var/lib/dpkg/available /var/lib/dpkg/available-bad
sudo cp /var/lib/dpkg/available-old /var/lib/dpkg/available
sudo rm -rf /var/lib/dpkg/updates/*
sudo rm -rf /var/lib/apt/lists
sudo rm /var/cache/apt/*.bin
sudo mkdir /var/lib/apt/lists
sudo mkdir /var/lib/apt/lists/partial
LANG=C;sudo apt-get clean
LANG=C;sudo apt-get autoclean
LANG=C;sudo apt-get --purge autoremove
LANG=C;sudo apt-get --fix-missing update -o APT::Cache-Limit=100000000
sudo dpkg --configure -a
sudo dpkg --clear-avail
LANG=C;sudo apt-get -f install
LANG=C;sudo apt-get --fix-missing install
LANG=C;sudo apt-get update -o APT::Cache-Limit=100000000 && sudo apt-get dist-upgrade
find /etc/apt -name '*.list' -exec bash -c 'echo -e "\n$1\n"; cat -n "$1"' _ '{}' \;

Dpkg Packages re-install

If lot's debency, md5sum, pacpage not found,... 

dpkg --get-selections > mypackages.txt

remove unwanted packages from *.txt so it looks list of packages to reinstall

Download all packages.

download: 

#!/bin/bash
filecontent=( `cat $1 `)
echo $1
cd /var/cache/apt/archives
 for t in "${filecontent[@]}"
   do
    apt-get download $t
 done
echo "Download end!"

Use dpkg to install all packages.

install: 

#!/bin/bash
filecontent=( `cat $1 `)
echo $1
cd /var/cache/apt/archives
 for t in "${filecontent[@]}"
   do
    dpkg -i $t*.deb
 done
echo "Install end!"

Bacula

RESTORE whitout DATABASE

bscan -u bacula -n bacula -P bacula73 -s -S -V ST3POOL-0125 /media/KurrolaStorage3

bscan -u bacula -n <DB username> -P <DB password> -s -S -V <backup tape/file> <tabe/file directory>

/opt/bacula/bin/bscan -c /opt/bacula/etc/bacula-sd.conf -u bacula -n bacula -P bacula73 -h 192.168.0.41 -t 3388 -s -v -V ST4POOL-0002\|ST4POOL-0003\|ST4POOL-0004\|ST4POOL-0005 /media/joni/ST4

/opt/bacula/bin/bscan -b test.bsr -c /opt/bacula/etc/bacula-sd.conf -u bacula -n bacula -P bacula73 -h 192.168.0.41 -t 3388 -s -v -V /media/joni/ST4

MYSQL

backup bacula DB's: 

 mysqldump -u steve -phuhaa bacula > bacula.sql

 mysqldump -u <DB's user> -p<DB's password> bacula > <backup file name>

Restore: 

 mysql   -u yourname -phuhaa bacula < bacula.sql

 mysql -u <DB's user> -p<DB's password> bacula < <backup file name>

Skip error; 

stop slave;
SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1;
start slave;
SHOW SLAVE STATUS\G;

Bacula repair: 

dbcheck -bvf /tmp bacula bacula <password>
sudo dbcheck -bvf /tmp bacula bacula bacula73 192.168.0.41 3388

MyIsam repair: 

mysqlcheck -u root -paptk#7315 -r drupal


Ring database: 

M1:
STOP SLAVE;
RESET SLAVE;
USE wanted; ### IF ONLY ONE
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;

M2:

rsync -  M1:/var/lib/mysql/ M2:/var/log/mysql/
start mysql

STOP SLAVE;
RESET SLAVE; 
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;
CHANGE MASTER TO MASTER_HOST='192.168.0.M1', MASTER_USER='slave_user', MASTER_PASSWORD='slave_password', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=19467;  
START SLAVE;
UNLOCK TABLES;

M1:

CHANGE MASTER TO MASTER_HOST='192.168.0.M2', MASTER_USER='slave_user', MASTER_PASSWORD='slave_password', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=19467;  
START SLAVE;
UNLOCK TABLES;
SHOW SLAVE STATUS\G;

M2;

SHOW SLAVE STATUS\G;

Status File Creation

http://www.solengtech.com/solengtech/opensource/howto-repair-ubuntu-dpkg-status-file

HowTo: Repair Ubuntu dpkg status file My Ubuntu 10.04.02 installs starting losing /var/lib/dpkg/status, and you can't update without it... Thanks to capink, but I ended up re-installing anyway so unfortunately losing this file you are near the end of the line.

http://ubuntuforums.org/archive/index.php/t-474587.html Using the script below will generate a new status file in your current directory (usually your home dir). Use this file in place of your status file after backing it up:

  1. !/bin/bash

get_control_info () { for i in /var/lib/apt/lists/*_Packages do sed '/Package: '"$1"'$/,/^$/!d' $i done }

for i in /var/lib/dpkg/info/*.list do package_name=`basename $i | sed 's/.list$//'` get_control_info $package_name >> status-new done

sed -i -e '/^Filename: .*/d' -e '/^MD5sum:/d' -e '/^SHA1:/d' -e '/^SHA256:/d' status-new sed -i '/^Package: /a\Status: install ok installed' status-new

TRUSTY

https://www.digitalocean.com/community/tutorials/how-to-create-a-calibre-ebook-server-on-ubuntu-14-04

TRUSTY MPI4YOU

New :
-load balancing / fail over for bind, postfix, dhcp, mysql ( mariadb-galera cluster ), apache

Basic Protection
4g CANTENNA & IPV6
SpaceNavigator
apacheDS LDAP & KERBEROS server
Bacula 7.0.5 From Sources Ubuntu Trusty 14.04 whit maridb (mysql)
mail in database - dbmail 3.1.1.7

Privacy:
ring.cx
https://www.torproject.org/

Android comp:
ORBOt, ORWEB https://guardianproject.info/apps/orbot https://guardianproject.info/releases/orweb-latest.apk
OSTEL https://ostel.co/about

Firefox Sync server

http://docs.services.mozilla.com/howtos/run-sync-1.5.html#howto-run-sync15

Alice

k = aiml.Kernel() k.learn("std-startup-old.xml") k.respond("load aiml y") k.setBotPredicate('name', 'Alice') k.setBotPredicate('master', 'Joni') k.setBotPredicate('email', 'alice@kurrola.dy.fi') k.setBotPredicate('domain', 'kurrola.dy.fi')

mailmessage = sys.stdin.read() msg = email.message_from_string(mailmessage) for part in msg.walk(): 

   # each part is a either non-multipart, or another multipart message
   # that contains further parts... Message is organized like a tree
   if part.get_content_type() == 'text/plain':
       textmsg = part.get_payload() # prints the raw text

print msg['From'] print msg['To'] print textmsg

response = k.respond( textmsg )

answer = "hi,\n\n" + response + "\n" + "\n\nAlice AI\n" + "Secretary of Office Kurrola\n\n" + "e-mail: alice@kurrola.dy.fi"

msgr = MIMEMultipart('alternative') msgr['Subject'] = "re:"+msg['Subject'] msgr['From'] = "alice@kurrola.dy.fi" msgr['To'] = msg['From'] part1 = MIMEText( answer , 'plain') part2 = MIMEText( mailmessage , 'email') msgr.attach(part1) msgr.attach(part2) s = smtplib.SMTP('2001:13b8:100:8353::4',25) s.sendmail("alice@kurrola.dy.fi", msg['From'], msgr.as_string()) s.quit()

apli

Project Managemtn http://taigaio.github.io/taiga-doc/dist/setup-production.html

RESERVED PORTS

Frontside:

80 HTTP 143 DBMAIL-IMAPTD 443 HTTPS

1024 AMAVIS 1025 DBMAIL-LMTPD

8080 Calibre Library 8088 Baculucum 3306 MariaDB Galera Cluster 3307 MariaDB for Bacula

BackStage:

9999 Sage

GAMES

http://www.msg.chem.iastate.edu/GAMESS/download/dist.source.shtml

LOGIN

https://wiki.ubuntu.com/LightDM#Show_Manual_Login_Box
sudo nano /etc/lightdm/lightdm.conf.d/50-manlogin.conf : 

[SeatDefaults]
greeter-show-manual-login=true

TRUSTY fail over load balance

haproxy cluster https://github.com/olafz/percona-clustercheck mariadb https://mariadb.com/blog/setup-mariadb-enterprise-cluster-part-3-setup-ha-proxy-load-balancer-read-and-write-pools

packages

sudo apt-get install cifs-utils\ nfs-4

apacheDS

Manual's:
http://directory.apache.org/studio/users-guide/ldap_browser/index.html

Install: 

http://mirror.netinch.com/pub/apache//directory/apacheds/dist/2.0.0-M17/apacheds-2.0.0-M17-64bit.bin

Install: 

apt-get install openjdk-7-jre
dpkg -i /media/ST1/software/apacheds-2.0.0-M17-64bit.deb

Start: 

  /opt/apacheds-2.0.0-M17/bin/apacheds console

DS-386 - OPEN Lapd

sudo apt-get install apache2-mpm-worker
apt-get intall 389-ds  ( all packages, servers )
/usr/sbin/setup-ds-admin

sudo /usr/sbin/setup-ds

DS-386 operating

sudo /usr/bin/389-console

OPEN Lapd

Orginal: http://hswong3i.net/blog/hswong3i/ldap-single-sign-webmin-ubuntu-12-04-howto

LDAP Single-Sign-On with Webmin on Ubuntu 12.04 HOWTO Submitted by hswong3i on Tue, 2013-01-08 00:01

In order to setup a LAMP cluster we usually need a way to share the master server uid/gid with other else member servers, for whatever NFS shared home directory, or running Apache2 + PHP5 in suexec style. Using LDAP + Webmin can simplify this Single-Sign-On (SSO) need in a handy way.

This HOWTO will guide you though installation of Webmin and OpenLDAP server, then use it as SSO between 2 server with nss-pam-ldapd. First of all let's fouce on making it works, and then enhence it with better security. Server Requirement

In this example let's assume we have 2 servers: dev6c1 and dev6c2, with domain name "localdomain" and IP information as below:

172.24.145.25 dev6c1.localdomain dev6c1 172.24.145.26 dev6c2.localdomain dev6c2

Where dev6c1 will be the master for OpenLDAP, where dev6c2 will be slave that using nss-pam-ldapd as SSO. Install Webmin Manually

Simply execute following commands:

cat >> /etc/apt/sources.list.d/virtualmin.list <<-EOF deb http://download.webmin.com/download/repository sarge contrib deb http://webmin.mirror.somersettechsolutions.co.uk/repository sarge contrib deb http://software.virtualmin.com/gpl/ubuntu/ virtualmin-precise main deb http://software.virtualmin.com/gpl/ubuntu/ virtualmin-universal main EOF

wget -q http://www.webmin.com/jcameron-key.asc -O- | sudo apt-key add - wget -q http://software.virtualmin.com/lib/RPM-GPG-KEY-virtualmin -O- | sudo apt-key add - wget -q http://software.virtualmin.com/lib/RPM-GPG-KEY-webmin -O- | sudo apt-key add -

aptitude -y install ubuntu-extras-keyring && \ aptitude update && \ aptitude -y full-upgrade && \ tasksel install openssh-server && \ tasksel install server && \ tasksel install mail-server && \ aptitude -y install usermin webmin && \ aptitude update && aptitude -y full-upgrade && aptitude autoclean && aptitude clean

Install OpenLDAP

Execute following commands at dev6c1 as OpenLDAP master:

aptitude -y install slapd libnet-ldap-perl ldap-utils

Don't care about the initial question, just feel free to answer them, as we will now reconfigure it with: 

   dpkg-reconfigure slapd
       Omit OpenLDAP server configuration? No
       DNS domain name: localdomain
       Organization name: localdomain
       Administrator password: (your password)
       Database backend to use: HDB
       Do you want the database to be removed when slapd is purged? No
       Move old database? Yes
       Allow LDAPv2 protocol? No

Configure LDAP Server for Webmin

Log into webmin with https://172.24.145.25:10000/, go to "Un-used Modules -> LDAP Server"

Go to "OpenLDAP Server Configuration", and update the password with "New administration password"

Go back and click on "Create Tree", create new DN as below: 

   For Unix user:
       Distinguished name: dc=Users,dc=localdomain
       Create example object under new DN? Unix user
   For Unix group:
       Distinguished name: dc=Groups,dc=localdomain
       Create example object under new DN? Unix group

Now verify the slapd setup by "Apply Configuration", "Stop Server", "Start Server"; also clock on "Browse Database" and should show sometime similar as below:

Install nss-pam-ldapd

Execute following commands on both machines as LDAP SSO client:

aptitude -y install libnet-ldap-perl libpam-ldapd libnss-ldapd ldap-utils nscd nslcd

Don't care about the initial question, just feel free to answer them.

Also disable nscd or else may generate unexpected result during LDAP lookup debug:

update-rc.d-insserv -f nscd remove /etc/init.d/nscd stop

Reconfigure all above packages as below: 

   dpkg-reconfigure libpam-ldapd
       PAM profiles to enable: Unix authentication, LDAP Authentication
   dpkg-reconfigure libnss-ldapd
       Name services to configure: group, passwd, shadow
   dpkg-reconfigure nslcd
       LDAP server URI: ldap://dev6c1/
       LDAP server search base: dc=localdomain
       LDAP authentication to use: none
       Use StartTLS? No

As long as example user and group created in above section, now we can check if LDAP lookup works:

getent passwd | grep example

Should show similar result as:

example:*:9999:9999:Example user:/home/example:/bin/sh

Don't forget to test with both machines. Configure LDAP Client for Webmin

Note that Webmin just support libpam-ldap/libnss-ldap officially, but not libpam-ldapd/libnss-ldapd, so the key point for setup LDAP Client at Webmin is to allow all else Webmin module able to access our LDAP server correctly (i.e. also benefit for Virtualmin setup with LDAP authentication backend).

Log into webmin with https://172.24.145.25:10000/, go to "Un-used Modules -> LDAP Client"

As it complaint configuration file not found (well, for sure, as we are using libpam-ldapd/libnss-ldapd), click on "Module Config" and update as: 

   LDAP client configuration file: /etc/ldap/ldap.conf
   PAM LDAP configuration file: Same as LDAP client file
   Root LDAP client password files: /etc/ldap/ldap.secret

Now it should looks fine. Before configure with Webmin interface let's key in some default value as reference to /etc/ldap/ldap.conf:

cat >> /etc/ldap/ldap.conf <<-EOF host dev6c1 rootbinddn cn=admin,dc=localdomain base dc=localdomain nss_base_passwd dc=Users,dc=localdomain nss_base_shadow dc=Users,dc=localdomain nss_base_group dc=Groups,dc=localdomain EOF

Access the Webmin interface again, and so click on "LDAP Server Configuration" and double confirm as below: 

   LDAP server hostnames: dev6c1
   Login for root user: cn=admin,dc=localdomain
   Password for root user: (your_secret_password)

Go back and click on "LDAP Search Bases" and double confirm as below: 

   Global search base: dc=localdomain
   Base for Unix users: dc=Users,dc=localdomain
   Base for Unix password: dc=Users,dc=localdomain
   Base for Unix groups: dc=Users,dc=localdomain

Go back and click on "Validate Configuration", if all works it should report as similar as below: 

   Don't forget to press "Refresh Modules" and check if Webmin able to detech all installed modules correctly.

Configure LDAP Users and Groups for Webmin and Usermin

Due to changed configuration file path for "LDAP client configuration file" on above section, plus Webmin hardcoded the "auth_ldap=/etc/ldap.conf" in /etc/webmin/ldap-useradmin/config (see http://sourceforge.net/p/webadmin/bugs/3714/?limit=10&page=1#9d30 for complete bug report since 2010), the "LDAP Users and Groups" tab will not show up even pressing "Refresh Modules", and also result as Usermin "Change Password" tab not functioning, too. So let's fix it ;-(

First of all update the auth_ldap value accordingly (since no GUI options for changing it):

sed -i 's/^auth_ldap=.*$/auth_ldap=\/etc\/ldap\/ldap.conf/g' /etc/webmin/ldap-useradmin/config

Next, go to "Webmin -> Usermin Configuration -> Usermin Module Configuration -> Change Password": 

   Change with: Use PAM or passwd command

Now pressing "Refresh Modules" once again, check if following tabs exists correctly: 

   System -> LDAP Client
   System -> LDAP Users and Groups
   Servers -> LDAP Server

Great! Webmin and Usermin should now working together with LDAP correctly ;-) Create LDAP Users by Webmin

Up to this part creating new LDAP user account would be very simple. Just go to "System -> LDAP Users and Groups" and click on "Add a new LDAP user" as show below:

Change LDAP Users Password by Usermin

Are you looking for a handy way for changing password by your users? It's show time for Usermin!

Log into webmin with https://172.24.145.25:20000/ with your user account, go to "Login -> Change Password" and now changing your LDAP user account password with GUI:

Next Step?

Oh if you are going use Virtualmin then now you can configure it as LDAP backend user storage. Virtualmin will therefore create all new user into your LDAP tree so can populate for any cluster setup usage.

If you are going to use NFS or other else cluster file system, using LDAP as SSO can now make other member machine share the same uid/gid with host, therefore Apache2 suexec can operate correctly. Reference 

   http://www.virtualmin.com/documentation/id%2Ccombining_virtualmin_and_ldap
   http://sourceforge.net/p/webadmin/bugs/3714/?limit=10&page=1#9d30
Retrieved from "http://oh5bzr.hamshack.info/wiki/index.php?title=Cluster_Kurrola&oldid=707"