Difference between revisions of "Cluster Kurrola"

From Radio Amteur Station OH5BZR - WIKI
Jump to: navigation, search
(→‎sageserver)
(→‎sageserver)
Line 490: Line 490:
  
 
apt-get install gnutsl
 
apt-get install gnutsl
−
 
  
  
Line 507: Line 506:
 
notebook startup script script /home/sageserver/startnotebook:
 
notebook startup script script /home/sageserver/startnotebook:
  
−
#!/bin/sh
+
#!/bin/sh
−
 
 
 
  ~/sage/sage -c "notebook(interface='localhost',  
 
  ~/sage/sage -c "notebook(interface='localhost',  
 
  directory='./sage_notebook.sagenb',
 
  directory='./sage_notebook.sagenb',

Revision as of 15:47, 22 March 2014

Installation

Cluster Kurrola desing targets: easy to manage ( webmin and LTSP are tested and compared ) , get parallel solvers work ( bind9,dhcp,nfs4 ) giganet whit IP4, stabilize system ( Twin Machine , "4G Cantenna mobile internet Antenna,.." server to work outside world at IPV6 and inside IPV4 / IPV6 (wwdial,hostapd,radvd,shoreline,shoreline6) basic servers ( mysql, postresql, apache2, mediawiki, mythweb, calibre ) and wlan services to workstations

To get parallel software like elmerfem and openfoam to work correctly you need full stack ( name services DNS and routing ) of network service and as learnt at practice at twin machine configuration. Twin machine configuration for DHCP ( both ipv4 and ipv6 ), BIND9 ( master and slave ). BIND9 namaserice is needed to have stable routing even connection to global internet breaks. DHCP is used to networkboot and network address managment. To get IPV6 and IPV4 filtered and routed correctly Shoreline and Shoreline6 are simple tool to configure firewall to Linux kernel. You need device level softaware like wvdial to get 4G modem/router to talk whit server routing and brdegeing software as well hostapd to open wlan services. To manage configuration of servers as well workstations webmin is good tool. To get network working well whit software you need common usernamre as well password base to get communication work between machines and services like NFS4 and rsh-redone.

For post prosessing you need Paraview as well support for 3d mouse. That's possible by building Praview 4.0 from sources whit vrpn and hidapi.

Cluster Kurrola ( MPI4YOU ) base parameters: 

/mpi3 is directory shared via NFS4 to all nodes
/mpi3/B3 Build Script's for version 3
/mpi3/S3 Source Code version 3
/mpi3/C3 Code Verion 3

Each node has Ubuntu 12.04 LTS ( Long Term Suport version, no need to change OS for 7 years )

Connections

NetworkConfiguration.jpg

Installing NetworkManager is simple way to get all right for single machine network. I got big supprise when I understood how complex it come to setup STABILE beawulf parallel network to home, you need to setup all network components.


Ubuntu 12.04 have much automation at protocol stack, it finds devices when connected ([udev]) and most cases setups hardware drivers and creates device like eth and ttyUSB. At Cluster Kurrola Installation NetworkManager is totaly removed due instability and some server software incompatibility issues (calibre,mpich3,rsh-redone). What have to do is to glue kernel IPV4 and IPV6 part's together whit devices and give correct parameter's to devices for operation.

Physical: 4G mobile network Saunanlahti & ELISA, HSPA+DC

Modem  : Huawei 398

Protocol stack:

DC-HSPA ppp 

   ipv4: rounting, bridge  Ubuntu kernel
         -shorewall, vwdial, isc-dhcp., bind9

   ipv6: System Six IPV6 tunnel over IPV4 network
         -shorewall6, isc-dhcp, bind9,aiccu


Server mpi2:


ppp0: wvdialD, wvdial, aiccu br0: eth0 shoreline shoreline6 aiccu raddvd, wlan0 (hostapd)

80 / http, drupal 443 / https, drupal 81 / nominatim-osm-MTK MAPs 8082 / owncloud 

/ bind9
/ dhcp
/ dhcp ipv6

3142 / apt-cacher-ng 8080 / calibre 8081 / internal drupal site 7000-7200 / mpi4you mpich3


hostapd

/etc/hostapd/hostapd.conf : 

interface=wlan0
bridge=br0
driver=nl80211
macaddr_acl=1
accept_mac_file=/etc/hostapd.accept
ssid=KURROAIR
hw_mode=g
channel=3 
wpa=2
wpa_passphrase=MyEncryptionKey
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP CCMP
wpa_ptk_rekey=600

wvdial

/etc/wvdial.conf : 

[Dialer connect]
Modem Type = Analog Modem
Modem = /dev/ttyUSB0
Phone = *99#
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
Init3 = AT^SYSCFGEX="0302",3fffffff,1,4,7fffffffffffffff,,
Init4 = AT+CGDCONT=1,"ip","internet",,0,0
ISDN = off
Ask Username = off
Username = no
Ask Password = off
Password = no
Baud = 9600
Stupid mode = on
Auto DNS = on
[Dialer signal]
Modem = /dev/ttyUSB1
Init1 = AT+CSQ
Init2 = AT^SYSINFO
Init3 = AT+CIND?

/etc/init.d/wvdialD : 

#!/bin/sh
NAME="wvdialD"
DESC="wvdial keep alive and startup"
test -x $DAEMON || exit 0
case "$1" in
start)
date >> /var/log/syslog
echo "[wvdialD] *** Starting Mobile Broadband Connection. wvdial up" >> /var/log/syslog
date >> /var/log/3g4gsession 
while true; do
/home/joni/webon >> /var/log/3g4gsession &
wvdial connect >> /var/log/3g4gsession
echo "RESTART" >> /var/log/3g4gsession 
date >> /var/log/3g4gsession
sleep 5 
done
date >> /var/log/syslog
/home/joni/weboff
echo "[wvdialD] Mobile Broadband Connection closed *******." >> /var/log/syslog
echo "STOP-DROP" > /var/log/3g4gsession
date >> /var/log/3g4gsession
;;
stop)
echo "[wvdialD] Stopping wvdial." >> /var/log/syslog  
killall wvdial
echo "[wvdialD] wvdial down." >> /var/log/syslog
echo "STOPED BY COMMAND" > /var/log/3g4gsession
date >> /var/log/3g4gsession
killall wvdialD
;;
status)
# Check network status with nmcli 
cat /var/log/3g4gsession
;;
*)
echo "[wvdialD] Keeps wvdial up"
echo $"Usage: $0 {start|stop|status}"
exit 1
esac
exit 0

IPV4,IPV6

/etc/network/interface : 

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
#Primary internet connection
auto ppp0
iface ppp0 inet wvdial
# The primary network interface
auto eth0
iface eth0 inet manual
manual wlan0
#address 192.168.0.40
iface wlan0 inet manual
auto br0
iface br0 inet static
network 192.168.0.0
address 192.168.0.XX
broadcast 192.168.0.255
netmask 255.255.255.0
bridge_ports eth0 wlan0
dns-nameservers 192.168.0.40 192.168.0.XX 
dns-search X.kurrola.dy.fi X.kurrola.dy.fi
dns-domain X.kurrola.dy.fi X.kurrola.dy.fi
gateway 192.168.0.254
#ipv6
iface eth0 inet6 static
address 2001:XXXX:100:363::40
netmask 64

ipv6 routing local wlan node: 

sysctl -w net.ipv6.conf.br0.disable_ipv6=0 ( echo1 > /proc/sys/net/ipv6/conf/br0/disable_ipv6 )
sudo ip -6 addr add 2001:14b8:100:363::41/64 dev br0

ipv6 routing at router whit aiccu and firewall: 

sysctl -w net.ipv6.conf.eth0.disable_ipv6=0
sudo ip -6 addr add 2001:14b8:100:363::40/64 dev eth0
sudo ip -6 route add 2001:14b8:100:363::40/64 dev eth0


[postfix]

https://help.ubuntu.com/community/Postfix

Orginal: http://christian.skala.me/blog/gmail-why-are-you-doing-this-to-me/#.Unt2oEOJRwY

Forcing IPv4 when sending to Gmail

If we want to force postfix to use IPv4 instead IPv6 when sending to Gmail, we need to add a line in the main configuration file of postfix like so 1 2

$ sudo nano /etc/postfix/main.cf transport_maps = hash:/etc/postfix/transport

Let’s create this new file and add a new entry in the transport table. Basically, we fore mail for gmail.com to use smtp-ipv4 (which we have to define later, see below) 1 2

$ sudo nano /etc/postfix/transport gmail.com smtp-ipv4:

Here comes the key part. We create a new rule for our new entry and define to use IPv4 protocol explicitly when this rule applies: 1 2 3

$ sudo nano /etc/postfix/master.cf smtp-ipv4 unix .. .. .. .. smtp 

-o inet_protocols=ipv4

We need to run the postmap command after the change and reload postfix 1 2

$ sudo postmap /etc/postfix/transport $ sudo postfix reload


Ltps

ACPI - SUSPEND - HIBERNATE START

/etc/udev/rules.d/

Install: 

apt-get install usb-modeswitch

http://www.draisberghof.de/usb_modeswitch/#install http://www.siptune.net/tiki-index.php?page=udev+rules+mokkuloille http://sysadminnotebook.blogspot.fi/2012/03/vodafone-k5005-huawei-e389-4g-modem-on.html

/lib/udev/rules.d/40-usb_modeswitch.rules : 

# Huawei 398 oma
ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="1506", RUN+="usb_modeswitch '%b/%k'"

/etc/usb_modeswitch.d : 

# Huawei E398
TargetVendor= 0x12d1
TargetProduct= 0x1506
MessageContent="55534243123456780000000000000011062000000100000000000000000000"


usb_modeswitch -R -v 12d1 -p 1505


New testing:

echo "12d1 1506" > /sys/bus/usb-serial/drivers/option1/new_id echo -e "AT^NDISDUP=1,1,\"online.telia.se\"\r" > /dev/ttyUSB0 dhclient wwan0

rsh-redone

apt-get install rsh-redone-client rsh-redone-server

/etc/hosts.equiv

  1. command access to your system .

+ + ~


/etc/inetd.conf

  1. BSD: Shell, login, exec and talk are BSD protocols.

shell stream tcp nowait root /usr/sbin/tcpd /usr/sbin/in.rshd login stream tcp nowait root /usr/sbin/tcpd /usr/sbin/in.rlogind


/home/joni/.rhosts 

kalle
mika
hei
paakone
192.168.0.77
192.168.0.78
192.168.0.132
192.168.0.233

/etc/pam.d/rsh 

#
# The PAM configuration file for the rsh (Remote Shell) service
#
# Due to limitations in the rsh protocol, modules depending on the conversation
# function to work cannot be used.  This includes authentication modules such
# as pam_unix.so.
#auth   required        pam_rhosts.so
auth    sufficient        pam_rhosts.so
#auth   required        pam_nologin.so
#auth   required        pam_env.so
#account        required        pam_unix.so
#session        required        pam_unix.so
auth    sufficient        pam_rhosts.so

Local Time keyboard

https://help.ubuntu.com/community/Howto%3A%20Custom%20keyboard%20layout%20definitions 

sudo apt-get install loacles kde-l10n-fi language-selector-gnome language-selector-common language-pack-gnome-fi-base language-pack-gnome-en-base language-pack-gnome-de-base language-pack-gnome-fr-base language-pack-fi-base language-pack-de-base language-pack-fr-base language-pack-en-base language-pack-fi language-pack-fr language-pack-de language-pack-en language-pack-gnome-de language-pack-gnome-fr language-pack-gnome-en language-pack-gnome-fi language-pack-en-base language-pack-fi-base language-pack-de-base language-pack-fr-base firefox-locale-fi thunderbird-locale-fi voikko-fi 
#timezone
dpkg-reconfigure tzdata
#local languages


Files:

/etc/default/locale: 

LANG=fi_FI.UTF-8

/etc/default/keyboard 
/etc/X11/xorg.conf
udevadm trigger --subsystem-match=input --action=change

THEN: 

 locale-gen purge

Debian not unbuntu: 


dpkg-reconfigure localeconf
sudo dpkg-reconfigure console-data
sudo dpkg-reconfigure console-setup



sudo dpkg-reconfigure keyboard-configuration - work's after reboot
sudo dpkg-reconfigure kdb

setxkbmap fi - change keyboard command line

Monitoring

sudo apt-get install lm-sensors gkrellmd
sudo sensors dedect
sudo service module-init-tools start
sudo nano /etc/gkrell*.conf
sudo /etc/init.d/gkrellmd start

Nvidia most recent

http://www.nvidia.com/download/driverResults.aspx/69378/en-us 

wget *

update-grub /dev/sda
update-grub /dev/sdb

MediaWiki

Running MediaWiki on Ubuntu [1]
PDF support [2]
Latex support sudo apt-get install ocml texlive and changes [3]

g++-4.7

sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install g++-4.7

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-4.6
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.7 40 --slave /usr/bin/g++ g++ /usr/bin/g++-4.7
sudo update-alternatives --config gcc


IPV6

Sixxs How To[4]
Ipv6 How to[5]
http://www.shorewall.net/IPv6Support.html
http://tldp.org/HOWTO/Linux+IPv6-HOWTO/x1083.html
http://www.shorewall.net/6to4.htm

ip -6 ro add 2001:14b8:100:363::/64 dev eth0
ip -6 route add default via 2001:14b8:100:363::40 dev eth0

File:Example.jpg

Space Navigator

See vrpn how to get it work whit Paraview.

http://docs.salome-platform.org/salome_4_1_5/visu/user/navigation_in_gauss_viewer_page.html http://code.google.com/p/liquid-galaxy/wiki/LinuxSpaceNavigator

ODPDS

SOURCES:
ODPDS wiki
[Orginal J-P's wiki https://sites.google.com/site/jpsdatareviewstheboy007/kindle-new-kindle-3/jetbook-whit-calibre-and-calibre2odpds]

Source Under construction: 

JetBook whit Calibre and calibre2odpds Installations note's

Download & install: Ubuntu 12.4 LTS Calibre calibre2odpds - manual version apache server - run: sudo apt-get install apache2 JAVA - run: sudo apt-get install openjdk-7-jre icedtea-7-plugin

Extra: Ectaco instructions to create by hand library

Directory structure of example: /home/calibre/CalibreBooks - calibre databse directory /var/www/odpds - link to /home/calibre/CalibreBooks, run: sudo ln -s /home/calibre/CalibreBooks /var/www/odpdss


Calibre2odpds configuration file is located /home/.calibre2opds/default.profile.xml just manually edit it:

gedit /home/joni/.calibre2opds/default.profile.xml

LINE: <entry key="UrlBase">http://yourip/</entry>


run: ./rungui.sh

Hard Disk

USB 3.0 - Sandberg Docking Station 705 Mb/s 

WD20EARX 52-122 Mb/s average 89 Mb/s
ST3 WD20EZRX 74-140 Mb/s average 120 Mb/s


SATA 2 

 sdb WD10EFRX 68-152 Mb/s average 59 Mb/s
myjohn2 ST 3500AS 42-91 Mb/s average 66 Mb/s
myjohn3 WD6402A 52-165 Mb/s 92 Mb/s
var/lib/mysql 268-278 Mb/s 273 Mb/s

MythTV

http://wiki.team-mediaportal.com/1_MEDIAPORTAL_1/11_Preparing_Your_System/01_Supported_TV_Cards http://parker1.co.uk/mythtv_freesat.php http://www.lyngsat.com/Astra-1KR-1L-1M-2C.html http://www.tbsdtv.com/products/tbs6928-dvb-s2-tv-tuner-ci-pcie-card.html

!!!No Trans bug solution https://github.com/MythTV/mythweb/commit/6e251346675db00a889fc922ab7f270c1d9b66c9

digikam

http://linuxg.net/how-to-install-digikam-3-3-0-on-ubuntu-linux-mint-debian-and-derivates/

precise dose not have:

OpenCV enough fresh
http://docs.opencv.org/doc/tutorials/introduction/linux_install/linux_install.html#linux-installation

CAN NOT WORK 12.4 due depencyes,...easyly!

sageserver

http://wiki.sagemath.org/SageServer#Other_Instructions above should work out whit below changes

You need extra user's sageserver, sage0 .. sage9 You need extra groups sageserver, sageuser Chesk that user sageserver and sage0..sage9 can access /mpi3/S3/sage-6.1 (sage source directory)


apt-get install gnutsl


bash script for userspace at server: 

sudo -u sageserver -i ssh-keygen -t dsa
for i in {0..9}
do
mkdir /home/sage$i/.ssh
cat /home/sageserver/.ssh/id_dsa.pub > /home/sage$i/.ssh/authorized_keys
chown -R sage$i:sageserver /home/sage$i/.ssh
chmod 700 -R /home/sage$i/.ssh
chmod 755 -R /home/sage$i/.ssh/authorized_keys
done

notebook startup script script /home/sageserver/startnotebook: 

#!/bin/sh
~/sage/sage -c "notebook(interface='localhost', 
directory='./sage_notebook.sagenb',
automatic_login=false,
secure=false,
port=9999, 
accounts=false, 
timeout=3600, 
server_pool=['sage%d@localhost'%i for i in range(10)], 
ulimit='-u 100 -t 36000 -v 500000')"


upstart for server: 

#!/bin/sh
NAME="sageserverD"
DESC="sageserver keep alive and startup"
test -x $DAEMON || exit 0
case "$1" in

start) 

          date >> /var/log/syslog

echo "[sageserverD] *** Starting SageServer" >> /var/log/syslog 

           exec su -l sageserver /home/sageserver/startnotebook > /dev/null
    	;;

stop) 

      echo "[sageserverD] Stopping sageserverD." >> /var/log/syslog  
       killall sage

;; status) 

     	;;

*) echo "SageServer" 

      echo $"Usage: $0 {start|stop|status}"
      exit 1
esac
exit 0

Running Server

MPI4YOU - elmer, netgen and openfoam whit mpich3
HYDRA
GIS - Installation openstreetmap, gdal, ogr2ogr, psql2osm
Library - Installation calibre, mendelay, recol
GPG - Finish

Server start & stop

/etc/init.d/networking restart
/etc/init.d/snmpd restart
service bind9 restart
sudo service isc-dhcp-server start
/etc/init.d/apache2 restart


SSL Update & create =

SERVER CERTIFICATE UPDATE at SSL DIRECTORY 

sudo openssl genrsa -out privkey.pem 4096
sudo openssl req -new -key privkey.pem -out cert.csr
  GO CACERT
sudo nano kurrola.dy.fi.crt

sudo cp privkey.pem kurrola.dy.fi.insecure
wget http://www.cacert.org/certs/root.crt

CLIENT SERTIFICATE GENERATION

BACULA

http://wiki.bacula.org/doku.php?id=faq 

What to backup:
/etc/bacula - you need configuration
/var/lib/bacula/bacula.sql - you need backup from database

Crete index at MySql database: 

   File.PathId
   File.FilenameId
   Job.FileSetId
   Job.ClientId

MariaDB - under test

Due pronlem's whit mysql started testong MariaDB

https://downloads.mariadb.org/mariadb/repositories/#mirror=netinch&distro=Ubuntu&distro_release=precise&version=10.0

<a href="http://mariadb.org"> 

 <img src="http://badges.mariadb.org/mariadb-badge-125x50.png"
   width="125" height="50" border="0" alt="Powered by MariaDB"
   title="Powered by MariaDB" />

</a> 

SELECT concat('ALTER TABLE `',TABLE_NAME,'` ENGINE=tokudb;')
FROM Information_schema.TABLES 
WHERE ENGINE != 'tokudb' AND TABLE_TYPE='BASE TABLE' 
AND TABLE_SCHEMA='bacula'

ALTER TABLE `BaseFiles` ENGINE=tokudb; 

ALTER TABLE `Client` ENGINE=tokudb;                  
ALTER TABLE `Counters` ENGINE=tokudb;                 
ALTER TABLE `Device` ENGINE=tokudb;                   
ALTER TABLE `File` ENGINE=tokudb;                     
ALTER TABLE `FileSet` ENGINE=tokudb;                  
ALTER TABLE `Filename` ENGINE=tokudb;                 
ALTER TABLE `Job` ENGINE=tokudb;                      
ALTER TABLE `JobHisto` ENGINE=tokudb;                
ALTER TABLE `JobMedia` ENGINE=tokudb;                 
ALTER TABLE `Location` ENGINE=tokudb;                 
ALTER TABLE `LocationLog` ENGINE=tokudb;              
ALTER TABLE `Log` ENGINE=tokudb;                      
ALTER TABLE `Media` ENGINE=tokudb;                    
ALTER TABLE `MediaType` ENGINE=tokudb;                
ALTER TABLE `Path` ENGINE=tokudb;                     
ALTER TABLE `PathHierarchy` ENGINE=tokudb;            
ALTER TABLE `PathVisibility` ENGINE=tokudb;           
ALTER TABLE `Pool` ENGINE=tokudb;                     
ALTER TABLE `RestoreObject` ENGINE=tokudb;            
ALTER TABLE `Status` ENGINE=tokudb; 
ALTER TABLE `Storage` ENGINE=tokudb;     
ALTER TABLE `UnsavedFiles` ENGINE=tokudb;
ALTER TABLE `Version` ENGINE=tokudb;

or 

SET @DATABASE_NAME = 'name_of_your_db';

SELECT  CONCAT('ALTER TABLE ', table_name, ' ENGINE=tokudb;') AS sql_statements
FROM    information_schema.tables AS tb
WHERE   table_schema = @DATABASE_NAME
AND     `ENGINE` = 'MyISAM'
AND     `TABLE_TYPE` = 'BASE TABLE'
ORDER BY table_name DESC;

MYSQL to SSD

Whit TokuDB prevent sw&hw buffering: 

hdparam -W0 /dev/SDD

Mysql run on two SSD at server mpi1 and mpi2, asyncronouse mode mpi2 as master.

Innodb's can not be copyed palce to another, directory name must be same if so you can copy from system to other. So do mysqldump, setup new storage system and re-install databases.


Setup:

/etc/mysql/my.cnf datadir=<sdd dir> 

datadir         = /media/dbs/mysql

/etc/apparmor.d/abstractions/mysql 

  /var/lib/mysql/mysql.sock rw,
  /media/myjohn2/mysql/mysql.sock rw,

/etc/apparmor.d/usr.sbin.mysqld 

 /media/dbs/mysql/ r,
 /media/dbs/mysql/** rwk,

/etc/passwd 

mysql:x:129:139:MySQL Server,,,:/media/dbs/mysql:/bin/false

cd /media/dbs/mysql/
cp -Rp /var/lib/mysql ./
chown -R mysql:mysql /media/dbs/mysql/
/etc/init.d/apparmor restart
stop mysql
start mysql

Mysql Replication

http://dev.mysql.com/doc/refman/5.6/en/replication-howto.html http://www.howtoforge.com/how-to-set-up-database-replication-in-mysql-on-ubuntu-9.10-p2 http://dev.mysql.com/doc/refman/5.0/en/replication-howto-masterbaseconfig.html http://www.howtoforge.com/mysql_database_replication

master my.cnf 

server-id               = 1
log_bin                 = /var/log/mysql/mysql-bin.log
expire_logs_days        = 10
max_binlog_size         = 100M
binlog_do_db            = bacula, mediawiki, kurrola

rights for slave to replicate: 

GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'%' IDENTIFIED BY 'slave_password';
FLUSH PRIVILEGES;
quit;

LOCK & UNLOCK DATABASE FOR COPYING: 

USE wanted; ### IF ONLY ONE
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;

....COPY DATABASE 

UNLOCK TABLES;
quit;

slave my.cnf 

server-id               =2
master-connect-retry    =60
#replicate-do-db         =bacula, mediawiki, kurrola

where to replicate and used user , password: 

STOP SLAVE;
RESET SLAVE;
UNLOCK TABLES;
CHANGE MASTER TO MASTER_HOST='192.168.0.100', MASTER_USER='slave_user', MASTER_PASSWORD='slave_password', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=19467;

I have also found that I needed to UNLOCK TABLES on the slave before starting the slave

In trouble do not forget: 

 mysql_install_db  -  can fix missing files sometimes

 nano /etc/apparmor.d/local/usr.sbin.mysqld  -  add rights to access new locations for mysql
 service apparmor reload

TUNE MYSQL

./tuning-primer.sh ./mysqltuner.pl

Emergency Repair

GPG

gedit ~/.gnupg/gpg.conf 

keyserver http://keyserver.ubuntu.com

run: 

apt-get update 2> /tmp/keymissing; for key in $(grep "NO_PUBKEY" /tmp/keymissing |sed "s/.*NO_PUBKEY //"); do echo -e "\nProcessing key: $key"; sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys $key ; done

HARD DISK'S whit bad blocks

repair bad blokcs: 

sudo badblocks /dev/sdc > /tmp/iambad
sudo fsck -l /tmp/iambad /dev/sdc

DPKG , packages

http://www.thepowerbase.com/2012/04/how-to-fix-broken-packages-in-ubuntu-or-debian/

Copy Installation: 

dpkg --get-selections > mypackages.txt
sudo dpkg --set-selections < mypackages.txt
sudo apt-get dselect-upgrade

or

Repair broken: 

debsums -s -a 2> /tmp/broken.log
sed -n 's/^.*\(checksum mismatch\|changed file\) \([^ ]*\) file.*$/\1/p;s/^.*t open \([^ ]*\) file.*$/\1/p' < /tmp/broken.log |  sort -u > /tmp/broken.pkgs
cd /var/cache/apt/archives
rm  `cat /tmp/broken.pkgs`
apt-get --reinstall install `cat /tmp/broken.pkgs`

or

Try organize dpkg: https://help.ubuntu.com/community/PackageManagerTroubleshootingProcedure 

ubuntu-support-status
sudo grep -R proxy /etc/apt/*
grep proxy  /etc/environment
echo $http_proxy
echo $ftp_proxy
grep proxy /etc/bash.bashrc
grep proxy ~/.bashrc
cat /etc/apt/apt.conf
sudo fuser -vvv /var/lib/dpkg/lock
cat /etc/lsb-release
uname -a
sudo rm /var/lib/apt/lists/lock 
sudo rm /var/lib/dpkg/lock
sudo cp -arf /var/lib/dpkg /var/lib/dpkg.backup
sudo mv /var/lib/dpkg/status /var/lib/dpkg/status-bad
sudo cp /var/lib/dpkg/status-old /var/lib/dpkg/status  ||  sudo cp /var/backups/apt.extended_states.0 /var/lib/dpkg/status
sudo mv /var/lib/dpkg/available /var/lib/dpkg/available-bad
sudo cp /var/lib/dpkg/available-old /var/lib/dpkg/available
sudo rm -rf /var/lib/dpkg/updates/*
sudo rm -rf /var/lib/apt/lists
sudo rm /var/cache/apt/*.bin
sudo mkdir /var/lib/apt/lists
sudo mkdir /var/lib/apt/lists/partial
LANG=C;sudo apt-get clean
LANG=C;sudo apt-get autoclean
LANG=C;sudo apt-get --purge autoremove
LANG=C;sudo apt-get --fix-missing update -o APT::Cache-Limit=100000000
sudo dpkg --configure -a
sudo dpkg --clear-avail
LANG=C;sudo apt-get -f install
LANG=C;sudo apt-get --fix-missing install
LANG=C;sudo apt-get update -o APT::Cache-Limit=100000000 && sudo apt-get dist-upgrade
find /etc/apt -name '*.list' -exec bash -c 'echo -e "\n$1\n"; cat -n "$1"' _ '{}' \;

Dpkg Packages re-install

If lot's debency, md5sum, pacpage not found,... 

dpkg --get-selections > mypackages.txt

remove unwanted packages from *.txt so it looks list of packages to reinstall

Download all packages.

download: 

#!/bin/bash
filecontent=( `cat $1 `)
echo $1
cd /var/cache/apt/archives
 for t in "${filecontent[@]}"
   do
    apt-get download $t
 done
echo "Download end!"

Use dpkg to install all packages.

install: 

#!/bin/bash
filecontent=( `cat $1 `)
echo $1
cd /var/cache/apt/archives
 for t in "${filecontent[@]}"
   do
    dpkg -i $t*.deb
 done
echo "Install end!"

Bacula

RESTORE whitout DATABASE

bscan -u bacula -n bacula -P bacula73 -s -S -V ST3POOL-0125 /media/KurrolaStorage3

bscan -u bacula -n <DB username> -P <DB password> -s -S -V <backup tape/file> <tabe/file directory>

MYSQL

backup bacula DB's: 

 mysqldump -u steve -phuhaa bacula > bacula.sql

 mysqldump -u <DB's user> -p<DB's password> bacula > <backup file name>

Restore: 

 mysql   -u yourname -phuhaa bacula < bacula.sql

 mysql -u <DB's user> -p<DB's password> bacula < <backup file name>

Skip error; 

stop slave;
SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1;
start slave;
SHOW SLAVE STATUS\G;

Bacula repair: 

dbcheck -bvf /tmp bacula bacula <password>

MyIsam repair: 

mysqlcheck -u root -paptk#7315 -r drupal


Ring database: 

M1:
STOP SLAVE;
RESET SLAVE;
USE wanted; ### IF ONLY ONE
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;

M2:

rsync -  M1:/var/lib/mysql/ M2:/var/log/mysql/
start mysql

STOP SLAVE;
RESET SLAVE; 
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;
CHANGE MASTER TO MASTER_HOST='192.168.0.M1', MASTER_USER='slave_user', MASTER_PASSWORD='slave_password', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=19467;  
START SLAVE;
UNLOCK TABLES;

M1:

CHANGE MASTER TO MASTER_HOST='192.168.0.M2', MASTER_USER='slave_user', MASTER_PASSWORD='slave_password', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=19467;  
START SLAVE;
UNLOCK TABLES;
SHOW SLAVE STATUS\G;

M2;

SHOW SLAVE STATUS\G;

Status File Creation

http://www.solengtech.com/solengtech/opensource/howto-repair-ubuntu-dpkg-status-file

HowTo: Repair Ubuntu dpkg status file My Ubuntu 10.04.02 installs starting losing /var/lib/dpkg/status, and you can't update without it... Thanks to capink, but I ended up re-installing anyway so unfortunately losing this file you are near the end of the line.

http://ubuntuforums.org/archive/index.php/t-474587.html Using the script below will generate a new status file in your current directory (usually your home dir). Use this file in place of your status file after backing it up:

  1. !/bin/bash

get_control_info () { for i in /var/lib/apt/lists/*_Packages do sed '/Package: '"$1"'$/,/^$/!d' $i done }

for i in /var/lib/dpkg/info/*.list do package_name=`basename $i | sed 's/.list$//'` get_control_info $package_name >> status-new done

sed -i -e '/^Filename: .*/d' -e '/^MD5sum:/d' -e '/^SHA1:/d' -e '/^SHA256:/d' status-new sed -i '/^Package: /a\Status: install ok installed' status-new

Retrieved from "http://oh5bzr.hamshack.info/wiki/index.php?title=Cluster_Kurrola&oldid=250"