Archiving script refactorisation

Due to the main previous archiving script falling over itself I have started to write a new version with the help of #bash on freenode.

Conceptual things I have changed:

  • Using a whitelist of UUIDs instead of mountpoints  to determine if a drive is plugged in.
  • Keyfiles instead of password so an expect script is not required.
  • Tar instead of rsync, even with limitations rsync was playing havoc with asterisk

This version uses gnupg instead of truecrypt, but I have plans on seeing if truecrypt can mount without requiring any sort of expect script. Also the script doesn’t have incremental archiving implemented, but tar does support incremental archiving.

In short expect a different version to emerge soon.

#!/bin/bash

#UUID used for drives (seperate drives using " " )
drives="9a3ffd9e-ee84-4475-ab0f-60b98910eeef dd519d4f-a63b-4579-bcc7-0feb073ba030"
mntloc="/mnt/bkdrive"

#check if drive is active: fuser ...

#step one umount folder
echo "attempting to unmount $mntloc"
umount "$mntloc"

#finds first acceptable uuid
found="0"
for i in $drives; do
    #sets the dev location
    devloc="$(blkid -U $i)"
   #checks drive is found
    if [ -n "$devloc" ]; then
        echo "suitable drive found. (UUID=$i $devloc) ..now mounting.."
        found="1"
        break;
    fi
done
#if drive not found. exit
if [[ ${found} -eq "0" ]]; then
    echo "Warning no suitable drive found!"
    exit 1
fi

#check mounting worked.
if (mount -t auto "$devloc" "$mntloc"); then
   echo "mounted.";
else
   echo "Failed to mount. Exiting."
   exit 1;
fi

#backup /ANAT/Common in a screen
file="$mntloc/ANATCommon_bk_$(date +%Y-%m-%d).tar.xz"

#screen -S Backup -d -m 
tar -cvJ /ANAT/Common | gpg -r [email protected] -e -o "${file}.gpg"
Posted in Articles | Leave a comment

Ubuntu vs Debian (sever side)

The situation

For a while now I have been considering changing from using Ubuntu server, whilst Ubuntu server does provide great support with updates.

Distribution upgrades are known to fail with Ubuntu and with a cycle of 2 distribution releases per year, it is not that feasible to be able to be upgrading by installing from scratch all the time.

Ubuntu does offer some other advantages such as a GUI  (gufw) for iptables which is not outdated and a pretty good community of support.

Why Debian?

The idea of using debian is centered around stability with updates. Distribution upgrades with Debian are known to work, distribution releases tend to have a much longer cycle than other linux distributions but at the same time it takes a lot longer too.

It might be argued that openSuse, fedora and arch linux are also known to work with a distribution upgrade. However, both fedora and arch linux tend to be more focused on cutting edge software than stable software and openSuse is no longer being supported by novell which may have a long term negative effect on the linux flavor as a whole.

By the way gentoo is out of the question, because hosting a server with the ability to compile is a bad idea from a security point of view.

So by a process of elimination of “core” linux flavors is the reason for choosing Debian.

How to migrate?

To migrate, I will have to use the backup server (this server only exists just incase the hardware on the existing server fails).

Firstly I will install all pre-existing and required software packages from the exisiting server.

Secondly I will take a dump of the /etc directory on the existing server, as the exisiting server is running ubuntu in theory the configurations should be similar enough to just dump the /etc folder into a debian system.

Finally I’ll transfer the data across.

Posted in Articles | 2 Comments

Embedding Videos in WordPress

Problem

Trying to add the embed code for something like YouTube or Vimeo gets stripped out by WordPress’s editor. On top of that, the standard embed option of leaving the URL to the website on a line of it’s own gets screwed up if you edit a published web page as the various text filters add the paragraph <p> </p> tags around the URL, meaning it’s no longer on the line on it’s own, and instead of being an embedded video it’s now

Fix v1 : Shortcode

If you simply want to embed videos from well known sites like YouTube, then you can use one of the plethora of WordPress plugins which use some sort of short code, like |YouTube#84631| to insert the embeded video.

Fix v2 : Allow the Embed Code

This process involves allowing the HTML tags used in the embed code into the HTML tab of the TinyMCE WordPress editor.

Step 1. Install the TinyMCE Valid Elements plugin.

Step 2. Add the following :

# embed
* allowfullscreen, allowscriptaccess, height, src, type, width

# iframe
* frameborder, height, marginheight, marginwidth, scrolling, src, width

# object
* embed, height, name, param, src, type, value, width

# param
* name, value

Step 3. Go to the video page and copy the embed code.

Step 4. Go to edit a page or post and do a [Shift] + Refresh [F5], this clears the browsers cache.

Step 5. Select the HTML tab of the page or post, scroll to the correct location and paste in the embed code.

Step 6. Preview the page and update or publish it if correct.

Note : When getting the embed code you may find the size of the embedded video is larger than the size of the theme’s column and you’ll have to use a smaller size.

Posted in Articles | Leave a comment

Link Lint (Broken HTTP links) using Parallel processing

This script is designed to check many different websites for broken links at once and uses a multi-process approach to speed up checking times.

#!/bin/bash
# This script was created by Michael Kubler
# On the 26th of May, 2010
# for ANAT, the Australian Network for Art and Technology
# Title : Link Lint website link checker cronjob
# Description : This should be run as a weekly cronjob checking the anat websites for invalid files or broken links.
# Crontab listing : 20 10 1 * * ~/.cron/LinkLint_site.sh
# Requires : LinkLint
# Optional : The LinkLint information as a site domain
DIR='/home/anat/linklint'
echo "This document is created by the LinkLint cronjob script on $HOST and contains a listing of all the errors on the listed websites" &gt; $DIR/1_all_errors.txt
EMAIL='[email protected]' # Where to send the email
SITES_ARRAY=(w00t.anat.org.au surfacetension.anat.org.au) #THE IMPORTANT BIT! A listing of the sites to check
SITE_DOMAIN='http://office.anat.org.au/linklint' # An address to access the other Link Lint info. Leave the trailing slash. If you don't want the information hosted by Apache to the world you could try including a network address, or local file address if run on your computer.
 
echo '- Running Linklint'
for ix in ${!SITES_ARRAY[*]}
do
 SITE=${SITES_ARRAY[$ix]}
 ((linklint -host $SITE -http -net -doc $DIR/$SITE/ -cache $DIR/$SITE/ /@) &amp;&gt; /dev/null) &amp;
 ## Above is the most important line. It spawns all the LinkLint processes. The &amp; at the end  causes it to be 
 echo Started linklint on $SITE
done
echo
echo '- Displaying number of still running LinkLint processes'
echo ' NB : This may take some time...'
grepfor='linklint'
## This section gives you something to look at when you are waiting for it to process ##
# Note if pgrep isn't working you can try the following to get a listing of the number of running processes.
# ps augx | grep /usr/bin/linklint | grep -v grep | wc -l
while [ `pgrep $grepfor | wc -l` != 0 ];
do
 echo -n `pgrep $grepfor | wc -l`
 echo -n ', '
 sleep 15
 ## Checks every 15 seconds to see how many link lint processes are still running. ##
done
echo '' # Used to add a new line
echo '- Done Link Linting'
## Collating error files
for ix in ${!SITES_ARRAY[*]}
do
 SITE=${SITES_ARRAY[$ix]}
 echo "#########  $SITE  #############" &gt;&gt; $DIR/1_all_errors.txt
 if [ -f $DIR/$SITE/error.txt ]
 then
 ## There's an error (or few), so display the listing of errors.
 echo "$SITE_DOMAIN/$SITE/index.html" &gt;&gt; $DIR/1_all_errors.txt
 cat $DIR/$SITE/error.txt &gt;&gt; $DIR/1_all_errors.txt
 fi
done
echo '- Done collating errors'
## Email a list of all the errors.
CURRENTDATE=$(date '+%d-%m-%Y') #The date (e.g 23-02-2008)
mail -s "[LinkLint] sites errors - $CURRENTDATE" $EMAIL &lt; $DIR/1_all_errors.txt
echo "- Done sending email to '$EMAIL' with the subject '[LinkLint] sites errors - $CURRENTDATE'"
echo "If any sites have errored you can check them individually by running :"
echo "&gt; linklint -host $SITE -http -net -doc $DIR/$SITE/ -cache $DIR/$SITE/ /@"
echo '- Script complete'
exit 0

Continue reading

Posted in Code | Leave a comment

Bash Prompt

For those that are regular Terminal, SSH, and/or Putty users, there will come a time when you want to change the Bash prompt.

Here is the new PS1 (Shell Prompt) which you can add into ~/.bashrc

if [ "$color_prompt" = yes ]; then<br />    PS1="\[$(tput setaf 4)\]\d [$(date +%H:%M)]\[$(tput setaf 2)\]\u@\h\[$(tput setaf 7)\]:\[$(tput setaf 5)\]\w\[$(tput setaf 7)\]$ \[$(tput sgr0)\]"<br />else<br />    PS1="${debian_chroot:+($debian_chroot)}\[\e[1;34m\][$(date +%H:%M)]\[\e[01;32m\]\u@\h\[\e[00m\]:\[\e[1;36m\]\w\[\e[00m\]\$ "<br />fi

Note : The code above is assuming you are using Ubuntu, or a similar Linux distro. Using tput may not work with some consoles.
Continue reading

Posted in Code | Leave a comment

Create Site Script

This is one of the most comprehensive bash scripts I’ve ever written.
Over 760 lines of code, but it allows an Ubuntu System Admin to create a new Website, Mailman mailing list, FTP account and/or  WordPress Blog by typing an email address, and pressing ‘y’ or ‘n’ a couple of times as they are answering some basic questions.

It will let you create the Apache config file with multiple top level domains (e.g anat.org.au anat.com anat.com.au, etc..).
It also does a decent, although not insane, amount of error checking.

Download it here : http://w00t.anat.org.au/files/scripts/create_site/create_site.sh

Or with the download link below

Download

Assumptions

This is important. I’ve made a number of sweeping assumptions when creating this script. DO NOT USE THIS SCRIPT UNLESS YOU HAVE READ THE INFORMATION BELOW!

  • Linux Distro : I assume you are using Ubuntu. If you aren’t, you’ll have to adjust this script yourself. It might still work, but advanced things like automated FTP account creation, will likely fail. It will also fail if Apache is configured differently on different distro’s.
  • Not a shared host : I’ve assumed that you are running this on a VPS (Virtual Private Server) and not a shared host.. Which usually won’t give you root access and has the apache web folders in different locations.
  • Sudo/Root Access : Some of the commands being run, such as changing Apache configs and restarting programs require root or sudo access.
  • FTP is a custom compiled version of PureFTPd : The default apt-get install of Pure-FTPd doesn’t include the ability to create virtual users, which is what most small time VPS admins want to be able to give their users. FTP access but not SSH.
  • Expects : Unfortunately with PureFTP you can’t send it the password on the command line or get it to pull it from a file. As such I had to use Expects to allow it to try and replace someone’s human hands having to type the FTP password. If you don’t have expects installed it’ll try and apt-get install it, if that fails you’ll have to type the ftp password in yourself (the script will warn you as such).
  • WordPress : The install of WordPress is whatever the latest on the site is. It assumes you can download the install file, that you have added the MySQL details, and that you are installing a single isolated setup of WordPress and not using an existing WordPress Multi-Site config. If you want this domain to share another WordPress’s multi-site, then you’ll have to configure that yourself.
  • MySQL user/pass : As mentioned above
  • Mailman : You are using Mailman for mailing list software. If you are using something else or haven’t installed it before then you’ll want to say ‘n’ to this.
  • Exim4 : If you aren’t using Exim4 for your SMTP server then you won’t get all the benefits out of the Mailman config.

[Please note : This script is not 100% finished. It currently doesn’t edit the Mailman mailing list’s Exim4 config for you]

Command line format

sudo ./create_site $DOMAIN_WITHOUT_TLD

e.g
sudo ./create_site anat
# Note that the $DOMAIN is the main name without the top level domain (it will ask you if you want to change from the default .com TLD, and you can have a whole list of space separated top level domains as alias’s.
Continue reading

Posted in Code, Downloads, OMFG | Leave a comment

ANAT Apache folders convention

I’ll start by getting right to the point. This is how I structure the folders for Apache.

Assumptions :

  • Using Ubuntu (or a similar distro)
  • Single main user, but can have multiple (virtual) FTP users
  • You have full root access (in this case using Sudo)
  • You aren’t a security nazi

The easiest way of explaining this is to give an example.

Example : anat.org.au

  • /var/www/anat/
  • /var/www/anat/www/ — Primary site root
  • /var/www/anat/list/ — Used as a placeholder for mailing lists
  • /var/www/anat/staging/www/ — Used as a testing site for new concepts.
  • /var/www/anat/stats/ — Optionally used to contain webalizer or other stats (usually of the main www site). Can also be used to re-direct to Google.com/analytics, or even a Cacti, MRTG, or Naigos style SNMP graphing app.

As you can see the format is :

/var/www/$DOMAIN/$SUBDOMAIN/$SUB_SUBDOMAIN/

Continue reading

Posted in Articles, Need to know | Leave a comment

MySQL Backup Script with User Permisions

This is a usefull script that performs a standard MySQL database dump but includes the functionality to backup users and user permisions in addition to the datain the database.

#!/bin/bash
# Copyright (c) 2005 nixCraft project 
# This script is licensed under GNU GPL version 2.0 or above
# Author Vivek Gite 
# Addapted to grab multiple databases and users by Dale Caon and output to file rather than another database
# ------------------------------------------------------------
# SETME First - local mysql user/pass
_lusr="root"
_lpass='*****'
_lhost="localhost"
 
_folder='/backups/mysql'
_file='mysql.sql'
#### Edit below at own risk! #####
 
# Set up folder and file to hold user and data dump
if [ -d $_folder ]
then
        #Don't do anything
        echo -n '' &gt; "$_folder/$_file"
else
        mkdir -p "$_folder/"
fi;
 
cd $_folder
 
# Make sure you can connect to local db server
mysqladmin -u "$_lusr" -p"$_lpass" -h "$_lhost"  ping &amp;&gt;/dev/null || { echo "Error: Mysql server is not online or incorrect values set for _lusr, _lpass, and _lhost"; exit 2; }
 
##### Step 1: Okay build .sql file with db dump ####
for i in $(mysql -u "$_lusr" -p"$_lpass" -h "$_lhost" -e'show databases' | sed '1,1d' |  grep -v 'information_schema' | grep -v 'mysql'); do
        echo "create database IF NOT EXISTS $i; " &gt;&gt; "$_file"
done;
 
#### Step 2: Build mysql query to grab all privs and user@host combo for given db_username ####
mysql -u "$_lusr" -p"$_lpass" -h "$_lhost" -B -N \
-e "SELECT DISTINCT CONCAT('SHOW GRANTS FOR ''',user,'''@''',host,''';') AS query FROM user" \
mysql \
| mysql  -u "$_lusr" -p"$_lpass" -h "$_lhost" \
| grep -v root | grep -v debian-sys-maint \
| sed 's/Grants for .*/#### &amp;/' | sed 's/$/;/'\
| sed 's/\\\\_/_/' &gt;&gt; "$_file"
 
#### Step 3: Dump database contents for all databases ####
for i in $(mysql -u "$_lusr" -p"$_lpass" -h "$_lhost" -e'show databases' | sed '1,1d' |  grep -v 'information_schema' | grep -v 'mysql'); do
        echo "use $i;" &gt;&gt; "$_file"
        mysqldump -u "$_lusr" -p"$_lpass" -h "$_lhost" "$i" &gt;&gt; "$_file"
done;
 
## Compressing SQL ##
tar -zcf $_file.tar.gz $_file
rm $_file

I hope you find this usefull.

Posted in Code, Need to know | Leave a comment

Usefull Code for Mass Renaming

Found this snippit of code and thought someone else would find it usefull

This was used to rename a bunch of log files with inconsistant naming schema to all lower case and with a .log extention instead of _log.

#!/bin/bash
### Rename all uppercase directories to lower case ##
# NB : Addapted from from http://www.linuxweblog.com/node/306
 
for x in `find * -type f -maxdepth 0`;
do
  # Translate Caps to Small letters
  y=$(echo $x | tr '[A-Z]' '[a-z]');
  if [ "$x" != "$y" ]; then
    mv $x $y;
  fi
done
 
for x in `find * -type f -maxdepth 0`;
do
  # Rename log files using .log instead of _log to conform to new naming schema
  y=`echo $x | sed "s/_log/.log/"`
  if [ "$x" != "$y" ]; then
    mv $x $y;
  fi
done

Enjoy

The ANAT Tech Department

Posted in Articles | Leave a comment

Download Buttons

Based on http://www.photoshopstar.com/web-graphics/professional-glossy-download-button/ I’ve attempted to create some download buttons.
I didn’t have the Myriad Pro font, and I know the look isn’t as swish as it could be, but it’s an attempt.

The Buttons

Download

Demo

Preview

Source Code

Continue reading

Posted in Downloads, Tutorials | Leave a comment