Category: IT & Technical

For most of us, we have some data on some computers that we’d like to keep. For some it is critical. However, it can be a minefield, and finding a suitable strategy can be difficult.

Having spent a lot of time in the past moving precious data from one hard drive to another, and using fault tolerant RAID arrays to protect important data, I decided that it would be a good idea to invest in a long term and repeatable backup, and tape seems to be the accepted way to do this.

Since I don’t want to invest the life-savings in a state of the art tape drive, an older (but still very capable) model should do for what I need.

Hence I have bought an HP LTO5 Drive – 1.5TB per tape native, and up to 3TB compressed, which depends entirely on what you are backing up. Assume 1.5TB to be on the safe side.

The LTO6 and 7 drives hold more data per tape, but the tapes are proportionately more expensive too, so LTO5 seemed like a good compromise.

However, if you’re used to using NAS or external Hard Drive as backup, tape seems to be an altogether more complicated thing and fraught with troubles.

As I have an HP server, I decided to buy an HP drive as that should ensure it was compatible. There are good second hand offers on eBay, but I found a “new unboxed” one for a good price and bought that.

Problem 1: SAS Connections

If you’re used only to SATA (as I am), SAS is quite a new and complicated thing. I checked the specs for the server I was going to put the drive in, and it says that it has SAS connections, so that should be all right? Wrong! While there are SAS connections in the server, the are all used in the 4 non-hot-swap drive bays. Ok, says I – I’ll plug in one of the HDDs into the SATA port, and free up a SAS port. Brilliant – the HDDs are fine, and the tape drive lights up.

Does Linux recognise it? Of course it doesn’t – the hard drive controller in the (albeit entry level) HP server offers SAS connectors and says SAS on the specification (I did check before buying the drive), but is only capable of driving SATA hardware. Thanks HP!

A new SAS controller card and suitable cables (an extra £200 odd) and I’m in business – or so I thought.

Problem 2: Warranty Support for Updates

So I try a backup – looks good, and then an error…

Try again… looks good… error.

Reading about the errors I was getting on various forums, the advise seems to be to update the tape drive firmware.

However, just because the product is advertised as “new” by the seller – don’t expect HPE to give you up-to-date firmware to go with it – this is “enterprise” kit you have to pay for support if the item is out of warranty – and there is no way of finding out if it is in warranty if you don’t buy from HPE directly, or from an authorised reseller. (i.e. not from eBay)

A Solution

Noting that the problems documented as fixed by the update didn’t include mine, I continued to search websites for the specific error I was getting which was only shown by dmesg, not by tar, and found that when using tar with an LTO5 drive, the block size is essential.

The advise given here was you should use at least 256kb blocksize for an LTO5 drive.

So, almost like magic, adding the parameter “-b 512” (i.e. 512 x 512b = 256kb) to the tar command fixed the problem, and I’m now backing up to tape, but only after several hours of scouring the web and pulling my hair out.

Conclusion – the Moral of this Story

Tape backup is a whole order of magnitude more complicated and expensive than an external HDD or NAS drive. Don’t start down the road of tape unless you really need it, and can invest the time and energy in getting it to work properly.

Do your research thoroughly before buying anything, especially 2nd hand, and don’t assume just because you see something on the documentation that it will work without additional hardware and/or software.

Was it all worth it? Well, I hope I’ll never need to find out…

 

I have been requested by a client to produce data extracts from an old bespoke system, which has been knocking about for many years and been left without upgrades for several years. A nice piece of work for me, but it’s got me thinking about the whole upgrade lifecycle issues and bespoke vs packaged solutions in general.

The reasons it’s not been upgraded are that “it ain’t broke – don’t fix it”, and that nobody wants to pay the costs to keep regression testing new upgrades.

However, there are disadvantages to letting a system continue on old techonolgy for a long period of time. Maintenance gets more difficult as fewer people are available who remember the older technology. The world moves on and the system doesn’t, so it looks antiquated, and leads users to feel irritated by the inadequate user-interface.

The net result is that a new proprietary packaged system is going to be deployed replacing the creaking bespoke system, at considerable cost. However, as with all packaged solutions, the “out of the box” fit with the business requirement is good, but not perfect. Ok, the bespoke system doesn’t fit with the business 100% either, since the business has moved on and it hasn’t, but I can’t help thinking that had the bespoke system been periodically upgraded to use the latest (or at least more recent) technologies, and the changes to the requirements, that the requirements fit could have been better and the overall cost less than the new replacement system.

A final disadvantage of remaining on older techonolgy, comes when carrying out the extract of data for migration, you realise that “that approach doesn’t work on that version, you need the next one”. Realising that several approaches aren’t possible due to the features I’m now used to being for more recent versions, it becomes frustrating to have to rethink each time.

I’ve spent the past few days implementing a serialisation process overriding the ObjectOutputStream and ObjectInputStream classes. However there seemed to be certain classes that cannot be deserialized even though they are serializable, as they have no default constructor.

The serialisation class documentation implies that the constructor for a serializable class isn’t called anyway, which makes sense as you don’t really want to set up a class from scratch when you are deserializing, just restore it.

Most replies to questions on this subject seemed to imply that it can’t be done except by the default serialization classes. This is no use if you want to override these, and since they are overridable, surely Sun woudn’t have made it impossible.

One hit during my search mentioned the ReflectionFactory class in passing, but didn’t have any detail, so I looked into it further, and sure enough, this seems to be the answer.

The Solution?

It would seem that the ReflectionFactory class can create a Constructor object on the fly, using the default constructor of the first non-serializable superclass, and the class you’re trying to instanciate.

So when overriding the deserialisation process, you can create an instance of an object that doesn’t use the default constructor, and doesn’t need one to exist by calling the constructor returned by something like the following:

ReflectionFactory rf = ReflectionFactory.getReflectionFactory();
Class type = HashMap.class; //
Class nonSerType;
for ( nonSerType = type; Serializable.class.isAssignableFrom(nonSerType); nonSerType = nonSerType.getSuperclass() ) {}
Constructor superCon, deSerCon;
superCon = nonSerType.getConstructor();
deSerCon = rf.newConstructorForSerialization(type, superCon);

where the deSerCon can be used to instanciate the deserialized object.

This process seems to work for my deserialization work, and so I’ve documented it here. I’ll admit, I’ve no idea if there are any specific limitations to this process, but for the limited range of classes I’ve tested so far, it works fine.

I have spent several days trying to figure out how to do server side XSLT using an Apache server using numerous suggested ways from articiles on the web about how to do it using Cocoon, and using mod-xslt2, all of which are out of date in one way or another, i.e. mod-xslt2 doesn’t work with Apache 2.2 yet, and the article describing how to do it with Cocoon used a previous version of Cocoon so didn’t help much with the current one.

However, I have now managed server side XSLT2 transformation using Apache, Tomcat, mod-jk, mod-rewrite and Saxon as the transformation engine.

The Theory

I noticed in the Saxon documentation, that one of the demos was a simple servlet. So, running this servlet on Tomcat, I should be able to do the transformations on the server.

The downside of this approach is that I was going to end up with really nasty looking URLs, such as

http://www.screwtape.co.uk/Servlets/SaxonServlet?source=xml/homePage.xml&style=xsl/main.xsl

which is hardly convenient when writing links, and while that could be automated in the xsl transformation, it would still look ugly in the browser address or status bars.

However, Apache has mod-rewrite which does lots of exciting things with URLs. This can rewrite any incoming URL to poke the relevant parameters into the servlet.

The Practice

Install Apache, Tomcat and mod-jk. There’s loads of instructions for that on the web, and anything I say will have been better said elsewhere so I won’t.

Everything here is rather distibution dependant, so your system may be slightly different, but the ideas should be similar. I’m using Centos 5.2 here.

1. Download and install SaxonB 9.1 for Java. Make sure you also get resources package, as that has the servlet sample.
2. Complile the SaxonServlet.java file, and create a .war file. To do this, I created a web application in Netbeans 6.5 and got it to do all the compiling and packaging etc.
3. Using the tomcat manager app, add and release the .war file.
4. Place or link your xml and xsl files in the servlet directory, so the servlet can access them. I have /usr/share/tomcat5/webapps/Servlets/xml and /usr/share/tomcat5/webapps/Servlets/xsl linked to my webserver directories, although you could have the xml and xsl files just in the servlet directory.
5. I use the default virtual host, and hold the majority of the website files in the apache html directory, therefore you can’t just use the auto-generated virtual host file.Update the /usr/share/tomcat5/conf/servers.xml file: the <Host> needs some <Alias> tags to allow Tomcat to see your website when not accessed through localhost. The host section look something like this:

   <Host name="localhost" appBase="webapps"
       unpackWARs="true" autoDeploy="true"
       xmlValidation="false" xmlNamespaceAware="false">
       <Alias>www.screwtape.co.uk</Alias>
       <Alias>screwtape.co.uk</Alias
       ...
   </Host>

6. Restart your tomcat instance, and this will create the “auto” subdirectory in the /usr/share/tomcat5/conf directory.
7. Look through the mod_jk.conf in the auto directory, and you should see a pair of lines starting JkMount. Copy these lines into the main httpd.conf file area, if like me you’re not using virtual hosts, or into the relevant <VirtualHost> element. You should have something like this:

   JkMount /Servlets ajp13
   JkMount /Servlets/* ajp13

8. Restart httpd, then we can test that you can run the servlet from Apache. So from your browser try

   http://localhost/Servlets/SaxonServelt

   This should show an error message about missing style parameter. If you get this message then Apache and Tomcat are linked and you can proceed.

   It is probably a good idea to try and access the servelet from other hostnames/domain names that might use it, since it may work for localhost and no other hosts. Thus trying

   http://<yourhostname>/Servlets/SaxonServlet

   may be a good idea too.

9. Now we need to rewrite the incoming URLs to use the servlet. I have all my URLs beginning xml/ rewritten to use the servlet, with a single xsl transformation. If you need different xsl files, then this will be slightly more tricky, and will need a separate rewrite rule for each xsl file, or a more complex servelet.Add the RewriteRule directives in the <Directory element appropriate for your site. I have

   <Directory "/var/www/html">
       ...
       RewriteEngine       on
       RewriteRule         (^xml/.*$)      /Servlets/SaxonServlet?source=$1&style=/xsl/main.xsl
   </Directory>

10. Restart httpd and assuming everything is ok, you should get your xml documents transformed using the specified xsl. That’s how this page works, as it’s entirely written in xml.

I hope this is useful to someone, and if anyone has a better way of doing it I’ll be interested to know. One of my friends insists PHP is the way to go, but I can’t see that being any easier than this.

Unfortunately, I failed to find any suitable instructions for doing this when I was looking to set up a single fetchmail daemon for Centos. It appears this is easy on debian flavours, and perhaps there are better ways of doing it than this, but I thought I’d share the method I used…

I’ve kept the steps fairly simple, so please forgive me if some of it is completely obvious…

1 – Install fetchmail if not already installed

Check if fetchmail is already installed, and perhaps update it if it is out of date.

yum list fetchmail

will show you if you’ve already got it installed, and which versions are available for install. I used the x86_64 package version 6.3.6-1.1.el5

Use yum to install or update this as required.

2 – Configure fetchmail

Why is nothing ever simple under Centos? I tried to find a RPM fetchmailconf, the gui fetchmailrc file configurator, for Centos 5 but couldn’t find one for the current version of fetchmail.

I ended up with fetchmailconf-6.3.4-1.1_6.0.1.el5.x86_64.rpm which wouldn’t install as it depends on the fetchmail-6.3.4 rather than the 6.3.6 version I had installed, so I had to install it using

rpm -i --nodeps fetchmailconf-6.3.4-1.1_6.0.1.el5.x86_64.rpm

This at least worked, and so I used it to configure my .fetchmailrc file for my mail servers etc. I then copied the ~/.fetchmailrc file to /etc/fetchmailrc for use by the daemon.

3 – Set up a script in /etc/init.d.

I largely copied the script for the Samba daemon.

#!/bin/sh
#
# chkconfig: - 91 35
# description: Starts and stops fetchmail in daemon mode
#

# Source function library.
if [ -f /etc/init.d/functions ] ; then
  . /etc/init.d/functions
elif [ -f /etc/rc.d/init.d/functions ] ; then
  . /etc/rc.d/init.d/functions
else
  exit 1
fi

# Avoid using root's TMPDIR
unset TMPDIR

# Check that fetchmailrc exists.
[ -f /etc/fetchmailrc ] || exit 6

RETVAL=0

start() {
        KIND="fetchmail"
	echo -n $"Starting $KIND services: "
	daemon --user fetchmail fetchmail -f /etc/fetchmailrc --syslog
	RETVAL=$?
	echo
	return $RETVAL
}	

stop() {
        KIND="fetchmail"
	echo -n $"Shutting down $KIND services: "
	runuser fetchmail -c 'fetchmail --syslog --quit >dev/null 2>&1'
	RETVAL=$?
	[ "$RETVAL" -eq 0 ] && success $"$base startup" || failure $"$base startup"
	echo
	return $RETVAL
}	

restart() {
	stop
	start
}	

reload() {
        echo -n $"Reloading fetchmailrc file: "
	RETVAL=$?
	echo
	return $RETVAL
}	

rhstatus() {
	status fetchmail
	RETVAL=$?
}	

# Allow status as non-root.
if [ "$1" = status ]; then
       rhstatus
       exit $?
fi

# Check that we can write to it... so non-root users stop here
[ -w /etc/fetchmailrc ] || exit 4

case "$1" in
  start)
  	start
	;;
  stop)
  	stop
	;;
  restart)
  	restart
	;;
  reload)
  	reload
	;;
  status)
  	rhstatus
	;;
  *)
	echo $"Usage: $0 {start|stop|restart|reload|status}"
	exit 2
esac

exit $?

You can download this file by right clicking here.

4 – Set up a fetchmail user.

Create a user on the system called fetchmail. I used the -r option with useradd but then I had to create a home directory as the fetchmail program expects one. You can set up an alternative directory using an environment variable, but I took the easy way and just created a home directory.

You then chown the /etc/fetchmailrc file to fetchmail, and the home directory.

5 – Start it up…

Now try and start the service.

service fetchmail start

With any luck, the service starts… I suggest you then try and stop it with

service fetchmail stop

to ensure the stop code works too.

Next steps…

I then added the service using the gui services control, and this works ok, allowing me to start, stop, and configure the service for automatic startup on level 5.

However, selinux is doing something I don’t quite understand. It works using the service command from the command line, but it doesn’t work during the normal start up process. I suspect this is to do with the context task initrc_exec_t, but what I’ve tried so far doesn’t work. Someone who understands selinux better than me can probably tell me what I’m doing wrong.

I hope this is useful to someone. Let me know if you find a better way…