PackageEnhancement2007Story

Acceptance Criteria

Write scripts to create and populate the database with basic data and, optionally, with sample fictional data (users, projects and tasks), supporting different languages for that basic data. Check and manage non-standard conditions, for example, if the database is already installed. Test the installation process in deep.

Make debian packages based on the former scripts. Check the package dependencies and the installation process in deep.

If there is time enough, design a method to handle database schema updates and add support for them into the debian installer.

Additional Specification Comments

Have a look at the current database schema and the default predefined queries. Study what user data can be translated without disturbing the proper application operation. Translate them to create multilingual installation data sets.

Learn about some standard ways for deploying a database and its data set in Postgres. Write a script, or enhance the curren one, to handle that deployment. Check as many possible non-standard install conditions and errors as possible. Create a tgz package with that install script included.

Learn about making debian packages. Study how the current v1.4 package is done and find a way to improve it with the scripts done before. The package install process should be user friendly and follow the guidelines of standard packages in debian. It should ask the user for her preferred install language and if she want to install the sample example data, check if there is a previous version already installed, give the opportunity to keep the old data, delete it or update it.

Think of a way to handle schema updates based in the old and new package versions and the language chosen by the user. Implement it in the Debian package and test the process in deep.

If there is plenty of free time, study how to create an rpm package as well and build it.

-- EnriqueOcana - 10 Jul 2007

As a hint to learn how to make debian packages as much "standard" as possible, have a look at the following documentation:

• http://www.debian.org/doc/maint-guide/
• http://webapps-common.alioth.debian.org/draft/html/
• http://webapps-common.alioth.debian.org/draft-php/html/
• http://people.debian.org/~seanius/policy/dbapp-policy.html/
• http://es.tldp.org/Presentaciones/200002hispalinux/conf-29/29-html/debian-paquetes.html
• http://www.ibm.com/developerworks/linux/library/l-debpkg.html?ca=dgr-lnxw01DebianLinux

Also, to find packaging examples, install and have a look to the debian packages named "hello" and "hello-debhelper".

Some notes for installing and dealing with debian packages:

• To see a description of a package: apt-cache show PACKAGE
• To see what files come with a package (once installed): dpkg -L PACKAGE
• To remove a package (purge is for config files deletion also): apt-get remove --purge PACKAGE
• More actions: have a look at http://crysol.org/node/59

-- EnriqueOcana - 17 Jul 2007

For the installer dialogs, the best way is to use debconf. You can find documentation an tutorials looking for "debconf tutorial" in Google:

• http://www.fifi.org/doc/debconf-doc/tutorial.html
• http://xarxantoni.net:8080/mediawiki/index.php/Creaci%C3%B3n_de_paquetes_Debian

Implementation Notes

-- AndresManeiro - 05 Jul 2007

Today, i started to write scripts and i had some problems with the shell. The scripts started with #!/bin/sh and i thought it's correct. But it wasn't. When i executed the script i had some problems and, finally, i found out that i was using the wrong shell, i was using dash, isntead of bash. Now all the scripts start with #!/bin/bash and it goes ok.

But this isn't the only problem: the next code doesn't print the variable $OPTIONS.. does anybody knows why?

COMANDO1=`mktemp`
cat > $COMANDO1 << END
 psql template1 -c "CREATE USER $DBUSER WITH ENCRYPTED PASSWORD '$DBPWD'" \
  || { echo "User exists. What do you want?:" 
        OPTIONS="foo"
        echo $OPTIONS
 }
END

Well.. i will wait one day. If you know the reason.. you can write to amaneiro@igalia.com See you later!

-- AndresManeiro - 06 Jul 2007

Yesterday i had problems with here-documents mechanism, for this the $OPTIONS variable don't print his value. It's because the here documents mechanism (<<) evaluates the variables present in the text piece in order to get standard input . So, the $OPTIONS variable gets evaluated just in the moment the text is taken.

To prevent this, you have to escape the "$" in this way: "echo \$OPTIONS". This copies the variable name literally and lets it to be evaluated later, when that piece of script is really executed:

COMANDO1=`mktemp`
cat > $COMANDO1 << END
 psql template1 -c "CREATE USER $DBUSER WITH ENCRYPTED PASSWORD '$DBPWD'" \
  || { echo "User exists. What do you want?:" 
        OPTIONS="foo"
        echo \$OPTIONS
 }
END

But it's very hard if you want to scape more than one variable. In this case, is useful quot or escape the "limit string" at the head of a here document. It disables parameter substitution within its body ( cat > $COMANDO1 << 'END' or cat > $COMANDO1 << \END):

COMANDO1=`mktemp`
cat > $COMANDO1 << 'END'
 psql template1 -c "CREATE USER $DBUSER WITH ENCRYPTED PASSWORD '$DBPWD'" \
  || { echo "User exists. What do you want?:" 
        OPTIONS="foo"
        echo $OPTIONS
 }
END

-- AndresManeiro - 09 Jul 2007

Today, i completed the deployment scripts to install phpreport aplication (the first iteration that works good). In this first iteration, i had some problems with the special case that a previous phpreport aplication already existed.

At first, the flow work script was:

# First Step:
#    If there is an existent user there are some options to choose:
#     - overwrite it
#     - keep the existent user and continue the installation process
#     - abort the installation
#
# Second Step:
#    If a database already exists there are some options to choose:
#     - overwrite it
#     - keep existent user and go on with the installing
#     - abort the installation
#
# Third: LOAD SCHEMA FROM $DBSCHEMA AND POPULATE IT WITH $DBEXAMPLEDATA

In the point 1.1 (overwrite user) the script needs to remove default user 'phpreport' and his owned objects. So, it can delete the existent database BEFORE the second step.

The clausule from postgres DROP OWNED only delete objects from current database, but it don't delete the database (and i need it because the database have as owner the default user 'phpreport'). And later, thinking about the matter, I had an idea: Why should we keep the work flow script design? Is it a compulsory requirement? ... and what do you think? Should I switch steps one and two? Can it improve the process? Does it make the installation easier?

-- AndresManeiro - 23 Jul 2007

I go back to Twiki. I´ve been doing a summer course for a couple of days and spent some time reading about debian packaging. I realized that debian packaging it´s not a trivial process at all, even though it´s simple when you have a global vision, it can be difficult for beginners

In the following days i'll try to document the process clearly for beginners.

FIRST, THE GOAL: DEBIAN PACKAGE, but ... what a debian package (package.deb) is? It's a compress file containing three files:

• control.tar.gz: contains configuration files and metadata
• data.tar.gz: contains all the relevant files to install in system
• debian-binary: contains the version number for the binary package

SECOND, THE PHASES ... now you know what a package is... how can we achieve it?

I will relate the process to create a package for web aplication. This process seems compiled aplication packaging.

Web aplication => (step1: to debianize) => Debianized web aplication => (step2: to package) => Package.deb

• Step 1: to debianize. In this point we only have to create a FHS (Filesystem Hierarchy Standard) with files that we want to put in final system. So, we have to create a few configuration files need for step 2.

• Step 2: to package. Here, we use the available software tools to create a debian package and other relation files from debianized program.

In the following days, I´ll write about each step in deep.

-- AndresManeiro - 26 Jul 2007

Today, i write about PACKAGE DEBIANIZING PROCESS for web aplication. In this point, you have two tasks: create a "paralell" FHS and some specific files. If you want to know more about this process, you can view a good document about build debian package without help, also in spanish.

• Create Filesystem Hierarchy Standard (FHS)

Well. Come on. Within your directory program to debianize, create another directory named debian. And within it duplicate the dirs where your place your web aplication files. This you can do with software tools (as dh_make) that automatize this process. Finally, the directory structure should look like this...

{previousdir}/packagename-version/

{previousdir}/packagename-version/debian (place here specific configuration files below)

{previousdir}/packagename-version/debian/etc/ (configuration files web aplication)

{previousdir}/packagename-version/debian/usr/share/ (share files web aplication)

{previousdir}/web_aplication/debian/var/www/ (static and dinamic files aplication)

You must read Debian policy about the good structure for your files.

• Create Specific files

Some specific files are necessary to customize the behaviour of the package. Like the previous step, this process can be done by means of software tools like dh_make. The four required files are:

- copyright: information about license and copyright

- control: info about the package to package management tools (his depends, who build it, description, ...)

- changelog: contains version number, revision, distribution and urgency of your package... and other changes over it

- rules: as you can suposse... contains the rules for building the package. It's a makefile.

Other files are useful too. You can view them in Debian New Maintainers' Guide

Well. Fortunately enough, there are some software tools to do the harder work. They're dh_make and other debhelper tools. Now, i'll explain HOW TO USE THESE TOOLS IN ORDER TO AUTOMATIZE THE DEBIANIZE PROCESS.

In a shell, go to your web aplication directory, and then type "dh_make". This creates a template structure in order to debianize the aplication. This template consist of a few scripts, for example it creates the four steps mentioned above (copyright, control, changelog and rules). Now, you must adapt them according to your needs. At this point, your must review the dh_make generated control file. So you already have your specific and required files. Well.. now we create the FHS.

For this, debhelper tools will help us. We will use dh_installdirs, dh_installdocs and dh_install.

(NOTE: You must know that is a makefile and how it goes for next explanation). If you review "rules" file you can view that dh_make has create some rules for you. In this point i checked that "rules" file had dh_installdirs in target "install" and dh_installdocs, dh_install in target "binary-arch".

Read the manual entries for each of the "dh_x" utilities mentioned above. In brief, they work in this way...

• dh_installdirs: create the directory that you indicate in "dirs" file in FHS (not in the system but in you package directory when build the package). I create a file "dirs" look like this..

usr/share/doc/phpreport/
var/www/phpreport/css
var/www/phpreport/images
var/www/phpreport/include

• dh_installdocs: install the docs indicates by "docs" file in usr/share/doc/package, copyright file and other useful scripts if exists too. I also create a "docs" file look like this ...

sql

• dh_install: install other files in directory as you indicate in file "install". By example, my "install" file look like this ...

*.php   var/www/phpreport
*.html  var/www/phpreport
*.dtd   var/www/phpreport
css     var/www/phpreport
images  var/www/phpreport
include var/www/phpreport

So, you can see that debhelper utilities and dh_make are very easy to use and they do your life as debian maintainer more easy!

-- AndresManeiro - 01 Aug 2007

Build a debian package is very easy once you had debianized the aplication (view above). Then, you can use dpkg-buildpackage. In your favourite shell write dpkg-buildpackage -rfakeroot -us -uc and wait for outcome (you must do it within your directory web aplication). It create the package.deb file. So, we already have a debian package ready to install.

So, you can install this package with dpkg -i package.deb (you must be root for this).

But the building process isn't complete yet. I want do that the user can choose some options during install (username and database name by example), for this, i use debconf utility (view also a briefly and useful tutorial) wich management install dialogs. Also, in order to manage database install, reinstall and update as good as posible... i choose dbconfig-common (view also technical reference) aplication for this.

They are an useful tools .. but i am having some problems when they fail... the debug process is hard for a beginner user. I will try to write about this tools and install process in-deepth in next days.

-- AndresManeiro - 03 Aug 2007

Today, i finish the last iteration for debian packaging. So... i have my debian package for web aplication phpreport runs with debconf and dbconfig-common, wich provided me a useful software tools to create and build the package. Now, I only have to debug and improvement some minor tasks. Today, i start the documentation story, for this story too (write about use debconf and dbconfig-common).

-- AndresManeiro - 03 Aug 2007

Debconf is an utility wich help you to use installer dialogs. You must write a template file that will be showed in installer dialog.

This templates have a specific and rigorous format. You must be careful with sintaxis, it can cause troubles for you. By the way, i had some bugs because i was use an space instead of a dot, as manual says.

Then, you must write a few of code in some scripts. In order to use debconf, you can use a config script for it. Before i write about this... you must know that it occurs when someone install a debian package. When you execute dpkg -i packagename.deb the next scripts are executed (if they exist) in this order (you can view the man page for more info):

• dpkg -i => (config, if debconf) => prerinst => postinst => package ok

So, you can write code in some script to prompt. I only use config script for this, how debian policy says... though dbconfig-common prompt his answers in postinst. In this useful tutorial you can view more info about use debconf.

About debugging with debconf:

• A variable exists for help you: DEBCONF_DEBUG. You must set it as DEBCONF_DEBUG="developer". Use it.

• I also recommend you set the answer priority to low (db_set debconf/priority low)

• Finally, you can view the source primitives (db_get, db_set, etc) in /usr/share/debconf/confmodule and go on. I could resolve some trouble quickly because i know source insight script. And never forget source this script in you config (or postinst) script!

-- AndresManeiro - 17 Sep 2007

EnriqueOcana did some improvement at phpreport aplication which modifies database and php scripts. I was installing a new dataset to probe this improvement and i was evaluating work in order to adapt installation process to this changes.

-- AndresManeiro - 18 Sep 2007

I translate database queries into english language. This work was very easy from i [[https://community.igalia.com/twiki/bin/view/PhpReport/AdminAndUseDocumentation2007Story]had identified id queries] and they was translanted into technical documentation.

I had some problems with purge process. They taken too much time and, finally, i wrote a mail to Sean Finney (mainteiner dbconfig-common package) about they. The purge process didnt purge configuration files properly and i had to do manually in shell:

ucf --purge /etc/dbconfig-common/phpreport-1.5.conf

I was also evaluating a solution in order to create a testing dataset.

-- AndresManeiro - 19 Sep 2007

Dbconfig-common package use debconf in order to management Dialog menu. Debconf have four priorities to show question to user: low, medium, high, critical. When dbconfig-common creates a postgres database, it shows next questions according to priority configured as goes:

Anyway, you can always reconfigure priority with:

dpkg-reconfigure -p your_priority_favourite_level _your_package_name_

Default values when questions aren't asked as goes:

Conection method (socket unix, tcp/ip, tcp/ip +ssl) = unix socket
Admin Autentication Method (ident/password) = ident
User Autentication Method (ident/password) = ident
Database Admin Name (for postgres database) = postgres
Database User Name (for your package) = packagename without characters as - or .For instance, 
if you package name is phpreport-1.5, default value is phpreport15.
Database Name (for you package) = phpreport15 (same as above)
Password (for you database package) = 12-random-characters

UNISTALL/PURGE APLICATION

In order to unintall phpreport, you must difference two actions: desintall (dpkg -r phpreport) and purge (dpkg -P phpreport). First, only uninstall aplication. Second, unistall aplication and purge data (database and configuration files; you can view phpreport.postrm).

-- AndresManeiro - 20 Sep 2007

We want install PHPreport with dataset in order to you can to realice some probes with them. So, we did a backup from igalia data (limited from date and others parameters) and change date with other invented date.

Today, i wrote a simple script wich changes igalian people names by 100 most frecuent girl spanish names (i get them from INE, Statistic National Institute from Spain).

#!/bin/bash
# Script to change words from file usersfile
# oldusers = path to file with words to replace
# newusers = path to file with words wich you want

numusers=`uniq -u usersfile | wc -l`
for i in `seq 1 $numusers`; do
    rpl -wq `sed -n "${i}p" oldusersfile` `sed -n "${i}p" newusersfile` usersfile;
done

Other way to do the same is:

#!/bin/bash
oldusers=${dirbase}/oldusersfile
newusers=${dirbase}/newusersfile
rplusers=${dirbase}/rplusersfile

exec 3<>$oldusers
exec 4<>$newusers

numusers=`uniq -u usersfile | wc -l`
for i in `seq 1 $numusers`; do
    read olduser <&3;
    read newuser <&4;

    rpl -wq $olduser $newuser $rplusers;
done

exec 3>&-
exec 4>&-

-- AndresManeiro - 21 Sep 2007

I improvement scripts above in order to abstract them and resolve some problems:

• files are passed as parameters
• lines can to contain more than a word (by example, a sustitution from bussines names: "bussines inc. from galician")
• protection when number words in old file is largest than a number word into new file (only into script which use rpl whitin sed clausules: the new file are readed recursively, once it is finished, it's started from init)

#!/bin/bash

# Script to change words from file usersfile
# Parameters:
#  * file old words ($1)
#  * file new words ($2)
#  * file to replace ($3)

exec 3<>$1
exec 4<>$2

# use uniq to count num_olds, because once sustituted in $3 never more will be sustituted
num_olds=`uniq -u "$1" | wc -l`
#num_news=`cat "$2" | wc -l`

for i in `seq 1 $num_olds`; do
    read old <&3
    read new <&4
    rpl -wq "$old" "$new" $3
done

exec 3>&-
exec 4>&-

# # WARNING: you must check that num_olds < num_news
# # otherway, sustitutions wich num_olds > num_news are made with "" 
# # for correct this, you can use comment code below (more ineficient)

# num_olds=`uniq -u "$1" | wc -l`
# num_news=`cat "$2" | wc -l`

# for i in `seq 1 $num_olds`; do
#     # it prevents errors because num_old > num_new
#     j=`expr $i % $num_news + 1`
#     rpl -wq "`sed -n "${i}p" "$1"`" "`sed -n "${j}p" "$2"`" $3
# done

In other hand, i am having some problem because when i change id_projects because some "projects id" fit in with "areas", and then, they are sustituted too.

-- AndresManeiro - 24 Sep 2007

In order to improvement script eficience i am probing this solution:

exec 3<>$1
exec 4<>$2

# use uniq to count num_olds, because once sustituted in $3 never more will be sustituted
num_olds=`uniq -u "$1" | wc -l`
#num_news=`cat "$2" | wc -l`

for i in `seq 1 $num_olds`; do
    read old <&3
    read new <&4

    pars[i-1]="-e s/\<$old\>/$new/g"
    #echo "${pars[i-1]}"
done

#echo "${pars[@]:0}"
sed `echo "${pars[@]:0}"` $3

exec 3>&-
exec 4>&-

I was some problem because i was probing with echo (view comment lines) but i don't use doble-quote to print out. If you print this with echo ${pars[i-1]} the out isn't that you expect. It will be only 's/$old/$new/' because -e is an echo option.

With this line _pars[i-1]="-e s/\<$old\>/$new/g" _ i had two problems:

• First, i wrote pars[i-1]="-e 's/$old/$new/g'" but it didn't work. Then, i delete simple-quote pars[i-1]="-e s/\<$old\>/$new/g" and it goes on.

• Second, i want replace only completely words (view problem here). For this, i wrapped $old variable between "<>" characters: pars[i-1]="-e s/\<$old\>/$new/g"

-- AndresManeiro - 03 Oct 2007

In order to finish installation package and basic dataset creation, i am creating some example data manually. Once i have to probe this dataset, i take up again to modify dataset from igalian data.

How create an example dataset:

• create parea label, csector label, type label
• create customer
• create project
• create city
• create periods to users
• add users to project
• add projects to customers
• insert tasks users

Delay Causes

Final or Pending Considerations

Tasks in this story

div class="twikiTopicInfo twikiRevInfo twikiGrayText twikiMoved"<&/div>-->