Project automation & dependency management for TYPO3.CMS projects

comments

typo3 cms project automation deployment build process

After some years developing web application projects with TYPO3.CMS I‘d like to share and discuss my thoughts about project deployment, building and dependency management with the current LTS version 6.2+ with a broader audience.

Deprecated article

This article is deprecated with the release of TYPO3 CMS version 7.4+.

The composer integration into TYPO3 CMS has made great progress and therefore the approach outlined in this article is no longer relevant.

Status quo

The TYPO3.CMS project made great process during the last months. Introducing namespaces and autoloading capabilities with 6.0 the 6.1 releases paved the way for further improvements regarding package management (via TYPO3.Flow Package manager) and composer integration in 6.2.

The project aims at easy installation and setup for people who like to install the CMS on shared hosting providers.

While TYPO3.Surf provides a good base for professional deployment strategies the concept is not a new one: Capistrano (Ruby, since 2006) or Fabric (Python, since 2009) comes with the same functionalities.

TYPO3.CMS itself doesn‘t ship with any deployment tools like - for example - similar to symfony 1. The rsync based approach there was very basic, but the application had one!

Additionally, it is very difficult to setup and maintain a TYPO3.CMS project from the command line. The setup / installer process of the system itself and extensions is completely GUI driven which makes it impossible to use database migrations or a command line interface to perform these tasks.

Everything is a component

I think building web applications with a CMS can be very well handled using a component based approach. Using an application like TYPO3.CMS as a component has many advantages:

1. Implementing framework agnostic

While being squeezed into typo3conf/ext/ this is sometimes (mentally) not possible. Furthermore, this location also differs from modern quasi standards known from frameworks like Symfony or Zend Framework: src/.

2. TDD - but fast and instant feedback

The TYPO3.CMS extension phpunit provides good functionality within a backend module for running tests and viewing the results.

Being an efficient developer you want instant feedback. Every request - response cycle steals time. Using the native Linux watch command coupled to phpunit execution gives you instant, continuous feedback.

3. DRY

Make usage of components get rid of project specifica. Imagine you want to replace GIFBUILDER by WideImage or historic parseFunc by HTML Purifier. It‘s easy to get those deps via Composer providing TYPO3.CMS hooks via glue code and make usage of modern, well proven and tested components.

4. Build locally, slim deployment

By using a build automation tool you are able to package the project suiting your needs and remove runtime irrelevant parts like unit tests. Using rsync is the most basic approach. But what if you want to deliver Debian or RPM packages? You need a custom build process allowing you to tighten the package and providing a slim runtime environment.

5. Separate business logic from dependencies

It‘s not enough to encapsulate everything in a TYPO3.CMS extension, you should also consider to keep a clean VCS repository. I‘ve seen many VCS repositories containing the whole TYPO3.CMS project. This isn‘t distraction free, every (framework) developer not living in the TYPO3.CMS „domain“ will despair at those structure: „Where‘s src/?“. It prevents other developers from focusing on real business logic problems.

Bending and breaking the shackles

To achieve these goals I‘d like to introduce 6 milestones:

  1. Setting up the project extension structure
  2. Integrate Composer autoloading
  3. Allow a custom project structure
  4. Provide an extension as a package
  5. Continuous integration / testing
  6. Simple, repeatable project setup during development

1. Setting up the project extension structure

Your code will live in src/extension_key/. This way you‘re out of typo3conf/ext/. Letting your head think in a more framework agnostic manner, possibly extracting all generic code into a proper Composer package and only writing down glue code in your extension.

The first thing is to create a composer.json for your project (you need it anyway to use other packages) and add the following lines:

{
	"name": "vendor/project",
	"description": "A TYPO3.CMS project",
	"type": "project",
	"license": "GPL-2.0",
	"autoload": {
		"classmap": [
			"src/"
		]
	}
}
composer.json - kickstarted

Attention: please wait with calling php composer.phar install. Proceed to milestone 3 in order to prevent getting a distracted project directory.

2. Integrate Composer autoloading

While the latest TYPO3.CMS version 6.2+ LTS comes with a composer.json and a working class- / autoloader the setup isn‘t ready for using TYPO3.CMS as a „package“ with the Composer autoloader. The „autoload“ key is missing in composer.json and you can‘t load classes from other sources than typo3conf/ext/, Packages/Libraries - which I think is too specific.

Extend the composer.json with the following lines:

{
	// snip
	"require": {
		"typo3/cms": "~6.2"
	},
	"autoload": {
		"classmap": [
			„src/“,
			"vendor/typo3/cms"
		]
	}
	// snap
}
composer.json - autoload configuration

3. Allow a custom project structure

This one really kept me running in circles. Maybe it is/was a Composer issue, but even by specifying --no-plugins during php composer.phar install/update the project‘s root directory gets polluted by some TYPO3.CMS specific files, directories and symlinks.

These entries were introduced by the Composer plugin typo3/cms-composer-installers, shipped with the composer.json of TYPO3.CMS.

To get rid of this custom installer logic, just extend your project‘s composer.json a third time by:

{
	// snip
	"replace": {
		"typo3/cms-composer-installers": "*"
	}
	// snap
}
composer.json - override custom cms installers

Now everything is ready to load TYPO3.CMS as a real project package dependency and make usage of the Composer autoloader for your project code living in src/ and running tests from the command line without the need to load the whole TYPO3.CMS application.

After calling php composer.phar install you have a proper autoload classmap for your project and TYPO3.CMS.

4. Provide an extension as a package

This task is rather complex. Because the TYPO3.CMS project still uses the TER and Extension Manager backend GUI for downloading and installing extensions a custom Composer repository needs to be introduced to manage your extension dependencies as real Composer packages.

The guys over at Lightwerk set up a pretty cool Composer repository, ready for usage within your own composer.json. They crawl the TER and create a proper Composer mapping and dependency resolution for each TER extension not marked as „insecure“.

The downside is: every package in this repository has the dependency lw/typo3cms-installers. This is just another example for forcing developers using a specific structure within their projects. Just because it works for one it doesn‘t mean it fits every need.

Furthermore, this is no official package repository, means that if this service gets closed, all of your projects can‘t be build. So I suggest to create your own Composer repository. This way you‘re independent, only register the packages you really need (at my company we‘re only using a small set of TER extensions; all others are custom built) and can create a simple and lightweight package structure.

All you need is Satis and a custom satis.json definition. Head over to an example package repository to see an example.

Just make sure, you specify proper vendor keys for all TYPO3.CMS extensions. Lightwerk uses typo3-ter, while my approach is to use typo3-cms-extension. This vendor key allows to cherry pick all TYPO3.CMS extensions from Composer‘s vendor/ directory and handle these packages during building your own project.

Your composer.json needs to be adjusted again, and it might look like this example:

{
	// snip	
	"repositories": [
		{
			"type": "composer",
			"url": "http://packages.example.org/"
		}
	],
	"require": {
		"typo3-cms-extension/realurl": "~1.12",
		"typo3-cms-extension/static_info_tables": "~6.0",
		"typo3-cms-extension/static_info_tables_de": "~6.0",
	}
	// snap
}
composer.json - Private package repository integration

5. Continuous integration / testing

Continuous integration and testing is a vital requirement while developing enterprise level web applications. Fortunately all necessary tools are available as Composer packages already. So make sure you specify the dev requirements in your composer.json as following:

{
	// snip
	"require-dev": {
		"phpunit/phpunit": "~4.0",
		"phpmd/phpmd": "~1.5",
		"pdepend/pdepend": "~1.1",
		"phploc/phploc": "~2.0",
		"squizlabs/php_codesniffer": "~1.5",
		"typo3-ci/typo3sniffpool": "1.0.0-alpha",
		"typo3-ci/typo3cms": "1.0.0-alpha",
		"sebastian/phpcpd": "~2.0"
	}
	// snap
}
composer.json - PHP QA tools integration

Updating the Composer dependencies without specify --no-dev now allows you to continuously run unit tests and ensure high quality software development by generating and analyzing code metrics.

The last milestone introduces a possibile solution in how to incorporate these quality assurance tools into the project build automation.

6. Simple, repeatable project setup during development

While the initial setup of a TYPO3.CMS project isn‘t simple at all (eg. one can not execute a simple command to install TYPO3.CMS), I think it‘s a good approach to provide simple and repeatable setup for all following work during the project‘s development and future maintenance iterations.

Initially you must use the TYPO3.CMS Install Tool to create a configuration file and some application specific folders and files (eg. typo3temp/, typo3conf/ etc.) and the Extension Manager to create the package states file.

web/ should be volatile! Think of it as a dist directory, holding the whole ready-to-use application after building your project. With this in mind, extract the configuration files into a data directory, now they are repeatedly installable into the web/ directory and also can be easily monitored by the VCS.

Start over - cleanly. Every developer must have the possibility to start over the whole project setup. Thus in mind, it must be possible to delete everything code related and keep all user generated or fixture content.

To achieve this goals I decided to use Phing as the project automation tool. With Phing its possible to create easy to extend and maintain tasks to perform the following operations:

  1. Run the QA toolchain
  2. Cleaning up the distribution directory
  3. Install TYPO3.CMS by copying from vendor/ to web/
  4. Copying all project and vendor extensions
  5. Copying configuration + fixture data
  6. Performing project specific tasks (like Grunt or other frontend-related stuff)
  7. Executing the release process which incorporates operations 1-6 and additionally a deployment toolchain like rsync or building and distributing operating system packages (eg. deb/rpm)

Please head over to this website‘s GitHub repository and review the build configuration as an example.

Composer integration during runtime

While we‘re using Composer from a development point of view, it‘s also very interesting and - of course - a must-have to make use of installed dependencies during runtime.

Because TYPO3.CMS brings is own autoload mechanism and instead of using its propriertary vendor paths the autoloader must be adjusted to suit your needs.

During running the unit tests, one can include the vendor/autoload.php as a bootstrap script within PHPUnit, but using this file during runtime will lead to a runtime exception: typo3/cms/ and typo3-cms-extension/* are moved out of the vendor/ directory during project building (because TYPO3.CMS needs runtime files in specific directories) and the Composer classloader is not usable in a runtime context.

Therefore, you can incorporate the Composer autoloading in a custom way into your TYPO3.CMS instance. Add the following lines to one of your project‘s „main“ extensions (eg. a „site“ extension, encapsulating site wide templates and TypoScript setup) ext_localconf.php:

// snip
require PATH_site . '/../vendor/composer/ClassLoader.php';

/* @var $composerAutoloader \Composer\Autoload\ClassLoader */
$composerAutoloader = new \Composer\Autoload\ClassLoader();

// add only things we actually need during runtime
$composerAutoloader->add('Net_', PATH_site . '/../vendor/net/http/src/');
$composerAutoloader->add('Illuminate\\Support', PATH_site . '/../vendor/illuminate/support/');
$composerAutoloader->add('Arg\\Tagcloud', PATH_site . '/../vendor/arg/tagcloud/src/');

// register autoloading
$composerAutoloader->register(TRUE);
// snap
Using Composer's autoloader from a TYPO3.CMS extension

Odds‘n‘ends

Extension isn‘t available after running Composer update + build

Currently it‘s not possible to install TYPO3.CMS extensions without using the backend GUI. Just run your build and open the Extension Manager for installing the extension. Keep in mind to locally sync the PackageStates and LocalConfiguration files which you should keep in VCS in a proper data/ directory.

Copying / installing TYPO3.CMS - whoot?!

Unfortunately the system must reside in the document root directory (web/ - dist/ - whatever). But the benefit of using TYPO3.CMS as a component (eg. an update process is done with just updating composer.json, git push and starting the release process - QA is run automatically!) is bigger than the copy process. For the future I‘d like to symlink the typo3/ directory and copying index.php as the „Front Controller“ into web/.

The problem, TYPO3.CMS not implementing the Front Controller pattern, is luckily under investigation. As Helmut Hummel states, he worked out a solution:

Comments

comments powered by Disqus