drupal

Ansistrano, deploying Drupal with Ansible

Last months have been pretty busy here at BBC Worldwide with some of the projects we've been involved in.

In my case, one of the main areas I’ve been trying to improve (apart of the daily busy routine) is our devops / CI approach. The previous approach was using Chef, but it was not really adopted by the developers (which meant that after some months, the system was broken) and we were lacking some fundamental things (like integration with the production system, a real CI flow, etc…).

On the other hand, my previous experience with puppet was that the system grew too much to be easily maintained, specially if you don’t have a big dedicated devops team. To sum up, during last months research I was pretty amazed with Ansible, it's amazingly simple, very flexible and a truly joy to maintain. So, to me, the way forward was Ansible.

So, we started rewriting some of the old broken features in Chef and, sooner than later, we found ourselves reinventing the wheel. Some features were custom things, like triggering "drush cc all”, executing grunt, phpcs, etc... but obviously lots others were not… well, we are not the first team in needing an apache and mysql database, are we?

And deployment is one of these things that I was not too keen to reinvent. It would be pretty simple with Ansible, to be honest, but why not use something that is already tested, works well and at the same time give some love back to the community? Enter Ansistrano.

Ansistrano is a deployment tool based in Ansible. The relation with Capistrano ends just here, with the name similarity (no code base was harmed during the process). For more information about Ansistrano please visit the official repository: https://github.com/ansistrano/deploy

- Step 0. Install ansistrano (deploy and you’ll probably be interested in the rollback too):

ansible-galaxy install carlosbuenosvinos.ansistrano-deploy carlosbuenosvinos.ansistrano-rollback

- Step 0. Clone the deploy repo in the folder you’ll be using the app. Let’s say we’ll be working in ~/my-app-build :

git clone https://github.com/ansistrano/deploy.git ansistrano-deploy

Notice that we are cloning the deploy repository.

We’ll be using the latest stable release, list tags:

vagrant@vagrant-ubuntu-trusty-64:~/my-app-build$ git tag
1.0.0
1.1.0
1.2.0
1.3.0
1.4.0
1.4.1

at the moment, the latest stable release is 1.4.1, so:

git checkout 1.4.1

Step 1. Now, the ‘tricky part’. We are going to write a file where we’ll execute all the magic. Let’s call it build-my-app.yml:

---
- name: Deploy example app to local.myapp.com
  hosts: all
  vars:
    ansistrano_deploy_to: "/var/www/myapp.local"
    ansistrano_keep_releases: 3
    ansistrano_deploy_via: "git"
    ## GIT pull strategy
    ansistrano_git_repo: git@bitbucket.org:myapp-in-git.git
    ansistrano_git_branch: develop
    ansistrano_git_identity_key_path: ""

  roles:
    - { role: ansistrano-deploy }

Notice that the role we have at the end is just the deploy repository cloned from Ansistrano (https://github.com/ansistrano/deploy). You can change the name, as I've done in this example (ansistrano-deploy) but you'll have to upgrade the roles variable in your yml (again, as shown in this example).  

Step 2. Run deploy:

ansible-playbook -i "localhost," -c local build-my-app.yml -vvv

Instead of specifying localhost you could also use a hosts file, as recommended in the official documentation. This is also going to give you the freedom of deploying your app not just in your local environment, but potentially in any remote server, i.e. aws, digital ocean, etc...

Step 3 (optional). From here, the limit is the sky. In our case, we have some complex deployments running grunt, drush, clearing cache… The advantage here is that ansible is pretty flexible, so you could for example add your own tasks in your deploy file (build-my-app.yml), i.e.:

tasks:
  - include: goodfood/tasks/apt.yml

The right, official way of doing this is using the hooks that ansistrano itself offers you:

  # Hooks: custom tasks if you need them
  ansistrano_before_setup_tasks_file: "{{ playbook_dir }}/<your-deployment-config>/my-before-setup-tasks.yml"
  ansistrano_after_setup_tasks_file: "{{ playbook_dir }}/<your-deployment-config>/my-after-setup-tasks.yml"
  ansistrano_before_update_code_tasks_file: "{{ playbook_dir }}/<your-deployment-config>/my-before-update-code-tasks.yml"
  ansistrano_after_update_code_tasks_file: "{{ playbook_dir }}/<your-deployment-config>/my-after-update-code-tasks.yml"
  ansistrano_before_symlink_shared_tasks_file: "{{ playbook_dir }}/<your-deployment-config>/my-before-symlink-shared-tasks.yml"
  ansistrano_after_symlink_shared_tasks_file: "{{ playbook_dir }}/<your-deployment-config>/my-after-symlink-shared-tasks.yml"
  ansistrano_before_symlink_tasks_file: "{{ playbook_dir }}/<your-deployment-config>/my-before-symlink-tasks.yml"
  ansistrano_after_symlink_tasks_file: "{{ playbook_dir }}/<your-deployment-config>/my-after-symlink-tasks.yml"
  ansistrano_before_cleanup_tasks_file: "{{ playbook_dir }}/<your-deployment-config>/my-before-cleanup-tasks.yml"
  ansistrano_after_cleanup_tasks_file: "{{ playbook_dir }}/<your-deployment-config>/my-after-cleanup-tasks.yml"

where playbook_dir is the folder where your yml file lives.

That's it. Simple, elegant, and really powerful, I love it :-). Next episode, we'll talk about the rollback artifact.

Notes: Amended Step 0, as you don't need to install the playbook through the ansible galaxy, just cloning would do. Thank you Carlos Buenosvinos.

Profiling PHP and Drupal with XDebug (In OSX)

Problems with a rogue app you built (maybe not you)?

Just some links which I found useful in order to debug/profiling your app:

Run it in you Mac with OSX:

Sometimes you need your profiler to be executed everytime. You can do that with this:

xdebug.profiler_enable=On

This is my whole xdebug.ini:

; Enable xdebug extension module

zend_extension=/usr/lib64/php/modules/xdebug.so

xdebug.remote_enable=1

xdebug.remote_host=192.168.33.1

xdebug.remote_port=9000

xdebug.remote_mode=req

xdebug.remote_connect_back=0

xdebug.profiler_enable=On

xdebug.profiler_output_dir=/www/profiling_data

xdebug.profiler_enable_trigger=1

 

Other general links:

 

Using Symfony Dependency Injection in Drupal 7: Software Engineering in Drupal

I still remember in a previous job discussing with a partner about doing things the proper way, using coding standards, commenting what you are doing, ... giving your code some love in definitive.

I'm still amazed of his answer, basically he was refusing to doing things the right way (the only way) because using his way he was a lot quicker... Well, probably, but on the meantime modifying some of the crap we had there (not just him, I have to say) could take us like 4 to 10 times more time.

That's basically because you needed to understand what was going on that code... and I promise you, when you have a single function of around 3000 lines, no comments, things happening without too much logic, no standards, dozens of lines commented... that is not easy at all, even for the owners of that code. That's why you use Coding Standards, code reviews, Pull Requests and all that stuff in serious environments of Software Engineering.

Anyway, developing using the Object Oriented paradigm is quite a lot more organized than doing it using simply .modules and .inc files in Drupal. It does not matter how organized you are, at the end you'll end with lots of files and functions in a nightmare to understand for third developers. And, when you are working in Enterprise environments, with lots of people sharing and creating code, things like coding standards and doing things the proper way becomes a very serious thing.

Solution? As I said, using Object Oriented programming paradigm. In Drupal is easy enough since Drupal 7 allows you to add directly classes included in .php, .inc/* files. Simply do:

files[] = lib/MyClass.php

Well, that's easy enough... but there is a better more sophisticated way of doint it: Symfony Dependency Injection.

There are some good advantages of doing this that way. First and maybe most important, you are starting to be ready for Drupal 8. Second one, you objects can use Lazy loading. That's a great thing, basically means that your objects will be using the Proxy Pattern, which allows your code to load things only when they are going to be needed.

That's a good trick to improve your Drupal site performance from your own code (a big big percentage of performance problems are derivated from bad coding practices).

Want more advantages? Did I say that you make your code ready to Drupal 8? Well, that's just the begining. Starting to use Symfony components like Drupal Symfony inject open us a world of thousands of bundles ready to use from Symfony, and just using them through Composer. Win win :-).

We will need basically two modules, Drupal Symfony inject which a college is developing here, in Capgemini, and Composer Manager as it is a dependancy:

It is important you install them using drush, so the vendors folder will be created for you (with all the dependencies that the module has):

drush pm-enable drupal_symfony_inject --y

Once installed, DSI makes you available three new hooks:

  • _namespace_register()
  • _symfony_yaml_config()
  • _symfony_yaml_config_params()

The first one, _namespace_register, allows you to register your new classes like that:

Now we will need to add the config.yaml, where we will declare the classes we will be using, parameters, etc...

Now you need to define your classes in the yaml file. Create the file in config/yaml/config.yaml and declare there the classes you'll be using:

And finally, the parameters:

That's it, Drupal is ready to use the Symfony Dependency Injection component. Lets move to create a new class, for example, CruiseHunter:

That's one of the Abstract classes I use in CruiseHunter from which I inherit to fetch the different contents the Cruise crawler uses. It is a lot more organized, it declares a common mechanism to fetch information from the crawler, etc...

Well, now that we have the class, we need to be able to import it and use it. And here is where the Dependency Injection appears:

Once you have all in place, you have to rebuild the symfony components you have just built:

drush rsc

If you see some error, you made a mistake, probably in your yaml. If you see something like this:

Symfony DI container compiled and cached successfully !!!

Congratulations, you are ready to use your new Symfony components :-).

Related articles:

Changing Short project name in a Drupal.org project

Typical problem, you have just gained your full project permissions role in Drupal.org, and you make your first mess... you have promoted your so loved first project without changing the Short project name.

Well, you have a problem, but not too big. Drupal.org will not allow you to rename your Short project name, so you will have to create a new one if you don't want to have an ugly url like that: https://drupal.org/project/2062343

Fortunately git can help. You will simply have to:

  • Create a new project, sandbox or not,
  • Go to the "Version control tab" and get the url of your new git project, something like: programadoresweb@git.drupal.org:project/2062343.git
  • Go to the folder of your old sandbox, the project in which you cannot change the Short name, and change the remote for this folder. For my url example, the command will be: git remote set-url origin programadoresweb@git.drupal.org:project/2062343.git
  • Nearly finished. You probably have to merge, as you will have to make your first commit in the new project: git pull
  • And finally, git push origin master will get the job done.
  • Remember to create a new branch, like 6.x-1.x and commit to that one to create a new official, downlodable release of your module. Congratulations :-).

 

Next time, take it easy cowboy ;-).

 

More information:

 

Drupal form api select elements returning integers instead of string

I always have the same problem when using select forms, and I always forget the solution.

Instead of returning an array, like: return $competitions; you have to return a drupal_map_assoc, like this: return drupal_map_assoc($competitions);

Otherwise, when selecting the form in hook_submit you will get a number, instead of a human readable string.

categorias: 

Customize a webform with

categorias: 

Convert dates from UNIXTIME in mysql

Reading a date in mysql can be frustrating, unless your brain is able to convert Dates from UNIXTIME formats. Let's see for example this scenario in Drupal:

SELECT * FROM `node` ORDER BY `node`.`created` DESC LIMIT 0 , 30

If you need to fetch, for example, when it was the last time that a node was created, this format doesn't help too much.

The solution is quite simple, FROM_UNIXTIME( field_with_date )

For example:

SELECT * , FROM_UNIXTIME( created ) FROM `node` ORDER BY `node`.`created` DESC LIMIT 0 , 30

categorias: 

Deploying Drupal projects: git + capistrano

 

removing some folders from git, file .gitignore with these contents:

 

# Ignore paths that contain generated content.

cache/

files/

sites/*/files

sites/*/private

 

 

 

Using symbolic links for /files directory:

 

 

namespace:custom do

    task:symlink do

      run "ln -s /var/www/crucerista/sites/crucerista.net/files/ /var/www/crsta/current/sites/crucerista.net/"

      run "echo 'finished'"

    end

end

after "deploy","custom:symlink"

 

 

Related links:

Similar technique, but aplied to Rails apps: http://www.practicalecommerce.com/blogs/post/468-Creating-Symlinks-When-Deploying-a-Rails-Application

Another idea, File Transfer actions in Capistrano: http://www.simonecarletti.com/blog/2009/02/capistrano-uploads-folder/

 

Optimizing a Drupal database: improving the performance

That is what you have when you don't optimize and maintenance properly your Drupal projects. A 4GB database which hits hugely the server performance. Just a couple of "magic tricks" and .... 194MB.

Magic? Not at all. If you have a problem with your server, with hangs up and things going very sloooow, check the cache. sessions and watchdog tables, and clear them.

It works like a charm :-)

Drupal maintenance tasks

 

How important is having a database optimized. Doing some "basic" optimizations like clearing the cache, runing the Drupal cron and a couple of things more, your database can change from having 4 GB to just around 200MB... incredible? Not at all, the importance of maintenance tasks :-)

Which tasks? Mainly:

 

  • Clean / truncate the sessions table. This table can grow up to several millions of registers. 
  • Carefull with the spammers. Your comment table can grow up to similar amounts if too many spammers take a look at you. If the problem is big, simply delete with phpmyadmin:
  • delete from comments where status = '1';  // unpublished comments
  • delete from comments where status = '0';  // all published comments

 

 

Related @links: