XHProf came from Facebook

Only does profiling
Made to run in a live site. Very little to no performance hit. Lightweight.
PECL library – XHProf (Easy to setup)
Brian Mercer php5-xhprof maintains PPA
PECL package is broken so you can’t sudo pecl install
1. Install the extension
2. Get the XHProf ???? from github
3. Devel module for Drupal

Now every page has the link at the bottom
Click it and see the profile data

Uses Access devel information so only those with permission see the link
He leaves enabled all the time

Inclusive Wall time: Generally what we need to look at. Includes wait time.
CPU time: only time to execute.
Exclusive: Only the time for that one call. probably want to sort by this.

If you see module_implements at the top… good indication you have too many modules. There is a pressflow branch to resolve this in D6. Use Catch’s version

** Environment Indicator module !!!! Use it. http://drupal.org/project/environment_indicator
$conf['environment_indicator_text'] = ‘DEVELOPMENT SERVER’;
$conf['environment_indicator_color'] = ‘dark-red’;
$conf['environment_indicator_enabled'] = FALSE;

Views cache plugin. Use as an example and write your own as needed. This is a huge performance help on sites with lots or large views.

Turn off dblog and replace with syslog. Big help

Working on Drupal UI version to review the XHProf data. Look for in future in the XHProf module project.
Also starting writing views 3 plugin for XHProf
Probably won’t be available for a while yet.

Drush enabled. You can profile drush runs for profiling s well. Link provided at the end of the run.
Drush migrate – can be long running migrations (hours)

http://jefflinwood.com

Drupal + mobile architecture
jQuery Mobiles + PhoneGap demo
Configuring Drupal 7 services (Service v3)

Native – Can use all the phone hardware as needed if it exists.
HTML 5 – Mobile framework (sensa touch, jQuery mobile) No hardware
Hybrid – Phonegap – wrapper that does allow access to some hardware devices. App store – Phonegap | Titanium – cross platform

Drupal – Provides, user management, permission/roles, CMS, etc… already. Leverage it
Services module
REST/JSON

Plist server module – Plist is native for iPhone but can also do JSON like Android does also.
Pick what makes sense. Don’t just say “everything” needs to be in the app

Caching and offline access
Cache data in SQLite
Cache JSON on the file system
Some functionality will be online only.

Configuring Services 3 in Drupal 7
Enable Serivces+REST server
resTt_server_plist
..
…

localhost/blog/node/1/json – blog=endpoint , node=resource, 1=nid

drupal-ios-sdk
drupalcloud – not as well maintained (android)
PhoneGap – immature right now but moving fast
jQuery Mobile – immature but moving fast

Photo By loop_oh

Two of our largest sites have run on Drupal since 2007. That was Drupal 4.7 for those who knew Drupal in those days. From day one, these sites allowed for online donations from anonymous users. Since we started on Drupal 4.7, LOTS of custom code was written. On Drupal 5 we began using the Secure Pages module to help the switching between HTTPS and HTTP for various URL paths and such. This wasn’t without it’s share of bugs which much has been fixed. Today we are on Drupal 6 but we’ve also grown up. Our hosting is now across multiple servers, dedicated DB server, memcache, etc… These performance and scalability changes has seemed to highlight some additional challenges with secure pages.

This is not a bash on the secure pages module, however it has caused me to think about architecture and strategy. See, these sites are not the typical “Here’s our non-profit and here is a form you can give donation through.” Instead we have 50+ different giving forms for different programs, partnerships and experiences with more being creating weekly it seems. Forcing a user to create an online account in order to donate is not an option. We have recently implemented integration between SalesForce.com and these Drupal sites. This integration with the CRM solution means that in short order, users on these Drupal sites will be able to see information about their partnership, contributions, pledges, update their contact information, make more decsions about how they interact and partner with this organization and more. Anticipated traffic spikes are going to require us to add Pressflow and varnish to our configuration in the near future.

All this to say that there is a definite need to secure a number pieces of these sites, but the majority of content is still really anonymous user content. So here is my question to the community of medium to large sites on Drupal that have the need for mixed HTTPS/HTTP traffic:

What’s the best recipe in Drupal 6 & 7 for ensuring certain, but not all pages, including login, user and other content types or paths, are delivered via HTTPS while the rest of the site is HTTP?

Actually wondering if a better approach is to have anything needing SSL to be delivered from a subdomain (secure.mydomain.com) that points to the same Drupal instance.

Curious how others deal with this for client projects. What’s your experience been?

Photo by The Library of Congress

[video of presentation]

Contextual Admin module – http://drupal.org/project/context_admin
Provide plastic, plugin driven, context aware administration tool
Wrapped in flexibilty of page manager
Exportable, Reusable, features compatible

Start with View inside a panel
VBO to administer other nodes of the same context
Customer: If I want to do something in a section, I just go to it.

This can allow users of a role the ability to add others users of a certain role. Drupal can’t….context admin can

Have an article, with now an add image tab, add a new image node, save it returnng to the article. Node reference automatically added as well.

In context admin settings.
Access = who can see the link
Selection = what they can see when they get here

Excellent presentation~! Lots of potential here for much more user friendly way of managing content but got to think ahead.

OpenPublish – Drupal 6.x
Tattler is on it’s way out. No business model for it.
OpenPublic – Drupal 7
Open Atrium – Acquired from Developmneet Seed
Managing News – Acquired from Development Seed

Companies that only do dev work are at risk.
Can only make as much money as code you can develop, hours you can charge.
Move from a VAR into also being an ISV.

Phase 2 Technology is the maintainer of the features module

Features Module
Kit Compliance module – Basic set of standards to ensure distributions, apps from cause problems with eachother when used together

All items relate to the ongoing support and development for http://www.economist.com/

Front page components:

• Frontpage made up of lots of pieces and parts
• Hero box.
• Each item in each area of FP is individually selected. Admin is a single place tabbed for each section so easily managed to “build the page”

London – New York – Austin

• Mailing list (Team), IRC (Team), Skype (1 to 1)
• Developers / Core Committers
• 1 Designer
• SCRUM manager

2 week sprints. 15 minute stand-up daily. (What I did yesterday, What I’m doing today, any impediments)
Demo what is done to customer.
Code review, Tested, Deployed.

Use Google Docs for multi-organization teams as the SCRUM tool. Four Kitchens uses JIRA internally.
Defined single person that is the final approved/decline of any feature request. (Product Owner)

The Most X Box (Most commented, Most recommended)

• Too query intensive to run on every page load. Ran in cron, cached every x minutes and then use the cache for display.
• No longer using it this way as it has gotten more intelligent.
• Jenkins runs jobs that pre-load cache.
• User page loads only every get this box content from cache. Never sets cache.
• If no cache, blank box.

Dev Ops

• Branching – BZR -
• Every feature starts in a feature branch Use launchpad to manage.
• Trunk -> Stage -> Live
• Fridays – Promote to stage. Teams signoff via email
• Tuesday – Push live.

Testing

• Selenium – More features than simpletest
• Wow! – Runs in a headless Ubuntu X-Window recorded with FFMPEG so you actually see what was happening on a failure.

Vendor Branching

• Each contrib module has a vendor branch
• Pristine version of contrib as base and changes tracked.

Topic trees

• At least ten pieces of content for a topic before it will be vailable as a topic.

Jenkins

• Web based system for automating builds. Used to be called Hudson.
• Now can do other things.
• Make calls to drush cron
• Take your imports and such and make them as drush cron calls
• Drupal CRON runs in order of module weight. If a module cron run fails, everything after will not run
• Database download and sanitize user data.

Performance

• Caching
• Varnish front-end (Pressflow for Drupal 6 or Drupal 7)
• Logged in session cookie, Varnish is setup to not serve cached is user is logged in.
• Database
• MySQL replication
• All write queries to master (Pressflow)
• reads are from one of the replicants

Migration

• Continual Integration
• All stories created in CCI – Oracle
• Custom code pulls from Oracle and then creates/updates nodes in drupal

They wrote menu_callback_cache as finer grain control of caching in drupal

Ran across this presentation from NorthPoint recently that I thought did a good job of summing some of the current state of implementation and thinking:

Just thought I’d put this out there for my personal reference and possibly your intrigue.

Featured image from: http://www.mobiledrupal.com

On our production boxes we installed our production site as well as a copy of production for staging purposes. For us, this is used for final QA testing prior to production deployment. This is our low tech solution to ensuring that staging is on an identical configuration as production to ensure no gotchas with slight differences in server builds, library versions, etc…

Things had been humming along nicely for about a week when a staff member came rushing into to my office to show me that the production homepage had the tag “STAGING SITE” in the site title. My heart dropped as my mind immediately went to “oh no, the production db is pointing at the staging DB! Has it been that way for a week…. ARGH!!” After a few frantic minutes of checking Drupal and apache site configurations, I realized that the worse case scenario was not the issue. Phew. So it got me thinking that this must be cache related, but how would cache cross sites? Immediately the new piece of memcache came to mind. We are running a single instance for all bins initially.

Doh! That had to be it. So a little Googling and I found this issue in the memcache module queue that was closed. Double Doh! All I had to do was read the README.txt file and the answer was staring me in the face. The cache keys are not always created uniquely with identifiers including URLs/Hostnames nor absolute files paths. So as an example telnet to your default bin.

>telnet 127.0.0.1 11211
>stats items
>stats cachedump 1 100

Let’s look at these. The stats items command lists all the slab and statistics about the items in each slab. Pick a slab number that has some items in it. The slab number is to the right of :items:“. The slab number is 37 in the example below. It indicates there are 3 items in the slab.

STAT items:37:number 3

Now if we want to see the contents of the items in the slab we issue the stats cachedump <slabnumber> <maxitems> command.

>stats cachedump 37 100
ITEM cache_views-views_data%3Aen [276200 b; 1307993360 s]
ITEM cache-variables [285776 b; 1307993360 s]
ITEM cache_content-content_type_info%3Aen [235095 b; 1307993360 s]
END

So here we see that in slab 37 there are 3 items. One of those items is cache-variables. Now here is where we have to understand. Drupal has created a cache item with a key of cache-variables. That key has nothing unique about it to let anyone know which of the many drupal sites running it is for. If more than one Drupal site is using this same memcache bin, then it will get/set the same cache item for the variables of the site. Whichever site gets to set this cache item first, the second or third site will use the variables cached by the first site.

In our case, the staging site was hit and set this cache item. Our production site then looked to see if a cache item with the same key existed. It did and so the production site used those values to help generate the requested page. This explains why we saw the site title including the STAGING SITE text on the production site.

Let’s get back to the README.txt file in the memcache module. It very explicitly explains that when running multiple sites in the same bin that you need to use the memcache_key_prefix configuration variable in the settings.php that uniquely identifies each site.

$conf['memcache_key_prefix'] = 'staging_';

Now when I look at the slab containing the cache-variable key, we’ll notice it is prefix with “staging_” which now only the staging site is going to look at. Adding a different prefix configuration to the production site ensures that it too has unique cache item keys.

>stats cachedump 37 100
ITEM staging_-cache_views-views_data%3Aen [276200 b; 1307993360 s]
ITEM staging_-cache-variables [285776 b; 1307993360 s]
ITEM staging_-cache_content-content_type_info%3Aen [235095 b; 1307993360 s]
END

This is obvious now having found the problem and solution however it definitely was one of those “Doh” moments. Hopefully this will help someone else.

Blessings

Drupal Commons (Connect theme homepage screenshot) - BrightLemon

For those who are familiar with Drupal, you are probably also familiar with Acquia, the company created by Dries Buytaert, Jay Batson and others. Acquia has worked to create an “installation profile” called Drupal Commons as an out of the box social community building site.

Drupal Commons is ready-to-use social software that provides familiar tools from the social web to help you create positive buzz, brand loyalty, and customer satisfaction in your community.

Looks like there is a fairly substantial dev release in the works unlike the previous, primarily security update focused releases. According to the release notes, it looks like the next version 1.6 will include a number of substantial changes. One of the most striking changes is the appearance of a new default theme. This new theme appears to be thanks to a partnership between Acquia and BrightLemon. I for one, love the new theme as it provides a much more visually appealing/compelling look when showing Drupal Commons to customers or users as an “out of the box” experience before discussing what is then possible via templates and themes.

Additionally, Context has fully replaced Panels module. In previous versions, the home page, user profile, and group home page were rendered by Panels but now will retrieve display components using Context. Apparently Context gave a few more features and using both was causing confusion.

NOTE: This is a DEV release. As Acquia warns, do not use it as a production implementation. There is currently no upgrade from previous versions, however Jay Batson confirms that an upgrade path will exist for the final release of this version.

Installation profiles provide site features and functions for a specific type of site as a single download containing Drupal core, contributed modules, themes, and pre-defined configuration. They make it possible to quickly set up a complex, use-specific site in fewer steps than if installing and configuring elements individually. For example, an installation profile for conferences might include modules for advanced user registration, attendee profiles, ticketing, and a list of speakers and sessions. Once installed, installation profiles can be configured and customized the same as traditionally-built Drupal sites. Refer to the installation profile documentation for more information.

Now, one of the biggest challenges is that there are multiple instances of the “image_attach” sections on a single content edit form, one for each category. The Drupal 5 version handled this by naming these by appending the category name to the fields. This allows the submitted form value for “iids” to be an array of categories. Each category itself being an array of image iid values.

For the life of me, my Drupal 6 version of this functionality failed to maintain the category arrays. After some frustrated hair pulling (hmmm…. no wonder my hair is so short), I remembered to take a look at the FAPI docs.

http://api.drupal.org/api/drupal/developer–topics–forms_api.html/6

An important thing to note: notice that $form['access'] has a '#tree' => TRUE attribute. this setting retains the full tree structure for all elements under it when it is passed to $form_state['values']. you must explicitly declare this anywhere you wish to retain an array’s full hierarchy when it is passed.

So, Drupal 6 collapses form elements where possible by default. By adding the #tree attribute, the hierarchy of form elements is maintained in the form_state submitted. Eureka!

Here’s an example of what I was getting without the #tree attribute:

And now with the #tree attribute set to true, the form hierarchy is retain and not collapsed:

I’m sure most of you doing Drupal development are saying “Duh!”. Well, let’s just say I really didn’t know. Thought I’d throw it on the blog so I don’t forget.

Blessings.