Categories
Uncategorized WordPress

WordPress Automated Testing Next Gen

Thinking more about next steps for my half-baked idea for automated WordPress testing. So far, I’ve had multiple hosts reach out with nothing but positive feedback. The next step in my eyes is to come up with some next steps, proposing this as a featured project, and then building the pieces and recruiting hosts.  In order to really be successful, it’s not just the big hosts that we need to reach out to (most of them already follow WordPress core development), but the long tail of hosts and places that self-host. In my eyes, here is an initial brain dump of the tasks (and sub tasks) needed.

  • Test Runner and Reporter
    • We need a script that runs the tests and formats the results for reporting purposes. This script needs to be able to detect and/or know information about the environment along with information about the results of the tests (passed, failed, errored, skipped) and the revision this was run against.
    • The script needs to report the results to a centralized source. This could include tracking test length time, slow tests (to see if certain tests are slow under certain circumstances).
    • This likely needs to be written in PHP which is the only language we can be sure is on a WordPress server.
  • Test results receiver.
    • We need a place that results can be sent to. This will need to store the results of each test.
    • This could possibly be done by creating a WordPress plugin with a CPT for test results, along with the Rest API Infrastructure in core for sending in the results.
    • Hosts need to be able to register to send in results.
      • The barrier here should be as low as possible. We want people that self host to be able to easiely participate.
    • If a host doesn’t send in results for a platform after X number of runs, there should be alerting of some kind to make sure they know it isn’t working properly (it could be on purpose, it could be an accident).
  • Test Results Viewer/Reporter
    • When a test fails, or passes a certain threshold (multiple runners reporting failure, multiple failures in a row for a single result), there should be a notification to the #core channel on slack.
    • Developers should be able to quickly scan for test failures.  1) We should be able to see which commits triggered multiple failures.  We should also be able to see which configurations are unstable (maybe there are changes to the unit test or core code to improve stability).
    • Some UI/UX design will be needed.
  • Documentation on the test runner needs to be written
    • This will help make it easier for people to contribute.
  • Outreach to hosts
  • Infrastructure for hosting the test results receiver and test results viewer. This could be wordpress.org (would simplify the slack notifications and logins), or it could be donated by someone else.

I think the next step is for me to propose this as a formal feature project. If accepted, I’ll setup a chat in #core to go over what needs to be done and see if people are intrested in volunteering to work on specific pieces. Here is how I would describe it for the feature project proposal:

Problem:  The current automated testing infrastructure doesn’t test a realistic matrix of where WordPress sites are hosted. What would we learn if the test matrix matched the hosting matrix?

Action: Exploring and building an infrastructure for automated testing that relies upon those that host sites.

Categories
Uncategorized WordPress

The variable pieces of the WordPress Stack

Yesterday I shared a half backed idea about having hosts help more with automated testing for WordPress. Continuing that thread, there are a handful of things that need to be figured out.  The first of which is what are the pieces of the WordPress Stack that we need to track and are the real variables. Here is my initial list (along with tickets for many showing when WordPress has been bitten).

  • PHP Version (all the way down)
  • MySQL Version (or MySQL variant)
  • OpenSSL Version. ( #30434 showed that different versions of OpenSSL respond differently to the order that SSL Certificates are presented)
  • ImageMagick version.  #36501 demonstrates that we can have issues that only show up in certain versions of ImageMagick.
  • Curl version. [37430] demonstrates this necessity.
  • PHP Extensions installed. #36629

What else?  I’m sure there are things that I’m missing.  What variable in the WordPress hosting stack have you noticed cause a bug?

Categories
Uncategorized WordPress

WordPress Testing: Hosts Matter

So this is a half-baked idea. Well, maybe 3/4 baked since I’ve mentioned it to a few people.

Right now, Automated testing in WordPress is based on Travis CI.  Which works pretty well.  Tests are run on 1 OS vs 8 versions of PHP (plus hhvm, but those don’t pass) which based on my completely unscientific estimates covers less than 0.1% of the actual WordPress userbase.  If WordPress really wants to do quality automated testing, we need to rely on the people hosting sites to test vs their stack. To do that, Core needs to provide an infrastructure that both encourages and enables easy automated testing.

It’s not that the current test matrix is bad, but it currently is only covering a small corner of the true test matrix.

What I would like to see is an API that hosts (both shared and dedicated) could use to register their unique setups for automated testing. Using PUBSUBHUB,  WordPress Core would signal that a change has taken place.  Then, the hosts could run their setup(s) vs the new version of trunk.  They could run both a plain version of trunk, but also perhaps the version that includes everything they install as a part of a new site and whatever “one click installer” they use.

Between versions and configurations of OpenSSL, Imagemagick, PHP, MySQL, MariaDB, CURL, and the operating system I wouldn’t be surprised to find out that WordPress runs on well over 100,000 different stacks. The only way for the automated tests to cover that is if the community participates.

 

Categories
Uncategorized WordPress

What You Don’t See on my WordPress Profile Activity Tab

A lot goes on in WordPress Core development that doesn’t make it into the WordPress Profile Activity log. Here is a handful of the things I’ve worked on over the last month to give you an idea of what I have worked on. Many other contributors work on things that also don’t make it into the report, so it’s import to recognize that this screen is far from a good recognition of effort put into WordPress.  This isn’t an exhaustive list (I wish I could remember everything).

  • Organize the Field-Guide posts.
    • Work with each author to help make sure they know the importance of the posts
    • Review drafts to ensure the posts have quality information and wording is clear.
  • Organize sending the email to all plugin authors about 4.5. This is worked on by the Core, Plugins Repo, and Meta teams.
    • Work with Polyglots team to help publicize Global Translation Day.
    • Work to help measure the impact of this email (so we can see if there are ways to be more effective in the future)
    • Start documenting how this happens to reduce the bus factor
  • During releases, remind people to not share the link to packages
  • Provide some mentorship to a guest committer
  • Review the credits list before the release
  • Review News posts announcing release
  • Draft Release Haikus (This is one of my favorite contributions)
  • Help document release activities and steps (aka, the release checklist)
  • Work with the Forums team on the OMGWTFBBQ! post
  • Review and test patches for other committers to commit
  • Help brainstorm the About Page and the release tagline
  • Attend weekly dev chats and contribute during those
  • Early-stage discussions around ways to help make core better serve the readers and commenters
  • Participate in the first Feature Projects chat.
  • Test various parts of WordPress
  • Work to Recruit new contributors
  • Be a sounding board for ideas from other contributors

As anyone who has ever worked on a project at scale knows, coding is just a small piece of the puzzle. While the work of WordPress generally happens in the open, it doesn’t always get logged.

Categories
Uncategorized WordPress

Job Hunting Tips for WordPress Engineers

I decided to setup a Ask Me Anything repo on github, where people can ask me anything they want (go ahead, ask). The second question asked was about WordPress Jobs.  While the response was aimed at the asker, much of the advice is general.

Let’s step back for a second, what does it mean to be a WordPress Engineer? I personally don’t want to work with people that paint themselves into the corner of just knowing WordPress. I like T-shaped people. Have you worked on things other than WordPress? Your resume doesn’t list much else technical. Are you most comfortable in PHP, JavaScript, or something else? Have you done much (or any) systems/server work? Resumes are about helping you prepare the interviewer to ask you good questions.

When finding jobs (and deciding between jobs), The question I often asked is “How will this prepare me for the job I want after it?”. Make sure that the position is going to help you become qualified for a job you want that you aren’t qualified yet.

Read the full response.

Categories
Uncategorized WordPress

Decisions, not options (suggested readings and watching)

Something incredibly common in every discussion about changing a default behavior in WordPress is someone suggesting “Just make it an option”.  WordPress has a core philosophy  “Decisions, not Options“.  This philosophy is one that is important for all software and is explained and enumerated over and over.  Here are some of the places that exemplify it best:

Dave Winer tweeted a photo of a weird, verbose, and confusing Android options screen. I love Android, but like most open source projects, it falls victim to the proliferation of options, rather than making decisions for its users.

When explaining this to developers at conferences, I generally mention the preference panels in Adium, a Mac OS X chat client. It practically has an Advanced tab for the Advanced tab. Stuff everywhere. Problem is, when there are too many options, your users can’t find any of them.

In Open Source, Learn to Decide – Andrew Nacin

The best UX will often be no UI at all: if the customer gets your product and their problem goes away instantly with no further steps, that’s amazing. Failing that, less UI beats more UI; and for many customers and problems, a traditional GUI won’t be right. Alternatives: command line, voice recognition, gestures, remote control, documentation, custom hardware, training, …

The Dangerous “UI Team” – Havoc Pennington

Preferences really substantively damage QA and testing. As someone who reads dozens of bug reports per day, and occasionally fixes a couple, I can tell you that it’s extremely common to find a bug that only happens if some certain combination of options are enabled. As a software developer, I can tell you that preferences frequently do add quite a bit of code complexity as well; some more than others, but even simple preferences can add a lot of complexity if there are 200 of them.

Upshot: more preferences means fewer real features, and more bugs.

Free Software UI – Havoc Pennington

note: I’ve referred to Havoc Pennington as the most influential person in the History of WordPress who has never been directly involved in the project and it’s essays like this that demonstrate it. 

One of the most important speeches at any WordCamp about UI, UX and development in general.

My thoughts are summed up in a tweet I made earlier today.

Categories
Uncategorized WordPress

The Future Stack: Running WordPress with Tomorrow’s Technologies Video

I have a longer post coming, but for now this is the video of the WordCamp US talk that Zack and I did.  Want more info? The #wpFutureStack hashtag on Twitter has it.  Add your resources to it.

Categories
Uncategorized WordPress

PHP 7 and WordPress hosts: An informal survey at #WCNYC

At WordCamp NYC this weekend I decided to do some impromptu research about how prepared different web hosts are for PHP 7 which is scheduled for release on the 12th of November. Without identifying myself or my role, I talked with most of the hosts present. I said hi and asked “What is your plan for PHP 7 support?”. For the hosts that gave poor answers, I’m not going to name names since it likely is just an education issue. The folks that attend events representing hosts are rarely the people with direct knowledge of what is in the pipeline.  For the two that gave great answers, kudos.

Dreamhost and WP Engine both gave me similar and solid answers: They have been doing a lot of internal testing of PHP 7 and while they don’t have a date or exact roll out plan yet, it is in works. This is a solid response in my book. Committing to a date is a hard thing to do. I’m glad these hosts have already started work and are doing testing on pre-release software.

Other hosts had less than stellar responses. These included:

  • Having no idea and taking my info down to get back to me.  I’m glad that they are willing to follow up. I’m ok with this answer (as long as they do follow up).
  • “I don’t know. What should our plan be for PHP 7?” This was less than ideal, but showed a willingness to learn and hopefully they take the information back and start working on a plan.
  • Stating that they weren’t sure when they were going to update since they had problems when they updated to PHP 6. I’m unsure why they had issues since PHP 6 was never released.
  • Asking if I meant PHP 5.7. There is no planned release numbered 5.7

I might try again at WordCamp US.  I wonder if the answers will be different?

Categories
Uncategorized WordPress

Automated Code Coverage of WordPress Core PHP code

Scott Taylor offered me “$3-4 million” to set up automated code coverage for WordPress core. While it is a bit below my usual hourly rate, I decided to give Scott a to celebrate WordPress 4.4 Beta 1 and did it for free. Running codecov as a part of the normal travis-ci runs wasn’t really an option since it can take an hour to do the phpunit run with codecoverage turned on and the builds are long enough as is, so I decided to setup a new script with a new travis config. This new config is largely a cutdown version of the current travis config combined with running phpunit with code coverage turned on.  An hourly cron checks to see if there have been any commits and if so, replaces the travis.yml file with the code coverage version and pushes that to github.  The two files are below.

Overall, it was a quick hacky way to hopefully help WordPress core contributors get a better sense of where unit tests are needed.  You can see the results for yourself. 

Categories
About Aaron Uncategorized WordPress

Fall Conferences – PHP Madison and WordCamp NYC

Today I get to announce two conferences that I’m speaking at this fall. The first is here in NYC and is the 2015 WordCamp NYC.  I’ll be giving a talk entitled:
Lessons from Science Fiction and Fantasy we can use in Creating Websites.  Here is a short synopsis.

Science Fiction and Fantasy can teach web creators many valuable lessons. From seeing how Daleks with too narrow of a goal always fail to understanding the Klingons value of honor, to hundreds of other we can become better web creators by borrowing lessons from Science Fiction and Fantasy.

Next, I’ll be traveling to Madison, Wisconsin for the first time in almost 10 years to present “How Not To Build A WordPress Plugin” at Madison PHP.  A short synopsis of this talk is

WordPress has a powerful plugin architecture that enables you to build almost anything on top of WordPress. This power though can lead to anti-patterns that slow down sites, confuse users, and make it hard to scale. Let’s look at the wrong way of building plugins so you can avoid these traps.

Tickets for both events are on sale.  If you are either one, make sure to say hi!