Friday, October 28, 2016

Usability improvements for Firefox automation initiative - Status update #8

NEW:
* Debugging on remote workers is not a main priority for this quarter since it got completed

* We've added investigating hyper chunking

On this update we will look at the progress made in the last two weeks.

This quarter’s main focus is on improving end to end times on Try (Thunder Try project).

For all bugs and priorities you can check out:

Thunder Try - Improve end to end times on try
---------------------------------------------

Project #1 - Artifact builds on automation
##########################################

Accomplished recently:
  • All --artifact build jobs now provide crash-reporter symbols. This means test jobs scheduled against linux64 debug work with --artifact.

Upcoming:
  • Patch in review - debug artifact builds on try. (Right now --artifact always results in an opt artifact build.)

Project #2 - S3 Cloud Compiler Cache
####################################

Accomplished recently:
  • Green builds on all platforms on Try, planning to compare build times vs. existing Python sccache now

Project #3 - Metrics
####################

Accomplished recently:

Upcoming:
  • Still has the “small population” problem, where outliers make the 90th percentile look big.
  • Add a TaskCluster view of End-to-End times
  • Start knocking off bugs in the dashboard

Project #4 - Build automation improvements
##########################################
Nothing new for this edition

Project #5 - Hyper chunking
###########################

Nothing new for this edition

Project #6 - Run Web platform tests from the source checkout
############################################################

Feature blocked on misconfigured EBS volumes in AWS (bug 1305174)


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Wednesday, October 12, 2016

Usability improvements for Firefox automation initiative - Status update #7

On this update we will look at the progress made in the last two weeks.

A reminder that this quarter’s main focus is on:
  • Debugging tests on interactive workers (only Linux on TaskCluster)
  • Improve end to end times on Try (Thunder Try project)

For all bugs and priorities you can check out the project management page for it:

Status update:
Debugging tests on interactive workers
---------------------------------------------------

Accomplished recently:
  • No new progress

Upcoming:
  • Android xpcshell
  • Blog/newsgroup post


Thunder Try - Improve end to end times on try
---------------------------------------------

Project #1 - Artifact builds on automation
##########################################

Accomplished recently:
  • The following platforms are now supported: linux, linux64, macosx64, win32, win64
  • An option was added to download symbols for our compiled artifacts during the artifact build

Upcoming:
  • Debug artifact builds on try. (Right now --artifact always results in an opt artifact build.)
  • Android artifact builds on try, thanks to nalexander.

Project #2 - S3 Cloud Compiler Cache
####################################

Some of the issues found last quarter for this project was around NSS which also was in need of replacing. This project was put on hold until the NSS work was completed. We’re going to resume this for Q4.

Project #3 - Metrics
####################

Accomplished recently:

Upcoming:
  • Figure out what to do with these small populations:
    • Ignore them - too small to be statistically significant
    • Aggregate them - All the rarely run suites can be pushed into a “Other” category
    • Show some other statistic:  Maybe median is better?
    • Show median of past day, and 90% for the week:  That can show the longer trend, and short term situation, for better overall feel.

Project #4 - Build automation improvements
##########################################
Accomplished recently:
  • Bug 1306167 - Updated build machines to use SSD. Linux PGO builds now take half the time

https://bugzilla.mozilla.org/show_bug.cgi?id=1306167#c11

Project #5 - Run Web platform tests from the source checkout
############################################################
Nothing to add on this edition.


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Wednesday, September 28, 2016

[NEW] Added build status updates - Usability improvements for Firefox automation initiative - Status update #6

[NEW] Starting on this newsletter we will start giving you build automation improvements since they help with the end to end time of your pushes

On this update we will look at the progress made in the last two weeks.

A reminder that this quarter’s main focus is on:
  • Debugging tests on interactive workers (only Linux on TaskCluster)
  • Improve end to end times on Try (Thunder Try project)

For all bugs and priorities you can check out the project management page for it:

Status update:
Debugging tests on interactive workers
---------------------------------------------------

Accomplished recently:
  • Fixed regression that broke the interactive wizard
  • Support for Android reftests landed

Upcoming:
  • Support for Android xpcshell
  • Video demonstration


Thunder Try - Improve end to end times on try
---------------------------------------------

Project #1 - Artifact builds on automation
##########################################

Accomplished recently:
  • Windows and Mac artifact builds are soon to land
  • |mach try| now supports --artifact option
  • Compiled-code tests jobs error-out early when run with --artifact on try

Upcoming:
  • Windows and Mac artifact builds available on Try
  • Fix triggering of test jobs on Buildbot with artifact build

Project #2 - S3 Cloud Compiler Cache
####################################

Nothing new in this edition.

Project #3 - Metrics
####################

Accomplished recently:

  • Drill-down charts:

  • Which lead to a detailed view:

  • With optional wait times included (missing 10% outliers, so looks almost the same):


Upcoming:
  • Iron out interactivity bugs
  • Show outliers
  • Post these (static) pages to my people page
  • Fix ActiveData production to handle these queries (I am currently using a development version of ActiveData, but that version has some nasty anomalies)

Project #4 - Build automation improvements
##########################################
Upcoming:


Project #5 - Run Web platform tests from the source checkout
############################################################
Accomplished recently:
  • WPT is now running from the source checkout in automation

Upcoming:
  • There are still parts in automation relying on a test zip. Next steps is to minimize those so you can get a loner, pull any revision from any repo, and test WPT changes in an environment that is exactly what the automation tests run in.

Other
#####
  • Bug 1300812 - Make Mozharness downloads and unpacks actions handle better intermittent S3/EC2 issues
    • This adds retry logic to reduce intermittent oranges


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.