Week notes - 2020 w13 - worklog - everything is broken - JOYK Joy of Geek, Geek News, Link all geek

everything is brokenotsukare Thoughts after a day of work

next Mozilla Toronto All Hands is canceled. That's a good thing.
Coronavirus had no impact on my working life for now. The same as usual.
1. Mozilla is working well in a distributed team.
2. Webcompat team is already a very distributed team. None of our team members are in the same location. So this is business as usual for us.
3. And as long as the economic impact is not affecting Mozilla, we will continue to work as normal. That will be a different story, if/when the economic crisis reaches us. We are for now a very privileged tribe.
4. In Japan, we do not have yet a lockdown. (This is silly, I think. People are having non responsible behaviors.)
  - Updates 2020-03-26: and here we are. Tokyo, Kanagawa, Chiba and Saitama provinces have asked people to stay home this coming week-end, because the numbers of cases are accelerating.
5. The school has already stopped for 2 weeks, but is supposed to restart beginning in April for the new year. (yeah Japan has a different schedule). But we had already removed our child the last 10 days of schools.
published my work notes left out for a while. I need to refocus my energy here to keep track of what I'm doing
One thing which is obvious with regards to Web compatibility and websites these last couple of days. If there are websites not working in one of your favorite browsers when working or in your daily activities to follow the impact of coronavirus on our daily lives, please report it on webcompat.com. Ksenia and I will try our best to diagnose it.
- Be as descriptive as possible of all the steps you did to encounter the issue.
- Try to reproduce it a couple of times.
- Try to reproduce it in a different browsers.
Ksenia has started an amazing work on refactoring the webcompat.com multi-steps form
FIXED! Latest version of Firefox no longer accepts MacOS text clippings in the google search field. Thanks to Masayuki.
FIXED! document.createEvent("KeyEvents"). This is now forbidden in Firefox. A site was failing they fixed it! Thanks.
I wished today (tuesday) I could type leadfoot commands directly into the browser console to prototype my tests. Maybe it's possible in some ways.
- intermediate solution proposed by Vlad is intern Recorder. I need to try this.
When someone is asking a Pull Request review on github, you receive an email with [email protected] as one of the recipients. Easy to discover with a dynamic mailbox. I usually set my filtering on Mail.app with plenty of dynamic mailboxes. I have a couple of criteria but one which is always very useful to improve the performance is to add a "Received date" criteria with something around a couple of days I usually set around 14 days to 21 days.
I wish there was an annotation mode in git or github. Update 2020-03-30: Check my week notes of week 2020-14.
- It would be even fun if you could share these annotations with people.
- I could imagine that locally you could do something like. git annotate --lines 34-35 -m 'blablababalabla' module/verydope.py
- Then you could later on see them again, extract again, share with someone. git readnotes hash_ref --from [email protected]
When inheriting a code from another, the first part is to read the code and comment it to try to piece things together.
Today, Friday, is diagnosis day.
- Remarkable report with a real website and a reduced test case. That's really cool to see this.
- I didn't handle as much as I wanted today. We will increase the load on Monday.

New form on webcompat.com

We had an issue with the new form design. We switched to 100% of our users on March 16, 2020. but indeed all the bugs received didn't get the label that they were actually reporting with the new form design. Probably only a third got the new form.

So that was the state when I fell asleep on Monday night. Mike pushed the bits a bit more during my night and opened.

My feeling is that if we are out of the experimental phase, we probably need to just not go through the AB code at all, and all of it becomes a lot simpler.

We can keep in place the code for future AB experiments and open a new issue for removing the old form code and tests once ksenia has finished refactoring the rest of the code for the new form.

So on this hypothesis, let's create a new PR. I expect tests to break badly. That's an Achilles' heel of our current setup. The AB experiment was an experiment at the beginning. Never let an experiment grows without the proper setup. We need to fix it.

(env) ~/code/webcompat.com % pytest
============================= test session starts ==============================
platform darwin -- Python 3.7.4, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /Users/karl/code/webcompat.com
collected 157 items

tests/unit/test_api_urls.py ...........                                  [  7%]
tests/unit/test_config.py ..                                             [  8%]
tests/unit/test_console_logs.py .....                                    [ 11%]
tests/unit/test_form.py .................                                [ 22%]
tests/unit/test_helpers.py ............................                  [ 40%]
tests/unit/test_http_caching.py ...                                      [ 42%]
tests/unit/test_issues.py ......                                         [ 45%]
tests/unit/test_rendering.py ......                                      [ 49%]
tests/unit/test_tools_changelog.py ..                                    [ 50%]
tests/unit/test_topsites.py ...                                          [ 52%]
tests/unit/test_uploads.py ...                                           [ 54%]
tests/unit/test_urls.py .................................                [ 75%]
tests/unit/test_webhook.py ......................................        [100%]

============================= 157 passed in 21.28s =============================

Unit testing seems to work. There was a simple fix. The big breakage should happen on the functional tests side. Let's create a PR and see how much CircleCI is not happy about it.

So this is breaking. My local functional testing was deeply broken, and after investigating in many directions, I found the solution. I guess this week is the week where everything is broken.

Questions about tools

I have been asked about the tools at Mozilla, I'm using for the job.

gather feedback from a semi-random set of engineers on various tooling that we have and translate it into actionable requests.

I was not sure which levels of details was needed and on which tools. This is what came from the top of my head.

mozregression for finding bugs.
How do we run mozregression for Firefox Preview (Fenix)?
The documentation is dry. A lot more practical examples would be good. I'm happy to review the ones I would be using.
it would be super cool to be able to run a mozregression on desktop and/or to have a remote mozregression on android.
It would be very cool to be able to have a preset profile (kind of about:config with specific parameters) to launch with each iterations of the regression.
Ice on top the cake: have a web driver set of instructions making the possibility to run the mozregression automatically. When running I sometimes needs to repeat the same set of interactions with a real webpage on a website. I want to be able to program these as instructions (aka enter this url, click here, type this text, etc.) And I probably could have a "run this js test" to see if it's good or bad version. So the regression could run automatically instead of having to repeat the things 20 times.
Searchfox.org
quite a cool tool, which can be very practical to find references sometimes.
I don't have specific requests for it.
Firefox devtools
I have extensive requests. That's my main tool for working on day to day basis. But all my requests are already tracked by the devtools team. Just to say that this is essential for being able to work in the webcompat team and diagnose bugs.
About contributing to mozilla central.
I honestly gave up. I don't even know what is the current way of doing things. Sometimes I see a bug that I could help tackle or start tackling in C++. (Something super simple). But the time it takes to setup the full thing when contributing once every 3 months is dreadful. Things have changed, the tool for review has changed, or something else. The try seems complicated.
As an aside, maybe it improved a lot since three years ago and I should give it another try with something like this
So to take my comments with a pinch of salt.

State of my mozilla central repo

short story version: I had to reinstall everything.

After writing this above, I wondered about the state of my mozilla central repo and how ready I would be to locally compile firefox. Plus I will need it very soon.

hg --version
Mercurial Distributed SCM (version 4.3-rc)
(see https://mercurial-scm.org for more information)

Copyright (C) 2005-2017 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

oops. Let's download the new version. Mercurial 5.2.2 for MacOS X 10.14+ it seems.

hg --version
*** failed to import extension firefoxtree from /Users/karl/.mozbuild/version-control-tools/hgext/firefoxtree: 'module' object has no attribute 'command'
*** failed to import extension reviewboard from /Users/karl/.mozbuild/version-control-tools/hgext/reviewboard/client.py: No module named wireproto
*** failed to import extension push-to-try from /Users/karl/.mozbuild/version-control-tools/hgext/push-to-try: 'module' object has no attribute 'command'
Mercurial Distributed SCM (version 5.2.2)
(see https://mercurial-scm.org for more information)

Copyright (C) 2005-2019 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Hmmm not sure it is better. Let explore the Mozilla Source Tree Documentation.

~/code/mozilla-central % hg pull
*** failed to import extension firefoxtree from /Users/karl/.mozbuild/version-control-tools/hgext/firefoxtree: 'module' object has no attribute 'command'
*** failed to import extension reviewboard from /Users/karl/.mozbuild/version-control-tools/hgext/reviewboard/client.py: No module named wireproto
*** failed to import extension push-to-try from /Users/karl/.mozbuild/version-control-tools/hgext/push-to-try: 'module' object has no attribute 'command'
pulling from https://hg.mozilla.org/mozilla-central/
abandon : certificate for hg.mozilla.org has unexpected fingerprint `sha****`
(check hostsecurity configuration)

maybe I should remove the .mozbuild directory.
There's the certificate issue.

ok I removed the line of [hostsecurity] section in .hgrc. Let's try again

[hostsecurity]
hg.mozilla.org:fingerprints = sha******

The update will take around 40 minutes. Perfect… it's time for lunch. hmmm another fail. let's remove the current error message.

% hg pull
pulling from https://hg.mozilla.org/mozilla-central/
searching for changes
adding changesets
adding manifests
transaction abort!
rollback completed
abandon : stream ended unexpectedly  (got 9447 bytes, expected 32768)

let's try again

hg pull
pulling from https://hg.mozilla.org/mozilla-central/
searching for changes
adding changesets
adding manifests
adding file changes
added 90459 changesets with 787071 changes to 208862 files
new changesets 7a4290ed6a61:9f3f88599fff
(run 'hg update' to get a working copy)

yeah it worked.

hg update
157775 files updated, 0 files merged, 42118 files removed, 0 files unresolved
updated to "9f3f88599fff: Bug 1624113: Explicitly flip pref block_Worker_with_wrong_mime for test browser_webconsole_non_javascript_mime_worker_error.js. r=baku"
4 other heads for branch "default"

./mach bootstrap

returns an error.

 ./mach bootstrap

Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends, or the GeckoView Java API. They are unsuitable
for those working on C++ code. For more information see:
https://developer.mozilla.org/en-US/docs/Artifact_builds.

Please choose the version of Firefox you want to build:
  1. Firefox for Desktop Artifact Mode
  2. Firefox for Desktop
  3. GeckoView/Firefox for Android Artifact Mode
  4. GeckoView/Firefox for Android
Your choice: 2

Looks like you have Homebrew installed. We will install all required packages via Homebrew.

Traceback (most recent call last):
    4: from /usr/local/Homebrew/Library/Homebrew/brew.rb:13:in `<main>'
    3: from /usr/local/Homebrew/Library/Homebrew/brew.rb:13:in `require_relative'
    2: from /usr/local/Homebrew/Library/Homebrew/global.rb:10:in `<top (required)>'
    1: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- active_support/core_ext/object/blank (LoadError)
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/usr/local/bin/brew', 'list']' returned non-zero exit status 1.

  File "/Users/karl/code/mozilla-central/python/mozboot/mozboot/mach_commands.py", line 44, in bootstrap
    bootstrapper.bootstrap()
  File "/Users/karl/code/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 442, in bootstrap
    self.instance.install_system_packages()
  File "/Users/karl/code/mozilla-central/python/mozboot/mozboot/osx.py", line 192, in install_system_packages
    getattr(self, 'ensure_%s_system_packages' % self.package_manager)(not hg_modern)
  File "/Users/karl/code/mozilla-central/python/mozboot/mozboot/osx.py", line 355, in ensure_homebrew_system_packages
    self._ensure_homebrew_packages(packages)
  File "/Users/karl/code/mozilla-central/python/mozboot/mozboot/osx.py", line 304, in _ensure_homebrew_packages
    universal_newlines=True).split()
  File "/Users/karl/code/mozilla-central/python/mozboot/mozboot/base.py", line 443, in check_output
    return subprocess.check_output(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)

I wonder if it's because I switched from python 2.7 to python 3.7 a while ago as a default on my machine. OK I had an old version of brew. I updated it with

brew update

And ran again

./mach bootstrap

hmmm it fails again.

In the process of being solved. I erased ~/.mozbuild/version-control-tools/

Then reinstalled

./mach vcs-setup

Ah understood. My old system was configured to use reviewboard but now they use phabricator. grmbl.

I wish there was a kind of reset configuration system. Maybe there is.

I remove from ~/.hgrc

reviewboard = /Users/karl/.mozbuild/version-control-tools/hgext/reviewboard/client.py

but short summary I needed to re-install everything… because everything was old and outdated.

And now the "funny" thing is that reinstalling from scratch from Japan takes a lot of time… twice it fails because the stream is just failing. :/

Finally!

hg clone https://hg.mozilla.org/mozilla-central/
destination directory: mozilla-central
applying clone bundle from https://hg.cdn.mozilla.net/mozilla-central/9f3f88599fffa54ddf0c744b98cc02df99f8d0b8.zstd-max.hg
adding changesets
adding manifests
adding file changes
added 520018 changesets with 3500251 changes to 567696 files
finished applying clone bundle
searching for changes
aucun changement trouvé
520018 local changesets published
updating to branch default
282924 files updated, 0 files merged, 0 files removed, 0 files unresolved

well that was not a smooth ride. After reinstalling from 0, I still had an issue because of the python 2.7 that brew installed in another location. Recommendations online were encouraging to reinstall it with brew. I really do not like brew. So I did the exact opposite and I removed the brew version of python 2.7. And it worked!

brew uninstall --ignore-dependencies python@2

./mach build

and success…

 3:22.01 0 compiler warnings present.
 3:22.17 Overall system resources - Wall time: 200s; CPU: 0%; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
To view resource usage of the build, run |mach resource-usage|.
 3:22.28 Your build was successful!
To take your build for a test drive, run: |mach run|
For more information on what to do now, see https://developer.mozilla.org/docs/Developer_Guide/So_You_Just_Built_Firefox

Otsukare!

Week notes - 2020 w13 - worklog - everything is broken

New form on webcompat.com

Questions about tools

State of my mozilla central repo

Recommend

Week notes - 2020 w14 - worklog - Human Core Dump

Just Write A Blog Post

Browser Wish List - Tabs Time Machine

Browser Wish List - Tab Splitting for Contextual Reading

A-localized work or distributed work

Formatted console.log lines. Stacktraces export wish.

Browser developer tools timeline

Browser Wish List - Distressful Content Filtering

Week notes - 2020 w39 - worklog - A new era

Week notes - 2020 w40 - worklog - slower and deeper

About Joyk