Sketch Keypad

https://blog.usejournal.com/how-i-created-a-physical-sketch-panel-to-enhance-my-design-system-thinking-93de49c7b832

Part of me hopes that Apple’s Pro Workflow group is exploring something like a Touch Pad (not Bar) for use with desktop apps. A multi-touch panel that augments the keyboard, mouse, and trackpad that can be used to execute application-specific actions using an interface provided by the application. It could be a great way to provide color pickers, shuttle controls, etc. The kind of things the Touch Bar provides on the laptops, but with greater potential because of the less restricted height. Fingerprint authentication would be nice to have too, I suppose.

Mojave Finder Ideas

The Gallery View

As a designer and photographer, I was particularly interested in Finder’s new Gallery View. While the functionality is solid, there is room for improvement, particularly in pointing device and keyboard interactions.

  • The QuickLook view does not scale down to the proper location on screen when filenames are displayed in the thumbnail strip.
The QuickLook preview is misaligned when scaling back down to the thumbnail.
  • The mousewheel might as well be mapped to scroll left/right when the cursor is over the thumbnail strip. As it stands, it’s useless unless you know that you can switch the scroll axis by holding Shift. If you already knew this, congratulations — you too have spent a lot of time at your Mac.
  • Similar to the interaction in Preview when in Single Page mode, people should be able to two-finger swipe left/right with the cursor over the selected image to navigate the thumbnail strip.
  • When you have both a trackpad and mouse connected to your Mac, you can’t grab the scroll thumb by horizontally scrolling with Shift+mousewheel and then moving the cursor into the scroll track; the scroll thumb disappears too quickly. You can easily do so using the trackpad. Once the trackpad is disconnected, the scrollbar is displayed persistently. I guess no one on the Finder QA team uses both a trackpad and mouse simultaneously.
  • I swear Option+mousewheel over the selected item preview to zoom was working at one point. It’s not, but it should.
  • The pinch-to-zoom gesture could be used over the thumbnail strip to switch between the three thumbnail sizes offered in View Options (⌘J), though they are fixed sizes and the pinch-to-zoom gesture is usually used for smooth, continuous zooming.
  • After disconnecting a trackpad, the thumbnail strip can be scrolled vertically such that the thumbnails (or filenames, if displayed) overlap the scroll track. See below.
Disconnecting a trackpad causes layout issues.
  • Using the Home/End keys to jump to the ends of the thumbnail strip causes the vertical position of the thumbnails to shift. They reset once you scroll or use an arrow key to select another item. See below.
Using the Home and End keys causes layout issues.
  • Unlike the three other Finder views, the Page Up/Down keys do not work in Gallery view.
  • The splitter between the Preview and file display panes (or preview and metadata in Gallery View) does not provide double-click behavior. Double-clicking it could either toggle between minimum (480 points) and maximum (960 points) widths for the Preview pane or, like Finder’s Sidebar, reset it to the default width.

If you are a designer and currently use just a mouse or just a trackpad, you are missing out. Assuming sufficient desk space (and hands), you can use the trackpad with one hand to pan and zoom in design apps while simultaneously using the mouse with the other to drag or resize objects (though screen update performance ranges from a bit janky to pretty smooth, depending on the app). Once you’ve tried it, you won’t want to go back.

Making Metadata More Useful

I’ve always found metadata interesting for the possibilities it presents to retrieve and explore information based on attributes, so I’m glad Apple added the ability to display more file metadata in Mojave’s Finder. The problem is that you can’t do much with it aside from view it and select an individual label or value. Below are videos of prototypes I built in Kite for how it could evolve to be more useful.

Showing Photo Locations

Assuming an image has GPS coordinates embedded in its Exif metadata, it would be nice to be able to show the image on a map without having to add it to Photos to use the Places view within that app.

Video of a prototype showing an image location on a map. Prototype built in Kite.
  • Command-clicking the pin icon in the metadata list could launch the Maps app directly rather than having to first access the map pane, then click Show in Maps.

Creating Smart Folders from Displayed Metadata

It would be great to be able to create a Spotlight-powered Smart Folder based on metadata displayed in Finder’s Preview pane. Currently, the process of creating Smart Folders from image metadata (other than the basics like filename and type) involves opening the “Other…” menu in the Spotlight query builder, scrolling to or searching for the metadata type you want to use, then clicking OK. It’s pretty well tucked away and the list of available metadata is extensive.

For example, say I want to create a Smart Folder of all the images captured with my Olympus and exported to JPEG. As you can see below, it’s potentially a fairly easy process when you can start with metadata that’s already displayed.

Building a Smart Folder in Mojave’s Finder from displayed metadata.

At narrow pane widths, the Search Quick Action would collapse into the Other menu.

I first thought of this design for Capture One, but it can fit into most any app that displays metadata in a label-value list and lets you build collections based on metadata.

Further Metadata Notes

  • I’d love it if a future version of Finder displays IPTC metadata (particularly keywords) in the Preview pane. IPTC metadata is already indexed by Spotlight and displayed in the Info window. Hopefully, it wouldn’t be too much work to add support to the Preview pane.
  • It would be possible to do at least rough location mappings of photos based solely on IPTC location metadata such as Location, Municipality, etc.
  • Embedded keywords are more useful than Finder Tags because they travel with the file, regardless of the filesystem.
  • Referring to the pane that displays metadata as Preview doesn’t really make sense in Gallery View, where the metadata and content preview are separate. “Details” would make sense.

Folder Content Previews

Rather than simply displaying a folder’s icon at a larger size when selected in Gallery View, Mojave’s Finder could give users a peek into the folder (information scent) by previewing the folder’s contents. In the case of populous folders, the items shown at the front could be based on the date last opened or modified to increase the odds they are more recent in the user’s memory.

A prototype showing Finder previewing a folder’s contents in Gallery View.

While this example only shows images (for which I think the feature would be most useful), it would work with any file type providing QuickLook previews.

Hexatious

As long as I’ve used Sketch, I’ve wanted a way to display the letters in hex color codes in lowercase, which I find easier to read. See the two contrived but possible examples below. The top inspector screenshot is Sketch, the bottom with a few typography adjustments.

First example comparing Sketch hex codes rendered in all upper case and in mixed case with typography enhancements
Second example comparing Sketch hex codes rendered in all upper case and in mixed case with typography enhancements

The adjusted hex codes are rendered in Apple SF Pro Text with the “High legibility” (note the open 4 and slashed zero) and “One storey a” settings enabled in the Typography panel accessed in Sketch from View > Show Fonts > gear menu > Typography. I also applied character spacing of 0.2 to give the glyphs a little more breathing room.

On a related note, it would be great if Sketch’s text styles accounted for settings enabled in the Typography panel. As of the 53 beta release (71998), things like the “Single storey a” are not saved to and applied by Sketch text styles.

Copy and Paste Resizing Constraints in Sketch

I know there are FPS-inspired shortcuts for assigning resizing constaints to layers in Sketch, but I’d like a way to simply copy those for one layer and paste them onto another.

  1. The Resizing Constraints commands would appear in the contextual menu when Control-clicking on a layer on the canvas (as depicted) or in the Layers list.
  2. I like the brevity of the contextual menu that appears when Control-clicking on a shadow in the Inspector. A similarly focused menu for the Resizing section would be nice.
  3. For those of us who use the keyboard heavily in Sketch, Command-Shift-C and Command-Shift-V to Copy and Paste constraints would be helpful.
  4. The Copy/Paste Resizing Constraints commands would also be available through the Edit > Paste menu. Reset is already in Layer > Constraints.
  5. It’s a bit inconsistent that the controls are labeled “Resizing” in the Inspector, “Constraints” in the Layer menu.

Sketch Missing Font Finder

I’ve opened a few Sketch documents lately that specify fonts I didn’t have installed. Something like the TypeKit integration in Adobe’s Creative Cloud apps could smooth the process of acquiring missing fonts.

Sketch find and install missing fonts dialog mockup

  1. I think it’s helpful to show the document name in the dialog because you can’t actually see the document open in Sketch yet. Maybe you opened a couple documents at once and aren’t sure which one has missing fonts.
  2. The Search Web for Font button would only appear if the specified font cannot be found on Google Fonts. Clicking it would open a search engine query (using the default engine in the user’s default web browser) with a string along the lines of “FontName-Weight+font”.
  3. Alternatively, a menu allowing the user to either search a generic search engine or a font site like FontSquirrel could appear.
  4. The dialog should be wide enough to accomodate most PostScript font names. Extremely lengthy font names should be truncated in the middle so the user can see the first bit of the family name and the weight. The whole name can be displayed on hover.
  5. Some variation of this interface should also appear in the Missing Fonts sheet.

James Young pointed me to the FontBuddy plugin by Anima.

Speech and Gesture Input

I would really like to be able to interact with applications in a simultaneous multi-modal fashion, taking advantage of pointing device, keyboard, voice, and gestures. In some cases, I'd like to direct speech commands to a background app while pointer and keyboard focus remains in the front app. Sometimes you need information from an app, not necessarily to use it.

Say I’m chatting with a friend to schedule dinner next week, but I don’t remember my schedule. Calendar is open and visible in another region of the screen, but it’s showing the current week. Assuming the Mac is always listening for the Siri invocation command, I just say “Hey, Siri, go to next week in Calendar” rather than switching to Calendar, navigating to next week, then switching back to the chat. This scenario is depicted in the video below.

SiriKit for macOS Flowchart
A flowchart for Siri interactions with background apps on macOS.

Speedy interpretation of speech commands is crucial to these interactions feeling fluid and natural. Incorporation of a speech co-processor (as with the iMac Pro) allows Macs to always listen for the “Hey, Siri” prompt rather than having to invoke Siri through the menubar item or keyboard shortcut.

If Apple builds a display (still hoping for a 40-inch 8K) to pair with the forthcoming Mac Pro, they should include the dot field hardware that enables Face Unlock on the iPhone X. Provide that on the Mac, but go further by using it to recognize hand gestures made in the space between the user and the display. It could also potentially be useful for those with motor impairments as a way to use blink patterns to execute commands. Maybe it could even be used as a way to translate sign languages to text, without having to wear a special glove.

Gestures

  1. Hold up a thumbs up or down, or one or more fingers to star rate the playing song
  2. Raise or lower your hand to control audio or video volume
  3. Make a pinch in or out gesture in mid-air to zoom in mapping or graphics apps; not everyone has a trackpad
  4. Make a “holding a camera, pressing the shutter button” gesture to take a screenshot

I wrote years ago about visual gesture interpretation, for which there now seems to be capable hardware. More recent thoughts on gestural interaction from David Rose and IDEO.

Tools and Resources

Subway Status

Once you’ve lived in New York for a while, you don’t necessarily need a specific route to get between two points. You know roughly which subway lines will you get from point A to point B. The important thing becomes knowing which trains are operating normally and which are delayed or rerouted (which seems pretty common these days). To that end, the MTA site provides a status widget providing an overview, but I’d like it to be a single click (or two-finger swipe) away in the Today view of the Notifications Center. A similar display would also be useful on phones.

Subway status widget prototype screenshot

Interactive subway widget prototype (requires a browser that supports CSS Grid)

The brief SVG morph from the 7 local circle to express diamond is done using anime.js.

Safari Responsive Design Mode Polish

Here's the UI as seen in Safari Technical Preview 41, released October 4, 2017.

Safari Technical Preview 41 Responsive Design Mode

Revised Responsive Design Mode

Revised Safari Responsive Design Mode
Safari Responsive Design Mode - device icon hover showing rotate icon
  • Add a bandwidth throttling menu:
    1. Default Network
    2. High-Speed Cellular
    3. Low-Speed Cellular
    4. Offline
  • Improve the contrast of the selected viewport setting in a style consistent with the Web Inspector
  • Show a rotate icon when hovering on a device icon
  • Align label baselines
  • Add a screen mode menu for iPads that would allow selection from:
    1. Full Screen
    2. Primary
    3. Secondary
    4. Split View

    This provides an explicit UI for using the iPad multi-tasking modes and would allow users to switch between portrait and landscape orientations of different iPad models while staying in a multi-tasking view. In STP 41, you can skip cycling through the iPad multi-tasking modes by Option-clicking the iPad icon.

Some Pros Want More Real Estate

I like the announced specs for the iMac Pro, but I’m waiting for Apple to unveil a large external display. I have a 27″ 5K display and I want more space for photos and UI designs without having to deal with multiple displays. Check out Dell’s 32″ 8K display. According to Sven Neuhaus’ PPI calculator, the Dell’s PPI is about 280, which is not ideal for macOS. To fit 8K of pixels at the 220 PPI that makes for comfortable rendering of the macOS interface, Apple would need to build a 40″ diagonal panel. Sounds good to me.

The other thing I’m really looking forward to is APFS in High Sierra. I had to reformat an external drive recently because of corruption that likely could have been avoided with a modern filesystem.