Built in Drama, which is a nice new macOS app. It doesn’t have the scriptability that Kite does, but I like the UI and drawing capabilities more than Principle.
Category: Interaction
How people interact with their world.
Sketch Keypad
Part of me hopes that Apple’s Pro Workflow group is exploring something like a Touch Pad (not Bar) for use with desktop apps. A multi-touch panel that augments the keyboard, mouse, and trackpad that can be used to execute application-specific actions using an interface provided by the application. It could be a great way to provide color pickers, shuttle controls, etc. The kind of things the Touch Bar provides on the laptops, but with greater potential because of the less restricted height. Fingerprint authentication would be nice to have too, I suppose.
Mojave Finder Ideas
The Gallery View
As a designer and photographer, I was particularly interested in Finder’s new Gallery View. While the functionality is solid, there is room for improvement, particularly in pointing device and keyboard interactions.
- The QuickLook view does not scale down to the proper location on screen when filenames are displayed in the thumbnail strip.
- The mousewheel might as well be mapped to scroll left/right when the cursor is over the thumbnail strip. As it stands, it’s useless unless you know that you can switch the scroll axis by holding Shift. If you already knew this, congratulations — you too have spent a lot of time at your Mac.
- Similar to the interaction in Preview when in Single Page mode, people should be able to two-finger swipe left/right with the cursor over the selected image to navigate the thumbnail strip.
- When you have both a trackpad and mouse connected to your Mac, you can’t grab the scroll thumb by horizontally scrolling with Shift+mousewheel and then moving the cursor into the scroll track; the scroll thumb disappears too quickly. You can easily do so using the trackpad. Once the trackpad is disconnected, the scrollbar is displayed persistently. I guess no one on the Finder QA team uses both a trackpad and mouse simultaneously.
- I swear Option+mousewheel over the selected item preview to zoom was working at one point. It’s not, but it should.
- The pinch-to-zoom gesture could be used over the thumbnail strip to switch between the three thumbnail sizes offered in View Options (⌘J), though they are fixed sizes and the pinch-to-zoom gesture is usually used for smooth, continuous zooming.
- After disconnecting a trackpad, the thumbnail strip can be scrolled vertically such that the thumbnails (or filenames, if displayed) overlap the scroll track. See below.
- Using the Home/End keys to jump to the ends of the thumbnail strip causes the vertical position of the thumbnails to shift. They reset once you scroll or use an arrow key to select another item. See below.
- Unlike the three other Finder views, the Page Up/Down keys do not work in Gallery view.
- The splitter between the Preview and file display panes (or preview and metadata in Gallery View) does not provide double-click behavior. Double-clicking it could either toggle between minimum (480 points) and maximum (960 points) widths for the Preview pane or, like Finder’s Sidebar, reset it to the default width.
If you are a designer and currently use just a mouse or just a trackpad, you are missing out. Assuming sufficient desk space (and hands), you can use the trackpad with one hand to pan and zoom in design apps while simultaneously using the mouse with the other to drag or resize objects (though screen update performance ranges from a bit janky to pretty smooth, depending on the app). Once you’ve tried it, you won’t want to go back.
Making Metadata More Useful
I’ve always found metadata interesting for the possibilities it presents to retrieve and explore information based on attributes, so I’m glad Apple added the ability to display more file metadata in Mojave’s Finder. The problem is that you can’t do much with it aside from view it and select an individual label or value. Below are videos of prototypes I built in Kite for how it could evolve to be more useful.
Showing Photo Locations
Assuming an image has GPS coordinates embedded in its Exif metadata, it would be nice to be able to show the image on a map without having to add it to Photos to use the Places view within that app.
- Command-clicking the pin icon in the metadata list could launch the Maps app directly rather than having to first access the map pane, then click Show in Maps.
Creating Smart Folders from Displayed Metadata
It would be great to be able to create a Spotlight-powered Smart Folder based on metadata displayed in Finder’s Preview pane. Currently, the process of creating Smart Folders from image metadata (other than the basics like filename and type) involves opening the “Other…” menu in the Spotlight query builder, scrolling to or searching for the metadata type you want to use, then clicking OK. It’s pretty well tucked away and the list of available metadata is extensive.
For example, say I want to create a Smart Folder of all the images captured with my Olympus and exported to JPEG. As you can see below, it’s potentially a fairly easy process when you can start with metadata that’s already displayed.
At narrow pane widths, the Search Quick Action would collapse into the Other menu.
I first thought of this design for Capture One, but it can fit into most any app that displays metadata in a label-value list and lets you build collections based on metadata.
Further Metadata Notes
- I’d love it if a future version of Finder displays IPTC metadata (particularly keywords) in the Preview pane. IPTC metadata is already indexed by Spotlight and displayed in the Info window. Hopefully, it wouldn’t be too much work to add support to the Preview pane.
- It would be possible to do at least rough location mappings of photos based solely on IPTC location metadata such as Location, Municipality, etc.
- Embedded keywords are more useful than Finder Tags because they travel with the file, regardless of the filesystem.
- Referring to the pane that displays metadata as Preview doesn’t really make sense in Gallery View, where the metadata and content preview are separate. “Details” would make sense.
Folder Content Previews
Rather than simply displaying a folder’s icon at a larger size when selected in Gallery View, Mojave’s Finder could give users a peek into the folder (information scent) by previewing the folder’s contents. In the case of populous folders, the items shown at the front could be based on the date last opened or modified to increase the odds they are more recent in the user’s memory.
While this example only shows images (for which I think the feature would be most useful), it would work with any file type providing QuickLook previews.
Cease to Increase
I imagine I’m not the only Sketch user who maximizes their document window. Unfortunately, this creates an interaction problem with Sketch’s scrubby steppers.
When the window is maximized (full-screen or zoomed), the Inspector is very close to the edge of the screen, meaning there isn’t much room to travel to increase the value by clicking and dragging a stepper to the right. As of version 51.2, the value stops increasing once the cursor reaches the edge.
Instead, Sketch should detect that the cursor has hit the edge of the screen and keep increasing the value as long as the cursor remains at the edge. Infinite depth!
Originally a few tweets, posted here for posterity. I filed a feature suggestion with the Sketch team.
Fireworks Begat Sketch
It has been great to watch Sketch evolve from a fairly simple drawing application (DrawIt, originally) to a key part of many digital designers' workflow and the center of a design tool ecosystem. Having been a heavy Fireworks user, I see a lot of it in Sketch.
The Bohemian Coding team was obviously aware of Fireworks and the community around it. One of the first Sketch plugins I knew of was Ale Muñoz's Sketch Commands, which he originally developed for Fireworks. He has since joined the Bohemian team. Bohemian Coding also asked illustrator and designer Isabel Aracama to create a vector illustration showing that Sketch could be used for such things. Prior to its demise, she was a vocal promoter of Fireworks. If you didn't know Fireworks could handle complex vector illustration, check out her work.
Fireworks wasn't just for slicing images. A number of features made it a good tool for designing any pixel-based interface, features that are among those designers coming from Photoshop to Sketch have found particularly welcome.
Flexible Canvas Sizing
Fireworks allowed you to creates pages within your design file of different resolutions. Ultimately, it's not as flexible as the Page > Artboard (or infinite canvas) arrangement in Sketch, but it was a lot more flexible than Photoshop allowed at the time. This made it easy to design different parts of interfaces while having access to the same symbols and styles.
Flexible Symbols
Fireworks allowed you to nest symbols, and scale and style them without affecting other instances. If configured with 9-slice scaling, you could stretch symbols without distorting the areas outside the center slice.
Buttons were a special symbol type in Fireworks, allowing you to create a base button and reuse it with custom label text. Within the symbol, you could create up, down, and hover states.
Rich Symbols were a pain to create, but provided some of the same functionality as Symbol Overrides do in Sketch. The prolific John Dunning wrote some tools to ease the process, but Fireworks was a bit buggy in handling custom Rich Symbols.
If you needed to share your symbols, you could just export a symbol library file.
Plugins
Plugins (known as Extensions in Fireworks) were an under-promoted feature in Fireworks. As designers find with Sketch, plugins can dramatically increase your efficiency. One of the first plugins I installed whenever I set up Fireworks was John Dunning's QuickFire, which put all the commands of other plugins a few keystrokes away rather than buried in a submenu of the Commands menu, much like Sketch Runner does in Sketch. I used Ale Muñoz's Orange Commands, John Dunning's keyboard resizing commands, and Matt Stow's math resizing commands a lot to improve my workflow.
The Twist and Fade plugin included with Fireworks is similar to Looper for Sketch.
Click-Through Prototyping
If you are primarily concerned with testing whether a navigation scheme makes sense, simple click-through prototyping is quite useful. It doesn't let you work with user input beyond clicks or transition smoothly between states and views, but it can help you address problems in layout, contrast, and labeling.
In Fireworks, you would draw arbitrary polygonal hotspots or place button symbols, then specify a page within your Fireworks file to which they should point. You couldn't specify a transition (push left, right, cross-fade, etc.); everything just swapped. You could export to HTML (which created an image map) or to a clickable PDF with everything rasterized. The prototyping functionality in the recent v49 release of Sketch is not terribly different, but it represents what Emanuel Sá referred to as “the tip of the iceberg.”
Miscellany
- Fireworks used an extended version of the PNG file format that made for pretty compact files, even with lots of pages. It's certainly not why I chose Fireworks as my design tool, but it was welcome when backing up or working on a machine without tons of memory.
- I miss Fireworks' Auto Shapes sometimes. There are a few in Sketch (star, polygon, arrow, triangle), but Fireworks provided a few more. A former colleague learned how to use Fireworks after seeing how easy it was to redline mockups using the Measure Auto Shape.
- I still miss the ability to simply see the distance between two guides by holding down a modifier key while the cursor is between two guides on the same axis.
- They wouldn't really make sense in Sketch, but Shared Layers were another huge advantage of Fireworks over Photoshop for UI design. These worked a bit like master pages in InDesign. Create a layer (a navigation bar, for instance), then share it to all the pages where it should appear. You could achieve the same basic thing with a symbol, but symbol positions don't carry across instances.
- There was a cool gradient fill type in Fireworks called Satin that could be used to create nice abstract graphics. It would be tough to translate to code (be it CSS or a native UI toolkit), but it would still be great for static graphics.
What Wasn't So Great
- There were stability problems that decreased over the time Adobe developed the product, but it never became rock-solid. Sketch has really improved in this regard.
- No native vector export. John Dunning wrote an extension to export SVG that worked quite well, but no built-in option existed.
- No multi-resolution export. This wasn't a big deal until the iPhone 4 came along with the high-PPI Retina display.
- Text anti-aliasing was always so-so.
- No springs and struts (or constraints, resizing rules, etc.).
I'm glad that David Malouf had adopted Fireworks for the team at my first job working as a designer. It served me well from 2006 until Adobe announced the end of development in 2013. Reading the writing on the wall, I switched to Sketch, even before it had many tools for reuse like symbols and shared styles. If they can just get that guide space measurement feature implemented…
Copy and Paste Resizing Constraints in Sketch
I know there are FPS-inspired shortcuts for assigning resizing constaints to layers in Sketch, but I’d like a way to simply copy those for one layer and paste them onto another.
- The Resizing Constraints commands would appear in the contextual menu when Control-clicking on a layer on the canvas (as depicted) or in the Layers list.
- I like the brevity of the contextual menu that appears when Control-clicking on a shadow in the Inspector. A similarly focused menu for the Resizing section would be nice.
- For those of us who use the keyboard heavily in Sketch, Command-Shift-C and Command-Shift-V to Copy and Paste constraints would be helpful.
- The Copy/Paste Resizing Constraints commands would also be available through the Edit > Paste menu. Reset is already in Layer > Constraints.
- It’s a bit inconsistent that the controls are labeled “Resizing” in the Inspector, “Constraints” in the Layer menu.
Sketch Missing Font Finder
I’ve opened a few Sketch documents lately that specify fonts I didn’t have installed. Something like the TypeKit integration in Adobe’s Creative Cloud apps could smooth the process of acquiring missing fonts.
- I think it’s helpful to show the document name in the dialog because you can’t actually see the document open in Sketch yet. Maybe you opened a couple documents at once and aren’t sure which one has missing fonts.
- The Search Web for Font button would only appear if the specified font cannot be found on Google Fonts. Clicking it would open a search engine query (using the default engine in the user’s default web browser) with a string along the lines of “FontName-Weight+font”.
- Alternatively, a menu allowing the user to either search a generic search engine or a font site like FontSquirrel could appear.
- The dialog should be wide enough to accomodate most PostScript font names. Extremely lengthy font names should be truncated in the middle so the user can see the first bit of the family name and the weight. The whole name can be displayed on hover.
- Some variation of this interface should also appear in the Missing Fonts sheet.
James Young pointed me to the FontBuddy plugin by Anima.
Speech and Gesture Input
I would really like to be able to interact with applications in a simultaneous multi-modal fashion, taking advantage of pointing device, keyboard, voice, and gestures. In some cases, I'd like to direct speech commands to a background app while pointer and keyboard focus remains in the front app. Sometimes you need information from an app, not necessarily to use it.
Say I’m chatting with a friend to schedule dinner next week, but I don’t remember my schedule. Calendar is open and visible in another region of the screen, but it’s showing the current week. Assuming the Mac is always listening for the Siri invocation command, I just say “Hey, Siri, go to next week in Calendar” rather than switching to Calendar, navigating to next week, then switching back to the chat. This scenario is depicted in the video below.
Speedy interpretation of speech commands is crucial to these interactions feeling fluid and natural. Incorporation of a speech co-processor (as with the iMac Pro) allows Macs to always listen for the “Hey, Siri” prompt rather than having to invoke Siri through the menubar item or keyboard shortcut.
If Apple builds a display (still hoping for a 40-inch 8K) to pair with the forthcoming Mac Pro, they should include the dot field hardware that enables Face Unlock on the iPhone X. Provide that on the Mac, but go further by using it to recognize hand gestures made in the space between the user and the display. It could also potentially be useful for those with motor impairments as a way to use blink patterns to execute commands. Maybe it could even be used as a way to translate sign languages to text, without having to wear a special glove.
Gestures
- Hold up a thumbs up or down, or one or more fingers to star rate the playing song
- Raise or lower your hand to control audio or video volume
- Make a pinch in or out gesture in mid-air to zoom in mapping or graphics apps; not everyone has a trackpad
- Make a “holding a camera, pressing the shutter button” gesture to take a screenshot
I wrote years ago about visual gesture interpretation, for which there now seems to be capable hardware. More recent thoughts on gestural interaction from David Rose and IDEO.
Tools and Resources
- Principle for the Siri demo animation
- ScreenFlow to combine the video exported from Principle with the voice command/Siri chime audio
- Facebook Design's macOS desktop kit for some of the UI elements
- Designer Matt McDaniel's Messages Sketch file, which I tweaked for High Sierra
Merging the Mergers in Affinity Photo
Say you drag a few images into one of Affinity Photo’s merging tools, only to realize you opened the wrong one. The commands are grouped in the menu, two have keyboard shortcuts right next to one another, and the tools have similar layouts, making this a fairly easy mistake to make.
Rather than having to hit Cancel, open the intended merge tool, then relocate the images you want to merge, what if you could just switch to it, with the images you added still in place?
Notes
- The functionality of the Batch tool seems different enough to warrant it remaining separate.
- To keep the various merging options visible to the user, it might be best to keep their individual items in the File menu. Each would just open the unified interface to the appropriate tool.
- It would be great to be able to drag-and-drop directly to the Images pane rather than having to open the file picker, then drop to it, then click OK.
- The visual styling of the window widgets is a bit confusing; the brighter widgets are disabled. This is contrary to most GUI conventions, including how the OK button is disabled in the tools themselves.
- If the Panorama tool isn’t resizable because the panorama previews would have to be re-rendered (a processor-intensive task), just fix the width of the preview pane while allowing the Images pane to expand.
- There is some ambiguity in the Panorama tool when you have stitched multiple panoramas together. If you don't explicitly select all of them in the preview pane before you hit OK, only the last selected one will build. A different button label or counter tallying how many panoramas will be rendered (Render 2 Panoramas) might be sufficient.
- NTH: a way to tell which images constitute which panorama after the panorama preview is generated. This could look something like the mockup below. Ideally, the number displays next to the panorama preview would handle discontiguous stitches nicely, so if you used three of 4 images out of sequence, something like “1 – 2, 4” would appear next to the thumbnail.
- NTH: the “Who you gonna call? Ghostbusters!” portion of the movie theme should play when you check the “Automatically remove ghosts” option in the HDR merge tool.
I'm looking forward to Affinity Publisher!
Subway Status
Once you’ve lived in New York for a while, you don’t necessarily need a specific route to get between two points. You know roughly which subway lines will you get from point A to point B. The important thing becomes knowing which trains are operating normally and which are delayed or rerouted (which seems pretty common these days). To that end, the MTA site provides a status widget providing an overview, but I’d like it to be a single click (or two-finger swipe) away in the Today view of the Notifications Center. A similar display would also be useful on phones.
Interactive subway widget prototype (requires a browser that supports CSS Grid)
The brief SVG morph from the 7 local circle to express diamond is done using anime.js.