Oct 2023, https://blog.sbensu.com/

For more like this one, see Friction logs

Setup: Trying to make Anki cards based on chat sessions from OpenAI

Result: Failed. Need to spend a lot more effort to do what I wanted.

Takeaways

• Overall perception
  • The product can do a lot of stuff because of its numerous integrations but it feels low quality. Using the UI feels clunky and not easy at all.
  • To have something truly massive (50% of the population can use it), you’d need much better product design and execution. My guess is that you’d have to be very motivated to get through Zapier and end up with a useful integration.
• UI choices
  • Zapier never shows code to the end-user unless they ask for it in “Code by Zapier”
    • Even when they are making bare API request to OpenAI that return nested JSON blobs
    • They represent basic control flow with the same Action construct as regular integrations. For example, they have “Filter” (if statements), “Delays”, etc.
  • The visual editor that is good at showing the steps that are executed and the order in which they are executed
    • The best part of the visual editor is that it asks you to run each step with real values as you are making it
      • which is a better debugging experience than most programming languages (where you write a long script, have to think about test values, and then run it to find it fail on a step that you were not thinking about)
    • Compared to text programs, having all the script choices be behind UI menus and dropdown hides most of the choices you’ve made and makes it hard to scan how the script is configured.
    • Compared to text programs, interacting with each step through the UI menus, makes it easier to see what the step can do in the first place and what options are even available.
  • Error handling is terrible! They dump an unreadable error and send you to an FAQ page.
  • The spreadsheet integration is very specialized. I bet spreadsheets are always at the core of what they do.
• LLM-usage within the app
  • When you are starting a Zap, you can give them a prompt “Summarize this from that” to generate the outline of a Zap with its steps
    • Terribly done, asked for “ChatGPT” got a different integration entirely despite OpenAI being one of the possible integrations.
  • Code generation in “Code by Zapier” → terribly done, unusable
  • They derive a nice name for each step (I think they do this with LLMs)
  • When you are trying to format data with Formatter by Zapier, you can prompt what you want and it will autocomplete all the options of the Formatter
    • Worked well but possibly because I prompted one of the examples they gave
• AI integrations
  • They have 106 AI integrations, and many of the ones you’d want (OpenAI, Anthropic, Cohere, etc)
  • Few on image or video generation
  • They are taking AI integrations seriously
  • They have many of the productivity ones like meeting notes, summarization, etc.
• Overall integrations
  • They have +5000 integrations and they’ve been at it fr ~10 years.
  • When it comes to productivity and utility they have most of what you’d want (all of Google Suite, Slack, CRMs, etc)

Raw notes

Create a Zap from natural language

I can send them a prompt and they’ll try to make a Zap out of it, but it doesn’t work well:

It recognized that there are some nouns that it doesn’t know what to do about: