Saturday, March 11, 2023

Playing with ChatGPT and Understanding How to Query ChatGPT

Like any new tool, ChatGPT and related generative AI tools require some amount of human learning.  Granted the latest generation of generative AI/chatbots is very sophisticated and we, as humans, know how to ask questions, yet suddenly the art of asking questions comes front stage.

As search tools improved over time and the interface landed itself to entering full sentences rather than just keywords, we probably all started naturally entering questions in Google search and other search tools.  I know I did.  Instead of entering "World Café" I could enter "What is a World Café"?  Ideally, there would be a difference in the results because with a simple keyword I am asking for everything that mentions World Cafés and with a simple "What is" question, I am looking for a description or definition.  

Enters ChatGPT and it's a new world, a new way of interacting with a query tool.  It may feel like a conversation but it is not. I would prefer to reserve the word conversation for interactions with humans. I don't care that it seems to acquire an attitude at times.  We should not fall into the trap of thinking it has human-like capabilities, or feelings of any kind.  It does not.  Neither should we be reacting to its answers as if it were a human.  It is probabilistic model. It does an impressive job of guessing what the next word should be but it has no understanding of what the sentences mean.  It is iterative in a useful way.  You can refine your query without starting over and the the tool remembers the initial parameters of your query. 

Let's explore further with my "World Café" query.  As a side note, I know enough about World Cafés to have a sense of the accuracy and meaningfulness of what I would normally find by searching the web.  I am not an expert who would have written the content that exists on the web about this topic.  I have also attended World Cafes and implemented adaptations of the model.  As with anything related to information and knowledge management, the prior knowledge and experience of the individual encountering and trying to absorb new information is relevant.

Here are a few questions I asked:

  • What are the main uses for a World Café? 
  • Is it different from a Knowledge Café? -- I learned a few nuances I was not aware of.
  • If I am planning a Knowledge Café, what are some of the questions I should ask?  -- this was a badly formulated question which resulted in an answer that was off the mark. The answer focused on the types of questions to ask within the Café rather than questions to ask myself as a planner of the Café. The same confusion could have happened in a conversation with a human.
  •  That's not what I was looking for. Let's rephrase.  What are some best practices in planning a world café or a knowledge café? -- See how I fell into the conversation mode.  I am not sure how ChatGPT interprets the fact that I tried to tell it the answer was not what I was looking for. I am not sure how ChatGPT can learn from that and what it would learn from that.  Ultimately, my question was not phrased properly. 
  • When is a World Café or Knowledge Café the most appropriate way to engage a group of people in a meaningful conversation?
  • What are some alternative stakeholder engagement methods? -- here, because I am still refining a query, ChatGPT knows that I am looking for alternatives to World Cafés/Knowledge Cafés and therefore it will not list these two methods in the answer. 
I came back to this query a few days later and tried something a little different. Note that you can return to a past query and start from where you left off or start a new query.  I tried it both ways with the following question:
  • I have been asked to plan a World Café for somewhere between 20 and 100 people.  How should I go about planning this World Café?
The answers to the same question were quite different.  The answer that came as a result to a completely new query was off-topic in the sense that it did not focus on the planning.  It described the entire process of implementing a World Café. It did so in such generic terms that with the exception of a single sentence, it could have applied to any meeting.  So, I follow up with: "I'm only interested in the planning stage."  The answer was a series of bullet points that would also apply to any meeting.  I guess I was looking for more specificity. So I insisted with "I am looking for more specific guidance".  And it worked.  Each bullet point was now accompanied by 3 or 4 useful sub-bullets.

The answer that came as a result of the existing query (the refinements from all the questions listed above) was concise yet much more precise and targeted.  Every bullet point mentioned World Cafés and was on point.  And yet, it completely failed to mention anything about identifying and inviting participants, which turns out to be a big gap. 

I am trying to imagine what happened with the two different queries.  The query starting from scratch is looking at a huge amount of materials mentioning World Cafes and the concise answer it is able to provide is the most generic.  It's not wrong, but not very useful either.   I am imagining that the refined query is looking at a more narrow set of materials determined to be relevant based on the previous questions.  Depending on the winding road of questions I asked, it may have eliminated resources that talked about participants.  I wonder.

So I asked a follow up question:  "What about participants?"  YES.  Very good answer to that.  

Final question:  What are the key sources for this information?  I had a strong negative reaction to the one bullet in the answer that suggested ChatGPT had "professional experience". 

Sorry, but ChatGPT has zero professional experience planning or implementing World Cafes. I don't think I would dare to say that I have professional experience in X if all I've done is read about X. ChatGPT is gaining a lot of experience answering questions and learning how to answer questions that satisfy the questioners, but until humans feed it the knowledge based on their experience, it can't learn.  Questions around what it can or cannot learn are fascinating. 

Lesson Learned with this set of ChatGPT queries:

  • Smart querying requires critical thinking.  This is particularly true as we (humans) learn to interact with this powerful tool. Until we fully understand its capabilities and its weaknesses, we need to treat our queries as practice runs.  Our practice runs are training materials for ChatGPT as well, so it is potentially learning how to answer pretty bad, beginners' queries.
  •  Don't give up too quickly when the answer seems off topic or too generic.  Refine the query until you get to the level of detail and specificity you need and accept that it is not perfect. 
  • Keep an eye on the full query, the series of questions and refinements, because it may represents a set of constraints that shapes the set of materials ChatGPT looks at.  If you veered off, like I did, with a question about alternative methods, you might need to later say "ignore question x" and focus on World Cafes.  When I asked "What are the key sources for this information?" I think ChatGPT answered for the entire question thread, not the last question in the thread. 
  • There is a lot I don't understand about HOW ChatGPT goes about selecting its sources in the context of a single question vs. full set of iterative queries.  I will continue testing and practicing asking good questions as a way to continue learning. 
Read also: