ideo / avocado-allegory Goto Github PK

Today's to-do list, before catching up with Fra:

• Make a sandbox app
• Expand the sandbox to have all controls, including how many entrants are in the contest.
• Dynamically create entrants
• Voting method where everyone picks their one favorite and majority wins
• Implement Jenna's numbers in the unittests
• Display rankings

Not everyone knew what was going on. People took the simulation for granted and “trusted us” despite not really understanding how it worked under the hood.

• The copy can be less metaphorical, more explicitly explaining what’s happening. We could even use the word agent
• The townspeople’s votes should stay up the whole time
• Include labels that explain what each thing is: Objective Score, Townspeople’s votes

Also, try adding a counter at the top that increments.

• Introduce voting demo
• Sketch out a conclusion
• Briefly explaining the vote N times thing.

If so, let's show, say, the top 5.

(I know that the Smith/Schwartz set may only have one candidate, but can we get rankings of those outside the set?)

• Use names or letters instead of numbers
• Use the word score on the napkin so we can then better explain how scores can be interpreted as implicit rankings

This way we can go as low as 5 guacs or as high as 50 guacs. Use a random seed so that the results are consistent.

This must be done before tuning so that we can have reproducible scenarios.

It’s the meat of the thing. Fill the page. It will also help visually distinguish things.

• Create test cases
• Consider redoing how you generate rankings. You could do it at the end from the counts dataframe.
• Update so people can rank just their top N candidates, not all of them.

We want to work towards the following story arch:

1. We have a problem. We’ve run a contest but it’s not working.
2. Here is a solution or here are some solutions we could try.
3. This is the takeaway you should have.

I think we have clarity on what the takeaway should be, but how to get there needs discussion. Here’s how I see the takeaway right now:

• What we thought was a problem – not enough guac to go around – was actually an opportunity because it made it easier to include more voters into the contest.

To get that flow, we need to decide on what set up we want to show first and how it will fail.

Potential starting points:

• We have a contest that works well, but only when just a few guacs participate. When we add more entrants it breaks down.
  • Solution could be limiting guacs and adding more people. Would be cool to show limiting guacs only partially fixes it but then adding more people takes us home.
  • Unsure of what the voting method should be for the first one. Perhaps ranking top three.
• We have a contest that works well, but only when just a few guacs participate, because all we do is pick our one favorite.
  • When we add more guacs, we move to ranking just the top three or 5 (RCV)
  • Then we also try score voting
  • Then we also try condorcet

These are the challenges we need to test throughout

• Not everyone assess things the same way – Pepe and Fra
• Collusion! – Carlos
• (Optional) Collusion through having a big team

One Idea for a story structure:

• Baseline scenario and how it doesn’t work.
• Proposed Solution
• Introduce Pepe/Fra – how does proposed solution hold up
• Propose another solution (once without, once with Pepe/Fra)
• Introuce Carlos – how do both solutions
• Conclusion

As you eat more guac the untasted one become less appealing?

• how does that change with increasing/decreasing the number of judges
• how does that change with increasing/decreasing the number of guacs

People who enter the contest also get to vote, like Carlos, but other people show up to taste and vote as well, like Pepe and Carlos.

• Perhaps it’ll be easier if we reduce it from 20 to 15? Less names to read?

• re-write Jenna's code
• test script on Jenna's ballot. Do we recover it?
• add logic for a tie. For now we pick a winner at random among the subset (how often does that occur over 200 simulations??)

Such as:

• A few more top contenders
• A whole bunch of top contenders
• Oh shit they're all roughly mediocre

For now, I think we should always have one objective winner for each scenario. But let's discuss that!

The story itself needs to be lighter, quicker, more concise. The instructional or explanatory elements needs to be more explicit.

• Experiment with further use of st.caption() to deliniate instructional or explanatory parts.

Some people understood what this was and some did not. The “god” character was received well, but the purpose of often unclear.

• Update the copy so we explicitly explain what these scores are and their purpose in the simulation, i.e. giving us a shared ground truth
• Include scenario selectors in each contest. People didn’t scroll back up to the top.

• with no effects
• include fullness factors
• include personas

Right now, we introduce two separate problems and show one solution. It’s not clear what the impact of each character is. Show their impact separately. But the big question here is how to fit them into a revised flow.

So, in addition to just awarding Carlos's guacamole a 10.0, they'll down vote all the one's that aren't his.

My concern is that right now when a Carlos Crony gets to vote on the best guac, with a objective score of 10.0, they'll still likely give it a high subjective score. This could lead to their ballot having two 10.0 scores on it. Not how a crony would likely actually behave.

We also need a name for it, because I don’t think we should call it the Condorcet method. Some ideas:

• Tallying implicit rankings

We want both versions

• Simulate 100 times
• How often is the proper winner recovered?

For both:

• Formalize code to pass parameters to a simulation. This must include the scenario and the number of candidates. It could be done by passing both of those as parameters to the simulation and letting it select the guac_df.

We need more variety of outcomes. Let’s re-run our tuning simulations at that value.

• Christina’s idea about people preferring their mother’s guac recipe represents the Grammy’s reliance on “safe” artists and reluctance to embrace new approaches, like Chance releasing his album for free.
• The Grammy’s is concerned about how songs that have a lot of collaborators tend to do well because they’ve essentially recruited a big team. Is there a way we can incorporate that? It’s different than the collusion problem, but perhaps it has the same solution.