Coverage comparison
We compare Vitas with random testing and Skillexplorer about the state space coverage, and the time limit is set as 10 minites for three techniques. The table below shows the state space coverage of each technique. Here, the manual test results are for reference because they have no time limit. The last column is the union of state space coverage of three techniques and users. We carry out this experiment on the benchmark. The skills selected here satisfy three conditions: 1. They can be opened by four methods. 2. Their behavior is consistent for four methods. 3. No network exception happens during the test.
To get the table below, run python experiment.py 7 in the experiment directory. The table is saved in fig7_coverge_rate_four.xlsx.
| skill's name | manual | Vitas | random | Skillexplorer | union |
|---|---|---|---|---|---|
| Build a PC Sales | 15 | 10 | 14 | 7 | 25 |
| Asthma Device Helper | 8 | 32 | 32 | 59 | 61 |
| Animal Sounds | 45 | 37 | 27 | 25 | 108 |
| Dr Speech | 14 | 21 | 12 | 18 | 28 |
| Song Quiz | 59 | 33 | 17 | 15 | 89 |
| Choose Your Adventure Game | 25 | 12 | 12 | 10 | 26 |
| Toe Tac Tic | 13 | 13 | 12 | 23 | 25 |
| Butterball | 33 | 22 | 16 | 38 | 77 |
| My Horoscope | 26 | 39 | 40 | 18 | 82 |
| Birthstone | 14 | 16 | 12 | 9 | 20 |
| Cooking Temperature Buddy | 8 | 13 | 15 | 11 | 17 |
| Yoga Teacher | 39 | 30 | 14 | 10 | 48 |
| NewsChest: Unofficial FC Barcelona news | 16 | 11 | 13 | 8 | 29 |
| Super Tech Support | 10 | 19 | 16 | 19 | 19 |
| Unofficial Simpson's Quote Finder | 8 | 9 | 9 | 8 | 9 |
| Funny Noises | 9 | 11 | 11 | 11 | 11 |
| Actor Wiki | 10 | 11 | 10 | 14 | 29 |
| LOST Geek Quiz | 70 | 98 | 85 | 17 | 180 |
| Grace Chapel | 16 | 17 | 15 | 17 | 28 |
| Scare off Burglars | 6 | 10 | 10 | 9 | 15 |
| NPR | 9 | 14 | 13 | 8 | 19 |
| Soccer Scores | 6 | 10 | 6 | 3 | 12 |
| The Answer Chicago | 2 | 2 | 2 | 1 | 2 |
| FireBoard - Cloud Connected Smart Thermometer | 5 | 7 | 7 | 7 | 7 |
| Travel Dates | 3 | 4 | 5 | 5 | 8 |
| Pico y Placa | 23 | 7 | 8 | 11 | 24 |
| I am Affirmations | 36 | 21 | 18 | 13 | 52 |
| The Rock of Northeast Arkansas | 15 | 19 | 22 | 18 | 25 |
| Lighthouse Apostolic Ministries, Saint Louis, MO | 15 | 15 | 22 | 17 | 23 |
| Fitness Encouragement | 69 | 28 | 21 | 16 | 80 |
| A path to forgiveness | 48 | 52 | 44 | 51 | 54 |
| Driving Etiquette | 62 | 11 | 8 | 5 | 69 |
| cook perfect eggs | 63 | 39 | 31 | 61 | 88 |
| Dracula Gossip | 3 | 20 | 12 | 17 | 33 |
| Never Have I Ever | 35 | 25 | 22 | 28 | 68 |
| Unofficial Cash4Life Winning Numbers | 22 | 28 | 17 | 19 | 44 |
| Backgammon Helper | 21 | 29 | 27 | 29 | 38 |
| India trivia game | 53 | 65 | 44 | 22 | 119 |
| Santa's Helper | 85 | 47 | 21 | 74 | 188 |
| Magic Pet | 95 | 35 | 32 | 29 | 157 |
| World Languages | 61 | 73 | 49 | 17 | 133 |
| Seventh Inning Stretch Baseball - Buttons Optional | 53 | 40 | 41 | 15 | 80 |
| Hurricane Center | 65 | 49 | 31 | 48 | 93 |
| Stanley's journey | 57 | 40 | 32 | 53 | 65 |
| Uninspiring Quotes | 20 | 20 | 8 | 16 | 33 |
| Travelers | unavailable | 4 | 10 | 10 | 14 |
| U.S. Bank | unavailable | 14 | 11 | unavailable | 16 |
| IBKR IBot | unavailable | 37 | 37 | unavailable | 48 |
| Stock Info | unavailable | 19 | unavailable | 13 | 26 |
| electraFi.com | unavailable | unavailable | unavailable | unavailable | 0 |
| JW.ORG® | 26 | 12 | 12 | unavailable | 28 |
| The Bible | unavailable | 16 | 55 | unavailable | 71 |
| Life Hacks | unavailable | 10 | 6 | 11 | 19 |
| New vocabulary word | unavailable | 42 | 22 | 4 | 60 |
| Ehrlich Pest Control | unavailable | unavailable | 14 | 33 | 39 |
| Today's special | unavailable | unavailable | unavailable | unavailable | 0 |
| Fox Chapel School Lunch | unavailable | 4 | unavailable | unavailable | 4 |
| Recipe Finder By Ingredients | unavailable | 8 | 6 | 10 | 16 |
| Go Irish | unavailable | 2 | 2 | 2 | 2 |
| Freedom Festival | unavailable | 23 | unavailable | unavailable | 23 |
| Skyrim Very Special Edition | unavailable | 14 | unavailable | unavailable | 14 |
| Beat the Intro | unavailable | 3 | unavailable | unavailable | 3 |
| The Floor is Lava | unavailable | unavailable | 55 | 37 | 84 |
| Destiny 2 Ghost | unavailable | 18 | unavailable | unavailable | 18 |
| Trivial Pursuit Tap | unavailable | 51 | unavailable | unavailable | 51 |
| Fortnite Skin Picker | unavailable | 16 | 9 | 12 | 26 |
| Simon Says | unavailable | 30 | 29 | 36 | 95 |
| My Capital Quiz | 10 | inconsistent behavior | inconsistent behavior | unavailable | 10 |
| Bender the Great | 13 | 22 | 21 | unavailable | 32 |
| Murder Mystery Theater | 44 | 27 | 19 | unavailable | 48 |
| Country Flag Quiz | unavailable | unavailable | unavailable | unavailable | 0 |
| Head Games | 50 | 22 | 12 | unavailable | 58 |
| The Bingo Caller | unavailable | 21 | unavailable | 85 | 106 |
| Naughty Or Nice Quiz | unavailable | unavailable | unavailable | 5 | 5 |
| Amazon Math | 2 | unavailable | 2 | unavailable | 2 |
| Escape mystery room | unavailable | unavailable | unavailable | unavailable | 0 |
| Trick or Treat | unavailable | 34 | 13 | 23 | 60 |
| Westworld Quotes | unavailable | unavailable | 3 | unavailable | 3 |
| AYRIAL: Feng Shui Tips | unavailable | 11 | 10 | 17 | 32 |
| Daily Horoscope | unavailable | 13 | 5 | unavailable | 14 |
| Cooking Temperatures | 24 | 22 | 22 | unavailable | 26 |
| Coffee Maker | unavailable | 30 | 20 | 13 | 38 |
| Daily Gemini Horoscope | 11 | 13 | unavailable | unavailable | 20 |
| Fortune Teller | unavailable | 13 | 14 | 11 | 33 |
| Daily Pisces Horoscope | 11 | 16 | 12 | unavailable | 29 |
| Allrecipes | unavailable | unavailable | unavailable | 23 | 23 |
| The Audio Chef | unavailable | 21 | 13 | unavailable | 32 |
| Baby Gender Reveal Themes | unavailable | unavailable | 13 | 59 | 72 |
| Grandmother Names | 1 | inconsistent behavior | inconsistent behavior | unavailable | 1 |
| Guest Reception | unavailable | 43 | 31 | unavailable | 65 |
| Utah Jazz | unavailable | 31 | unavailable | unavailable | 31 |
| 48 Laws of Power | unavailable | 21 | 51 | 22 | 84 |
| I spilt lipstick in your Valentino bag | unavailable | 26 | 4 | unavailable | 30 |
| Halloween Jokes | unavailable | 32 | 16 | 18 | 61 |
| Rain Storm for Healing | unavailable | 8 | 3 | 6 | 15 |
| U.S. Postal Service® Informed Delivery® | unavailable | 1 | 1 | unavailable | 1 |
| Smart Home | unavailable | 9 | 5 | 8 | 12 |
| Good Vibes | unavailable | unavailable | 7 | 2 | 9 |
| Chat Bot | 1 | unavailable | inconsistent behavior | inconsistent behavior | 1 |
| ha ha burns | unavailable | unavailable | unavailable | unavailable | 0 |
| My Speaker | unavailable | 19 | unavailable | unavailable | 19 |
| My Metronome | unavailable | 13 | unavailable | unavailable | 13 |
| Casey Neistat | unavailable | 37 | 26 | unavailable | 46 |
| Circle of Fifths Facts | unavailable | 12 | 6 | 6 | 20 |
| The Weather Channel | unavailable | 6 | unavailable | 4 | 9 |
| Parkview Christian Church Homer Glen Campus | 16 | unavailable | 22 | unavailable | 24 |
| SaiSkill | 8 | unavailable | 6 | unavailable | 13 |
| The Rookie's Guide to the NFL | unavailable | unavailable | unavailable | 2 | 2 |
| Share The Facts | 7 | unavailable | 6 | unavailable | 7 |
| LaMetric TIME – connected clock for smart home | 14 | unavailable | 16 | 15 | 17 |
| Vivitar Home Cameras | unavailable | 1 | 1 | 1 | 1 |
| Adventure Game | 70 | unavailable | inconsistent behavior | unavailable | 70 |
| 3 Cards | 17 | 17 | unavailable | 14 | 21 |
| Ten Pin Bowling | unavailable | 49 | 53 | 43 | 72 |
| The Shocking Truth | unavailable | unavailable | unavailable | unavailable | 0 |
| 66 Tolls | 14 | unavailable | 14 | unavailable | 17 |
| Magnificent Facts | 16 | unavailable | 11 | 19 | 23 |
| Explain like I'm five | 6 | unavailable | unavailable | unavailable | 6 |
| My Quest To Teach | 78 | unavailable | 17 | unavailable | 92 |
| Beer Bot | 3 | inconsistent behavior | inconsistent behavior | inconsistent behavior | 3 |
| Park Creek Menu | 3 | inconsistent behavior | unavailable | 3 | 4 |
| Wine Down | 69 | 49 | 28 | unavailable | 74 |
| All Cats | unavailable | unavailable | 7 | unavailable | 7 |
| This Day in History | unavailable | 45 | unavailable | 47 | 89 |
| Crisis Response | 162 | unavailable | 85 | unavailable | 240 |
| Mayor Karen Best | unavailable | 39 | 33 | 50 | 68 |
| Recommend a Romance Author | 20 | unavailable | 13 | 14 | 28 |
| Rock Paper Scissors | 62 | 20 | 33 | inconsistent behavior | 102 |
| Treasure Words | unavailable | 19 | 10 | 17 | 33 |
| Creepy Quest | unavailable | unavailable | 33 | unavailable | 33 |
| Skylanders Imaginators Game Helper | unavailable | 80 | 78 | 68 | 174 |
| RPG Dungeon Room Generator | unavailable | unavailable | 12 | 8 | 15 |
| Space Adventure | unavailable | 100 | 44 | 112 | 218 |
| Russian Escape | unavailable | unavailable | 28 | unavailable | 28 |
| Horrible Movie Plots | unavailable | unavailable | 37 | unavailable | 37 |
| Provinces of Spain | 71 | 59 | unavailable | unavailable | 96 |
| Listening Quiz | unavailable | unavailable | 7 | unavailable | 7 |
| OtakuGeneration | unavailable | unavailable | 5 | unavailable | 5 |
| A Musical Christmas | unavailable | unavailable | unavailable | unavailable | 0 |
| I Am A Triangle | 100 | 96 | inconsistent behavior | unavailable | 130 |
| My Compliments | unavailable | 6 | 3 | 5 | 12 |
| Teams Quiz | unavailable | unavailable | unavailable | unavailable | 0 |
| Creative New Mexico Blog Reader | unavailable | 29 | 30 | 23 | 54 |
| PrimeResi | 19 | 28 | 25 | unavailable | 56 |
| Parrot History | unavailable | unavailable | 23 | unavailable | 23 |
| The Crying Baby | unavailable | 34 | 40 | 38 | 82 |
| Virtual Trump | unavailable | unavailable | unavailable | unavailable | 0 |
| Fairy Milk | unavailable | unavailable | unavailable | unavailable | 0 |
| Quote Me | 55 | inconsistent behavior | 18 | 38 | 107 |
| Fundamental Valuation | 12 | inconsistent behavior | inconsistent behavior | inconsistent behavior | 12 |