C3-IoC job role matching accuracy increases monotonically with the number of questionnaire questions answered, reaching about 80% accuracy with four questions and nearly 100% with 24¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2–r3 · q2
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Community assignment accuracy for dummy user profiles increases from about 80% with four questions to nearly 100% with 24 questions answered. → Adán José-García 2023 q2 i? Success rate of placing the true job role in the top 10 matches increases rapidly with more questions; a percentage equal to or higher than 90% is obtained from 10 questions onwards. → Adán José-García 2023 (2)

Evidence¶

Adán José-García 2023¶

Adán José-García, Alison Sneyd, Ana Melro, Anaïs Ollagnier, Georgina Tarling, Haiyang Zhang, Mark Stevenson, Richard Everson, Rudy Arthur. (2023). C3-IoC: A Career Guidance System for Assessing Student Skills using Machine Learning and Network Visualisation. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-022-00317-y

q2 · i? · design · r3

Validation experiment constructing a dummy user for each of the 381 general job corpus job roles and varying answered questions Q from 4 to 24. The article reports the accuracy increase "from about 80% when answering four questions to nearly 100%" with 24 questions; no standardised effect size is printed.

"In general, it is evident that the performance of the assignment of dummy user profiles to the correct communities continually increases as more questions are answered. The accuracy increases from about 80% when answering four questions to nearly 100% when the number of questions answered increases to 24 questions."

Adán José-García 2023 (2)¶

Adán José-García, Alison Sneyd, Ana Melro, Anaïs Ollagnier, Georgina Tarling, Haiyang Zhang, Mark Stevenson, Richard Everson, Rudy Arthur. (2023). C3-IoC: A Career Guidance System for Assessing Student Skills using Machine Learning and Network Visualisation. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-022-00317-y

q2 · i? · design · r2

Numerical evaluation of Success Rate Percentage SRP(k) for k = 1, 5 and 10 over dummy users for Q = 4 to 24. The article reports the top-5/top-10 success rate "increases rapidly with the number of questions answered"; no effect size printed.

"As expected, the success rate for matching the true job in the top 5 or 10 best matches increases rapidly with the number of questions answered. However, when the exact match k = 1 is required, the success rate is initially proportional to Q"

Discussion¶

Related Claims¶

• Exact-match (k=1) job role prediction requires approximately 24 questionnaire questions to achieve an average success rate of 90%, while at least six questions yield a top-10 success rate above 70% — related
• LLM accuracy increases monotonically with prompt sophistication across semantically equivalent medical questions, with Low Literacy users scoring significantly below Prompt Engineer users — related
• User trial participants rated the C3-IoC system positively for usability, with the Job roles map and Job roles overview the highest-rated components — related