How big are the orca, shark, etc. models parameter wise? We know OpenAI’s davinci model is around 175B, and AI21’s jumbo model is around 178B. How big is orca?
Hi Delano, welcome to the community!
hey, that’s a shame. I understand your need to keep data confidential, but when you have at least several models to choose from (OpenAI and AI21 being the most prominent), you can’t just rely on arbitrary measures of “performance”…
I hear you, Vova. I’d invite you to try it out for your specific use case and let us know what kind of results you get. The team will be happy to advise on prompt engineering and finetuning file structure, if needed.
I agree, the performance seems pretty good.
But I’m sure the number of parameters are far fewer than OpenAI and AI21. How are you guys able to keep the quality this high?
IMHO parameter count isn’t actually a very meaningful measure of utility for LLMs. A lot of work (for instance PET or T0) suggests that model size isn’t indicative of performance when the data distribution is changed. Compression work shows I can delete ninety-something percent of parameters and preserve task performance.
Parameter count definitely gives you a rough idea of performance, but the real number we care about is task-performance on our users’ tasks. We have a “feedback” feature on the roadmap where a user could let us know if the model’s response is right or wrong so we can metric ourselves better and constantly aim to optimise those more meaningful performance metrics.