A Scouting Manifesto

A Scouting Manifesto

If you’re reading this it’s too late you really like basketball. More specifically, you probably are really interested in the NBA draft, and evaluating future NBA talent. The NBA Draft is fascinating for many reasons, but the inaccuracy is part of what makes it so compelling. Coming into the year, not a single person would have suggested that Minnesota was the league’s 2nd best team, or that Ramon Sessions would be the league’s second best player. However, if you go back and look at the 2009 NBA Draft, you can see that Hasheem Thabeet was picked second overall! An actual NBA front office had looked at the hundred or so potential NBA-level guys who they could draft that year, and decided Hasheem Thabeet was the second best option of that entire player pool. This is a player pool that also included MVP frontrunners James Harden and Stephen Curry, while Thabeet isn’t even in the league anymore. What makes this so incredibly crazy is that no one thought this was crazy at the time. Sure, some people disagreed and maybe would have suggested taking Thabeet a couple picks later or so, but by and large, the smartest basketball minds in the world thought this was a reasonable decision.

The NBA is not some solved experiment, no one knew the Warriors would be this good or that Hassan Whiteside would shatter our collective minds, but nowhere is the NBA more unpredictable than the draft. There are obvious reasons for this; anyone predicting the futures of 19 and 20 year-olds is up against formidable odds, and NBA success doesn’t always reflect where a player “should’ve” been drafted. Klay Thompson probably would’ve turned into a good NBA player on whichever team he went to, but it’s doubtful he would have become the borderline star that he is if it hadn’t been for a variety of factors specific to what he’s gone through with Golden State. The NBA Draft will never be even close to fully accurate, but by refining and self-evaluating the way in which we scout NBA prospects, we should be able to improve the accuracy of the process.

There are four fundamental elements to evaluating any prospect: the eye test, their tool package, predictive models based on their numbers, and off-the-court stuff. The eye test is what we think of as traditional scouting; you watch a player play, and from that draw conclusions about how good an NBA player they can be. People can have varying “eye tests” on a player, and the eye test has the most variation of any form of player evaluation. Predictive models take a given prospect’s stats from wherever they’re playing, and then project that player to the NBA based on how players’ stats traditionally translate to the NBA. Oftentimes player analysis is just split up into those two categories, but in terms of the draft, players are often evaluated strictly based on their “tools”.

You’ve probably heard something like “he’s a good shooter, has secondary creation skills, and has the athleticism to be a good defender, he can be a 3-and-D guy in the league.” Analysis like this is obviously founded in the eye test, but you don’t really have to watch a player to figure this out. In this example, if you just looked at a player’s stats and saw that they shot 40% from three, had a decent amount of 2-point attempts, and averaged a couple steals or blocks per game you could come up with an identical statement about the player. People don’t actually evaluate prospects like that, but it often is something similar. On some level this is just a part of the eye test, but I think it is an important distinction. When people say “Kelly Oubre has good tools” they don’t mean they’re impressed with him when they watch him, they’re saying that he has a skill set that easily fits in the NBA.

Most of the time a player’s off-court actions and personality don’t impact their draft status very much, but in the context of someone like Robert Upshaw, who has been dismissed from his college, it makes the player a bigger risk. On the flip side, when someone is known as a hard worker who really loves the game, that should contribute positively to their prospect status. When determining a player’s value as an NBA player, it is imperative to look at all four of these things, but figuring out how to balance them is the tricky part.

The classic debate is obviously the one between the eye test and analytics. Draft analytics are not really a part of the public perception of the draft; you might see a player’s PER or RPM shown on TV, but at this point you would never even hear draft models discussed, much less see his EWP or WARP projection. However, there isn’t really any obvious reason for this. Draft models are much worse at predicting player value than a metric like RPM is at representing it, but the same goes for eye test evaluation of draft prospects in comparison to eye test opinions on NBA players. Draft models like Layne Vashro’s EWP, Kevin Pelton’s WARP, or even John Hollinger’s Draft Rater are not better than the eye test, and definitely should not replace the eye test. Still, draft models have consistently done a good job of predicting NBA player performance, and in many cases been more “right” about a prospect than scouting consensus.

In Vashro’s EWP model, Hasheem Thabeet was not 2nd overall, but rated as the #9 prospect, behind both Stephen Curry and James Harden. Thabeet was not the 9th best player in his class, but that’s a lot closer to reality than 2nd. Now I can also find countless examples where scouting consensus was superior to draft model performance, but the crucial point is that both have value. Simply ignoring draft model performance would be foolish; models have clearly shown they have predictive success and value that the eye test does not always capture. Factoring in draft model performance is easier said than done. Tyus Jones is widely considered a late first-round level prospect, so most people have him somewhere between 20-30 on their boards. Looking at draft models paints a different picture; he is 2nd in Kevin Pelton’s numbers, and 10th in Layne’s. Tyus shouldn’t now be considered a top 10 prospect, but it means there is a decent chance he is being underrated.

It is easy, and sometimes even correct, to rationalize dismissing the numbers in a case like this if you personally don’t think Tyus can succeed in the NBA, but his numbers should not be ignored. There is a clear history of draft model performance having value, and you are just hindering yourself if you don’t pay attention to them.

There are cases in which someone’s numbers should be subjectively given less weight, and Tyus could be an example of that. Draft models are inevitably going to have a harder time figuring out how players translate to the defensive end of the floor than the offensive end. The only widely available defensive stats in college are steals, blocks, and rebounds. A player can put up good or solid numbers in those categories and still be a poor defender, and likewise, a player can look pedestrian in those measures when, in fact, they’re a great defensive prospect. In many cases, looking at a player’s defensive numbers should cause you to rethink your perception of them. Draymond Green was widely considered a poor defensive prospect, but his underlying college numbers were actually very impressive. Scouts saw Draymond as someone without a typical NBA body, and therefore cognitive biases caused them not to notice his startlingly quick feet and incredible defensive awareness.

The numbers are not always representative of defensive ability, but they should at least make you think harder before you come to a conclusion on a player. Numbers can be misleading in other ways too; if a guard plays zone, it can inflate his rebounding and steal numbers, and that should be kept in mind when doing draft analysis. On the other hand, if a player was battling through injury for part of the season, then you may have reason to think model scores are lower than they should be, and you can choose to place less emphasis on them. Unfortunately, there is no one way to do this. In some cases the numbers are going to be right, and in other cases, the eye test will be right. Sometimes the eye test and the numbers will agree, but the player will still under or over-perform expectations.

Tools are a tricky thing; at times they can have almost no value, but they also can be a crucial part of player evaluation. Tool evaluations boil a player down to their basic skills. They ask: what is a player going to be able to do to fit in at an NBA level? Michael Beasley is a great example of this; he looked like a really good NBA player, and the numbers backed that up (#2 in Layne’s model that year). He failed partially because of off the court stuff, but also because he just doesn’t have the tools to really fit in at the NBA level. He’s not a great shooter, he’s not an elite athlete, he’s not quick enough to guard 3’s, and he’s not big enough to guard 4’s. Obviously, it’s easier to say that now than it was in 2008 because there was hope he could be a good shooter, or that he was quick enough to defend small forwards. These were questions at the time, and are a big reason why he never turned into a useful NBA player.

Many of the same questions can be applied to Jabari Parker, but Jabari could just as easily become an example of the flaws of looking at a player’s tools. It’s possible Jabari is just so good it doesn’t matter; if Jabari can be a high usage scorer with decent efficiency, he is going to be a valuable player, tools and fit be damned.

Last year around the draft, Jabari was often described as the draft’s “safest” player, but this seemed ridiculous to me. Jabari had big questions tools-wise, and was not loved by analytical models. I’ve come to believe a player’s “safety” can be measured through how well rounded he is in each different aspect of player evaluation. Looking at last year’s prospects, no one was truly safe; no one was loved by scouts, had great tools, looked good in draft models, and didn’t have off the court concerns. Jabari had questions both tools-wise, and draft model-wise, making him a somewhat risky prospect. Joel Embiid was liked by scouts and draft models, and had great tools, but his foot injury is a massive off the court risk. Under my evaluation process, Andrew Wiggins would be the safest prospect of the group. He wasn’t loved by draft models, but they didn’t hate him, and many thought Bill Self’s system suppressed his numbers. In my mind this is the best way to evaluate player safety or “floor” in the draft, but it is not a way to rank prospects.

A player’s ceiling or upside is arguably more important, and floor doesn’t have much to do with that. My system for evaluating ceiling is relatively simple, a player’s ceiling is basically how good they seem at their best category. If a player has out of this world tools, puts up incredible numbers, or is viewed as a potential strong #1 pick by scouts he undoubtedly has a super high ceiling. Being strong in multiple categories should only increase a player’s ceiling. Embiid had top-5 pick level numbers, #1 pick level scouting, and #1 pick level tools, so it’s fair to say he had the highest ceiling of any draftee last year. Wiggins had #1 pick level scouting, top-5 pick level tools, and top-20 pick level numbers, but being exceptional in just one category meant he had a super high ceiling, just not quite as high as Embiid. Jabari had #1 pick level scouting, top-10 pick level numbers, and merely first rounder level tools. Him being considered a possible #1 pick alone meant he had a very high ceiling, but not as high as Wiggins’ or Embiid’s.

Floor and ceiling are not universal, and different people can grade players differently in any of the four categories. Draft model evaluation can even differ based on how someone weights different models, and if they think there are reasons to adjust a player’s numbers. Getting a feel for a player’s floor and ceiling is key, but is not the end of the story. Jabari had both a lower floor, and a lower ceiling than Wiggins, but that doesn’t automatically mean Jabari was a worse prospect. Picking players in the NBA Draft at a basic level is just risk/reward assessment. A player can have both a lower ceiling and lower floor than another prospect, but if I judge his median outcome to be higher, he still might be a better pick. There is no “right” way to predict NBA draft prospect success, but by making sure you have a good evaluation process, I believe both fans and teams can do a better job in the draft. Over the next couple months, I’ll be applying this process, and attempting to both predict and write about this year’s draft class as best as I can.


One thought on “A Scouting Manifesto

  1. Pingback: Karl Towns Is Awesome, How Does He Stack Up With the Best Recent Big Man Prospects? | Wingspan Addicts

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s