How to Game the Best Company to Work For Awards

This is continued from a "Gloves Off Friday Post" by Mike West on Linkedin Pulse here:  How the Best Company Award is Wrong

How The Best Companies To Work For Are Ranked.

Different newspapers, magazines and institutes have different methodologies to rank order companies, however the thing they have in common is that a large portion of the rating is based on an employee survey. 

Below is what it states on the Fortune website:

"To identify the 100 Best Companies to Work For, each year Fortune partners with Great Place to Work to conduct the most extensive employee survey in corporate America.

Two-thirds of a company’s survey score is based on the results of the Trust Index Employee Survey, which is sent to a random sample of employees from each company. This survey asks questions related to employees’ attitudes about management’s credibility, overall job satisfaction, and camaraderie. The other third is based on responses to the Culture Audit, which includes detailed questions about pay and benefit programs and a series of open-ended questions about hiring practices, methods of internal communication, training, recognition programs, and diversity efforts.

Glass Door also bases its rating on survey questions.

"To determine the best places to work, Glassdoor looks at company reviews provided by employees between November 13, 2013 and November 2, 2014, in which individuals are asked to consider and rate such factors as overall satisfaction, CEO leadership, career opportunities, compensation, and work-life balance." 

To the credit of the rating agencies, asking employees at different employers the same questions seems like the fairest and most scientifically valid way to compare employers to each other. 

How to Game it.

Knowing surveys are the most substantial part of the ranking, the key to gaming Best Company to Work For Awards is to know the natural distribution of attitude by employee characteristics and use this information to increase the % of employee surveyed in segments that have higher positive attitude than average, while decreasing the sample rate in those with lower positive attitude than average. 

Relatively unsophisticated rating agencies, such as local newspapers, could easily be exploited in this manner. On the other hand some rating agencies, like Fortune and the "Great Company to Work For Institute" will reply to this with, "we use a random sample". Unscrupulous HR bosses do not be deterred. While using a "random sample" seems like a great way to prevent manipulation, this too can be beat. The unscrupulous HR boss could beat random sampling by proportionally manipulating the quantity of email addresses from different segments he/she provided to the rating agency (determined by an understanding of the natural response of varied segments) and/or he/she could just filter which emails allowed to pass through the email server.  To be clear - I am making a point - I am not suggesting a company should cheat, however if a company wanted to do so this would be how they could do so without directly standing behind employees shoulders while they fill out the survey or offering to throw pizza parties for the groups with the highest results. By the way, you better believe that some companies and managers DO do things like that. 

I am not aware of a specific circumstance where a company has deliberately manipulated a survey process, however I am aware of circumstances where companies have benefited indirectly without the knowledge of the rating agency.

Here is the Problem.  

How positively people respond to questions at their employer varies reliably by certain employee characteristics and these characteristics are not uniformly sampled across all employers.   Characteristics that may statistically matter extend from natural demographic distributions (age, gender, ethnicity..), to natural job type distributions (professional, skilled labor and unskilled labor), to natural geographic distributions, to other characteristics we may not even typically record. I left out the most important. The characteristic that I have found to consistently vary between segments by substantial margin, unrelated to the actual quality of the company, is Tenure Group. Company Tenure is calculated something like this (Current Date - Start Date) and is usually grouped something like this (0-1 Year, 1.1 to 3 Years, 3.1 to 5 years, 5.1 years to 10, 10+ Years).

What it Looks Like.

A typical tenure group pattern looks something like this:

Typical Employee Engagement Pattern by time in job

It is worth mentioning that Tenure Distribution is at least partly driving geographic and industry differences. You can see this if you consider that the labor market characteristics of geographies and industries have a relationship with the proportional distribution of Company Tenures. Faster growing local economies and industries have lower overall tenure so these populations would also have proportionally more people in low tenure groups. 

In the graph below note the growth characteristics of our leading industries. Think about the growth characteristics of the industries to the right.  See it?

PWC 2015 Report

PWC 2015 Report

Does it Matter That Much?

You might say, "Come on!, How much could this problem really matter? Actually a lot!

The phenomenon can be observed in rare cases when the #1 Company Award unexplainably flips away from a company in one year and returns to them in a future year.  What explains the difference is decreased hiring rate, relative to other nearby companies on the list.

Statista

This is a little far fetched but the other thing you could do to game the award is to hire a large number of people right before the time of year of the awards and/or right before you apply for the award the first time.  I can't say anybody does that to win awards of this nature intentionally, or anyone ever would, but some benefit from a massive growth rate that ensures this will happen for them whether they try to do it or not.

Should We Care?

"Google has been on the list for 10 years with this being its seventh time at No. 1, thanks to sparking the imagination of its talented and highly compensated workers, and by adding perks to an already dizzying array of freebies." 

The first reason we should care is that the companies that win these awards receive substantial press and as a result receive a remarkable increase in the amount of job applicants. Think something akin to a million new applications to Google. If this is coupled with an increased ability to filter job applicant pools to identify high quality candidates then these #1 picked employers have a substantial competitive advantage in ability to select the most highly qualified workers. Further, these employers have a PR gains from which to take key talent away from other companies and keep their own key talent.

Another reason we should care is that many organizations try to imitate the "Best Practices" of the companies that are highly ranked on these lists. The companies that want to be like them may unfortunately be imitating characteristics that have no actual relationship to what makes a great company to work for, or the reported survey results and therefore arbitrary.  Recall, correlation does not imply causality. Trying to imitate all of the practices of the purported great companies may result in investments that generate no return and simultaneously decrease margin, thus making it more difficult for the imitating company to compete in the future. This could provide substantial advantages to companies that can make the top of the list AND afford to give up a small portion of big margin to spend above average on salary and unusual employee perks.

I have written extensively about how Best Practices lead us astray in a prior blog post : 7 Reasons Best Practices Are Not Best For You. 

What can be done about it?

For starters the newspapers, magazines and rating agencies could sample survey responses in tenure groups to ensure an apples to apples comparison. Instead of a random sample this would be called a "Stratified Random Sample".  If they really wanted to step up their game they could also just put all of the data into a single multiple regression model, to isolate a company effect from tenure effects and any other variables that may skew results, be they demographic, job related, geography or whatever. This sounds complicated but actually any undergraduate statistics major or any graduate behavioral level science major could run this analysis.  

Now, as I state repeatedly, I am not suggesting anyone should really try to game the Best Company to Work For Awards, however I can understand why you would want to up your game to truly improve employee engagement and be a truly great employer.  The best way to do so is to look across data sets and use the engagement data in increasingly better ways to get better at actually moving engagement. Survey providers are good at managing the process of constructing a good survey and collecting data but provide a very limited view of the data and no survey providers work with their survey data, plus your other data sources to provide a single longitudinal view of the truth.  No survey provider maintains ongoing relationship with your sources that adjust with underlying structural changes automatically and that you can query in real time. I know of some employers who can look at survey data in this way, however they cannot do so while maintaining employee confidentiality as a survey provider would.  The world is now in luck - One Model can take data from a survey provider (or tool) and join it into a single view of the truth with other employee related data, allowing a longitudinal view, update automatically, and most importantly, can do this while maintain employee confidentiality just a survey provider would (by not allowing you to report data below a sample size threshold) :-)  If this interests you, let us know and we would be happy to provide you with a demo so you can see for yourself what sort of new advantages this can give you!

----------------------------------------------------------------------------------------

This is a continuation of a "Gloves Off Friday" post : How the Best Company Award is Wrong

More like it:

Why Josh Bersin is Wrong About Embedded Analytics?

The Most Dangerous Technology in HR Today

What Your HR Technology Sales Rep Doesn't Want You To Know

----------------------------------------------------------------------------------------

Who is Mike West?

Mike's passion is figuring out how to create an analysis strategy for difficult HR problems. 

Mike has 15 years of experience building People Analytics from the ground up as an employee at the founding of Merck HR Decision Support, PetSmart Talent Analytics, Google People Analytics, Children's Medical (Dallas) HR Analytics, and PeopleAnalyst - the first People Analytics design firm - working with Jawbone, Otsuka and several People Analytics technology startups. Mike is currently the VP of Product Strategy for One Model -the first cloud data warehouse platform designed for People Analytics.

Connect with Mike West on Linkedin | follow Misc-People Analytics | or join One.PeopleAnalytics.Community