On 29th September 2015, the MHRD announced its decision to come up with a National Institute Ranking Framework (NIRF), “outlining a methodology to rank institutions across the country.” Every year since then, the MHRD has released these rankings, which are calculated using a set of metrics that have been amended throughout the years. The latest annual ranking exercise NIRF has shown a few consistent performers in the Engineering Category with IIT Madras, IIT Delhi, and IIT Bombay sharing the top three positions for the third consecutive year, while IIT Kanpur moved up a spot to fourth.
The MHRD claims that the NIRF score is meant to be “a reflection of where the institution is standing vis-a-vis other institutions in the same category.” But are these scores a fair assessment of an Institute’s overall performance? How reliable are these rankings? We try to answer these questions by considering two different, but equally important aspects of the ranking framework- the data, and the parameters used to calculate the rankings.
Errors in data submitted by institutes
A significant component of the NIRF rankings is derived from data that had been submitted by the institutions themselves and not been verified by NIRF itself at any stage. To allay this serious concern over the integrity of the rankings, NIRF had claimed that it would be conducting periodic random verification of the submitted data and mandated the institutions to publicly display it. With different rankings getting released each year and the fourth edition having been released recently, so far no reports have been released on the verifications which NIRF had claimed it would conduct.
We studied the data from the Top 10 institutes, looking through the various metrics under which they had submitted their data.
IIT Bombay had submitted data to NIRF under 3 categories: Overall, Engineering, and Management. Under the Overall Category, the Median Salary of a student in the UG 4 Year course, graduating in the year 2019, had been reported as Rs. 27,87,638[1]. This value is abnormally greater than similar data from other peer institutes like IIT Delhi(Rs. 15,70,000)[2], IIT Madras(Rs. 13,60,000)[3], but more than that, it is nearly triple the amount reported by IIT Bombay itself for the previous years (For eg, in 2018- Rs. 10,11,000 and in 2017- Rs. 9,56,757)[1]. Similar is the case for the data presented under the Overall Category for the Median Salary of a student in the PG 2 Year course graduating in the year 2019, which has been reported to be Rs. 50,93,691[1] while the same value for the previous 2 years had been under Rs. 32 Lakhs[1].
This is suggestive of an abnormality in the data submitted by IIT Bombay which necessitated further investigation. A report concerning the placements for the Year 2019, published in July 2019, by Placement Office, IIT Bombay, reports only 204 offers having been made across the entire graduating batch, with a salary over Rs. 16,00,000[4]. This is at odds with the data submitted to NIRF because, on inferring that, we must have had at least (459(Placed UG 4 Year students, NIRF)+577(Placed UG 4 Year students, NIRF))/2=518 offers made of over Rs. 27,00,000[1].
(Source: [1])
(Source: [4])
The saving grace in the entirety of this has been the fact that incidentally, NIRF does not take into account the Median Salary for calculating the overall rankings[5]. Apart from this particular instance of misrepresentation of data this year, there have been similar reports coming in with previous editions of NIRF Rankings. According to a report by The Telegraph, The University of Calcutta, which was placed at 5th amongst all universities in NIRF 2019, had reported the median salary for undergraduates who graduated in 2015-16 to be Rs. 8,04,400 but in the data presented for NIRF 2017, the same data was much lower at Rs 4,00,000 for the same set of students who had graduated in 2015-16 [6]. Considering the fact that our own study involved only a very small subset from the 1000s of institutions in the rankings, it is possible that other instances of erroneous data might exist.
Due to the nature of this data coming from official sources, it poses a problem in which any person could end up getting misled by such false data regarding the placement statistics of the outgoing batch. It is even worse a situation if institutions had been deliberately sending in inflated figures, seeking to raise their rankings. Even the universities and the institutes are a stakeholder in this since these rankings might determine the amount of funding they are able to secure for themselves[7], perhaps something which might motivate further fudging of data.
Are the parameters fair?
At first glance, the metrics used by the MHRD to rank the institutes look holistic and reasonable. As explained in one of our earlier infographic articles concerning the NIRF rankings[8], the framework takes into account the following parameters: “Teaching, Learning and Resources,” “Research and Professional Practices,” “Graduation Outcomes,” “Outreach and Inclusivity,” and “Perception”. But an in-depth analysis of the parameters and sub-parameters reveals some underlying flaws in the ranking framework.
The Dual-Degree and IIT K advantage:
One of the shortcomings of the rankings lies with its failure to properly accommodate different academic structures under a common framework. This inability to standardize diverse structures results in some anomalies in the scores, and as a consequence, distorts the rankings to some extent. An example of this is evident in the NIRF’s treatment of Dual Degree programs. Most IITs offer explicit 5-year Dual Degree programs during the time of admission itself, whereas IITK offers no such program. Rather, IITK Undergraduate students are offered a chance to switch to a Dual Degree program during their 3rd and 4th year.
Due to this, the NIRF data submitted by IITK has no separate section concerning UG 5-year students, unlike other IITs. Instead, these Dual degree students are listed under the category of “UG 4-year students opting for higher studies”. This induces a spike in IITK’s numbers in the category. The number of UG (4 years) students opting for higher studies in the other (top 4) IITs varies between 50-150, while for IITK, this number is to the north of 300.
This anomaly gives a small boost to IITK’s score in the Graduation Outcome(GO) parameter.
(Source: [9])
Consider the following two scenarios:
Scenario 1: A total of 800 students take admission under the UG 4 year program in 2016. Out of this, 200 students opt to convert to a dual degree program at a later stage. Out of the remaining 600 students, 500 students get placed/opt for higher studies from a different institute.
Scenario 2: 600 students take admission under the UG 4 year program, while 200 students get admitted to a dual degree program in 2016. 500 out of the 600 students get placed/opt for higher studies from a different institute.
The two scenarios are virtually similar, albeit a difference in the academic structure of the institute. We would expect a similar graduation outcome metric for these two scenarios. But, the ranking metric used in NIRF results in significantly different outcomes.
In the first scenario, the graduation outcome score (for the 2020 rankings) will be proportional to the ratio ⅞, while in the second scenario, it will be a factor of ⅚ . And although this may sound like an insignificant difference, the scaling of these factors leads to a significant spike in an institute’s score.
In general, there is a huge diversity in the academic framework of institutions. A failure to normalize these diverse structures may cause a significant distortion in the institute rankings.
Problems with State Diversity Index:
Another problematic metric in the ranking framework is the State Diversity Index. The metric intends to quantify the regional diversity of an institute, by taking into account the fraction of students from other states enrolled in the institute. But the parameter tends to unfairly penalize institutes located in states with a large population. For example, Uttar Pradesh, being the most populous state, houses more than 17% of the Indian population. As a result, a random distribution of students will tend to have a greater percentage of students from Uttar Pradesh, as compared to any other state. Consequently, IITK has the highest percentage (~28%) of native students, among the top 5 IITs. This naturally affects IITK’s performance in the metric in a negative way. Similarly, IITB also ends up having a low State Diversity Index, owing to the large population of the state of Maharashtra. A better metric could be formed by normalizing the diversity index with the population of the state in which the institute is located. This would remove the bias towards states with a large population.
The number of PhDs graduating:
The NIRF also has a metric for the number of PhD students graduated. The metric takes into consideration the mean of the number of PhD students that have graduated from the institute in the past 3 years. In the past years, IITK has consistently reported a lower number of PhD students as compared to the other top 4 IITs.
(Source: [9])
The number of research scholars that an institute can host is heavily governed by the number of faculty at that institute. A larger teaching community can naturally supervise a larger student community but the metric stated above takes into account only the raw number of students graduating over the past three years into consideration. A smaller number of graduating students could set a wrong perception of the institute and could question the capabilities of the faculties and infrastructure provided. IITK has the lowest number of faculty members among the top 5 IITs.This number is influenced by a lot of factors, one of them being the faculty strength position sanctioned by MHRD. The sanctioned faculty strength for IITK is considerably less than some of its counterparts. Thus it is natural to assume that IITK would lag behind in the number of students graduating from the institute as well.
(Source: [10]
The PhD program is not as aligned as the conventional UG and PG programs. The graduation outcome of a PhD program is dependent on a number of variables such as the faculty strength, the average time taken for completion of the degree, and the number of departments in an institute among others.
The current method of using the mean of the number of PhD students graduating over the last three years, seems a bit primitive. NIRF should try to incorporate the different variables listed above into their framework as well, in order to construct a metric which can give better insights into the PhD program of an institute.
Conclusion
The possibility of errors in the data submitted by institutes, combined with the unfairness of some parameters, raises some genuine concerns regarding the reliability of the NIRF rankings. The extent of the combined distortion these factors can cause is significant. Considering the wide media coverage these rankings receive, they can be misleading sometimes and might have an unfair impact on some Institutes’ reputations. At best, these rankings are just indicative and should be taken with a pinch of salt.
Writing Credits: Aayush Priyadarshi, Akshat Goyal, Shiven Tripathi
Editing: Aditya Sonthalia, Pulkit Deshmukh
References:
[1]. Data submitted by IIT Bombay to (NIRF) in the year 2020, NIRF-Overall
http://www.iitb.ac.in/sites/default/files/DATA_SUBMITTED_FOR_OVERALL_CATEGORY.pdf
[2]. Data submitted by IIT Delhi to (NIRF) in the year 2020, NIRF-Overall
https://home.iitd.ac.in/uploads/NIRF-Overall-2020.pdf
[3]. Data submitted by IIT Madras to (NIRF) in the year 2020, NIRF-Overall
https://www.iitm.ac.in/sites/default/files/NIRF/Overall-final.pdf
[4]. Placement and Internship Report,2018-2019, Placement Office, Indian Institute of Technology, Bombay, July 2019
http://placements.iitb.ac.in/files/IIT_Bombay_Placement_and_Internship_Report_2018-19.pdf
[5]. Methodology for Ranking of Academic Institutions in India, (RANKING METRICS FOR OVERALL), NIRF India Rankings 2020, Ministry of Human Resource Development
https://www.nirfindia.org/nirfpdfcdn/2020/framework/Overall.pdf
[6]. Samantak Das, “The idea of the university is devalued under the current government”, The Telegraph Online, 26th April 2019
https://www.telegraphindia.com/opinion/the-idea-of-the-university-is-devalued-under-the-modi-government/cid/1689408#
[7]. M Saraswathy, “NIRF ranking may soon determine funding for educational institutes”, Moneycontrol.com, 13th September 2019
https://www.moneycontrol.com/news/business/economy/nirf-ranking-may-soon-determine-funding-for-educational-institutes-4435201.html
[8]. Vox Populi, “IITK in NIRF rankings”
http://voxiitk.com/iitk-in-nirf-rankings/
[9]. Metrics for ranking institutions in the Engineering category
https://www.nirfindia.org/nirfpdfcdn/2020/framework/Engineering.pdf
[10]. Faculty strength statistics
https://www.iitsystem.ac.in/?q=facultystas/pview&year=2019-2020
