http://picasaweb.google.com/vchilana
Saturday, July 10, 2010
Photos of Amsterdam and IACAT Conference
It has been quite sometime that I added here. Will post soon the link to my paper presented in the First IACAT Conference in Arnhem from June 7-9, 2010. The link to the pictures is:
Wednesday, March 24, 2010
Testing Organisations 2020
How Technology will reshape Recruitment/Selection Testing Organisations
Prof.VipinK Chilana
Recruitment testing organizations world over have different operating structure. Some of them are Government bodies, some are created by Public Sector organizations and some are privately owned. Besides the ownership and different clientele, the major way they differ from each other is the verticals and functions contained and performed in-house, in-sourced and out-sourced and the prominent flavor on which these were started/incorporated.
Institute of Banking Personnel Selection
A brief introduction of IBPS would not be out of place to understand the context.
The Institute of Banking Personnel Selection (IBPS) had begun its operations as Personnel Selection Services (PSS) Unit of National Institute of Bank Management (NIBM) in 1969. At that time the idea was to demonstrate to the Banks how these services – recruitment, promotion and placement could be handled. The function of providing the prototype was extended to providing the actual services with all the academic and operational inputs as well as managing the projects. In 1984 this unit became an independent entity - IBPS. While carving it out from NIBM, the first Chairman of its Governing Board was Dr Manmohan Singh who was the Governor RBI at that time. IBPS was envisioned as both an academically and research oriented institute and an operation handling organization, with respect to its mission of optimizing human resource development through personnel assessment. The academics revolve around learning, teaching, conducting research, practicing and publishing. Each has its own importance in the spiral. An ideal situation is Researcher-as-a-Professor and Professor-as-a-Researcher. Additionally, the operations element was included to provide academic grounding and enable direct application of the knowledge gained through research. The operations were also based on the foundation of Psychometrics. The model was Researcher-Professor as Operations Chief.
If an international parallel is to be drawn, IBPS was patterned on ETS – Educational Testing Services, Princeton known as Mecca of testing. The effort was to build and steer an operation based organization with research and academic flavor. It was a fine balancing act in which the operations, academics and research were to support each other to strengthen each component. While academics provided backbone for the content, the action research strengthened operations as well as content by way of validation and demographic data studies. In an organisation like IBPS the research may not be a “typical” hypothesis testing type. It is more of an action research. The objective is to improve the academic and operational processes. This type of research may not earn accolades from academia but adds value and efficiency both to IBPS operations and the beneficiaries of its services. It is rare that when the work prominently is of operations the fulcrum is academic and research oriented. For that reason, the designations and pay scales are commensurate with an academic body rather than a large corporation.
The initial years were of out-sourcing both printing and result-processing work but content development chiefly was in-house. The Psychometrics involved in the processes was also driven internally and rigors of item writing, item review/pre-testing, item analysis after the exam and validation studies etc. were maintained. After some time the limitations of out-sourcing printing and processing through data punching were realized. These pertained to some aspects of Security or Speed or Accuracy or Control or quickly adapting to the unique requirements of the client. During that period OMR technology was also growing and it was felt that scanning and printing should be built as in-house capabilities, given the number of exams and diversity of requirements. Having its own high capacity printing press and 100% in-house high volume scanning capability were the hallmarks that distinguished IBPS from any other such organization in India, be it UPSC, SSC, Railway recruitment Boards, other State level Boards or the private organizations in this business. The ownership of these two essential capabilities enabled the development of many unique processes, in-house software and operational practices to cater to the diverse requirements of IBPS clients. The solution and delivery became an integrated whole rather than sum of different parts.
To bring home the advantages of such integration, I would narrate one experience. IBPS had good association with Prof. John (Jack) E. Hunter of Michigan State University. Jack had exceptional mathematical and quantitative facility. He wrote his own computer programs for analysis and very generously distributed the software. I, being one of the regular users of his software, once shared with him that I found it easier to run Factor Analysis using his Package compared to even SPSS. His matter of fact response was that it was written by a Psychologist who also knew programming and not the other way round. What it meant was that two functions when integrated either at individual level or Institutional level result in a better solution more suited to the unique requirements. Such an integration of purpose underlies the balancing maintained at IBPS between research and operations, as well as solution and delivery.
Technology and IBPS
The last decade has seen the advent of web based technology in testing as an alternative to the conventional paper-and-OMR approach.. There are two major areas in which the paradigm has shifted. One is the application (applying) process of candidates for jobs, including communication with the candidates for exam related information and the other is delivery of exams online. Another very significant growth area, enabled by technology, is the feasibility of Computer Adaptive Testing (CAT) for mass level selection. In fact it is this marvel of psychometrics on the ever improving backbone of technology that is best suited model for large scale online exams.
One area in which IBPS has helped change the rules of the game is online applications. In fact almost 100% recruitment projects of Public Sector Banks (PSBs) are online. The online application has been adopted by several different Banks with IBPS providing customisation suiting to the requirements of the level of jobs, type of recruitment and branch network of the bank. Added to the online advantage is the payment of fee in any branch of the bank rather than purchasing Demand Draft or other such mode of payment. There are many advantages to the applicant in each of these methods. The summary of two major variations is as follows:
Fully online - no print out or document or fee receipt to be sent
In this mode, the candidates go to any designated branch of the bank to pay the fee against payment challan which can be downloaded from Bank’s web-site. Then they fill the application online with payment details and retain a copy of the print-out and challan with them. In this mode the biggest advantage is that it is not affected by postal delay or loss.
Application online but print-out of registered application with DD/payment challan required to be sent
In this mode all the above steps are followed but unless the print-out with DD etc is received at the given post bag number within the stipulated time, the application is not valid.
IBPS will coordinate more than 8.5 million registrations for PSBs in 2009-10. For one clerical project of SBI for 11000 vacancies, about 3.7 million registrations took place in a span of 40 days. Sify, a leading technology provider, partnered with IBPS in handling this project. IBPS also built its in-house capability for hosting this service albeit on a smaller scale to get an insight into this area.
IBPS had also conducted online exams through Sify and other technology partners for volumes upto 2000 candidates in a session. One of the earlier IBPS articles-“Will CAT be CAT?”- had the following opening sentence - “The decision of the Indian Institute of Managements (IIMs) to conduct CAT-2009 online has the potential to change the landscape of testing - if it works well, technological glitches notwithstanding.” I don’t subscribe to the CBT mode in which the exam was conducted for such a big number and when the scheme of equating scores is not shared. However, the general public was more concerned with the technology based delivery rather than more pertinent Psychometric questions. In spite of all the glitches and despite it not being an Adaptive test model, it has paved way for online exams in a big and bold way. Someone had to start at this level with such a big volume. Who else if not IIMs?
Another trend is that more and more technology companies are coming into the sphere of testing. For example, a new company is set up only to provide online testing services and all the capability is generated around that. For such companies the initiation and core competence is technology for delivery of exams and not Psychometrics of testing. This lack of integration, as discussed previously in the context of IBPS, can mean not only a lessened psychometric component but a lessened overall service.
Technology, Testing, and the Future
What would be the enhanced and next version of this technology-enabled online application and testing paradigm? With pressure also from “save the planet” campaign; the shift will be towards a paperless process. This mode will also take away the role of Post Office and will significantly reduce the recruitment cycle to less than half the time it takes now. The major steps and time frame may be:
Apply online after payment of fee in the Bank branch (many other simpler modes of fee payment would evolve): A window of about 20 days is provided for this.
Download call letter within 15 days of closing of application (modes of immediate allotment of test centers would also evolve)
Take the test on-line based on adaptive testing (about 10 days window), commencing from within 7 days of hosting call letter downloads. (The dates would be indicated in the notification itself)
Result within 2 days of closing of online test window.
The whole process is reduced to less than two months as compared to more than five months now.
The basic question is which of the following organizational models fits the bill:
A pure technology company
The traditional organization with outside technology partner(s)
A technology initiated company trying to acquire basic Psychometric inputs
A Psychometric core competence based company evolving as a technology company.
The first two types of organizations would not be able to sustain and cope up with the dynamic situation of ever evolving requirements and diversity. The last two would be better suited. But though technology will bring and enable all these changes, the driving force will be technology driven Psychometrics.
This in any way doesn’t imply just creating one vertical called Technology in the organisation or acquiring a technology company to deliver the online function. Simply acquiring a technology-enabling partner, like ETS did with Prometric, would help adapt better and improve response time. But this in itself is not the integration. A Testing company essentially will have to become a technology company. What this means is that it has to be integrated at individual as well as organizational level. The thinking and process of Technology and Psychometrics will go hand-in-hand with the latter being the initiating force for the new mode.
Institute of Banking Personnel Selection
A brief introduction of IBPS would not be out of place to understand the context.
The Institute of Banking Personnel Selection (IBPS) had begun its operations as Personnel Selection Services (PSS) Unit of National Institute of Bank Management (NIBM) in 1969. At that time the idea was to demonstrate to the Banks how these services – recruitment, promotion and placement could be handled. The function of providing the prototype was extended to providing the actual services with all the academic and operational inputs as well as managing the projects. In 1984 this unit became an independent entity - IBPS. While carving it out from NIBM, the first Chairman of its Governing Board was Dr Manmohan Singh who was the Governor RBI at that time. IBPS was envisioned as both an academically and research oriented institute and an operation handling organization, with respect to its mission of optimizing human resource development through personnel assessment. The academics revolve around learning, teaching, conducting research, practicing and publishing. Each has its own importance in the spiral. An ideal situation is Researcher-as-a-Professor and Professor-as-a-Researcher. Additionally, the operations element was included to provide academic grounding and enable direct application of the knowledge gained through research. The operations were also based on the foundation of Psychometrics. The model was Researcher-Professor as Operations Chief.
If an international parallel is to be drawn, IBPS was patterned on ETS – Educational Testing Services, Princeton known as Mecca of testing. The effort was to build and steer an operation based organization with research and academic flavor. It was a fine balancing act in which the operations, academics and research were to support each other to strengthen each component. While academics provided backbone for the content, the action research strengthened operations as well as content by way of validation and demographic data studies. In an organisation like IBPS the research may not be a “typical” hypothesis testing type. It is more of an action research. The objective is to improve the academic and operational processes. This type of research may not earn accolades from academia but adds value and efficiency both to IBPS operations and the beneficiaries of its services. It is rare that when the work prominently is of operations the fulcrum is academic and research oriented. For that reason, the designations and pay scales are commensurate with an academic body rather than a large corporation.
The initial years were of out-sourcing both printing and result-processing work but content development chiefly was in-house. The Psychometrics involved in the processes was also driven internally and rigors of item writing, item review/pre-testing, item analysis after the exam and validation studies etc. were maintained. After some time the limitations of out-sourcing printing and processing through data punching were realized. These pertained to some aspects of Security or Speed or Accuracy or Control or quickly adapting to the unique requirements of the client. During that period OMR technology was also growing and it was felt that scanning and printing should be built as in-house capabilities, given the number of exams and diversity of requirements. Having its own high capacity printing press and 100% in-house high volume scanning capability were the hallmarks that distinguished IBPS from any other such organization in India, be it UPSC, SSC, Railway recruitment Boards, other State level Boards or the private organizations in this business. The ownership of these two essential capabilities enabled the development of many unique processes, in-house software and operational practices to cater to the diverse requirements of IBPS clients. The solution and delivery became an integrated whole rather than sum of different parts.
To bring home the advantages of such integration, I would narrate one experience. IBPS had good association with Prof. John (Jack) E. Hunter of Michigan State University. Jack had exceptional mathematical and quantitative facility. He wrote his own computer programs for analysis and very generously distributed the software. I, being one of the regular users of his software, once shared with him that I found it easier to run Factor Analysis using his Package compared to even SPSS. His matter of fact response was that it was written by a Psychologist who also knew programming and not the other way round. What it meant was that two functions when integrated either at individual level or Institutional level result in a better solution more suited to the unique requirements. Such an integration of purpose underlies the balancing maintained at IBPS between research and operations, as well as solution and delivery.
Technology and IBPS
The last decade has seen the advent of web based technology in testing as an alternative to the conventional paper-and-OMR approach.. There are two major areas in which the paradigm has shifted. One is the application (applying) process of candidates for jobs, including communication with the candidates for exam related information and the other is delivery of exams online. Another very significant growth area, enabled by technology, is the feasibility of Computer Adaptive Testing (CAT) for mass level selection. In fact it is this marvel of psychometrics on the ever improving backbone of technology that is best suited model for large scale online exams.
One area in which IBPS has helped change the rules of the game is online applications. In fact almost 100% recruitment projects of Public Sector Banks (PSBs) are online. The online application has been adopted by several different Banks with IBPS providing customisation suiting to the requirements of the level of jobs, type of recruitment and branch network of the bank. Added to the online advantage is the payment of fee in any branch of the bank rather than purchasing Demand Draft or other such mode of payment. There are many advantages to the applicant in each of these methods. The summary of two major variations is as follows:
Fully online - no print out or document or fee receipt to be sent
In this mode, the candidates go to any designated branch of the bank to pay the fee against payment challan which can be downloaded from Bank’s web-site. Then they fill the application online with payment details and retain a copy of the print-out and challan with them. In this mode the biggest advantage is that it is not affected by postal delay or loss.
Application online but print-out of registered application with DD/payment challan required to be sent
In this mode all the above steps are followed but unless the print-out with DD etc is received at the given post bag number within the stipulated time, the application is not valid.
IBPS will coordinate more than 8.5 million registrations for PSBs in 2009-10. For one clerical project of SBI for 11000 vacancies, about 3.7 million registrations took place in a span of 40 days. Sify, a leading technology provider, partnered with IBPS in handling this project. IBPS also built its in-house capability for hosting this service albeit on a smaller scale to get an insight into this area.
IBPS had also conducted online exams through Sify and other technology partners for volumes upto 2000 candidates in a session. One of the earlier IBPS articles-“Will CAT be CAT?”- had the following opening sentence - “The decision of the Indian Institute of Managements (IIMs) to conduct CAT-2009 online has the potential to change the landscape of testing - if it works well, technological glitches notwithstanding.” I don’t subscribe to the CBT mode in which the exam was conducted for such a big number and when the scheme of equating scores is not shared. However, the general public was more concerned with the technology based delivery rather than more pertinent Psychometric questions. In spite of all the glitches and despite it not being an Adaptive test model, it has paved way for online exams in a big and bold way. Someone had to start at this level with such a big volume. Who else if not IIMs?
Another trend is that more and more technology companies are coming into the sphere of testing. For example, a new company is set up only to provide online testing services and all the capability is generated around that. For such companies the initiation and core competence is technology for delivery of exams and not Psychometrics of testing. This lack of integration, as discussed previously in the context of IBPS, can mean not only a lessened psychometric component but a lessened overall service.
Technology, Testing, and the Future
What would be the enhanced and next version of this technology-enabled online application and testing paradigm? With pressure also from “save the planet” campaign; the shift will be towards a paperless process. This mode will also take away the role of Post Office and will significantly reduce the recruitment cycle to less than half the time it takes now. The major steps and time frame may be:
Apply online after payment of fee in the Bank branch (many other simpler modes of fee payment would evolve): A window of about 20 days is provided for this.
Download call letter within 15 days of closing of application (modes of immediate allotment of test centers would also evolve)
Take the test on-line based on adaptive testing (about 10 days window), commencing from within 7 days of hosting call letter downloads. (The dates would be indicated in the notification itself)
Result within 2 days of closing of online test window.
The whole process is reduced to less than two months as compared to more than five months now.
The basic question is which of the following organizational models fits the bill:
A pure technology company
The traditional organization with outside technology partner(s)
A technology initiated company trying to acquire basic Psychometric inputs
A Psychometric core competence based company evolving as a technology company.
The first two types of organizations would not be able to sustain and cope up with the dynamic situation of ever evolving requirements and diversity. The last two would be better suited. But though technology will bring and enable all these changes, the driving force will be technology driven Psychometrics.
This in any way doesn’t imply just creating one vertical called Technology in the organisation or acquiring a technology company to deliver the online function. Simply acquiring a technology-enabling partner, like ETS did with Prometric, would help adapt better and improve response time. But this in itself is not the integration. A Testing company essentially will have to become a technology company. What this means is that it has to be integrated at individual as well as organizational level. The thinking and process of Technology and Psychometrics will go hand-in-hand with the latter being the initiating force for the new mode.
Testing Organizations 2020
How technology will reshape recruitment testing organisationsVipin K. Chilana 2010-03-03 14:01:55. How technology will reshape ...sify.com/.../how-technology-will-reshape-recruitment-testing-organisations-...
Wednesday, September 2, 2009
Monday, August 31, 2009
Dr Nate's Paper
If you are still interested in more of the same, you might want to read http://www.testpublishers.org/Documents/08-002Thompson-Final.pdf. The drawback to this article is that it is purely psychometric and does not discuss any issues endemic to India.
Thank you and regards,
Nate Thompson
______________________
Nathan A. Thompson, Ph.D. Vice President, Assessment Systems Corporation
Adjunct Faculty, University of CincinnatiPhone: 651-647-9220
Thank you and regards,
Nate Thompson
______________________
Nathan A. Thompson, Ph.D. Vice President, Assessment Systems Corporation
Adjunct Faculty, University of CincinnatiPhone: 651-647-9220
Sunday, August 30, 2009
Links to two articles at sify.com
Will CAT-2009 follow Computer Adaptive Testing model?
2009-08-26 11:58:14
The decision of the Indian Institute of Managements (IIMs) to conduct……
-------------------------------------------------------------------------------------
Applying online for jobs in public sector banks
2009-07-23 13:09:56
Prof. Vipin K. Chilana, Head - Processing & Technology ....
2009-08-26 11:58:14
The decision of the Indian Institute of Managements (IIMs) to conduct……
-------------------------------------------------------------------------------------
Applying online for jobs in public sector banks
2009-07-23 13:09:56
Prof. Vipin K. Chilana, Head - Processing & Technology ....
Dr Nate's 5 Day Workshop on IRT/CAT
From July 29 to Aug 1, 2009, IBPS organized 5 days workshop on IRT/CAT. This is what I wrote to Dr Weiss as a feedback. You can get information about other scheduled workshops at http://www.assess.com/
At 02:05 AM 8/3/2009
Dear Dr Weiss,I provide below part of what I spoke on the concluding session of the 5 day India workshop. A group picture is also attached to test if you can spot me there and see how much has my appearance changed after a decade! It was around 16 July 1999 when I got my first tutorial on IRT and CAT from you in your cabin in the Minnesota University. Dr Nate is bringing with him whole set of pictures taken here and his impressions about India.
At 02:05 AM 8/3/2009
Dear Dr Weiss,I provide below part of what I spoke on the concluding session of the 5 day India workshop. A group picture is also attached to test if you can spot me there and see how much has my appearance changed after a decade! It was around 16 July 1999 when I got my first tutorial on IRT and CAT from you in your cabin in the Minnesota University. Dr Nate is bringing with him whole set of pictures taken here and his impressions about India.
A mother persuades her son to go to school. She says, “Darling, Are you not ready to go to school?” The son says, “Mummy nobody at school likes me.” “But darling, you must go to school”, says the mother. “But the teachers don’t like me at school”, says the son. “But sweetheart, you are 45 years old, and you are the principal of the school. You have to go to school” Well, had I been in Dr Nate’s place for this workshop, my mother would have persuaded me like that from the second day onwards and I would have told her that they have got a bunch of people together who understand the subject somewhat and another bunch who don’t at all. But Dr Nate didn’t need any that kind of prodding from any one. His enthusiasm on the 5th day also didn’t wane and the zeal was the same in the last session as it was in the first. He balanced very well the content for the novice and those who knew about the subject a bit. He handled the question posed at -3 Theta with equal aplomb as that of +3 Theta without losing his cool smile and without letting anyone feel that unrelated or very basic questions are not welcome. I fact when we were planning this program we secretly wished that Dr Weiss himself should be a part of the workshop and had sent him an email to consider our request . The way Dr Nate has conducted this workshop for a divergent group like ours is an indirect compliment to Dr Weiss also as Dr Nate happens to be his student. We wish Dr Weiss and Dr Nate visit India in 2010 and the 2011 GMAC CAT conference is also held in India.
Regards,
Vipin
--------------------------------------
Vipin:
--------------------------------------
Vipin:
Thanks very much for the photo (you're the guy in the middle with the gray beard and the black sleveless vest). I also very much appreciate the comments on Nate's performance. Although he is young, as you indicated he is very knowledgeable as well as very personable. I'm pleased to hear that he did a good job!Where the next CAT conference is held is up to the people who sponsor it (GMAC). However, once the new CAT organization starts functioning, there is always the possibility that they could hold conferences in other locations in the years that GMAC doesn't sponsor a conference.
Dave
**********************************************
David J. Weiss, PresidentAssessment Systems Corporation
2233 University Avenue, Suite 200St. Paul Minnesota 55114, U.S.A.
Email: dweiss@assess.comPhone: 651-647-9220 Fax: 651-647-0412
Worldwide Web: http://www.assess.com/
**********************************************
Subscribe to:
Comments (Atom)