How to Extract Online Data for Research
While a fair bit of data is actually available online, it's not in a machine readable format. The data is either in a .pdf file or inside tables on multiple web pages. Data available in these formats is difficult to analyze as it usually cannot be read by any statistical software.
A large part of our work at the JustJobs Network is to address the challenges arising in research processes, one of which is the lack of relevant, usable data.
The lack of data has been a recurrent problem for me while researching skill development and vocational training in India. This is one reason why we engage in primary data collection on these issues.
While a fair bit of data is actually available online, it’s not in a machine readable format. The data is either in a .pdf file or inside tables on multiple web pages. Data available in these formats is difficult to analyze as it usually cannot be read by any statistical software.
Data presented in .pdf files is particularly challenging to work with. However it is possible to extract data inside tables in .html files, such as the data on Industrial Training Institutes (ITIs) in India.
A useful way of extracting such data is by using a a python based data scraping framework, Scrapy, which can be set up using pip or easy_install. A good introduction to Scrapy can be found here: http://doc.scrapy.org/en/latest/intro/tutorial.html.
In some cases, you will need to interact with the javascript of the page before extracting data. For these situations, Splash – a light-weight browser developed by Scraping Hub – can be used.
Once installed, using Scrapy to extract data is quite easy. First you will need to get the urls for each ITI. This can be done in a few different ways. The easiest way is to use the following url format: https://ncvtmis.gov.in/Pages/ITI/Detail.aspx?code=iti_code
The website also lists the urls in the following format: ncvtmis.gov.in/Pages/ITI/Detail.aspx?ITI=xxxxx , where the url is completed by what looks like a combination of state and district codes, but I found the ITI codes easier to work with.
Once you have the codes, you can use the standard scrapy classes and functions to gather data on each ITI. First make the urls from above, the start_urls, then insert the wait time in the Splash code as a precaution to make sure the .html loads before the scraping begins (this step is optional), and finally insert the xpaths for the data you want to extract in your parse function. You will also need to make the corresponding changes in the items.py file. Now run the spider you have just written and save the data to a JSON file.
Data from the JSON file is easy to extract – just convert it to a Pandas dataframe and then outsheet to a .csv file.
While the process of data extraction using Scrapy may seem a bit complicated, it is still very useful. As demonstrated by the example above, the process allows us to have a dataset of every ITI in India along with data on the courses offered and the enrollment at the course level.
Stay tuned for interesting findings from this data.
The full code can be found here: https://github.com/amanbirs/data-scraping-projects/tree/master/itis_list
You may also be interested in
FutureWORKS Collective: Charting New Frontiers in the Global South
As global labour markets continue to undergo transition induced by rapid technological advancement and climate change, there is not enough research that captures the diverse experiences of the Global South. The FutureWORKS Collective, funded by IDRC Canada, is a consortium of research organisations that will fill this lacuna. The JustJobs Network is proud to be the coordinator of this initiative that underscores the urgency of adapting research and policies to the realities of the Global South.
Bihar—India’s latest urban frontier
Bihar’s unconventional urbanisation is not characterised by the development of industrial hubs that generally characterise the process of urbanisation. Despite being famous for its out-migration, Bihar is experiencing something that the author describes as “urbanisation from within” i.e. the natural growth and densification of small towns and villages, increased population density and a shift from agriculture to non-farm activities. This article points out challenges in this form of urbanisation and suggests changes that will transform these urbanising villages into centers of job-rich economic development.
Towards a GREEN future: Conserving Meghalaya’s forests through PES
A government initiative to promote conservation and build climate resilience, the GREEN Meghalaya scheme is based on the payment for ecosystems model (PES). This article introduces the PES model and poses inquiries about its operation and implementation, especially against the changing landscape of forest conservation in India.
Embracing Transgender Inclusion in the Workplace: A Call for Action and Acceptance
The article highlights challenges transgender individuals encounter in workplaces and society. It emphasises holistic recognition by linking food, shelter, and employment for inclusive efforts in their livelihoods.
Will Gender Quotas in India’s Legislatures Deliver Gender Equality?
The newly enacted law to reserve one-third seats for women in the Lok Sabha and state legislatures could advance gender equality in India. Certain conditions however need to be met to ensure that numerical representation translates to substantive representation.
Time for a National Urban Employment Guarantee Act? Lessons from Odisha’s Urban Wage Employment Initiative
The success of UWEI and MUKTA, Odisha’s initiatives to curb pandemic-induced urban employment could well be the blueprint for a National Urban Employment Guarantee Act that emphasizes worker security, community involvement and a gender-inclusive design.
Constructing within planetary boundaries: Warnings from Mumbai’s coastal road project
The building and construction sector in India is marred with inherent contradictions – it is at once an employment generator and a highly polluting sector. It stresses the question: How do we balance human needs without compromising earth systems?
Big Problems, Small Wins: Social Security for Rajasthan’s Gig Workers
Despite good intentions, much in the new Rajasthan gig workers act is built on weak foundations, which will make it difficult to implement. This makes it unlikely that the workers will actually get what they deserve.
Technology and Work: Pursuing Equitable Distribution Amidst Digitalisation
As technology continues to change the world of work irreversibly, its impact on the workforce and the need for equitable distribution of digital work gains continues to gain emphasis. This document, drafted through the consultations with JJN’s global research consortium highlights the role of public systems, the classification of platform workers, the challenges of collective bargaining and the implications for women in platform work. It outlines that path for further research and policy intervention to support worker well-being and sustainable platform models.
Rajasthan Drafts Bill For Gig Workers: Will It Hold Aggregators Accountable?
On 10 February, the Rajasthan government announced that it would introduce the Rajasthan Platform-based Gig Workers (Registration and Welfare) Bill, 2023. The draft bill, accessed by The Quint, envisages a social security and welfare board for platform-based workers.
How India’s population exploded to overtake China’s and what’s next
So how did India’s population get so big, and how long will it last? Can India's population be a tremendous productive force for the economy?
Gig work must do more than expand female labour force participation
Just bringing more women into the labour market isn’t enough. Whether the work provides pathways for professional growth, the quality and conditions of work are also critical considerations if we want to reap the economic and social benefits of women’s participation in the labour force